![\"\"](\"https://i.ibb.co/KmkbVk1/001-max.jpg\")
FAIR just released ConvNeXt V2 about a week ago. I have done some experiments with it and the CV score is about the same as efficientnet. I will have some LB tests i in the coming days.
\nInstalling it from the official release is a little bit hard since it will do some compilation, luckily timm already has it, just install timm from the source then we can use it.
Another related discussion: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/375791
\npaper: https://arxiv.org/abs/2301.00808
\nofficial source: https://github.com/facebookresearch/ConvNeXt-V2
\ntimm: https://github.com/rwightman/pytorch-image-models/blob/main/timm/models/convnext.py#L580
FAIR just released ConvNeXt V2 about a week ago. I have done some experiments with it and the CV score is about the same as efficientnet. I will have some LB tests i in the coming days.
\nInstalling it from the official release is a little bit hard since it will do some compilation, luckily timm already has it, just install timm from the source then we can use it.
Another related discussion: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/375791
\npaper: https://arxiv.org/abs/2301.00808
\nofficial source: https://github.com/facebookresearch/ConvNeXt-V2
\ntimm: https://github.com/rwightman/pytorch-image-models/blob/main/timm/models/convnext.py#L580
https://github.com/rwightman/pytorch-image-models
\nrwightman retrained convnext v1 and they become as good as v2.
\nv1 can be used for commerical. (but v2 cannot)
i tested some models, they are good
\nconvnext_tiny.in12k_ft_in1k_384 - 85.1 @ 384
\nconvnext_small.in12k_ft_in1k_384 - 86.2 @ 384
I tested convnext_v1 and compared it to maxvit. Still have problem with site=1 score. Probably this is why my single model is unable to jump over 0.5 (have not tried blend so far - still looking for better single model).
\n
![\"\"](\"https://i.ibb.co/WvnBnBk/001-conv.jpg\")
What image size did you use for these archs? I am having a lot of trouble getting a stable training on bigger architectures (especially transformers) even with a 3090 without lowering image size significantly
", "rawMarkdown": "What image size did you use for these archs? I am having a lot of trouble getting a stable training on bigger architectures (especially transformers) even with a 3090 without lowering image size significantly" }, { "id": 2112696, "postDate": "2023-01-23T18:58:53.973Z", "content": "(4 * 384, 2 * 384) - I use A100 (80GB RAM GPU) or A6000 - bs=16 no matter what machine I use
", "rawMarkdown": "(4 * 384, 2 * 384) - I use A100 (80GB RAM GPU) or A6000 - bs=16 no matter what machine I use ", "votes": 2 }, { "id": 2112699, "postDate": "2023-01-23T19:07:07.577Z", "content": "😨😲😲😲😲
", "rawMarkdown": "😨😲😲😲😲" }, { "id": 2112702, "postDate": "2023-01-23T19:11:29.593Z", "content": "But do not worry - I am sure people have less resources and certainly better results 😂
", "rawMarkdown": "But do not worry - I am sure people have less resources and certainly better results 😂", "votes": 1 }, { "id": 2113151, "postDate": "2023-01-24T05:37:18.260Z", "content": "this is for base
\n![\"https://i.ibb.co/wNkZgx4/Selection-673.png\"](\"https://i.ibb.co/wNkZgx4/Selection-673.png\")
\nonly half an epoch
\n(training in progress)
this is for small
\n![\"https://i.ibb.co/sg78K60/Selection-674.png\"](\"https://i.ibb.co/sg78K60/Selection-674.png\")
yes i need to think of how to improve site1 performance
", "rawMarkdown": "this is for base\n\nonly half an epoch\n(training in progress)\n\nthis is for small\n\n\nyes i need to think of how to improve site1 performance", "votes": 1 }, { "id": 2113271, "postDate": "2023-01-24T07:19:11.980Z", "content": "Here is another example:
\n![\"\"](\"https://i.ibb.co/MPbdG4z/s001.jpg\")
site2 results is very good
\ni am trying the following:
\nThank you very much for prompts. I will report changes when they appear.
", "rawMarkdown": "Thank you very much for prompts. I will report changes when they appear. " }, { "id": 2113462, "postDate": "2023-01-24T09:44:54.013Z", "content": "I thnink that model could be good as well for site_1 but …
\n![\"\"](\"https://i.ibb.co/0QmdXgr/si-001.jpg\")
you can do the following:
\nif the case is a, then the best way is replace mean() to learned function to combine results , e.g. attention pooling over multiple images, multiple instance, e.g …
\nif the lesion is actually visible in many views but the score is only one view, you should enforce score consistency. (e.g. consistency loss, feature consistency, or better augmentation to move from one view to another, … mixup, etc)
\nbut you only show results for positive case. you should also analyze results for negative image or false positive. It is not known that poor site 1 results is due to difficult pos images or difficult neg images
", "rawMarkdown": "you can do the following:\n1. show the class activation map for all images of the same one case. There are a few possibility\na. the lower score is due the lesion being visible in the top score view and occluded other low score view\nb. actually the high score is due to \"noise\", e.g letters, corners ... so the results is wrong \n\nif the case is a, then the best way is replace mean() to learned function to combine results , e.g. attention pooling over multiple images, multiple instance, e.g ...\n\nif the lesion is actually visible in many views but the score is only one view, you should enforce score consistency. (e.g. consistency loss, feature consistency, or better augmentation to move from one view to another, ... mixup, etc)\n\n---\n\nbut you only show results for positive case. you should also analyze results for negative image or false positive. It is not known that poor site 1 results is due to difficult pos images or difficult neg images", "votes": 1 }, { "id": 2113491, "postDate": "2023-01-24T10:01:03.973Z", "content": "\"I thnink that model could be good as well for site_1 but \"
\ncheck why site 2 don't have this problem. i think site 2 pos images have higher score (hence averaging, the score is still high) or site 2 has less number of images and hence higher score
\nmaybe we need number aware averaging, aka attention pool …
", "rawMarkdown": "\"I thnink that model could be good as well for site_1 but \"\n\ncheck why site 2 don't have this problem. i think site 2 pos images have higher score (hence averaging, the score is still high) or site 2 has less number of images and hence higher score\n\nmaybe we need number aware averaging, aka attention pool ...", "votes": 1 }, { "id": 2113511, "postDate": "2023-01-24T10:19:12.540Z", "content": "\nmaybe you have discovered some magic
\n![\"https://i.ibb.co/276Wxq9/Selection-676.png\"](\"https://i.ibb.co/276Wxq9/Selection-676.png\")
a quick test of attention pooling
\nhttps://stackoverflow.com/questions/20069009/pandas-get-topmost-n-records-within-each-group
\n # group by mean of top3\n gb = site_df[['site_id', 'patient_id', 'laterality', 'cancer_t', 'cancer_p']].groupby(\n ['patient_id', 'laterality']).head(3).reset_index(drop=True)\n gb = gb[['site_id', 'patient_id', 'laterality', 'cancer_t', 'cancer_p']].groupby(\n ['patient_id', 'laterality']).mean()\n\n gb.loc[:, 'cancer_t'] = gb.cancer_t.astype(int)\n m = compute_metric(gb.cancer_p, gb.cancer_t)\n text = f'{\"top3 mean()\": <16} [{site_id}]'\n text += f'\\t{m[\"auc\"]:0.5f}'\nmean is wrong!!!!
\nmean = numerator/ denominator = sum/num of images
the more images, the greater is the denominator
\nthe better score for occluded lesion (i.e. should have more negative score), the smaller is the numerator
d1 = train_df[train_df.site_id == 1]\n d2 = train_df[train_df.site_id == 2]\n\n gb = d1.groupby(['patient_id', 'laterality'])['image_id'].count()\n gb.value_counts()\n gb = d2.groupby(['patient_id', 'laterality'])['image_id'].count()\n gb.value_counts()\n\n\n '''\n site1\n 2 6990\n 3 3442\n 4 893\n 5 239\n 6 60\n 7 10\n 8 2\n\n\n site2\n 2 11394\n 3 785\n 4 11\n\n '''\nyes, exactly … in my tests in case of more images per patient I tried to take two largest proba but did not work.
\ndf.groupby(['patient_id','laterality']).apply(lambda x: x.nlargest(2,'cancer_p'))\nlargest 2 does not work for me.
\ni need 3.
but the correct method is to learn a pooling function.
\nnow i am doing experiments and searching for paper.
i think there are some kaggle competition that try to classify product for e-commerce based on a set of images (rather than single image)
", "rawMarkdown": "largest 2 does not work for me.\ni need 3.\n\nbut the correct method is to learn a pooling function.\nnow i am doing experiments and searching for paper.\n\ni think there are some kaggle competition that try to classify product for e-commerce based on a set of images (rather than single image)", "votes": 2 }, { "id": 2113638, "postDate": "2023-01-24T11:33:40.533Z", "content": "For me it works as well. Some small gain.
\n![\"\"](\"https://i.ibb.co/2FDq7W2/si009.jpg\")
\"some small gain.\"
\nthis is true gain (both recall and precision improve). i am making a submission now and have to wait for results.
\nif LB also improves, it would means that not all images are useful for site1. correctly removing some images may improve results or we should train as bags?
\nyou can submit single site only to see if LB score for site2 is better than site1.
\nand also if the instability in LB is due to site1
I submitted with top3 and waiting for results.
\nRest tests in progress - let you know if find something.
BTW: this model (single model) gave me 0.51 LB (without top3)
", "rawMarkdown": "I submitted with top3 and waiting for results. \nRest tests in progress - let you know if find something.\n\nBTW: this model (single model) gave me 0.51 LB (without top3)" }, { "id": 2114328, "postDate": "2023-01-24T22:09:34.157Z", "content": "@hengck23 - for me head(3) on LB did not increase score. For sure thinking about multiple images per side should be investigated.
", "rawMarkdown": "@hengck23 - for me head(3) on LB did not increase score. For sure thinking about multiple images per side should be investigated.", "votes": 1 }, { "id": 2114386, "postDate": "2023-01-25T00:13:12.733Z", "content": "for me , it results in the same score
\nthis is a possibility that the public test has more site2 images that is why LB score is (much) higher than CV?
\nmaybe some kaggler will probe and report?
", "rawMarkdown": "for me , it results in the same score\n\nthis is a possibility that the public test has more site2 images that is why LB score is (much) higher than CV?\n\nmaybe some kaggler will probe and report?", "votes": 1 }, { "id": 2114650, "postDate": "2023-01-25T06:34:43.840Z", "content": "Look here: https://www.kaggle.com/code/tomooinubushi/some-lb-probing-results-to-share/notebook
\n\n\nThere are no new site ID in test dataset.
\n
\n No. of images in site ID 1 > No. of images in site ID 2. (by @yujiariyasu)
Most of my false alarms and misses come from machine_id = 49
\n\n", "rawMarkdown": "Look here: https://www.kaggle.com/code/tomooinubushi/some-lb-probing-results-to-share/notebook\n\n>There are no new site ID in test dataset. \n>No. of images in site ID 1 > No. of images in site ID 2. (by @yujiariyasu)\n\nMost of my false alarms and misses come from machine_id = 49\n\n>ore than 40% of images are from machine ID 49 (43% for train dataset) (by @kaggleqrdl)" }, { "id": 2120468, "postDate": "2023-01-29T16:08:29.167Z", "content": "\nore than 40% of images are from machine ID 49 (43% for train dataset) (by @kaggleqrdl)
\n
here is a good method to do multiple image prediction.
\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370333#2120459
Thank you! I am reading.
\nI still working using single model (no blend) - now it is LB 0.57 but today made some changes (I had some bug in inference part) and I think in two days should be over 0.6 (I hope).
\nI am afraid … shakeup in this competition is really possible. Score is very sensitive on th. Still looking for some postprocessing ideas.
\nNow changes I made:
\n\"shakeup in this competition is really possible.\"
\nthat is very true, but you should be able to compute the range of shakeup
\"Still looking for some postprocessing ideas.\"
\nspend some time to think how to use probing to produce a more stable results too.
\n(there aren't many +ve cases in the hidden test and i think the top kagglers have a way to make the best guess)
ignore the issue of license aside, i think ConvNeXt-V2 is ggod.
\nMy training is still in porgress but training/validation log looks good.
\nI am working on ConvNeXt-V2-small
it seems that large model (imagenet top-1 0.86 and above) of vision transformer and 2nd geration of cnn (like ConvNeXt-V2) uses low initial learning rate and just need few epoches. This is very much like transfer learning of large language model BERT given a few ten thousands of text train samples.
", "rawMarkdown": "ignore the issue of license aside, i think ConvNeXt-V2 is ggod.\nMy training is still in porgress but training/validation log looks good.\nI am working on ConvNeXt-V2-small \n\nit seems that large model (imagenet top-1 0.86 and above) of vision transformer and 2nd geration of cnn (like ConvNeXt-V2) uses low initial learning rate and just need few epoches. This is very much like transfer learning of large language model BERT given a few ten thousands of text train samples.\n\n ", "votes": 4, "replies": [ { "id": 2102261, "postDate": "2023-01-16T13:45:51.303Z", "content": "I have the same observation with the low initial learning rate.
", "rawMarkdown": "I have the same observation with the low initial learning rate.", "replies": [ { "id": 2102272, "postDate": "2023-01-16T13:53:09.553Z", "content": "so the good results come from low learning rate of transfer learning.
\nthis is the same case for NextVIT.
but i am surprise that imagenet/natural image can represent mammography images.
\ni though this can be a new research area: https://hirokatsukataoka16.github.io/Pretraining-without-Natural-Images/
\nsome synthetic images gneration for medical images if you understand how the medical devices woprk
any kaggler want to try transfer learning for giant VIT:
\nhttps://github.com/rwightman/pytorch-image-models/blob/main/results/results-imagenet.csv
\neva_giant_patch14_560.m30m_ft_in22k_in1k
\ntop1=89.796
\nparams=1,014.45
thanks! the paper seems to suggest it is on par with vision trasnformer
\nwhich of the variant did you try?
\ne.g. huge, base, tiny …
i am more concern about the speed. what is the speed of ConvNeXt V2 compared to efficientnet?
\n(i got a few good results with vision transformer, but their speed are slower than efficient especially for large ersolution beyond 1024)
e.g. 1x single fold transformer nextVIT-B for 1536x960 has LB 0.56 but takes 9hrs (local cv 0.50, imagenet top1 83.2 at 224x224).it uses up all my time.
", "rawMarkdown": "thanks! the paper seems to suggest it is on par with vision trasnformer\n\nwhich of the variant did you try?\ne.g. huge, base, tiny ...\n\ni am more concern about the speed. what is the speed of ConvNeXt V2 compared to efficientnet?\n(i got a few good results with vision transformer, but their speed are slower than efficient especially for large ersolution beyond 1024)\n\ne.g. 1x single fold transformer nextVIT-B for 1536x960 has LB 0.56 but takes 9hrs (local cv 0.50, imagenet top1 83.2 at 224x224).it uses up all my time.", "votes": 4, "replies": [ { "id": 2096732, "postDate": "2023-01-12T08:25:52.150Z", "content": "I used nano and 1024 input, it's faster than efficientnetv2_s, which is the main model I've been used so far. And CV score is 0.05 better than efficientnetv2_s with the same input.
\nI'll have more experiments on this and update the results.
", "rawMarkdown": "I used nano and 1024 input, it's faster than efficientnetv2_s, which is the main model I've been used so far. And CV score is 0.05 better than efficientnetv2_s with the same input.\n\nI'll have more experiments on this and update the results.", "votes": 5, "replies": [ { "id": 2096764, "postDate": "2023-01-12T08:48:06.080Z", "content": "\"I used nano and 1024 input\"
\nthanks!
\nit is interesting that i never get good results for 1024.
\n1024 never give LB greater than 0.50 for me (for small or big models)
\nthis is different from other kagglers.
maybe i need to think more about it
", "rawMarkdown": "\"I used nano and 1024 input\"\n\nthanks! \nit is interesting that i never get good results for 1024. \n1024 never give LB greater than 0.50 for me (for small or big models)\nthis is different from other kagglers.\n\nmaybe i need to think more about it\n\n" } ] }, { "id": 2096758, "postDate": "2023-01-12T08:39:42.617Z", "content": "My experience with ConvNext has been very bad in the past in terms of speed, especially inference. Maybe there have been some improvements recently, interesting that nano is faster for you there.
", "rawMarkdown": "My experience with ConvNext has been very bad in the past in terms of speed, especially inference. Maybe there have been some improvements recently, interesting that nano is faster for you there.", "votes": 4 }, { "id": 2097854, "postDate": "2023-01-13T02:08:24.750Z", "content": "@hengck23 Did you use something similar to this paper (https://arxiv.org/pdf/2111.11429.pdf) aka windowed attention + few global attention blocks ? Otherwise, I guess it's impossible to train and infer ViT-B at 1536x960.
", "rawMarkdown": "@hengck23 Did you use something similar to this paper (https://arxiv.org/pdf/2111.11429.pdf) aka windowed attention + few global attention blocks ? Otherwise, I guess it's impossible to train and infer ViT-B at 1536x960.", "replies": [ { "id": 2097900, "postDate": "2023-01-13T03:36:08.590Z", "content": "Nope. I was using Z8 by HP workstation with 2x A6000 48GB Nvidia GPU cards
", "rawMarkdown": "Nope. I was using Z8 by HP workstation with 2x A6000 48GB Nvidia GPU cards", "votes": 2 }, { "id": 2097901, "postDate": "2023-01-13T03:38:27.170Z", "content": "maybe you can try gradient checkpointing … a bit slower
", "rawMarkdown": "maybe you can try gradient checkpointing ... a bit slower", "votes": 1 } ] } ] }, { "id": 2096756, "postDate": "2023-01-12T08:38:34.373Z", "content": "Was already discussed here: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/375791
\nAnd keep an eye out on the license (it has same license in timm)
", "rawMarkdown": "Was already discussed here: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/375791\n\nAnd keep an eye out on the license (it has same license in timm)", "votes": 2, "replies": [ { "id": 2096761, "postDate": "2023-01-12T08:44:03.377Z", "content": "Oh I missed that, I'll add the link in the post. Good to know the license issue.
", "rawMarkdown": "Oh I missed that, I'll add the link in the post. Good to know the license issue.", "replies": [ { "id": 2096798, "postDate": "2023-01-12T09:11:06.633Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2096799, "postDate": "2023-01-12T09:11:21.573Z", "content": "so according to the license, can we use convnextv2 at last?
", "rawMarkdown": "so according to the license, can we use convnextv2 at last?" } ] }, { "id": 2121220, "postDate": "2023-01-30T06:15:01.150Z", "content": "I switched to ConvNextv2, but it seems to be slower than eff-net-v2.
", "rawMarkdown": "I switched to ConvNextv2, but it seems to be slower than eff-net-v2.\n" }, { "id": 2099747, "postDate": "2023-01-14T17:13:51.200Z", "content": "Any news about LB score?
\nThank you for sharing this BTW
\nThe Devastator.
", "rawMarkdown": "Any news about LB score?\n\nThank you for sharing this BTW\n\nThe Devastator.\n", "replies": [ { "id": 2101485, "postDate": "2023-01-16T00:25:17.830Z", "content": "It's not stable, but I already saw 0.01 boost on LB than efficientnetv2_s
My mistake, the differences are not coming from the version of timm.
this is a common fact (for efficientnet)
\nyou need to retrain or finetune with the new version timm.
alternatively you can:
\nor
\ndef function_one():\n import xxx #old version\n\ndef function_two():\n import xxx #new version\nyou your notebook import use_old_timm.py and use_new_timm.py
\nor you can use torch jit to save the model and load (the model definition is saved in jit model)
", "rawMarkdown": "this is a common fact (for efficientnet)\nyou need to retrain or finetune with the new version timm.\n\nalternatively you can:\n1. create one py file call use_old_timm.py. import the the old timm and define your efficientnet model there\n2. create another py file call use_new_timm.py. import the new timm here.\n\n---\n\nor\n\n```\ndef function_one():\n import xxx #old version\n\ndef function_two():\n import xxx #new version\n```\n\nyou your notebook import use_old_timm.py and use_new_timm.py\n\n---\n\nor you can use torch jit to save the model and load (the model definition is saved in jit model)\n", "votes": 1, "replies": [ { "id": 2098454, "postDate": "2023-01-13T15:17:49.020Z", "content": "Thanks @hengck23 , haven't been thought about using jit but it's truly a brilliant way to handle the version issue here.
why I pip install the --pre timm, using the create_model(), error that \"the model name\" is not found?
\nthough they are listed:
\n 'convnextv2_atto.fcmae',
\n 'convnextv2_atto.fcmae_ft_in1k',
\n 'convnextv2_base.fcmae',
\n 'convnextv2_base.fcmae_ft_in1k',
\n 'convnextv2_base.fcmae_ft_in22k_in1k',
\n 'convnextv2_base.fcmae_ft_in22k_in1k_384',
\n 'convnextv2_femto.fcmae',
\n 'convnextv2_femto.fcmae_ft_in1k',
\n 'convnextv2_huge.fcmae',
\n 'convnextv2_huge.fcmae_ft_in1k',
\n 'convnextv2_huge.fcmae_ft_in22k_in1k_384',
\n 'convnextv2_huge.fcmae_ft_in22k_in1k_512',
\n 'convnextv2_large.fcmae',
\n 'convnextv2_large.fcmae_ft_in1k',
\n 'convnextv2_large.fcmae_ft_in22k_in1k',
\n 'convnextv2_large.fcmae_ft_in22k_in1k_384',
\n 'convnextv2_nano.fcmae',
\n 'convnextv2_nano.fcmae_ft_in1k',
\n 'convnextv2_nano.fcmae_ft_in22k_in1k',
\n 'convnextv2_nano.fcmae_ft_in22k_in1k_384',
\n 'convnextv2_pico.fcmae',
\n 'convnextv2_pico.fcmae_ft_in1k',
\n 'convnextv2_tiny.fcmae',
\n 'convnextv2_tiny.fcmae_ft_in1k',
\n 'convnextv2_tiny.fcmae_ft_in22k_in1k',
\n 'convnextv2_tiny.fcmae_ft_in22k_in1k_384',
this worked pip install git+https://github.com/rwightman/pytorch-image-models.git
Thanks. I may try it later.
", "rawMarkdown": "Thanks. I may try it later.", "votes": 1 } ] } ] }, { "id": 2099204, "postDate": "2023-01-14T08:47:26.767Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 2112342, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-23T14:59:33.367000", "content": "https://github.com/rwightman/pytorch-image-models
\nrwightman retrained convnext v1 and they become as good as v2.
\nv1 can be used for commerical. (but v2 cannot)
i tested some models, they are good
\nconvnext_tiny.in12k_ft_in1k_384 - 85.1 @ 384
\nconvnext_small.in12k_ft_in1k_384 - 86.2 @ 384
I tested convnext_v1 and compared it to maxvit. Still have problem with site=1 score. Probably this is why my single model is unable to jump over 0.5 (have not tried blend so far - still looking for better single model).
\n
What image size did you use for these archs? I am having a lot of trouble getting a stable training on bigger architectures (especially transformers) even with a 3090 without lowering image size significantly
", "votes": 0, "replies": [] }, { "id": 2112696, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-23T18:58:53.973000", "content": "(4 * 384, 2 * 384) - I use A100 (80GB RAM GPU) or A6000 - bs=16 no matter what machine I use
", "votes": 2, "replies": [] }, { "id": 2112699, "author_name": "outwrest", "author_url": "", "post_date": "2023-01-23T19:07:07.577000", "content": "😨😲😲😲😲
", "votes": 0, "replies": [] }, { "id": 2112702, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-23T19:11:29.593000", "content": "But do not worry - I am sure people have less resources and certainly better results 😂
", "votes": 1, "replies": [] }, { "id": 2113151, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-24T05:37:18.260000", "content": "this is for base
\n
\nonly half an epoch
\n(training in progress)
this is for small
\n
yes i need to think of how to improve site1 performance
", "votes": 1, "replies": [] }, { "id": 2113271, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-24T07:19:11.980000", "content": "Here is another example:
\n
site2 results is very good
\ni am trying the following:
\nThank you very much for prompts. I will report changes when they appear.
", "votes": 0, "replies": [] }, { "id": 2113462, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-24T09:44:54.013000", "content": "I thnink that model could be good as well for site_1 but …
\n
you can do the following:
\nif the case is a, then the best way is replace mean() to learned function to combine results , e.g. attention pooling over multiple images, multiple instance, e.g …
\nif the lesion is actually visible in many views but the score is only one view, you should enforce score consistency. (e.g. consistency loss, feature consistency, or better augmentation to move from one view to another, … mixup, etc)
\nbut you only show results for positive case. you should also analyze results for negative image or false positive. It is not known that poor site 1 results is due to difficult pos images or difficult neg images
", "votes": 1, "replies": [] }, { "id": 2113491, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-24T10:01:03.973000", "content": "\"I thnink that model could be good as well for site_1 but \"
\ncheck why site 2 don't have this problem. i think site 2 pos images have higher score (hence averaging, the score is still high) or site 2 has less number of images and hence higher score
\nmaybe we need number aware averaging, aka attention pool …
", "votes": 1, "replies": [] }, { "id": 2113511, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-24T10:19:12.540000", "content": "\nmaybe you have discovered some magic
\n
a quick test of attention pooling
\nhttps://stackoverflow.com/questions/20069009/pandas-get-topmost-n-records-within-each-group
\n # group by mean of top3\n gb = site_df[['site_id', 'patient_id', 'laterality', 'cancer_t', 'cancer_p']].groupby(\n ['patient_id', 'laterality']).head(3).reset_index(drop=True)\n gb = gb[['site_id', 'patient_id', 'laterality', 'cancer_t', 'cancer_p']].groupby(\n ['patient_id', 'laterality']).mean()\n\n gb.loc[:, 'cancer_t'] = gb.cancer_t.astype(int)\n m = compute_metric(gb.cancer_p, gb.cancer_t)\n text = f'{\"top3 mean()\": <16} [{site_id}]'\n text += f'\\t{m[\"auc\"]:0.5f}'\nmean is wrong!!!!
\nmean = numerator/ denominator = sum/num of images
the more images, the greater is the denominator
\nthe better score for occluded lesion (i.e. should have more negative score), the smaller is the numerator
d1 = train_df[train_df.site_id == 1]\n d2 = train_df[train_df.site_id == 2]\n\n gb = d1.groupby(['patient_id', 'laterality'])['image_id'].count()\n gb.value_counts()\n gb = d2.groupby(['patient_id', 'laterality'])['image_id'].count()\n gb.value_counts()\n\n\n '''\n site1\n 2 6990\n 3 3442\n 4 893\n 5 239\n 6 60\n 7 10\n 8 2\n\n\n site2\n 2 11394\n 3 785\n 4 11\n\n '''\nyes, exactly … in my tests in case of more images per patient I tried to take two largest proba but did not work.
\ndf.groupby(['patient_id','laterality']).apply(lambda x: x.nlargest(2,'cancer_p'))\nlargest 2 does not work for me.
\ni need 3.
but the correct method is to learn a pooling function.
\nnow i am doing experiments and searching for paper.
i think there are some kaggle competition that try to classify product for e-commerce based on a set of images (rather than single image)
", "votes": 2, "replies": [] }, { "id": 2113638, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-24T11:33:40.533000", "content": "For me it works as well. Some small gain.
\n
\"some small gain.\"
\nthis is true gain (both recall and precision improve). i am making a submission now and have to wait for results.
\nif LB also improves, it would means that not all images are useful for site1. correctly removing some images may improve results or we should train as bags?
\nyou can submit single site only to see if LB score for site2 is better than site1.
\nand also if the instability in LB is due to site1
I submitted with top3 and waiting for results.
\nRest tests in progress - let you know if find something.
BTW: this model (single model) gave me 0.51 LB (without top3)
", "votes": 0, "replies": [] }, { "id": 2114328, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-24T22:09:34.157000", "content": "@hengck23 - for me head(3) on LB did not increase score. For sure thinking about multiple images per side should be investigated.
", "votes": 1, "replies": [] }, { "id": 2114386, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-25T00:13:12.733000", "content": "for me , it results in the same score
\nthis is a possibility that the public test has more site2 images that is why LB score is (much) higher than CV?
\nmaybe some kaggler will probe and report?
", "votes": 1, "replies": [] }, { "id": 2114650, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-25T06:34:43.840000", "content": "Look here: https://www.kaggle.com/code/tomooinubushi/some-lb-probing-results-to-share/notebook
\n\n\nThere are no new site ID in test dataset.
\n
\n No. of images in site ID 1 > No. of images in site ID 2. (by @yujiariyasu)
Most of my false alarms and misses come from machine_id = 49
\n\n", "votes": 0, "replies": [] }, { "id": 2120468, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-29T16:08:29.167000", "content": "\nore than 40% of images are from machine ID 49 (43% for train dataset) (by @kaggleqrdl)
\n
here is a good method to do multiple image prediction.
\nhttps://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370333#2120459
Thank you! I am reading.
\nI still working using single model (no blend) - now it is LB 0.57 but today made some changes (I had some bug in inference part) and I think in two days should be over 0.6 (I hope).
\nI am afraid … shakeup in this competition is really possible. Score is very sensitive on th. Still looking for some postprocessing ideas.
\nNow changes I made:
\n\"shakeup in this competition is really possible.\"
\nthat is very true, but you should be able to compute the range of shakeup
\"Still looking for some postprocessing ideas.\"
\nspend some time to think how to use probing to produce a more stable results too.
\n(there aren't many +ve cases in the hidden test and i think the top kagglers have a way to make the best guess)
ignore the issue of license aside, i think ConvNeXt-V2 is ggod.
\nMy training is still in porgress but training/validation log looks good.
\nI am working on ConvNeXt-V2-small
it seems that large model (imagenet top-1 0.86 and above) of vision transformer and 2nd geration of cnn (like ConvNeXt-V2) uses low initial learning rate and just need few epoches. This is very much like transfer learning of large language model BERT given a few ten thousands of text train samples.
", "votes": 4, "replies": [ { "id": 2102261, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-16T13:45:51.303000", "content": "I have the same observation with the low initial learning rate.
", "votes": 0, "replies": [ { "id": 2102272, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-16T13:53:09.553000", "content": "so the good results come from low learning rate of transfer learning.
\nthis is the same case for NextVIT.
but i am surprise that imagenet/natural image can represent mammography images.
\ni though this can be a new research area: https://hirokatsukataoka16.github.io/Pretraining-without-Natural-Images/
\nsome synthetic images gneration for medical images if you understand how the medical devices woprk
any kaggler want to try transfer learning for giant VIT:
\nhttps://github.com/rwightman/pytorch-image-models/blob/main/results/results-imagenet.csv
\neva_giant_patch14_560.m30m_ft_in22k_in1k
\ntop1=89.796
\nparams=1,014.45
thanks! the paper seems to suggest it is on par with vision trasnformer
\nwhich of the variant did you try?
\ne.g. huge, base, tiny …
i am more concern about the speed. what is the speed of ConvNeXt V2 compared to efficientnet?
\n(i got a few good results with vision transformer, but their speed are slower than efficient especially for large ersolution beyond 1024)
e.g. 1x single fold transformer nextVIT-B for 1536x960 has LB 0.56 but takes 9hrs (local cv 0.50, imagenet top1 83.2 at 224x224).it uses up all my time.
", "votes": 4, "replies": [ { "id": 2096732, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-12T08:25:52.150000", "content": "I used nano and 1024 input, it's faster than efficientnetv2_s, which is the main model I've been used so far. And CV score is 0.05 better than efficientnetv2_s with the same input.
\nI'll have more experiments on this and update the results.
", "votes": 5, "replies": [ { "id": 2096764, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-12T08:48:06.080000", "content": "\"I used nano and 1024 input\"
\nthanks!
\nit is interesting that i never get good results for 1024.
\n1024 never give LB greater than 0.50 for me (for small or big models)
\nthis is different from other kagglers.
maybe i need to think more about it
", "votes": 0, "replies": [] } ] }, { "id": 2096758, "author_name": "Psi", "author_url": "", "post_date": "2023-01-12T08:39:42.617000", "content": "My experience with ConvNext has been very bad in the past in terms of speed, especially inference. Maybe there have been some improvements recently, interesting that nano is faster for you there.
", "votes": 4, "replies": [] }, { "id": 2097854, "author_name": "NguyenThanhNhan", "author_url": "", "post_date": "2023-01-13T02:08:24.750000", "content": "@hengck23 Did you use something similar to this paper (https://arxiv.org/pdf/2111.11429.pdf) aka windowed attention + few global attention blocks ? Otherwise, I guess it's impossible to train and infer ViT-B at 1536x960.
", "votes": 0, "replies": [ { "id": 2097900, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-13T03:36:08.590000", "content": "Nope. I was using Z8 by HP workstation with 2x A6000 48GB Nvidia GPU cards
", "votes": 2, "replies": [] }, { "id": 2097901, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-13T03:38:27.170000", "content": "maybe you can try gradient checkpointing … a bit slower
", "votes": 1, "replies": [] } ] } ] }, { "id": 2096756, "author_name": "Psi", "author_url": "", "post_date": "2023-01-12T08:38:34.373000", "content": "Was already discussed here: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/375791
\nAnd keep an eye out on the license (it has same license in timm)
", "votes": 2, "replies": [ { "id": 2096761, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-12T08:44:03.377000", "content": "Oh I missed that, I'll add the link in the post. Good to know the license issue.
", "votes": 0, "replies": [ { "id": 2096798, "author_name": "", "author_url": "", "post_date": "2023-01-12T09:11:06.633000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2096799, "author_name": "Mr.Fire", "author_url": "", "post_date": "2023-01-12T09:11:21.573000", "content": "so according to the license, can we use convnextv2 at last?
", "votes": 0, "replies": [] } ] }, { "id": 2121220, "author_name": "ECO", "author_url": "", "post_date": "2023-01-30T06:15:01.150000", "content": "I switched to ConvNextv2, but it seems to be slower than eff-net-v2.
", "votes": 0, "replies": [] }, { "id": 2099747, "author_name": "The Devastator", "author_url": "", "post_date": "2023-01-14T17:13:51.200000", "content": "Any news about LB score?
\nThank you for sharing this BTW
\nThe Devastator.
", "votes": 0, "replies": [ { "id": 2101485, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-16T00:25:17.830000", "content": "It's not stable, but I already saw 0.01 boost on LB than efficientnetv2_s
My mistake, the differences are not coming from the version of timm.
this is a common fact (for efficientnet)
\nyou need to retrain or finetune with the new version timm.
alternatively you can:
\nor
\ndef function_one():\n import xxx #old version\n\ndef function_two():\n import xxx #new version\nyou your notebook import use_old_timm.py and use_new_timm.py
\nor you can use torch jit to save the model and load (the model definition is saved in jit model)
", "votes": 1, "replies": [ { "id": 2098454, "author_name": "Chenglu", "author_url": "", "post_date": "2023-01-13T15:17:49.020000", "content": "Thanks @hengck23 , haven't been thought about using jit but it's truly a brilliant way to handle the version issue here.
why I pip install the --pre timm, using the create_model(), error that \"the model name\" is not found?
\nthough they are listed:
\n 'convnextv2_atto.fcmae',
\n 'convnextv2_atto.fcmae_ft_in1k',
\n 'convnextv2_base.fcmae',
\n 'convnextv2_base.fcmae_ft_in1k',
\n 'convnextv2_base.fcmae_ft_in22k_in1k',
\n 'convnextv2_base.fcmae_ft_in22k_in1k_384',
\n 'convnextv2_femto.fcmae',
\n 'convnextv2_femto.fcmae_ft_in1k',
\n 'convnextv2_huge.fcmae',
\n 'convnextv2_huge.fcmae_ft_in1k',
\n 'convnextv2_huge.fcmae_ft_in22k_in1k_384',
\n 'convnextv2_huge.fcmae_ft_in22k_in1k_512',
\n 'convnextv2_large.fcmae',
\n 'convnextv2_large.fcmae_ft_in1k',
\n 'convnextv2_large.fcmae_ft_in22k_in1k',
\n 'convnextv2_large.fcmae_ft_in22k_in1k_384',
\n 'convnextv2_nano.fcmae',
\n 'convnextv2_nano.fcmae_ft_in1k',
\n 'convnextv2_nano.fcmae_ft_in22k_in1k',
\n 'convnextv2_nano.fcmae_ft_in22k_in1k_384',
\n 'convnextv2_pico.fcmae',
\n 'convnextv2_pico.fcmae_ft_in1k',
\n 'convnextv2_tiny.fcmae',
\n 'convnextv2_tiny.fcmae_ft_in1k',
\n 'convnextv2_tiny.fcmae_ft_in22k_in1k',
\n 'convnextv2_tiny.fcmae_ft_in22k_in1k_384',
this worked pip install git+https://github.com/rwightman/pytorch-image-models.git
Thanks. I may try it later.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2099204, "author_name": "", "author_url": "", "post_date": "2023-01-14T08:47:26.767000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "2096684": "FAIR just released ConvNeXt V2 about a week ago. I have done some experiments with it and the CV score is about the same as efficientnet. I will have some LB tests i in the coming days.\n\nInstalling it from the official release is a little bit hard since it will do some compilation, luckily `timm` already has it, just install `timm` from the source then we can use it.\n\nAnother related discussion: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/375791\n\npaper: https://arxiv.org/abs/2301.00808\nofficial source: https://github.com/facebookresearch/ConvNeXt-V2\ntimm: https://github.com/rwightman/pytorch-image-models/blob/main/timm/models/convnext.py#L580", "2112342": "https://github.com/rwightman/pytorch-image-models\n\nrwightman retrained convnext v1 and they become as good as v2.\nv1 can be used for commerical. (but v2 cannot)\n\ni tested some models, they are good\nconvnext_tiny.in12k_ft_in1k_384 - 85.1 @ 384\nconvnext_small.in12k_ft_in1k_384 - 86.2 @ 384", "2102253": "ignore the issue of license aside, i think ConvNeXt-V2 is ggod.\nMy training is still in porgress but training/validation log looks good.\nI am working on ConvNeXt-V2-small \n\nit seems that large model (imagenet top-1 0.86 and above) of vision transformer and 2nd geration of cnn (like ConvNeXt-V2) uses low initial learning rate and just need few epoches. This is very much like transfer learning of large language model BERT given a few ten thousands of text train samples.\n\n ", "2096691": "thanks! the paper seems to suggest it is on par with vision trasnformer\n\nwhich of the variant did you try?\ne.g. huge, base, tiny ...\n\ni am more concern about the speed. what is the speed of ConvNeXt V2 compared to efficientnet?\n(i got a few good results with vision transformer, but their speed are slower than efficient especially for large ersolution beyond 1024)\n\ne.g. 1x single fold transformer nextVIT-B for 1536x960 has LB 0.56 but takes 9hrs (local cv 0.50, imagenet top1 83.2 at 224x224).it uses up all my time.", "2096756": "Was already discussed here: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/375791\n\nAnd keep an eye out on the license (it has same license in timm)", "2121220": "I switched to ConvNextv2, but it seems to be slower than eff-net-v2.\n", "2099747": "Any news about LB score?\n\nThank you for sharing this BTW\n\nThe Devastator.\n", "2098389": "~~Be careful that after installing `timm` from source (`0.8.6.dev` version), my efficientnet can not reproduce the previous experiments with `0.6.12` version, CV fp1 drops a little bit.~~\n\n~~There has been a lot of changes between the two version and I'm not sure why is this hapenning.~~\n\nMy mistake, the differences are not coming from the version of `timm`.", "2099204": "" } }