\n
![\"https://i.imgur.com/EHAeCy9.png\"](\"https://i.imgur.com/EHAeCy9.png\")
Hey there!
\nUnfortunately I'm stuck with my work at a pf1 score of 0.04 - 0.05, although I think I did everything I should do with the problem.
\nI'm using fastai. I'm using grouped stratified kfold (5 splits) for my folds. I'm training on 512x512 images, upsampling the positive class 3x, training an efficientnet_b2 with weighted BCELoss (positive_weight=20) and a learning rate of 1e-3 - 1e-5 (tried everything).
\nThat's exactly what e.g. Radek did in his notebook or what others have also reported.
\nFrom what I read I should be able to get an thresholded pf1 score of around 0.2 with 512x512 images, but I'm still stuck at 0.05, with thresholding maybe 0.07, but not more.
\nDo you have any tips for me?
\nThanks in advance
", "messages": [ { "id": 2081670, "postDate": "2022-12-31T13:46:03.920Z", "content": "Hey there!
\nUnfortunately I'm stuck with my work at a pf1 score of 0.04 - 0.05, although I think I did everything I should do with the problem.
\nI'm using fastai. I'm using grouped stratified kfold (5 splits) for my folds. I'm training on 512x512 images, upsampling the positive class 3x, training an efficientnet_b2 with weighted BCELoss (positive_weight=20) and a learning rate of 1e-3 - 1e-5 (tried everything).
\nThat's exactly what e.g. Radek did in his notebook or what others have also reported.
\nFrom what I read I should be able to get an thresholded pf1 score of around 0.2 with 512x512 images, but I'm still stuck at 0.05, with thresholding maybe 0.07, but not more.
\nDo you have any tips for me?
\nThanks in advance
", "rawMarkdown": "Hey there!\n\nUnfortunately I'm stuck with my work at a pf1 score of 0.04 - 0.05, although I think I did everything I should do with the problem. \n\nI'm using fastai. I'm using grouped stratified kfold (5 splits) for my folds. I'm training on 512x512 images, upsampling the positive class 3x, training an efficientnet_b2 with weighted BCELoss (positive_weight=20) and a learning rate of 1e-3 - 1e-5 (tried everything). \n\nThat's exactly what e.g. Radek did in his notebook or what others have also reported.\n\nFrom what I read I should be able to get an thresholded pf1 score of around 0.2 with 512x512 images, but I'm still stuck at 0.05, with thresholding maybe 0.07, but not more.\n\nDo you have any tips for me?\n\nThanks in advance", "votes": 11 }, { "id": 2081921, "postDate": "2022-12-31T20:24:19.920Z", "content": "hey @rasmus01610! This competition is quite brutal and unforgiving, there are so many things that can go wrong!
\nYou might have a bug in the inference pipeline (which is very hard to pick up on), there could be an issue with data preprocessing, etc.
\nA pf1 of 0.05 likely means that the model is not learning, the threshold also should generally be much higher. Might be good to introduce another metric such as AUC or precision and recall that would tell you more about the situation. But my guess is that for some reason the model is just not training.
\nImpossible to tell what the reason might be. This setting is very tough for CV, so I wouldn't be too hard on yourself 🙂 And in general, starting with a working pipeline really goes a long way. Working on this nearly drove me crazy, but I drawing inspiration and validating my work against some of the published notebooks was very helpful, maybe that is a path worth exploring!
\nWishing you the best of luck with getting your pipeline to work!
", "rawMarkdown": "hey @rasmus01610! This competition is quite brutal and unforgiving, there are so many things that can go wrong!\n\nYou might have a bug in the inference pipeline (which is very hard to pick up on), there could be an issue with data preprocessing, etc.\n\nA pf1 of 0.05 likely means that the model is not learning, the threshold also should generally be much higher. Might be good to introduce another metric such as AUC or precision and recall that would tell you more about the situation. But my guess is that for some reason the model is just not training.\n\nImpossible to tell what the reason might be. This setting is very tough for CV, so I wouldn't be too hard on yourself 🙂 And in general, starting with a working pipeline really goes a long way. Working on this nearly drove me crazy, but I drawing inspiration and validating my work against some of the published notebooks was very helpful, maybe that is a path worth exploring!\n\nWishing you the best of luck with getting your pipeline to work!", "votes": 7, "replies": [ { "id": 2082656, "postDate": "2023-01-01T19:07:15.633Z", "content": "Now I'm at the point where I take your notebook and use it to train from scratch, and it still doesn't learn. It is your notebook without any changes. I don't understand the world anymore :D
", "rawMarkdown": "Now I'm at the point where I take your notebook and use it to train from scratch, and it still doesn't learn. It is your notebook without any changes. I don't understand the world anymore :D", "votes": 2 } ] }, { "id": 2087510, "postDate": "2023-01-05T16:34:49.613Z", "content": "I guess I cracked the problem. I rewrote everything in plain pytorch using a weighted sampler, but somehow the biggest impact was that I converted the grayscale images to RGB (3 channels) and I don't normalize them but only scale them to (0,1). Especially the RGB conversion doesn't make any sense in my eyes, but whatever, it trains now.
\nI load the pos samples in a ratio of 1/15 and have a pos_weight of 3 for my BCELoss. This trains now normally actually.
\nI believe the biggest issue was with how I handled the image data before feeding it to the neural net.
Thank you for all of your suggestions.
", "rawMarkdown": "I guess I cracked the problem. I rewrote everything in plain pytorch using a weighted sampler, but somehow the biggest impact was that I converted the grayscale images to RGB (3 channels) and I don't normalize them but only scale them to (0,1). Especially the RGB conversion doesn't make any sense in my eyes, but whatever, it trains now. \n\nI load the pos samples in a ratio of 1/15 and have a pos_weight of 3 for my BCELoss. This trains now normally actually. \nI believe the biggest issue was with how I handled the image data before feeding it to the neural net.\n\nThank you for all of your suggestions. ", "votes": 5, "replies": [ { "id": 2087515, "postDate": "2023-01-05T16:36:42.850Z", "content": "Give us feedback when you finish training - metrics. How it progress.
", "rawMarkdown": "Give us feedback when you finish training - metrics. How it progress. " }, { "id": 2093873, "postDate": "2023-01-10T11:44:38.890Z", "content": "Has your score improved?
", "rawMarkdown": "Has your score improved? ", "replies": [ { "id": 2100631, "postDate": "2023-01-15T10:08:45.943Z", "content": "yep, didn't submit the score yet, but more local CV score improved to 0.15 or so.
", "rawMarkdown": "yep, didn't submit the score yet, but more local CV score improved to 0.15 or so. " } ] } ] }, { "id": 2081682, "postDate": "2022-12-31T14:02:50Z", "content": "Try unweighted BCEloss with 10x upsampling of positives.
", "rawMarkdown": "Try unweighted BCEloss with 10x upsampling of positives.", "votes": 5, "replies": [ { "id": 2082657, "postDate": "2023-01-01T19:07:26.120Z", "content": "Nope. Not really working. stuck at around 0.08 thresholded pfbeta score.
", "rawMarkdown": "Nope. Not really working. stuck at around 0.08 thresholded pfbeta score." } ] }, { "id": 2100396, "postDate": "2023-01-15T06:11:47.327Z", "content": "def efficientnet_b4():
\n model = timm.create_model(
\n 'efficientnet_b4', pretrained=True, in_chans=3, num_classes=1)
\n return model
Please set \" pretrained=True\"
\nI set it False. I sucked.
@rasmus01610 I was having the same issue and in my case I wasn't loading the pretrained weights, and was just training everything from scratch. In some of the shared notebooks you may have to set pretrained=True.
Good Job!
", "rawMarkdown": "Good Job!\n", "votes": -1 }, { "id": 2096391, "postDate": "2023-01-12T02:34:39.450Z", "content": "Maybe do some hyperparameter tuning and try different learning rates, batch sizes..
\nThere are usually low hanging fruits playing with them..
The Devastator.
", "rawMarkdown": "Maybe do some hyperparameter tuning and try different learning rates, batch sizes.. \nThere are usually low hanging fruits playing with them..\n\n\nThe Devastator.\n" }, { "id": 2087586, "postDate": "2023-01-05T17:34:07.050Z", "content": "check if you have equal distribution of cases and controls in each batch. The batch sample may get biased towards the controls.
", "rawMarkdown": "check if you have equal distribution of cases and controls in each batch. The batch sample may get biased towards the controls." }, { "id": 2087097, "postDate": "2023-01-05T10:23:56.703Z", "content": "What is your batch size?
", "rawMarkdown": "What is your batch size?", "replies": [ { "id": 2087496, "postDate": "2023-01-05T16:27:54.817Z", "content": "its 32 at 1024x1024.
", "rawMarkdown": "its 32 at 1024x1024.", "replies": [ { "id": 2123474, "postDate": "2023-01-31T14:38:28.060Z", "content": "How you make it? For 1024 I can reach batch size 4 only here for GPU.
", "rawMarkdown": "How you make it? For 1024 I can reach batch size 4 only here for GPU." }, { "id": 2124572, "postDate": "2023-02-01T04:43:40.017Z", "content": "It is better to use TPU, and if you want to know how to use it, please have a look at the following kaggle notebook https://www.kaggle.com/code/awsaf49/rsna-bcd-efficientnet-tf-tpu-1vm-train
", "rawMarkdown": "It is better to use TPU, and if you want to know how to use it, please have a look at the following kaggle notebook [https://www.kaggle.com/code/awsaf49/rsna-bcd-efficientnet-tf-tpu-1vm-train](https://www.kaggle.com/code/awsaf49/rsna-bcd-efficientnet-tf-tpu-1vm-train)" } ] } ] }, { "id": 2085680, "postDate": "2023-01-04T10:18:58.303Z", "content": "is it this reason?
\n
Nope, I use a weighted datasampler that gives me a constant ratio of pos and neg samples.
", "rawMarkdown": "Nope, I use a weighted datasampler that gives me a constant ratio of pos and neg samples." } ] }, { "id": 2084195, "postDate": "2023-01-03T09:18:01.257Z", "content": "Hi @rasmus01610, I have a similar problem. Have you manage to solve this?
", "rawMarkdown": "Hi @rasmus01610, I have a similar problem. Have you manage to solve this? ", "replies": [ { "id": 2084595, "postDate": "2023-01-03T15:58:38.447Z", "content": "not yet. I tried everything. I wrote the whole pipeline in pure pytorch. There seems to be a really subtle bug somewhere. Don't know where my problem is. I'll keep you updated.
", "rawMarkdown": "not yet. I tried everything. I wrote the whole pipeline in pure pytorch. There seems to be a really subtle bug somewhere. Don't know where my problem is. I'll keep you updated. " } ] } ], "comments": [ { "id": 2081921, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-31T20:24:19.920000", "content": "hey @rasmus01610! This competition is quite brutal and unforgiving, there are so many things that can go wrong!
\nYou might have a bug in the inference pipeline (which is very hard to pick up on), there could be an issue with data preprocessing, etc.
\nA pf1 of 0.05 likely means that the model is not learning, the threshold also should generally be much higher. Might be good to introduce another metric such as AUC or precision and recall that would tell you more about the situation. But my guess is that for some reason the model is just not training.
\nImpossible to tell what the reason might be. This setting is very tough for CV, so I wouldn't be too hard on yourself 🙂 And in general, starting with a working pipeline really goes a long way. Working on this nearly drove me crazy, but I drawing inspiration and validating my work against some of the published notebooks was very helpful, maybe that is a path worth exploring!
\nWishing you the best of luck with getting your pipeline to work!
", "votes": 7, "replies": [ { "id": 2082656, "author_name": "Marius ", "author_url": "", "post_date": "2023-01-01T19:07:15.633000", "content": "Now I'm at the point where I take your notebook and use it to train from scratch, and it still doesn't learn. It is your notebook without any changes. I don't understand the world anymore :D
", "votes": 2, "replies": [] } ] }, { "id": 2087510, "author_name": "Marius ", "author_url": "", "post_date": "2023-01-05T16:34:49.613000", "content": "I guess I cracked the problem. I rewrote everything in plain pytorch using a weighted sampler, but somehow the biggest impact was that I converted the grayscale images to RGB (3 channels) and I don't normalize them but only scale them to (0,1). Especially the RGB conversion doesn't make any sense in my eyes, but whatever, it trains now.
\nI load the pos samples in a ratio of 1/15 and have a pos_weight of 3 for my BCELoss. This trains now normally actually.
\nI believe the biggest issue was with how I handled the image data before feeding it to the neural net.
Thank you for all of your suggestions.
", "votes": 5, "replies": [ { "id": 2087515, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-05T16:36:42.850000", "content": "Give us feedback when you finish training - metrics. How it progress.
", "votes": 0, "replies": [] }, { "id": 2093873, "author_name": "Zincc", "author_url": "", "post_date": "2023-01-10T11:44:38.890000", "content": "Has your score improved?
", "votes": 0, "replies": [ { "id": 2100631, "author_name": "Marius ", "author_url": "", "post_date": "2023-01-15T10:08:45.943000", "content": "yep, didn't submit the score yet, but more local CV score improved to 0.15 or so.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2081682, "author_name": "James Howard", "author_url": "", "post_date": "2022-12-31T14:02:50", "content": "Try unweighted BCEloss with 10x upsampling of positives.
", "votes": 5, "replies": [ { "id": 2082657, "author_name": "Marius ", "author_url": "", "post_date": "2023-01-01T19:07:26.120000", "content": "Nope. Not really working. stuck at around 0.08 thresholded pfbeta score.
", "votes": 0, "replies": [] } ] }, { "id": 2100396, "author_name": "ChenxiangSun@NJU", "author_url": "", "post_date": "2023-01-15T06:11:47.327000", "content": "def efficientnet_b4():
\n model = timm.create_model(
\n 'efficientnet_b4', pretrained=True, in_chans=3, num_classes=1)
\n return model
Please set \" pretrained=True\"
\nI set it False. I sucked.
@rasmus01610 I was having the same issue and in my case I wasn't loading the pretrained weights, and was just training everything from scratch. In some of the shared notebooks you may have to set pretrained=True.
Good Job!
", "votes": -1, "replies": [] }, { "id": 2096391, "author_name": "The Devastator", "author_url": "", "post_date": "2023-01-12T02:34:39.450000", "content": "Maybe do some hyperparameter tuning and try different learning rates, batch sizes..
\nThere are usually low hanging fruits playing with them..
The Devastator.
", "votes": 0, "replies": [] }, { "id": 2087586, "author_name": "dhinesh", "author_url": "", "post_date": "2023-01-05T17:34:07.050000", "content": "check if you have equal distribution of cases and controls in each batch. The batch sample may get biased towards the controls.
", "votes": 0, "replies": [] }, { "id": 2087097, "author_name": "Pranay Barkataki", "author_url": "", "post_date": "2023-01-05T10:23:56.703000", "content": "What is your batch size?
", "votes": 0, "replies": [ { "id": 2087496, "author_name": "Marius ", "author_url": "", "post_date": "2023-01-05T16:27:54.817000", "content": "its 32 at 1024x1024.
", "votes": 0, "replies": [ { "id": 2123474, "author_name": "Igor Litvin", "author_url": "", "post_date": "2023-01-31T14:38:28.060000", "content": "How you make it? For 1024 I can reach batch size 4 only here for GPU.
", "votes": 0, "replies": [] }, { "id": 2124572, "author_name": "Pranay Barkataki", "author_url": "", "post_date": "2023-02-01T04:43:40.017000", "content": "It is better to use TPU, and if you want to know how to use it, please have a look at the following kaggle notebook https://www.kaggle.com/code/awsaf49/rsna-bcd-efficientnet-tf-tpu-1vm-train
", "votes": 0, "replies": [] } ] } ] }, { "id": 2085680, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-04T10:18:58.303000", "content": "is it this reason?
\n
Nope, I use a weighted datasampler that gives me a constant ratio of pos and neg samples.
", "votes": 0, "replies": [] } ] }, { "id": 2084195, "author_name": "Zincc", "author_url": "", "post_date": "2023-01-03T09:18:01.257000", "content": "Hi @rasmus01610, I have a similar problem. Have you manage to solve this?
", "votes": 0, "replies": [ { "id": 2084595, "author_name": "Marius ", "author_url": "", "post_date": "2023-01-03T15:58:38.447000", "content": "not yet. I tried everything. I wrote the whole pipeline in pure pytorch. There seems to be a really subtle bug somewhere. Don't know where my problem is. I'll keep you updated.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2081670": "Hey there!\n\nUnfortunately I'm stuck with my work at a pf1 score of 0.04 - 0.05, although I think I did everything I should do with the problem. \n\nI'm using fastai. I'm using grouped stratified kfold (5 splits) for my folds. I'm training on 512x512 images, upsampling the positive class 3x, training an efficientnet_b2 with weighted BCELoss (positive_weight=20) and a learning rate of 1e-3 - 1e-5 (tried everything). \n\nThat's exactly what e.g. Radek did in his notebook or what others have also reported.\n\nFrom what I read I should be able to get an thresholded pf1 score of around 0.2 with 512x512 images, but I'm still stuck at 0.05, with thresholding maybe 0.07, but not more.\n\nDo you have any tips for me?\n\nThanks in advance", "2081921": "hey @rasmus01610! This competition is quite brutal and unforgiving, there are so many things that can go wrong!\n\nYou might have a bug in the inference pipeline (which is very hard to pick up on), there could be an issue with data preprocessing, etc.\n\nA pf1 of 0.05 likely means that the model is not learning, the threshold also should generally be much higher. Might be good to introduce another metric such as AUC or precision and recall that would tell you more about the situation. But my guess is that for some reason the model is just not training.\n\nImpossible to tell what the reason might be. This setting is very tough for CV, so I wouldn't be too hard on yourself 🙂 And in general, starting with a working pipeline really goes a long way. Working on this nearly drove me crazy, but I drawing inspiration and validating my work against some of the published notebooks was very helpful, maybe that is a path worth exploring!\n\nWishing you the best of luck with getting your pipeline to work!", "2087510": "I guess I cracked the problem. I rewrote everything in plain pytorch using a weighted sampler, but somehow the biggest impact was that I converted the grayscale images to RGB (3 channels) and I don't normalize them but only scale them to (0,1). Especially the RGB conversion doesn't make any sense in my eyes, but whatever, it trains now. \n\nI load the pos samples in a ratio of 1/15 and have a pos_weight of 3 for my BCELoss. This trains now normally actually. \nI believe the biggest issue was with how I handled the image data before feeding it to the neural net.\n\nThank you for all of your suggestions. ", "2081682": "Try unweighted BCEloss with 10x upsampling of positives.", "2100396": "def efficientnet_b4():\n model = timm.create_model(\n 'efficientnet_b4', pretrained=True, in_chans=3, num_classes=1)\n return model\n\nPlease set \" pretrained=True\" \nI set it False. I sucked.", "2090106": "@rasmus01610 I was having the same issue and in my case I wasn't loading the pretrained weights, and was just training everything from scratch. In some of the shared notebooks you may have to set `pretrained=True`.", "2101254": "Good Job!\n", "2096391": "Maybe do some hyperparameter tuning and try different learning rates, batch sizes.. \nThere are usually low hanging fruits playing with them..\n\n\nThe Devastator.\n", "2087586": "check if you have equal distribution of cases and controls in each batch. The batch sample may get biased towards the controls.", "2087097": "What is your batch size?", "2085680": "is it this reason?\n", "2084195": "Hi @rasmus01610, I have a similar problem. Have you manage to solve this? " } }