\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2Fcddfda9cbba4cea840ae042a1a2dc420%2FScreenshot%202022-12-05%20210820.png?generation=1670238556819606&alt=media\")
The following training logs come from training on the same split with exactly the same params:
\nExhibit A:
\n
Exhibit B:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F66bfc5351425ddd1b1eae3f2cff2cb49%2FScreenshot%202022-12-05%20211144.png?generation=1670238721164093&alt=media\")
This is so crazy. How can you trust any of your runs? How can you trust your results will generalize to unseen data? How can you tell what works and what doesn't?
\nHere are a couple of ways that should allow our model to learn better:
\nA couple of ideas here. First of all, normalization. Should we normalize each image to between 0 and 1? That is not great for neural networks in general (they learn better on zero-centered data). Equally importantly, this way we lose the relative pixel intensities between images.
\nMaybe one scan being lighter than the other is important?
\nA related idea here is putting different window in each of the 3 channels (though I haven't had much luck with this approach in the past).
\nThe bigger the batch, the higher the chance the loss will be stable.
\nSimilar argument to the above -- if any one step that we make is less likely to take us too far in the wrong direction, our training might be more stable.
\nRemoving images that are malformed or mislabeled can go a very long way to stabilizing (and improving) results.
\nThat is especially true if the other dataset is large and of good quality. If we can start with our neural network being able to \"see\" what's in the image better, its training might be much more consistent.
\nLabel Smoothing helps calibrate the probabilities our model outputs which can be very useful in the context of this competition. But it also improves training outcomes on lower-quality data. Reason being that the loss increases disproportionately as our model becomes overconfident (outputs predictions closer to 0 or 1) and gets an example wrong.
\nA couple of such training examples in succession can throw the weights off to the extent that our model will not be able to recover. Label Smoothing helps with that.
\nBigger images might allow our model to pick up on signal better (there is some indication in the results and discussions on the forums) BUT they come at a cost of longer training and longer inference.
\nThis is what people in this competition have tried and I believe to good results 🙂 Why multitask learning might not improve the final results, it can help bootstrap the training by helping the model pick up on the signal in the data.
\nThis is a big one! Only 2% of images in the train set come from patients with cancer! A model can go through batches without seeing a positive example!
\nBut how to address this in the context of this data? Upsampling, weighted loss? More aggressive augmentations for positive class? This is something that definitely requires more experimentation.
\nI hope you will find the above of use 🙂 I am still working on
\n🤖 [fast.ai starter pack] train + inference 🚀.
\nThe training stability issue is hitting me really hard.
\nBut hoping to get through it soon so that I can put some of the techniques above to further testing 🙂
\nEDIT: Obviously the metric calculation from the 2nd training log had a bug 🤦♂️🤦♂️🤦♂️ That is generally not helpful to figuring out which way is up. But anyhow, even absent that bug, there is a lot of instability in the training, but I have now verified some of the methods from this thread work 🙂 More on this soon!
\nThe following training logs come from training on the same split with exactly the same params:
\nExhibit A:
\n
Exhibit B:
\n
This is so crazy. How can you trust any of your runs? How can you trust your results will generalize to unseen data? How can you tell what works and what doesn't?
\nHere are a couple of ways that should allow our model to learn better:
\nA couple of ideas here. First of all, normalization. Should we normalize each image to between 0 and 1? That is not great for neural networks in general (they learn better on zero-centered data). Equally importantly, this way we lose the relative pixel intensities between images.
\nMaybe one scan being lighter than the other is important?
\nA related idea here is putting different window in each of the 3 channels (though I haven't had much luck with this approach in the past).
\nThe bigger the batch, the higher the chance the loss will be stable.
\nSimilar argument to the above -- if any one step that we make is less likely to take us too far in the wrong direction, our training might be more stable.
\nRemoving images that are malformed or mislabeled can go a very long way to stabilizing (and improving) results.
\nThat is especially true if the other dataset is large and of good quality. If we can start with our neural network being able to \"see\" what's in the image better, its training might be much more consistent.
\nLabel Smoothing helps calibrate the probabilities our model outputs which can be very useful in the context of this competition. But it also improves training outcomes on lower-quality data. Reason being that the loss increases disproportionately as our model becomes overconfident (outputs predictions closer to 0 or 1) and gets an example wrong.
\nA couple of such training examples in succession can throw the weights off to the extent that our model will not be able to recover. Label Smoothing helps with that.
\nBigger images might allow our model to pick up on signal better (there is some indication in the results and discussions on the forums) BUT they come at a cost of longer training and longer inference.
\nThis is what people in this competition have tried and I believe to good results 🙂 Why multitask learning might not improve the final results, it can help bootstrap the training by helping the model pick up on the signal in the data.
\nThis is a big one! Only 2% of images in the train set come from patients with cancer! A model can go through batches without seeing a positive example!
\nBut how to address this in the context of this data? Upsampling, weighted loss? More aggressive augmentations for positive class? This is something that definitely requires more experimentation.
\nI hope you will find the above of use 🙂 I am still working on
\n🤖 [fast.ai starter pack] train + inference 🚀.
\nThe training stability issue is hitting me really hard.
\nBut hoping to get through it soon so that I can put some of the techniques above to further testing 🙂
\nEDIT: Obviously the metric calculation from the 2nd training log had a bug 🤦♂️🤦♂️🤦♂️ That is generally not helpful to figuring out which way is up. But anyhow, even absent that bug, there is a lot of instability in the training, but I have now verified some of the methods from this thread work 🙂 More on this soon!
\nGreat work, Excellent notebook. Upvoted .
", "rawMarkdown": "Great work, Excellent notebook. Upvoted .\n\n", "votes": 1, "replies": [ { "id": 2059511, "postDate": "2022-12-09T00:13:28.653Z", "content": "Thank you, @asif05amu! Appreciate it! Glad you are finding this useful 🙂
", "rawMarkdown": "Thank you, @asif05amu! Appreciate it! Glad you are finding this useful 🙂 " } ] }, { "id": 2057248, "postDate": "2022-12-06T22:21:09.383Z", "content": "\n\nWhy multitask learning might not improve the final results, it can help bootstrap the training by helping the model pick up on the signal in the data.
\n
This hits me for a sec.
", "rawMarkdown": "> Why multitask learning might not improve the final results, it can help bootstrap the training by helping the model pick up on the signal in the data.\n\nThis hits me for a sec.", "votes": 1 }, { "id": 2057252, "postDate": "2022-12-06T22:23:25.070Z", "content": "I find this class imbalance notebook quite interesting: tackling-class-imbalance
", "rawMarkdown": "I find this class imbalance notebook quite interesting: [tackling-class-imbalance](https://www.kaggle.com/code/shahules/tackling-class-imbalance)", "votes": 2, "replies": [ { "id": 2057300, "postDate": "2022-12-07T00:41:28.040Z", "content": "That is a very interesting notebook, thanks for suggesting it! 🙂
", "rawMarkdown": "That is a very interesting notebook, thanks for suggesting it! 🙂" }, { "id": 2059504, "postDate": "2022-12-08T23:51:35.593Z", "content": "i would like in this competition to give my attention to data more :
\nthe first step is : to convert data into PNG format and this task was handle by you . thank you for sharing
\nnext use DeepSmote generative model to balance the data
Great work, Excellent notebook. Upvoted .
", "votes": 1, "replies": [ { "id": 2059511, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-09T00:13:28.653000", "content": "Thank you, @asif05amu! Appreciate it! Glad you are finding this useful 🙂
", "votes": 0, "replies": [] } ] }, { "id": 2057248, "author_name": "Saurav Solanki", "author_url": "", "post_date": "2022-12-06T22:21:09.383000", "content": "\n\nWhy multitask learning might not improve the final results, it can help bootstrap the training by helping the model pick up on the signal in the data.
\n
This hits me for a sec.
", "votes": 1, "replies": [] }, { "id": 2057252, "author_name": "Saurav Solanki", "author_url": "", "post_date": "2022-12-06T22:23:25.070000", "content": "I find this class imbalance notebook quite interesting: tackling-class-imbalance
", "votes": 2, "replies": [ { "id": 2057300, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-07T00:41:28.040000", "content": "That is a very interesting notebook, thanks for suggesting it! 🙂
", "votes": 0, "replies": [] }, { "id": 2059504, "author_name": "Youness EL BRAG", "author_url": "", "post_date": "2022-12-08T23:51:35.593000", "content": "i would like in this competition to give my attention to data more :
\nthe first step is : to convert data into PNG format and this task was handle by you . thank you for sharing
\nnext use DeepSmote generative model to balance the data