![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2755695%2F0bf39e12f3004d2d843bea9aa2944dee%2F2023-08-20%2000.42.48.jpg?generation=1692481385008583&alt=media\")
This was a fantastic competition - thanks to organizers and my team. Past 3 months have not been easy, but I’m glad that in the end it was worth it.
\nQuite early we realized that heavy models and long training work well here, so we mostly ran experiments with backbones like effnet_v2_large and heavier. The final solution includes:
\nIt takes about a week to train with 4xA6000 (most of the training time goes into creation of good pseudo-labels).
\nAfter we figured out that some geometric augmentations don’t really work (for example, vertical flips), we removed them from the augmentation pipeline. Additionally, the 4TTA validation and regular one didn’t have a perfect correlation, so the validation during training was performed with 4TTA, which gave a significant boost early in the competition.
\nIn terms of split - the splitting proposed by organizers was used.
\nThe original dataset (of 20519 records, each of 8 images) is already quite big for U-Net like models, however we wanted to expand it further. That’s why we created pseudo-labels for the whole dataset by training 3 models with 4 folds for 2 rounds. The following scheme illustrates the idea:
\n
Then we train final models with a sampler: 50% original data, 50% soft pseudo labels.
\nNot a lot of things worked well here, so we added some simple tricks to improve the pipeline a bit. Those things include:
\nThis was a fantastic competition - thanks to organizers and my team. Past 3 months have not been easy, but I’m glad that in the end it was worth it.
\nQuite early we realized that heavy models and long training work well here, so we mostly ran experiments with backbones like effnet_v2_large and heavier. The final solution includes:
\nIt takes about a week to train with 4xA6000 (most of the training time goes into creation of good pseudo-labels).
\nAfter we figured out that some geometric augmentations don’t really work (for example, vertical flips), we removed them from the augmentation pipeline. Additionally, the 4TTA validation and regular one didn’t have a perfect correlation, so the validation during training was performed with 4TTA, which gave a significant boost early in the competition.
\nIn terms of split - the splitting proposed by organizers was used.
\nThe original dataset (of 20519 records, each of 8 images) is already quite big for U-Net like models, however we wanted to expand it further. That’s why we created pseudo-labels for the whole dataset by training 3 models with 4 folds for 2 rounds. The following scheme illustrates the idea:
\n
Then we train final models with a sampler: 50% original data, 50% soft pseudo labels.
\nNot a lot of things worked well here, so we added some simple tricks to improve the pipeline a bit. Those things include:
\nCongratulations and thanks for sharing, it is a very impressive result, and even more w/o correcting the labels and using the mean of annotations.
\nIn this case, what would be the advantage of doing pseudolabeling using folds (as you did) vs doing pseudolabeling using all training data at once (i.e. train the 3 models with all available 4th slice training data to predict the 5-8 slices)?
", "rawMarkdown": "Congratulations and thanks for sharing, it is a very impressive result, and even more w/o correcting the labels and using the mean of annotations.\n\nIn this case, what would be the advantage of doing pseudolabeling using folds (as you did) vs doing pseudolabeling using all training data at once (i.e. train the 3 models with all available 4th slice training data to predict the 5-8 slices)? ", "votes": 1, "replies": [ { "id": 2401580, "postDate": "2023-08-21T17:02:24.637Z", "content": "With OOF one can run a few rounds of pseudolabeling.
\nOtherwise confirmation bias will lead to zero improvements after the 1st round.
@selimsef could you clarify the thing about the OOF please? Am I correct, that you firstly train the model on 3 folds, leaving the 4th fold for the validation? Then you do pseudo. Thereafter you train models again on the previous 3 folds+pseudo. And then you are doing pseudo again (the 2nd round)? So basically, do you leave the same fold for the validation each time?
", "rawMarkdown": "@selimsef could you clarify the thing about the OOF please? Am I correct, that you firstly train the model on 3 folds, leaving the 4th fold for the validation? Then you do pseudo. Thereafter you train models again on the previous 3 folds+pseudo. And then you are doing pseudo again (the 2nd round)? So basically, do you leave the same fold for the validation each time? " }, { "id": 2403591, "postDate": "2023-08-22T18:22:27.100Z", "content": "Generally, it's crucial to make out-of-fold (OOF) predictions at each stage (standard k-fold approach). That's the only requirement in this process. So it will work even if you train with 3-4-5-6-7 folds in 5 stage pseudo it will work as well.
", "rawMarkdown": "Generally, it's crucial to make out-of-fold (OOF) predictions at each stage (standard k-fold approach). That's the only requirement in this process. So it will work even if you train with 3-4-5-6-7 folds in 5 stage pseudo it will work as well.", "votes": 1 } ] } ] }, { "id": 2398747, "postDate": "2023-08-19T22:26:09.033Z", "content": "Wow! what a laborious project! Congratulations @ivanpan
\nTransformers seem to be very useful in various AI problems. Can you please suggest me a good source to learn them?
Sure. The main idea that you should understand before diving into any transformers (whether tailored for Computer Vision or NLP or anything else) is attention mechanism that was proposed (or at least popularized) in this paper.
\nI heavily recommend this article that explains it the best.
\nAfter you're done with the fundamentals, just google \"TRANSFORMER_NAME explained medium\" to understand any particular architecture. For implementations - look either at timm or HuggingFace.
\nTo give you pointers, for NLP check out BERT, T5 and GPT models. For CV - ViT and Swin.
", "rawMarkdown": "Sure. The main idea that you should understand before diving into any transformers (whether tailored for Computer Vision or NLP or anything else) is attention mechanism that was proposed (or at least popularized) in [this paper](https://arxiv.org/abs/1706.03762).\n\nI heavily recommend [this article](http://jalammar.github.io/illustrated-transformer/) that explains it the best. \n\nAfter you're done with the fundamentals, just google \"TRANSFORMER_NAME explained medium\" to understand any particular architecture. For implementations - look either at [timm](https://github.com/huggingface/pytorch-image-models) or [HuggingFace](http://huggingface.co/). \n\nTo give you pointers, for NLP check out BERT, T5 and GPT models. For CV - ViT and Swin. ", "votes": 2, "replies": [ { "id": 2402888, "postDate": "2023-08-22T11:38:37.103Z", "content": "Thanks a lot. you really helped me🙏
", "rawMarkdown": "Thanks a lot. you really helped me🙏", "votes": 1 } ] } ] }, { "id": 2413580, "postDate": "2023-08-29T03:04:52.660Z", "content": "I'm sorry to ask three very novice questions:
\nThank you very much, if you can answer
", "rawMarkdown": "I'm sorry to ask three very novice questions:\n\n1. Is TTA only used during the testing phase? If so, what needs to be done during the validation phase?\n2. Where does the data from the pseudo label stage come from? What is the confidence level of the pseudo label to be used? And what does OOF in the picture mean?\n3. How did you find hyperparameters such as Composite loss and backbone? Did you use some automl tools? If so, which tool are you using?\n\nThank you very much, if you can answer" }, { "id": 2398881, "postDate": "2023-08-20T03:29:36.413Z", "content": "Wow very complex and great work @ivanpan !
", "rawMarkdown": "Wow very complex and great work @ivanpan !" }, { "id": 2413575, "postDate": "2023-08-29T02:51:33.860Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2413572, "postDate": "2023-08-29T02:47:23.953Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2401561, "author_name": "delai50", "author_url": "", "post_date": "2023-08-21T16:53:22.817000", "content": "Congratulations and thanks for sharing, it is a very impressive result, and even more w/o correcting the labels and using the mean of annotations.
\nIn this case, what would be the advantage of doing pseudolabeling using folds (as you did) vs doing pseudolabeling using all training data at once (i.e. train the 3 models with all available 4th slice training data to predict the 5-8 slices)?
", "votes": 1, "replies": [ { "id": 2401580, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2023-08-21T17:02:24.637000", "content": "With OOF one can run a few rounds of pseudolabeling.
\nOtherwise confirmation bias will lead to zero improvements after the 1st round.
@selimsef could you clarify the thing about the OOF please? Am I correct, that you firstly train the model on 3 folds, leaving the 4th fold for the validation? Then you do pseudo. Thereafter you train models again on the previous 3 folds+pseudo. And then you are doing pseudo again (the 2nd round)? So basically, do you leave the same fold for the validation each time?
", "votes": 0, "replies": [] }, { "id": 2403591, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2023-08-22T18:22:27.100000", "content": "Generally, it's crucial to make out-of-fold (OOF) predictions at each stage (standard k-fold approach). That's the only requirement in this process. So it will work even if you train with 3-4-5-6-7 folds in 5 stage pseudo it will work as well.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2398747, "author_name": "NewToAI_i", "author_url": "", "post_date": "2023-08-19T22:26:09.033000", "content": "Wow! what a laborious project! Congratulations @ivanpan
\nTransformers seem to be very useful in various AI problems. Can you please suggest me a good source to learn them?
Sure. The main idea that you should understand before diving into any transformers (whether tailored for Computer Vision or NLP or anything else) is attention mechanism that was proposed (or at least popularized) in this paper.
\nI heavily recommend this article that explains it the best.
\nAfter you're done with the fundamentals, just google \"TRANSFORMER_NAME explained medium\" to understand any particular architecture. For implementations - look either at timm or HuggingFace.
\nTo give you pointers, for NLP check out BERT, T5 and GPT models. For CV - ViT and Swin.
", "votes": 2, "replies": [ { "id": 2402888, "author_name": "NewToAI_i", "author_url": "", "post_date": "2023-08-22T11:38:37.103000", "content": "Thanks a lot. you really helped me🙏
", "votes": 1, "replies": [] } ] } ] }, { "id": 2413580, "author_name": "kongweihao", "author_url": "", "post_date": "2023-08-29T03:04:52.660000", "content": "I'm sorry to ask three very novice questions:
\nThank you very much, if you can answer
", "votes": 0, "replies": [] }, { "id": 2398881, "author_name": "Arvind (Yetirajan) Narayanan Iyengar", "author_url": "", "post_date": "2023-08-20T03:29:36.413000", "content": "Wow very complex and great work @ivanpan !
", "votes": 0, "replies": [] }, { "id": 2413575, "author_name": "", "author_url": "", "post_date": "2023-08-29T02:51:33.860000", "content": "", "votes": 0, "replies": [] }, { "id": 2413572, "author_name": "", "author_url": "", "post_date": "2023-08-29T02:47:23.953000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2398737": "This was a fantastic competition - thanks to organizers and my team. Past 3 months have not been easy, but I’m glad that in the end it was worth it.\n\n### Overview\n\n- U-Net models with a diverse set of encoders (both convolution-based, and transformer-based).\n- Creating pseudo-labels for the unlabelled part of the data (7/8 of all the data) using folds and ensembles.\n- Composite loss (CE, Dice, Focal).\n- EMA + SWA.\n- High resolution (512 + 768).\n- 4TTA (hflip, rot90, rot270).\n\n### Base models\n\nQuite early we realized that heavy models and long training work well here, so we mostly ran experiments with backbones like effnet_v2_large and heavier. The final solution includes:\n\n- effnet_v2_l.\n- effnet_v2_xl.\n- effnet_l2.\n- maxvit.\n\nIt takes about a week to train with 4xA6000 (most of the training time goes into creation of good pseudo-labels).\n\n### Validation\n\nAfter we figured out that some geometric augmentations don’t really work (for example, vertical flips), we removed them from the augmentation pipeline. Additionally, the 4TTA validation and regular one didn’t have a perfect correlation, so the validation during training was performed with 4TTA, which gave a significant boost early in the competition. \n\nIn terms of split - the splitting proposed by organizers was used. \n\n### Pseudo labels\n\nThe original dataset (of 20519 records, each of 8 images) is already quite big for U-Net like models, however we wanted to expand it further. That’s why we created pseudo-labels for the whole dataset by training 3 models with 4 folds for 2 rounds. The following scheme illustrates the idea:\n\n\n\nThen we train final models with a sampler: 50% original data, 50% soft pseudo labels.\n\n### Additional tricks\nNot a lot of things worked well here, so we added some simple tricks to improve the pipeline a bit. Those things include:\n\n- Weighted loss between CE, Dice, and Focal. \n- EMA during training.\n- SWA on checkpoints after training.\n- Remove BN from the decoder in U-Net. \n- Add different resolutions in final ensemble (512 + 768).\n\n### Things that didn’t work\n\n- Twersky loss (with focus on either FN or FP).\n- Post-processing to remove FP. \n- 3D models (including conv-lstm).\n- Training on individual annotations (shame on us for not trying to train with mean of annotations).\n- Figuring-out why geometric augmentations don’t work (our guess was annotators’ bias. Turns out, it was conversion bias that other top teams found).\n- Heavy augmentations.\n- Adding classifier.\n- Predicting additional frames or using additional channels during training.\n- Validation based on geography. \n- Training only with positive data.\n\n### Links \n\t\n- Inference kernel: [link](https://www.kaggle.com/code/selimsef/kdl-unet-768-inference-contrails).\n- GitHub repo with training code: [link](https://github.com/selimsef/kaggle-identify-contrails-4th/).\n", "2401561": "Congratulations and thanks for sharing, it is a very impressive result, and even more w/o correcting the labels and using the mean of annotations.\n\nIn this case, what would be the advantage of doing pseudolabeling using folds (as you did) vs doing pseudolabeling using all training data at once (i.e. train the 3 models with all available 4th slice training data to predict the 5-8 slices)? ", "2398747": "Wow! what a laborious project! Congratulations @ivanpan \nTransformers seem to be very useful in various AI problems. Can you please suggest me a good source to learn them?", "2413580": "I'm sorry to ask three very novice questions:\n\n1. Is TTA only used during the testing phase? If so, what needs to be done during the validation phase?\n2. Where does the data from the pseudo label stage come from? What is the confidence level of the pseudo label to be used? And what does OOF in the picture mean?\n3. How did you find hyperparameters such as Composite loss and backbone? Did you use some automl tools? If so, which tool are you using?\n\nThank you very much, if you can answer", "2398881": "Wow very complex and great work @ivanpan !", "2413575": "", "2413572": "" } }