![\"vertebrae](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5688805%2F6a7e79b4096dd655bd93f71ef6a74f42%2Fstage1_vertebrae.png?generation=1667127778021148&alt=media\")
Congratulations to all the winners! Thanks to Kaggle and RSNA for hosting this interesting competition.
\nThis is my first gold medal and I am glad to become a Kaggle Master, though there is still a long way to go. Many thanks to my teammates yokuyama and sht for giving me the chance to learn so much and inspiring me with new ideas which I could have never come up with as a solo player.
\nOn behalf of our team, let me introduce our solution.
\nThis competition shared a lot of similarity to the past RSNA competitions(2019/2020). Thus we spent a lot of time gathering the wisdom shared by the top teams in the past competitions like aux loss, 2stage model, EMA, etc. Special thanks to all of you for sharing such important information!
\nWe also spent a lot of time reading the paper and watching youtube to learn about Cervical Spine fracture, finally we found the paper which contains the key idea to win this competition: Cropping !!
\nDeep Sequential Learning for Cervical Spine Fracture Detection on Computed Tomography Imaging
\nWe chose seresnext50_32x4d pretrained on imagenet as the backbone for Unet. The reason why we did not use 3D segmentation model is that in the 2nd stage, the feature extracted from vertebrae model boosted CV about 0.02, thus we chose 2D model to get image level feature.
\nThanks timm and segmentation_models_pytorch.
\n
We have to admit that we are lucky enough to get the gold medal. Our submission was an ensemble of mixup seresnext50(CV 0.263, LB 0.23) and no mixup seresnext50(CV 0.269, LB0.24), getting public LB 0.22(private LB 0.28).
\nIn the last two days, we experimentally trained a efficientnet_b2 got CV 0.269 and LB 0.25, indicating that maybe our seresnext50 fit the public LB well and may show a poor performance in private LB. Thus we trained an efficientnet_b4 in the last day(got CV 0.260) and replaced the no mixup seresnext50 with mixup efficientnet_b4. That pushed our public LB to 0.21 and private LB to 0.27 (the gold medal zone).
\nWe spent too much time improving a single model and did not try many other backbones. In the last day we finally found efficientnet was a better choice because the bottleneck of inference is on transfering extracted feature from GPU to CPU(feature len of seresnext50 is 2048…), not the model inference time.
\n![\"fracture](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5688805%2F927d2bf7c4f17d11586c163a9d2b500b%2Fstage1_fracture.png?generation=1667128199554100&alt=media\")
In stage2, we used several slice sequence models for study level classification.
\nFinally, we used four RNN/Transformer models. CV scores for each model were about the same, but the RNN and Transformer ensembles improved the scores slightly.
\nThe architecture of each model is based on past RSNA competitions(2019/2020). Again special thanks to all of you for sharing such important information!
\n\n$$P_{overall} = k_{1} P_{model} + (1-k_{1})(1-\\prod_{i=1}^{7}(1 - P_{\\rm Ci}))$$
\n![\"stage2\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5688805%2F480954ad32c9759aea1961bc933d978f%2Fstage2.png?generation=1667124473522707&alt=media\")
Congratulations to all the winners! Thanks to Kaggle and RSNA for hosting this interesting competition.
\nThis is my first gold medal and I am glad to become a Kaggle Master, though there is still a long way to go. Many thanks to my teammates yokuyama and sht for giving me the chance to learn so much and inspiring me with new ideas which I could have never come up with as a solo player.
\nOn behalf of our team, let me introduce our solution.
\nThis competition shared a lot of similarity to the past RSNA competitions(2019/2020). Thus we spent a lot of time gathering the wisdom shared by the top teams in the past competitions like aux loss, 2stage model, EMA, etc. Special thanks to all of you for sharing such important information!
\nWe also spent a lot of time reading the paper and watching youtube to learn about Cervical Spine fracture, finally we found the paper which contains the key idea to win this competition: Cropping !!
\nDeep Sequential Learning for Cervical Spine Fracture Detection on Computed Tomography Imaging
\nWe chose seresnext50_32x4d pretrained on imagenet as the backbone for Unet. The reason why we did not use 3D segmentation model is that in the 2nd stage, the feature extracted from vertebrae model boosted CV about 0.02, thus we chose 2D model to get image level feature.
\nThanks timm and segmentation_models_pytorch.
\n
We have to admit that we are lucky enough to get the gold medal. Our submission was an ensemble of mixup seresnext50(CV 0.263, LB 0.23) and no mixup seresnext50(CV 0.269, LB0.24), getting public LB 0.22(private LB 0.28).
\nIn the last two days, we experimentally trained a efficientnet_b2 got CV 0.269 and LB 0.25, indicating that maybe our seresnext50 fit the public LB well and may show a poor performance in private LB. Thus we trained an efficientnet_b4 in the last day(got CV 0.260) and replaced the no mixup seresnext50 with mixup efficientnet_b4. That pushed our public LB to 0.21 and private LB to 0.27 (the gold medal zone).
\nWe spent too much time improving a single model and did not try many other backbones. In the last day we finally found efficientnet was a better choice because the bottleneck of inference is on transfering extracted feature from GPU to CPU(feature len of seresnext50 is 2048…), not the model inference time.
\n
In stage2, we used several slice sequence models for study level classification.
\nFinally, we used four RNN/Transformer models. CV scores for each model were about the same, but the RNN and Transformer ensembles improved the scores slightly.
\nThe architecture of each model is based on past RSNA competitions(2019/2020). Again special thanks to all of you for sharing such important information!
\n\n$$P_{overall} = k_{1} P_{model} + (1-k_{1})(1-\\prod_{i=1}^{7}(1 - P_{\\rm Ci}))$$
\n
Great work @honglihang and team! Your approach note is a good learning experience for me. Thanks for sharing it and all the best!
\nHearty congratulations for the result and the progression too!
Congratulations @honglihang and thanks for sharing some wisdom!
", "rawMarkdown": "Congratulations @honglihang and thanks for sharing some wisdom!", "votes": 1 }, { "id": 2010111, "postDate": "2022-10-30T14:21:28.840Z", "content": "Congratulations on winning, it was close, but certainly not just luck, do you train on 235 + 258 patients only?, is it like how I choose my patients in my approach? if so, did you try pseudo labeling the rest of the dataset to get more boost?
", "rawMarkdown": "Congratulations on winning, it was close, but certainly not just luck, do you train on 235 + 258 patients only?, is it like how I choose my patients in my approach? if so, did you try pseudo labeling the rest of the dataset to get more boost?", "votes": 2, "replies": [ { "id": 2010913, "postDate": "2022-10-31T08:41:13.533Z", "content": "Congratulations to you too, nice solo gold!
\nWe trained 1st stage on only 235 + 280 patients, no pseudo lableing, and we used the 1490+ patients left at the 2nd stage. I think we share the same approach in balancing the samples.
Did you filter the C1-C7 area before training fracture classification using segentation model? I saw that you mentioned you got AUC 0.91 but when we filtered the C1-C7 area and trained the model only on C1-C7 image, the AUC decreased to around 0.8. We took this as a correct approach because the gap in AUC indicates that if feeding non C1-C7 image to the model, it gave the model extra task of classifying whether it is C1-C7 (meaning fracture classification task implicitly contains the task to classify C1-C7, resulting in the AUC 0.91, which is overestimated. The feature extracted from such model may cause extra difficulty in 2nd stage).
\nThus, we think the difference in private LB is caused by backbone and pseduo labeling.
", "rawMarkdown": "Congratulations to you too, nice solo gold!\nWe trained 1st stage on only 235 + 280 patients, no pseudo lableing, and we used the 1490+ patients left at the 2nd stage. I think we share the same approach in balancing the samples.\n\nDid you filter the C1-C7 area before training fracture classification using segentation model? I saw that you mentioned you got AUC 0.91 but when we filtered the C1-C7 area and trained the model only on C1-C7 image, the AUC decreased to around 0.8. We took this as a correct approach because the gap in AUC indicates that if feeding non C1-C7 image to the model, it gave the model extra task of classifying whether it is C1-C7 (meaning fracture classification task implicitly contains the task to classify C1-C7, resulting in the AUC 0.91, which is overestimated. The feature extracted from such model may cause extra difficulty in 2nd stage).\n\nThus, we think the difference in private LB is caused by backbone and pseduo labeling.", "votes": 2 }, { "id": 2010961, "postDate": "2022-10-31T09:25:32.943Z", "content": "I did filter the dataset so that I only have images which had atleast one of the seven vertebraes, I did not try it otherwise so I don't know what effect it had on AUC
\nVery interesting remarks, I think we might be not on the same page on something, I do not use the image-level model for giving me a (BS, 7) output, I only use it as binary, so it gives me output of (BS, 1) 1 being fractured and 0 being non-fractured, I do not ask for it to classify vertebrae… A fracture is a fracture, not classifying the vertebrae with the same model does give me a boost.
\nAlthough my features did give me a lot of difficulty in developing the Study level model, I think it was due to my inexperience in the task.
\nI did try both, using the whole set for study-level model gave me about .26-.27 public, using part of it as pseudo for image-level which boosted image-level to .94 AUC as mentioned, and then making a study-level on those features did push the score to .23 public
", "rawMarkdown": "I did filter the dataset so that I only have images which had atleast one of the seven vertebraes, I did not try it otherwise so I don't know what effect it had on AUC\n\nVery interesting remarks, I think we might be not on the same page on something, I do not use the image-level model for giving me a (BS, 7) output, I only use it as binary, so it gives me output of (BS, 1) 1 being fractured and 0 being non-fractured, I do not ask for it to classify vertebrae... A fracture is a fracture, not classifying the vertebrae with the same model does give me a boost.\n\nAlthough my features did give me a lot of difficulty in developing the Study level model, I think it was due to my inexperience in the task.\n\nI did try both, using the whole set for study-level model gave me about .26-.27 public, using part of it as pseudo for image-level which boosted image-level to .94 AUC as mentioned, and then making a study-level on those features did push the score to .23 public", "votes": 1 } ] } ], "comments": [ { "id": 2009942, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-10-30T12:15:17.810000", "content": "Great work @honglihang and team! Your approach note is a good learning experience for me. Thanks for sharing it and all the best!
\nHearty congratulations for the result and the progression too!
Congratulations @honglihang and thanks for sharing some wisdom!
", "votes": 1, "replies": [] }, { "id": 2010111, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-10-30T14:21:28.840000", "content": "Congratulations on winning, it was close, but certainly not just luck, do you train on 235 + 258 patients only?, is it like how I choose my patients in my approach? if so, did you try pseudo labeling the rest of the dataset to get more boost?
", "votes": 2, "replies": [ { "id": 2010913, "author_name": "RihanPiggy", "author_url": "", "post_date": "2022-10-31T08:41:13.533000", "content": "Congratulations to you too, nice solo gold!
\nWe trained 1st stage on only 235 + 280 patients, no pseudo lableing, and we used the 1490+ patients left at the 2nd stage. I think we share the same approach in balancing the samples.
Did you filter the C1-C7 area before training fracture classification using segentation model? I saw that you mentioned you got AUC 0.91 but when we filtered the C1-C7 area and trained the model only on C1-C7 image, the AUC decreased to around 0.8. We took this as a correct approach because the gap in AUC indicates that if feeding non C1-C7 image to the model, it gave the model extra task of classifying whether it is C1-C7 (meaning fracture classification task implicitly contains the task to classify C1-C7, resulting in the AUC 0.91, which is overestimated. The feature extracted from such model may cause extra difficulty in 2nd stage).
\nThus, we think the difference in private LB is caused by backbone and pseduo labeling.
", "votes": 2, "replies": [] }, { "id": 2010961, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-10-31T09:25:32.943000", "content": "I did filter the dataset so that I only have images which had atleast one of the seven vertebraes, I did not try it otherwise so I don't know what effect it had on AUC
\nVery interesting remarks, I think we might be not on the same page on something, I do not use the image-level model for giving me a (BS, 7) output, I only use it as binary, so it gives me output of (BS, 1) 1 being fractured and 0 being non-fractured, I do not ask for it to classify vertebrae… A fracture is a fracture, not classifying the vertebrae with the same model does give me a boost.
\nAlthough my features did give me a lot of difficulty in developing the Study level model, I think it was due to my inexperience in the task.
\nI did try both, using the whole set for study-level model gave me about .26-.27 public, using part of it as pseudo for image-level which boosted image-level to .94 AUC as mentioned, and then making a study-level on those features did push the score to .23 public
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2009857": "Congratulations to all the winners! Thanks to Kaggle and RSNA for hosting this interesting competition.\n\nThis is my first gold medal and I am glad to become a Kaggle Master, though there is still a long way to go. Many thanks to my teammates [yokuyama](https://www.kaggle.com/yokuyama) and [sht](https://www.kaggle.com/shotaosaki) for giving me the chance to learn so much and inspiring me with new ideas which I could have never come up with as a solo player.\n\nOn behalf of our team, let me introduce our solution.\n\n# Stage1: Image Level Model\n\nThis competition shared a lot of similarity to the past RSNA competitions(2019/2020). Thus we spent a lot of time gathering the wisdom shared by the top teams in the past competitions like aux loss, 2stage model, EMA, etc. Special thanks to all of you for sharing such important information!\n\nWe also spent a lot of time reading the paper and watching youtube to learn about Cervical Spine fracture, finally we found the paper which contains the key idea to win this competition: Cropping !!\n\n[Deep Sequential Learning for Cervical Spine Fracture Detection on Computed Tomography Imaging](https://arxiv.org/abs/2010.13336)\n\n## Pipeline\n\n1. Pretrain vertebrae model using Verse2020 (79cases).\n2. Train vertebrae model using Competition data (89cases).\n3. Prepare training data of fracture model. Randomly sampling 258 negative cases, filtering C1-C7 area and croping the vertebrae using vertebrae model trained above.\n4. Train fracture model.\n\n## CV strategy\n\n- Not using CV strategy in stage1.\n\n## Vertebrae model\n\n### Augmentation\n\n- Reused the training pipeline implemented in [UWMGI](https://www.kaggle.com/code/awsaf49/uwmgi-unet-train-pytorch). Big thanks to [awsaf49](https://www.kaggle.com/awsaf49) !\n\n### Model\n\nWe chose seresnext50_32x4d pretrained on imagenet as the backbone for Unet. The reason why we did not use 3D segmentation model is that in the 2nd stage, the feature extracted from vertebrae model boosted CV about 0.02, thus we chose 2D model to get image level feature.\n\nThanks timm and segmentation_models_pytorch.\n\n### Training strategy\n\n- Pretrain 18 epochs using Verse2020 because according to the experience in UWMGI, it takes long epoch to get a good segmentation model.\n- Did not use 2.5D image in vertebrae model considering the computing time, cause we are not competing on the dice loss.\n- train 1 epoch using competition data.\n- CosineAnnealingLR\n\n### What did not work\n\n- LovaszLoss\n- long epoch training using competition data\n\n\n\n## Fracture model\n\n### Augmentation\n\n- HorizontalFlip\n- VerticalFlip\n- Rotate\n- RandomBrightnessContrast\n\n### Model\n\n- seresnext50_32x4d pretrained on imagenet\n- efficientnet_b4 pretrained on noisy student\n\n### Training strategy\n\n- [Important] Mixup(beta=1.0, p=0.6) and long epoch(18-25 epochs) which improved the model performance.(mixup CV 0.263 while no mixup CV 0.269). CV of no mixup model got worse in long epoch while mixup model did not.\n- CosineAnnealingLR\n- EMA(decay=0.9999)\n- 2.5D image\n\n### What did not work\n\n- Randomly dropping the channel containing neighbour image\n- bone view and soft tissue view, used by radiologist to detect fracture (maybe AI has found somewhat new mechanism, hmmm..)\n- large image size (we used 640 x 640, but not useful...)\n\nWe have to admit that we are lucky enough to get the gold medal. Our submission was an ensemble of mixup seresnext50(CV 0.263, LB 0.23) and no mixup seresnext50(CV 0.269, LB0.24), getting public LB 0.22(private LB 0.28).\n\nIn the last two days, we experimentally trained a efficientnet_b2 got CV 0.269 and LB 0.25, indicating that maybe our seresnext50 fit the public LB well and may show a poor performance in private LB. Thus we trained an efficientnet_b4 in the last day(got CV 0.260) and replaced the no mixup seresnext50 with mixup efficientnet_b4. That pushed our public LB to 0.21 and private LB to 0.27 (the gold medal zone).\n\nWe spent too much time improving a single model and did not try many other backbones. In the last day we finally found efficientnet was a better choice because the bottleneck of inference is on transfering extracted feature from GPU to CPU(feature len of seresnext50 is 2048...), not the model inference time.\n\n\n\n# Stage2: Study Level Model\n\nIn stage2, we used several slice sequence models for study level classification.\n\n## Pipeline\n\n1. Extract image-level features from each of the stage 1 fracture and vertebrae models.\n2. Concatenate the fracture and vertebrae features.\n3. Stack the features that belong to one study series together.\n4. Resize or pad the stacked features to feat_dim x 256.\n5. Feed the features into the model.\n\n## CV strategy\n\n- 5 folds. Stratified by patient_overall.\n\n### Augmentation\n\n- [Important] Mixup(beta=1.0, p=0.7) to the stacked features.\n\n### Model\n\nFinally, we used four RNN/Transformer models. CV scores for each model were about the same, but the RNN and Transformer ensembles improved the scores slightly.\n\nThe architecture of each model is based on past RSNA competitions(2019/2020). Again special thanks to all of you for sharing such important information!\n\n- [Attention biGRU](https://www.kaggle.com/competitions/rsna-str-pulmonary-embolism-detection/discussion/194145)\n- [Attention biLSTM](https://www.kaggle.com/competitions/rsna-str-pulmonary-embolism-detection/discussion/194145)\n- [BERT-like Transformer Encoder](https://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/117228)\n- [Dense biLSTM](https://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/117228)\n\n### Post-processing\n\n- Calibrate the models predicted patient_overall with the predicted probabilities of C1-C7.\n\n$$P_{overall} = k_{1} P_{model} + (1-k_{1})(1-\\prod_{i=1}^{7}(1 - P_{\\rm Ci}))$$\n\n\n", "2009942": "Great work @honglihang and team! Your approach note is a good learning experience for me. Thanks for sharing it and all the best!\nHearty congratulations for the result and the progression too!", "2009879": "Congratulations @honglihang and thanks for sharing some wisdom!", "2010111": "Congratulations on winning, it was close, but certainly not just luck, do you train on 235 + 258 patients only?, is it like how I choose my patients in my approach? if so, did you try pseudo labeling the rest of the dataset to get more boost?" } }