![\"sub1.png\"](\"https://i.postimg.cc/nLCdrC3s/sub1.png\")
Thank you to all the participants for your hard work in the competition.We are honored to have achieved a good result, coming in third place in this competition. We also want to express our deepest gratitude to the organizers for putting together such a fantastic event.Thank you very much.
\nFinally, I want to thank my excellent teammates @haqishen , @boliu0 , @kevin1742064161 . On behalf of my teammates, I would like to introduce part of our solution, and another part is presented by @boliu0 in another thread.
We use 4 external data in total. Not all models used all the external data. Some models only used CBIS-DDSM + CMMD, while the remaining models used all four external data. Although these external data appear to be different from the competition data, they can improve the CV and significantly enhance the stability of the training.
\n1) CBIS-DDSM
\nThe classification labels of CBIS-DDSM are: MALIGNANT, BENIGN WITHOUT CALLBACK, BENIGN. We consider MALIGNANT as positive and the others as negative, resulting in 1,350 positive and 1,753 negative samples.
2) CMMD
\nThe classification labels of CMMD are: MALIGNANT, BENIGN. We consider MALIGNANT as positive and BENIGN as negative, resulting in 4,094 positive and 1,108 negative samples.
3) Vindr
\nVindr did not provide classification labels, but provided BI-RADS indices. The detailed explanation of the indices can be found here:
\nWe finally chose to consider BI-RADS-4 and BI-RADS-5 as positive, BI-RADS-2 and BI-RADS-3 as negative, and discard other categories. Finally, we obtained 988 positive and 5,606 negative samples.
4) Mini-DDSM
\n The classification labels of Mini-DDSM are: Benign, Cancer, Normal. Since we found that the detection model had a large number of wrong bboxes in the Normal category, we finally chose to discard all Normal images, consider Cancer as positive, and Benign as negative. Finally, we obtained 2,716 positive and 2,684 negative samples.
1) Firstly, we use 1,000 annotated images, 500 of which are labeled using the open-source annotations provided by @remekkinas and 500 of which are manually annotated by us (referred to as BCD_1k), to train YOLOX-m. No validation set is used, and all data is used for both training and validation.
\n2) Using the model from step 1, we predict the whole dataset using YOLOX-m, which will be used as the ROI crop for the official dataset. Meanwhile, the detection boxes will be saved and a pseudo-label dataset (referred to as BCD_all) will be created.
\n3) Using the BCD_all dataset obtained in step 2, YOLOX-nano and YOLOX-x are trained, with no validation set. YOLOX-nano will be used for final online submissions, while YOLOX-x will be used for ROI crop on external datasets.
\n4) Images were cropped according to the detection boxes and resized to a 1.6:1 aspect ratio (1536960 or 1280800) with padding.
The link to the BCD_1k and BCD_all datasets:
\nhttps://www.kaggle.com/datasets/kevin1742064161/bcd-dataset
\nThe link to the YOLOX code:
\nhttps://www.kaggle.com/datasets/kevin1742064161/yolo-x
\nThe link to the bboxes of official datasets and external datasets:
\nhttps://www.kaggle.com/datasets/kevin1742064161/yolox-bboxes
HorizontalFlip, VerticalFlip, RandomBrightnessContrast, ShiftScaleRotate, MedianBlur, GaussianBlur, GaussNoise, ElasticTransform, GridDistortion, OpticalDistortion, CoarseDropout, Mixup
\nWe used two types of classification models.
\nOne type is a CNN ( EfficientNet or Convnext) that integrates metadata at the image-level and uses Mean to generate prediction scores.
\nThe other is a multi-view model based on CNN+LSTM, which applies the idea of multiple instance learning. (We will call this CNN+LSTM models LSTM for convenience in the following). @boliu0 has provided a more detailed explanation of this model.
There is a significant score difference between the two sites, which is essentially caused by different machines.
\nn folds in the submitted code, it was given a weight of n. Avoiding overemphasis on the weight of individual models was a key strategy for avoiding overfitting.Private LB: 0.51
\nPublic LB: 0.67
\nCV: 0.549
\nLocal Private score: 0.521
Private LB: 0.53
\nPublic LB: 0.67
\nCV: 0.554
\nLocal private Score: 0.524
![\"sub2.png\"](\"https://i.postimg.cc/63VCpKx2/sub2.png\")
1) LSTM has a much more stable threshold in CV, but performs slightly worse than Mean in LB. For the Mean model, a threshold change of ±0.1 can cause a fluctuation of about 0.05 in pF1. For example, if the optimal threshold is 0.5, the worst pF1 can be around 0.45 when the threshold is 0.4~0.6. However, for the LSTM model, a threshold change of 0.3 may only cause a fluctuation of 0.05.
\n2) We had 14 submissions that scored above 0.55 in private leaderboard, including two submissions that ranked highest in the Public Leaderboard (0.68). However, their CV and simulated Private scores were not as good as our final two submissions, so we didn't choose them.
\n3) We believe that the LSTM model is the better model (than Image-level models). Although the local pF1 and AUC are similar, using LSTM is obviously a better choice from a theoretical standpoint. Moreover, the LSTM model has a more stable threshold.
\nInference Code is public!
\nThe code of the two submissions has been publicly available in here and here
Thank you to all the participants for your hard work in the competition.We are honored to have achieved a good result, coming in third place in this competition. We also want to express our deepest gratitude to the organizers for putting together such a fantastic event.Thank you very much.
\nFinally, I want to thank my excellent teammates @haqishen , @boliu0 , @kevin1742064161 . On behalf of my teammates, I would like to introduce part of our solution, and another part is presented by @boliu0 in another thread.
We use 4 external data in total. Not all models used all the external data. Some models only used CBIS-DDSM + CMMD, while the remaining models used all four external data. Although these external data appear to be different from the competition data, they can improve the CV and significantly enhance the stability of the training.
\n1) CBIS-DDSM
\nThe classification labels of CBIS-DDSM are: MALIGNANT, BENIGN WITHOUT CALLBACK, BENIGN. We consider MALIGNANT as positive and the others as negative, resulting in 1,350 positive and 1,753 negative samples.
2) CMMD
\nThe classification labels of CMMD are: MALIGNANT, BENIGN. We consider MALIGNANT as positive and BENIGN as negative, resulting in 4,094 positive and 1,108 negative samples.
3) Vindr
\nVindr did not provide classification labels, but provided BI-RADS indices. The detailed explanation of the indices can be found here:
\nWe finally chose to consider BI-RADS-4 and BI-RADS-5 as positive, BI-RADS-2 and BI-RADS-3 as negative, and discard other categories. Finally, we obtained 988 positive and 5,606 negative samples.
4) Mini-DDSM
\n The classification labels of Mini-DDSM are: Benign, Cancer, Normal. Since we found that the detection model had a large number of wrong bboxes in the Normal category, we finally chose to discard all Normal images, consider Cancer as positive, and Benign as negative. Finally, we obtained 2,716 positive and 2,684 negative samples.
1) Firstly, we use 1,000 annotated images, 500 of which are labeled using the open-source annotations provided by @remekkinas and 500 of which are manually annotated by us (referred to as BCD_1k), to train YOLOX-m. No validation set is used, and all data is used for both training and validation.
\n2) Using the model from step 1, we predict the whole dataset using YOLOX-m, which will be used as the ROI crop for the official dataset. Meanwhile, the detection boxes will be saved and a pseudo-label dataset (referred to as BCD_all) will be created.
\n3) Using the BCD_all dataset obtained in step 2, YOLOX-nano and YOLOX-x are trained, with no validation set. YOLOX-nano will be used for final online submissions, while YOLOX-x will be used for ROI crop on external datasets.
\n4) Images were cropped according to the detection boxes and resized to a 1.6:1 aspect ratio (1536960 or 1280800) with padding.
The link to the BCD_1k and BCD_all datasets:
\nhttps://www.kaggle.com/datasets/kevin1742064161/bcd-dataset
\nThe link to the YOLOX code:
\nhttps://www.kaggle.com/datasets/kevin1742064161/yolo-x
\nThe link to the bboxes of official datasets and external datasets:
\nhttps://www.kaggle.com/datasets/kevin1742064161/yolox-bboxes
HorizontalFlip, VerticalFlip, RandomBrightnessContrast, ShiftScaleRotate, MedianBlur, GaussianBlur, GaussNoise, ElasticTransform, GridDistortion, OpticalDistortion, CoarseDropout, Mixup
\nWe used two types of classification models.
\nOne type is a CNN ( EfficientNet or Convnext) that integrates metadata at the image-level and uses Mean to generate prediction scores.
\nThe other is a multi-view model based on CNN+LSTM, which applies the idea of multiple instance learning. (We will call this CNN+LSTM models LSTM for convenience in the following). @boliu0 has provided a more detailed explanation of this model.
There is a significant score difference between the two sites, which is essentially caused by different machines.
\nn folds in the submitted code, it was given a weight of n. Avoiding overemphasis on the weight of individual models was a key strategy for avoiding overfitting.Private LB: 0.51
\nPublic LB: 0.67
\nCV: 0.549
\nLocal Private score: 0.521
Private LB: 0.53
\nPublic LB: 0.67
\nCV: 0.554
\nLocal private Score: 0.524
1) LSTM has a much more stable threshold in CV, but performs slightly worse than Mean in LB. For the Mean model, a threshold change of ±0.1 can cause a fluctuation of about 0.05 in pF1. For example, if the optimal threshold is 0.5, the worst pF1 can be around 0.45 when the threshold is 0.4~0.6. However, for the LSTM model, a threshold change of 0.3 may only cause a fluctuation of 0.05.
\n2) We had 14 submissions that scored above 0.55 in private leaderboard, including two submissions that ranked highest in the Public Leaderboard (0.68). However, their CV and simulated Private scores were not as good as our final two submissions, so we didn't choose them.
\n3) We believe that the LSTM model is the better model (than Image-level models). Although the local pF1 and AUC are similar, using LSTM is obviously a better choice from a theoretical standpoint. Moreover, the LSTM model has a more stable threshold.
\nInference Code is public!
\nThe code of the two submissions has been publicly available in here and here
\n\nWe believe that the LSTM model is the better model (than Image-level models). Although the local pF1 and AUC are similar, using LSTM is obviously a better choice from a theoretical standpoint. Moreover, the LSTM model has a more stable threshold.
\n
Hope you get to investigate this further during the follow-up analysis. Congrats and thanks for sharing, your deliverables/datasets are very comprehensive.
", "rawMarkdown": ">We believe that the LSTM model is the better model (than Image-level models). Although the local pF1 and AUC are similar, using LSTM is obviously a better choice from a theoretical standpoint. Moreover, the LSTM model has a more stable threshold.\n\nHope you get to investigate this further during the follow-up analysis. Congrats and thanks for sharing, your deliverables/datasets are very comprehensive.", "votes": 1, "replies": [ { "id": 2165987, "postDate": "2023-03-02T15:10:56.087Z", "content": "Thank you for your suggestions and congratulations.
", "rawMarkdown": "Thank you for your suggestions and congratulations.", "votes": 1 } ] }, { "id": 2173131, "postDate": "2023-03-08T05:49:03.253Z", "content": "I don't know yolox, you can help me?
", "rawMarkdown": "I don't know yolox, you can help me?" } ], "comments": [ { "id": 2165938, "author_name": "Antti Isosalo", "author_url": "", "post_date": "2023-03-02T14:47:27.910000", "content": "\n\nWe believe that the LSTM model is the better model (than Image-level models). Although the local pF1 and AUC are similar, using LSTM is obviously a better choice from a theoretical standpoint. Moreover, the LSTM model has a more stable threshold.
\n
Hope you get to investigate this further during the follow-up analysis. Congrats and thanks for sharing, your deliverables/datasets are very comprehensive.
", "votes": 1, "replies": [ { "id": 2165987, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-03-02T15:10:56.087000", "content": "Thank you for your suggestions and congratulations.
", "votes": 1, "replies": [] } ] }, { "id": 2173131, "author_name": "Qx Nam", "author_url": "", "post_date": "2023-03-08T05:49:03.253000", "content": "I don't know yolox, you can help me?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2165801": "## Introduction\nThank you to all the participants for your hard work in the competition.We are honored to have achieved a good result, coming in third place in this competition. We also want to express our deepest gratitude to the organizers for putting together such a fantastic event.Thank you very much.\nFinally, I want to thank my excellent teammates @haqishen , @boliu0 , @kevin1742064161 . On behalf of my teammates, I would like to introduce part of our solution, and another part is presented by @boliu0 [in another thread](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/391779).\n\n## 1. Overview of the pipeline\n- Extract ROI with a fixed aspect ratio(1.6:1) using YOLOX\n- Feed the ROI into different classification models\n- Average weighting fusion of the results from the classification models\n\n## 2. External Data\nWe use 4 external data in total. Not all models used all the external data. Some models only used CBIS-DDSM + CMMD, while the remaining models used all four external data. Although these external data appear to be different from the competition data, they can improve the CV and significantly enhance the stability of the training.\n\n1) [CBIS-DDSM](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=22516629)\nThe classification labels of CBIS-DDSM are: MALIGNANT, BENIGN WITHOUT CALLBACK, BENIGN. We consider MALIGNANT as positive and the others as negative, resulting in 1,350 positive and 1,753 negative samples.\n\n2) [CMMD](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=70230508)\nThe classification labels of CMMD are: MALIGNANT, BENIGN. We consider MALIGNANT as positive and BENIGN as negative, resulting in 4,094 positive and 1,108 negative samples.\n\n3) [Vindr](https://physionet.org/content/vindr-mammo/1.0.0/)\nVindr did not provide classification labels, but provided BI-RADS indices. The detailed explanation of the indices can be found [here](https://radiopaedia.org/articles/breast-imaging-reporting-and-data-system-bi-rads): \nWe finally chose to consider BI-RADS-4 and BI-RADS-5 as positive, BI-RADS-2 and BI-RADS-3 as negative, and discard other categories. Finally, we obtained 988 positive and 5,606 negative samples.\n\n4) [Mini-DDSM](https://www.kaggle.com/datasets/cheddad/miniddsm2)\n The classification labels of Mini-DDSM are: Benign, Cancer, Normal. Since we found that the detection model had a large number of wrong bboxes in the Normal category, we finally chose to discard all Normal images, consider Cancer as positive, and Benign as negative. Finally, we obtained 2,716 positive and 2,684 negative samples.\n\n## 3. Preprocessing\n1) Firstly, we use 1,000 annotated images, 500 of which are labeled using the open-source annotations provided by @remekkinas and 500 of which are manually annotated by us (referred to as BCD_1k), to train YOLOX-m. No validation set is used, and all data is used for both training and validation. \n2) Using the model from step 1, we predict the whole dataset using YOLOX-m, which will be used as the ROI crop for the official dataset. Meanwhile, the detection boxes will be saved and a pseudo-label dataset (referred to as BCD_all) will be created. \n3) Using the BCD_all dataset obtained in step 2, YOLOX-nano and YOLOX-x are trained, with no validation set. YOLOX-nano will be used for final online submissions, while YOLOX-x will be used for ROI crop on external datasets.\n4) Images were cropped according to the detection boxes and resized to a 1.6:1 aspect ratio (1536*960 or 1280*800) with padding.\n\nThe link to the BCD_1k and BCD_all datasets:\n[https://www.kaggle.com/datasets/kevin1742064161/bcd-dataset](https://www.kaggle.com/datasets/kevin1742064161/bcd-dataset)\nThe link to the YOLOX code:\n[https://www.kaggle.com/datasets/kevin1742064161/yolo-x](https://www.kaggle.com/datasets/kevin1742064161/yolo-x)\nThe link to the bboxes of official datasets and external datasets:\n[https://www.kaggle.com/datasets/kevin1742064161/yolox-bboxes](https://www.kaggle.com/datasets/kevin1742064161/yolox-bboxes)\n\n## 4. Data augmentation\nHorizontalFlip, VerticalFlip, RandomBrightnessContrast, ShiftScaleRotate, MedianBlur, GaussianBlur, GaussNoise, ElasticTransform, GridDistortion, OpticalDistortion, CoarseDropout, Mixup\n\n## 5. Model\nWe used two types of classification models. \nOne type is a CNN ( EfficientNet or Convnext) that integrates metadata at the image-level and uses Mean to generate prediction scores. \nThe other is a multi-view model based on CNN+LSTM, which applies the idea of multiple instance learning. (We will call this CNN+LSTM models `LSTM` for convenience in the following). @boliu0 has provided a [more detailed explanation of this model](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/391779). \n\n### 5.1 Meta data\n#### Motivation of meta data\nThere is a significant score difference between the two sites, which is essentially caused by different machines.\n\n#### How to encode and insert meta data\n- Considering that there are machines in the test set that have not appeared in the training set, @boliu0 suggests to use one-hot encoding for machines. Since there are 10 machines in the training set, the encoding is a 1x10 vector. If it is external data or a machine that has not appeared in the training set, it is a vector of all zeros.\n- Site and view are encoded in the same way, as a 1x2 vector.\n- The machine or machine+site+view vectors are sent to an MLP for encoding, and the encoded features are concatenated with the CNN backbone features and sent to another MLP for classification.\n\n## 6. Ensemble\n- Considering the significant risk of shake in this competition, we adopted a relatively conservative strategy for our final submissions. One submission focused on the Mean model and included 7 image-level models and 4 LSTM models. The other submission focused on LSTM and included 7 LSTM models and 4 image-level models. The threshold for combining the two types of models was optimized using the best cross-validation (CV) threshold obtained from the out-of-fold (OOF) data. \n- The image-level models were trained using a 4-fold split, while the LSTM models were trained using a 5-fold split. When searching for the threshold, the entire OOF was used. If a model utilized `n` folds in the submitted code, it was given a weight of `n`. Avoiding overemphasis on the weight of individual models was a key strategy for avoiding overfitting.\n- In the end, the LSTM-focused model achieved a higher score in both submissions, which was consistent with our CV and simulated private scores. For more details about our simulated private score, please check [here](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/390958). We are not sure that we simulated in the right way, just a funny story.\n\n### 6.1 First submission: 7 Image-level(Mean) + 4 LSTM\nPrivate LB: 0.51 \nPublic LB: 0.67 \nCV: 0.549\nLocal Private score: 0.521\n\n[](https://postimg.cc/Y4BNyqKk)\n\n## 6.2 Second submission: 7 LSTM + 4 Image-level(Mean)\nPrivate LB: 0.53\nPublic LB: 0.67\nCV: 0.554\nLocal private Score: 0.524\n\n[](https://postimg.cc/9RMDLvMC)\n\n## 7. Some interesting thoughts\n\n1) LSTM has a much more stable threshold in CV, but performs slightly worse than Mean in LB. For the Mean model, a threshold change of ±0.1 can cause a fluctuation of about 0.05 in pF1. For example, if the optimal threshold is 0.5, the worst pF1 can be around 0.45 when the threshold is 0.4~0.6. However, for the LSTM model, a threshold change of 0.3 may only cause a fluctuation of 0.05.\n\n2) We had 14 submissions that scored above 0.55 in private leaderboard, including two submissions that ranked highest in the Public Leaderboard (0.68). However, their CV and simulated Private scores were not as good as our final two submissions, so we didn't choose them. \n\n3) We believe that the LSTM model is the better model (than Image-level models). Although the local pF1 and AUC are similar, using LSTM is obviously a better choice from a theoretical standpoint. Moreover, the LSTM model has a more stable threshold.\n\n**Inference Code is public!**\nThe code of the two submissions has been publicly available in [here](https://www.kaggle.com/code/forcewithme/0226-yoloxnano-yoloxs-mean2) and [here](https://www.kaggle.com/code/forcewithme/final-lstm2)\n", "2165938": ">We believe that the LSTM model is the better model (than Image-level models). Although the local pF1 and AUC are similar, using LSTM is obviously a better choice from a theoretical standpoint. Moreover, the LSTM model has a more stable threshold.\n\nHope you get to investigate this further during the follow-up analysis. Congrats and thanks for sharing, your deliverables/datasets are very comprehensive.", "2173131": "I don't know yolox, you can help me?" } }