![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3706526%2Ff93bb633bb5bcfeffde33cb6931f5bd3%2Fsolution.png?generation=1677589173238328&alt=media\")
First of all, we would like to express our respect to all the participants and thank the organizers for making this competition possible. This competition was tough for us due to the volatility of the metrics and the fact that we had to complete our inference in less than 9 hours. In the end, by choosing our best local cv model (CV0.534), we were able to win 7th place.
\nHere we will describe how we increased our CV score and how we made our entire inference faster.
We obtained the ROI breast in a rule-based way. In brief, after converting values below 40 to 0, uniform columns and rows were removed because they were assumed to be background. This process was so simple yet effective and fast. After this preprocessing, we obtained images with an aspect ratio of 1:1.6~2 on average, and which were then resized to 1520x912. This resolution was determined after comparing four options: ①640x640, ②1024x1024, ③1520x912, and ④2689x1569. The order of CV score was ① < ② < ③ > ④. Additionally, we generated images with sigmoid windowing applied as well, which did not have a significant effect on improving the score, and even on ensemble. However, we trained some models with these images and included them in the ensemble to make our prediction more robust.
\nThe most important thing was to generate 2 types of input images, namely breast-level and laterality-level. The breast-level consisted of one breast per image, while the laterality-level had two or more breasts per image by simply concatenating breasts in columns. The models trained with the laterality-level was much weaker (-0.04) than those with the breast-level, but the ensemble of the two levels was very effective.
We found that larger models did not necessarily score better (EfficientNetB2 < EfficientNetB5 > EfficientNetB7). Out of several models, including EfficientNetV2, EfficientNetB5, SeResNeXt50, ConvNeXt tiny/small, and NextViT base, EfficientNetB5 performed the best (CV0.47 on 1520x912), and EfficientNetV2S came in second-best (CV0.45 on 1520x912). Additionally, we replaced the pooling layer from 'average' to 'generalized mean' (GeM), which resulted in a slight improvement in score (0.005~0.01). Note that we ultimately used EfficientNetV2S in inference because EfficientNetB5 takes longer to infer than EfficientNetV2S. However, the difference in metrics between the two was negligible (CV0.49, single model) by using B5's predictions on external data when training V2S, as described later.
\nWe used VinDr-Mammo dataset as external data, whose labels were defined by breast-level predictions after aggregating them into laterality-level. We found a significant improvement in the CV score by 0.02 using this dataset.
\nThe inference time limit was very tight, so we made efforts to ensure inference completed in time. First, we used DALI to decode images and preprocessed most of them on GPU. This significantly increased processing speed. However, as mentioned here, some images could not be processed using DALI, so we used dicomSDL and cupy as a fallback method for decoding and preprocessing images. Additionally, we implemented a 2-stage method as shown in the following image. Through data analysis, we observed that the data contained many explicit negative samples which did not require prediction by a strong model or ensemble. By filtering out those data with the threshold of 0.01, we were able to reduce the number of data (to 25%) which need to be saved to disk and predicted without losing accuracy.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3706526%2Fcfd0afe964fcafd12ca8460914bd6816%2Fsolution1.png?generation=1677589474115470&alt=media\")
\nTo speed up prediction, we compiled our models using tensorRT. Through our experiments, we observed that the fp16 model was 1.3 times faster than fp32, but its numerical error is not negligible. In the end, we used models compiled in fp32.
train: https://github.com/Masaaaato/RSNABreast7thPlace
\ninference: https://www.kaggle.com/code/masato114/2stage-ensemble/notebook
First of all, we would like to express our respect to all the participants and thank the organizers for making this competition possible. This competition was tough for us due to the volatility of the metrics and the fact that we had to complete our inference in less than 9 hours. In the end, by choosing our best local cv model (CV0.534), we were able to win 7th place.
\nHere we will describe how we increased our CV score and how we made our entire inference faster.
We obtained the ROI breast in a rule-based way. In brief, after converting values below 40 to 0, uniform columns and rows were removed because they were assumed to be background. This process was so simple yet effective and fast. After this preprocessing, we obtained images with an aspect ratio of 1:1.6~2 on average, and which were then resized to 1520x912. This resolution was determined after comparing four options: ①640x640, ②1024x1024, ③1520x912, and ④2689x1569. The order of CV score was ① < ② < ③ > ④. Additionally, we generated images with sigmoid windowing applied as well, which did not have a significant effect on improving the score, and even on ensemble. However, we trained some models with these images and included them in the ensemble to make our prediction more robust.
\nThe most important thing was to generate 2 types of input images, namely breast-level and laterality-level. The breast-level consisted of one breast per image, while the laterality-level had two or more breasts per image by simply concatenating breasts in columns. The models trained with the laterality-level was much weaker (-0.04) than those with the breast-level, but the ensemble of the two levels was very effective.
We found that larger models did not necessarily score better (EfficientNetB2 < EfficientNetB5 > EfficientNetB7). Out of several models, including EfficientNetV2, EfficientNetB5, SeResNeXt50, ConvNeXt tiny/small, and NextViT base, EfficientNetB5 performed the best (CV0.47 on 1520x912), and EfficientNetV2S came in second-best (CV0.45 on 1520x912). Additionally, we replaced the pooling layer from 'average' to 'generalized mean' (GeM), which resulted in a slight improvement in score (0.005~0.01). Note that we ultimately used EfficientNetV2S in inference because EfficientNetB5 takes longer to infer than EfficientNetV2S. However, the difference in metrics between the two was negligible (CV0.49, single model) by using B5's predictions on external data when training V2S, as described later.
\nWe used VinDr-Mammo dataset as external data, whose labels were defined by breast-level predictions after aggregating them into laterality-level. We found a significant improvement in the CV score by 0.02 using this dataset.
\nThe inference time limit was very tight, so we made efforts to ensure inference completed in time. First, we used DALI to decode images and preprocessed most of them on GPU. This significantly increased processing speed. However, as mentioned here, some images could not be processed using DALI, so we used dicomSDL and cupy as a fallback method for decoding and preprocessing images. Additionally, we implemented a 2-stage method as shown in the following image. Through data analysis, we observed that the data contained many explicit negative samples which did not require prediction by a strong model or ensemble. By filtering out those data with the threshold of 0.01, we were able to reduce the number of data (to 25%) which need to be saved to disk and predicted without losing accuracy.
\n
\nTo speed up prediction, we compiled our models using tensorRT. Through our experiments, we observed that the fp16 model was 1.3 times faster than fp32, but its numerical error is not negligible. In the end, we used models compiled in fp32.
train: https://github.com/Masaaaato/RSNABreast7thPlace
\ninference: https://www.kaggle.com/code/masato114/2stage-ensemble/notebook
Great work and congrats!
\n\"The models trained with the laterality-level was much weaker (-0.04) than those with the breast-level, but the ensemble of the two levels was very effective.\"
\nI have the first observation too but I just stopped there… 😑
", "rawMarkdown": "Great work and congrats!\n\n\"The models trained with the laterality-level was much weaker (-0.04) than those with the breast-level, but the ensemble of the two levels was very effective.\"\n\nI have the first observation too but I just stopped there... 😑", "votes": 1, "replies": [ { "id": 2165561, "postDate": "2023-03-02T09:23:04.330Z", "content": "Of course I considered discarding that weaker models, but I thought surely the lower covariance between breast- and laterality-level would be beneficial to the ensemble, and I was reassured by the fact that the scores were up in LB😏
", "rawMarkdown": "Of course I considered discarding that weaker models, but I thought surely the lower covariance between breast- and laterality-level would be beneficial to the ensemble, and I was reassured by the fact that the scores were up in LB😏", "votes": 1 } ] }, { "id": 2263351, "postDate": "2023-05-17T14:21:00.723Z", "content": "Hello @masato114 i checked your code. So in the instruction after training with rsna images (step 1), creating pseudo labels (step 2), you instructed the below step 3 instruction:
\nChange the config file as follows according to your purpose.
\nFor breast-level/external dataset/no windowing: config1.yaml
\nFor laterality-level/external dataest/no windowing: config2.yaml
\nFor breast-level/external dataset/windowing: config3.yaml
\nFor laterality-level/external dataset/windowing: config4.yaml
So my question is: for step 3 i will use train.py with the above configs right? Also do i need to finetune with lateral with vindr and then breast with vindr as shown in the figure or i can finetune them in any order and ensemble later?
", "rawMarkdown": "Hello @masato114 i checked your code. So in the instruction after training with rsna images (step 1), creating pseudo labels (step 2), you instructed the below step 3 instruction: \nChange the config file as follows according to your purpose.\nFor breast-level/external dataset/no windowing: config1.yaml\nFor laterality-level/external dataest/no windowing: config2.yaml\nFor breast-level/external dataset/windowing: config3.yaml\nFor laterality-level/external dataset/windowing: config4.yaml\n\nSo my question is: for step 3 i will use train.py with the above configs right? Also do i need to finetune with lateral with vindr and then breast with vindr as shown in the figure or i can finetune them in any order and ensemble later? ", "replies": [ { "id": 2265147, "postDate": "2023-05-19T02:49:34.330Z", "content": "Hi, thanks for your comment.
\nYes, you can use train.py with the configs you indicated for step 3. Once you successfully generated pseudolabels on vindr, which are labeled for each breast of each image, you can make lateral-level labels by aggregating them on both patient_id and laterality. This should be done before proceeding to step 3.
\nAs you are saying, you can finetune them in any order and ensemble later.
", "rawMarkdown": "Hi, thanks for your comment.\n\nYes, you can use train.py with the configs you indicated for step 3. Once you successfully generated pseudolabels on vindr, which are labeled for each breast of each image, you can make lateral-level labels by aggregating them on both patient_id and laterality. This should be done before proceeding to step 3.\n\nAs you are saying, you can finetune them in any order and ensemble later.", "replies": [ { "id": 2265182, "postDate": "2023-05-19T03:50:09.807Z", "content": "@masato114 Thanks for the reply. My further question is:
\n--This should be done before proceeding to step 3.
\nSo is this the pseudo label generation step as mentioned in your github as:
\nConduct pseudolabeling on external dataset.
\npython -u src/external_pseudolabeling.py configs/config0.yaml
These are the steps from your github readme.md:
\nIn short, are the 3 steps specified in the github readme.md file are sufficient right? or u want any intermediate step as for generating lateral-level labels by aggregating them on both patient_id and laterality? Can you please clarify where to put any extra steps in the above-mentioned three steps as i am seeing
\nyou have generated the lateral labels in the external_pseudolabeling.py file (in VinDrMammo_breast-level_annotations.csv)? Can u please confirm?
Hi, I apologize for making you confused.
\nThere are no extra steps between step 2 and 3. But perhaps you need to rename columns of vindr annotation file so that they are the same as of train.csv. I may have overlooked something about vindr whose patient column is named as 'study_id' not 'patient_id', I guess.
\nIf you have hit errors, please kindly share the concrete problems.
Thanks a lot..
", "rawMarkdown": "Thanks a lot.." }, { "id": 2294255, "postDate": "2023-06-09T23:12:39.713Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2305764, "postDate": "2023-06-16T22:39:44.253Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2305778, "postDate": "2023-06-16T22:48:08.277Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2305791, "postDate": "2023-06-16T23:05:07.107Z", "content": "Yes, I did the same process as what is applied to test images in inference notebook.
", "rawMarkdown": "Yes, I did the same process as what is applied to test images in inference notebook." }, { "id": 2305800, "postDate": "2023-06-16T23:22:50.380Z", "content": "Hi @masato114 , did u perform the same preprocessing for vindr dataset as u did for kaggle? can u please share the preprocessing for vindr dataset?
", "rawMarkdown": "Hi @masato114 , did u perform the same preprocessing for vindr dataset as u did for kaggle? can u please share the preprocessing for vindr dataset?" }, { "id": 2305811, "postDate": "2023-06-16T23:30:55.130Z", "content": "You can find it in https://www.kaggle.com/code/masato114/rsna-generate-train-images/notebook.
", "rawMarkdown": "You can find it in https://www.kaggle.com/code/masato114/rsna-generate-train-images/notebook." }, { "id": 2305832, "postDate": "2023-06-16T23:57:39.507Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2305854, "postDate": "2023-06-17T00:33:03.157Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2305968, "postDate": "2023-06-17T03:18:22.597Z", "content": "@masato114 the TransferSyntaxUID is 1.2.840.10008.1.2.1, it is different from the RSNA dataset. So, if condition of convert_dicom_to_j2k is not executed for vindr and format is also .dicom for vindr instead of .dcm for rsna. So either u have commented the if check for PhotometricInterpretation and TransferSyntaxUID. Can u please check?
\nAlso because Vindr images have different formats, the dali is throwing an error jp2:
\n[/opt/dali/dali/imgcodec/image_decoder.cc:448] Cannot parse the image: /ocean/projects/asc170022p/shg121/PhD/RSNA_Breast_Imaging/Dataset/External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/tmp/j2k/42ec6b7233c3e984cec02a1dd702dcb0_33c3590bb1738c0b9512630d9741bf65.jp2
The temporary jp2 image is actually not forming up correctly.
\nSo, i am doing like this:
\nimport cv2
\nSIZE = (912, 1520)
\npixel_data = ds.pixel_array
\nnormalized_data = ((pixel_data - np.min(pixel_data)) / np.ptp(pixel_data) * 255).astype(np.uint8)
\nimg = cv2.resize(normalized_data, SIZE, interpolation=cv2.INTER_AREA)
\nplt.imshow(img, cmap=plt.cm.bone)
\nplt.show()
Try the following.
\n ():\n img_copy = img.copy()\n img = np.where(img <= , , img) \n height, _ = img.shape\n\n \n y_a = height // + (height*)\n y_b = height // - (height*)\n b_arr = img[y_b:y_a].std(axis=) != \n continuing_ones = CountUpContinuingOnes(b_arr)\n \n col_ind = np.where(continuing_ones == continuing_ones.())[]\n img = img[:, col_ind]\n\n \n _, width = img.shape\n x_a = width // + (width*)\n x_b = width // - (width*)\n b_arr = img[:,x_b:x_a].std(axis=) != \n continuing_ones = CountUpContinuingOnes(b_arr)\n \n row_ind = np.where(continuing_ones == continuing_ones.())[]\n\n img_copy[row_ind][:, col_ind]\n\n ():\n dicom = dicomsdl.(in_path)\n data = dicom.pixelData()\n data = data[:-, :-]\n dicom.getPixelDataInfo()[] == :\n data = np.amax(data) - data\n\n data = data - np.(data)\n data = data / np.(data)\n data = (data * ).astype(np.uint8)\n\n img = ExtractBreast(data)\n img = cv2.resize(img, SIZE, interpolation = cv2.INTER_AREA)\n cv2.imwrite(out_path, img)\nGreat thanks @masato114
", "rawMarkdown": "Great thanks @masato114 " }, { "id": 2333448, "postDate": "2023-07-07T01:44:45.600Z", "content": "Hi @masato114, thanks for all the clarification for the training. I really appreciate your help and i am able to
\nsuccessfully replicate your training scripts for all the four cases.
Now after training, following is the version number of all the models and their corresponding architectures:
\n| Configs | \nBreast/Laterity | \nDataset type | \nModel type | \nModel version | \n
|---|---|---|---|---|
| config0.yaml | \nBreast Level | \nKaggle Data | \nEfficientnet_b5 | \nmodel_ver_084 | \n
| config1.yaml | \nBreast Level | \nKaggle + External Data | \nEfficientnet_b2 | \nmodel_ver_084 | \n
| config2.yaml | \nLateral Level | \nKaggle + External Data | \nEfficientnet_b2 | \nmodel_ver_085 | \n
| config3.yaml | \nBreast Level | \nKaggle + External Data (with Sigmoid windowing) | \nEfficientnet_b2 | \nmodel_ver_099 | \n
| config4.yaml | \nLateral Level | \nKaggle + External Data (with Sigmoid windowing) | \nEfficientnet_b2 | \nmodel_ver_100 | \n
Now when I am trying to replicate the code for ensemble as per https://www.kaggle.com/code/masato114/2stage-ensemble/notebook, in the 20^th notebook, you are using the model kaggle/input/compile-ver49-83-83/ver49-{fold}.ts' to fill df['prediction'] data.
\nThen in 23rd notebook for breast level, /kaggle/input/efficientnetv2s-exp084-tensorrt/efficientnetv2s_ver084-seed10-{fold}-swa.ts, /kaggle/input/efficientnetv2s-exp084-tensorrt/efficientnetv2s_ver084-seed660-{fold}-swa.ts and /kaggle/input/rsna-efficientnetv2s-pseudo-sig-tensorrt/efficientnetv2s_ver099-seed10-{fold}.ts are used to predict test_preds_1, test_preds_2 and test_preds_3 respectively. These 3 values are ensembled to calculate second_df['prediction_ver84'].
\nSimilary in 29^th notebook for laterality Level, /kaggle/input/efficientnetv2s-exp085-tensorrt/efficientnetv2s_ver085-{fold}.ts, kaggle/input/efficientnetv2s-exp085-tensorrt/efficientnetv2s_ver085-seed660-{fold}.ts, and /kaggle/input/rsna-efficientnetv2s-pseudo-sig-tensorrt/efficientnetv2s_ver100-seed10-{fold}.ts are used predict test_preds_1, test_preds_2 and test_preds_3 respectively. These 3 values are ensembled to calculate second_df['prediction_ver85'].
\nNow my questions are as follows:
\nIt will be nice if you make these points and ensembling a bit more clear.
", "rawMarkdown": "Hi @masato114, thanks for all the clarification for the training. I really appreciate your help and i am able to\nsuccessfully replicate your training scripts for all the four cases.\n\nNow after training, following is the version number of all the models and their corresponding architectures:\n\n| Configs | Breast/Laterity | Dataset type | Model type | Model version |\n|--------------|-----------------|-------------------------------------------------|-----------------|---------------|\n| config0.yaml | Breast Level | Kaggle Data | Efficientnet_b5 | model_ver_084 |\n| config1.yaml | Breast Level | Kaggle + External Data | Efficientnet_b2 | model_ver_084 |\n| config2.yaml | Lateral Level | Kaggle + External Data | Efficientnet_b2 | model_ver_085 |\n| config3.yaml | Breast Level | Kaggle + External Data (with Sigmoid windowing) | Efficientnet_b2 | model_ver_099 |\n| config4.yaml | Lateral Level | Kaggle + External Data (with Sigmoid windowing) | Efficientnet_b2 | model_ver_100 | \n\nNow when I am trying to replicate the code for ensemble as per https://www.kaggle.com/code/masato114/2stage-ensemble/notebook, in the 20^th notebook, you are using the model kaggle/input/compile-ver49-83-83/ver49-{fold}.ts' to fill df['prediction'] data.\n\nThen in 23rd notebook for breast level, /kaggle/input/efficientnetv2s-exp084-tensorrt/efficientnetv2s_ver084-seed10-{fold}-swa.ts, /kaggle/input/efficientnetv2s-exp084-tensorrt/efficientnetv2s_ver084-seed660-{fold}-swa.ts and /kaggle/input/rsna-efficientnetv2s-pseudo-sig-tensorrt/efficientnetv2s_ver099-seed10-{fold}.ts are used to predict test_preds_1, test_preds_2 and test_preds_3 respectively. These 3 values are ensembled to calculate second_df['prediction_ver84'].\n\nSimilary in 29^th notebook for laterality Level, /kaggle/input/efficientnetv2s-exp085-tensorrt/efficientnetv2s_ver085-{fold}.ts, kaggle/input/efficientnetv2s-exp085-tensorrt/efficientnetv2s_ver085-seed660-{fold}.ts, and /kaggle/input/rsna-efficientnetv2s-pseudo-sig-tensorrt/efficientnetv2s_ver100-seed10-{fold}.ts are used predict test_preds_1, test_preds_2 and test_preds_3 respectively. These 3 values are ensembled to calculate second_df['prediction_ver85'].\n\nNow my questions are as follows:\n1. How did you get model with version 049?\n2. To get the models with seed660 did you again train on breast and lateral level with config1 and config2? Is there any reason to use seed660?\n3. It means you only use efficientnetb5 to get the pseudo labels? And, you did not use efficientnetb5 for ensembling?\n\nIt will be nice if you make these points and ensembling a bit more clear." }, { "id": 2333483, "postDate": "2023-07-07T03:00:54.890Z", "content": "In short,
\nGreat. Thanks for your help. Happy to collaborate.
", "rawMarkdown": "Great. Thanks for your help. Happy to collaborate." } ] } ] }, { "id": 2164074, "postDate": "2023-03-01T09:30:50.307Z", "content": "\n\nThrough data analysis, we observed that the data contained many explicit negative samples which did not require prediction by a strong model or ensemble. By filtering out those data with the threshold of 0.01, we were able to reduce the number of data (to 25%) which need to be saved to disk and predicted without losing accuracy.
\n
Congratulations and thank you for sharing your work! May I know what data analysis you applied to filter the test (hidden) images?
", "rawMarkdown": ">Through data analysis, we observed that the data contained many explicit negative samples which did not require prediction by a strong model or ensemble. By filtering out those data with the threshold of 0.01, we were able to reduce the number of data (to 25%) which need to be saved to disk and predicted without losing accuracy.\n\nCongratulations and thank you for sharing your work! May I know what data analysis you applied to filter the test (hidden) images?", "replies": [ { "id": 2164931, "postDate": "2023-03-01T21:41:33.947Z", "content": "Thank you for your question.
\nI did an analysis against oof predictions like this.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9268451%2F980fe355177fd722e585bd5d7e5dcae2%2Fanasysis.png?generation=1677707481367684&alt=media\")
\n\nThrough data analysis, we observed that the data contained many explicit negative samples which did not require prediction by a strong model or ensemble. By filtering out those data with the threshold of 0.01, we were able to reduce the number of data (to 25%) which need to be saved to disk and predicted without losing accuracy.
\n
That is very inspiring👍
", "rawMarkdown": ">Through data analysis, we observed that the data contained many explicit negative samples which did not require prediction by a strong model or ensemble. By filtering out those data with the threshold of 0.01, we were able to reduce the number of data (to 25%) which need to be saved to disk and predicted without losing accuracy.\n\nThat is very inspiring👍" }, { "id": 2163005, "postDate": "2023-02-28T14:27:02.817Z", "content": "Thanks for the thorough explanation! 👍 Was DALI used for the augmentation also? Elastic deformation can be quite resource consuming, did you have your own implementation for that one?
", "rawMarkdown": "Thanks for the thorough explanation! 👍 Was DALI used for the augmentation also? Elastic deformation can be quite resource consuming, did you have your own implementation for that one?", "replies": [ { "id": 2163071, "postDate": "2023-02-28T15:11:12.577Z", "content": "Thanks. Once images were loaded with DALI and saved in storage, DALI was no longer used, even for augmentation. Elastic deformation was implemented by albumentations.augmentations.geometric.transforms.ElasticTransform with alpha=10 and sigma=15.
Great work and congrats!
\n\"The models trained with the laterality-level was much weaker (-0.04) than those with the breast-level, but the ensemble of the two levels was very effective.\"
\nI have the first observation too but I just stopped there… 😑
", "votes": 1, "replies": [ { "id": 2165561, "author_name": "Mt.Panda", "author_url": "", "post_date": "2023-03-02T09:23:04.330000", "content": "Of course I considered discarding that weaker models, but I thought surely the lower covariance between breast- and laterality-level would be beneficial to the ensemble, and I was reassured by the fact that the scores were up in LB😏
", "votes": 1, "replies": [] } ] }, { "id": 2263351, "author_name": "Shantanu Ghosh", "author_url": "", "post_date": "2023-05-17T14:21:00.723000", "content": "Hello @masato114 i checked your code. So in the instruction after training with rsna images (step 1), creating pseudo labels (step 2), you instructed the below step 3 instruction:
\nChange the config file as follows according to your purpose.
\nFor breast-level/external dataset/no windowing: config1.yaml
\nFor laterality-level/external dataest/no windowing: config2.yaml
\nFor breast-level/external dataset/windowing: config3.yaml
\nFor laterality-level/external dataset/windowing: config4.yaml
So my question is: for step 3 i will use train.py with the above configs right? Also do i need to finetune with lateral with vindr and then breast with vindr as shown in the figure or i can finetune them in any order and ensemble later?
", "votes": 0, "replies": [ { "id": 2265147, "author_name": "Mt.Panda", "author_url": "", "post_date": "2023-05-19T02:49:34.330000", "content": "Hi, thanks for your comment.
\nYes, you can use train.py with the configs you indicated for step 3. Once you successfully generated pseudolabels on vindr, which are labeled for each breast of each image, you can make lateral-level labels by aggregating them on both patient_id and laterality. This should be done before proceeding to step 3.
\nAs you are saying, you can finetune them in any order and ensemble later.
", "votes": 0, "replies": [ { "id": 2265182, "author_name": "Shantanu Ghosh", "author_url": "", "post_date": "2023-05-19T03:50:09.807000", "content": "@masato114 Thanks for the reply. My further question is:
\n--This should be done before proceeding to step 3.
\nSo is this the pseudo label generation step as mentioned in your github as:
\nConduct pseudolabeling on external dataset.
\npython -u src/external_pseudolabeling.py configs/config0.yaml
These are the steps from your github readme.md:
\nIn short, are the 3 steps specified in the github readme.md file are sufficient right? or u want any intermediate step as for generating lateral-level labels by aggregating them on both patient_id and laterality? Can you please clarify where to put any extra steps in the above-mentioned three steps as i am seeing
\nyou have generated the lateral labels in the external_pseudolabeling.py file (in VinDrMammo_breast-level_annotations.csv)? Can u please confirm?
Hi, I apologize for making you confused.
\nThere are no extra steps between step 2 and 3. But perhaps you need to rename columns of vindr annotation file so that they are the same as of train.csv. I may have overlooked something about vindr whose patient column is named as 'study_id' not 'patient_id', I guess.
\nIf you have hit errors, please kindly share the concrete problems.
Thanks a lot..
", "votes": 0, "replies": [] }, { "id": 2294255, "author_name": "", "author_url": "", "post_date": "2023-06-09T23:12:39.713000", "content": "", "votes": 0, "replies": [] }, { "id": 2305764, "author_name": "", "author_url": "", "post_date": "2023-06-16T22:39:44.253000", "content": "", "votes": 0, "replies": [] }, { "id": 2305778, "author_name": "", "author_url": "", "post_date": "2023-06-16T22:48:08.277000", "content": "", "votes": 0, "replies": [] }, { "id": 2305791, "author_name": "Mt.Panda", "author_url": "", "post_date": "2023-06-16T23:05:07.107000", "content": "Yes, I did the same process as what is applied to test images in inference notebook.
", "votes": 0, "replies": [] }, { "id": 2305800, "author_name": "Shantanu Ghosh", "author_url": "", "post_date": "2023-06-16T23:22:50.380000", "content": "Hi @masato114 , did u perform the same preprocessing for vindr dataset as u did for kaggle? can u please share the preprocessing for vindr dataset?
", "votes": 0, "replies": [] }, { "id": 2305811, "author_name": "Mt.Panda", "author_url": "", "post_date": "2023-06-16T23:30:55.130000", "content": "You can find it in https://www.kaggle.com/code/masato114/rsna-generate-train-images/notebook.
", "votes": 0, "replies": [] }, { "id": 2305832, "author_name": "", "author_url": "", "post_date": "2023-06-16T23:57:39.507000", "content": "", "votes": 0, "replies": [] }, { "id": 2305854, "author_name": "", "author_url": "", "post_date": "2023-06-17T00:33:03.157000", "content": "", "votes": 0, "replies": [] }, { "id": 2305968, "author_name": "Shantanu Ghosh", "author_url": "", "post_date": "2023-06-17T03:18:22.597000", "content": "@masato114 the TransferSyntaxUID is 1.2.840.10008.1.2.1, it is different from the RSNA dataset. So, if condition of convert_dicom_to_j2k is not executed for vindr and format is also .dicom for vindr instead of .dcm for rsna. So either u have commented the if check for PhotometricInterpretation and TransferSyntaxUID. Can u please check?
\nAlso because Vindr images have different formats, the dali is throwing an error jp2:
\n[/opt/dali/dali/imgcodec/image_decoder.cc:448] Cannot parse the image: /ocean/projects/asc170022p/shg121/PhD/RSNA_Breast_Imaging/Dataset/External/Vindr/vindr-mammo-a-large-scale-benchmark-dataset-for-computer-aided-detection-and-diagnosis-in-full-field-digital-mammography-1.0.0/tmp/j2k/42ec6b7233c3e984cec02a1dd702dcb0_33c3590bb1738c0b9512630d9741bf65.jp2
The temporary jp2 image is actually not forming up correctly.
\nSo, i am doing like this:
\nimport cv2
\nSIZE = (912, 1520)
\npixel_data = ds.pixel_array
\nnormalized_data = ((pixel_data - np.min(pixel_data)) / np.ptp(pixel_data) * 255).astype(np.uint8)
\nimg = cv2.resize(normalized_data, SIZE, interpolation=cv2.INTER_AREA)
\nplt.imshow(img, cmap=plt.cm.bone)
\nplt.show()
Try the following.
\n ():\n img_copy = img.copy()\n img = np.where(img <= , , img) \n height, _ = img.shape\n\n \n y_a = height // + (height*)\n y_b = height // - (height*)\n b_arr = img[y_b:y_a].std(axis=) != \n continuing_ones = CountUpContinuingOnes(b_arr)\n \n col_ind = np.where(continuing_ones == continuing_ones.())[]\n img = img[:, col_ind]\n\n \n _, width = img.shape\n x_a = width // + (width*)\n x_b = width // - (width*)\n b_arr = img[:,x_b:x_a].std(axis=) != \n continuing_ones = CountUpContinuingOnes(b_arr)\n \n row_ind = np.where(continuing_ones == continuing_ones.())[]\n\n img_copy[row_ind][:, col_ind]\n\n ():\n dicom = dicomsdl.(in_path)\n data = dicom.pixelData()\n data = data[:-, :-]\n dicom.getPixelDataInfo()[] == :\n data = np.amax(data) - data\n\n data = data - np.(data)\n data = data / np.(data)\n data = (data * ).astype(np.uint8)\n\n img = ExtractBreast(data)\n img = cv2.resize(img, SIZE, interpolation = cv2.INTER_AREA)\n cv2.imwrite(out_path, img)\nGreat thanks @masato114
", "votes": 0, "replies": [] }, { "id": 2333448, "author_name": "Shantanu Ghosh", "author_url": "", "post_date": "2023-07-07T01:44:45.600000", "content": "Hi @masato114, thanks for all the clarification for the training. I really appreciate your help and i am able to
\nsuccessfully replicate your training scripts for all the four cases.
Now after training, following is the version number of all the models and their corresponding architectures:
\n| Configs | \nBreast/Laterity | \nDataset type | \nModel type | \nModel version | \n
|---|---|---|---|---|
| config0.yaml | \nBreast Level | \nKaggle Data | \nEfficientnet_b5 | \nmodel_ver_084 | \n
| config1.yaml | \nBreast Level | \nKaggle + External Data | \nEfficientnet_b2 | \nmodel_ver_084 | \n
| config2.yaml | \nLateral Level | \nKaggle + External Data | \nEfficientnet_b2 | \nmodel_ver_085 | \n
| config3.yaml | \nBreast Level | \nKaggle + External Data (with Sigmoid windowing) | \nEfficientnet_b2 | \nmodel_ver_099 | \n
| config4.yaml | \nLateral Level | \nKaggle + External Data (with Sigmoid windowing) | \nEfficientnet_b2 | \nmodel_ver_100 | \n
Now when I am trying to replicate the code for ensemble as per https://www.kaggle.com/code/masato114/2stage-ensemble/notebook, in the 20^th notebook, you are using the model kaggle/input/compile-ver49-83-83/ver49-{fold}.ts' to fill df['prediction'] data.
\nThen in 23rd notebook for breast level, /kaggle/input/efficientnetv2s-exp084-tensorrt/efficientnetv2s_ver084-seed10-{fold}-swa.ts, /kaggle/input/efficientnetv2s-exp084-tensorrt/efficientnetv2s_ver084-seed660-{fold}-swa.ts and /kaggle/input/rsna-efficientnetv2s-pseudo-sig-tensorrt/efficientnetv2s_ver099-seed10-{fold}.ts are used to predict test_preds_1, test_preds_2 and test_preds_3 respectively. These 3 values are ensembled to calculate second_df['prediction_ver84'].
\nSimilary in 29^th notebook for laterality Level, /kaggle/input/efficientnetv2s-exp085-tensorrt/efficientnetv2s_ver085-{fold}.ts, kaggle/input/efficientnetv2s-exp085-tensorrt/efficientnetv2s_ver085-seed660-{fold}.ts, and /kaggle/input/rsna-efficientnetv2s-pseudo-sig-tensorrt/efficientnetv2s_ver100-seed10-{fold}.ts are used predict test_preds_1, test_preds_2 and test_preds_3 respectively. These 3 values are ensembled to calculate second_df['prediction_ver85'].
\nNow my questions are as follows:
\nIt will be nice if you make these points and ensembling a bit more clear.
", "votes": 0, "replies": [] }, { "id": 2333483, "author_name": "Mt.Panda", "author_url": "", "post_date": "2023-07-07T03:00:54.890000", "content": "In short,
\nGreat. Thanks for your help. Happy to collaborate.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2164074, "author_name": "Chua Jia Qing Isaiah", "author_url": "", "post_date": "2023-03-01T09:30:50.307000", "content": "\n\nThrough data analysis, we observed that the data contained many explicit negative samples which did not require prediction by a strong model or ensemble. By filtering out those data with the threshold of 0.01, we were able to reduce the number of data (to 25%) which need to be saved to disk and predicted without losing accuracy.
\n
Congratulations and thank you for sharing your work! May I know what data analysis you applied to filter the test (hidden) images?
", "votes": 0, "replies": [ { "id": 2164931, "author_name": "luddite^", "author_url": "", "post_date": "2023-03-01T21:41:33.947000", "content": "Thank you for your question.
\nI did an analysis against oof predictions like this.
\n
\n\nThrough data analysis, we observed that the data contained many explicit negative samples which did not require prediction by a strong model or ensemble. By filtering out those data with the threshold of 0.01, we were able to reduce the number of data (to 25%) which need to be saved to disk and predicted without losing accuracy.
\n
That is very inspiring👍
", "votes": 0, "replies": [] }, { "id": 2163005, "author_name": "Antti Isosalo", "author_url": "", "post_date": "2023-02-28T14:27:02.817000", "content": "Thanks for the thorough explanation! 👍 Was DALI used for the augmentation also? Elastic deformation can be quite resource consuming, did you have your own implementation for that one?
", "votes": 0, "replies": [ { "id": 2163071, "author_name": "Mt.Panda", "author_url": "", "post_date": "2023-02-28T15:11:12.577000", "content": "Thanks. Once images were loaded with DALI and saved in storage, DALI was no longer used, even for augmentation. Elastic deformation was implemented by albumentations.augmentations.geometric.transforms.ElasticTransform with alpha=10 and sigma=15.