Training source code: https://github.com/louis-she/rsna-2022-public
\nInfernece notebook: https://www.kaggle.com/code/snaker/rsna-infer-t4-threshold?scriptVersionId=120550686
Thank everyone who contributed to the competition and thanks Kaggle for hosting.
\nMy solution didn't involve complicated pipeline, just as most does, single image classification. Here are some settings you may be interested in:
\nI realized that all the label tagged for laterality + patient_id is the same. That means the label is actually for patient + laterality, not image. So the easiest way to deal with this, is to train OOF and drop False Negatives with high confidence( e.g. the label is 1 but model predict probability is like 0.1 or 0.01). The CV score boost huge after this is done( about 0.03 to 0.04 ).
The performance matters a lot in the competition. I'm happy that I take almost one month to optimize the inference pipeline, and I really learned something, like open-sourced the nvjpeg2k-python ( first time writing some cuda code, it feels good ). To utilze the maximum of the hardware, I use the 2 x T4 kernel and some multiprocessing tricks. You can see that in the inference notebook.
\nI use the percentile threshold trick I mentioned in the thread, cause it boost my LB score about ~0.015 . I know that in this way I will have a very high chance to overfit LB but I still chose it as one of my last selection sub, because it is just too good on LB… The result of manual threshold without percentile is 0.61 , the 0.63 score of LB is so attractive that I left everything behind.
\nCongrats to all the winners! And thanks for the great community, let's meet in another competition.
", "messages": [ { "id": 2162323, "postDate": "2023-02-28T06:04:23.740Z", "content": "Training source code: https://github.com/louis-she/rsna-2022-public
\nInfernece notebook: https://www.kaggle.com/code/snaker/rsna-infer-t4-threshold?scriptVersionId=120550686
Thank everyone who contributed to the competition and thanks Kaggle for hosting.
\nMy solution didn't involve complicated pipeline, just as most does, single image classification. Here are some settings you may be interested in:
\nI realized that all the label tagged for laterality + patient_id is the same. That means the label is actually for patient + laterality, not image. So the easiest way to deal with this, is to train OOF and drop False Negatives with high confidence( e.g. the label is 1 but model predict probability is like 0.1 or 0.01). The CV score boost huge after this is done( about 0.03 to 0.04 ).
The performance matters a lot in the competition. I'm happy that I take almost one month to optimize the inference pipeline, and I really learned something, like open-sourced the nvjpeg2k-python ( first time writing some cuda code, it feels good ). To utilze the maximum of the hardware, I use the 2 x T4 kernel and some multiprocessing tricks. You can see that in the inference notebook.
\nI use the percentile threshold trick I mentioned in the thread, cause it boost my LB score about ~0.015 . I know that in this way I will have a very high chance to overfit LB but I still chose it as one of my last selection sub, because it is just too good on LB… The result of manual threshold without percentile is 0.61 , the 0.63 score of LB is so attractive that I left everything behind.
\nCongrats to all the winners! And thanks for the great community, let's meet in another competition.
", "rawMarkdown": "Training source code: https://github.com/louis-she/rsna-2022-public\nInfernece notebook: https://www.kaggle.com/code/snaker/rsna-infer-t4-threshold?scriptVersionId=120550686\n\nThank everyone who contributed to the competition and thanks Kaggle for hosting.\n\nMy solution didn't involve complicated pipeline, just as most does, single image classification. Here are some settings you may be interested in:\n\n* **Augmentation** RandomRotate90, HorizontalFlip, VerticalFlip\n* **Models and Resolutions** \n 1. ConvnextV2_nano @ 1536\n 2. ConvnextV2_nano_another_seed @ 1536\n 3. ConvnextV2_nano @ 2048\n 4. EfficientnetV2_s @ 1536\n* **Optimizer and Scheduler** Adam with 3e-5 + OneCycleLR\n* **Epochs** 33\n* **Agg** by maximum\n* Aux Losses: no\n* Tablet Features: no\n* SWA: no\n* External Dataset: no\n\n## The only trick: drop false positives\n\nI realized that all the label tagged for `laterality + patient_id` is the same. That means the label is actually for `patient + laterality`, not image. So the easiest way to deal with this, is to train OOF and drop False Negatives with high confidence( e.g. the label is 1 but model predict probability is like 0.1 or 0.01). The CV score boost huge after this is done( about 0.03 to 0.04 ).\n\n## Things I have learned\n\nThe performance matters a lot in the competition. I'm happy that I take almost one month to optimize the inference pipeline, and I really learned something, like open-sourced the [nvjpeg2k-python](https://github.com/louis-she/nvjpeg2k-python) ( first time writing some cuda code, it feels good ). To utilze the maximum of the hardware, I use the 2 x T4 kernel and some multiprocessing tricks. You can see that in the inference notebook.\n\nI use the percentile threshold trick I mentioned in the thread, cause it boost my LB score about ~0.015 . I know that in this way I will have a very high chance to overfit LB but I still chose it as one of my last selection sub, because it is just too good on LB... The result of manual threshold without percentile is 0.61 , the 0.63 score of LB is so attractive that I left everything behind.\n\n------------------------------------------\n\nCongrats to all the winners! And thanks for the great community, let's meet in another competition.\n", "votes": 28 }, { "id": 2162597, "postDate": "2023-02-28T10:01:01.740Z", "content": "I was looking for the trick, The only trick: drop false positives to remove incorrect positive labels, for that I tried Multiple Instance Learning, Triplet Loss, Prototype Net, Saliency Map Inconsistency, but never thought it could be as simple as what you did! 🙂
\nThanks for sharing the trick! It is another take away from this competition for me.
Glad I can help… Some words lists by you are unfamiliar to me, I'll have a look at these new techs, thanks for sharing.
", "rawMarkdown": "Glad I can help... Some words lists by you are unfamiliar to me, I'll have a look at these new techs, thanks for sharing.", "votes": 1 } ] }, { "id": 2162373, "postDate": "2023-02-28T06:51:51.230Z", "content": "Congrats! And thanks for open-sourcing the nvjpeg2k-python!
", "rawMarkdown": "Congrats! And thanks for open-sourcing the nvjpeg2k-python!", "votes": 1 }, { "id": 2163596, "postDate": "2023-02-28T23:51:00.653Z", "content": "Thank you for sharing a great solution!
\nI don't understand 'drop false positive'. Do you mean you drop false positive in train data, and train another model from scratch with train data without false positive?
\nI would appreciate it if you answer the question.
", "rawMarkdown": "Thank you for sharing a great solution!\n\nI don't understand 'drop false positive'. Do you mean you drop false positive in train data, and train another model from scratch with train data without false positive? \n\nI would appreciate it if you answer the question.", "replies": [ { "id": 2163633, "postDate": "2023-03-01T00:57:48.120Z", "content": "\"Do you mean you drop false negative in train data, and train another model from scratch without train data without false negative\"
\nthat's correct, train model without the FPs, validation should use all the data.
", "rawMarkdown": "\"Do you mean you drop false negative in train data, and train another model from scratch without train data without false negative\"\n\nthat's correct, train model without the FPs, validation should use all the data.", "votes": 1, "replies": [ { "id": 2163714, "postDate": "2023-03-01T02:43:17.400Z", "content": "Thank you for answering! Would I ask one more question?
\nI would like you to explain why you decided to remove false positives in train data.
\nBecause I was trying to make the model's prediction of false positive data close to zero, I couldn't think of removing false positive in train data.
\nI was wondering what's the effect of removing false positive in train data.
remove is based on a fact: label is tagged on laterality + patient_id, not on every image. If one of the images which has the same laterality + patient_id is positive, then all the image belongs to the same laterality + patient_id are marked as positive. So there must be many FPs in the dataset.
Thank you very much!!
", "rawMarkdown": "Thank you very much!!" } ] } ] }, { "id": 2162850, "postDate": "2023-02-28T12:53:39.033Z", "content": "Thanks for sharing, I have one question, how did you not overfit with 33 epochs? My models would overfit near the 6th-7th epoch usually. I find it interesting since you also applied very few augmentations, perhaps is the architecture?
", "rawMarkdown": "Thanks for sharing, I have one question, how did you not overfit with 33 epochs? My models would overfit near the 6th-7th epoch usually. I find it interesting since you also applied very few augmentations, perhaps is the architecture?", "replies": [ { "id": 2162856, "postDate": "2023-02-28T12:58:09.830Z", "content": "To my understanding he didn't use pretrained weights (as they are not allowed for commercial use).
", "rawMarkdown": "To my understanding he didn't use pretrained weights (as they are not allowed for commercial use)." }, { "id": 2162999, "postDate": "2023-02-28T14:16:43.430Z", "content": "huh i missed something here.. one epoch for me does not mean the whole dataset, instead, it is number_of_positives * (negtive_ratios + 1), I normally chose negative_ratios as 4 to 6. So here one epochs means all the positive samples and some of the negative samples, some = number_of_positives * negative_ratios
Thanks for overview your solution. Did you use original data for training stage or some cropped/preprocessed version?
", "rawMarkdown": "Thanks for overview your solution. Did you use original data for training stage or some cropped/preprocessed version?", "replies": [ { "id": 2162776, "postDate": "2023-02-28T12:05:24.937Z", "content": "image = utils.crop_roi(image)\nlong_edge = (image.shape[:])\npad_fn = A.PadIfNeeded(long_edge, long_edge, border_mode=cv2.BORDER_CONSTANT, position= self.training , value=, always_apply=, p=)\nfor utils.crop_roi, you can find the source here: https://github.com/louis-she/rsna-2022-public/blob/03e9c542b756cb16b8ee6de2bb79d1b2bd346e1f/utils.py#L24
I was looking for the trick, The only trick: drop false positives to remove incorrect positive labels, for that I tried Multiple Instance Learning, Triplet Loss, Prototype Net, Saliency Map Inconsistency, but never thought it could be as simple as what you did! 🙂
\nThanks for sharing the trick! It is another take away from this competition for me.
Glad I can help… Some words lists by you are unfamiliar to me, I'll have a look at these new techs, thanks for sharing.
", "votes": 1, "replies": [] } ] }, { "id": 2162373, "author_name": "Antti Isosalo", "author_url": "", "post_date": "2023-02-28T06:51:51.230000", "content": "Congrats! And thanks for open-sourcing the nvjpeg2k-python!
", "votes": 1, "replies": [] }, { "id": 2163596, "author_name": "Ju7on9", "author_url": "", "post_date": "2023-02-28T23:51:00.653000", "content": "Thank you for sharing a great solution!
\nI don't understand 'drop false positive'. Do you mean you drop false positive in train data, and train another model from scratch with train data without false positive?
\nI would appreciate it if you answer the question.
", "votes": 0, "replies": [ { "id": 2163633, "author_name": "Chenglu", "author_url": "", "post_date": "2023-03-01T00:57:48.120000", "content": "\"Do you mean you drop false negative in train data, and train another model from scratch without train data without false negative\"
\nthat's correct, train model without the FPs, validation should use all the data.
", "votes": 1, "replies": [ { "id": 2163714, "author_name": "Ju7on9", "author_url": "", "post_date": "2023-03-01T02:43:17.400000", "content": "Thank you for answering! Would I ask one more question?
\nI would like you to explain why you decided to remove false positives in train data.
\nBecause I was trying to make the model's prediction of false positive data close to zero, I couldn't think of removing false positive in train data.
\nI was wondering what's the effect of removing false positive in train data.
remove is based on a fact: label is tagged on laterality + patient_id, not on every image. If one of the images which has the same laterality + patient_id is positive, then all the image belongs to the same laterality + patient_id are marked as positive. So there must be many FPs in the dataset.
Thank you very much!!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2162850, "author_name": "moth", "author_url": "", "post_date": "2023-02-28T12:53:39.033000", "content": "Thanks for sharing, I have one question, how did you not overfit with 33 epochs? My models would overfit near the 6th-7th epoch usually. I find it interesting since you also applied very few augmentations, perhaps is the architecture?
", "votes": 0, "replies": [ { "id": 2162856, "author_name": "Rasoul Mojtahedzadeh", "author_url": "", "post_date": "2023-02-28T12:58:09.830000", "content": "To my understanding he didn't use pretrained weights (as they are not allowed for commercial use).
", "votes": 0, "replies": [] }, { "id": 2162999, "author_name": "Chenglu", "author_url": "", "post_date": "2023-02-28T14:16:43.430000", "content": "huh i missed something here.. one epoch for me does not mean the whole dataset, instead, it is number_of_positives * (negtive_ratios + 1), I normally chose negative_ratios as 4 to 6. So here one epochs means all the positive samples and some of the negative samples, some = number_of_positives * negative_ratios
Thanks for overview your solution. Did you use original data for training stage or some cropped/preprocessed version?
", "votes": 0, "replies": [ { "id": 2162776, "author_name": "Chenglu", "author_url": "", "post_date": "2023-02-28T12:05:24.937000", "content": "image = utils.crop_roi(image)\nlong_edge = (image.shape[:])\npad_fn = A.PadIfNeeded(long_edge, long_edge, border_mode=cv2.BORDER_CONSTANT, position= self.training , value=, always_apply=, p=)\nfor utils.crop_roi, you can find the source here: https://github.com/louis-she/rsna-2022-public/blob/03e9c542b756cb16b8ee6de2bb79d1b2bd346e1f/utils.py#L24