{ "id": 388735, "title": "Beware of cases like these!", "url": "/competitions/rsna-breast-cancer-detection/discussion/388735", "author_name": "Anil Thomas", "post_date": "2023-02-19T08:35:59.315000", "votes": 36, "comment_count": 17, "views": 0, "content": "

The ground truth may not be as reliable as we think. If a breast is identified as positive, all of its images are tagged as positive. Even in cases where a view entirely missed the tumor… This is especially problematic if your model is trained with image-level labels. If you use multi-instance learning, you're probably fine.

\n

As an example, let’s look at patient 11094, who has cancer on the left side. In image 1882170663, a tumor is clearly visible, while in image 1926447510, it cannot be seen.

\n

As I am not a radiologist, I rely on a trained model to locate the tumor. I have attached the saliency map for the first image below. This was produced on the Akridata Data Explorer platform that was discussed in another thread (consider this a sneak preview of model analysis features that we have in the pipeline).
\n\"\"

\n

In the second image (below), the tumor seems to be out of the X-ray’s field of view. I guess this has a high chance of happening if the tumor is close to the rib cage.
\n\"\"

\n

Patient 28989 would be another example. The view shown below has the tumor visible.

\n

\"\"

\n

In one of the other views of the same breast, the tumor pointed out by the saliency map is not present:
\n\"\"

\n

Training with such examples can cause the model to associate irrelevant patterns with a positive label. When hard labels are used, even a few bad labels can have an impact.

\n

Other than MIL that combines images of each breast, I don’t know if there is a good solution to this. It seems hard to weed out such bad examples automatically. I tried looking for discrepancies between CC and MLO predictions in this notebook. A difference in predicted probabilities does not seem to be a great indicator of this issue.

", "messages": [ { "id": 2150417, "postDate": "2023-02-19T08:35:59.317Z", "content": "

The ground truth may not be as reliable as we think. If a breast is identified as positive, all of its images are tagged as positive. Even in cases where a view entirely missed the tumor… This is especially problematic if your model is trained with image-level labels. If you use multi-instance learning, you're probably fine.

\n

As an example, let’s look at patient 11094, who has cancer on the left side. In image 1882170663, a tumor is clearly visible, while in image 1926447510, it cannot be seen.

\n

As I am not a radiologist, I rely on a trained model to locate the tumor. I have attached the saliency map for the first image below. This was produced on the Akridata Data Explorer platform that was discussed in another thread (consider this a sneak preview of model analysis features that we have in the pipeline).
\n\"\"

\n

In the second image (below), the tumor seems to be out of the X-ray’s field of view. I guess this has a high chance of happening if the tumor is close to the rib cage.
\n\"\"

\n

Patient 28989 would be another example. The view shown below has the tumor visible.

\n

\"\"

\n

In one of the other views of the same breast, the tumor pointed out by the saliency map is not present:
\n\"\"

\n

Training with such examples can cause the model to associate irrelevant patterns with a positive label. When hard labels are used, even a few bad labels can have an impact.

\n

Other than MIL that combines images of each breast, I don’t know if there is a good solution to this. It seems hard to weed out such bad examples automatically. I tried looking for discrepancies between CC and MLO predictions in this notebook. A difference in predicted probabilities does not seem to be a great indicator of this issue.

", "rawMarkdown": "The ground truth may not be as reliable as we think. If a breast is identified as positive, all of its images are tagged as positive. Even in cases where a view entirely missed the tumor… This is especially problematic if your model is trained with image-level labels. If you use multi-instance learning, you're probably fine. \n\nAs an example, let’s look at patient 11094, who has cancer on the left side. In image 1882170663, a tumor is clearly visible, while in image 1926447510, it cannot be seen.\n\nAs I am not a radiologist, I rely on a trained model to locate the tumor. I have attached the saliency map for the first image below. This was produced on the [Akridata Data Explorer](https://akridata.ai/data-explorer/) platform that was discussed in [another thread](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/383466) (consider this a sneak preview of model analysis features that we have in the pipeline).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F70039698959cd9e8fce83f7517d4a869%2Fdownload6.png?generation=1676795073862040&alt=media)\n\nIn the second image (below), the tumor seems to be out of the X-ray’s field of view. I guess this has a high chance of happening if the tumor is close to the rib cage.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F9b3d28c9773d2dde1b7ab1b58dc0b7c4%2Fdownload7.png?generation=1676795134432432&alt=media)\n\nPatient 28989 would be another example. The view shown below has the tumor visible.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F33fc649b37aa90c856ca21059955dc0d%2Fdownload10.png?generation=1676795171616908&alt=media)\n\nIn one of the other views of the same breast, the tumor pointed out by the saliency map is not present:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2Fd7c5cd61fb0c6f87ace448d2c4564bad%2Fdownload11.png?generation=1676795194856271&alt=media)\n\nTraining with such examples can cause the model to associate irrelevant patterns with a positive label. When hard labels are used, even a few bad labels can have an impact.\n\nOther than MIL that combines images of each breast, I don’t know if there is a good solution to this. It seems hard to weed out such bad examples automatically. I tried looking for [discrepancies between CC and MLO predictions in this notebook](https://www.kaggle.com/code/anlthms/rsna-mammogram-model-error-analysis). A difference in predicted probabilities does not seem to be a great indicator of this issue.", "votes": 36 }, { "id": 2150710, "postDate": "2023-02-19T14:15:22.823Z", "content": "

I am a general radiologist, not a breast specialist, and I attempted to evaluate whether a tumor was present in the images. However, it was very challenging. In many cases, tumors are very small and difficult to localize in multi-view images especially in dense breast.

", "rawMarkdown": "I am a general radiologist, not a breast specialist, and I attempted to evaluate whether a tumor was present in the images. However, it was very challenging. In many cases, tumors are very small and difficult to localize in multi-view images especially in dense breast.", "votes": 7, "replies": [ { "id": 2156051, "postDate": "2023-02-23T04:28:55.340Z", "content": "

It's an interesting story that although you are a general doctor, it's difficult to determine clearing the position of tumors on breast images.. The description makes me understand why this screening decision is longer than other procedure.

", "rawMarkdown": "It's an interesting story that although you are a general doctor, it's difficult to determine clearing the position of tumors on breast images.. The description makes me understand why this screening decision is longer than other procedure.", "replies": [ { "id": 2157170, "postDate": "2023-02-23T18:37:09.240Z", "content": "

Isn't it simply because the tumors are not in those images? And that's why you are having a hard time finding them?

\n

If you look at both examples above, the top pair of images shows a saliency map focused on a point near the edge of the image, and the bottom image appears to have that point cropped out.

\n

So in sets of images where the tumor is near the edge of the images, you run the risk of some of the angles not including the tumor.

", "rawMarkdown": "Isn't it simply because the tumors are not in those images? And that's why you are having a hard time finding them?\n\nIf you look at both examples above, the top pair of images shows a saliency map focused on a point near the edge of the image, and the bottom image appears to have that point cropped out.\n\nSo in sets of images where the tumor is near the edge of the images, you run the risk of some of the angles not including the tumor." }, { "id": 2157176, "postDate": "2023-02-23T18:49:53.443Z", "content": "

@drluke Always assuming that we trust this model's saliency and that there isn't any other spot in the image that reveals cancer I suppose.

", "rawMarkdown": "@drluke Always assuming that we trust this model's saliency and that there isn't any other spot in the image that reveals cancer I suppose." } ] } ] }, { "id": 2157257, "postDate": "2023-02-23T21:01:19.533Z", "content": "

A few other cases that may (or may not!) indicate bad labels:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
#Image & saliency mapPossibly negative view of the same breast
1\"\"\"\"
2\"\"\"\"
3\"\"\"\"
4\"\"\"\"
5\"\"\"\"
", "rawMarkdown": "A few other cases that may (or may not!) indicate bad labels:\n\n|#| Image & saliency map | Possibly negative view of the same breast|\n|---| --- | --- |\n|1|![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F3f3d86ae69398b94ba8f43e95c060b20%2Fdownload3.png?generation=1677184780179259&alt=media) |![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F51ea1af49ba6dadb546f307819de0b3c%2Fdownload4.png?generation=1677184799364633&alt=media) |\n|2|![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F2310330f062f1eb262ab6440879d24ec%2Fdownload5.png?generation=1677184946857984&alt=media) | ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F55b6bbe4676aa9d4ea3bb1da50449e43%2Fdownload20.png?generation=1677184994972413&alt=media)|\n|3|![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F9e5816698d175f81ceb8c88b8f418f20%2Fdownload0.png?generation=1677185118620924&alt=media) |![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F6bf310e066a442d3eaeb15d254857214%2Fdownload1.png?generation=1677185143648860&alt=media)|\n|4| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2Fe021fdcaf60da300e39cc2a480be412a%2FScreenshot%20from%202023-02-21%2013-20-32.png?generation=1677185200569721&alt=media)| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F85431e29a9d382c74a1daa2665802376%2FScreenshot%20from%202023-02-21%2013-20-47.png?generation=1677185977520449&alt=media)|\n|5|![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F7030f932d5bc5b75654d004b3287ec60%2F43615_214035779.jpg?generation=1677185771013418&alt=media)| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F6214898a423ac16cc83058b8e1ba6a8e%2F43615_1907428658.jpg?generation=1677186001496130&alt=media)|\n", "votes": 3, "replies": [ { "id": 2157543, "postDate": "2023-02-24T05:31:37.040Z", "content": "

if you really want to check the accuary of CAM heatmap, you need ground truth.
\nuse external data mentioned in the furm with pixel ground truth for that.

\n

run you model on these external data, you should see your model is quite accurate for some cases

", "rawMarkdown": "if you really want to check the accuary of CAM heatmap, you need ground truth.\nuse external data mentioned in the furm with pixel ground truth for that.\n\nrun you model on these external data, you should see your model is quite accurate for some cases\n", "votes": 2 } ] }, { "id": 2154463, "postDate": "2023-02-22T03:09:43.643Z", "content": "

thanks for sharing your finding. how did you find those images ? i also looked your notebook https://www.kaggle.com/code/anlthms/rsna-mammogram-model-error-analysis, is there a way ( code ) to search the images, do a reverse prediction on these images to find the possible images which will miss guide the model training ( build wrong pattern ) ?

", "rawMarkdown": "thanks for sharing your finding. how did you find those images ? i also looked your notebook https://www.kaggle.com/code/anlthms/rsna-mammogram-model-error-analysis, is there a way ( code ) to search the images, do a reverse prediction on these images to find the possible images which will miss guide the model training ( build wrong pattern ) ?", "votes": 1, "replies": [ { "id": 2154609, "postDate": "2023-02-22T06:15:06.073Z", "content": "

That notebook is a good starting point. If you feed it a CSV file from model validation (example), it will list images with labels that are suspect. It works by pulling out cases where different views of the same breast lead to very different predictions. Once you have a shortlist of such cases, you would need some sort of saliency map generation to confirm.

", "rawMarkdown": "That notebook is a good starting point. If you feed it a CSV file from model validation ([example](https://www.kaggle.com/code/anlthms/rsna-mammogram-model-error-analysis/data?select=validation-fold0.csv)), it will list images with labels that are suspect. It works by pulling out cases where different views of the same breast lead to very different predictions. Once you have a shortlist of such cases, you would need some sort of [saliency map generation](https://github.com/jacobgil/pytorch-grad-cam) to confirm.", "votes": 3, "replies": [ { "id": 2154751, "postDate": "2023-02-22T08:25:32.630Z", "content": "

thank you very much for the explanation, i looked that validation file, was thinking that file may take some time to build ( didn't check if they are related to the data provided by the host ) and thanks for the pointer, that is what i am looking for, feel much more fun from that end.

", "rawMarkdown": "thank you very much for the explanation, i looked that validation file, was thinking that file may take some time to build ( didn't check if they are related to the data provided by the host ) and thanks for the pointer, that is what i am looking for, feel much more fun from that end. ", "votes": 1 } ] } ] }, { "id": 2155988, "postDate": "2023-02-23T03:09:40.123Z", "content": "

\"Other than MIL that combines images of each breast, I don’t know if there is a good solution to this.\"
\nrelabel each image in the set

", "rawMarkdown": "\"Other than MIL that combines images of each breast, I don’t know if there is a good solution to this.\"\nrelabel each image in the set ", "votes": 2 }, { "id": 2154909, "postDate": "2023-02-22T10:16:49.320Z", "content": "

thank you. did you train excluding some of them? and if yes, did you see any significant difference in your cv / lb ?

", "rawMarkdown": "thank you. did you train excluding some of them? and if yes, did you see any significant difference in your cv / lb ?", "votes": 2, "replies": [ { "id": 2155915, "postDate": "2023-02-23T01:30:23.107Z", "content": "

Good question! I haven't had a chance to experiment much. Let me see if I can find a few more mislabeled examples, clean the dataset and retrain.

", "rawMarkdown": "Good question! I haven't had a chance to experiment much. Let me see if I can find a few more mislabeled examples, clean the dataset and retrain.", "votes": 3, "replies": [ { "id": 2157209, "postDate": "2023-02-23T19:35:58.667Z", "content": "

I have visually inspected all positive images, and excluded those that had inconsistent saliency map. For me it didn't improve the results (it actually reduced the LB score by 0.01). I might have removed some of the good positive cases, though.

", "rawMarkdown": "I have visually inspected all positive images, and excluded those that had inconsistent **saliency map**. For me it didn't improve the results (it actually reduced the LB score by 0.01). I might have removed some of the good positive cases, though.", "votes": 2 }, { "id": 2157228, "postDate": "2023-02-23T20:01:06.123Z", "content": "

thank you for letting us know!

", "rawMarkdown": "thank you for letting us know!" }, { "id": 2157510, "postDate": "2023-02-24T04:38:42.543Z", "content": "

I compared both cases and the results are inconclusive. The original learning curves look like these:
\n\"\"

\n

And this is what the curves look like after cleaning:
\n\"\"

\n

The loss values are noisy and the differences could simply be attributed to randomness.

\n

For this experiment, I removed these images from the training set.
\n\n11094_1926447510.png\n28989_1223569992.png\n16668_1015929339.png\n21867_831671840.png\n60653_2052987229.png\n19750_684882869.png\n
\nMaybe this is too few to make a difference?

", "rawMarkdown": "I compared both cases and the results are inconclusive. The original learning curves look like these:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2Fc1f4dacfb5552b8bfa7781b32e996a20%2Floss-curves1.png?generation=1677212851036713&alt=media)\n\nAnd this is what the curves look like after cleaning:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F5531e8d6f2dc7e38007ae72d613dec30%2Floss-curves2.png?generation=1677212916148562&alt=media)\n\nThe loss values are noisy and the differences could simply be attributed to randomness.\n\nFor this experiment, I removed these images from the training set. \n``\n11094_1926447510.png\n28989_1223569992.png\n16668_1015929339.png\n21867_831671840.png\n60653_2052987229.png\n19750_684882869.png\n``\nMaybe this is too few to make a difference? ", "votes": 1 }, { "id": 2157704, "postDate": "2023-02-24T08:41:19.983Z", "content": "

@anlthms
\nThank you for your nice sharing
\nI guess these removal make the CV fold data change unless you fix valid data.
\nthese data change effect the valid loss.

\n

anyway, did these removal worked for LB score?

", "rawMarkdown": "@anlthms \nThank you for your nice sharing\nI guess these removal make the CV fold data change unless you fix valid data.\nthese data change effect the valid loss.\n\nanyway, did these removal worked for LB score?", "votes": 1 } ] } ] }, { "id": 2150465, "postDate": "2023-02-19T09:37:37.250Z", "content": "

Thanks for your information. Fortunately, our data does not have much positive data. But, as likely you say, we need to find something solution to improve our job. At this time, I can only think about upsampling the orignal data and getting the new one as being distorted.

", "rawMarkdown": "Thanks for your information. Fortunately, our data does not have much positive data. But, as likely you say, we need to find something solution to improve our job. At this time, I can only think about upsampling the orignal data and getting the new one as being distorted." } ], "comments": [ { "id": 2150710, "author_name": "ECO", "author_url": "", "post_date": "2023-02-19T14:15:22.823000", "content": "

I am a general radiologist, not a breast specialist, and I attempted to evaluate whether a tumor was present in the images. However, it was very challenging. In many cases, tumors are very small and difficult to localize in multi-view images especially in dense breast.

", "votes": 7, "replies": [ { "id": 2156051, "author_name": "Hyunsoo Lee 1010", "author_url": "", "post_date": "2023-02-23T04:28:55.340000", "content": "

It's an interesting story that although you are a general doctor, it's difficult to determine clearing the position of tumors on breast images.. The description makes me understand why this screening decision is longer than other procedure.

", "votes": 0, "replies": [ { "id": 2157170, "author_name": "Luke Hornof", "author_url": "", "post_date": "2023-02-23T18:37:09.240000", "content": "

Isn't it simply because the tumors are not in those images? And that's why you are having a hard time finding them?

\n

If you look at both examples above, the top pair of images shows a saliency map focused on a point near the edge of the image, and the bottom image appears to have that point cropped out.

\n

So in sets of images where the tumor is near the edge of the images, you run the risk of some of the angles not including the tumor.

", "votes": 0, "replies": [] }, { "id": 2157176, "author_name": "Eleftherios Fanioudakis", "author_url": "", "post_date": "2023-02-23T18:49:53.443000", "content": "

@drluke Always assuming that we trust this model's saliency and that there isn't any other spot in the image that reveals cancer I suppose.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2157257, "author_name": "Anil Thomas", "author_url": "", "post_date": "2023-02-23T21:01:19.533000", "content": "

A few other cases that may (or may not!) indicate bad labels:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
#Image & saliency mapPossibly negative view of the same breast
1\"\"\"\"
2\"\"\"\"
3\"\"\"\"
4\"\"\"\"
5\"\"\"\"
", "votes": 3, "replies": [ { "id": 2157543, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-24T05:31:37.040000", "content": "

if you really want to check the accuary of CAM heatmap, you need ground truth.
\nuse external data mentioned in the furm with pixel ground truth for that.

\n

run you model on these external data, you should see your model is quite accurate for some cases

", "votes": 2, "replies": [] } ] }, { "id": 2154463, "author_name": "Kefan Xu", "author_url": "", "post_date": "2023-02-22T03:09:43.643000", "content": "

thanks for sharing your finding. how did you find those images ? i also looked your notebook https://www.kaggle.com/code/anlthms/rsna-mammogram-model-error-analysis, is there a way ( code ) to search the images, do a reverse prediction on these images to find the possible images which will miss guide the model training ( build wrong pattern ) ?

", "votes": 1, "replies": [ { "id": 2154609, "author_name": "Anil Thomas", "author_url": "", "post_date": "2023-02-22T06:15:06.073000", "content": "

That notebook is a good starting point. If you feed it a CSV file from model validation (example), it will list images with labels that are suspect. It works by pulling out cases where different views of the same breast lead to very different predictions. Once you have a shortlist of such cases, you would need some sort of saliency map generation to confirm.

", "votes": 3, "replies": [ { "id": 2154751, "author_name": "Kefan Xu", "author_url": "", "post_date": "2023-02-22T08:25:32.630000", "content": "

thank you very much for the explanation, i looked that validation file, was thinking that file may take some time to build ( didn't check if they are related to the data provided by the host ) and thanks for the pointer, that is what i am looking for, feel much more fun from that end.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2155988, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-23T03:09:40.123000", "content": "

\"Other than MIL that combines images of each breast, I don’t know if there is a good solution to this.\"
\nrelabel each image in the set

", "votes": 2, "replies": [] }, { "id": 2154909, "author_name": "Eleftherios Fanioudakis", "author_url": "", "post_date": "2023-02-22T10:16:49.320000", "content": "

thank you. did you train excluding some of them? and if yes, did you see any significant difference in your cv / lb ?

", "votes": 2, "replies": [ { "id": 2155915, "author_name": "Anil Thomas", "author_url": "", "post_date": "2023-02-23T01:30:23.107000", "content": "

Good question! I haven't had a chance to experiment much. Let me see if I can find a few more mislabeled examples, clean the dataset and retrain.

", "votes": 3, "replies": [ { "id": 2157209, "author_name": "Rasoul Mojtahedzadeh", "author_url": "", "post_date": "2023-02-23T19:35:58.667000", "content": "

I have visually inspected all positive images, and excluded those that had inconsistent saliency map. For me it didn't improve the results (it actually reduced the LB score by 0.01). I might have removed some of the good positive cases, though.

", "votes": 2, "replies": [] }, { "id": 2157228, "author_name": "Eleftherios Fanioudakis", "author_url": "", "post_date": "2023-02-23T20:01:06.123000", "content": "

thank you for letting us know!

", "votes": 0, "replies": [] }, { "id": 2157510, "author_name": "Anil Thomas", "author_url": "", "post_date": "2023-02-24T04:38:42.543000", "content": "

I compared both cases and the results are inconclusive. The original learning curves look like these:
\n\"\"

\n

And this is what the curves look like after cleaning:
\n\"\"

\n

The loss values are noisy and the differences could simply be attributed to randomness.

\n

For this experiment, I removed these images from the training set.
\n\n11094_1926447510.png\n28989_1223569992.png\n16668_1015929339.png\n21867_831671840.png\n60653_2052987229.png\n19750_684882869.png\n
\nMaybe this is too few to make a difference?

", "votes": 1, "replies": [] }, { "id": 2157704, "author_name": "taruto", "author_url": "", "post_date": "2023-02-24T08:41:19.983000", "content": "

@anlthms
\nThank you for your nice sharing
\nI guess these removal make the CV fold data change unless you fix valid data.
\nthese data change effect the valid loss.

\n

anyway, did these removal worked for LB score?

", "votes": 1, "replies": [] } ] } ] }, { "id": 2150465, "author_name": "Hyunsoo Lee 1010", "author_url": "", "post_date": "2023-02-19T09:37:37.250000", "content": "

Thanks for your information. Fortunately, our data does not have much positive data. But, as likely you say, we need to find something solution to improve our job. At this time, I can only think about upsampling the orignal data and getting the new one as being distorted.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2150417": "The ground truth may not be as reliable as we think. If a breast is identified as positive, all of its images are tagged as positive. Even in cases where a view entirely missed the tumor… This is especially problematic if your model is trained with image-level labels. If you use multi-instance learning, you're probably fine. \n\nAs an example, let’s look at patient 11094, who has cancer on the left side. In image 1882170663, a tumor is clearly visible, while in image 1926447510, it cannot be seen.\n\nAs I am not a radiologist, I rely on a trained model to locate the tumor. I have attached the saliency map for the first image below. This was produced on the [Akridata Data Explorer](https://akridata.ai/data-explorer/) platform that was discussed in [another thread](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/383466) (consider this a sneak preview of model analysis features that we have in the pipeline).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F70039698959cd9e8fce83f7517d4a869%2Fdownload6.png?generation=1676795073862040&alt=media)\n\nIn the second image (below), the tumor seems to be out of the X-ray’s field of view. I guess this has a high chance of happening if the tumor is close to the rib cage.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F9b3d28c9773d2dde1b7ab1b58dc0b7c4%2Fdownload7.png?generation=1676795134432432&alt=media)\n\nPatient 28989 would be another example. The view shown below has the tumor visible.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F33fc649b37aa90c856ca21059955dc0d%2Fdownload10.png?generation=1676795171616908&alt=media)\n\nIn one of the other views of the same breast, the tumor pointed out by the saliency map is not present:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2Fd7c5cd61fb0c6f87ace448d2c4564bad%2Fdownload11.png?generation=1676795194856271&alt=media)\n\nTraining with such examples can cause the model to associate irrelevant patterns with a positive label. When hard labels are used, even a few bad labels can have an impact.\n\nOther than MIL that combines images of each breast, I don’t know if there is a good solution to this. It seems hard to weed out such bad examples automatically. I tried looking for [discrepancies between CC and MLO predictions in this notebook](https://www.kaggle.com/code/anlthms/rsna-mammogram-model-error-analysis). A difference in predicted probabilities does not seem to be a great indicator of this issue.", "2150710": "I am a general radiologist, not a breast specialist, and I attempted to evaluate whether a tumor was present in the images. However, it was very challenging. In many cases, tumors are very small and difficult to localize in multi-view images especially in dense breast.", "2157257": "A few other cases that may (or may not!) indicate bad labels:\n\n|#| Image & saliency map | Possibly negative view of the same breast|\n|---| --- | --- |\n|1|![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F3f3d86ae69398b94ba8f43e95c060b20%2Fdownload3.png?generation=1677184780179259&alt=media) |![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F51ea1af49ba6dadb546f307819de0b3c%2Fdownload4.png?generation=1677184799364633&alt=media) |\n|2|![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F2310330f062f1eb262ab6440879d24ec%2Fdownload5.png?generation=1677184946857984&alt=media) | ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F55b6bbe4676aa9d4ea3bb1da50449e43%2Fdownload20.png?generation=1677184994972413&alt=media)|\n|3|![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F9e5816698d175f81ceb8c88b8f418f20%2Fdownload0.png?generation=1677185118620924&alt=media) |![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F6bf310e066a442d3eaeb15d254857214%2Fdownload1.png?generation=1677185143648860&alt=media)|\n|4| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2Fe021fdcaf60da300e39cc2a480be412a%2FScreenshot%20from%202023-02-21%2013-20-32.png?generation=1677185200569721&alt=media)| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F85431e29a9d382c74a1daa2665802376%2FScreenshot%20from%202023-02-21%2013-20-47.png?generation=1677185977520449&alt=media)|\n|5|![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F7030f932d5bc5b75654d004b3287ec60%2F43615_214035779.jpg?generation=1677185771013418&alt=media)| ![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7837%2F6214898a423ac16cc83058b8e1ba6a8e%2F43615_1907428658.jpg?generation=1677186001496130&alt=media)|\n", "2154463": "thanks for sharing your finding. how did you find those images ? i also looked your notebook https://www.kaggle.com/code/anlthms/rsna-mammogram-model-error-analysis, is there a way ( code ) to search the images, do a reverse prediction on these images to find the possible images which will miss guide the model training ( build wrong pattern ) ?", "2155988": "\"Other than MIL that combines images of each breast, I don’t know if there is a good solution to this.\"\nrelabel each image in the set ", "2154909": "thank you. did you train excluding some of them? and if yes, did you see any significant difference in your cv / lb ?", "2150465": "Thanks for your information. Fortunately, our data does not have much positive data. But, as likely you say, we need to find something solution to improve our job. At this time, I can only think about upsampling the orignal data and getting the new one as being distorted." } }