From the HOST CONFIRMED - Label Consistency Check notebook I see that:
\n\n\nIf there is at least one image per
\nStudyInstanceUIDwithpe_present_on_image> 0.5, then…
refers to situations in which we believe the study indicates positive for PE.
\nI'm unsure about this though. What if all N images in the study have pe_present_on_image = 0.49. Then, assuming independence of images, the probability that the study indicates negative for PE is (1-0.49)^N, which with just N=2 becomes around 0.25. And I assume you now see where I'm going with this.
So I don't think it's necessary that at least one image to have pe_present_on_image > 0.5 for us to conclude that the study has a more than 0.5 chance of indicating PE. Therefore I think it would make sense to be able to specify things like RV/LV ratio, the location of the PE, or the nature of the condition (chronic/acute).
From the HOST CONFIRMED - Label Consistency Check notebook I see that:
\n\n\nIf there is at least one image per
\nStudyInstanceUIDwithpe_present_on_image> 0.5, then…
refers to situations in which we believe the study indicates positive for PE.
\nI'm unsure about this though. What if all N images in the study have pe_present_on_image = 0.49. Then, assuming independence of images, the probability that the study indicates negative for PE is (1-0.49)^N, which with just N=2 becomes around 0.25. And I assume you now see where I'm going with this.
So I don't think it's necessary that at least one image to have pe_present_on_image > 0.5 for us to conclude that the study has a more than 0.5 chance of indicating PE. Therefore I think it would make sense to be able to specify things like RV/LV ratio, the location of the PE, or the nature of the condition (chronic/acute).
I agree with @alexandersoare. These consistency rules add unnecessary complexity to the problem. I don't find it contradictory that a model can identify a suspicious image to have >50% chance of a PE, but when reviewing the entire stack of images as a whole conclude that the study is negative for the abnormality. What this inadvertently will cause is that people will manually write codes on top of model outputs, which will paradoxically make AI outputs even more difficult to explain.
", "rawMarkdown": "I agree with @alexandersoare. These consistency rules add unnecessary complexity to the problem. I don't find it contradictory that a model can identify a suspicious image to have >50% chance of a PE, but when reviewing the entire stack of images as a whole conclude that the study is negative for the abnormality. What this inadvertently will cause is that people will manually write codes on top of model outputs, which will paradoxically make AI outputs even more difficult to explain. ", "votes": 4, "replies": [ { "id": 1050729, "postDate": "2020-10-15T17:06:24.543Z", "content": "@yeeseng when trying to met label consistency on above of the AI model than getting higher loss. Did you get your current leaderboard loss with or without meeting label consistency requirements?
", "rawMarkdown": "@yeeseng when trying to met label consistency on above of the AI model than getting higher loss. Did you get your current leaderboard loss with or without meeting label consistency requirements?" } ] }, { "id": 1057689, "postDate": "2020-10-22T21:49:38.867Z", "content": "These issues were previously raised and discussed. @richardepstein nicely summarized the rationale for the rules as currently written.
", "rawMarkdown": "These issues were previously raised and [discussed](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473). @richardepstein nicely [summarized the rationale](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473#1014525) for the rules as currently written.", "replies": [ { "id": 1058035, "postDate": "2020-10-23T09:11:03.080Z", "content": "Hi @anthracene thanks for the response. I've read that summary and I agree. What I'm referring to is how we define \"medically consistent\". I think that if all images in a study have 49% predicted probability of having PE present, then it is medically inconsistent to then have to say that the overall study has more than 50% chance of being negative for PE.
", "rawMarkdown": "Hi @anthracene thanks for the response. I've read that summary and I agree. What I'm referring to is how we define \"medically consistent\". I think that if all images in a study have 49% predicted probability of having PE present, then it is medically **inconsistent** to then have to say that the overall study has more than 50% chance of being negative for PE." } ] }, { "id": 1047006, "postDate": "2020-10-12T07:13:30.780Z", "content": "@alexandersoare may be mean could make sense as all images belong to same CT ..
", "rawMarkdown": "@alexandersoare may be mean could make sense as all images belong to same CT .." } ], "comments": [ { "id": 1045249, "author_name": "Yee Ng", "author_url": "", "post_date": "2020-10-10T13:05:02.353000", "content": "I agree with @alexandersoare. These consistency rules add unnecessary complexity to the problem. I don't find it contradictory that a model can identify a suspicious image to have >50% chance of a PE, but when reviewing the entire stack of images as a whole conclude that the study is negative for the abnormality. What this inadvertently will cause is that people will manually write codes on top of model outputs, which will paradoxically make AI outputs even more difficult to explain.
", "votes": 4, "replies": [ { "id": 1050729, "author_name": "SumanSudhir", "author_url": "", "post_date": "2020-10-15T17:06:24.543000", "content": "@yeeseng when trying to met label consistency on above of the AI model than getting higher loss. Did you get your current leaderboard loss with or without meeting label consistency requirements?
", "votes": 0, "replies": [] } ] }, { "id": 1057689, "author_name": "John Mongan", "author_url": "", "post_date": "2020-10-22T21:49:38.867000", "content": "These issues were previously raised and discussed. @richardepstein nicely summarized the rationale for the rules as currently written.
", "votes": 0, "replies": [ { "id": 1058035, "author_name": "Alexander Soare", "author_url": "", "post_date": "2020-10-23T09:11:03.080000", "content": "Hi @anthracene thanks for the response. I've read that summary and I agree. What I'm referring to is how we define \"medically consistent\". I think that if all images in a study have 49% predicted probability of having PE present, then it is medically inconsistent to then have to say that the overall study has more than 50% chance of being negative for PE.
", "votes": 0, "replies": [] } ] }, { "id": 1047006, "author_name": "Jaideep", "author_url": "", "post_date": "2020-10-12T07:13:30.780000", "content": "@alexandersoare may be mean could make sense as all images belong to same CT ..
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1045184": "From the [HOST CONFIRMED - Label Consistency Check](https://www.kaggle.com/anthracene/host-confirmed-label-consistency-check) notebook I see that:\n\n> If there is at least one image per `StudyInstanceUID` with `pe_present_on_image` > 0.5, then...\n\nrefers to situations in which we believe the study indicates positive for PE.\n\nI'm unsure about this though. What if all N images in the study have `pe_present_on_image` = 0.49. Then, assuming independence of images, the probability that the study indicates negative for PE is (1-0.49)^N, which with just N=2 becomes around 0.25. And I assume you now see where I'm going with this.\n\nSo I don't think it's necessary that at least one image to have `pe_present_on_image` > 0.5 for us to conclude that the study has a more than 0.5 chance of indicating PE. Therefore I think it would make sense to be able to specify things like RV/LV ratio, the location of the PE, or the nature of the condition (chronic/acute).", "1045249": "I agree with @alexandersoare. These consistency rules add unnecessary complexity to the problem. I don't find it contradictory that a model can identify a suspicious image to have >50% chance of a PE, but when reviewing the entire stack of images as a whole conclude that the study is negative for the abnormality. What this inadvertently will cause is that people will manually write codes on top of model outputs, which will paradoxically make AI outputs even more difficult to explain. ", "1057689": "These issues were previously raised and [discussed](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473). @richardepstein nicely [summarized the rationale](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473#1014525) for the rules as currently written.", "1047006": "@alexandersoare may be mean could make sense as all images belong to same CT .." } }