I made a submission with all the predictions as '-' and got back a score of 0
\nIf the dice coefficient is 1.0 when the predicted mask and the ground truth is empty, and there are some samples in the hidden test set which contain empty masks, then my score would not have been zero. So I'm asking to confirm,
\nWhat happens when the predicted mask and the ground truth is empty?
I made a submission with all the predictions as '-' and got back a score of 0
\nIf the dice coefficient is 1.0 when the predicted mask and the ground truth is empty, and there are some samples in the hidden test set which contain empty masks, then my score would not have been zero. So I'm asking to confirm,
\nWhat happens when the predicted mask and the ground truth is empty?
If you look at the metric definition at https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/overview/evaluation, you see that the numerator is the size of the intersection of X and Y, where X is the set of predicted contrail pixels and Y is the set of groundtruth contrail pixels. In your case X is the empty set, so the size of the intersection will be 0.
", "rawMarkdown": "If you look at the metric definition at https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/overview/evaluation, you see that the numerator is the size of the intersection of X and Y, where X is the set of predicted contrail pixels and Y is the set of groundtruth contrail pixels. In your case X is the empty set, so the size of the intersection will be 0.", "replies": [ { "id": 2273405, "postDate": "2023-05-25T06:29:42.137Z", "content": "Thanks, I understand it now.
\nOne question, does this mean that,
\nif 10% of the hidden test set are empty masks, then the maximum possible score is 0.9 ?
\nAnd if 20% of the hidden test set are empty masks, then the maximum possible score is 0.8 ?
\nTherefore the upper limit score of this competition is not 1.0, and it is determined by the number of empty masks in the hidden test set, right?
No, that's not correct. If you predict exactly the correct set of pixels your score will be 1.0. The number of empty masks doesn't affect that other than that if you predict contrail pixels where the groundtruth mask is empty then your score will go down.
", "rawMarkdown": "No, that's not correct. If you predict exactly the correct set of pixels your score will be 1.0. The number of empty masks doesn't affect that other than that if you predict contrail pixels where the groundtruth mask is empty then your score will go down.", "votes": 1 }, { "id": 2274798, "postDate": "2023-05-26T08:55:51.027Z", "content": "This is how I calculated:
\nLet's assume there are 1000 samples, among which 900 contain contrails and 100 contain no contrail.
\nAnd let's assume we have a model that can predict pixels as contrails with 100% accuracy.
Then the dice coefficient on 900 samples will be 1.0
\nAnd the dice coefficient on 100 samples will be 0 as they contain no contrail so there is no intersection between the predicted contrail and the ground truth.
So the average score is = (900*1.0 + 100*0)/1000 = 0.90
\nConclusion: A model that is 100% accurate will not get 1.0 LB score because of empty masks present in the hidden test set
\nIs this wrong? Can you please identify where did I make a mistake?
", "rawMarkdown": "This is how I calculated:\n\nLet's assume there are 1000 samples, among which 900 contain contrails and 100 contain no contrail.\nAnd let's assume we have a model that can predict pixels as contrails with 100% accuracy.\n\nThen the dice coefficient on 900 samples will be 1.0 \nAnd the dice coefficient on 100 samples will be 0 as they contain no contrail so there is no intersection between the predicted contrail and the ground truth.\n\nSo the average score is = (900\\*1.0 + 100\\*0)/1000 = 0.90\n\nConclusion: A model that is 100% accurate will not get 1.0 LB score because of empty masks present in the hidden test set\n\nIs this wrong? Can you please identify where did I make a mistake?" }, { "id": 2277517, "postDate": "2023-05-27T20:44:33.087Z", "content": "Hi,
\nI thought about it too, and along with the host's comment I think the way it works is that if there is a sample that doesn't have contrails, it cannot contribute to the numerator or the denominator as Y.
\nHowever, if you predict a contrail where there isn't one via X, the value will only contribute to the denominator, reducing your score. if you don't predict anything via X, you don't contribute neither to the numerator or the denominator, leaving the potential maximum score at 1.0
Perhaps the issue with your example is that you take each sample individually, whereas the metric is specified as global (so you effectively sum all the samples together)
\nLet me know if there's anything I missed here.
", "rawMarkdown": "Hi,\nI thought about it too, and along with the host's comment I think the way it works is that if there is a sample that doesn't have contrails, it cannot contribute to the numerator or the denominator as Y. \nHowever, if you predict a contrail where there isn't one via X, the value will only contribute to the denominator, reducing your score. if you don't predict anything via X, you don't contribute neither to the numerator or the denominator, leaving the potential maximum score at 1.0\n\nPerhaps the issue with your example is that you take each sample individually, whereas the metric is specified as global (so you effectively sum all the samples together)\n\nLet me know if there's anything I missed here.", "votes": 1 }, { "id": 2277618, "postDate": "2023-05-27T23:03:23.663Z", "content": "@raistorx Thanks for the reply, that makes sense.
\n@aaronsarna Can you please confirm if this is correct or not?
Generally, @raistorx is correct.
\nThe one point that I'd state even more explicitly is that the metric does not consider samples at all. The score is computed globally over the pixels in the entire dataset. Y is just the set of all groundtruth contrail pixels in the entire dataset, and X is the set of all predicted contrail pixels in the entire dataset. There is no averaging over scores per-sample. It might help you to think about it as each pixel in the dataset being its own independent sample from the metric's perspective.
", "rawMarkdown": "Generally, @raistorx is correct.\n\nThe one point that I'd state even more explicitly is that the metric does not consider samples at all. The score is computed globally over the *pixels* in the entire dataset. Y is just the set of all groundtruth contrail pixels in the entire dataset, and X is the set of all predicted contrail pixels in the entire dataset. There is no averaging over scores per-sample. It might help you to think about it as each pixel in the dataset being its own independent sample from the metric's perspective.", "votes": 4 }, { "id": 2281054, "postDate": "2023-05-30T14:42:20.247Z", "content": "@aaronsarna That clears all my doubts, thanks.
", "rawMarkdown": "@aaronsarna That clears all my doubts, thanks." } ] } ] } ], "comments": [ { "id": 2272898, "author_name": "Aaron Sarna", "author_url": "", "post_date": "2023-05-24T20:27:57.453000", "content": "If you look at the metric definition at https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/overview/evaluation, you see that the numerator is the size of the intersection of X and Y, where X is the set of predicted contrail pixels and Y is the set of groundtruth contrail pixels. In your case X is the empty set, so the size of the intersection will be 0.
", "votes": 0, "replies": [ { "id": 2273405, "author_name": "MD Mushfirat Mohaimin", "author_url": "", "post_date": "2023-05-25T06:29:42.137000", "content": "Thanks, I understand it now.
\nOne question, does this mean that,
\nif 10% of the hidden test set are empty masks, then the maximum possible score is 0.9 ?
\nAnd if 20% of the hidden test set are empty masks, then the maximum possible score is 0.8 ?
\nTherefore the upper limit score of this competition is not 1.0, and it is determined by the number of empty masks in the hidden test set, right?
No, that's not correct. If you predict exactly the correct set of pixels your score will be 1.0. The number of empty masks doesn't affect that other than that if you predict contrail pixels where the groundtruth mask is empty then your score will go down.
", "votes": 1, "replies": [] }, { "id": 2274798, "author_name": "MD Mushfirat Mohaimin", "author_url": "", "post_date": "2023-05-26T08:55:51.027000", "content": "This is how I calculated:
\nLet's assume there are 1000 samples, among which 900 contain contrails and 100 contain no contrail.
\nAnd let's assume we have a model that can predict pixels as contrails with 100% accuracy.
Then the dice coefficient on 900 samples will be 1.0
\nAnd the dice coefficient on 100 samples will be 0 as they contain no contrail so there is no intersection between the predicted contrail and the ground truth.
So the average score is = (900*1.0 + 100*0)/1000 = 0.90
\nConclusion: A model that is 100% accurate will not get 1.0 LB score because of empty masks present in the hidden test set
\nIs this wrong? Can you please identify where did I make a mistake?
", "votes": 0, "replies": [] }, { "id": 2277517, "author_name": "Alex B", "author_url": "", "post_date": "2023-05-27T20:44:33.087000", "content": "Hi,
\nI thought about it too, and along with the host's comment I think the way it works is that if there is a sample that doesn't have contrails, it cannot contribute to the numerator or the denominator as Y.
\nHowever, if you predict a contrail where there isn't one via X, the value will only contribute to the denominator, reducing your score. if you don't predict anything via X, you don't contribute neither to the numerator or the denominator, leaving the potential maximum score at 1.0
Perhaps the issue with your example is that you take each sample individually, whereas the metric is specified as global (so you effectively sum all the samples together)
\nLet me know if there's anything I missed here.
", "votes": 1, "replies": [] }, { "id": 2277618, "author_name": "MD Mushfirat Mohaimin", "author_url": "", "post_date": "2023-05-27T23:03:23.663000", "content": "@raistorx Thanks for the reply, that makes sense.
\n@aaronsarna Can you please confirm if this is correct or not?
Generally, @raistorx is correct.
\nThe one point that I'd state even more explicitly is that the metric does not consider samples at all. The score is computed globally over the pixels in the entire dataset. Y is just the set of all groundtruth contrail pixels in the entire dataset, and X is the set of all predicted contrail pixels in the entire dataset. There is no averaging over scores per-sample. It might help you to think about it as each pixel in the dataset being its own independent sample from the metric's perspective.
", "votes": 4, "replies": [] }, { "id": 2281054, "author_name": "MD Mushfirat Mohaimin", "author_url": "", "post_date": "2023-05-30T14:42:20.247000", "content": "@aaronsarna That clears all my doubts, thanks.
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2272799": "I made a submission with all the predictions as '-' and got back a score of 0\n\nIf the dice coefficient is 1.0 when the predicted mask and the ground truth is empty, and there are some samples in the hidden test set which contain empty masks, then my score would not have been zero. So I'm asking to confirm,\nWhat happens when the predicted mask and the ground truth is empty?", "2272898": "If you look at the metric definition at https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/overview/evaluation, you see that the numerator is the size of the intersection of X and Y, where X is the set of predicted contrail pixels and Y is the set of groundtruth contrail pixels. In your case X is the empty set, so the size of the intersection will be 0." } }