![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2062758%2Fe3103107aac5e8a998650368fc03a800%2Ff.png?generation=1603738226066418&alt=media\")
I think many participants have a temptation to select a sub having higher LB but violating the label assignment rules, especially if they are far from the gold zone. Even if u see \"Winning submissions will be inspected to ensure label predictions adhere to the expected label hierarchy defined by the diagram on the Data page\", it does not mean that kaggle cannot check all selected submissions, it is very simple to do. And what will happen next? Check the rules:
\n\n\nCompetition Sponsor reserves the right to disqualify any participant who makes conflicting label predictions that do not adhere to the expected label hierarchy defined by the diagram on the Data page. This includes:
\n
\n Submissions that make study predictions which conflict with that study's image predictions.
\n Submissions that make conflicting or disallowed sub-label predictions (i.e. simultaneously predicting RV/LV ratio >=1 and <1, more than one label for PE type, etc.)
In other words entire team may be disqualified for cheating if even one selected sub is violating the rules, and there is no need to be in top 10 for this. I expect kaggle would try to make this competition fair, and for us it is better to play a fair game too.
", "messages": [ { "id": 1061097, "postDate": "2020-10-26T18:31:45.143Z", "content": "I think many participants have a temptation to select a sub having higher LB but violating the label assignment rules, especially if they are far from the gold zone. Even if u see \"Winning submissions will be inspected to ensure label predictions adhere to the expected label hierarchy defined by the diagram on the Data page\", it does not mean that kaggle cannot check all selected submissions, it is very simple to do. And what will happen next? Check the rules:
\n\n\nCompetition Sponsor reserves the right to disqualify any participant who makes conflicting label predictions that do not adhere to the expected label hierarchy defined by the diagram on the Data page. This includes:
\n
\n Submissions that make study predictions which conflict with that study's image predictions.
\n Submissions that make conflicting or disallowed sub-label predictions (i.e. simultaneously predicting RV/LV ratio >=1 and <1, more than one label for PE type, etc.)
In other words entire team may be disqualified for cheating if even one selected sub is violating the rules, and there is no need to be in top 10 for this. I expect kaggle would try to make this competition fair, and for us it is better to play a fair game too.
", "rawMarkdown": "I think many participants have a temptation to select a sub having higher LB but violating the label assignment rules, especially if they are far from the gold zone. Even if u see \"Winning submissions will be inspected to ensure label predictions adhere to the expected label hierarchy defined by the diagram on the Data page\", it does not mean that kaggle cannot check all selected submissions, it is very simple to do. And what will happen next? Check the rules:\n\n> Competition **Sponsor reserves the right to disqualify any participant who makes conflicting label predictions** that do not adhere to the expected label hierarchy defined by the diagram on the Data page. This includes:\n> Submissions that make study predictions which conflict with that study's image predictions.\nSubmissions that make conflicting or disallowed sub-label predictions (i.e. simultaneously predicting RV/LV ratio >=1 and <1, more than one label for PE type, etc.)\n\n**In other words entire team may be disqualified for cheating if even one selected sub is violating the rules, and there is no need to be in top 10 for this**. I expect kaggle would try to make this competition fair, and for us it is better to play a fair game too.", "votes": 15 }, { "id": 1061156, "postDate": "2020-10-26T19:19:29.450Z", "content": "How much LB difference do people observe with and without applying rules? For me, my CV decreases 0.002 but my LB stays the same. For my final two subs, I will be choosing submissions that conform. But I realize now that that may decrease my private LB by 0.002 š
", "rawMarkdown": "How much LB difference do people observe with and without applying rules? For me, my CV decreases 0.002 but my LB stays the same. For my final two subs, I will be choosing submissions that conform. But I realize now that that may decrease my private LB by 0.002 š", "votes": 5, "replies": [ { "id": 1061162, "postDate": "2020-10-26T19:23:16.637Z", "content": "On the only submission that ran without timeout I got - 0.003 LB (- 0.002 CV).
\n0.002 won't play a big role for either of us though, we're quite far from the gold zone
", "rawMarkdown": "On the only submission that ran without timeout I got - 0.003 LB (- 0.002 CV).\n\n0.002 won't play a big role for either of us though, we're quite far from the gold zone", "votes": 5 }, { "id": 1061181, "postDate": "2020-10-26T19:38:58.230Z", "content": "Yes @chris000, we also experienced a similar thing. In local CV not much drop -+ 0.001 and in LB around +-0.002/3.
", "rawMarkdown": "Yes @chris000, we also experienced a similar thing. In local CV not much drop -+ 0.001 and in LB around +-0.002/3.", "votes": 1 }, { "id": 1061190, "postDate": "2020-10-26T19:44:45.570Z", "content": "From that I saw the difference is getting larger if the performance of the original model is worse. For example my very first sub in this competition got 0.24/0.27, but right now I see the difference of only <~0.01, and even better models wd likely give 0.002, like u wrote.
", "rawMarkdown": "From that I saw the difference is getting larger if the performance of the original model is worse. For example my very first sub in this competition got 0.24/0.27, but right now I see the difference of only <~0.01, and even better models wd likely give 0.002, like u wrote.", "votes": 1 }, { "id": 1061198, "postDate": "2020-10-26T19:51:26.817Z", "content": "Yes that may be true. Also notice that if your model predicts negative exams pretty well adjusting wrong image preds, e.g. clipping them to 0.5 to be <= 0.5 doesn't penalize the score at all. Other rules also plays a role for sure but this also another observation.
", "rawMarkdown": "Yes that may be true. Also notice that if your model predicts negative exams pretty well adjusting wrong image preds, e.g. clipping them to 0.5 to be <= 0.5 doesn't penalize the score at all. Other rules also plays a role for sure but this also another observation." } ] }, { "id": 1061118, "postDate": "2020-10-26T18:56:08.150Z", "content": "@juliaelliott @philculliton are you going to apply the rule on top 10 teams or all the teams? I didn't see any clear answer to that in the forums.
", "rawMarkdown": "@juliaelliott @philculliton are you going to apply the rule on top 10 teams or all the teams? I didn't see any clear answer to that in the forums.", "votes": 6, "replies": [ { "id": 1061134, "postDate": "2020-10-26T19:08:14.853Z", "content": "I asked which teams would be checked in the deleted post here and Julia responded
\n\n\nPrize winners' winning submissions will be checked and disqualified/removed if non-conforming. They should not expect to rely on any secondary selected submission as a fall-back.
\n
(I also asked what happens if 1 sub complies and 1 sub does not comply). Julia also replied here
", "rawMarkdown": "I asked which teams would be checked in the deleted post [here][1] and Julia responded\n> **Prize winners'** winning submissions will be checked and disqualified/removed if non-conforming. They should not expect to rely on any secondary selected submission as a fall-back.\n\n(I also asked what happens if 1 sub complies and 1 sub does not comply). Julia also replied [here][2]\n\n[1]: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/192696\n[2]: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473#1057644", "votes": 6 }, { "id": 1061151, "postDate": "2020-10-26T19:17:08.047Z", "content": "@cdeotte Thank you for providing details on that.
", "rawMarkdown": "@cdeotte Thank you for providing details on that.", "votes": 3 }, { "id": 1061178, "postDate": "2020-10-26T19:38:27.353Z", "content": "@cdeotte Thanks. You are a native speaker, can you help me with this? \"Prize winning submissions will be checked\" doesn't mean that \"non-prize winning submissions won't be checked\", right?
", "rawMarkdown": "@cdeotte Thanks. You are a native speaker, can you help me with this? \"Prize winning submissions will be checked\" doesn't mean that \"non-prize winning submissions won't be checked\", right?", "votes": 2 }, { "id": 1061184, "postDate": "2020-10-26T19:39:46.040Z", "content": "@aerdem4 To me, it implies that only prize winners' submissions will be checked. But you're right- it doesn't explicitly say that non-prize winners' submissions will not be checked.
", "rawMarkdown": "@aerdem4 To me, it implies that only prize winners' submissions will be checked. But you're right- it doesn't explicitly say that non-prize winners' submissions will not be checked. ", "votes": 1 }, { "id": 1061185, "postDate": "2020-10-26T19:39:55.183Z", "content": "@aerdem4 My interpretation is that only the top 10 will be checked, I assume medals are not \"prizes\", but neither does it state non-prize submissions would be checkedā¦
\nEdit: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473#1024452 seems to suggest yes, top 10 only
", "rawMarkdown": "@aerdem4 My interpretation is that only the top 10 will be checked, I assume medals are not \"prizes\", but neither does it state non-prize submissions would be checked...\n\nEdit: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473#1024452 seems to suggest yes, top 10 only", "votes": 1 }, { "id": 1061196, "postDate": "2020-10-26T19:49:13.920Z", "content": "@aerdem4 The checks will be applied to the top 10. But note that if someone in the original top 10 is disqualified, the host will work their way down to reach a top 10, which is why I made the distinction that these would be \"prize winners\" vs. \"top 10.\"
", "rawMarkdown": "@aerdem4 The checks will be applied to the top 10. But note that if someone in the original top 10 is disqualified, the host will work their way down to reach a top 10, which is why I made the distinction that these would be \"prize winners\" vs. \"top 10.\"", "votes": 7 }, { "id": 1061757, "postDate": "2020-10-27T09:51:04.150Z", "content": "@cdeotte : I could not find the link easily, but it seems that if one out of the two submissions have inconsistencies, the team/competitor will be disqualified. Am I right ?
", "rawMarkdown": "@cdeotte : I could not find the link easily, but it seems that if one out of the two submissions have inconsistencies, the team/competitor will be disqualified. Am I right ? " } ] }, { "id": 1061120, "postDate": "2020-10-26T18:57:05.253Z", "content": "
I'm currently hoping the Kaggle gods let my compliant subs commit :'(
", "rawMarkdown": "\n\nI'm currently hoping the Kaggle gods let my compliant subs commit :'(", "votes": 3, "replies": [ { "id": 1061131, "postDate": "2020-10-26T19:05:55.390Z", "content": "I have a similar issueā¦ I woke up extra early to finish up my last notebook and submit it, but when it came time to commit, I couldn't, and I submitted it with only 8 hours left until competition ends. Now praying to the Kaggle gods too, or else I won't have a compliant subā¦
", "rawMarkdown": "I have a similar issue... I woke up extra early to finish up my last notebook and submit it, but when it came time to commit, I couldn't, and I submitted it with only 8 hours left until competition ends. Now praying to the Kaggle gods too, or else I won't have a compliant sub...", "votes": 1 }, { "id": 1061158, "postDate": "2020-10-26T19:20:06.520Z", "content": "My inference kernels used to spend 4-5 hours, but today I afraid that they will not fulfill the inference time limit( @theoviel, I'm crossing my fingers for u
", "rawMarkdown": "My inference kernels used to spend 4-5 hours, but today I afraid that they will not fulfill the inference time limit( @theoviel, I'm crossing my fingers for u", "votes": 3 }, { "id": 1061160, "postDate": "2020-10-26T19:21:43.517Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1061163, "postDate": "2020-10-26T19:25:28.893Z", "content": "Because it is really not helping much here( And I also hoped to include several more things, like 3d models that miserably failed for me(
", "rawMarkdown": "Because it is really not helping much here( And I also hoped to include several more things, like 3d models that miserably failed for me(", "votes": 3 }, { "id": 1061172, "postDate": "2020-10-26T19:36:44.590Z", "content": "One of the two actually finished successfully :)
", "rawMarkdown": "One of the two actually finished successfully :)", "votes": 1 } ] }, { "id": 1061137, "postDate": "2020-10-26T19:09:07.793Z", "content": "@iafoss Are you sure that only one conflicting final submission will result in full disqualification of all final submissions of the team? From Global Wheat, teams with one illegal submission were allowed to keep their other one.
\nCan anyone confirm whether this is the case?
\nEdit: Answered by Chris Deotte below. Thanks Chris!
", "rawMarkdown": "@iafoss Are you sure that only one conflicting final submission will result in full disqualification of all final submissions of the team? From Global Wheat, teams with one illegal submission were allowed to keep their other one. \n\nCan anyone confirm whether this is the case? \n\nEdit: Answered by Chris Deotte below. Thanks Chris!", "votes": 1, "replies": [ { "id": 1061140, "postDate": "2020-10-26T19:10:51.830Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1061156, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-10-26T19:19:29.450000", "content": "How much LB difference do people observe with and without applying rules? For me, my CV decreases 0.002 but my LB stays the same. For my final two subs, I will be choosing submissions that conform. But I realize now that that may decrease my private LB by 0.002 š
", "votes": 5, "replies": [ { "id": 1061162, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-10-26T19:23:16.637000", "content": "On the only submission that ran without timeout I got - 0.003 LB (- 0.002 CV).
\n0.002 won't play a big role for either of us though, we're quite far from the gold zone
", "votes": 5, "replies": [] }, { "id": 1061181, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-10-26T19:38:58.230000", "content": "Yes @chris000, we also experienced a similar thing. In local CV not much drop -+ 0.001 and in LB around +-0.002/3.
", "votes": 1, "replies": [] }, { "id": 1061190, "author_name": "Iafoss", "author_url": "", "post_date": "2020-10-26T19:44:45.570000", "content": "From that I saw the difference is getting larger if the performance of the original model is worse. For example my very first sub in this competition got 0.24/0.27, but right now I see the difference of only <~0.01, and even better models wd likely give 0.002, like u wrote.
", "votes": 1, "replies": [] }, { "id": 1061198, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-10-26T19:51:26.817000", "content": "Yes that may be true. Also notice that if your model predicts negative exams pretty well adjusting wrong image preds, e.g. clipping them to 0.5 to be <= 0.5 doesn't penalize the score at all. Other rules also plays a role for sure but this also another observation.
", "votes": 0, "replies": [] } ] }, { "id": 1061118, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-10-26T18:56:08.150000", "content": "@juliaelliott @philculliton are you going to apply the rule on top 10 teams or all the teams? I didn't see any clear answer to that in the forums.
", "votes": 6, "replies": [ { "id": 1061134, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-10-26T19:08:14.853000", "content": "I asked which teams would be checked in the deleted post here and Julia responded
\n\n\nPrize winners' winning submissions will be checked and disqualified/removed if non-conforming. They should not expect to rely on any secondary selected submission as a fall-back.
\n
(I also asked what happens if 1 sub complies and 1 sub does not comply). Julia also replied here
", "votes": 6, "replies": [] }, { "id": 1061151, "author_name": "Iafoss", "author_url": "", "post_date": "2020-10-26T19:17:08.047000", "content": "@cdeotte Thank you for providing details on that.
", "votes": 3, "replies": [] }, { "id": 1061178, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-10-26T19:38:27.353000", "content": "@cdeotte Thanks. You are a native speaker, can you help me with this? \"Prize winning submissions will be checked\" doesn't mean that \"non-prize winning submissions won't be checked\", right?
", "votes": 2, "replies": [] }, { "id": 1061184, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-26T19:39:46.040000", "content": "@aerdem4 To me, it implies that only prize winners' submissions will be checked. But you're right- it doesn't explicitly say that non-prize winners' submissions will not be checked.
", "votes": 1, "replies": [] }, { "id": 1061185, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-26T19:39:55.183000", "content": "@aerdem4 My interpretation is that only the top 10 will be checked, I assume medals are not \"prizes\", but neither does it state non-prize submissions would be checkedā¦
\nEdit: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473#1024452 seems to suggest yes, top 10 only
", "votes": 1, "replies": [] }, { "id": 1061196, "author_name": "Julia Elliott", "author_url": "", "post_date": "2020-10-26T19:49:13.920000", "content": "@aerdem4 The checks will be applied to the top 10. But note that if someone in the original top 10 is disqualified, the host will work their way down to reach a top 10, which is why I made the distinction that these would be \"prize winners\" vs. \"top 10.\"
", "votes": 7, "replies": [] }, { "id": 1061757, "author_name": "Catadanna", "author_url": "", "post_date": "2020-10-27T09:51:04.150000", "content": "@cdeotte : I could not find the link easily, but it seems that if one out of the two submissions have inconsistencies, the team/competitor will be disqualified. Am I right ?
", "votes": 0, "replies": [] } ] }, { "id": 1061120, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-10-26T18:57:05.253000", "content": "
I'm currently hoping the Kaggle gods let my compliant subs commit :'(
", "votes": 3, "replies": [ { "id": 1061131, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-26T19:05:55.390000", "content": "I have a similar issueā¦ I woke up extra early to finish up my last notebook and submit it, but when it came time to commit, I couldn't, and I submitted it with only 8 hours left until competition ends. Now praying to the Kaggle gods too, or else I won't have a compliant subā¦
", "votes": 1, "replies": [] }, { "id": 1061158, "author_name": "Iafoss", "author_url": "", "post_date": "2020-10-26T19:20:06.520000", "content": "My inference kernels used to spend 4-5 hours, but today I afraid that they will not fulfill the inference time limit( @theoviel, I'm crossing my fingers for u
", "votes": 3, "replies": [] }, { "id": 1061160, "author_name": "", "author_url": "", "post_date": "2020-10-26T19:21:43.517000", "content": "", "votes": 0, "replies": [] }, { "id": 1061163, "author_name": "Iafoss", "author_url": "", "post_date": "2020-10-26T19:25:28.893000", "content": "Because it is really not helping much here( And I also hoped to include several more things, like 3d models that miserably failed for me(
", "votes": 3, "replies": [] }, { "id": 1061172, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-10-26T19:36:44.590000", "content": "One of the two actually finished successfully :)
", "votes": 1, "replies": [] } ] }, { "id": 1061137, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2020-10-26T19:09:07.793000", "content": "@iafoss Are you sure that only one conflicting final submission will result in full disqualification of all final submissions of the team? From Global Wheat, teams with one illegal submission were allowed to keep their other one.
\nCan anyone confirm whether this is the case?
\nEdit: Answered by Chris Deotte below. Thanks Chris!
", "votes": 1, "replies": [ { "id": 1061140, "author_name": "", "author_url": "", "post_date": "2020-10-26T19:10:51.830000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1061097": "I think many participants have a temptation to select a sub having higher LB but violating the label assignment rules, especially if they are far from the gold zone. Even if u see \"Winning submissions will be inspected to ensure label predictions adhere to the expected label hierarchy defined by the diagram on the Data page\", it does not mean that kaggle cannot check all selected submissions, it is very simple to do. And what will happen next? Check the rules:\n\n> Competition **Sponsor reserves the right to disqualify any participant who makes conflicting label predictions** that do not adhere to the expected label hierarchy defined by the diagram on the Data page. This includes:\n> Submissions that make study predictions which conflict with that study's image predictions.\nSubmissions that make conflicting or disallowed sub-label predictions (i.e. simultaneously predicting RV/LV ratio >=1 and <1, more than one label for PE type, etc.)\n\n**In other words entire team may be disqualified for cheating if even one selected sub is violating the rules, and there is no need to be in top 10 for this**. I expect kaggle would try to make this competition fair, and for us it is better to play a fair game too.", "1061156": "How much LB difference do people observe with and without applying rules? For me, my CV decreases 0.002 but my LB stays the same. For my final two subs, I will be choosing submissions that conform. But I realize now that that may decrease my private LB by 0.002 š", "1061118": "@juliaelliott @philculliton are you going to apply the rule on top 10 teams or all the teams? I didn't see any clear answer to that in the forums.", "1061120": "\n\nI'm currently hoping the Kaggle gods let my compliant subs commit :'(", "1061137": "@iafoss Are you sure that only one conflicting final submission will result in full disqualification of all final submissions of the team? From Global Wheat, teams with one illegal submission were allowed to keep their other one. \n\nCan anyone confirm whether this is the case? \n\nEdit: Answered by Chris Deotte below. Thanks Chris!" } }