My team has dropped 10 places since the leak. As of yet, we have not used the leak.
\nI presume the organizers will have a hard time identifying all who use the leak as teams may generate predictions with the leak offline such that the code cannot be interrogated.
\nWould like to know if the leak is now defacto legal or if users of the leak will be disqualified, and if that is the plan, how?
", "messages": [ { "id": 2918738, "postDate": "2024-07-12T13:24:40.500Z", "content": "My team has dropped 10 places since the leak. As of yet, we have not used the leak.
\nI presume the organizers will have a hard time identifying all who use the leak as teams may generate predictions with the leak offline such that the code cannot be interrogated.
\nWould like to know if the leak is now defacto legal or if users of the leak will be disqualified, and if that is the plan, how?
", "rawMarkdown": "My team has dropped 10 places since the leak. As of yet, we have not used the leak.\n\nI presume the organizers will have a hard time identifying all who use the leak as teams may generate predictions with the leak offline such that the code cannot be interrogated.\n\nWould like to know if the leak is now defacto legal or if users of the leak will be disqualified, and if that is the plan, how?", "votes": 20 }, { "id": 2919525, "postDate": "2024-07-13T01:40:17.853Z", "content": "May I ask for a clarification?
\nThe action you need to take is to ensure that if a reviewer's solution submission is found to have a leak, then it should be removed from the leaderboard. This is because for most participants, the focus is on their leaderboard position and Kaggle medal, rather than money.
\nYour action will lead to significantly different decisions down the line.
\nFor teams that have the capability to identify leaks but actually follow your competition guidelines by not using them, please remember that they initially trusted you and stood with you. Now, they need a strong assurance from you to be able to sleep well.
Furthermore, based on my above point, I would suggest expanding the scope of the review to include the gold medal zone, not just the prize area. @jerrylin96
", "rawMarkdown": "May I ask for a clarification?\nThe action you need to take is to ensure that if a reviewer's solution submission is found to have a leak, then it should be removed from the leaderboard. This is because for most participants, the focus is on their leaderboard position and Kaggle medal, rather than money.\nYour action will lead to significantly different decisions down the line.\nFor teams that have the capability to identify leaks but actually follow your competition guidelines by not using them, please remember that they initially trusted you and stood with you. Now, they need a strong assurance from you to be able to sleep well.\n\nFurthermore, based on my above point, I would suggest expanding the scope of the review to include the gold medal zone, not just the prize area. @jerrylin96 ", "votes": 5, "replies": [ { "id": 2919543, "postDate": "2024-07-13T02:03:23.130Z", "content": "Hi @jerrylin96 , What we care about is the honor, rank and medal, not the prize. Gold medals without prizes and with prizes are almost the same important, except for their corresponding points on the kaggle rankings. So please at least make sure all the gold solutions are clean.
\nIf you want to verify all the gold solutions, but Kaggle thinks it's too exhausting and expensive, and wants to only verify the prize zone. Please make this situation public. Most of the participants will stand with you.
", "rawMarkdown": "Hi @jerrylin96 , What we care about is the honor, rank and medal, not the prize. Gold medals without prizes and with prizes are almost the same important, except for their corresponding points on the [kaggle rankings](https://www.kaggle.com/rankings?group=competitions). So please at least make sure all the gold solutions are clean.\n\n-------------------------------\n\nIf you want to verify all the gold solutions, but Kaggle thinks it's too exhausting and expensive, and wants to only verify the prize zone. Please make this situation public. Most of the participants will stand with you. ", "votes": 4, "replies": [ { "id": 2919551, "postDate": "2024-07-13T02:09:19.023Z", "content": "Some of us also care about silver and bronze and a solution is needed for participants in contention for those places that have chosen not to use the leak.
", "rawMarkdown": "Some of us also care about silver and bronze and a solution is needed for participants in contention for those places that have chosen not to use the leak.", "votes": 1 }, { "id": 2919559, "postDate": "2024-07-13T02:25:16.747Z", "content": "Hi @seanmacmaghnusa What I can say is that I stand with you and all the participants obey the rules. This competition is so frustrating. Hope you all can find a better competition in the future…
", "rawMarkdown": "Hi @seanmacmaghnusa What I can say is that I stand with you and all the participants obey the rules. This competition is so frustrating. Hope you all can find a better competition in the future... " }, { "id": 2920684, "postDate": "2024-07-13T18:21:53.600Z", "content": "If allowed, I would like to vet the solutions in the gold and award zone. I do not have the resources to go past that.
", "rawMarkdown": "If allowed, I would like to vet the solutions in the gold and award zone. I do not have the resources to go past that." }, { "id": 2920801, "postDate": "2024-07-13T19:42:36.827Z", "content": "@jerrylin96 Is that something you have gotten confirmation from the Kaggle admins? I am not sure if this is possible/ if host has the right to inspect non-prize solutions.
", "rawMarkdown": "@jerrylin96 Is that something you have gotten confirmation from the Kaggle admins? I am not sure if this is possible/ if host has the right to inspect non-prize solutions.", "votes": 1 }, { "id": 2920818, "postDate": "2024-07-13T20:00:05.203Z", "content": "\n\nI am not sure if this is possible/ if host has the right to inspect non-prize solutions.
\n
And so the plot thickens 🤣
\nThis is better than game of thrones.
This is genuinely hilarious and would explain a lot. Not sure if I want the curtains pulled back more or to just move on with this information in hand for future competitions 😅
", "rawMarkdown": "This is genuinely hilarious and would explain a lot. Not sure if I want the curtains pulled back more or to just move on with this information in hand for future competitions 😅\n" } ] } ] }, { "id": 2918843, "postDate": "2024-07-12T14:46:39.550Z", "content": "I also think that non-gold medalists fall under the table in this entire discussion. The top 10 ranks in a competition may be the most important ones, especially for the host, but the solution is not great for anyone in the silver range or below.
", "rawMarkdown": "I also think that non-gold medalists fall under the table in this entire discussion. The top 10 ranks in a competition may be the most important ones, especially for the host, but the solution is not great for anyone in the silver range or below.", "votes": 2, "replies": [ { "id": 2918855, "postDate": "2024-07-12T14:52:27.270Z", "content": "We all know that the real value of Kaggle is the learning. Ive learnt some new things from my teammates on this competition, and in return ive made some marginal contribution to our performance. The learning and exchange is the most important thing long term.
\nGiving people a chance at progression is part of the hook of Kaggle. It would be a shame if we cant make that happen here without cheating.
", "rawMarkdown": "We all know that the real value of Kaggle is the learning. Ive learnt some new things from my teammates on this competition, and in return ive made some marginal contribution to our performance. The learning and exchange is the most important thing long term.\n\nGiving people a chance at progression is part of the hook of Kaggle. It would be a shame if we cant make that happen here without cheating.", "votes": 1 } ] }, { "id": 2919051, "postDate": "2024-07-12T16:45:39.037Z", "content": "i think we should expect hosts verify only gold medals, IMHO what is below top 20 doesn't matter for them,manually check code of every competitor is to expensive, so probably teams that use leak in winning position for silver or bronze medals, personally i am not using leak.
", "rawMarkdown": "i think we should expect hosts verify only gold medals, IMHO what is below top 20 doesn't matter for them,manually check code of every competitor is to expensive, so probably teams that use leak in winning position for silver or bronze medals, personally i am not using leak.", "replies": [ { "id": 2919062, "postDate": "2024-07-12T16:56:26.113Z", "content": "That would be unfortunate. If that is the plan hosts should be explicit about it now so that we can freely use the leak.
\nThe communication is currently leaving many in a no-mans land.
", "rawMarkdown": "That would be unfortunate. If that is the plan hosts should be explicit about it now so that we can freely use the leak.\n\nThe communication is currently leaving many in a no-mans land. ", "votes": 2 } ] }, { "id": 2918776, "postDate": "2024-07-12T13:46:57.507Z", "content": "If I may suggest a solution, sudden 0.01 to 0.03 improvements in scores since posting of the leak should all be manually reviewed including the contents of all flat files used for ensembling.
\nThis wont capture teams using the leak who were aware prior to posting unfortunately, it should facilitate a thorough and fair review of submissions for the majority of participants.
", "rawMarkdown": "If I may suggest a solution, sudden 0.01 to 0.03 improvements in scores since posting of the leak should all be manually reviewed including the contents of all flat files used for ensembling.\n\nThis wont capture teams using the leak who were aware prior to posting unfortunately, it should facilitate a thorough and fair review of submissions for the majority of participants.", "votes": -1, "replies": [ { "id": 2918829, "postDate": "2024-07-12T14:37:21.453Z", "content": "In this competition, does the score improvement from using a leak range from 0.01 to 0.03?
", "rawMarkdown": "In this competition, does the score improvement from using a leak range from 0.01 to 0.03?", "votes": 1, "replies": [ { "id": 2918832, "postDate": "2024-07-12T14:39:51.413Z", "content": "My estimate is most are coming out as a 0.01 to 0.02 boost.
", "rawMarkdown": "My estimate is most are coming out as a 0.01 to 0.02 boost.", "votes": -2 }, { "id": 2918848, "postDate": "2024-07-12T14:48:38.610Z", "content": "I see. Thank you for reply. Since the leak, our team has dropped 15 places suddenly…
", "rawMarkdown": "I see. Thank you for reply. Since the leak, our team has dropped 15 places suddenly... ", "votes": 2 } ] }, { "id": 2919040, "postDate": "2024-07-12T16:41:38.270Z", "content": "People are seeing boosts in that range from ensembles and other things
", "rawMarkdown": "People are seeing boosts in that range from ensembles and other things", "replies": [ { "id": 2919052, "postDate": "2024-07-12T16:46:27.560Z", "content": "Do not disagree. However, there was very little movement in upper to mid bronze in the 5-10 days preceeding the leak despite teams actively submitting. Post leak a number of teams that were actively submitting and not making progress suddenly saw a jump thats commensurate with the boost from the leak.
\nBoth things can be true and thats what makes this difficult.
", "rawMarkdown": "Do not disagree. However, there was very little movement in upper to mid bronze in the 5-10 days preceeding the leak despite teams actively submitting. Post leak a number of teams that were actively submitting and not making progress suddenly saw a jump thats commensurate with the boost from the leak.\n\nBoth things can be true and thats what makes this difficult. \n\n", "votes": 2 } ] } ] }, { "id": 2919872, "postDate": "2024-07-13T09:36:10.033Z", "content": "I believe that any explicit use of timestamp and location information constitutes exploiting a loophole, which goes against the original intention of the hosts and the competition. Based on my estimation, the teams in the gold medal zone have likely used timestamp and location information explicitly, which is unfair to every other participating team. It is truly disheartening.
", "rawMarkdown": "I believe that any explicit use of timestamp and location information constitutes exploiting a loophole, which goes against the original intention of the hosts and the competition. Based on my estimation, the teams in the gold medal zone have likely used timestamp and location information explicitly, which is unfair to every other participating team. It is truly disheartening.", "votes": -3, "replies": [ { "id": 2919890, "postDate": "2024-07-13T09:47:56.517Z", "content": "\n\nI believe that any explicit use of timestamp and location information constitutes exploiting a loophole, which goes against the original intention of the hosts and the competition
\n
Only for test set/inference. For the train set there is no problem (remember also that this data was explicitly available for train set), and it's also confirmed by host.
", "rawMarkdown": ">I believe that any explicit use of timestamp and location information constitutes exploiting a loophole, which goes against the original intention of the hosts and the competition \n\nOnly for test set/inference. For the train set there is no problem (remember also that this data was explicitly available for train set), and it's [also confirmed by host](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519772#2919233).", "votes": 2 }, { "id": 2919931, "postDate": "2024-07-13T10:31:53.393Z", "content": "\n\nthe teams in the gold medal zone have likely used timestamp and location information explicitly
\n
no 😅
\nbased on my estimation, greySnow used a leak, but maybe my model is just a little worse and his approach is something I'll have to study after [most likely scenario] :D
", "rawMarkdown": "> the teams in the gold medal zone have likely used timestamp and location information explicitly\n\nno 😅\n\nbased on my estimation, greySnow used a leak, but maybe my model is just a little worse and his approach is something I'll have to study after [most likely scenario] :D", "replies": [ { "id": 2919948, "postDate": "2024-07-13T10:45:08.157Z", "content": "\n\nbased on my estimation, greySnow used a leak
\n
Your estimation is wrong :)
\nAt the very least, I intend to publish a complete Kaggle pipeline with single model 0.79+ LB. The rest is ensembling (and if you tried ensembling you probably know that +~0.004 is very achievable).
\nSo don't worry lol
We won't be able to check submissions not in the running for prize money, unfortunately. That being said, submissions discovered to use a leak will be disqualified from the leaderboard.
\nThis comment has been edited in response to community feedback.
", "rawMarkdown": "We won't be able to check submissions not in the running for prize money, unfortunately. That being said, submissions discovered to use a leak will be disqualified from the leaderboard.\n\nThis comment has been edited in response to community feedback.", "votes": -32, "replies": [ { "id": 2919448, "postDate": "2024-07-12T21:36:41.663Z", "content": "@jerrylin96 There really needs to be a clear ruling regarding this. Earlier, you said that leaky solutions won't be considered for Gold and cash prizes. But this suggests we can use it? It would be terrible if there are huge discussions about the gold zone after competition end because some kagglers thought it was allowed and others thought wasn't.
", "rawMarkdown": "@jerrylin96 There really needs to be a clear ruling regarding this. Earlier, you said that leaky solutions won't be considered for Gold and cash prizes. But this suggests we can use it? It would be terrible if there are huge discussions about the gold zone after competition end because some kagglers thought it was allowed and others thought wasn't.", "votes": 5, "replies": [ { "id": 2919456, "postDate": "2024-07-12T21:58:45.143Z", "content": "Hi Fritz, ideally we would not allow leak based solutions period. In practice, I cannot guarantee I will be able to vet every solution that could possibly have used a leak.
", "rawMarkdown": "Hi Fritz, ideally we would not allow leak based solutions period. In practice, I cannot guarantee I will be able to vet every solution that could possibly have used a leak. ", "votes": -10 } ] }, { "id": 2919467, "postDate": "2024-07-12T22:26:55.560Z", "content": "I understand and agree you can't check all submission and check only in-prize submissions. But kaggle is competition. we compete not only to win prize cash. I think it is not appropriate for a host to make a statement affirming a rule violation.
\nSo could you please take back this statement?
\n\nIf you are treating it as a learning exercise and your LB score is sufficiently far from prize range I think you should be okay experimenting with new techniques.
\n
I understand \"host\" won't check submission. On the other hand, is there a possibility of validating the submissions to the gold zone sub by kaggle staff? Even just \"possibly\" would be a deterrent against using leak.
", "rawMarkdown": "I understand and agree you can't check all submission and check only in-prize submissions. But kaggle is competition. we compete not only to win prize cash. I think it is not appropriate for a host to make a statement affirming a rule violation. \nSo could you please take back this statement?\n\n> If you are treating it as a learning exercise and your LB score is sufficiently far from prize range I think you should be okay experimenting with new techniques.\n\n\nI understand \"host\" won't check submission. On the other hand, is there a possibility of validating the submissions to the gold zone sub by kaggle staff? Even just \"possibly\" would be a deterrent against using leak.", "votes": 14, "replies": [ { "id": 2920841, "postDate": "2024-07-13T20:44:18.077Z", "content": "Hello, I will take back this statement.
", "rawMarkdown": "Hello, I will take back this statement.", "votes": 2 }, { "id": 2920971, "postDate": "2024-07-14T00:15:31.323Z", "content": "Hello, thank you for understanding and taking back☺️ !
", "rawMarkdown": "Hello, thank you for understanding and taking back☺️ !" } ] }, { "id": 2919486, "postDate": "2024-07-12T23:54:56.677Z", "content": "@jerrylin96
\nI interpreted that the host allowed the use of leaks as long as the submission doesn't reach the prize range. Could you explain what the spirit of the competition mentioned in the discussion below is?
\n https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/513193
This is not ML competition. Chicken game.
", "rawMarkdown": "This is not ML competition. Chicken game.", "votes": 10 }, { "id": 2919496, "postDate": "2024-07-13T00:27:29.170Z", "content": "It seems to be OK to use 'leak' up to 0.7890. However, those who see this comment will likely aim for 0.7891. However, those who see this comment will likely aim for 0.7892. However…
", "rawMarkdown": "It seems to be OK to use 'leak' up to 0.7890. However, those who see this comment will likely aim for 0.7891. However, those who see this comment will likely aim for 0.7892. However...", "votes": 3 }, { "id": 2919503, "postDate": "2024-07-13T00:47:47.007Z", "content": "What a terrible competition, the worst of the 78 competitions I have participated in on kaggle!
", "rawMarkdown": "What a terrible competition, the worst of the 78 competitions I have participated in on kaggle!", "votes": 9 }, { "id": 2919511, "postDate": "2024-07-13T01:14:55.500Z", "content": "I don't know what changed your mind in only one day.
\nThis current situation is not good at all. It is like Game theory. Gold teams try to use leaked information to improve their ranking without entering the prize zone.
\nwhat a mess!
", "rawMarkdown": "I don't know what changed your mind in only one day. \n\n- if it is the time/machine resource, we have 11 golds and 5 prize position, only twice workload. I don't think it is a big problem\n- if it is kaggle who don't allow you check gold solution, I will feel really disappointed on kaggle. No flexibility at all. Very rigid.\n\nThis current situation is not good at all. It is like Game theory. Gold teams try to use leaked information to improve their ranking without entering the prize zone. \n\nwhat a mess! ", "votes": 3 }, { "id": 2919517, "postDate": "2024-07-13T01:24:30.717Z", "content": "@charmq Can you share your 0.800 submission? It will save us time.
", "rawMarkdown": "@charmq Can you share your 0.800 submission? It will save us time.", "votes": 2 }, { "id": 2919523, "postDate": "2024-07-13T01:36:06.787Z", "content": "@jerrylin96 - Appreciate you're doing your best here. I also appreciate you are finishing your PhD, which is no small undertaking and is in and of itself stressful without adding extra stress.
\nI do not mean to pile on here but I feel like some polite push back is appropriate. You're using the term 'sufficiently' here - but what exactly constitutes sufficiently is left completely open to interpretation.
\nIn aggregate 10s, if not 100s, of thousands of human hours have been committed to this competition. Participant's efforts (all of them) deserve to be recognized with a more robust response and solution.
", "rawMarkdown": "@jerrylin96 - Appreciate you're doing your best here. I also appreciate you are finishing your PhD, which is no small undertaking and is in and of itself stressful without adding extra stress. \n\nI do not mean to pile on here but I feel like some polite push back is appropriate. You're using the term 'sufficiently' here - but what exactly constitutes sufficiently is left completely open to interpretation. \n\nIn aggregate 10s, if not 100s, of thousands of human hours have been committed to this competition. Participant's efforts (all of them) deserve to be recognized with a more robust response and solution.", "votes": 4 }, { "id": 2919573, "postDate": "2024-07-13T02:37:41.797Z", "content": "@jerrylin96
\nI can understand your opinion, I guess that it is difficult to check the implementation of all participants.
\nHowever our purpose is not only prize, but also medals, especially kaggle GM is one of the honor title, then gold medal is very great achievement.
if you will disqualify team that use leak solutions, you should check gold medals at least.
", "rawMarkdown": "@jerrylin96 \nI can understand your opinion, I guess that it is difficult to check the implementation of all participants. \nHowever our purpose is not only prize, but also medals, especially kaggle GM is one of the honor title, then gold medal is very great achievement. \n\nif you will disqualify team that use leak solutions, you should check gold medals at least. ", "votes": 7, "replies": [ { "id": 2919613, "postDate": "2024-07-13T03:37:13.723Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2919615, "postDate": "2024-07-13T03:37:47.577Z", "content": "@jerrylin96
\nEven if it's difficult to check all submissions and they are not perfect, you could broaden the scope a bit more and at least request the submission of source code for those within the gold range. You should be able to check as much as possible.
At the very least, I am more focused on whether I can get a gold medal than whether I can get a prize because the title of GM is at stake for me. There are many other Kagglers who also value medals highly. I think the current decision disregards the feelings of these Kagglers.
\nI believe that only with a good relationship between the host and Kagglers can good solutions be provided. Is it acceptable for someone who could genuinely provide a good solution to use a leak, thus making their solution worthless to you?
\nPlease reconsider
", "rawMarkdown": "@jerrylin96\nEven if it's difficult to check all submissions and they are not perfect, you could broaden the scope a bit more and at least request the submission of source code for those within the gold range. You should be able to check as much as possible.\n\nAt the very least, I am more focused on whether I can get a gold medal than whether I can get a prize because the title of GM is at stake for me. There are many other Kagglers who also value medals highly. I think the current decision disregards the feelings of these Kagglers.\n\nI believe that only with a good relationship between the host and Kagglers can good solutions be provided. Is it acceptable for someone who could genuinely provide a good solution to use a leak, thus making their solution worthless to you?\n\nPlease reconsider", "votes": 7 }, { "id": 2920440, "postDate": "2024-07-13T16:32:54.187Z", "content": "I emailed Kaggle staff last night to see if we could vet at least the gold medal submissions. You are all correct that in an ideal scenario we would vet every submission. I can say for a fact that I personally do not have the resources to do that, though.
", "rawMarkdown": "I emailed Kaggle staff last night to see if we could vet at least the gold medal submissions. You are all correct that in an ideal scenario we would vet every submission. I can say for a fact that I personally do not have the resources to do that, though.", "votes": 5, "replies": [ { "id": 2920450, "postDate": "2024-07-13T16:36:00.470Z", "content": "That is understood. Can you please work with Kaggle to ensure those who would have medalled as of the date of the leak, will be awarded the medal they would have won at least for those in bronze (and maybe silver). Results can be indexed to the lowest bronze medal as of date of the leak.
", "rawMarkdown": "That is understood. Can you please work with Kaggle to ensure those who would have medalled as of the date of the leak, will be awarded the medal they would have won at least for those in bronze (and maybe silver). Results can be indexed to the lowest bronze medal as of date of the leak.", "votes": -1 }, { "id": 2920610, "postDate": "2024-07-13T17:55:41.647Z", "content": "@seanmacmaghnusa
\nI understand your ungry.
\nBut I think it's impossible for both Kaggle and the participants. To check whether the team is legitimate or not, they must prepare documentation and show reproducibility of their solution, which is the same amount of work as receiving a prize. In my opinion, gold zone teams have enough motivation to do that, but enforcing this on silver and bronze teams may lead to undesirable situations.
@bamps53 True. I will add that this is a difficult leak to find, and difficult no less to exploit (building 2D->1D model is significantly harder than 1D->1D. At least in my guess, I did not build one myself. It's not the kind of leak that give you easy win with few lines of code). I don't believe there are many teams that have a good leaky solution, and most of them are probably good enough to enter silver zone with 1D->1D solution too. So people below gold range really should not be too concerned, imho.
", "rawMarkdown": "@bamps53 True. I will add that this is a difficult leak to find, and difficult no less to exploit (building 2D->1D model is significantly harder than 1D->1D. At least in my guess, I did not build one myself. It's not the kind of leak that give you easy win with few lines of code). I don't believe there are many teams that have a good leaky solution, and most of them are probably good enough to enter silver zone with 1D->1D solution too. So people below gold range really should not be too concerned, imho.", "votes": 3 }, { "id": 2920693, "postDate": "2024-07-13T18:25:10.963Z", "content": "\nI understand that you are currently in a difficult situation. Despite this, I genuinely feel that you are doing your best to ensure the solutions you obtain are the best for you.
\nSome people say that checking the prize zone and the gold zone simply doubles the cost, but that's not the case. The former was originally planned, while the latter incurs additional costs. There is a significant gap here.
\nIf these costs are unacceptable, I have a proposal.
\nThe gold zone check can be limited to validation of the inference code. Checking the training code is more than ten times harder than checking the inference code. However, the 2D-1D or 2D-2D prediction that you really want to prevent should be able to be confirmed only by checking the inference code.
\nI understand this is not perfect (at least you should not declare so to the participants in advance), but I think it is a realistic method.
\nI propose this as a compromise if there is a conflict between what we want to achieve and the costs while negotiation with Kaggle.
Despite various troubles, I have generally enjoyed the challenging tasks of this competition.
\nLet's do our best for the remaining period!
Not an unreasonable comment. There were other options here though. Whether the intent was to check beyond gold 'zone' or not, publicly stating they wouldn't check lead to an acceleration of new submissions with 0.005 to 0.02 improvements. Those will shake up bronze and move people who are not using the leak out of medal position. That was an unforced error and is hugely unfortunate.
\nThere also were also other options here including early end post leak or another extension.
\nMaybe I should just go work on the leak for the day.
", "rawMarkdown": "Not an unreasonable comment. There were other options here though. Whether the intent was to check beyond gold 'zone' or not, publicly stating they wouldn't check lead to an acceleration of new submissions with 0.005 to 0.02 improvements. Those will shake up bronze and move people who are not using the leak out of medal position. That was an unforced error and is hugely unfortunate.\n\nThere also were also other options here including early end post leak or another extension. \n\nMaybe I should just go work on the leak for the day." } ] }, { "id": 2921279, "postDate": "2024-07-14T09:29:44.657Z", "content": "Competition has turned into a variant of Thaler's 'number guessing game' :
\nchoose a leaked submission with the goal of making your submission as close as possible to the first five participants in the contest that don't use the leak.
", "rawMarkdown": "Competition has turned into a variant of Thaler's 'number guessing game' :\n\nchoose a leaked submission with the goal of making your submission as close as possible to the first five participants in the contest that don't use the leak." } ] }, { "id": 2919268, "postDate": "2024-07-12T19:04:48.150Z", "content": "Looking into this and will get back to you.
", "rawMarkdown": "Looking into this and will get back to you.", "votes": -4 }, { "id": 2919623, "postDate": "2024-07-13T03:43:21.730Z", "content": "Before any leakage, my score was 0.731, and I had eight different results that all roughly achieved a score of 0.731. I hadn't used Ensemble methods before. Now, using four of these results in an ensemble allowed me to reach 0.737, and using another four, I reached 0.738. Although the increase wasn't as much as 0.01, I believe that an improvement of 0.01 is possible solely through ensembling. Therefore, it's difficult to definitively attribute the 0.01 improvement to any specific cause.
", "rawMarkdown": "Before any leakage, my score was 0.731, and I had eight different results that all roughly achieved a score of 0.731. I hadn't used Ensemble methods before. Now, using four of these results in an ensemble allowed me to reach 0.737, and using another four, I reached 0.738. Although the increase wasn't as much as 0.01, I believe that an improvement of 0.01 is possible solely through ensembling. Therefore, it's difficult to definitively attribute the 0.01 improvement to any specific cause." }, { "id": 2923561, "postDate": "2024-07-16T01:46:11.557Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2919525, "author_name": "Max2020", "author_url": "", "post_date": "2024-07-13T01:40:17.853000", "content": "May I ask for a clarification?
\nThe action you need to take is to ensure that if a reviewer's solution submission is found to have a leak, then it should be removed from the leaderboard. This is because for most participants, the focus is on their leaderboard position and Kaggle medal, rather than money.
\nYour action will lead to significantly different decisions down the line.
\nFor teams that have the capability to identify leaks but actually follow your competition guidelines by not using them, please remember that they initially trusted you and stood with you. Now, they need a strong assurance from you to be able to sleep well.
Furthermore, based on my above point, I would suggest expanding the scope of the review to include the gold medal zone, not just the prize area. @jerrylin96
", "votes": 5, "replies": [ { "id": 2919543, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-13T02:03:23.130000", "content": "Hi @jerrylin96 , What we care about is the honor, rank and medal, not the prize. Gold medals without prizes and with prizes are almost the same important, except for their corresponding points on the kaggle rankings. So please at least make sure all the gold solutions are clean.
\nIf you want to verify all the gold solutions, but Kaggle thinks it's too exhausting and expensive, and wants to only verify the prize zone. Please make this situation public. Most of the participants will stand with you.
", "votes": 4, "replies": [ { "id": 2919551, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-13T02:09:19.023000", "content": "Some of us also care about silver and bronze and a solution is needed for participants in contention for those places that have chosen not to use the leak.
", "votes": 1, "replies": [] }, { "id": 2919559, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-13T02:25:16.747000", "content": "Hi @seanmacmaghnusa What I can say is that I stand with you and all the participants obey the rules. This competition is so frustrating. Hope you all can find a better competition in the future…
", "votes": 0, "replies": [] }, { "id": 2920684, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-13T18:21:53.600000", "content": "If allowed, I would like to vet the solutions in the gold and award zone. I do not have the resources to go past that.
", "votes": 0, "replies": [] }, { "id": 2920801, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-07-13T19:42:36.827000", "content": "@jerrylin96 Is that something you have gotten confirmation from the Kaggle admins? I am not sure if this is possible/ if host has the right to inspect non-prize solutions.
", "votes": 1, "replies": [] }, { "id": 2920818, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-13T20:00:05.203000", "content": "\n\nI am not sure if this is possible/ if host has the right to inspect non-prize solutions.
\n
And so the plot thickens 🤣
\nThis is better than game of thrones.
This is genuinely hilarious and would explain a lot. Not sure if I want the curtains pulled back more or to just move on with this information in hand for future competitions 😅
", "votes": 0, "replies": [] } ] } ] }, { "id": 2918843, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-07-12T14:46:39.550000", "content": "I also think that non-gold medalists fall under the table in this entire discussion. The top 10 ranks in a competition may be the most important ones, especially for the host, but the solution is not great for anyone in the silver range or below.
", "votes": 2, "replies": [ { "id": 2918855, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-12T14:52:27.270000", "content": "We all know that the real value of Kaggle is the learning. Ive learnt some new things from my teammates on this competition, and in return ive made some marginal contribution to our performance. The learning and exchange is the most important thing long term.
\nGiving people a chance at progression is part of the hook of Kaggle. It would be a shame if we cant make that happen here without cheating.
", "votes": 1, "replies": [] } ] }, { "id": 2919051, "author_name": "VITALIY", "author_url": "", "post_date": "2024-07-12T16:45:39.037000", "content": "i think we should expect hosts verify only gold medals, IMHO what is below top 20 doesn't matter for them,manually check code of every competitor is to expensive, so probably teams that use leak in winning position for silver or bronze medals, personally i am not using leak.
", "votes": 0, "replies": [ { "id": 2919062, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-12T16:56:26.113000", "content": "That would be unfortunate. If that is the plan hosts should be explicit about it now so that we can freely use the leak.
\nThe communication is currently leaving many in a no-mans land.
", "votes": 2, "replies": [] } ] }, { "id": 2918776, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-12T13:46:57.507000", "content": "If I may suggest a solution, sudden 0.01 to 0.03 improvements in scores since posting of the leak should all be manually reviewed including the contents of all flat files used for ensembling.
\nThis wont capture teams using the leak who were aware prior to posting unfortunately, it should facilitate a thorough and fair review of submissions for the majority of participants.
", "votes": -1, "replies": [ { "id": 2918829, "author_name": "shisa07", "author_url": "", "post_date": "2024-07-12T14:37:21.453000", "content": "In this competition, does the score improvement from using a leak range from 0.01 to 0.03?
", "votes": 1, "replies": [ { "id": 2918832, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-12T14:39:51.413000", "content": "My estimate is most are coming out as a 0.01 to 0.02 boost.
", "votes": -2, "replies": [] }, { "id": 2918848, "author_name": "shisa07", "author_url": "", "post_date": "2024-07-12T14:48:38.610000", "content": "I see. Thank you for reply. Since the leak, our team has dropped 15 places suddenly…
", "votes": 2, "replies": [] } ] }, { "id": 2919040, "author_name": "Rob Freeman", "author_url": "", "post_date": "2024-07-12T16:41:38.270000", "content": "People are seeing boosts in that range from ensembles and other things
", "votes": 0, "replies": [ { "id": 2919052, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-12T16:46:27.560000", "content": "Do not disagree. However, there was very little movement in upper to mid bronze in the 5-10 days preceeding the leak despite teams actively submitting. Post leak a number of teams that were actively submitting and not making progress suddenly saw a jump thats commensurate with the boost from the leak.
\nBoth things can be true and thats what makes this difficult.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2919872, "author_name": "Zhuoqun Li", "author_url": "", "post_date": "2024-07-13T09:36:10.033000", "content": "I believe that any explicit use of timestamp and location information constitutes exploiting a loophole, which goes against the original intention of the hosts and the competition. Based on my estimation, the teams in the gold medal zone have likely used timestamp and location information explicitly, which is unfair to every other participating team. It is truly disheartening.
", "votes": -3, "replies": [ { "id": 2919890, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-13T09:47:56.517000", "content": "\n\nI believe that any explicit use of timestamp and location information constitutes exploiting a loophole, which goes against the original intention of the hosts and the competition
\n
Only for test set/inference. For the train set there is no problem (remember also that this data was explicitly available for train set), and it's also confirmed by host.
", "votes": 2, "replies": [] }, { "id": 2919931, "author_name": "slime", "author_url": "", "post_date": "2024-07-13T10:31:53.393000", "content": "\n\nthe teams in the gold medal zone have likely used timestamp and location information explicitly
\n
no 😅
\nbased on my estimation, greySnow used a leak, but maybe my model is just a little worse and his approach is something I'll have to study after [most likely scenario] :D
", "votes": 0, "replies": [ { "id": 2919948, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-13T10:45:08.157000", "content": "\n\nbased on my estimation, greySnow used a leak
\n
Your estimation is wrong :)
\nAt the very least, I intend to publish a complete Kaggle pipeline with single model 0.79+ LB. The rest is ensembling (and if you tried ensembling you probably know that +~0.004 is very achievable).
\nSo don't worry lol
We won't be able to check submissions not in the running for prize money, unfortunately. That being said, submissions discovered to use a leak will be disqualified from the leaderboard.
\nThis comment has been edited in response to community feedback.
", "votes": -32, "replies": [ { "id": 2919448, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-07-12T21:36:41.663000", "content": "@jerrylin96 There really needs to be a clear ruling regarding this. Earlier, you said that leaky solutions won't be considered for Gold and cash prizes. But this suggests we can use it? It would be terrible if there are huge discussions about the gold zone after competition end because some kagglers thought it was allowed and others thought wasn't.
", "votes": 5, "replies": [ { "id": 2919456, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-12T21:58:45.143000", "content": "Hi Fritz, ideally we would not allow leak based solutions period. In practice, I cannot guarantee I will be able to vet every solution that could possibly have used a leak.
", "votes": -10, "replies": [] } ] }, { "id": 2919467, "author_name": "kurupical", "author_url": "", "post_date": "2024-07-12T22:26:55.560000", "content": "I understand and agree you can't check all submission and check only in-prize submissions. But kaggle is competition. we compete not only to win prize cash. I think it is not appropriate for a host to make a statement affirming a rule violation.
\nSo could you please take back this statement?
\n\nIf you are treating it as a learning exercise and your LB score is sufficiently far from prize range I think you should be okay experimenting with new techniques.
\n
I understand \"host\" won't check submission. On the other hand, is there a possibility of validating the submissions to the gold zone sub by kaggle staff? Even just \"possibly\" would be a deterrent against using leak.
", "votes": 14, "replies": [ { "id": 2920841, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-13T20:44:18.077000", "content": "Hello, I will take back this statement.
", "votes": 2, "replies": [] }, { "id": 2920971, "author_name": "kurupical", "author_url": "", "post_date": "2024-07-14T00:15:31.323000", "content": "Hello, thank you for understanding and taking back☺️ !
", "votes": 0, "replies": [] } ] }, { "id": 2919486, "author_name": "Takoi", "author_url": "", "post_date": "2024-07-12T23:54:56.677000", "content": "@jerrylin96
\nI interpreted that the host allowed the use of leaks as long as the submission doesn't reach the prize range. Could you explain what the spirit of the competition mentioned in the discussion below is?
\n https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/513193
This is not ML competition. Chicken game.
", "votes": 10, "replies": [] }, { "id": 2919496, "author_name": "ONODERA", "author_url": "", "post_date": "2024-07-13T00:27:29.170000", "content": "It seems to be OK to use 'leak' up to 0.7890. However, those who see this comment will likely aim for 0.7891. However, those who see this comment will likely aim for 0.7892. However…
", "votes": 3, "replies": [] }, { "id": 2919503, "author_name": "heng", "author_url": "", "post_date": "2024-07-13T00:47:47.007000", "content": "What a terrible competition, the worst of the 78 competitions I have participated in on kaggle!
", "votes": 9, "replies": [] }, { "id": 2919511, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-13T01:14:55.500000", "content": "I don't know what changed your mind in only one day.
\nThis current situation is not good at all. It is like Game theory. Gold teams try to use leaked information to improve their ranking without entering the prize zone.
\nwhat a mess!
", "votes": 3, "replies": [] }, { "id": 2919517, "author_name": "ONODERA", "author_url": "", "post_date": "2024-07-13T01:24:30.717000", "content": "@charmq Can you share your 0.800 submission? It will save us time.
", "votes": 2, "replies": [] }, { "id": 2919523, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-13T01:36:06.787000", "content": "@jerrylin96 - Appreciate you're doing your best here. I also appreciate you are finishing your PhD, which is no small undertaking and is in and of itself stressful without adding extra stress.
\nI do not mean to pile on here but I feel like some polite push back is appropriate. You're using the term 'sufficiently' here - but what exactly constitutes sufficiently is left completely open to interpretation.
\nIn aggregate 10s, if not 100s, of thousands of human hours have been committed to this competition. Participant's efforts (all of them) deserve to be recognized with a more robust response and solution.
", "votes": 4, "replies": [] }, { "id": 2919573, "author_name": "tereka", "author_url": "", "post_date": "2024-07-13T02:37:41.797000", "content": "@jerrylin96
\nI can understand your opinion, I guess that it is difficult to check the implementation of all participants.
\nHowever our purpose is not only prize, but also medals, especially kaggle GM is one of the honor title, then gold medal is very great achievement.
if you will disqualify team that use leak solutions, you should check gold medals at least.
", "votes": 7, "replies": [ { "id": 2919613, "author_name": "", "author_url": "", "post_date": "2024-07-13T03:37:13.723000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2919615, "author_name": "pao", "author_url": "", "post_date": "2024-07-13T03:37:47.577000", "content": "@jerrylin96
\nEven if it's difficult to check all submissions and they are not perfect, you could broaden the scope a bit more and at least request the submission of source code for those within the gold range. You should be able to check as much as possible.
At the very least, I am more focused on whether I can get a gold medal than whether I can get a prize because the title of GM is at stake for me. There are many other Kagglers who also value medals highly. I think the current decision disregards the feelings of these Kagglers.
\nI believe that only with a good relationship between the host and Kagglers can good solutions be provided. Is it acceptable for someone who could genuinely provide a good solution to use a leak, thus making their solution worthless to you?
\nPlease reconsider
", "votes": 7, "replies": [] }, { "id": 2920440, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-13T16:32:54.187000", "content": "I emailed Kaggle staff last night to see if we could vet at least the gold medal submissions. You are all correct that in an ideal scenario we would vet every submission. I can say for a fact that I personally do not have the resources to do that, though.
", "votes": 5, "replies": [ { "id": 2920450, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-13T16:36:00.470000", "content": "That is understood. Can you please work with Kaggle to ensure those who would have medalled as of the date of the leak, will be awarded the medal they would have won at least for those in bronze (and maybe silver). Results can be indexed to the lowest bronze medal as of date of the leak.
", "votes": -1, "replies": [] }, { "id": 2920610, "author_name": "Camaro", "author_url": "", "post_date": "2024-07-13T17:55:41.647000", "content": "@seanmacmaghnusa
\nI understand your ungry.
\nBut I think it's impossible for both Kaggle and the participants. To check whether the team is legitimate or not, they must prepare documentation and show reproducibility of their solution, which is the same amount of work as receiving a prize. In my opinion, gold zone teams have enough motivation to do that, but enforcing this on silver and bronze teams may lead to undesirable situations.
@bamps53 True. I will add that this is a difficult leak to find, and difficult no less to exploit (building 2D->1D model is significantly harder than 1D->1D. At least in my guess, I did not build one myself. It's not the kind of leak that give you easy win with few lines of code). I don't believe there are many teams that have a good leaky solution, and most of them are probably good enough to enter silver zone with 1D->1D solution too. So people below gold range really should not be too concerned, imho.
", "votes": 3, "replies": [] }, { "id": 2920693, "author_name": "Camaro", "author_url": "", "post_date": "2024-07-13T18:25:10.963000", "content": "\nI understand that you are currently in a difficult situation. Despite this, I genuinely feel that you are doing your best to ensure the solutions you obtain are the best for you.
\nSome people say that checking the prize zone and the gold zone simply doubles the cost, but that's not the case. The former was originally planned, while the latter incurs additional costs. There is a significant gap here.
\nIf these costs are unacceptable, I have a proposal.
\nThe gold zone check can be limited to validation of the inference code. Checking the training code is more than ten times harder than checking the inference code. However, the 2D-1D or 2D-2D prediction that you really want to prevent should be able to be confirmed only by checking the inference code.
\nI understand this is not perfect (at least you should not declare so to the participants in advance), but I think it is a realistic method.
\nI propose this as a compromise if there is a conflict between what we want to achieve and the costs while negotiation with Kaggle.
Despite various troubles, I have generally enjoyed the challenging tasks of this competition.
\nLet's do our best for the remaining period!
Not an unreasonable comment. There were other options here though. Whether the intent was to check beyond gold 'zone' or not, publicly stating they wouldn't check lead to an acceleration of new submissions with 0.005 to 0.02 improvements. Those will shake up bronze and move people who are not using the leak out of medal position. That was an unforced error and is hugely unfortunate.
\nThere also were also other options here including early end post leak or another extension.
\nMaybe I should just go work on the leak for the day.
", "votes": 0, "replies": [] } ] }, { "id": 2921279, "author_name": "steubk", "author_url": "", "post_date": "2024-07-14T09:29:44.657000", "content": "Competition has turned into a variant of Thaler's 'number guessing game' :
\nchoose a leaked submission with the goal of making your submission as close as possible to the first five participants in the contest that don't use the leak.
", "votes": 0, "replies": [] } ] }, { "id": 2919268, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-12T19:04:48.150000", "content": "Looking into this and will get back to you.
", "votes": -4, "replies": [] }, { "id": 2919623, "author_name": "Chengwei Yan", "author_url": "", "post_date": "2024-07-13T03:43:21.730000", "content": "Before any leakage, my score was 0.731, and I had eight different results that all roughly achieved a score of 0.731. I hadn't used Ensemble methods before. Now, using four of these results in an ensemble allowed me to reach 0.737, and using another four, I reached 0.738. Although the increase wasn't as much as 0.01, I believe that an improvement of 0.01 is possible solely through ensembling. Therefore, it's difficult to definitively attribute the 0.01 improvement to any specific cause.
", "votes": 0, "replies": [] }, { "id": 2923561, "author_name": "", "author_url": "", "post_date": "2024-07-16T01:46:11.557000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2918738": "My team has dropped 10 places since the leak. As of yet, we have not used the leak.\n\nI presume the organizers will have a hard time identifying all who use the leak as teams may generate predictions with the leak offline such that the code cannot be interrogated.\n\nWould like to know if the leak is now defacto legal or if users of the leak will be disqualified, and if that is the plan, how?", "2919525": "May I ask for a clarification?\nThe action you need to take is to ensure that if a reviewer's solution submission is found to have a leak, then it should be removed from the leaderboard. This is because for most participants, the focus is on their leaderboard position and Kaggle medal, rather than money.\nYour action will lead to significantly different decisions down the line.\nFor teams that have the capability to identify leaks but actually follow your competition guidelines by not using them, please remember that they initially trusted you and stood with you. Now, they need a strong assurance from you to be able to sleep well.\n\nFurthermore, based on my above point, I would suggest expanding the scope of the review to include the gold medal zone, not just the prize area. @jerrylin96 ", "2918843": "I also think that non-gold medalists fall under the table in this entire discussion. The top 10 ranks in a competition may be the most important ones, especially for the host, but the solution is not great for anyone in the silver range or below.", "2919051": "i think we should expect hosts verify only gold medals, IMHO what is below top 20 doesn't matter for them,manually check code of every competitor is to expensive, so probably teams that use leak in winning position for silver or bronze medals, personally i am not using leak.", "2918776": "If I may suggest a solution, sudden 0.01 to 0.03 improvements in scores since posting of the leak should all be manually reviewed including the contents of all flat files used for ensembling.\n\nThis wont capture teams using the leak who were aware prior to posting unfortunately, it should facilitate a thorough and fair review of submissions for the majority of participants.", "2919872": "I believe that any explicit use of timestamp and location information constitutes exploiting a loophole, which goes against the original intention of the hosts and the competition. Based on my estimation, the teams in the gold medal zone have likely used timestamp and location information explicitly, which is unfair to every other participating team. It is truly disheartening.", "2919334": "We won't be able to check submissions not in the running for prize money, unfortunately. That being said, submissions discovered to use a leak will be disqualified from the leaderboard.\n\nThis comment has been edited in response to community feedback.", "2919268": "Looking into this and will get back to you.", "2919623": "Before any leakage, my score was 0.731, and I had eight different results that all roughly achieved a score of 0.731. I hadn't used Ensemble methods before. Now, using four of these results in an ensemble allowed me to reach 0.737, and using another four, I reached 0.738. Although the increase wasn't as much as 0.01, I believe that an improvement of 0.01 is possible solely through ensembling. Therefore, it's difficult to definitively attribute the 0.01 improvement to any specific cause.", "2923561": "" } }