{ "id": 519772, "title": "Kaggle Competition Update [NEW]", "url": "/competitions/leap-atmospheric-physics-ai-climsim/discussion/519772", "author_name": "Jerry Lin", "post_date": "2024-07-12T18:26:01.570000", "votes": 13, "comment_count": 45, "views": 0, "content": "

Dear Kaggle community,

\n

We’re almost at the end of the LEAP competition! Thank you so much for your persistent, hard work and for the engaging intellectual conversations you’ve fostered over the past few months. Scientists at LEAP (and elsewhere) are waiting with eager anticipation to see what you’ve learned and innovated as part of this competition. Even if your submission did not make it to the top 5, please do not hesitate to reach out to our community to foster collaborations. There will likely be opportunities to be involved in publications, which would be a way for your innovations to endure past this competition!

\n

As some of you may know by reading the literature associated with this competition, our end goal is to design a machine learning emulator capable of acting as a drop-in replacement for the cloud-resolving model in E3SM-MMF. We do this because explicitly resolving subgrid physics can be prohibitively expensive while large-scale atmospheric motion is much more tractable in comparison.

\n

To that end, our insistence on 1D-1D solutions is informed by the fact that 2D-1D or 2D-2D solutions would be solving for a fundamentally different problem. We would like to outsource to machine learning only what is absolutely necessary due to computational constraints for our hybrid physics-ML approach, and this means constraining what information these emulators have access to. Because of this, winning solutions that take advantage of the leak will be disqualified. However, we will not be releasing a new test set or extending the deadline.

\n

This should not come as a surprise as we refreshed our test set and extended the competition to address the first leak, and this new leak is more severe than the first. Given how it was released only a few days before the competition deadline, options for addressing it in a way that satisfies all participants are limited, and we apologize for negatively impacting participants who may have been making an honest attempt at pushing the state-of-the-art. This competition has been a learning experience for us as well, and we hope to mitigate instances of problematic leaks in the future.

\n

Thank you again for your amazing work. We’re almost at the end. We look forward to learning from you, and we wish you the best of luck!

\n

Best regards,

\n

LEAP competition hosts

", "messages": [ { "id": 2919546, "postDate": "2024-07-13T02:04:53.833Z", "content": "

Hi @jerrylin96 , What we care about is the honor, rank and medal, not the prize. Gold medals without prizes and with prizes are almost the same important, except for their corresponding points on the kaggle rankings. So please at least make sure all the gold solutions are clean.

", "rawMarkdown": "Hi @jerrylin96 , What we care about is the honor, rank and medal, not the prize. Gold medals without prizes and with prizes are almost the same important, except for their corresponding points on the [kaggle rankings](https://www.kaggle.com/rankings?group=competitions). So please at least make sure all the gold solutions are clean.", "votes": 13, "replies": [ { "id": 2919548, "postDate": "2024-07-13T02:05:34.310Z", "content": "

If you want to verify all the gold solutions, but Kaggle thinks it's too exhausting and expensive, and wants to only verify the prize zone. Please make this situation public. Most of the participants that obey the rules will stand with you.

", "rawMarkdown": "If you want to verify all the gold solutions, but Kaggle thinks it's too exhausting and expensive, and wants to only verify the prize zone. Please make this situation public. Most of the participants that obey the rules will stand with you. ", "replies": [ { "id": 2919654, "postDate": "2024-07-13T05:07:47.677Z", "content": "

I would personally like to verify all submissions in the gold and prize zone. I was given the impression that was not possible , but I will bring it up again.

", "rawMarkdown": "I would personally like to verify all submissions in the gold and prize zone. I was given the impression that was not possible , but I will bring it up again.", "votes": 14 }, { "id": 2919663, "postDate": "2024-07-13T05:23:26.270Z", "content": "

Thanks for your confirmation and efforts. I was wonderring why you changed your mind so quickly. The only assumption I made before is that kaggle does not allow you do that. Now it is proved.

\n

Disappointed. :(

", "rawMarkdown": "Thanks for your confirmation and efforts. I was wonderring why you changed your mind so quickly. The only assumption I made before is that kaggle does not allow you do that. Now it is proved. \n\nDisappointed. :(", "votes": 2 }, { "id": 2919672, "postDate": "2024-07-13T05:34:24.350Z", "content": "

Okay. We stand with you @jerrylin96 .

\n

We hope the kaggle staff will think carefully about this issue. We all know it brings external effort to verify all the gold solutions.

\n

However, as mentioned here, it's credibility that makes kaggle so different from other competition platforms. Honestly, if we want to earn money from competitions, there are many other choices that can earn more money with less effort than Kaggle.

\n

We love Kaggle because of its credibility. Please don't hurt it just to reduce some workload. It's not worth it . Your credibility is muuuuuuuuuuuuuuuuuuuuch more valuable than this workload.

", "rawMarkdown": "Okay. We stand with you @jerrylin96 . \n\nWe hope the kaggle staff will think carefully about this issue. We all know it brings external effort to verify all the gold solutions. \n\nHowever, as mentioned [here](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519772#2919557), it's credibility that makes kaggle so different from other competition platforms. Honestly, if we want to earn money from competitions, there are many other choices that can earn more money with less effort than Kaggle. \n\nWe love Kaggle because of its credibility. Please don't hurt it just to reduce some workload. It's not worth it . Your credibility is muuuuuuuuuuuuuuuuuuuuch more valuable than this workload.", "votes": 3 }, { "id": 2919675, "postDate": "2024-07-13T05:40:22.673Z", "content": "

Even if we are in a team that wins a championship, we may earn 5k~10k dollars. But the effort is so huge, and we can make much more money if we make the same effort on other things, than playing on Kaggle. What we care about is the medal and the honor.

", "rawMarkdown": "Even if we are in a team that wins a championship, we may earn 5k~10k dollars. But the effort is so huge, and we can make much more money if we make the same effort on other things, than playing on Kaggle. What we care about is the medal and the honor. ", "votes": 3 } ] }, { "id": 2919553, "postDate": "2024-07-13T02:13:59.793Z", "content": "

To the participants who want to use the leak:

\n

Most of you are Grandmasters or Masters. Before using leaks in your final submissions, please consider carefully: is this gold medal without a prize, more important than the reputation you've cultivated in the community over all this time?

\n

If someone profits from leaks under these circumstances, even if they finally end up top1 on Kaggle rankings, we(I believe the majority of participants in Kaggle) – will have no respect for him.

\n

Would a throne without the support of the community truly hold any meaning for you?

", "rawMarkdown": "To the participants who want to use the leak:\n---------------------------------\nMost of you are Grandmasters or Masters. Before using leaks in your final submissions, please consider carefully: is this gold medal without a prize, more important than the reputation you've cultivated in the community over all this time?\n\n If someone profits from leaks under these circumstances, even if they finally end up top1 on [Kaggle rankings](https://www.kaggle.com/rankings?group=competitions), we(I believe the majority of participants in Kaggle) – will have no respect for him. \n\nWould a throne without the support of the community truly hold any meaning for you?", "votes": 12, "replies": [ { "id": 2920012, "postDate": "2024-07-13T11:27:15.993Z", "content": "

I am a bit confused why this discussion is coming up now. In past competitions, people used leaks much worse than this one to get top ranks and were pretty open about it (e.g. Optiver realized volatility competition, top 12 solutions were leak-based and completely useless to the host and people sided with the leak users). It was always deemed as part of the competition and noone lost their reputation because of it (and I am not referring to cases of clear cheating like in the PetFinder competition).
\nIn this competition the leak was announced 4 days before end, but in other competitions the leak wasn't made public at all and only known by the leak users, I am not sure if this makes it any better.
\nI respect the decision of any team deciding to use the leak, if the host states it is okay for non-prize positions. It has always been like that on Kaggle. It also wouldn't be fair to get punished by trying to not use the leak, but therefore getting a worse place.
\nMy current solution is non-leak based so I am not saying this to profit from this, but I think it is a) hard to enforce, b) will just lead to even more unfairness, and c) inconsistent with how such leaks were handled in past competitions.

", "rawMarkdown": "I am a bit confused why this discussion is coming up now. In past competitions, people used leaks much worse than this one to get top ranks and were pretty open about it (e.g. Optiver realized volatility competition, top 12 solutions were leak-based and completely useless to the host and people sided with the leak users). It was always deemed as part of the competition and noone lost their reputation because of it (and I am not referring to cases of clear cheating like in the PetFinder competition). \nIn this competition the leak was announced 4 days before end, but in other competitions the leak wasn't made public at all and only known by the leak users, I am not sure if this makes it any better.\nI respect the decision of any team deciding to use the leak, if the host states it is okay for non-prize positions. It has always been like that on Kaggle. It also wouldn't be fair to get punished by trying to not use the leak, but therefore getting a worse place.\nMy current solution is non-leak based so I am not saying this to profit from this, but I think it is a) hard to enforce, b) will just lead to even more unfairness, and c) inconsistent with how such leaks were handled in past competitions." }, { "id": 2920071, "postDate": "2024-07-13T12:06:26.430Z", "content": "

It's indeed interesting that the reaction of the community is so different. Do you have any ideas why? @fritzcremer

", "rawMarkdown": "It's indeed interesting that the reaction of the community is so different. Do you have any ideas why? @fritzcremer " }, { "id": 2920093, "postDate": "2024-07-13T12:29:31.297Z", "content": "

I really don't know, perhaps because the host has explicitly stated that they don't want the leak to be used? But it should be clear that no host in any competition will be in favor of a leaky solution haha

", "rawMarkdown": "I really don't know, perhaps because the host has explicitly stated that they don't want the leak to be used? But it should be clear that no host in any competition will be in favor of a leaky solution haha", "votes": 1 }, { "id": 2920100, "postDate": "2024-07-13T12:33:33.073Z", "content": "

Alot of it is this. We're effectively left with:

\n

Leaks are explicitly illegal, but sometimes they're defacto legal. We'll figure out the exact conditions where they're defacto legal once the competition closes. Goodluck!

", "rawMarkdown": "Alot of it is this. We're effectively left with:\n\nLeaks are explicitly illegal, but sometimes they're defacto legal. We'll figure out the exact conditions where they're defacto legal once the competition closes. Goodluck!" }, { "id": 2920114, "postDate": "2024-07-13T12:40:33.770Z", "content": "

I'm not sure myself. But it is interesting that the non-leak 'side' has more support from the community in this competition. Maybe it's just different folks, idk

", "rawMarkdown": "I'm not sure myself. But it is interesting that the non-leak 'side' has more support from the community in this competition. Maybe it's just different folks, idk", "votes": 1 }, { "id": 2920206, "postDate": "2024-07-13T14:00:35.683Z", "content": "

Maybe many participants have lost their patience after experiencing an extension period before, and this competition consumes too many machine resources, so they may just want to end it on schedule, because whether allow leaks or not, there will be another extension.

", "rawMarkdown": "Maybe many participants have lost their patience after experiencing an extension period before, and this competition consumes too many machine resources, so they may just want to end it on schedule, because whether allow leaks or not, there will be another extension.", "votes": 3 }, { "id": 2920932, "postDate": "2024-07-13T23:15:59.003Z", "content": "

I dont have that strong of a stance on it but for me its different because we all know of the leak and have been explictly told not to use it by the organizer. It is very hard to regulate and make sure that people dont actually use it now that the cat is out of the bag.

\n

It is one thing if you find a special data trick and find a way to outsmart things and then win because of it, but that isnt really the case here. We knew the intention of the competition, the host explictly said not to do something. We did a reset because of it. It was pretty clear something in the data would be able to leak out the original time or lat/lon if you plotted the series in the training individually. At that point you arent really outsmarting the group, youre just intentionally going around the hosts intention for your own gain

", "rawMarkdown": "I dont have that strong of a stance on it but for me its different because we all know of the leak and have been explictly told not to use it by the organizer. It is very hard to regulate and make sure that people dont actually use it now that the cat is out of the bag.\n\nIt is one thing if you find a special data trick and find a way to outsmart things and then win because of it, but that isnt really the case here. We knew the intention of the competition, the host explictly said not to do something. We did a reset because of it. It was pretty clear something in the data would be able to leak out the original time or lat/lon if you plotted the series in the training individually. At that point you arent really outsmarting the group, youre just intentionally going around the hosts intention for your own gain", "votes": 4 }, { "id": 2920969, "postDate": "2024-07-14T00:05:48.507Z", "content": "

Still come up with new leaky submission 😰

", "rawMarkdown": "Still come up with new leaky submission 😰", "votes": 1 }, { "id": 2920973, "postDate": "2024-07-14T00:20:27.557Z", "content": "

I really don't understand why they still submit a leaky solution. 😅

", "rawMarkdown": "I really don't understand why they still submit a leaky solution. 😅", "votes": 3 }, { "id": 2920980, "postDate": "2024-07-14T01:12:14.477Z", "content": "

They are just making foolish of Kaggle and the host.

", "rawMarkdown": "They are just making foolish of Kaggle and the host.", "votes": 4 } ] }, { "id": 2919557, "postDate": "2024-07-13T02:20:51.903Z", "content": "

To Kaggle stuff:

\n

Kaggle is the best ML competition cummunity in the world (at least for now). For a long time, a Kaggle Gold Medal has carried significant weight in school applications and job markets. This prestige stems from Kaggle's rigorous scrutiny of gold-medal solutions and its strong credibility. Building credibility is a long and arduous process, yet its collapse can often occur in an instant.

\n
\n

In this competition, the workload difference between rigorously reviewing submissions in the prize zone and doing so for all gold medal zone solutions is only two times—or perhaps less than that, considering higher-ranked solutions tend to be more complex, requiring greater effort and resources to verify.

\n

Are you sure you want to risk the substantial credibility that Kaggle has built up over time just to save on this work?

", "rawMarkdown": "To Kaggle stuff:\n----------------------------\nKaggle is the best ML competition cummunity in the world (at least for now). For a long time, a Kaggle Gold Medal has carried significant weight in school applications and job markets. This prestige stems from Kaggle's rigorous scrutiny of gold-medal solutions and its strong credibility. Building credibility is a long and arduous process, yet its collapse can often occur in an instant.\n\n----------------------------\nIn this competition, the workload difference between rigorously reviewing submissions in the prize zone and doing so for all gold medal zone solutions is only two times—or perhaps less than that, considering higher-ranked solutions tend to be more complex, requiring greater effort and resources to verify. \n\n**Are you sure you want to risk the substantial credibility that Kaggle has built up over time just to save on this work?**", "votes": 13, "replies": [ { "id": 2919572, "postDate": "2024-07-13T02:37:23.933Z", "content": "

@forcewithme
\nWhile there may have been differences in rule interpretation, I respect your consistent stance against utilizing leaks in competitions.
\nI share your concerns regarding the potential damage to Kaggle's reputation, especially since the host has clearly prohibited the use of leaks as a rule.

\n

@inversion @mylesoneill @ashleychow
\nCould you please check this post and provide your opinion or approach for addressing it?

", "rawMarkdown": "@forcewithme \nWhile there may have been differences in rule interpretation, I respect your consistent stance against utilizing leaks in competitions. \nI share your concerns regarding the potential damage to Kaggle's reputation, especially since the host has clearly prohibited the use of leaks as a rule.\n\n@inversion @mylesoneill @ashleychow \nCould you please check this post and provide your opinion or approach for addressing it?", "votes": 10 } ] } ] }, { "id": 2919198, "postDate": "2024-07-12T18:26:01.570Z", "content": "

Dear Kaggle community,

\n

We’re almost at the end of the LEAP competition! Thank you so much for your persistent, hard work and for the engaging intellectual conversations you’ve fostered over the past few months. Scientists at LEAP (and elsewhere) are waiting with eager anticipation to see what you’ve learned and innovated as part of this competition. Even if your submission did not make it to the top 5, please do not hesitate to reach out to our community to foster collaborations. There will likely be opportunities to be involved in publications, which would be a way for your innovations to endure past this competition!

\n

As some of you may know by reading the literature associated with this competition, our end goal is to design a machine learning emulator capable of acting as a drop-in replacement for the cloud-resolving model in E3SM-MMF. We do this because explicitly resolving subgrid physics can be prohibitively expensive while large-scale atmospheric motion is much more tractable in comparison.

\n

To that end, our insistence on 1D-1D solutions is informed by the fact that 2D-1D or 2D-2D solutions would be solving for a fundamentally different problem. We would like to outsource to machine learning only what is absolutely necessary due to computational constraints for our hybrid physics-ML approach, and this means constraining what information these emulators have access to. Because of this, winning solutions that take advantage of the leak will be disqualified. However, we will not be releasing a new test set or extending the deadline.

\n

This should not come as a surprise as we refreshed our test set and extended the competition to address the first leak, and this new leak is more severe than the first. Given how it was released only a few days before the competition deadline, options for addressing it in a way that satisfies all participants are limited, and we apologize for negatively impacting participants who may have been making an honest attempt at pushing the state-of-the-art. This competition has been a learning experience for us as well, and we hope to mitigate instances of problematic leaks in the future.

\n

Thank you again for your amazing work. We’re almost at the end. We look forward to learning from you, and we wish you the best of luck!

\n

Best regards,

\n

LEAP competition hosts

", "rawMarkdown": "Dear Kaggle community,\n\nWe’re almost at the end of the LEAP competition! Thank you so much for your persistent, hard work and for the engaging intellectual conversations you’ve fostered over the past few months. Scientists at LEAP (and elsewhere) are waiting with eager anticipation to see what you’ve learned and innovated as part of this competition. Even if your submission did not make it to the top 5, please do not hesitate to reach out to our community to foster collaborations. There will likely be opportunities to be involved in publications, which would be a way for your innovations to endure past this competition!\n\nAs some of you may know by reading the literature associated with this competition, our end goal is to design a machine learning emulator capable of acting as a drop-in replacement for the cloud-resolving model in E3SM-MMF. We do this because explicitly resolving subgrid physics can be prohibitively expensive while large-scale atmospheric motion is much more tractable in comparison.\n\nTo that end, our insistence on 1D-1D solutions is informed by the fact that 2D-1D or 2D-2D solutions would be solving for a fundamentally different problem. We would like to outsource to machine learning only what is absolutely necessary due to computational constraints for our hybrid physics-ML approach, and this means constraining what information these emulators have access to. Because of this, winning solutions that take advantage of the leak will be disqualified. However, we will not be releasing a new test set or extending the deadline.\n\nThis should not come as a surprise as we refreshed our test set and extended the competition to address the first leak, and this new leak is more severe than the first. Given how it was released only a few days before the competition deadline, options for addressing it in a way that satisfies all participants are limited, and we apologize for negatively impacting participants who may have been making an honest attempt at pushing the state-of-the-art. This competition has been a learning experience for us as well, and we hope to mitigate instances of problematic leaks in the future.\n\nThank you again for your amazing work. We’re almost at the end. We look forward to learning from you, and we wish you the best of luck!\n\nBest regards,\n\nLEAP competition hosts", "votes": 13 }, { "id": 2921040, "postDate": "2024-07-14T03:51:27.983Z", "content": "

hi @jerrylin96 ,did u see Did you see the LB won the bronze medal increase from 0.69 to 0.70 to 0.71 ?It's probably going to be 0.72 by the end of the day.
\nI went through this process and it was nice to see the progress I was making, but more often than not I felt helpless，just like:

\n
\n

So we beat on, boats against the current, borne back ceaselessly into the past.

\n
\n

Check if leak is used,In most cases it may only be necessary to check
\ndf_p_test[] = -df_test[]

\n

If he uses the integration model（ensemble),Just need it to present a dataset or a program that generates a csv.

\n

Ensuring the reliability of the gold medal is fairness in my opinion, ensuring the reliability of all participants is fairness to a greater extent.

", "rawMarkdown": "hi @jerrylin96 ,did u see Did you see the LB won the bronze medal increase from 0.69 to 0.70 to 0.71 ?It's probably going to be 0.72 by the end of the day.\nI went through this process and it was nice to see the progress I was making, but more often than not I felt helpless，just like:\n\n>So we beat on, boats against the current, borne back ceaselessly into the past.\n\nCheck if leak is used,In most cases it may only be necessary to check\n`df_p_test[] = -df_test[]`\n\nIf he uses the integration model（ensemble),Just need it to present a dataset or a program that generates a csv.\n\nEnsuring the reliability of the gold medal is fairness in my opinion, ensuring the reliability of all participants is fairness to a greater extent.", "votes": 6, "replies": [ { "id": 2922449, "postDate": "2024-07-15T07:37:52.423Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2924921, "postDate": "2024-07-16T19:52:27.520Z", "content": "

Hi @jerrylin96! Thank you for running such a great competition 😁 Even as complications arose, you handled them as gracefully as possible by making appropriate decisions and communicating them in a timely fashion. It was a fantastic learning experience and a privilege to contribute to research that ultimately impacts our entire planet.

", "rawMarkdown": "Hi @jerrylin96! Thank you for running such a great competition 😁 Even as complications arose, you handled them as gracefully as possible by making appropriate decisions and communicating them in a timely fashion. It was a fantastic learning experience and a privilege to contribute to research that ultimately impacts our entire planet.", "votes": 5 }, { "id": 2919233, "postDate": "2024-07-12T18:42:18.807Z", "content": "

@jerrylin96 Judging by your main aim (1D->1D model) I guess that using timespace during training (in 1D->1D scenario) is not a problem. Still, it would be nice to get a confirmation. I specifically used timespace data as auxiliary loss during training. It was just a small thing that gave a small bump, and I saw no problem with it (especially as this data was explicitly available on HF low-res dataset and at no point was referred to as something to avoid). Nevertheless, with recent discussions, I started to get worried. See also this comment, which asks for clarifications regarding this matter and similar. Thank you.

", "rawMarkdown": "@jerrylin96 Judging by your main aim (1D->1D model) I guess that using timespace during training (in 1D->1D scenario) is not a problem. Still, it would be nice to get a confirmation. I specifically used timespace data as auxiliary loss during training. It was just a small thing that gave a small bump, and I saw no problem with it (especially as this data was explicitly available on HF low-res dataset and at no point was referred to as something to avoid). Nevertheless, with recent discussions, I started to get worried. See also [this comment](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519575#2918339), which asks for clarifications regarding this matter and similar. Thank you.", "votes": 4, "replies": [ { "id": 2919278, "postDate": "2024-07-12T19:10:42.420Z", "content": "

Hi greySnow, can you clarify what you mean by using timespace data? Hoping to clarify so that you and others aren't left in the dark.

", "rawMarkdown": "Hi greySnow, can you clarify what you mean by using timespace data? Hoping to clarify so that you and others aren't left in the dark.", "votes": 2, "replies": [ { "id": 2919304, "postDate": "2024-07-12T19:22:01.373Z", "content": "

We know the exact time stamp and lon/lat for the training set (which was not shuffled in Kaggle and came with this explicit information on HF). During training, we can use this data. One of the most common practices for data that exists for the training set but not for the test set is to use it as an auxiliary loss. That is, trying to predict the time and lat/lon of each sample during training. I want to stress that it was not a leak (data was explicitly public), and using it was genuinely done in good faith.
\nMoreover, I did not use it to try to infer the test set and rearrange it (turning it into an unwanted 2D-1D problem). It was just a simple auxiliary loss. Honestly, it was just a small bump. I'm pretty sure I can get a very similar score without it. The problem is that the deadline is three days, so retraining from scratch is a problem.

", "rawMarkdown": "We know the exact time stamp and lon/lat for *the training set* (which was not shuffled in Kaggle and came with this explicit information on HF). During training, we can use this data. One of the most common practices for data that exists for the training set but not for the test set is to use it as an auxiliary loss. That is, trying to predict the time and lat/lon of each sample during training. I want to stress that it was not a leak (data was explicitly public), and using it was genuinely done in good faith.\nMoreover, I did not use it to try to infer the test set and rearrange it (turning it into an unwanted 2D-1D problem). It was just a simple auxiliary loss. Honestly, it was just a small bump. I'm pretty sure I can get a very similar score without it. The problem is that the deadline is three days, so retraining from scratch is a problem.", "votes": 2 }, { "id": 2919322, "postDate": "2024-07-12T19:34:11.763Z", "content": "

If you are not reverse engineering the test set, that seems fine to me.

", "rawMarkdown": "If you are not reverse engineering the test set, that seems fine to me.", "votes": 5 }, { "id": 2919343, "postDate": "2024-07-12T19:37:21.740Z", "content": "

Thank you for the confirmation. Now I can go to sleep without worrying 🤣

", "rawMarkdown": "Thank you for the confirmation. Now I can go to sleep without worrying 🤣 ", "votes": 3 } ] } ] }, { "id": 2919413, "postDate": "2024-07-12T20:52:26.033Z", "content": "

If you would like to be considered for a cash prize, please submit your NON-LEAK based solution. Do not submit a leak based solution and report the score without the leak in your team name.

", "rawMarkdown": "If you would like to be considered for a cash prize, please submit your NON-LEAK based solution. Do not submit a leak based solution and report the score without the leak in your team name.", "votes": 2, "replies": [ { "id": 2919425, "postDate": "2024-07-12T21:02:16.663Z", "content": "

Does that mean teams with leaky solutions would still secure their positions on the leaderboard?

", "rawMarkdown": "Does that mean teams with leaky solutions would still secure their positions on the leaderboard? ", "votes": 2, "replies": [ { "id": 2919427, "postDate": "2024-07-12T21:07:06.297Z", "content": "
\n

Does that mean teams with leaky solutions would still secure their positions on the leaderboard?

\n
\n

Hmmm I sure hope it won't be top 10 leaky gold medals and then prize money winners with silver 😅

", "rawMarkdown": ">Does that mean teams with leaky solutions would still secure their positions on the leaderboard? \n\nHmmm I sure hope it won't be top 10 leaky gold medals and then prize money winners with silver 😅", "votes": 3 } ] }, { "id": 2919497, "postDate": "2024-07-13T00:28:18.983Z", "content": "
\n

If you would like to be considered for a cash prize, please submit your NON-LEAK based solution.

\n
\n

Only the top 5 (cash zone) teams, not all the gold zone teams?

", "rawMarkdown": ">If you would like to be considered for a cash prize, please submit your NON-LEAK based solution.\n\nOnly the top 5 (cash zone) teams, not all the gold zone teams?", "votes": 1, "replies": [ { "id": 2919500, "postDate": "2024-07-13T00:45:09.827Z", "content": "

@jerrylin96

\n

Please let me emphasize this point. Many participants are competing for the gold medal rather than the prize money. It is a proof of skill by Kaggle and is worth as much or even more than the monetary prize.

\n

If you simply say, \"Don't use leaks to receive the prize, but it's okay to use them if you aim for rankings or medals,\" many people would take that approach (although it may be too late now…).

\n

What I want to assert are the following two points:
\n1) All solutions from teams within the gold medal range should be checked.
\n2) If the use of leaks is confirmed, that team should be excluded from the leaderboard.

", "rawMarkdown": "@jerrylin96 \n\nPlease let me emphasize this point. Many participants are competing for the gold medal rather than the prize money. It is a proof of skill by Kaggle and is worth as much or even more than the monetary prize. \n\nIf you simply say, \"Don't use leaks to receive the prize, but it's okay to use them if you aim for rankings or medals,\" many people would take that approach (although it may be too late now...).\n\nWhat I want to assert are the following two points:\n1) All solutions from teams within the gold medal range should be checked.\n2) If the use of leaks is confirmed, that team should be excluded from the leaderboard.", "votes": 6 }, { "id": 2919507, "postDate": "2024-07-13T01:00:43.343Z", "content": "

I think the discussion process between jerry and kaggle staff was:

\n

jerry: We should check all submissions.
\nkaggle: We don’t have that much energy and resources to do it.
\njerry: Can we at least check the gold medal zone?
\nkaggle: Sorry, bro.
\njerry: Cash zone?
\nkaggle: OK 😎

", "rawMarkdown": "I think the discussion process between jerry and kaggle staff was:\n\njerry: We should check all submissions.\nkaggle: We don’t have that much energy and resources to do it.\njerry: Can we at least check the gold medal zone?\nkaggle: Sorry, bro.\njerry: Cash zone?\nkaggle: OK 😎", "votes": 3 } ] }, { "id": 2919513, "postDate": "2024-07-13T01:16:06.753Z", "content": "

I don't know what changed your mind in only one day.

\n\n

This current situation is not good at all. It is like Game theory. Gold teams try to use leaked information to improve their ranking without entering the prize zone.

\n

what a mess!

", "rawMarkdown": "I don't know what changed your mind in only one day.\n\n- if it is kaggle who don't allow you check gold solution, I will feel really disappointed on kaggle. No flexibility at all. Very rigid.\n\n- if it is the time/machine resource, we have 11 golds and 5 prize position, only twice workload. I don't think it is a big problem\n\nThis current situation is not good at all. It is like Game theory. Gold teams try to use leaked information to improve their ranking without entering the prize zone.\n\nwhat a mess!", "votes": 2 }, { "id": 2921125, "postDate": "2024-07-14T05:49:58.370Z", "content": "

Dear @jerrylin96
\nIn IMC 2024, cheating was rampant among the medal-winning participants, so Kaggle spent a week after the competition to clean up the medal-winning participants. Why is cheating (using leaks to increase scores) allowed in this competition?

", "rawMarkdown": "Dear @jerrylin96\nIn [IMC 2024](https://www.kaggle.com/competitions/image-matching-challenge-2024), [cheating](https://www.kaggle.com/competitions/image-matching-challenge-2024/discussion/509696) was rampant among the medal-winning participants, so Kaggle spent a week after the competition to clean up the medal-winning participants. Why is cheating (using leaks to increase scores) allowed in this competition?", "votes": 1 } ] }, { "id": 2919388, "postDate": "2024-07-12T20:22:55.050Z", "content": "

@jerrylin96

\n

Can you disambiguate, does winning solution == medalling solution?

\n

If not can you qualify the approach you will take for bronze and silver participants?

", "rawMarkdown": "@jerrylin96 \n\nCan you disambiguate, does winning solution == medalling solution?\n\nIf not can you qualify the approach you will take for bronze and silver participants?", "votes": 1, "replies": [ { "id": 2919409, "postDate": "2024-07-12T20:50:07.377Z", "content": "

He answered it here

", "rawMarkdown": "He answered it [here](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519720#2919334)" } ] }, { "id": 2919499, "postDate": "2024-07-13T00:34:36.607Z", "content": "

@jerrylin96

\n

We can submit two entries. If we submit both a submission with leaks and a submission without leaks, and both are within the prize range, what will happen? (The current 3rd place corresponds to this situation)

\n

1) The leaderboard and medals will not be affected. The prize money will not be awarded to that team and will be passed down.
\n2) If any submission used leaks, that team will be completely excluded from the leaderboard, and the rankings will be recalculated.
\n3) If any submission used leaks, the score will be calculated with the other submission, and the rankings will be recalculated.

", "rawMarkdown": "@jerrylin96 \n\nWe can submit two entries. If we submit both a submission with leaks and a submission without leaks, and both are within the prize range, what will happen? (The current 3rd place corresponds to this situation)\n\n1) The leaderboard and medals will not be affected. The prize money will not be awarded to that team and will be passed down.\n2) If any submission used leaks, that team will be completely excluded from the leaderboard, and the rankings will be recalculated.\n3) If any submission used leaks, the score will be calculated with the other submission, and the rankings will be recalculated.", "votes": 2 }, { "id": 2923765, "postDate": "2024-07-16T06:11:44.903Z", "content": "

Hi everyone, I am in learning process, Can anyone explain what the \"leak\" discussion going on this competition (if anyone have time)?!! Is it means the output of the test data released(leaked accidentally)(or any cheatcode)?!!
\nThanks!!, I am grateful for your time and the explanations/guidance you will provide!!!!!!

", "rawMarkdown": "Hi everyone, I am in learning process, Can anyone explain what the \"leak\" discussion going on this competition (if anyone have time)?!! Is it means the output of the test data released(leaked accidentally)(or any cheatcode)?!!\nThanks!!, I am grateful for your time and the explanations/guidance you will provide!!!!!!" }, { "id": 2920210, "postDate": "2024-07-13T14:04:11.233Z", "content": "

hello, is training with high-res data allowed?

", "rawMarkdown": "hello, is training with high-res data allowed?", "replies": [ { "id": 2920361, "postDate": "2024-07-13T15:50:39.390Z", "content": "

Yes, but I don’t think it would help you.

", "rawMarkdown": "Yes, but I don’t think it would help you.", "votes": 2, "replies": [ { "id": 2922737, "postDate": "2024-07-15T12:58:13.087Z", "content": "

what will happen if a team use multi sample feature to pretend using 1d->1d model? Like \"preprocessing\" data instead of simply BN/LN. It can be hidden between thousands lines of code.

", "rawMarkdown": "what will happen if a team use multi sample feature to pretend using 1d->1d model? Like \"preprocessing\" data instead of simply BN/LN. It can be hidden between thousands lines of code." }, { "id": 2922748, "postDate": "2024-07-15T13:09:35.217Z", "content": "

It's not so easy to hide such a thing. Inference is easy to follow, in order to make use of 2D in inference one would need to rearrange the test set according to leak time and space. This is something that is easy to catch.
\nIf you are concerned about cheating, you should be much more concerned about someone hiding 2D->1D test pseudo-labels in the training set and training 1D->1D model on them. The only way to catch it is creating the dataset from scratch, ensuring no hiding of test pseudo-labels, and training on it from scratch too. Which is what host would probably have to do. And this is why validation would take so much effort.

", "rawMarkdown": "It's not so easy to hide such a thing. Inference is easy to follow, in order to make use of 2D in inference one would need to rearrange the test set according to leak time and space. This is something that is easy to catch.\nIf you are concerned about cheating, you should be much more concerned about someone hiding 2D->1D test pseudo-labels in the training set and training 1D->1D model on them. The only way to catch it is creating the dataset from scratch, ensuring no hiding of test pseudo-labels, and training on it from scratch too. Which is what host would probably have to do. And this is why validation would take so much effort.", "votes": 1 }, { "id": 2923138, "postDate": "2024-07-15T17:21:45.370Z", "content": "

Yeah my biggest concern is pseudolabeling the test set with a 2d-> 1d model and then distilling in a 1d model those labels. Not a simple check to verify that model

", "rawMarkdown": "Yeah my biggest concern is pseudolabeling the test set with a 2d-> 1d model and then distilling in a 1d model those labels. Not a simple check to verify that model", "votes": 2 } ] } ] }, { "id": 2925017, "postDate": "2024-07-16T21:16:47.690Z", "rawMarkdown": "", "votes": -1, "isDeleted": true } ], "comments": [ { "id": 2919546, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-13T02:04:53.833000", "content": "

Hi @jerrylin96 , What we care about is the honor, rank and medal, not the prize. Gold medals without prizes and with prizes are almost the same important, except for their corresponding points on the kaggle rankings. So please at least make sure all the gold solutions are clean.

", "votes": 13, "replies": [ { "id": 2919548, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-13T02:05:34.310000", "content": "

If you want to verify all the gold solutions, but Kaggle thinks it's too exhausting and expensive, and wants to only verify the prize zone. Please make this situation public. Most of the participants that obey the rules will stand with you.

", "votes": 0, "replies": [ { "id": 2919654, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-13T05:07:47.677000", "content": "

I would personally like to verify all submissions in the gold and prize zone. I was given the impression that was not possible , but I will bring it up again.

", "votes": 14, "replies": [] }, { "id": 2919663, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-13T05:23:26.270000", "content": "

Thanks for your confirmation and efforts. I was wonderring why you changed your mind so quickly. The only assumption I made before is that kaggle does not allow you do that. Now it is proved.

\n

Disappointed. :(

", "votes": 2, "replies": [] }, { "id": 2919672, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-13T05:34:24.350000", "content": "

Okay. We stand with you @jerrylin96 .

\n

We hope the kaggle staff will think carefully about this issue. We all know it brings external effort to verify all the gold solutions.

\n

However, as mentioned here, it's credibility that makes kaggle so different from other competition platforms. Honestly, if we want to earn money from competitions, there are many other choices that can earn more money with less effort than Kaggle.

\n

We love Kaggle because of its credibility. Please don't hurt it just to reduce some workload. It's not worth it . Your credibility is muuuuuuuuuuuuuuuuuuuuch more valuable than this workload.

", "votes": 3, "replies": [] }, { "id": 2919675, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-13T05:40:22.673000", "content": "

Even if we are in a team that wins a championship, we may earn 5k~10k dollars. But the effort is so huge, and we can make much more money if we make the same effort on other things, than playing on Kaggle. What we care about is the medal and the honor.

", "votes": 3, "replies": [] } ] }, { "id": 2919553, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-13T02:13:59.793000", "content": "

To the participants who want to use the leak:

\n

Most of you are Grandmasters or Masters. Before using leaks in your final submissions, please consider carefully: is this gold medal without a prize, more important than the reputation you've cultivated in the community over all this time?

\n

If someone profits from leaks under these circumstances, even if they finally end up top1 on Kaggle rankings, we(I believe the majority of participants in Kaggle) – will have no respect for him.

\n

Would a throne without the support of the community truly hold any meaning for you?

", "votes": 12, "replies": [ { "id": 2920012, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-07-13T11:27:15.993000", "content": "

I am a bit confused why this discussion is coming up now. In past competitions, people used leaks much worse than this one to get top ranks and were pretty open about it (e.g. Optiver realized volatility competition, top 12 solutions were leak-based and completely useless to the host and people sided with the leak users). It was always deemed as part of the competition and noone lost their reputation because of it (and I am not referring to cases of clear cheating like in the PetFinder competition).
\nIn this competition the leak was announced 4 days before end, but in other competitions the leak wasn't made public at all and only known by the leak users, I am not sure if this makes it any better.
\nI respect the decision of any team deciding to use the leak, if the host states it is okay for non-prize positions. It has always been like that on Kaggle. It also wouldn't be fair to get punished by trying to not use the leak, but therefore getting a worse place.
\nMy current solution is non-leak based so I am not saying this to profit from this, but I think it is a) hard to enforce, b) will just lead to even more unfairness, and c) inconsistent with how such leaks were handled in past competitions.

", "votes": 0, "replies": [] }, { "id": 2920071, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-13T12:06:26.430000", "content": "

It's indeed interesting that the reaction of the community is so different. Do you have any ideas why? @fritzcremer

", "votes": 0, "replies": [] }, { "id": 2920093, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2024-07-13T12:29:31.297000", "content": "

I really don't know, perhaps because the host has explicitly stated that they don't want the leak to be used? But it should be clear that no host in any competition will be in favor of a leaky solution haha

", "votes": 1, "replies": [] }, { "id": 2920100, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-13T12:33:33.073000", "content": "

Alot of it is this. We're effectively left with:

\n

Leaks are explicitly illegal, but sometimes they're defacto legal. We'll figure out the exact conditions where they're defacto legal once the competition closes. Goodluck!

", "votes": 0, "replies": [] }, { "id": 2920114, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-13T12:40:33.770000", "content": "

I'm not sure myself. But it is interesting that the non-leak 'side' has more support from the community in this competition. Maybe it's just different folks, idk

", "votes": 1, "replies": [] }, { "id": 2920206, "author_name": "heng", "author_url": "", "post_date": "2024-07-13T14:00:35.683000", "content": "

Maybe many participants have lost their patience after experiencing an extension period before, and this competition consumes too many machine resources, so they may just want to end it on schedule, because whether allow leaks or not, there will be another extension.

", "votes": 3, "replies": [] }, { "id": 2920932, "author_name": "ryches", "author_url": "", "post_date": "2024-07-13T23:15:59.003000", "content": "

I dont have that strong of a stance on it but for me its different because we all know of the leak and have been explictly told not to use it by the organizer. It is very hard to regulate and make sure that people dont actually use it now that the cat is out of the bag.

\n

It is one thing if you find a special data trick and find a way to outsmart things and then win because of it, but that isnt really the case here. We knew the intention of the competition, the host explictly said not to do something. We did a reset because of it. It was pretty clear something in the data would be able to leak out the original time or lat/lon if you plotted the series in the training individually. At that point you arent really outsmarting the group, youre just intentionally going around the hosts intention for your own gain

", "votes": 4, "replies": [] }, { "id": 2920969, "author_name": "heng", "author_url": "", "post_date": "2024-07-14T00:05:48.507000", "content": "

Still come up with new leaky submission 😰

", "votes": 1, "replies": [] }, { "id": 2920973, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-14T00:20:27.557000", "content": "

I really don't understand why they still submit a leaky solution. 😅

", "votes": 3, "replies": [] }, { "id": 2920980, "author_name": "Camaro", "author_url": "", "post_date": "2024-07-14T01:12:14.477000", "content": "

They are just making foolish of Kaggle and the host.

", "votes": 4, "replies": [] } ] }, { "id": 2919557, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-07-13T02:20:51.903000", "content": "

To Kaggle stuff:

\n

Kaggle is the best ML competition cummunity in the world (at least for now). For a long time, a Kaggle Gold Medal has carried significant weight in school applications and job markets. This prestige stems from Kaggle's rigorous scrutiny of gold-medal solutions and its strong credibility. Building credibility is a long and arduous process, yet its collapse can often occur in an instant.

\n
\n

In this competition, the workload difference between rigorously reviewing submissions in the prize zone and doing so for all gold medal zone solutions is only two times—or perhaps less than that, considering higher-ranked solutions tend to be more complex, requiring greater effort and resources to verify.

\n

Are you sure you want to risk the substantial credibility that Kaggle has built up over time just to save on this work?

", "votes": 13, "replies": [ { "id": 2919572, "author_name": "kami", "author_url": "", "post_date": "2024-07-13T02:37:23.933000", "content": "

@forcewithme
\nWhile there may have been differences in rule interpretation, I respect your consistent stance against utilizing leaks in competitions.
\nI share your concerns regarding the potential damage to Kaggle's reputation, especially since the host has clearly prohibited the use of leaks as a rule.

\n

@inversion @mylesoneill @ashleychow
\nCould you please check this post and provide your opinion or approach for addressing it?

", "votes": 10, "replies": [] } ] } ] }, { "id": 2921040, "author_name": "Aristotle.Chen", "author_url": "", "post_date": "2024-07-14T03:51:27.983000", "content": "

hi @jerrylin96 ,did u see Did you see the LB won the bronze medal increase from 0.69 to 0.70 to 0.71 ?It's probably going to be 0.72 by the end of the day.
\nI went through this process and it was nice to see the progress I was making, but more often than not I felt helpless，just like:

\n
\n

So we beat on, boats against the current, borne back ceaselessly into the past.

\n
\n

Check if leak is used,In most cases it may only be necessary to check
\ndf_p_test[] = -df_test[]

\n

If he uses the integration model（ensemble),Just need it to present a dataset or a program that generates a csv.

\n

Ensuring the reliability of the gold medal is fairness in my opinion, ensuring the reliability of all participants is fairness to a greater extent.

", "votes": 6, "replies": [ { "id": 2922449, "author_name": "", "author_url": "", "post_date": "2024-07-15T07:37:52.423000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2924921, "author_name": "Marec Serlin", "author_url": "", "post_date": "2024-07-16T19:52:27.520000", "content": "

Hi @jerrylin96! Thank you for running such a great competition 😁 Even as complications arose, you handled them as gracefully as possible by making appropriate decisions and communicating them in a timely fashion. It was a fantastic learning experience and a privilege to contribute to research that ultimately impacts our entire planet.

", "votes": 5, "replies": [] }, { "id": 2919233, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-12T18:42:18.807000", "content": "

@jerrylin96 Judging by your main aim (1D->1D model) I guess that using timespace during training (in 1D->1D scenario) is not a problem. Still, it would be nice to get a confirmation. I specifically used timespace data as auxiliary loss during training. It was just a small thing that gave a small bump, and I saw no problem with it (especially as this data was explicitly available on HF low-res dataset and at no point was referred to as something to avoid). Nevertheless, with recent discussions, I started to get worried. See also this comment, which asks for clarifications regarding this matter and similar. Thank you.

", "votes": 4, "replies": [ { "id": 2919278, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-12T19:10:42.420000", "content": "

Hi greySnow, can you clarify what you mean by using timespace data? Hoping to clarify so that you and others aren't left in the dark.

", "votes": 2, "replies": [ { "id": 2919304, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-12T19:22:01.373000", "content": "

We know the exact time stamp and lon/lat for the training set (which was not shuffled in Kaggle and came with this explicit information on HF). During training, we can use this data. One of the most common practices for data that exists for the training set but not for the test set is to use it as an auxiliary loss. That is, trying to predict the time and lat/lon of each sample during training. I want to stress that it was not a leak (data was explicitly public), and using it was genuinely done in good faith.
\nMoreover, I did not use it to try to infer the test set and rearrange it (turning it into an unwanted 2D-1D problem). It was just a simple auxiliary loss. Honestly, it was just a small bump. I'm pretty sure I can get a very similar score without it. The problem is that the deadline is three days, so retraining from scratch is a problem.

", "votes": 2, "replies": [] }, { "id": 2919322, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-12T19:34:11.763000", "content": "

If you are not reverse engineering the test set, that seems fine to me.

", "votes": 5, "replies": [] }, { "id": 2919343, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-12T19:37:21.740000", "content": "

Thank you for the confirmation. Now I can go to sleep without worrying 🤣

", "votes": 3, "replies": [] } ] } ] }, { "id": 2919413, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-12T20:52:26.033000", "content": "

If you would like to be considered for a cash prize, please submit your NON-LEAK based solution. Do not submit a leak based solution and report the score without the leak in your team name.

", "votes": 2, "replies": [ { "id": 2919425, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2024-07-12T21:02:16.663000", "content": "

Does that mean teams with leaky solutions would still secure their positions on the leaderboard?

", "votes": 2, "replies": [ { "id": 2919427, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-12T21:07:06.297000", "content": "
\n

Does that mean teams with leaky solutions would still secure their positions on the leaderboard?

\n
\n

Hmmm I sure hope it won't be top 10 leaky gold medals and then prize money winners with silver 😅

", "votes": 3, "replies": [] } ] }, { "id": 2919497, "author_name": "heng", "author_url": "", "post_date": "2024-07-13T00:28:18.983000", "content": "
\n

If you would like to be considered for a cash prize, please submit your NON-LEAK based solution.

\n
\n

Only the top 5 (cash zone) teams, not all the gold zone teams?

", "votes": 1, "replies": [ { "id": 2919500, "author_name": "Camaro", "author_url": "", "post_date": "2024-07-13T00:45:09.827000", "content": "

@jerrylin96

\n

Please let me emphasize this point. Many participants are competing for the gold medal rather than the prize money. It is a proof of skill by Kaggle and is worth as much or even more than the monetary prize.

\n

If you simply say, \"Don't use leaks to receive the prize, but it's okay to use them if you aim for rankings or medals,\" many people would take that approach (although it may be too late now…).

\n

What I want to assert are the following two points:
\n1) All solutions from teams within the gold medal range should be checked.
\n2) If the use of leaks is confirmed, that team should be excluded from the leaderboard.

", "votes": 6, "replies": [] }, { "id": 2919507, "author_name": "heng", "author_url": "", "post_date": "2024-07-13T01:00:43.343000", "content": "

I think the discussion process between jerry and kaggle staff was:

\n

jerry: We should check all submissions.
\nkaggle: We don’t have that much energy and resources to do it.
\njerry: Can we at least check the gold medal zone?
\nkaggle: Sorry, bro.
\njerry: Cash zone?
\nkaggle: OK 😎

", "votes": 3, "replies": [] } ] }, { "id": 2919513, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-13T01:16:06.753000", "content": "

I don't know what changed your mind in only one day.

\n\n

This current situation is not good at all. It is like Game theory. Gold teams try to use leaked information to improve their ranking without entering the prize zone.

\n

what a mess!

", "votes": 2, "replies": [] }, { "id": 2921125, "author_name": "mh", "author_url": "", "post_date": "2024-07-14T05:49:58.370000", "content": "

Dear @jerrylin96
\nIn IMC 2024, cheating was rampant among the medal-winning participants, so Kaggle spent a week after the competition to clean up the medal-winning participants. Why is cheating (using leaks to increase scores) allowed in this competition?

", "votes": 1, "replies": [] } ] }, { "id": 2919388, "author_name": "Sean McManus", "author_url": "", "post_date": "2024-07-12T20:22:55.050000", "content": "

@jerrylin96

\n

Can you disambiguate, does winning solution == medalling solution?

\n

If not can you qualify the approach you will take for bronze and silver participants?

", "votes": 1, "replies": [ { "id": 2919409, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-12T20:50:07.377000", "content": "

He answered it here

", "votes": 0, "replies": [] } ] }, { "id": 2919499, "author_name": "Camaro", "author_url": "", "post_date": "2024-07-13T00:34:36.607000", "content": "

@jerrylin96

\n

We can submit two entries. If we submit both a submission with leaks and a submission without leaks, and both are within the prize range, what will happen? (The current 3rd place corresponds to this situation)

\n

1) The leaderboard and medals will not be affected. The prize money will not be awarded to that team and will be passed down.
\n2) If any submission used leaks, that team will be completely excluded from the leaderboard, and the rankings will be recalculated.
\n3) If any submission used leaks, the score will be calculated with the other submission, and the rankings will be recalculated.

", "votes": 2, "replies": [] }, { "id": 2923765, "author_name": "Gnanaprakash R", "author_url": "", "post_date": "2024-07-16T06:11:44.903000", "content": "

Hi everyone, I am in learning process, Can anyone explain what the \"leak\" discussion going on this competition (if anyone have time)?!! Is it means the output of the test data released(leaked accidentally)(or any cheatcode)?!!
\nThanks!!, I am grateful for your time and the explanations/guidance you will provide!!!!!!

", "votes": 0, "replies": [] }, { "id": 2920210, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-07-13T14:04:11.233000", "content": "

hello, is training with high-res data allowed?

", "votes": 0, "replies": [ { "id": 2920361, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-13T15:50:39.390000", "content": "

Yes, but I don’t think it would help you.

", "votes": 2, "replies": [ { "id": 2922737, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-07-15T12:58:13.087000", "content": "

what will happen if a team use multi sample feature to pretend using 1d->1d model? Like \"preprocessing\" data instead of simply BN/LN. It can be hidden between thousands lines of code.

", "votes": 0, "replies": [] }, { "id": 2922748, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-15T13:09:35.217000", "content": "

It's not so easy to hide such a thing. Inference is easy to follow, in order to make use of 2D in inference one would need to rearrange the test set according to leak time and space. This is something that is easy to catch.
\nIf you are concerned about cheating, you should be much more concerned about someone hiding 2D->1D test pseudo-labels in the training set and training 1D->1D model on them. The only way to catch it is creating the dataset from scratch, ensuring no hiding of test pseudo-labels, and training on it from scratch too. Which is what host would probably have to do. And this is why validation would take so much effort.

", "votes": 1, "replies": [] }, { "id": 2923138, "author_name": "ryches", "author_url": "", "post_date": "2024-07-15T17:21:45.370000", "content": "

Yeah my biggest concern is pseudolabeling the test set with a 2d-> 1d model and then distilling in a 1d model those labels. Not a simple check to verify that model

", "votes": 2, "replies": [] } ] } ] }, { "id": 2925017, "author_name": "", "author_url": "", "post_date": "2024-07-16T21:16:47.690000", "content": "", "votes": -1, "replies": [] } ], "raw_markdown_by_id": { "2919546": "Hi @jerrylin96 , What we care about is the honor, rank and medal, not the prize. Gold medals without prizes and with prizes are almost the same important, except for their corresponding points on the [kaggle rankings](https://www.kaggle.com/rankings?group=competitions). So please at least make sure all the gold solutions are clean.", "2919198": "Dear Kaggle community,\n\nWe’re almost at the end of the LEAP competition! Thank you so much for your persistent, hard work and for the engaging intellectual conversations you’ve fostered over the past few months. Scientists at LEAP (and elsewhere) are waiting with eager anticipation to see what you’ve learned and innovated as part of this competition. Even if your submission did not make it to the top 5, please do not hesitate to reach out to our community to foster collaborations. There will likely be opportunities to be involved in publications, which would be a way for your innovations to endure past this competition!\n\nAs some of you may know by reading the literature associated with this competition, our end goal is to design a machine learning emulator capable of acting as a drop-in replacement for the cloud-resolving model in E3SM-MMF. We do this because explicitly resolving subgrid physics can be prohibitively expensive while large-scale atmospheric motion is much more tractable in comparison.\n\nTo that end, our insistence on 1D-1D solutions is informed by the fact that 2D-1D or 2D-2D solutions would be solving for a fundamentally different problem. We would like to outsource to machine learning only what is absolutely necessary due to computational constraints for our hybrid physics-ML approach, and this means constraining what information these emulators have access to. Because of this, winning solutions that take advantage of the leak will be disqualified. However, we will not be releasing a new test set or extending the deadline.\n\nThis should not come as a surprise as we refreshed our test set and extended the competition to address the first leak, and this new leak is more severe than the first. Given how it was released only a few days before the competition deadline, options for addressing it in a way that satisfies all participants are limited, and we apologize for negatively impacting participants who may have been making an honest attempt at pushing the state-of-the-art. This competition has been a learning experience for us as well, and we hope to mitigate instances of problematic leaks in the future.\n\nThank you again for your amazing work. We’re almost at the end. We look forward to learning from you, and we wish you the best of luck!\n\nBest regards,\n\nLEAP competition hosts", "2921040": "hi @jerrylin96 ,did u see Did you see the LB won the bronze medal increase from 0.69 to 0.70 to 0.71 ?It's probably going to be 0.72 by the end of the day.\nI went through this process and it was nice to see the progress I was making, but more often than not I felt helpless，just like:\n\n>So we beat on, boats against the current, borne back ceaselessly into the past.\n\nCheck if leak is used,In most cases it may only be necessary to check\n`df_p_test[] = -df_test[]`\n\nIf he uses the integration model（ensemble),Just need it to present a dataset or a program that generates a csv.\n\nEnsuring the reliability of the gold medal is fairness in my opinion, ensuring the reliability of all participants is fairness to a greater extent.", "2924921": "Hi @jerrylin96! Thank you for running such a great competition 😁 Even as complications arose, you handled them as gracefully as possible by making appropriate decisions and communicating them in a timely fashion. It was a fantastic learning experience and a privilege to contribute to research that ultimately impacts our entire planet.", "2919233": "@jerrylin96 Judging by your main aim (1D->1D model) I guess that using timespace during training (in 1D->1D scenario) is not a problem. Still, it would be nice to get a confirmation. I specifically used timespace data as auxiliary loss during training. It was just a small thing that gave a small bump, and I saw no problem with it (especially as this data was explicitly available on HF low-res dataset and at no point was referred to as something to avoid). Nevertheless, with recent discussions, I started to get worried. See also [this comment](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/519575#2918339), which asks for clarifications regarding this matter and similar. Thank you.", "2919413": "If you would like to be considered for a cash prize, please submit your NON-LEAK based solution. Do not submit a leak based solution and report the score without the leak in your team name.", "2919388": "@jerrylin96 \n\nCan you disambiguate, does winning solution == medalling solution?\n\nIf not can you qualify the approach you will take for bronze and silver participants?", "2919499": "@jerrylin96 \n\nWe can submit two entries. If we submit both a submission with leaks and a submission without leaks, and both are within the prize range, what will happen? (The current 3rd place corresponds to this situation)\n\n1) The leaderboard and medals will not be affected. The prize money will not be awarded to that team and will be passed down.\n2) If any submission used leaks, that team will be completely excluded from the leaderboard, and the rankings will be recalculated.\n3) If any submission used leaks, the score will be calculated with the other submission, and the rankings will be recalculated.", "2923765": "Hi everyone, I am in learning process, Can anyone explain what the \"leak\" discussion going on this competition (if anyone have time)?!! Is it means the output of the test data released(leaked accidentally)(or any cheatcode)?!!\nThanks!!, I am grateful for your time and the explanations/guidance you will provide!!!!!!", "2920210": "hello, is training with high-res data allowed?", "2925017": "" } }