Zhang San has a result of 0.7930 from a non-leaked source and a result of 0.8000 from a leaked source. By using a 9:1 fusion, he achieved a score of 0.794, which just happened to secure him the first place. However, when reproducing the results, Zhang San only submitted the non-leaked code with a result of 0.7930. Even if the result is reproduced, it can be considered that this 0.001 difference is due to fluctuations. Are there any methods to sanction Zhang San?
", "messages": [ { "id": 2918218, "postDate": "2024-07-12T04:57:54.837Z", "content": "Zhang San has a result of 0.7930 from a non-leaked source and a result of 0.8000 from a leaked source. By using a 9:1 fusion, he achieved a score of 0.794, which just happened to secure him the first place. However, when reproducing the results, Zhang San only submitted the non-leaked code with a result of 0.7930. Even if the result is reproduced, it can be considered that this 0.001 difference is due to fluctuations. Are there any methods to sanction Zhang San?
", "rawMarkdown": "Zhang San has a result of 0.7930 from a non-leaked source and a result of 0.8000 from a leaked source. By using a 9:1 fusion, he achieved a score of 0.794, which just happened to secure him the first place. However, when reproducing the results, Zhang San only submitted the non-leaked code with a result of 0.7930. Even if the result is reproduced, it can be considered that this 0.001 difference is due to fluctuations. Are there any methods to sanction Zhang San?", "votes": 12 }, { "id": 2918652, "postDate": "2024-07-12T12:03:39.843Z", "content": "All the top teams mostly masters and grandmasters are well respected members of the community, and I believe almost certainly no one will do it. Lets play this in good faith
", "rawMarkdown": "All the top teams mostly masters and grandmasters are well respected members of the community, and I believe almost certainly no one will do it. Lets play this in good faith", "votes": 7 }, { "id": 2918265, "postDate": "2024-07-12T05:48:23.173Z", "content": "Yes. It is almost impossible to have 100% perfect solution in this case. Every decision comes with some drawbacks. At least, I believe most kagglers in top rank would not use leak after the host shows his attitude.
\nBut I am also not suprised if \"Zhang San\" did like this. I cannot figure out perfect ways to avoid those cases. I can only feel shamed on them. Maybe host can rerun all codes and use reproduced score to rank, but it's also not a perfect solution. It needs lots of resources and also if the reproduced result is beyond some expectatitions, it must cause lots of further argues. Maybe the host would do this way, I don't know.
\nIf you suggest we need to allow leaks and have an extension, but it's not what host wants. They paid money to host this competition for good solutions and they try their best to make thing in the right way. I think it's good enough.
\nSo, IMHO the current decision might be a really good choice. If you have any good suggestion, feel free to say it.
\nIf my team drops rank because of some cheated team who is good at hidding leaks, that's it and I could only accept it. This is life and life is not totally fair. Just a competition game, not a big deal.
", "rawMarkdown": "Yes. It is almost impossible to have 100% perfect solution in this case. Every decision comes with some drawbacks. At least, I believe most kagglers in top rank would not use leak after the host shows his attitude. \n\nBut I am also not suprised if \"Zhang San\" did like this. I cannot figure out perfect ways to avoid those cases. I can only feel shamed on them. Maybe host can rerun all codes and use reproduced score to rank, but it's also not a perfect solution. It needs lots of resources and also if the reproduced result is beyond some expectatitions, it must cause lots of further argues. Maybe the host would do this way, I don't know.\n\nIf you suggest we need to allow leaks and have an extension, but it's not what host wants. They paid money to host this competition for good solutions and they try their best to make thing in the right way. I think it's good enough.\n\nSo, IMHO the current decision might be a really good choice. If you have any good suggestion, feel free to say it.\n\nIf my team drops rank because of some cheated team who is good at hidding leaks, that's it and I could only accept it. This is life and life is not totally fair. Just a competition game, not a big deal. ", "votes": 7 }, { "id": 2932473, "postDate": "2024-07-23T02:20:55.533Z", "content": "If we get a lower score from fluctuations, we can ask Zhang San to send us multiple saved models (perhaps saved at different checkpoints to prevent training from scratch) and perform a one sample t test.
", "rawMarkdown": "If we get a lower score from fluctuations, we can ask Zhang San to send us multiple saved models (perhaps saved at different checkpoints to prevent training from scratch) and perform a one sample t test. ", "votes": 3 }, { "id": 2923651, "postDate": "2024-07-16T03:48:01.280Z", "content": "Tom, 張三 :zhang-san
\nDick, 李四
\nHarry, 王五
https://chinese.yabla.com/chinese-english-pinyin-dictionary.php?define=Zhang+San
\nname for an unspecified person, second of a series of three
but you can use generative Ai to generate their facial images today
\nhttps://zh.wikipedia.org/zh-cn/%E7%84%A1%E5%90%8D%E6%B0%8F
\nchina: 刘一、陈二、张三、李四、王五、赵六 、孙七、周八、吴九、郑十
\nus: Tom, Dick and Harry or Alice, Bob, Charlie, Dave, Eve, Frank, ….
\nitaly: Mario Rossi, Tizio, Caio, Sempronio, Pinco Pallino
\nfrance: Jean Dupont, Paul Martin, Monsieur Durand
\nuk: Tom, Dick and Harry or Jack and Jill
\njapan: 山田太郎（やまだ たろう，Yamada Taro）、山田花子（やまだ はなこ，Yamada Hanako）、名無しの権兵衛（ななしのごんべい Nanashi no Gonbei）、何野某（なんの なにがし，Nanno Nanigashi）、土左卫门（どざえもん，Dozaemon）
Although it may have a negative impact, the best strategy for top teams now is to ensemble with a leaky solution.
\nThe ensembling weight is decided by last day's ranking, but no more than 0.1 is pretty safe.🤣
If some teams haven't used leaked data before, I believe the best strategy is to conserve their energy, organize their reproducible code, and facilitate the organizers' review as much as possible. It's better not to risk exploring the leak.😀
", "rawMarkdown": "If some teams haven't used leaked data before, I believe the best strategy is to conserve their energy, organize their reproducible code, and facilitate the organizers' review as much as possible. It's better not to risk exploring the leak.😀\n", "votes": 6, "replies": [ { "id": 2918327, "postDate": "2024-07-12T06:36:07.523Z", "content": "Yes, this's what top teams will do. I do not worry about this comp that all top teams have good reputation.
\nAnd i can understand top teams and hosts are running out of patience and want this comp to end in time. But code checking is really a bad start, I just want to point this out. Hope code checking will not happend again~
What \"fluctuations\"? Random seeds should be fixed to get reproducible results.
", "rawMarkdown": "What \"fluctuations\"? Random seeds should be fixed to get reproducible results.", "replies": [ { "id": 2918271, "postDate": "2024-07-12T05:55:03.090Z", "content": "In my experience, even seeds are fixed, some results are also a little fluctuated.
", "rawMarkdown": "In my experience, even seeds are fixed, some results are also a little fluctuated." }, { "id": 2918275, "postDate": "2024-07-12T05:58:19.883Z", "content": "Naaa I don't use fixed seed for training, it's too prone for problems imo. And yet I'm confident that my solution can be reproduced with enough accuracy.
", "rawMarkdown": "Naaa I don't use fixed seed for training, it's too prone for problems imo. And yet I'm confident that my solution can be reproduced with enough accuracy." } ] }, { "id": 2918231, "postDate": "2024-07-12T05:20:37.477Z", "content": "I don't see a \"Zhang San\" in first place. What are you referring to exactly?
", "rawMarkdown": "I don't see a \"Zhang San\" in first place. What are you referring to exactly?", "replies": [ { "id": 2918238, "postDate": "2024-07-12T05:28:30.533Z", "content": "Zhang San is not real one, it means someone.
", "rawMarkdown": "Zhang San is not real one, it means someone.", "votes": 3 }, { "id": 2918239, "postDate": "2024-07-12T05:28:42.800Z", "content": "\"Zhang San\" is usually a fictional name in Chinese, equivalent to \"John Doe\" or \"Jane Smith\" in English. It is often used to represent an ordinary person or as a character's name in examples and stories.
", "rawMarkdown": "\"Zhang San\" is usually a fictional name in Chinese, equivalent to \"John Doe\" or \"Jane Smith\" in English. It is often used to represent an ordinary person or as a character's name in examples and stories. ", "votes": 5 } ] }, { "id": 2918229, "postDate": "2024-07-12T05:17:07.977Z", "content": "yes indeed a concern if people don't select leaky models in the final submission, or some slight weighted blend. Right that people can claimthat random fluctuations cannot be explained
", "rawMarkdown": "yes indeed a concern if people don't select leaky models in the final submission, or some slight weighted blend. Right that people can claimthat random fluctuations cannot be explained" }, { "id": 2918286, "postDate": "2024-07-12T06:07:25.503Z", "rawMarkdown": "", "votes": -2, "isDeleted": true, "replies": [ { "id": 2918296, "postDate": "2024-07-12T06:16:06.680Z", "rawMarkdown": "", "votes": -2, "isDeleted": true, "replies": [ { "id": 2918305, "postDate": "2024-07-12T06:21:53.017Z", "content": "Just don't announce this threshold beforehand, least 'Zhang San' micro-manages to stay within the threshold.
\nI think after the host reproduce top 10, he will have a pretty good idea of flactuation. So if a team is way above most other flactuation, it's a good reason to re-run it, and if he stay way above…well…this is a pretty strong indicator for something fishy.
All the top teams mostly masters and grandmasters are well respected members of the community, and I believe almost certainly no one will do it. Lets play this in good faith
", "votes": 7, "replies": [] }, { "id": 2918265, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-12T05:48:23.173000", "content": "Yes. It is almost impossible to have 100% perfect solution in this case. Every decision comes with some drawbacks. At least, I believe most kagglers in top rank would not use leak after the host shows his attitude.
\nBut I am also not suprised if \"Zhang San\" did like this. I cannot figure out perfect ways to avoid those cases. I can only feel shamed on them. Maybe host can rerun all codes and use reproduced score to rank, but it's also not a perfect solution. It needs lots of resources and also if the reproduced result is beyond some expectatitions, it must cause lots of further argues. Maybe the host would do this way, I don't know.
\nIf you suggest we need to allow leaks and have an extension, but it's not what host wants. They paid money to host this competition for good solutions and they try their best to make thing in the right way. I think it's good enough.
\nSo, IMHO the current decision might be a really good choice. If you have any good suggestion, feel free to say it.
\nIf my team drops rank because of some cheated team who is good at hidding leaks, that's it and I could only accept it. This is life and life is not totally fair. Just a competition game, not a big deal.
", "votes": 7, "replies": [] }, { "id": 2932473, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-23T02:20:55.533000", "content": "If we get a lower score from fluctuations, we can ask Zhang San to send us multiple saved models (perhaps saved at different checkpoints to prevent training from scratch) and perform a one sample t test.
", "votes": 3, "replies": [] }, { "id": 2923651, "author_name": "hengck23", "author_url": "", "post_date": "2024-07-16T03:48:01.280000", "content": "Tom, 張三 :zhang-san
\nDick, 李四
\nHarry, 王五
https://chinese.yabla.com/chinese-english-pinyin-dictionary.php?define=Zhang+San
\nname for an unspecified person, second of a series of three
but you can use generative Ai to generate their facial images today
\nhttps://zh.wikipedia.org/zh-cn/%E7%84%A1%E5%90%8D%E6%B0%8F
\nchina: 刘一、陈二、张三、李四、王五、赵六 、孙七、周八、吴九、郑十
\nus: Tom, Dick and Harry or Alice, Bob, Charlie, Dave, Eve, Frank, ….
\nitaly: Mario Rossi, Tizio, Caio, Sempronio, Pinco Pallino
\nfrance: Jean Dupont, Paul Martin, Monsieur Durand
\nuk: Tom, Dick and Harry or Jack and Jill
\njapan: 山田太郎（やまだ たろう，Yamada Taro）、山田花子（やまだ はなこ，Yamada Hanako）、名無しの権兵衛（ななしのごんべい Nanashi no Gonbei）、何野某（なんの なにがし，Nanno Nanigashi）、土左卫门（どざえもん，Dozaemon）
Although it may have a negative impact, the best strategy for top teams now is to ensemble with a leaky solution.
\nThe ensembling weight is decided by last day's ranking, but no more than 0.1 is pretty safe.🤣
If some teams haven't used leaked data before, I believe the best strategy is to conserve their energy, organize their reproducible code, and facilitate the organizers' review as much as possible. It's better not to risk exploring the leak.😀
", "votes": 6, "replies": [ { "id": 2918327, "author_name": "hyd", "author_url": "", "post_date": "2024-07-12T06:36:07.523000", "content": "Yes, this's what top teams will do. I do not worry about this comp that all top teams have good reputation.
\nAnd i can understand top teams and hosts are running out of patience and want this comp to end in time. But code checking is really a bad start, I just want to point this out. Hope code checking will not happend again~
What \"fluctuations\"? Random seeds should be fixed to get reproducible results.
", "votes": 0, "replies": [ { "id": 2918271, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-12T05:55:03.090000", "content": "In my experience, even seeds are fixed, some results are also a little fluctuated.
", "votes": 0, "replies": [] }, { "id": 2918275, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-12T05:58:19.883000", "content": "Naaa I don't use fixed seed for training, it's too prone for problems imo. And yet I'm confident that my solution can be reproduced with enough accuracy.
", "votes": 0, "replies": [] } ] }, { "id": 2918231, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-07-12T05:20:37.477000", "content": "I don't see a \"Zhang San\" in first place. What are you referring to exactly?
", "votes": 0, "replies": [ { "id": 2918238, "author_name": "ADAM.", "author_url": "", "post_date": "2024-07-12T05:28:30.533000", "content": "Zhang San is not real one, it means someone.
", "votes": 3, "replies": [] }, { "id": 2918239, "author_name": "Rib~", "author_url": "", "post_date": "2024-07-12T05:28:42.800000", "content": "\"Zhang San\" is usually a fictional name in Chinese, equivalent to \"John Doe\" or \"Jane Smith\" in English. It is often used to represent an ordinary person or as a character's name in examples and stories.
", "votes": 5, "replies": [] } ] }, { "id": 2918229, "author_name": "NikhilMishra", "author_url": "", "post_date": "2024-07-12T05:17:07.977000", "content": "yes indeed a concern if people don't select leaky models in the final submission, or some slight weighted blend. Right that people can claimthat random fluctuations cannot be explained
", "votes": 0, "replies": [] }, { "id": 2918286, "author_name": "", "author_url": "", "post_date": "2024-07-12T06:07:25.503000", "content": "", "votes": -2, "replies": [ { "id": 2918296, "author_name": "", "author_url": "", "post_date": "2024-07-12T06:16:06.680000", "content": "", "votes": -2, "replies": [ { "id": 2918305, "author_name": "greySnow", "author_url": "", "post_date": "2024-07-12T06:21:53.017000", "content": "Just don't announce this threshold beforehand, least 'Zhang San' micro-manages to stay within the threshold.
\nI think after the host reproduce top 10, he will have a pretty good idea of flactuation. So if a team is way above most other flactuation, it's a good reason to re-run it, and if he stay way above…well…this is a pretty strong indicator for something fishy.