![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4616875%2Fa210ca77b5b7424c94e4fdb26869ce87%2FScreenshot%202026-04-01%20at%201.53.56PM.png?generation=1775066074720419&alt=media\")
Hi guys,
\nHope the community still remembers that before competition closes, Team d4t4 was ranked among the top 10 on public LB with a score 0.460, now on the Public LB it shows they ranked 176th.
and if you do the math here you will find out their private ranking is computed relative to this new Public score, not the original one before competition closed.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4616875%2F94939e6a27fa4916cf77aa9a1b0df88a%2FScreenshot%202026-04-01%20at%201.54.58PM.png?generation=1775066159108921&alt=media\")
There are many other cases like this so you should check that against the evidence you have. Thus, it can be inferred that the rerun took place twice, once to rerank the public LB, and once to rank the private LB. However, there is no where in the overview page of communication from the host that said it will be run twice.
\nI understand that the private rerun should take place once. But why rerun on the public test as well? Is this a fair competition? What do you think?
", "messages": [ { "id": 3433444, "postDate": "2026-04-01T18:00:23.643Z", "content": "Hi guys,
\nHope the community still remembers that before competition closes, Team d4t4 was ranked among the top 10 on public LB with a score 0.460, now on the Public LB it shows they ranked 176th.
and if you do the math here you will find out their private ranking is computed relative to this new Public score, not the original one before competition closed.
There are many other cases like this so you should check that against the evidence you have. Thus, it can be inferred that the rerun took place twice, once to rerank the public LB, and once to rank the private LB. However, there is no where in the overview page of communication from the host that said it will be run twice.
\nI understand that the private rerun should take place once. But why rerun on the public test as well? Is this a fair competition? What do you think?
", "rawMarkdown": "Hi guys,\n\nHope the community still remembers that before competition closes, Team d4t4 was ranked among the top 10 on public LB with a score 0.460, now on the Public LB it shows they ranked 176th. \n\nand if you do the math here you will find out their private ranking is computed relative to this new Public score, not the original one before competition closed.\n\nThere are many other cases like this so you should check that against the evidence you have. Thus, it can be inferred that the rerun took place twice, once to rerank the public LB, and once to rank the private LB. However, there is no where in the overview page of communication from the host that said it will be run twice. \n\nI understand that the private rerun should take place once. But why rerun on the public test as well? Is this a fair competition? What do you think?", "votes": 6 }, { "id": 3433587, "postDate": "2026-04-01T20:53:00.723Z", "content": "@inversion My models were deterministic and I ran the same model thrice during the competition which yielded the same score 0.460 , after rerun suddenly my public lb is down and private too , the same approach in last Stanford rna competition yielded in 9 th rank bit frustrated
", "rawMarkdown": "@inversion My models were deterministic and I ran the same model thrice during the competition which yielded the same score 0.460 , after rerun suddenly my public lb is down and private too , the same approach in last Stanford rna competition yielded in 9 th rank bit frustrated ", "votes": 1, "replies": [ { "id": 3433688, "postDate": "2026-04-02T00:17:59.493Z", "content": "Thanks for bringing this up.
\nThe targets in Public LB did not change. But their order within the test set may have changed.
\nWould that explain the fluctuation in score? (You can test by running your notebook with the validation_sequences.csv and a permutation in order.)
@rhijudas, I tested the same model local server with validation_sequences.csv and computed metrics using the competition metrics, 0.501 was my score
", "rawMarkdown": "@rhijudas, I tested the same model local server with validation_sequences.csv and computed metrics using the competition metrics, 0.501 was my score " }, { "id": 3434023, "postDate": "2026-04-02T13:35:32.363Z", "content": "Thanks! Can you now prepare another validation set that permutes the order of the 28 validation sequences, and run your notebook again?
", "rawMarkdown": "Thanks! Can you now prepare another validation set that permutes the order of the 28 validation sequences, and run your notebook again?" }, { "id": 3434125, "postDate": "2026-04-02T16:23:14.720Z", "content": "sure i will do that
", "rawMarkdown": "sure i will do that " }, { "id": 3435270, "postDate": "2026-04-03T22:22:38.583Z", "content": "Thanks for sharing @arunodhayan, did you run your model again? I am curious to know the result too. Thanks
", "rawMarkdown": "Thanks for sharing @arunodhayan, did you run your model again? I am curious to know the result too. Thanks" } ] } ] }, { "id": 3433469, "postDate": "2026-04-01T18:21:30.637Z", "content": "Let me clarify . . . there are at least two types of reruns that are possible. The first is a standard Code competition rerun, where we copy your notebook and rerun it on a hidden Test dataset at submission time. In this case, the Public and Private scores are fixed after scoring, and the Private leaderboard is released at the end of the competition.
\nA second type occurs when the hidden Test dataset is updated, as in the case of this competition. When this happens, we rerun selected submission notebooks. This reruns for the entire dataset, both Public and Private. When the Kaggle backend reruns and scores a submission, there is no way to de-couple the Private from the Public observations; both get updated with a rerun. Even when the Public observations are exactly the same, models that exhibit non-determinism may have different scores.
", "rawMarkdown": "Let me clarify . . . there are at least two types of reruns that are possible. The first is a standard Code competition rerun, where we copy your notebook and rerun it on a hidden Test dataset at submission time. In this case, the Public and Private scores are fixed after scoring, and the Private leaderboard is released at the end of the competition.\n\nA second type occurs when the hidden Test dataset is updated, as in the case of this competition. When this happens, we rerun selected submission notebooks. This reruns for the entire dataset, both Public and Private. When the Kaggle backend reruns and scores a submission, there is no way to de-couple the Private from the Public observations; both get updated with a rerun. Even when the Public observations are exactly the same, models that exhibit non-determinism may have different scores.", "replies": [ { "id": 3433473, "postDate": "2026-04-01T18:25:50.340Z", "content": "Thanks for the reply. Can you elaborate on why the private test set for this competition is updated? The host says in another post that the first test is too \"easy.\" Is this the reason why you updated the test set?
", "rawMarkdown": "Thanks for the reply. Can you elaborate on why the private test set for this competition is updated? The host says in another post that the first test is too \"easy.\" Is this the reason why you updated the test set?", "votes": 1, "replies": [ { "id": 3433492, "postDate": "2026-04-01T18:38:54.340Z", "content": "The final private test set reflects the structures solved by an incredible group of cryo-electron microscopists, in parallel with the competition.
\nThis final private test set was not curated or filtered based on Kaggle notebook performance.
\nThe final private turned out to be on average slightly harder than the private test set that we had available during the competition. But this was not known to the microscopists or to the hosts until we got the results that you see now on the Private Leaderboard.
", "rawMarkdown": "The final private test set reflects the structures solved by an incredible group of cryo-electron microscopists, in parallel with the competition. \n\nThis final private test set was not curated or filtered based on Kaggle notebook performance. \n\nThe final private turned out to be on average slightly harder than the private test set that we had available during the competition. But this was not known to the microscopists or to the hosts until we got the results that you see now on the Private Leaderboard." }, { "id": 3433499, "postDate": "2026-04-01T18:53:25.397Z", "content": "I trust your integrity and appreciate your reply. But you could communicate this more clearly yesterday when the outcome is announced rather than ignoring every comment and leaving the community in giant confusion. It is not normal to see public LB cleared and have two private scores.
", "rawMarkdown": "I trust your integrity and appreciate your reply. But you could communicate this more clearly yesterday when the outcome is announced rather than ignoring every comment and leaving the community in giant confusion. It is not normal to see public LB cleared and have two private scores. ", "votes": 2 }, { "id": 3433505, "postDate": "2026-04-01T19:01:57.293Z", "content": "Thank you for this feedback. We could have communicated much better, and will ensure future competitions are clear in this regard.
", "rawMarkdown": "Thank you for this feedback. We could have communicated much better, and will ensure future competitions are clear in this regard." }, { "id": 3433553, "postDate": "2026-04-01T20:01:23.957Z", "content": "@inversion, yes you should have and the RNA team definitely should have and still can. They ignored every comment that was asking uncomfortable questions.
\nYou, me and many others that have been on this platform for years know that transparency is super important even in difficult situations. I have seen Kaggle navigate those very well in the past. This one is unique in the lack of transparency imho.
\nFor the record, my models are deterministic i.e. for the same settings gives the same result every time.
", "rawMarkdown": "@inversion, yes you should have and the RNA team definitely should have and still can. They ignored every comment that was asking uncomfortable questions. \n\nYou, me and many others that have been on this platform for years know that transparency is super important even in difficult situations. I have seen Kaggle navigate those very well in the past. This one is unique in the lack of transparency imho.\n\nFor the record, my models are deterministic i.e. for the same settings gives the same result every time.", "votes": 2 } ] } ] }, { "id": 3434299, "postDate": "2026-04-02T19:37:15.487Z", "content": "Congratulations on securing the 9th position!
", "rawMarkdown": "Congratulations on securing the 9th position!" } ], "comments": [ { "id": 3433587, "author_name": "Arunodhayan", "author_url": "", "post_date": "2026-04-01T20:53:00.723000", "content": "@inversion My models were deterministic and I ran the same model thrice during the competition which yielded the same score 0.460 , after rerun suddenly my public lb is down and private too , the same approach in last Stanford rna competition yielded in 9 th rank bit frustrated
", "votes": 1, "replies": [ { "id": 3433688, "author_name": "Rhiju Das", "author_url": "", "post_date": "2026-04-02T00:17:59.493000", "content": "Thanks for bringing this up.
\nThe targets in Public LB did not change. But their order within the test set may have changed.
\nWould that explain the fluctuation in score? (You can test by running your notebook with the validation_sequences.csv and a permutation in order.)
@rhijudas, I tested the same model local server with validation_sequences.csv and computed metrics using the competition metrics, 0.501 was my score
", "votes": 0, "replies": [] }, { "id": 3434023, "author_name": "Rhiju Das", "author_url": "", "post_date": "2026-04-02T13:35:32.363000", "content": "Thanks! Can you now prepare another validation set that permutes the order of the 28 validation sequences, and run your notebook again?
", "votes": 0, "replies": [] }, { "id": 3434125, "author_name": "Arunodhayan", "author_url": "", "post_date": "2026-04-02T16:23:14.720000", "content": "sure i will do that
", "votes": 0, "replies": [] }, { "id": 3435270, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2026-04-03T22:22:38.583000", "content": "Thanks for sharing @arunodhayan, did you run your model again? I am curious to know the result too. Thanks
", "votes": 0, "replies": [] } ] } ] }, { "id": 3433469, "author_name": "inversion", "author_url": "", "post_date": "2026-04-01T18:21:30.637000", "content": "Let me clarify . . . there are at least two types of reruns that are possible. The first is a standard Code competition rerun, where we copy your notebook and rerun it on a hidden Test dataset at submission time. In this case, the Public and Private scores are fixed after scoring, and the Private leaderboard is released at the end of the competition.
\nA second type occurs when the hidden Test dataset is updated, as in the case of this competition. When this happens, we rerun selected submission notebooks. This reruns for the entire dataset, both Public and Private. When the Kaggle backend reruns and scores a submission, there is no way to de-couple the Private from the Public observations; both get updated with a rerun. Even when the Public observations are exactly the same, models that exhibit non-determinism may have different scores.
", "votes": 0, "replies": [ { "id": 3433473, "author_name": "hongan", "author_url": "", "post_date": "2026-04-01T18:25:50.340000", "content": "Thanks for the reply. Can you elaborate on why the private test set for this competition is updated? The host says in another post that the first test is too \"easy.\" Is this the reason why you updated the test set?
", "votes": 1, "replies": [ { "id": 3433492, "author_name": "Rhiju Das", "author_url": "", "post_date": "2026-04-01T18:38:54.340000", "content": "The final private test set reflects the structures solved by an incredible group of cryo-electron microscopists, in parallel with the competition.
\nThis final private test set was not curated or filtered based on Kaggle notebook performance.
\nThe final private turned out to be on average slightly harder than the private test set that we had available during the competition. But this was not known to the microscopists or to the hosts until we got the results that you see now on the Private Leaderboard.
", "votes": 0, "replies": [] }, { "id": 3433499, "author_name": "hongan", "author_url": "", "post_date": "2026-04-01T18:53:25.397000", "content": "I trust your integrity and appreciate your reply. But you could communicate this more clearly yesterday when the outcome is announced rather than ignoring every comment and leaving the community in giant confusion. It is not normal to see public LB cleared and have two private scores.
", "votes": 2, "replies": [] }, { "id": 3433505, "author_name": "inversion", "author_url": "", "post_date": "2026-04-01T19:01:57.293000", "content": "Thank you for this feedback. We could have communicated much better, and will ensure future competitions are clear in this regard.
", "votes": 0, "replies": [] }, { "id": 3433553, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2026-04-01T20:01:23.957000", "content": "@inversion, yes you should have and the RNA team definitely should have and still can. They ignored every comment that was asking uncomfortable questions.
\nYou, me and many others that have been on this platform for years know that transparency is super important even in difficult situations. I have seen Kaggle navigate those very well in the past. This one is unique in the lack of transparency imho.
\nFor the record, my models are deterministic i.e. for the same settings gives the same result every time.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3434299, "author_name": "OverfitOracle", "author_url": "", "post_date": "2026-04-02T19:37:15.487000", "content": "Congratulations on securing the 9th position!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3433444": "Hi guys,\n\nHope the community still remembers that before competition closes, Team d4t4 was ranked among the top 10 on public LB with a score 0.460, now on the Public LB it shows they ranked 176th. \n\nand if you do the math here you will find out their private ranking is computed relative to this new Public score, not the original one before competition closed.\n\nThere are many other cases like this so you should check that against the evidence you have. Thus, it can be inferred that the rerun took place twice, once to rerank the public LB, and once to rank the private LB. However, there is no where in the overview page of communication from the host that said it will be run twice. \n\nI understand that the private rerun should take place once. But why rerun on the public test as well? Is this a fair competition? What do you think?", "3433587": "@inversion My models were deterministic and I ran the same model thrice during the competition which yielded the same score 0.460 , after rerun suddenly my public lb is down and private too , the same approach in last Stanford rna competition yielded in 9 th rank bit frustrated ", "3433469": "Let me clarify . . . there are at least two types of reruns that are possible. The first is a standard Code competition rerun, where we copy your notebook and rerun it on a hidden Test dataset at submission time. In this case, the Public and Private scores are fixed after scoring, and the Private leaderboard is released at the end of the competition.\n\nA second type occurs when the hidden Test dataset is updated, as in the case of this competition. When this happens, we rerun selected submission notebooks. This reruns for the entire dataset, both Public and Private. When the Kaggle backend reruns and scores a submission, there is no way to de-couple the Private from the Public observations; both get updated with a rerun. Even when the Public observations are exactly the same, models that exhibit non-determinism may have different scores.", "3434299": "Congratulations on securing the 9th position!" } }