![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2099265%2F2ea73c8d7116dc9d88f1f9526c6b34c2%2FImage%208-21-23%20at%205.17%20PM.jpeg?generation=1692663529867804&alt=media\")
Note that this is substantially similar to my answer here:
\nBroadly, teams near the top of the LB are there for one of two reasons:
\n[1] They are bioinformatics professionals (academic or biotech) who know what they're doing and have excellent scores because they have strong models. These will remain near the top.
\nBased on previous CAFAs, I'd expect 50-100 'professional' teams to end up in the medals. Most of those will have decent public scores too come submission time, even if they post their models only today (the last day). Noted that this is essentially the same point as @tilii7's comment that I think there will be a shakeup, but maybe for half the competitors in top 50. I would assume that many of the quiet guys in top 20 that have been in this competition before will stay as they are or move upward.
\n[2] They are have overfitted the public LB and are heading for an almighty fall when the results come out.
\nMy team has dropped ~200 places on the public LB in the last week or so, at the same time as higher-scoring public codes appeared. My judgement is that there's little new innovation in those codes beyond what has been shared earlier. rather a tweaking of parameters. Given how small a set the public LB is probably based on, fitting multiple parameters to maximise public score seems foolhardy.
\nI'd also expect that any genuinely good model would do reasonably well on the public LB too. I previously wrote that I would be surprised if many teams come 'from nowhere' (outside the medals, ~top 150) into the gold zone, unless they're deliberately sandbagging their scores on the public LB. Given that I think there are 150-200 seriously overfitted models in the top 250 now, I'd probably be less surprised now to see risers from further down. After all, we will be coming from 236th place or worse.
" }, { "id": 2401625, "authorName": "Alexander Chervov", "votes": 1, "postDate": "2023-08-21T17:35:57.353000", "content": "By the way , I guess if you \"merge\" your solution with quick-go taking same params as jn top public notebook.
\nThen, I guess your LB score would jump back on top. I guess many people doing that. I had wrong impression it will not work on private at all. Looking more carefully I guess there's some chance for at least not a complete shake down for that quick-go thing. It is also unexpected for me that small changes they do in params change LB score that much. Any way you may choose different solution for final private judgment.
\n\nMy team has dropped ~200 places on the public LB in the last week or so, at the same time as higher-scoring public codes appeared
\n
The interesting thing here is that there are some identical scores (for instance, 0.58938 or 0.59144) demonstrated by ~10 teams with no corresponding public notebooks.
\n\nThe interesting thing here is that there are some identical scores (for instance, 0.58938 or 0.59144) demonstrated by ~10 teams with no corresponding public notebooks.
\n
This would have been near-impossible if the public LB had a very large number of data points, or if the number of hyperparameters in those public notebooks was large. For 100 or so data points, it is possible that several groups made identical changes to public notebooks that got them identical scores.
" }, { "id": 2401854, "authorName": "Oleksiy Kononenko", "votes": 0, "postDate": "2023-08-21T20:50:52.793000", "content": "Yes, however, those high scoring public notebooks don’t have any hyperparameters to tune, they only combine several submissions into one. The number of combinations is pretty large, so I find it highly unlikely that ~10 teams would end up with identical scores.
\nMay be a submission was shared somewhere as a dataset, or some public notebook got deleted, not sure.
" }, { "id": 2401904, "authorName": "Alexander Chervov", "votes": 2, "postDate": "2023-08-21T21:55:16.213000", "content": "Public notebooks have simple integer hyperparameter.
\nWhich everybody tunes - \"top\" - how many labels allowed for protein
Originally MT put 35,later others used 45.
\nNow they checked more options.
\nSee different versions of the top public notebook.
\nI guess these teams with same score put the other integer values.
\n\nI guess these teams with same score put other integer values.
\n
That was also my thinking, and perhaps they used a slightly different combination of public models.
" }, { "id": 2401917, "authorName": "Alexander Chervov", "votes": 0, "postDate": "2023-08-21T22:04:58.370000", "content": "I think just changing \"top\" otherwise score might change more.
\nBut the newest version of public contains an option to have different \"top\" for different ontologies.
\nSo now we have 3 integers and appropriate choice is less obvious.
\nSo those who have more luck chosen better integers.
PS
\nIn my mind it is overfishing to LB.
Ah, I thought it was the model hyperparameter that was meant by @tilii7. Then, there is also a bonus that could be adjusted. Still, I'm not sure why ten teams would change these in exactly the same way.
\n\nStill, I'm not sure why ten teams would change these in exactly the same way.
\n
A relatively limited number of choices, ~1500 teams, 5 submissions per day. I have no trouble accepting that 10 teams would stumble upon the same combination of parameters.
" }, { "id": 2401946, "authorName": "Alexander Chervov", "votes": 1, "postDate": "2023-08-21T22:57:47.277000", "content": "People always look on what is working well, and try push it further.
\nThat top public changed \"top\" from 42 to 39 and got uplift - compare early version with the recent version.
So it is natural to try 38 37 36
\nNot so many options
\nThus we see several clusters.
PS
\n\"Bonus\" does not seen to be used by any one. Probably it is not working well
@alexandervc I was curious to try it some time ago and even bonus == 0.001 makes things worse.
I used bonus=0.05, but I'm not just kept the top score when add_prediction, I think it can increased the public LB.
Just updated cafa5-competition-statistics notebook and here is the final public LB / medal thresholds plot:
\n
What's the general feeling of a shakeup happening?
" }, { "id": 2400301, "authorName": "Oleksiy Kononenko", "votes": 0, "postDate": "2023-08-21T01:07:30.330000", "content": "I’ve joined very late, so not sure. May be those who have a better feeling of the competition could comment.
" }, { "id": 2400329, "authorName": "", "votes": 0, "postDate": "2023-08-21T02:08:40.037000", "content": "" }, { "id": 2400341, "authorName": "Oleksiy Kononenko", "votes": 1, "postDate": "2023-08-21T02:29:48.003000", "content": "For ICR the shake-up was caused by a significant difference between the public and private test data distributions. Public/private scores discrepancy was major even for those who used their own models. The key was to use minimal feature engineering and very basic classifier parameters, sometimes even just defaults.
" }, { "id": 2400351, "authorName": "Tilii", "votes": 7, "postDate": "2023-08-21T02:50:14.597000", "content": "It is impossible to tell how big of a shakeup will happen, as not even the organizers know the private test at this point. It will depend to some degree on luck. My understanding is that an objectively better model could end up looking worse if the other model better predicts the private set of proteins that acquire new annotations by the end of this year. To a degree, we are all at the mercy of those providing solid annotations.
\nI think there will be a shakeup, but maybe for half the competitors in top 50. I would assume that many of the quiet guys in top 20 that have been in this competition before will stay as they are or move upward.
" }, { "id": 2400117, "authorName": "Michalina Hulak", "votes": 1, "postDate": "2023-08-20T19:33:34.790000", "content": "Nice :) thanks
" } ], "index": { "id": "433184", "title": "CAFA5: last day competition stats", "authorName": "", "commentCount": "20", "votes": "10", "postDate": "2023-08-20 16:49:15.619000" }, "competition": "cafa-5-protein-function-prediction" }