![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F561669%2F1777525344603766131b9c367f0bfdb7%2FStage1_Stage2_Public_Top_LB_Scores.jpg?generation=1573161974251321&alt=media\")
Just wanted to share some clues of the score differences between stage 1 and stage 2 public LB. The purpose is to comfort those who see very high stage 2 scores. So don't be panic if you see your stage 2 scores many times higher than your stage 1.
\n\nThis is not a complete list of the top teams in Stage 1 public LB, just to show ten examples. Some teams may have only submitted for testing and not necessarily the predictions from their best model. I don't have a complete list of the final stage 1 public LB. It's about one week before the deadline (not accurate, but should be similar). Of course, only show those who have already submitted their stage 2 predictions.
\n\n
Just wanted to share some clues of the score differences between stage 1 and stage 2 public LB. The purpose is to comfort those who see very high stage 2 scores. So don't be panic if you see your stage 2 scores many times higher than your stage 1.
\n\nThis is not a complete list of the top teams in Stage 1 public LB, just to show ten examples. Some teams may have only submitted for testing and not necessarily the predictions from their best model. I don't have a complete list of the final stage 1 public LB. It's about one week before the deadline (not accurate, but should be similar). Of course, only show those who have already submitted their stage 2 predictions.
\n\n
@catlet , thanks for sharing. My partially retrained with stage-2 data model scored 0.864 on this meaningless public LB and it has the best CV score. So ignoring the current LB I think is the best option.
", "rawMarkdown": "@catlet , thanks for sharing. My partially retrained with stage-2 data model scored 0.864 on this meaningless public LB and it has the best CV score. So ignoring the current LB I think is the best option.", "votes": 1, "replies": [ { "id": 668630, "postDate": "2019-11-08T16:36:52.237Z", "content": "You are right. Don't worry about the high loss scores. Mine is between 0.5-0.8.
", "rawMarkdown": "You are right. Don't worry about the high loss scores. Mine is between 0.5-0.8." }, { "id": 668653, "postDate": "2019-11-08T17:11:53.973Z", "content": "Yeah @catlet. I just submitted my 3rd model, partially retrained because I use google colab which disconnected during the night. This one scored has the highest validation AUC score of all of my models and got meaningless current LB score of 1.176 which is interesting.
", "rawMarkdown": "Yeah @catlet. I just submitted my 3rd model, partially retrained because I use google colab which disconnected during the night. This one scored has the highest validation AUC score of all of my models and got meaningless current LB score of 1.176 which is interesting." } ] }, { "id": 668253, "postDate": "2019-11-08T07:06:29.387Z", "content": "\n\n\nI don't have a complete list of the final stage 1 public LB.
\n
You can look here:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/115941#latest-666526
\n\nP.S. I don't think public LB scores are meaningful. It seems that 0.5-0.7 is ok too.
", "rawMarkdown": "> I don't have a complete list of the final stage 1 public LB. \n\nYou can look here:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/115941#latest-666526\n\nP.S. I don't think public LB scores are meaningful. It seems that 0.5-0.7 is ok too.\n\n", "votes": 1, "replies": [ { "id": 668460, "postDate": "2019-11-08T13:01:52.193Z", "content": "Hi @sergeyzlobin - correct. Public LB scores are not (and aren't intended to be) meaningful. They're just there to provide minimal debugging feedback.
", "rawMarkdown": "Hi @sergeyzlobin - correct. Public LB scores are not (and aren't intended to be) meaningful. They're just there to provide minimal debugging feedback." }, { "id": 668482, "postDate": "2019-11-08T13:24:07.600Z", "content": "@philculliton Unfortunately, they provide zero debugging feedback. Which bug can we find with it?
", "rawMarkdown": "\n@philculliton Unfortunately, they provide zero debugging feedback. Which bug can we find with it?", "votes": 3 }, { "id": 668508, "postDate": "2019-11-08T14:01:13.473Z", "content": "I know that I have submitted and that my submission was successfully received because I can see a score there - just like in any other stage of any other competition. I don't need to worry about correctly interpreting (and Kaggle does not need to implement) some other mechanism that shows my submission has been successfully received. I think when Phil says minimal, he really does mean minimal.
", "rawMarkdown": "I know that I have submitted and that my submission was successfully received because I can see a score there - just like in any other stage of any other competition. I don't need to worry about correctly interpreting (and Kaggle does not need to implement) some other mechanism that shows my submission has been successfully received. I think when Phil says minimal, he really does mean **minimal**. " }, { "id": 668514, "postDate": "2019-11-08T14:11:58.417Z", "content": "@cherring in that case a simple 'Submission Successful' would have been a better response. If all the information you get is the fact that the submission has all the ID's it should have and nothing but the IDs + all labels are in the range 0-1 (currently this is the only debugging you get). A simple, OK would have been better then a number that doesn't give any indication.
@cherring you can be sure that your submitted file was parsed without errors\nedit: oh i see it was already answered, nvm.
", "rawMarkdown": "@cherring you can be sure that your submitted file was parsed without errors\nedit: oh i see it was already answered, nvm.", "votes": 1 }, { "id": 668522, "postDate": "2019-11-08T14:21:01.770Z", "content": "Yeah you are not wrong, the Success that you get when you upload is sufficient, for me at least. I feel like lots of people would be confused tho if there was no leader board. People would be unhappy with that. Really no win for Kaggle here.
\n\nThe direct reason is more likely that Kaggle does not currently have a mechanism in which they can have absolutely no samples in the PB. Regardless of it is in their backlog or not - there are other things that I much prefer them to be working on.
", "rawMarkdown": "Yeah you are not wrong, the Success that you get when you upload is sufficient, for me at least. I feel like lots of people would be confused tho if there was no leader board. People would be unhappy with that. Really no win for Kaggle here. \n\nThe direct reason is more likely that Kaggle does not currently have a mechanism in which they can have absolutely no samples in the PB. Regardless of it is in their backlog or not - there are other things that I much prefer them to be working on." }, { "id": 668526, "postDate": "2019-11-08T14:25:34.300Z", "content": "Screenshots were taken an hour prior to stage 1 end.
", "rawMarkdown": "Screenshots were taken an hour prior to stage 1 end. " }, { "id": 668609, "postDate": "2019-11-08T16:04:41.023Z", "content": "Thanks for the complete list of the stage 1 public LB, there was +-0.02 improvement for some teams during the last week.
", "rawMarkdown": "Thanks for the complete list of the stage 1 public LB, there was +-0.02 improvement for some teams during the last week. " }, { "id": 668654, "postDate": "2019-11-08T17:12:27.660Z", "content": "Thanks for the question @yuval6967 - you can find bugs where your predictions are outside of the expected range. It's not intended to find code bugs - in Stage 2, code changes should be limited to those required to import the new data.
\n\n\n\n\nI think when Phil says minimal, he really does mean minimal.
\n
This is correct! Thanks for all the discussion, everyone.
", "rawMarkdown": "Thanks for the question @yuval6967 - you can find bugs where your predictions are outside of the expected range. It's not intended to find code bugs - in Stage 2, code changes should be limited to those required to import the new data.\n\n> I think when Phil says minimal, he really does mean minimal.\n\nThis is correct! Thanks for all the discussion, everyone." } ] }, { "id": 672303, "postDate": "2019-11-13T18:37:26.017Z", "content": "Thanks for sharing. Here is my thought about stage 2 public leaderboard:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117168#latest-672298
", "rawMarkdown": "Thanks for sharing. Here is my thought about stage 2 public leaderboard:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117168#latest-672298" } ], "comments": [ { "id": 668626, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-11-08T16:31:53.613000", "content": "@catlet , thanks for sharing. My partially retrained with stage-2 data model scored 0.864 on this meaningless public LB and it has the best CV score. So ignoring the current LB I think is the best option.
", "votes": 1, "replies": [ { "id": 668630, "author_name": "catlet", "author_url": "", "post_date": "2019-11-08T16:36:52.237000", "content": "You are right. Don't worry about the high loss scores. Mine is between 0.5-0.8.
", "votes": 0, "replies": [] }, { "id": 668653, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-11-08T17:11:53.973000", "content": "Yeah @catlet. I just submitted my 3rd model, partially retrained because I use google colab which disconnected during the night. This one scored has the highest validation AUC score of all of my models and got meaningless current LB score of 1.176 which is interesting.
", "votes": 0, "replies": [] } ] }, { "id": 668253, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2019-11-08T07:06:29.387000", "content": "\n\n\nI don't have a complete list of the final stage 1 public LB.
\n
You can look here:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/115941#latest-666526
\n\nP.S. I don't think public LB scores are meaningful. It seems that 0.5-0.7 is ok too.
", "votes": 1, "replies": [ { "id": 668460, "author_name": "Phil Culliton", "author_url": "", "post_date": "2019-11-08T13:01:52.193000", "content": "Hi @sergeyzlobin - correct. Public LB scores are not (and aren't intended to be) meaningful. They're just there to provide minimal debugging feedback.
", "votes": 0, "replies": [] }, { "id": 668482, "author_name": "yuval reina", "author_url": "", "post_date": "2019-11-08T13:24:07.600000", "content": "@philculliton Unfortunately, they provide zero debugging feedback. Which bug can we find with it?
", "votes": 3, "replies": [] }, { "id": 668508, "author_name": "cherring", "author_url": "", "post_date": "2019-11-08T14:01:13.473000", "content": "I know that I have submitted and that my submission was successfully received because I can see a score there - just like in any other stage of any other competition. I don't need to worry about correctly interpreting (and Kaggle does not need to implement) some other mechanism that shows my submission has been successfully received. I think when Phil says minimal, he really does mean minimal.
", "votes": 0, "replies": [] }, { "id": 668514, "author_name": "yuval reina", "author_url": "", "post_date": "2019-11-08T14:11:58.417000", "content": "@cherring in that case a simple 'Submission Successful' would have been a better response. If all the information you get is the fact that the submission has all the ID's it should have and nothing but the IDs + all labels are in the range 0-1 (currently this is the only debugging you get). A simple, OK would have been better then a number that doesn't give any indication.
@cherring you can be sure that your submitted file was parsed without errors\nedit: oh i see it was already answered, nvm.
", "votes": 1, "replies": [] }, { "id": 668522, "author_name": "cherring", "author_url": "", "post_date": "2019-11-08T14:21:01.770000", "content": "Yeah you are not wrong, the Success that you get when you upload is sufficient, for me at least. I feel like lots of people would be confused tho if there was no leader board. People would be unhappy with that. Really no win for Kaggle here.
\n\nThe direct reason is more likely that Kaggle does not currently have a mechanism in which they can have absolutely no samples in the PB. Regardless of it is in their backlog or not - there are other things that I much prefer them to be working on.
", "votes": 0, "replies": [] }, { "id": 668526, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-11-08T14:25:34.300000", "content": "Screenshots were taken an hour prior to stage 1 end.
", "votes": 0, "replies": [] }, { "id": 668609, "author_name": "catlet", "author_url": "", "post_date": "2019-11-08T16:04:41.023000", "content": "Thanks for the complete list of the stage 1 public LB, there was +-0.02 improvement for some teams during the last week.
", "votes": 0, "replies": [] }, { "id": 668654, "author_name": "Phil Culliton", "author_url": "", "post_date": "2019-11-08T17:12:27.660000", "content": "Thanks for the question @yuval6967 - you can find bugs where your predictions are outside of the expected range. It's not intended to find code bugs - in Stage 2, code changes should be limited to those required to import the new data.
\n\n\n\n\nI think when Phil says minimal, he really does mean minimal.
\n
This is correct! Thanks for all the discussion, everyone.
", "votes": 0, "replies": [] } ] }, { "id": 672303, "author_name": "arctic", "author_url": "", "post_date": "2019-11-13T18:37:26.017000", "content": "Thanks for sharing. Here is my thought about stage 2 public leaderboard:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117168#latest-672298
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "667998": "Just wanted to share some clues of the score differences between stage 1 and stage 2 public LB. The purpose is to comfort those who see very high stage 2 scores. So don't be panic if you see your stage 2 scores many times higher than your stage 1. \n\nThis is not a complete list of the top teams in Stage 1 public LB, just to show ten examples. Some teams may have only submitted for testing and not necessarily the predictions from their best model. I don't have a complete list of the final stage 1 public LB. It's about one week before the deadline (not accurate, but should be similar). Of course, only show those who have already submitted their stage 2 predictions.\n\n\n\n\n\n", "668626": "@catlet , thanks for sharing. My partially retrained with stage-2 data model scored 0.864 on this meaningless public LB and it has the best CV score. So ignoring the current LB I think is the best option.", "668253": "> I don't have a complete list of the final stage 1 public LB. \n\nYou can look here:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/115941#latest-666526\n\nP.S. I don't think public LB scores are meaningful. It seems that 0.5-0.7 is ok too.\n\n", "672303": "Thanks for sharing. Here is my thought about stage 2 public leaderboard:\nhttps://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117168#latest-672298" } }