It seems like 85-90% of the scores on LB are 'baseline' submissions that were done by forking and submitting the public notebook. Owing to some of these submissions, the LB is not the right reflection of the participants.
\nWould (and should) these submissions be removed from consideration to create the final Leaderboard?
EDIT:
\nNot sure why the downvotes, but two points:
In my opinion, it is true that some people are trying to get a good score by forking public notebooks, but just forking public notebooks does not allow us to get a medal.
", "rawMarkdown": "In my opinion, it is true that some people are trying to get a good score by forking public notebooks, but just forking public notebooks does not allow us to get a medal.", "votes": 4 }, { "id": 1061622, "postDate": "2020-10-27T06:36:40.777Z", "content": "Baselines are shared almost in every competition and they are great for building on top of. They are also very useful for others to learn and even change their perspective. If you can get a little edge by putting some effort over these great resources you can even win a medal. In fact, 3D CNN baseline gave me a lot of ideas and helped me not to give up on this competition.
\nOn the other hand, lot of people might choose not to do the extra work to build something on top of it but instead submitting it as it is. This would only be unfortunate for them and wouldn’t affect anyone who is willing to put the extra work. However, if someone shares a very high scoring baseline by the very end of competition then that would be considered as a bad intention since it wouldn’t give enough time for others to digest it and learn from it, it would only help people who fork and submit. That’s why Kaggle warns competitors not to share high scoring kernels by the end.
\nAlso people only get removed from the leaderboard if there is a violation such as cheating. You may see 2 competitors from top 10 public LB were removed.
", "rawMarkdown": "Baselines are shared almost in every competition and they are great for building on top of. They are also very useful for others to learn and even change their perspective. If you can get a little edge by putting some effort over these great resources you can even win a medal. In fact, 3D CNN baseline gave me a lot of ideas and helped me not to give up on this competition.\n\nOn the other hand, lot of people might choose not to do the extra work to build something on top of it but instead submitting it as it is. This would only be unfortunate for them and wouldn’t affect anyone who is willing to put the extra work. However, if someone shares a very high scoring baseline by the very end of competition then that would be considered as a bad intention since it wouldn’t give enough time for others to digest it and learn from it, it would only help people who fork and submit. That’s why Kaggle warns competitors not to share high scoring kernels by the end.\n\nAlso people only get removed from the leaderboard if there is a violation such as cheating. You may see 2 competitors from top 10 public LB were removed.", "votes": 2 }, { "id": 1061526, "postDate": "2020-10-27T04:07:43.337Z", "content": "It seems like 85-90% of the scores on LB are 'baseline' submissions that were done by forking and submitting the public notebook. Owing to some of these submissions, the LB is not the right reflection of the participants.
\nWould (and should) these submissions be removed from consideration to create the final Leaderboard?
EDIT:
\nNot sure why the downvotes, but two points:
I love your point @keremt, but I still feel the forked and submit baseline notebooks without any edits would deflate the scores of those who wish to take an original approach (i.e. different programming framework or model architecture).
\nI feel it would be nice if you could only submit a baseline notebook if you have improved upon its score, not just submitting as-is, which can't be said to be a learning experience.
", "rawMarkdown": "I love your point @keremt, but I still feel the forked and submit baseline notebooks without any edits would deflate the scores of those who wish to take an original approach (i.e. different programming framework or model architecture). \n\nI feel it would be nice if you could only submit a baseline notebook if you have improved upon its score, not just submitting as-is, which can't be said to be a learning experience.", "votes": -1, "replies": [ { "id": 1063396, "postDate": "2020-10-28T20:56:45.907Z", "content": "no real way to differentiate a \"forked\" notebook from a new notebook. Easy enough to copy the notebook and change enough cosmetically, but without changing the actual algorithm. Easy enough to add some random small changes to the submission.csv so it doesn't match exactly. Would be very difficult to police.
\nFor good or bad, this is part of the competition. Fortunately, in many competitions it just leads to over-fitting the public leaderboard which don't do well on the private leaderboard.
", "rawMarkdown": "no real way to differentiate a \"forked\" notebook from a new notebook. Easy enough to copy the notebook and change enough cosmetically, but without changing the actual algorithm. Easy enough to add some random small changes to the submission.csv so it doesn't match exactly. Would be very difficult to police.\n\nFor good or bad, this is part of the competition. Fortunately, in many competitions it just leads to over-fitting the public leaderboard which don't do well on the private leaderboard.", "votes": 3 }, { "id": 1063424, "postDate": "2020-10-28T22:29:38.720Z", "content": "@adamdmehdi I appreciate your view about taking an original approach and I think it's valuable to try different things. But still you should keep in mind that this is a competition and as long as you are following the rules anything is fair game. On the other, if you can come up with a novel approach that may very well qualify as a paper and that would still make me happy even though I don't finish at top.
", "rawMarkdown": "@adamdmehdi I appreciate your view about taking an original approach and I think it's valuable to try different things. But still you should keep in mind that this is a competition and as long as you are following the rules anything is fair game. On the other, if you can come up with a novel approach that may very well qualify as a paper and that would still make me happy even though I don't finish at top." } ] }, { "id": 1063166, "postDate": "2020-10-28T15:21:54.153Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1061639, "author_name": "Kaito", "author_url": "", "post_date": "2020-10-27T06:49:37.783000", "content": "In my opinion, it is true that some people are trying to get a good score by forking public notebooks, but just forking public notebooks does not allow us to get a medal.
", "votes": 4, "replies": [] }, { "id": 1061622, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-10-27T06:36:40.777000", "content": "Baselines are shared almost in every competition and they are great for building on top of. They are also very useful for others to learn and even change their perspective. If you can get a little edge by putting some effort over these great resources you can even win a medal. In fact, 3D CNN baseline gave me a lot of ideas and helped me not to give up on this competition.
\nOn the other hand, lot of people might choose not to do the extra work to build something on top of it but instead submitting it as it is. This would only be unfortunate for them and wouldn’t affect anyone who is willing to put the extra work. However, if someone shares a very high scoring baseline by the very end of competition then that would be considered as a bad intention since it wouldn’t give enough time for others to digest it and learn from it, it would only help people who fork and submit. That’s why Kaggle warns competitors not to share high scoring kernels by the end.
\nAlso people only get removed from the leaderboard if there is a violation such as cheating. You may see 2 competitors from top 10 public LB were removed.
", "votes": 2, "replies": [] }, { "id": 1063038, "author_name": "Adam Mehdi", "author_url": "", "post_date": "2020-10-28T12:59:09.343000", "content": "I love your point @keremt, but I still feel the forked and submit baseline notebooks without any edits would deflate the scores of those who wish to take an original approach (i.e. different programming framework or model architecture).
\nI feel it would be nice if you could only submit a baseline notebook if you have improved upon its score, not just submitting as-is, which can't be said to be a learning experience.
", "votes": -1, "replies": [ { "id": 1063396, "author_name": "quadcore/Richard Epstein", "author_url": "", "post_date": "2020-10-28T20:56:45.907000", "content": "no real way to differentiate a \"forked\" notebook from a new notebook. Easy enough to copy the notebook and change enough cosmetically, but without changing the actual algorithm. Easy enough to add some random small changes to the submission.csv so it doesn't match exactly. Would be very difficult to police.
\nFor good or bad, this is part of the competition. Fortunately, in many competitions it just leads to over-fitting the public leaderboard which don't do well on the private leaderboard.
", "votes": 3, "replies": [] }, { "id": 1063424, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-10-28T22:29:38.720000", "content": "@adamdmehdi I appreciate your view about taking an original approach and I think it's valuable to try different things. But still you should keep in mind that this is a competition and as long as you are following the rules anything is fair game. On the other, if you can come up with a novel approach that may very well qualify as a paper and that would still make me happy even though I don't finish at top.
", "votes": 0, "replies": [] } ] }, { "id": 1063166, "author_name": "", "author_url": "", "post_date": "2020-10-28T15:21:54.153000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1061639": "In my opinion, it is true that some people are trying to get a good score by forking public notebooks, but just forking public notebooks does not allow us to get a medal.", "1061622": "Baselines are shared almost in every competition and they are great for building on top of. They are also very useful for others to learn and even change their perspective. If you can get a little edge by putting some effort over these great resources you can even win a medal. In fact, 3D CNN baseline gave me a lot of ideas and helped me not to give up on this competition.\n\nOn the other hand, lot of people might choose not to do the extra work to build something on top of it but instead submitting it as it is. This would only be unfortunate for them and wouldn’t affect anyone who is willing to put the extra work. However, if someone shares a very high scoring baseline by the very end of competition then that would be considered as a bad intention since it wouldn’t give enough time for others to digest it and learn from it, it would only help people who fork and submit. That’s why Kaggle warns competitors not to share high scoring kernels by the end.\n\nAlso people only get removed from the leaderboard if there is a violation such as cheating. You may see 2 competitors from top 10 public LB were removed.", "1061526": "It seems like 85-90% of the scores on LB are 'baseline' submissions that were done by forking and submitting the public notebook. Owing to some of these submissions, the LB is not the right reflection of the participants.\nWould (and should) these submissions be removed from consideration to create the final Leaderboard?\n\nEDIT: \nNot sure why the downvotes, but two points:\n- I may have an old Kaggle account, but this is my first competition after 5-6 years? So, it is a genuine question since I don't know what Kaggle Admin does after the competition ends\n- Even I have a 'forked and submitted' score on LB (that got selected by default), but that doesn't take away my objectivity of what happened in this competition", "1063038": "I love your point @keremt, but I still feel the forked and submit baseline notebooks without any edits would deflate the scores of those who wish to take an original approach (i.e. different programming framework or model architecture). \n\nI feel it would be nice if you could only submit a baseline notebook if you have improved upon its score, not just submitting as-is, which can't be said to be a learning experience.", "1063166": "" } }