{ "id": 115970, "title": "large LB scores", "url": "/competitions/rsna-intracranial-hemorrhage-detection/discussion/115970", "author_name": "Yee Ng", "post_date": "2019-11-06T09:44:44.920000", "votes": 8, "comment_count": 13, "views": 0, "content": "

What's up with the large LB scores. I know its calculated based on <1% of images but there is almost 10x difference from stage 1 as if the metric has changed.

", "messages": [ { "id": 666629, "postDate": "2019-11-06T09:44:44.920Z", "content": "

What's up with the large LB scores. I know its calculated based on <1% of images but there is almost 10x difference from stage 1 as if the metric has changed.

", "rawMarkdown": "What's up with the large LB scores. I know its calculated based on <1% of images but there is almost 10x difference from stage 1 as if the metric has changed.", "votes": 8 }, { "id": 666863, "postDate": "2019-11-06T15:11:29.300Z", "content": "

0.000 is amazing score🙄

", "rawMarkdown": "0.000 is amazing score🙄", "votes": 1 }, { "id": 666648, "postDate": "2019-11-06T10:16:08.250Z", "content": "

What was your previous score?

\n\n

Unfortunately, I forgot to download/screenshot the previous leaderboard data. Anyone has the data about stage 1 leaderboard?

", "rawMarkdown": "What was your previous score? \n\nUnfortunately, I forgot to download/screenshot the previous leaderboard data. Anyone has the data about stage 1 leaderboard?", "votes": 1 }, { "id": 666897, "postDate": "2019-11-06T15:49:43.547Z", "content": "

@juliaelliott The Leaderboard in stage 2 is meant to help us spot bugs in our submissions. But the current score is calculated on a very small part of the data and the scores are just all over the place, which gives no indication. can it be enlarged a little (5%-10%, so we can get some indication?

", "rawMarkdown": "@juliaelliott The Leaderboard in stage 2 is meant to help us spot bugs in our submissions. But the current score is calculated on a very small part of the data and the scores are just all over the place, which gives no indication. can it be enlarged a little (5%-10%, so we can get some indication?" }, { "id": 666653, "postDate": "2019-11-06T10:20:34.380Z", "content": "

Thats' okay

", "rawMarkdown": "Thats' okay", "votes": -1, "replies": [ { "id": 666805, "postDate": "2019-11-06T14:15:33.470Z", "content": "

@yaroshevskiy Oleg

\n\n

Is an LB score of 16.353 even possible? I thought the max LB you can get is 15.0 since there is a hard limit on the log loss ---- max(min(p,1−10e−15),10e−15)

\n\n

Eg. Worst case scenario. Correct label is 1 and you guess 0.\nLog loss = -(1) log(max(0, 10e-15)) = 15

", "rawMarkdown": "@yaroshevskiy Oleg\n\nIs an LB score of 16.353 even possible? I thought the max LB you can get is 15.0 since there is a hard limit on the log loss ---- max(min(p,1−10e−15),10e−15)\n\nEg. Worst case scenario. Correct label is 1 and you guess 0.\nLog loss = -(1) log(max(0, 10e-15)) = 15 " }, { "id": 666808, "postDate": "2019-11-06T14:17:38.173Z", "content": "

Even 18 is possible :D

", "rawMarkdown": "Even 18 is possible :D", "votes": 1 }, { "id": 666833, "postDate": "2019-11-06T14:46:31.227Z", "content": "

It's base e.

", "rawMarkdown": "It's base e." }, { "id": 666845, "postDate": "2019-11-06T14:56:53.503Z", "content": "

I see... That made sense. Thanks.

", "rawMarkdown": "I see... That made sense. Thanks." } ] }, { "id": 667653, "postDate": "2019-11-07T14:12:32.260Z", "content": "

@anginscribe , hi, it's amazing that you have got 0.000 loss. Could you tell us your loss scores in stage 1?

", "rawMarkdown": "@anginscribe , hi, it's amazing that you have got 0.000 loss. Could you tell us your loss scores in stage 1?" }, { "id": 667320, "postDate": "2019-11-07T05:48:28.703Z", "content": "
\n

Yee Ng wrote:

\n \n

What's up with the large LB scores. I know its calculated based on <1% of images but there is almost 10x difference from stage 1 as if the metric has changed.\n how can you know that it is calculated based on less than 1% of images

\n
", "rawMarkdown": "> **Yee Ng wrote:**\n> \n> What's up with the large LB scores. I know its calculated based on <1% of images but there is almost 10x difference from stage 1 as if the metric has changed.\nhow can you know that it is calculated based on less than 1% of images" }, { "id": 667117, "postDate": "2019-11-06T20:47:33.293Z", "content": "

Probably less \"any\" in the 1% than on stage 1 LB. As often, expect some changes with the Private LB.

", "rawMarkdown": "Probably less \"any\" in the 1% than on stage 1 LB. As often, expect some changes with the Private LB." }, { "id": 667068, "postDate": "2019-11-06T19:26:13.743Z", "content": "

10x+ scores, so unreliable on the <1% test set.

", "rawMarkdown": "10x+ scores, so unreliable on the <1% test set." }, { "id": 666806, "postDate": "2019-11-06T14:16:20.107Z", "content": "

same result, 10x LB scores

", "rawMarkdown": "same result, 10x LB scores " } ], "comments": [ { "id": 666863, "author_name": "takuoko", "author_url": "", "post_date": "2019-11-06T15:11:29.300000", "content": "

0.000 is amazing score🙄

", "votes": 1, "replies": [] }, { "id": 666648, "author_name": "arctic", "author_url": "", "post_date": "2019-11-06T10:16:08.250000", "content": "

What was your previous score?

\n\n

Unfortunately, I forgot to download/screenshot the previous leaderboard data. Anyone has the data about stage 1 leaderboard?

", "votes": 1, "replies": [] }, { "id": 666897, "author_name": "yuval reina", "author_url": "", "post_date": "2019-11-06T15:49:43.547000", "content": "

@juliaelliott The Leaderboard in stage 2 is meant to help us spot bugs in our submissions. But the current score is calculated on a very small part of the data and the scores are just all over the place, which gives no indication. can it be enlarged a little (5%-10%, so we can get some indication?

", "votes": 0, "replies": [] }, { "id": 666653, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2019-11-06T10:20:34.380000", "content": "

Thats' okay

", "votes": -1, "replies": [ { "id": 666805, "author_name": "Yee Ng", "author_url": "", "post_date": "2019-11-06T14:15:33.470000", "content": "

@yaroshevskiy Oleg

\n\n

Is an LB score of 16.353 even possible? I thought the max LB you can get is 15.0 since there is a hard limit on the log loss ---- max(min(p,1−10e−15),10e−15)

\n\n

Eg. Worst case scenario. Correct label is 1 and you guess 0.\nLog loss = -(1) log(max(0, 10e-15)) = 15

", "votes": 0, "replies": [] }, { "id": 666808, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2019-11-06T14:17:38.173000", "content": "

Even 18 is possible :D

", "votes": 1, "replies": [] }, { "id": 666833, "author_name": "César Parra Rojas", "author_url": "", "post_date": "2019-11-06T14:46:31.227000", "content": "

It's base e.

", "votes": 0, "replies": [] }, { "id": 666845, "author_name": "Yee Ng", "author_url": "", "post_date": "2019-11-06T14:56:53.503000", "content": "

I see... That made sense. Thanks.

", "votes": 0, "replies": [] } ] }, { "id": 667653, "author_name": "qzw", "author_url": "", "post_date": "2019-11-07T14:12:32.260000", "content": "

@anginscribe , hi, it's amazing that you have got 0.000 loss. Could you tell us your loss scores in stage 1?

", "votes": 0, "replies": [] }, { "id": 667320, "author_name": "hqhz1817", "author_url": "", "post_date": "2019-11-07T05:48:28.703000", "content": "
\n

Yee Ng wrote:

\n \n

What's up with the large LB scores. I know its calculated based on <1% of images but there is almost 10x difference from stage 1 as if the metric has changed.\n how can you know that it is calculated based on less than 1% of images

\n
", "votes": 0, "replies": [] }, { "id": 667117, "author_name": "Cogitae _ Thomas Soumarmon", "author_url": "", "post_date": "2019-11-06T20:47:33.293000", "content": "

Probably less \"any\" in the 1% than on stage 1 LB. As often, expect some changes with the Private LB.

", "votes": 0, "replies": [] }, { "id": 667068, "author_name": "catlet", "author_url": "", "post_date": "2019-11-06T19:26:13.743000", "content": "

10x+ scores, so unreliable on the <1% test set.

", "votes": 0, "replies": [] }, { "id": 666806, "author_name": "liuzhangzhen", "author_url": "", "post_date": "2019-11-06T14:16:20.107000", "content": "

same result, 10x LB scores

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "666629": "What's up with the large LB scores. I know its calculated based on <1% of images but there is almost 10x difference from stage 1 as if the metric has changed.", "666863": "0.000 is amazing score🙄", "666648": "What was your previous score? \n\nUnfortunately, I forgot to download/screenshot the previous leaderboard data. Anyone has the data about stage 1 leaderboard?", "666897": "@juliaelliott The Leaderboard in stage 2 is meant to help us spot bugs in our submissions. But the current score is calculated on a very small part of the data and the scores are just all over the place, which gives no indication. can it be enlarged a little (5%-10%, so we can get some indication?", "666653": "Thats' okay", "667653": "@anginscribe , hi, it's amazing that you have got 0.000 loss. Could you tell us your loss scores in stage 1?", "667320": "> **Yee Ng wrote:**\n> \n> What's up with the large LB scores. I know its calculated based on <1% of images but there is almost 10x difference from stage 1 as if the metric has changed.\nhow can you know that it is calculated based on less than 1% of images", "667117": "Probably less \"any\" in the 1% than on stage 1 LB. As often, expect some changes with the Private LB.", "667068": "10x+ scores, so unreliable on the <1% test set.", "666806": "same result, 10x LB scores " } }