![\"\"](\"https://lh3.googleusercontent.com/QyaseGU8MQd8h6cvQok1Vi7SE9k62UV2Hz1oqhB0Ct178H1T4kIusuhObZw3CSifDPaR1N035GtLVA\")
the kaggle metric is log loss and not accuracy.
\n\nthis means that the submitted probability must be calibrated. If stage2 test data and train2 (or train1) set has different probability, maybe we will all be dead at the final ranking
", "messages": [ { "id": 672030, "postDate": "2019-11-13T13:10:44.363Z", "content": "the kaggle metric is log loss and not accuracy.
\n\nthis means that the submitted probability must be calibrated. If stage2 test data and train2 (or train1) set has different probability, maybe we will all be dead at the final ranking
", "rawMarkdown": "the kaggle metric is log loss and not accuracy.\n\nthis means that the submitted probability must be calibrated. If stage2 test data and train2 (or train1) set has different probability, maybe we will all be dead at the final ranking", "votes": 7 }, { "id": 672415, "postDate": "2019-11-13T21:14:38.307Z", "content": "Haha cool meme picture @hengck23.
\n\nCan someone explain what is the practical difference between targetting log loss scores vs accuracy?
\n\nFrom my basic understanding, log-loss has a higher penalty on the scores for wrong predictions.
", "rawMarkdown": "Haha cool meme picture @hengck23.\n\nCan someone explain what is the practical difference between targetting log loss scores vs accuracy?\n\nFrom my basic understanding, log-loss has a higher penalty on the scores for wrong predictions.", "votes": 1, "replies": [ { "id": 672470, "postDate": "2019-11-13T23:40:30.413Z", "content": "assuming that we have the ground truth\n(x1,y1), (x2,y2), (x3,y3)
\n\nassuming that the image are very close and indistinguishable (i.e. we are going to have some error). we write x1=x2=x3=x. let assume y1=1,y2=0 and y3=0.
\n\nso now our model estimate p(y|x).\nfor binary classification, we would threshold at 0.5
\n\ncase one:\np(y|x) =0.33. \nlog loss = ..., accuracy = 2/3
\n\ncase two:\np(y|x) = 0.44.\nlog loss = ..., accuracy = 2/3
\n\ncase three:\np(y|x) = 0.0001.\nlog loss = ..., accuracy = 2/3
\n\nyou can see that accuracy can be constant, but only case one will have optimal log loss.
", "rawMarkdown": "assuming that we have the ground truth\n(x1,y1), (x2,y2), (x3,y3)\n\nassuming that the image are very close and indistinguishable (i.e. we are going to have some error). we write x1=x2=x3=x. let assume y1=1,y2=0 and y3=0.\n\nso now our model estimate p(y|x).\nfor binary classification, we would threshold at 0.5\n\ncase one:\np(y|x) =0.33. \nlog loss = ..., accuracy = 2/3\n\ncase two:\np(y|x) = 0.44.\nlog loss = ..., accuracy = 2/3\n\ncase three:\np(y|x) = 0.0001.\nlog loss = ..., accuracy = 2/3\n\n\n\nyou can see that accuracy can be constant, but only case one will have optimal log loss.\n", "votes": 3 }, { "id": 672996, "postDate": "2019-11-14T11:22:34.117Z", "content": "Thanks for breaking it down! I will need to examine this better.
", "rawMarkdown": "Thanks for breaking it down! I will need to examine this better. " } ] }, { "id": 672141, "postDate": "2019-11-13T14:53:46.257Z", "content": "Well, so far it can't be worse than ~500/1300 ¯\\__(ツ)__/¯
", "rawMarkdown": "Well, so far it can't be worse than ~500/1300 ¯\\\\_\\_(ツ)__/¯", "votes": 1, "replies": [ { "id": 672195, "postDate": "2019-11-13T15:51:45.093Z", "content": "That‘s true. 😄
", "rawMarkdown": "That‘s true. 😄 " } ] }, { "id": 672068, "postDate": "2019-11-13T13:40:27.313Z", "content": "So how do we calibrate the probability of submission? I have tried to make the predicted probability less than a certain value equal to 0. If it is greater than a certain probability value, it will be equal to 1, but such loss value will rise a lot. If you are not convenient to disclose, can you teach us after the game is over? Thanks a lot！
", "rawMarkdown": "So how do we calibrate the probability of submission? I have tried to make the predicted probability less than a certain value equal to 0. If it is greater than a certain probability value, it will be equal to 1, but such loss value will rise a lot. If you are not convenient to disclose, can you teach us after the game is over? Thanks a lot！", "votes": 1 }, { "id": 672132, "postDate": "2019-11-13T14:44:48.980Z", "content": "Heng, \ndid you try to see if the proportion of positive cases are similar between stage 1 test and stage 2 test submissions? I’ve got reassuringly similar percentages.
", "rawMarkdown": "Heng, \ndid you try to see if the proportion of positive cases are similar between stage 1 test and stage 2 test submissions? I’ve got reassuringly similar percentages." } ], "comments": [ { "id": 672415, "author_name": "David Tang", "author_url": "", "post_date": "2019-11-13T21:14:38.307000", "content": "Haha cool meme picture @hengck23.
\n\nCan someone explain what is the practical difference between targetting log loss scores vs accuracy?
\n\nFrom my basic understanding, log-loss has a higher penalty on the scores for wrong predictions.
", "votes": 1, "replies": [ { "id": 672470, "author_name": "hengck23", "author_url": "", "post_date": "2019-11-13T23:40:30.413000", "content": "assuming that we have the ground truth\n(x1,y1), (x2,y2), (x3,y3)
\n\nassuming that the image are very close and indistinguishable (i.e. we are going to have some error). we write x1=x2=x3=x. let assume y1=1,y2=0 and y3=0.
\n\nso now our model estimate p(y|x).\nfor binary classification, we would threshold at 0.5
\n\ncase one:\np(y|x) =0.33. \nlog loss = ..., accuracy = 2/3
\n\ncase two:\np(y|x) = 0.44.\nlog loss = ..., accuracy = 2/3
\n\ncase three:\np(y|x) = 0.0001.\nlog loss = ..., accuracy = 2/3
\n\nyou can see that accuracy can be constant, but only case one will have optimal log loss.
", "votes": 3, "replies": [] }, { "id": 672996, "author_name": "David Tang", "author_url": "", "post_date": "2019-11-14T11:22:34.117000", "content": "Thanks for breaking it down! I will need to examine this better.
", "votes": 0, "replies": [] } ] }, { "id": 672141, "author_name": "César Parra Rojas", "author_url": "", "post_date": "2019-11-13T14:53:46.257000", "content": "Well, so far it can't be worse than ~500/1300 ¯\\__(ツ)__/¯
", "votes": 1, "replies": [ { "id": 672195, "author_name": "Yiyuan", "author_url": "", "post_date": "2019-11-13T15:51:45.093000", "content": "That‘s true. 😄
", "votes": 0, "replies": [] } ] }, { "id": 672068, "author_name": "Yiyuan", "author_url": "", "post_date": "2019-11-13T13:40:27.313000", "content": "So how do we calibrate the probability of submission? I have tried to make the predicted probability less than a certain value equal to 0. If it is greater than a certain probability value, it will be equal to 1, but such loss value will rise a lot. If you are not convenient to disclose, can you teach us after the game is over? Thanks a lot！
", "votes": 1, "replies": [] }, { "id": 672132, "author_name": "hengck23", "author_url": "", "post_date": "2019-11-13T14:44:48.980000", "content": "Heng, \ndid you try to see if the proportion of positive cases are similar between stage 1 test and stage 2 test submissions? I’ve got reassuringly similar percentages.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "672030": "the kaggle metric is log loss and not accuracy.\n\nthis means that the submitted probability must be calibrated. If stage2 test data and train2 (or train1) set has different probability, maybe we will all be dead at the final ranking", "672415": "Haha cool meme picture @hengck23.\n\nCan someone explain what is the practical difference between targetting log loss scores vs accuracy?\n\nFrom my basic understanding, log-loss has a higher penalty on the scores for wrong predictions.", "672141": "Well, so far it can't be worse than ~500/1300 ¯\\\\_\\_(ツ)__/¯", "672068": "So how do we calibrate the probability of submission? I have tried to make the predicted probability less than a certain value equal to 0. If it is greater than a certain probability value, it will be equal to 1, but such loss value will rise a lot. If you are not convenient to disclose, can you teach us after the game is over? Thanks a lot！", "672132": "", "672070": "Heng, \ndid you try to see if the proportion of positive cases are similar between stage 1 test and stage 2 test submissions? I’ve got reassuringly similar percentages." } }