![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F1e6fb740684b6cdb77eef4f26d56f8e2%2FD62E03A7-9571-4737-B728-BD5D7DD57AE4.png?generation=1572716480280286&alt=media\")
sampling/weighing is important in training
\n\nwe sometimes use loss weighing (e.g focal loss) or class balancing to speed up training or improve accuracy. but improving accuracy is \"not the same\" as reducing loss. the loss is modified to improve accuracy by artificially changing the train distribution and affecting the decision boundary.
\n\nin summary. we care only about the boundary, and care less about the decision value
\n\nfor this challenge, we care about the actual log loss itself. so it is important to \"calibrate your probability\" to the actual one you expect in the test during training. you may have to use random sampling (instead of balanced sampling) at the final fine tuning. likewise you may need to disable any sample weighing.
\n\nthere will be some different between probability expressed in float 32 bit and 64 bit
\n\nthis is is very difficult (and very risky in scoring). e.g. a test sample is predicted to be negative with probability 0.0001. you may round it to zero. you will gain a little if the truth is really negative, but you will lose big if it is truth is instead positive.
\n\nif you can train with \"large margin\" and use some probability calibration trick such that you are almost 100% sure that there cannot be positive truth below a certain low values, you can zero out the values.
", "messages": [ { "id": 659671, "postDate": "2019-10-28T04:10:42.453Z", "content": "sampling/weighing is important in training
\n\nwe sometimes use loss weighing (e.g focal loss) or class balancing to speed up training or improve accuracy. but improving accuracy is \"not the same\" as reducing loss. the loss is modified to improve accuracy by artificially changing the train distribution and affecting the decision boundary.
\n\nin summary. we care only about the boundary, and care less about the decision value
\n\nfor this challenge, we care about the actual log loss itself. so it is important to \"calibrate your probability\" to the actual one you expect in the test during training. you may have to use random sampling (instead of balanced sampling) at the final fine tuning. likewise you may need to disable any sample weighing.
\n\nthere will be some different between probability expressed in float 32 bit and 64 bit
\n\nthis is is very difficult (and very risky in scoring). e.g. a test sample is predicted to be negative with probability 0.0001. you may round it to zero. you will gain a little if the truth is really negative, but you will lose big if it is truth is instead positive.
\n\nif you can train with \"large margin\" and use some probability calibration trick such that you are almost 100% sure that there cannot be positive truth below a certain low values, you can zero out the values.
", "rawMarkdown": "1. sampling/weighing is important in training\n\n- when the metric is accuracy, we usually use : predict\\_label = predict\\_prob > threshold\nwe do not care about predict\\_prob, but rather thresholded prediction\n\nwe sometimes use loss weighing (e.g focal loss) or class balancing to speed up training or improve accuracy. but improving accuracy is \"not the same\" as reducing loss. the loss is modified to improve accuracy by artificially changing the train distribution and affecting the decision boundary. \n\n**in summary. we care only about the boundary, and care less about the decision value**\n\n\nfor this challenge, we care about the actual log loss itself. so it is important to **\"calibrate your probability\"** to the actual one you expect in the test during training. you may have to use random sampling (instead of balanced sampling) at the final fine tuning. likewise you may need to disable any sample weighing.\n\n---\n\n2. decimal is important\n\nthere will be some different between probability expressed in float 32 bit and 64 bit\n\n\n---\n\n3. probability shaping\n\nthis is is very difficult (and very risky in scoring). e.g. a test sample is predicted to be negative with probability 0.0001. you may round it to zero. you will gain a little if the truth is really negative, but you will lose big if it is truth is instead positive.\n\nif you can train with \"large margin\" and use some probability calibration trick such that you are almost 100% sure that there cannot be positive truth below a certain low values, you can zero out the values. \n", "votes": 20 }, { "id": 663815, "postDate": "2019-11-02T17:41:25.107Z", "content": "
![\"\"](\"https://d3i71xaburhd42.cloudfront.net/68e8584f9a26ec2db828ea0cd13b8fc78a0e9a79/3-Figure2-1.png\")
First time working with log loss 🙀…Will try your ideas.Thanks for sharing
", "rawMarkdown": "First time working with log loss 🙀...Will try your ideas.Thanks for sharing " }, { "id": 662138, "postDate": "2019-10-31T06:04:32.090Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F113660%2F86746f8aa2a49bfbebe61d3ebf5370d9%2FSelection_060.png?generation=1572501869456145&alt=media\")
@Jeremy Howard has a notebook to detect such images. he used it to remove useless images without brain tissue to reduce training set
", "rawMarkdown": "\n\n\n@Jeremy Howard has a notebook to detect such images. he used it to remove useless images without brain tissue to reduce training set" }, { "id": 659893, "postDate": "2019-10-28T11:55:29.973Z", "content": "one example of probability shaping:
\n\n```
\n\nNAME_TO_LABEL={\n'any' : 0,\n'epidural' : 1,\n'intraparenchymal' : 2,\n'intraventricular' : 3,\n'subarachnoid' : 4,\n'subdural' : 5,\n}\n if 1: #reshape
\n pos = probability_label[:,1:].max(-1)\n neg = probability_label[:,1:].min(-1)\n p0 = probability_label[:,0]
probability_label[:,0][p0>0.99]=pos[p0>0.99]\n probability_label[:,0][p0<0.01]=neg[p0<0.01]\n```
\n\nquestion ask:
\n\na network to learning the shaping is better than heuristic guess, e.g. lstm layer label refinement
\n\nrelated: https://github.com/gpleiss/temperature_scaling
", "rawMarkdown": "one example of probability shaping:\n\n```\n\nNAME_TO_LABEL={\n'any' : 0,\n'epidural' : 1,\n'intraparenchymal' : 2,\n'intraventricular' : 3,\n'subarachnoid' : 4,\n'subdural' : 5,\n}\n if 1: #reshape \n pos = probability_label[:,1:].max(-1)\n neg = probability_label[:,1:].min(-1)\n p0 = probability_label[:,0]\n\n probability_label[:,0][p0>0.99]=pos[p0>0.99]\n probability_label[:,0][p0<0.01]=neg[p0<0.01]\n\n \n\n```\n\n\nquestion ask:\n\n- can p(any) be less or greater then p(epidural), p(intraparenchymal), etc \n- if sum(p(epidural), p(intraparenchymal)) is large, can p(any) be small?\n\na network to learning the shaping is better than heuristic guess, e.g. lstm layer label refinement\n\nrelated: https://github.com/gpleiss/temperature_scaling" }, { "id": 659758, "postDate": "2019-10-28T08:23:12.383Z", "content": "@hengck23 About the probability shaping, I'm asking with an open mind: Wouldn't it just \"even out\"? The 99 predictions you correctly round to 0 and the 1 prediction you incorrectly round to 0? So right now I'm thinking, it's better just to trust the Neural net :-D
\n\nAnother thought: because the any=0 is highly over-represented, the Neural net might tend to make overly pessimistic predictions for any=1 (i.e. give relatively low probabilities for them). Do you think this is True? If yes, would there be a way to solve this in a post-processing step? :-)
", "rawMarkdown": "@hengck23 About the *probability shaping*, I'm asking with an open mind: Wouldn't it just \"even out\"? The 99 predictions you correctly round to 0 and the 1 prediction you incorrectly round to 0? So right now I'm thinking, it's better just to trust the Neural net :-D\n\nAnother thought: because the any=0 is highly over-represented, the Neural net might tend to make overly pessimistic predictions for any=1 (i.e. give relatively low probabilities for them). Do you think this is True? If yes, would there be a way to solve this in a post-processing step? :-) ", "replies": [ { "id": 659761, "postDate": "2019-10-28T08:27:15.620Z", "content": "\"Wouldn't it just \"even out\"?\"
\n\nthe log loss is designed to even out. but if you have external post processing, then maybe you can make a gain.
\n\na modification of the log loss may help .... e.g if we know there are more negative samples than positive ...
\n\n\" So right now I'm thinking, it's better just to trust the Neural net :-D\"
\n\nso the sampling will be important for this
", "rawMarkdown": "\"Wouldn't it just \"even out\"?\" \n\nthe log loss is designed to even out. but if you have external post processing, then maybe you can make a gain.\n\na modification of the log loss may help .... e.g if we know there are more negative samples than positive ...\n\n---\n\n\" So right now I'm thinking, it's better just to trust the Neural net :-D\"\n\nso the sampling will be important for this", "votes": 1 } ] } ], "comments": [ { "id": 663815, "author_name": "hengck23", "author_url": "", "post_date": "2019-11-02T17:41:25.107000", "content": "
First time working with log loss 🙀…Will try your ideas.Thanks for sharing
", "votes": 0, "replies": [] }, { "id": 662138, "author_name": "hengck23", "author_url": "", "post_date": "2019-10-31T06:04:32.090000", "content": "
@Jeremy Howard has a notebook to detect such images. he used it to remove useless images without brain tissue to reduce training set
", "votes": 0, "replies": [] }, { "id": 659893, "author_name": "hengck23", "author_url": "", "post_date": "2019-10-28T11:55:29.973000", "content": "one example of probability shaping:
\n\n```
\n\nNAME_TO_LABEL={\n'any' : 0,\n'epidural' : 1,\n'intraparenchymal' : 2,\n'intraventricular' : 3,\n'subarachnoid' : 4,\n'subdural' : 5,\n}\n if 1: #reshape
\n pos = probability_label[:,1:].max(-1)\n neg = probability_label[:,1:].min(-1)\n p0 = probability_label[:,0]
probability_label[:,0][p0>0.99]=pos[p0>0.99]\n probability_label[:,0][p0<0.01]=neg[p0<0.01]\n```
\n\nquestion ask:
\n\na network to learning the shaping is better than heuristic guess, e.g. lstm layer label refinement
\n\nrelated: https://github.com/gpleiss/temperature_scaling
", "votes": 0, "replies": [] }, { "id": 659758, "author_name": "Alex", "author_url": "", "post_date": "2019-10-28T08:23:12.383000", "content": "@hengck23 About the probability shaping, I'm asking with an open mind: Wouldn't it just \"even out\"? The 99 predictions you correctly round to 0 and the 1 prediction you incorrectly round to 0? So right now I'm thinking, it's better just to trust the Neural net :-D
\n\nAnother thought: because the any=0 is highly over-represented, the Neural net might tend to make overly pessimistic predictions for any=1 (i.e. give relatively low probabilities for them). Do you think this is True? If yes, would there be a way to solve this in a post-processing step? :-)
", "votes": 0, "replies": [ { "id": 659761, "author_name": "hengck23", "author_url": "", "post_date": "2019-10-28T08:27:15.620000", "content": "\"Wouldn't it just \"even out\"?\"
\n\nthe log loss is designed to even out. but if you have external post processing, then maybe you can make a gain.
\n\na modification of the log loss may help .... e.g if we know there are more negative samples than positive ...
\n\n\" So right now I'm thinking, it's better just to trust the Neural net :-D\"
\n\nso the sampling will be important for this
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "659671": "1. sampling/weighing is important in training\n\n- when the metric is accuracy, we usually use : predict\\_label = predict\\_prob > threshold\nwe do not care about predict\\_prob, but rather thresholded prediction\n\nwe sometimes use loss weighing (e.g focal loss) or class balancing to speed up training or improve accuracy. but improving accuracy is \"not the same\" as reducing loss. the loss is modified to improve accuracy by artificially changing the train distribution and affecting the decision boundary. \n\n**in summary. we care only about the boundary, and care less about the decision value**\n\n\nfor this challenge, we care about the actual log loss itself. so it is important to **\"calibrate your probability\"** to the actual one you expect in the test during training. you may have to use random sampling (instead of balanced sampling) at the final fine tuning. likewise you may need to disable any sample weighing.\n\n---\n\n2. decimal is important\n\nthere will be some different between probability expressed in float 32 bit and 64 bit\n\n\n---\n\n3. probability shaping\n\nthis is is very difficult (and very risky in scoring). e.g. a test sample is predicted to be negative with probability 0.0001. you may round it to zero. you will gain a little if the truth is really negative, but you will lose big if it is truth is instead positive.\n\nif you can train with \"large margin\" and use some probability calibration trick such that you are almost 100% sure that there cannot be positive truth below a certain low values, you can zero out the values. \n", "663815": "\n", "659680": "", "3299349": "First time working with log loss 🙀...Will try your ideas.Thanks for sharing ", "662138": "\n\n\n@Jeremy Howard has a notebook to detect such images. he used it to remove useless images without brain tissue to reduce training set", "659893": "one example of probability shaping:\n\n```\n\nNAME_TO_LABEL={\n'any' : 0,\n'epidural' : 1,\n'intraparenchymal' : 2,\n'intraventricular' : 3,\n'subarachnoid' : 4,\n'subdural' : 5,\n}\n if 1: #reshape \n pos = probability_label[:,1:].max(-1)\n neg = probability_label[:,1:].min(-1)\n p0 = probability_label[:,0]\n\n probability_label[:,0][p0>0.99]=pos[p0>0.99]\n probability_label[:,0][p0<0.01]=neg[p0<0.01]\n\n \n\n```\n\n\nquestion ask:\n\n- can p(any) be less or greater then p(epidural), p(intraparenchymal), etc \n- if sum(p(epidural), p(intraparenchymal)) is large, can p(any) be small?\n\na network to learning the shaping is better than heuristic guess, e.g. lstm layer label refinement\n\nrelated: https://github.com/gpleiss/temperature_scaling", "659758": "@hengck23 About the *probability shaping*, I'm asking with an open mind: Wouldn't it just \"even out\"? The 99 predictions you correctly round to 0 and the 1 prediction you incorrectly round to 0? So right now I'm thinking, it's better just to trust the Neural net :-D\n\nAnother thought: because the any=0 is highly over-represented, the Neural net might tend to make overly pessimistic predictions for any=1 (i.e. give relatively low probabilities for them). Do you think this is True? If yes, would there be a way to solve this in a post-processing step? :-) " } }