{
  "id": 109996,
  "title": "How to best handle imbalanced data?",
  "url": "/competitions/rsna-intracranial-hemorrhage-detection/discussion/109996",
  "author_name": "Bo Peng",
  "post_date": "2019-09-24T03:45:22.409000",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>The dataset is imbalanced with a high number of images labeled as 0 correct? So how best to handle this imbalance?</p>",
  "messages": [
    {
      "id": 632989,
      "postDate": "2019-09-24T10:00:14.423Z",
      "content": "<p>This paper might be helpful:\nA systematic study of the class imbalance problem in convolutional neural networks\n<a href=\"https://arxiv.org/abs/1710.05381\">https://arxiv.org/abs/1710.05381</a></p>",
      "rawMarkdown": "This paper might be helpful:\nA systematic study of the class imbalance problem in convolutional neural networks\nhttps://arxiv.org/abs/1710.05381",
      "votes": 5,
      "replies": [
        {
          "id": 634153,
          "postDate": "2019-09-26T00:13:55.937Z",
          "content": "<p>Great paper! Thank you!</p>",
          "rawMarkdown": "Great paper! Thank you!",
          "votes": 2
        }
      ]
    },
    {
      "id": 641763,
      "postDate": "2019-10-05T04:05:12.560Z",
      "content": "<p>I have try both under and over sampling, and currently under sampling work better for me. </p>",
      "rawMarkdown": "I have try both under and over sampling, and currently under sampling work better for me. ",
      "votes": 1,
      "replies": [
        {
          "id": 644085,
          "postDate": "2019-10-08T09:49:23.887Z",
          "rawMarkdown": "",
          "votes": 1
        }
      ]
    },
    {
      "id": 641623,
      "postDate": "2019-10-04T21:18:47.370Z",
      "content": "<p>IMHO, there is no rule about which strategy you should follow! it depends completely on the data. Sometimes down/oversampling both harm the model training</p>",
      "rawMarkdown": "IMHO, there is no rule about which strategy you should follow! it depends completely on the data. Sometimes down/oversampling both harm the model training",
      "votes": 1
    },
    {
      "id": 634387,
      "postDate": "2019-09-26T08:14:37.337Z",
      "content": "<p>There was a  <a href=\"https://twitter.com/TheWolfBrain/status/1118211455604674565?s=20\">poll on twitter</a> earlier this year on how to handle class imbalances to do ML in Neuroscience. Unfortunately not too many explanations on why these approaches are used. Almost 80 people responded and respondents are split between over- and under-sampling. \nFor ML-applications in Neuroscience linear models seem to be more common and more efficient than Deep-Learning models, which is mostly due to the limited amount of very noisy data (see e.g. <a href=\"https://hal.archives-ouvertes.fr/hal-02276649/document\">https://hal.archives-ouvertes.fr/hal-02276649/document</a> ).\nSo I am not sure how much the above poll is related or helpful for DL models.</p>",
      "rawMarkdown": "There was a  [poll on twitter](https://twitter.com/TheWolfBrain/status/1118211455604674565?s=20) earlier this year on how to handle class imbalances to do ML in Neuroscience. Unfortunately not too many explanations on why these approaches are used. Almost 80 people responded and respondents are split between over- and under-sampling. \nFor ML-applications in Neuroscience linear models seem to be more common and more efficient than Deep-Learning models, which is mostly due to the limited amount of very noisy data (see e.g. [https://hal.archives-ouvertes.fr/hal-02276649/document](https://hal.archives-ouvertes.fr/hal-02276649/document) ).\nSo I am not sure how much the above poll is related or helpful for DL models.",
      "votes": 2
    },
    {
      "id": 632779,
      "postDate": "2019-09-24T03:45:22.410Z",
      "content": "<p>The dataset is imbalanced with a high number of images labeled as 0 correct? So how best to handle this imbalance?</p>",
      "rawMarkdown": "The dataset is imbalanced with a high number of images labeled as 0 correct? So how best to handle this imbalance?",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 632989,
      "author_name": "vbookshelf",
      "author_url": "",
      "post_date": "2019-09-24T10:00:14.423000",
      "content": "<p>This paper might be helpful:\nA systematic study of the class imbalance problem in convolutional neural networks\n<a href=\"https://arxiv.org/abs/1710.05381\">https://arxiv.org/abs/1710.05381</a></p>",
      "votes": 5,
      "replies": [
        {
          "id": 634153,
          "author_name": "Tom H.",
          "author_url": "",
          "post_date": "2019-09-26T00:13:55.937000",
          "content": "<p>Great paper! Thank you!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 641763,
      "author_name": "nan",
      "author_url": "",
      "post_date": "2019-10-05T04:05:12.560000",
      "content": "<p>I have try both under and over sampling, and currently under sampling work better for me. </p>",
      "votes": 1,
      "replies": [
        {
          "id": 644085,
          "author_name": "Akash Sharma",
          "author_url": "",
          "post_date": "2019-10-08T09:49:23.887000",
          "content": "",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 641623,
      "author_name": "sm_erlo",
      "author_url": "",
      "post_date": "2019-10-04T21:18:47.370000",
      "content": "<p>IMHO, there is no rule about which strategy you should follow! it depends completely on the data. Sometimes down/oversampling both harm the model training</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 634387,
      "author_name": "srs",
      "author_url": "",
      "post_date": "2019-09-26T08:14:37.337000",
      "content": "<p>There was a  <a href=\"https://twitter.com/TheWolfBrain/status/1118211455604674565?s=20\">poll on twitter</a> earlier this year on how to handle class imbalances to do ML in Neuroscience. Unfortunately not too many explanations on why these approaches are used. Almost 80 people responded and respondents are split between over- and under-sampling. \nFor ML-applications in Neuroscience linear models seem to be more common and more efficient than Deep-Learning models, which is mostly due to the limited amount of very noisy data (see e.g. <a href=\"https://hal.archives-ouvertes.fr/hal-02276649/document\">https://hal.archives-ouvertes.fr/hal-02276649/document</a> ).\nSo I am not sure how much the above poll is related or helpful for DL models.</p>",
      "votes": 2,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "632989": "This paper might be helpful:\nA systematic study of the class imbalance problem in convolutional neural networks\nhttps://arxiv.org/abs/1710.05381",
    "641763": "I have try both under and over sampling, and currently under sampling work better for me. ",
    "641623": "IMHO, there is no rule about which strategy you should follow! it depends completely on the data. Sometimes down/oversampling both harm the model training",
    "634387": "There was a  [poll on twitter](https://twitter.com/TheWolfBrain/status/1118211455604674565?s=20) earlier this year on how to handle class imbalances to do ML in Neuroscience. Unfortunately not too many explanations on why these approaches are used. Almost 80 people responded and respondents are split between over- and under-sampling. \nFor ML-applications in Neuroscience linear models seem to be more common and more efficient than Deep-Learning models, which is mostly due to the limited amount of very noisy data (see e.g. [https://hal.archives-ouvertes.fr/hal-02276649/document](https://hal.archives-ouvertes.fr/hal-02276649/document) ).\nSo I am not sure how much the above poll is related or helpful for DL models.",
    "632779": "The dataset is imbalanced with a high number of images labeled as 0 correct? So how best to handle this imbalance?"
  }
}