{
  "id": 343969,
  "title": "Competition metric details",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/343969",
  "author_name": "Samuel Cortinhas",
  "post_date": "2022-08-13T11:08:40.895000",
  "votes": 5,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I tried to replicate the competition metric but a score of 0.56 on the LB gave me 1.28 on my metric so I think I'm missing something. Please tell me if my understanding is incorrect:</p>\n<p>If we have labels <code>[C1-C7,overall]=[1,0,0,0,1,0,0,1]</code>, then the weights for that patient are <code>[2,1,1,1,2,1,1,14]</code>.</p>\n<p>So the loss for that patient is  <br>\n<code>-2log(p1)-log(1-p2)-log(1-p3)-log(1-p4)-2log(p5)-log(1-p6)-log(1-p7)-14log(p_overall)</code></p>\n<p>Finally, we sum the loss of every patient and divide by the number of patients.</p>\n<p>Is that everything? Do we need to normalise the weights on a by-patient basis, or scale it any other way? <a href=\"https://www.kaggle.com/felipekitamura\" target=\"_blank\">@felipekitamura</a> </p>",
  "messages": [
    {
      "id": 1896993,
      "postDate": "2022-08-13T11:08:40.897Z",
      "content": "<p>I tried to replicate the competition metric but a score of 0.56 on the LB gave me 1.28 on my metric so I think I'm missing something. Please tell me if my understanding is incorrect:</p>\n<p>If we have labels <code>[C1-C7,overall]=[1,0,0,0,1,0,0,1]</code>, then the weights for that patient are <code>[2,1,1,1,2,1,1,14]</code>.</p>\n<p>So the loss for that patient is  <br>\n<code>-2log(p1)-log(1-p2)-log(1-p3)-log(1-p4)-2log(p5)-log(1-p6)-log(1-p7)-14log(p_overall)</code></p>\n<p>Finally, we sum the loss of every patient and divide by the number of patients.</p>\n<p>Is that everything? Do we need to normalise the weights on a by-patient basis, or scale it any other way? <a href=\"https://www.kaggle.com/felipekitamura\" target=\"_blank\">@felipekitamura</a> </p>",
      "rawMarkdown": "I tried to replicate the competition metric but a score of 0.56 on the LB gave me 1.28 on my metric so I think I'm missing something. Please tell me if my understanding is incorrect:\n\nIf we have labels `[C1-C7,overall]=[1,0,0,0,1,0,0,1]`, then the weights for that patient are `[2,1,1,1,2,1,1,14]`.\n\nSo the loss for that patient is  \n`-2log(p1)-log(1-p2)-log(1-p3)-log(1-p4)-2log(p5)-log(1-p6)-log(1-p7)-14log(p_overall)`\n\nFinally, we sum the loss of every patient and divide by the number of patients.\n\nIs that everything? Do we need to normalise the weights on a by-patient basis, or scale it any other way? @felipekitamura ",
      "votes": 5
    },
    {
      "id": 1897022,
      "postDate": "2022-08-13T11:36:44.853Z",
      "content": "<p>I believe that you need to divide your summed loss for each patient by the sum of all the weights. See<br>\n<a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392</a><br>\nI'm surprised that you only got 1.28 by dividing by number of patients rather than total weights.</p>",
      "rawMarkdown": "I believe that you need to divide your summed loss for each patient by the sum of all the weights. See\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\nI'm surprised that you only got 1.28 by dividing by number of patients rather than total weights.\n\n",
      "votes": 2,
      "replies": [
        {
          "id": 1897063,
          "postDate": "2022-08-13T12:04:57.713Z",
          "content": "<p>I think you're right, thanks. It would be good if the competition organisers could confirm this. </p>",
          "rawMarkdown": "I think you're right, thanks. It would be good if the competition organisers could confirm this. ",
          "votes": 1
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1897022,
      "author_name": "SolverWorld",
      "author_url": "",
      "post_date": "2022-08-13T11:36:44.853000",
      "content": "<p>I believe that you need to divide your summed loss for each patient by the sum of all the weights. See<br>\n<a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392</a><br>\nI'm surprised that you only got 1.28 by dividing by number of patients rather than total weights.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1897063,
          "author_name": "Samuel Cortinhas",
          "author_url": "",
          "post_date": "2022-08-13T12:04:57.713000",
          "content": "<p>I think you're right, thanks. It would be good if the competition organisers could confirm this. </p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1896993": "I tried to replicate the competition metric but a score of 0.56 on the LB gave me 1.28 on my metric so I think I'm missing something. Please tell me if my understanding is incorrect:\n\nIf we have labels `[C1-C7,overall]=[1,0,0,0,1,0,0,1]`, then the weights for that patient are `[2,1,1,1,2,1,1,14]`.\n\nSo the loss for that patient is  \n`-2log(p1)-log(1-p2)-log(1-p3)-log(1-p4)-2log(p5)-log(1-p6)-log(1-p7)-14log(p_overall)`\n\nFinally, we sum the loss of every patient and divide by the number of patients.\n\nIs that everything? Do we need to normalise the weights on a by-patient basis, or scale it any other way? @felipekitamura ",
    "1897022": "I believe that you need to divide your summed loss for each patient by the sum of all the weights. See\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\nI'm surprised that you only got 1.28 by dividing by number of patients rather than total weights.\n\n"
  }
}