{
  "id": 341854,
  "title": "Competition metric in PyTorch",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/341854",
  "author_name": "Bartek Sadlej",
  "post_date": "2022-08-04T13:58:40.368000",
  "votes": 25,
  "comment_count": 4,
  "views": 0,
  "content": "<p>edit added <code>return loss / weights.sum()</code> <a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329</a><br>\nedit (29/08/2022) Regarding loss normalization. It should in fact be done on instance leven, not batch level. Addition and division are not commutative. Sorry for not rethinking it while updating with normalization. Now I get really correlated LB score.</p>\n<p>Code to compute comptetition metric in PyTorch according to this host's post with weights<br>\n<a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392</a></p>\n<pre><code>loss_fn = nn.BCEWithLogitsLoss(reduction='none')\n\ncompetition_weights = {\n    '-' : torch.tensor([7, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float, device=device),\n    '+' : torch.tensor([14, 2, 2, 2, 2, 2, 2, 2], dtype=torch.float, device=device),\n}\n\n# def competiton_loss(y_hat, y):\n#     loss = loss_fn(y_hat, y)\n#     weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n#     loss = (loss * weights).sum(axis=1).mean()\n\n#     return loss / weights.sum()\n\n\ndef competiton_loss(y_hat, y):\n    loss = loss_fn(y_hat, y)\n    weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n    loss = (loss * weights).sum(axis=1)\n    loss = loss / weights.sum(axis=1)\n    loss = loss.mean()\n\n    return loss \n</code></pre>",
  "messages": [
    {
      "id": 1884562,
      "postDate": "2022-08-04T13:58:40.370Z",
      "content": "<p>edit added <code>return loss / weights.sum()</code> <a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329</a><br>\nedit (29/08/2022) Regarding loss normalization. It should in fact be done on instance leven, not batch level. Addition and division are not commutative. Sorry for not rethinking it while updating with normalization. Now I get really correlated LB score.</p>\n<p>Code to compute comptetition metric in PyTorch according to this host's post with weights<br>\n<a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392</a></p>\n<pre><code>loss_fn = nn.BCEWithLogitsLoss(reduction='none')\n\ncompetition_weights = {\n    '-' : torch.tensor([7, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float, device=device),\n    '+' : torch.tensor([14, 2, 2, 2, 2, 2, 2, 2], dtype=torch.float, device=device),\n}\n\n# def competiton_loss(y_hat, y):\n#     loss = loss_fn(y_hat, y)\n#     weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n#     loss = (loss * weights).sum(axis=1).mean()\n\n#     return loss / weights.sum()\n\n\ndef competiton_loss(y_hat, y):\n    loss = loss_fn(y_hat, y)\n    weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n    loss = (loss * weights).sum(axis=1)\n    loss = loss / weights.sum(axis=1)\n    loss = loss.mean()\n\n    return loss \n</code></pre>",
      "rawMarkdown": "edit added `return loss / weights.sum()` https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329\nedit (29/08/2022) Regarding loss normalization. It should in fact be done on instance leven, not batch level. Addition and division are not commutative. Sorry for not rethinking it while updating with normalization. Now I get really correlated LB score.\n\n\nCode to compute comptetition metric in PyTorch according to this host's post with weights\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\n\n```\nloss_fn = nn.BCEWithLogitsLoss(reduction='none')\n\ncompetition_weights = {\n    '-' : torch.tensor([7, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float, device=device),\n    '+' : torch.tensor([14, 2, 2, 2, 2, 2, 2, 2], dtype=torch.float, device=device),\n}\n\n# def competiton_loss(y_hat, y):\n#     loss = loss_fn(y_hat, y)\n#     weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n#     loss = (loss * weights).sum(axis=1).mean()\n    \n#     return loss / weights.sum()\n\n\ndef competiton_loss(y_hat, y):\n    loss = loss_fn(y_hat, y)\n    weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n    loss = (loss * weights).sum(axis=1)\n    loss = loss / weights.sum(axis=1)\n    loss = loss.mean()\n    \n    return loss \n```",
      "votes": 24
    },
    {
      "id": 1890222,
      "postDate": "2022-08-08T15:58:59.543Z",
      "content": "<p>correct me if I am wrong but according to this: </p>\n<pre><code>Vertebrae negative: 1\nVertebrae positive: 2\nPatient negative: 7\nPatient positive: 14\n</code></pre>\n<p>if we have labels <code>0, 0, 0, 0, 0, 1, 0, 1</code> (<code>C1-C7</code>, <code>overall patient</code>) i thought the weights will be <code>1, 1, 1, 1, 1, 2, 1, 14</code>? </p>",
      "rawMarkdown": "correct me if I am wrong but according to this: \n```\nVertebrae negative: 1\nVertebrae positive: 2\nPatient negative: 7\nPatient positive: 14\n```\nif we have labels `0, 0, 0, 0, 0, 1, 0, 1 ` (`C1-C7`, `overall patient`) i thought the weights will be `1, 1, 1, 1, 1, 2, 1, 14`? \n\n",
      "votes": 1,
      "replies": [
        {
          "id": 1890247,
          "postDate": "2022-08-08T16:16:20.803Z",
          "content": "<p>That's correct, though I believe the overall prediction comes after the vertebrae-level predictions in the actual submission file.</p>",
          "rawMarkdown": "That's correct, though I believe the overall prediction comes after the vertebrae-level predictions in the actual submission file.",
          "votes": 3
        },
        {
          "id": 1890251,
          "postDate": "2022-08-08T16:18:52.947Z",
          "content": "<p>Thank you for the replay. I edited my comment<br>\n!</p>",
          "rawMarkdown": "Thank you for the replay. I edited my comment\n!",
          "votes": 1
        }
      ]
    },
    {
      "id": 1885303,
      "postDate": "2022-08-05T04:47:15.197Z",
      "content": "<p>You can normalize this loss by using <code>return loss / weights.sum()</code></p>",
      "rawMarkdown": "You can normalize this loss by using `return loss / weights.sum()`",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 1890222,
      "author_name": "DrHB",
      "author_url": "",
      "post_date": "2022-08-08T15:58:59.543000",
      "content": "<p>correct me if I am wrong but according to this: </p>\n<pre><code>Vertebrae negative: 1\nVertebrae positive: 2\nPatient negative: 7\nPatient positive: 14\n</code></pre>\n<p>if we have labels <code>0, 0, 0, 0, 0, 1, 0, 1</code> (<code>C1-C7</code>, <code>overall patient</code>) i thought the weights will be <code>1, 1, 1, 1, 1, 2, 1, 14</code>? </p>",
      "votes": 1,
      "replies": [
        {
          "id": 1890247,
          "author_name": "Sohier Dane",
          "author_url": "",
          "post_date": "2022-08-08T16:16:20.803000",
          "content": "<p>That's correct, though I believe the overall prediction comes after the vertebrae-level predictions in the actual submission file.</p>",
          "votes": 3,
          "replies": []
        },
        {
          "id": 1890251,
          "author_name": "DrHB",
          "author_url": "",
          "post_date": "2022-08-08T16:18:52.947000",
          "content": "<p>Thank you for the replay. I edited my comment<br>\n!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 1885303,
      "author_name": "Harshit Sheoran",
      "author_url": "",
      "post_date": "2022-08-05T04:47:15.197000",
      "content": "<p>You can normalize this loss by using <code>return loss / weights.sum()</code></p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1884562": "edit added `return loss / weights.sum()` https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392#1885329\nedit (29/08/2022) Regarding loss normalization. It should in fact be done on instance leven, not batch level. Addition and division are not commutative. Sorry for not rethinking it while updating with normalization. Now I get really correlated LB score.\n\n\nCode to compute comptetition metric in PyTorch according to this host's post with weights\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340392\n\n```\nloss_fn = nn.BCEWithLogitsLoss(reduction='none')\n\ncompetition_weights = {\n    '-' : torch.tensor([7, 1, 1, 1, 1, 1, 1, 1], dtype=torch.float, device=device),\n    '+' : torch.tensor([14, 2, 2, 2, 2, 2, 2, 2], dtype=torch.float, device=device),\n}\n\n# def competiton_loss(y_hat, y):\n#     loss = loss_fn(y_hat, y)\n#     weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n#     loss = (loss * weights).sum(axis=1).mean()\n    \n#     return loss / weights.sum()\n\n\ndef competiton_loss(y_hat, y):\n    loss = loss_fn(y_hat, y)\n    weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n    loss = (loss * weights).sum(axis=1)\n    loss = loss / weights.sum(axis=1)\n    loss = loss.mean()\n    \n    return loss \n```",
    "1890222": "correct me if I am wrong but according to this: \n```\nVertebrae negative: 1\nVertebrae positive: 2\nPatient negative: 7\nPatient positive: 14\n```\nif we have labels `0, 0, 0, 0, 0, 1, 0, 1 ` (`C1-C7`, `overall patient`) i thought the weights will be `1, 1, 1, 1, 1, 2, 1, 14`? \n\n",
    "1885303": "You can normalize this loss by using `return loss / weights.sum()`"
  }
}