{
  "id": 388547,
  "title": "CV and LB do not correlate.",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/388547",
  "author_name": "Nana-chan",
  "post_date": "2023-02-18T05:52:52.322000",
  "votes": 5,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I am suffering from the fact that CV and LB do not correlate in this competition.</p>\n<p>my cv strategy is below. (N_FOLDS=5)<br>\nI refer <a href=\"https://www.kaggle.com/code/vslaykovsky/train-pytorch-aux-targets-weighted-loss-thres?scriptVersionId=114303403\" target=\"_blank\">this notebook</a></p>\n<pre><code>split = StratifiedGroupKFold(N_FOLDS)\n k, (_, test_idx)  (split.split(df_train, df_train.cancer, groups=df_train.patient_id)):\n    df_train.loc[test_idx, ] = k\ndf_train.split = df_train.split.astype()\ndf_train.groupby().cancer.mean()\n</code></pre>\n<p>I use the code below. And I am saving the model with the largest score</p>\n<pre><code> ():\n    y_true_count = \n    ctp = \n    cfp = \n\n     idx  ((labels)):\n        prediction = ((predictions[idx], ), )\n         (labels[idx]):\n            y_true_count += \n            ctp += prediction\n        :\n            cfp += prediction\n\n    beta_squared = beta * beta\n    c_precision = ctp / (ctp + cfp)\n    c_recall = ctp / (y_true_count, )  \n     (c_precision &gt;   c_recall &gt; ):\n        result = ( + beta_squared) * (c_precision * c_recall) / (beta_squared * c_precision + c_recall)\n         result\n    :\n         \n\n ():\n    thres = np.linspace(, , )\n    f1s = [pfbeta(labels, predictions &gt; thr)  thr  thres]\n    idx = np.argmax(f1s)\n     f1s[idx], thres[idx]\n</code></pre>\n<p>The results of my experiment are as follows</p>\n<table>\n<thead>\n<tr>\n<th>POSITIVE_TARGET_WEIGHT</th>\n<th>AUX_LOSS_WEIGHT</th>\n<th>optimal_f1</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>20</td>\n<td>0</td>\n<td>0.395</td>\n<td>0.51</td>\n</tr>\n<tr>\n<td>20</td>\n<td>94</td>\n<td>0.412</td>\n<td>0.45</td>\n</tr>\n</tbody>\n</table>\n<p>Am I doing something wrong?<br>\nthanks.</p>",
  "messages": [
    {
      "id": 2149247,
      "postDate": "2023-02-18T05:52:52.323Z",
      "content": "<p>I am suffering from the fact that CV and LB do not correlate in this competition.</p>\n<p>my cv strategy is below. (N_FOLDS=5)<br>\nI refer <a href=\"https://www.kaggle.com/code/vslaykovsky/train-pytorch-aux-targets-weighted-loss-thres?scriptVersionId=114303403\" target=\"_blank\">this notebook</a></p>\n<pre><code>split = StratifiedGroupKFold(N_FOLDS)\n k, (_, test_idx)  (split.split(df_train, df_train.cancer, groups=df_train.patient_id)):\n    df_train.loc[test_idx, ] = k\ndf_train.split = df_train.split.astype()\ndf_train.groupby().cancer.mean()\n</code></pre>\n<p>I use the code below. And I am saving the model with the largest score</p>\n<pre><code> ():\n    y_true_count = \n    ctp = \n    cfp = \n\n     idx  ((labels)):\n        prediction = ((predictions[idx], ), )\n         (labels[idx]):\n            y_true_count += \n            ctp += prediction\n        :\n            cfp += prediction\n\n    beta_squared = beta * beta\n    c_precision = ctp / (ctp + cfp)\n    c_recall = ctp / (y_true_count, )  \n     (c_precision &gt;   c_recall &gt; ):\n        result = ( + beta_squared) * (c_precision * c_recall) / (beta_squared * c_precision + c_recall)\n         result\n    :\n         \n\n ():\n    thres = np.linspace(, , )\n    f1s = [pfbeta(labels, predictions &gt; thr)  thr  thres]\n    idx = np.argmax(f1s)\n     f1s[idx], thres[idx]\n</code></pre>\n<p>The results of my experiment are as follows</p>\n<table>\n<thead>\n<tr>\n<th>POSITIVE_TARGET_WEIGHT</th>\n<th>AUX_LOSS_WEIGHT</th>\n<th>optimal_f1</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>20</td>\n<td>0</td>\n<td>0.395</td>\n<td>0.51</td>\n</tr>\n<tr>\n<td>20</td>\n<td>94</td>\n<td>0.412</td>\n<td>0.45</td>\n</tr>\n</tbody>\n</table>\n<p>Am I doing something wrong?<br>\nthanks.</p>",
      "rawMarkdown": "I am suffering from the fact that CV and LB do not correlate in this competition.\n\nmy cv strategy is below. (N_FOLDS=5)\nI refer [this notebook](https://www.kaggle.com/code/vslaykovsky/train-pytorch-aux-targets-weighted-loss-thres?scriptVersionId=114303403)\n\n```python\nsplit = StratifiedGroupKFold(N_FOLDS)\nfor k, (_, test_idx) in enumerate(split.split(df_train, df_train.cancer, groups=df_train.patient_id)):\n    df_train.loc[test_idx, 'split'] = k\ndf_train.split = df_train.split.astype(int)\ndf_train.groupby('split').cancer.mean()\n```\n\nI use the code below. And I am saving the model with the largest score\n\n```python\ndef pfbeta(labels, predictions, beta=1.):\n    y_true_count = 0\n    ctp = 0\n    cfp = 0\n\n    for idx in range(len(labels)):\n        prediction = min(max(predictions[idx], 0), 1)\n        if (labels[idx]):\n            y_true_count += 1\n            ctp += prediction\n        else:\n            cfp += prediction\n\n    beta_squared = beta * beta\n    c_precision = ctp / (ctp + cfp)\n    c_recall = ctp / max(y_true_count, 1)  # avoid / 0\n    if (c_precision > 0 and c_recall > 0):\n        result = (1 + beta_squared) * (c_precision * c_recall) / (beta_squared * c_precision + c_recall)\n        return result\n    else:\n        return 0\n\ndef optimal_f1(labels, predictions):\n    thres = np.linspace(0, 1, 101)\n    f1s = [pfbeta(labels, predictions > thr) for thr in thres]\n    idx = np.argmax(f1s)\n    return f1s[idx], thres[idx]\n```\n\nThe results of my experiment are as follows\n\n| POSITIVE_TARGET_WEIGHT|AUX_LOSS_WEIGHT  |optimal_f1|LB|\n| --- | --- |--- |--- |\n| 20 | 0 |0.395 | 0.51|\n| 20 | 94 | 0.412 | 0.45|\n\nAm I doing something wrong?\nthanks.\n",
      "votes": 5
    },
    {
      "id": 2149713,
      "postDate": "2023-02-18T15:41:23.690Z",
      "content": "<p>Did you aggregate your predictions by patient id before measure score?</p>",
      "rawMarkdown": "Did you aggregate your predictions by patient id before measure score?",
      "votes": 1,
      "replies": [
        {
          "id": 2150599,
          "postDate": "2023-02-19T12:40:02.207Z",
          "content": "<p>Yes. I aggregate by mean.</p>",
          "rawMarkdown": "Yes. I aggregate by mean."
        }
      ]
    },
    {
      "id": 2149629,
      "postDate": "2023-02-18T14:07:59.727Z",
      "content": "<p>I have the same results, cv 0.31 but lb 0.5, and cv 0.6 but lb 0.3, so I think at the end of the competition, the rank will change a lot.</p>",
      "rawMarkdown": "I have the same results, cv 0.31 but lb 0.5, and cv 0.6 but lb 0.3, so I think at the end of the competition, the rank will change a lot.",
      "votes": 1,
      "replies": [
        {
          "id": 2149971,
          "postDate": "2023-02-18T20:35:03.053Z",
          "content": "<p>Could you tell me what was you validation loss for both the submissions. I noticed that reducing the validation loss regardless of the CV F1 score improved my performance on the LB</p>",
          "rawMarkdown": "Could you tell me what was you validation loss for both the submissions. I noticed that reducing the validation loss regardless of the CV F1 score improved my performance on the LB",
          "replies": [
            {
              "id": 2150159,
              "postDate": "2023-02-19T01:53:46.783Z",
              "content": "<p>Different loss functions usually get different loss values, but in my experiment, no linear relationship between eval loss and lb has been observed :(</p>",
              "rawMarkdown": "Different loss functions usually get different loss values, but in my experiment, no linear relationship between eval loss and lb has been observed :("
            }
          ]
        },
        {
          "id": 2150605,
          "postDate": "2023-02-19T12:42:47.827Z",
          "content": "<p>Thank you for your advice.</p>\n<p>I would choose the model with the highest CV and the model with the highest LB.</p>",
          "rawMarkdown": "Thank you for your advice.\n\nI would choose the model with the highest CV and the model with the highest LB."
        }
      ]
    },
    {
      "id": 2149727,
      "postDate": "2023-02-18T16:07:22.953Z",
      "content": "<p>This link may help you understand the situation. <a href=\"https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198</a></p>",
      "rawMarkdown": "This link may help you understand the situation. https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198",
      "votes": 2,
      "replies": [
        {
          "id": 2150590,
          "postDate": "2023-02-19T12:20:23.843Z",
          "content": "<p>Thank you for your advice.</p>\n<p>My cv strategy was different from his strategy.<br>\nI will experiment.</p>",
          "rawMarkdown": "Thank you for your advice.\n\nMy cv strategy was different from his strategy.\nI will experiment.",
          "votes": 1
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2149713,
      "author_name": "A.P.",
      "author_url": "",
      "post_date": "2023-02-18T15:41:23.690000",
      "content": "<p>Did you aggregate your predictions by patient id before measure score?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2150599,
          "author_name": "Nana-chan",
          "author_url": "",
          "post_date": "2023-02-19T12:40:02.207000",
          "content": "<p>Yes. I aggregate by mean.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2149629,
      "author_name": "Jijie Li",
      "author_url": "",
      "post_date": "2023-02-18T14:07:59.727000",
      "content": "<p>I have the same results, cv 0.31 but lb 0.5, and cv 0.6 but lb 0.3, so I think at the end of the competition, the rank will change a lot.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2149971,
          "author_name": "Naman Makkar",
          "author_url": "",
          "post_date": "2023-02-18T20:35:03.053000",
          "content": "<p>Could you tell me what was you validation loss for both the submissions. I noticed that reducing the validation loss regardless of the CV F1 score improved my performance on the LB</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2150159,
              "author_name": "Jijie Li",
              "author_url": "",
              "post_date": "2023-02-19T01:53:46.783000",
              "content": "<p>Different loss functions usually get different loss values, but in my experiment, no linear relationship between eval loss and lb has been observed :(</p>",
              "votes": 0,
              "replies": []
            }
          ]
        },
        {
          "id": 2150605,
          "author_name": "Nana-chan",
          "author_url": "",
          "post_date": "2023-02-19T12:42:47.827000",
          "content": "<p>Thank you for your advice.</p>\n<p>I would choose the model with the highest CV and the model with the highest LB.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2149727,
      "author_name": "Hyunsoo Lee 1010",
      "author_url": "",
      "post_date": "2023-02-18T16:07:22.953000",
      "content": "<p>This link may help you understand the situation. <a href=\"https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198</a></p>",
      "votes": 2,
      "replies": [
        {
          "id": 2150590,
          "author_name": "Nana-chan",
          "author_url": "",
          "post_date": "2023-02-19T12:20:23.843000",
          "content": "<p>Thank you for your advice.</p>\n<p>My cv strategy was different from his strategy.<br>\nI will experiment.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2149247": "I am suffering from the fact that CV and LB do not correlate in this competition.\n\nmy cv strategy is below. (N_FOLDS=5)\nI refer [this notebook](https://www.kaggle.com/code/vslaykovsky/train-pytorch-aux-targets-weighted-loss-thres?scriptVersionId=114303403)\n\n```python\nsplit = StratifiedGroupKFold(N_FOLDS)\nfor k, (_, test_idx) in enumerate(split.split(df_train, df_train.cancer, groups=df_train.patient_id)):\n    df_train.loc[test_idx, 'split'] = k\ndf_train.split = df_train.split.astype(int)\ndf_train.groupby('split').cancer.mean()\n```\n\nI use the code below. And I am saving the model with the largest score\n\n```python\ndef pfbeta(labels, predictions, beta=1.):\n    y_true_count = 0\n    ctp = 0\n    cfp = 0\n\n    for idx in range(len(labels)):\n        prediction = min(max(predictions[idx], 0), 1)\n        if (labels[idx]):\n            y_true_count += 1\n            ctp += prediction\n        else:\n            cfp += prediction\n\n    beta_squared = beta * beta\n    c_precision = ctp / (ctp + cfp)\n    c_recall = ctp / max(y_true_count, 1)  # avoid / 0\n    if (c_precision > 0 and c_recall > 0):\n        result = (1 + beta_squared) * (c_precision * c_recall) / (beta_squared * c_precision + c_recall)\n        return result\n    else:\n        return 0\n\ndef optimal_f1(labels, predictions):\n    thres = np.linspace(0, 1, 101)\n    f1s = [pfbeta(labels, predictions > thr) for thr in thres]\n    idx = np.argmax(f1s)\n    return f1s[idx], thres[idx]\n```\n\nThe results of my experiment are as follows\n\n| POSITIVE_TARGET_WEIGHT|AUX_LOSS_WEIGHT  |optimal_f1|LB|\n| --- | --- |--- |--- |\n| 20 | 0 |0.395 | 0.51|\n| 20 | 94 | 0.412 | 0.45|\n\nAm I doing something wrong?\nthanks.\n",
    "2149713": "Did you aggregate your predictions by patient id before measure score?",
    "2149629": "I have the same results, cv 0.31 but lb 0.5, and cv 0.6 but lb 0.3, so I think at the end of the competition, the rank will change a lot.",
    "2149727": "This link may help you understand the situation. https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/382198"
  }
}