{
  "id": 155876,
  "title": "Need for advice: random results on validation",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/155876",
  "author_name": "Dmitry A. Grechka",
  "post_date": "2020-06-03T11:09:40.577000",
  "votes": 0,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi all,</p>\n\n<p>I experience a strange phenomenon.\nStarting from some epoch the model performance on validation set becomes completely random.</p>\n\n<p>Consider the following log:</p>\n\n<p><code>\nepoch      kappa      loss    val_kappa  val_loss\n    0 0.07548171 0.9414767  0.048463047 1.2982789\n    1 0.28877169 0.8146502  0.184879363 0.9672309\n    2 0.35040712 0.7698837  0.153245032 0.8517976\n    3 0.40159923 0.7240623  0.208555162 0.9004576\n    4 0.44049644 0.6871418  0.127046525 0.8334766\n    5 0.48340845 0.6535359  0.168894529 0.8990936\n    6 0.51358300 0.6271099 -0.123218894 1.2090751\n    7 0.53834927 0.6048317 -0.116280079 1.1294516\n    8 0.55006683 0.5934765 -0.200695872 1.1731856\n    9 0.55658513 0.5901295 -0.151679993 1.1789567\n   10 0.57193851 0.5709416 -0.012553096 1.0003899\n   11 0.56137598 0.5726194 -0.006297827 1.0315917\n   12 0.61165726 0.5244491 -0.061643839 1.0894785\n   13 0.62888157 0.5081555 -0.052022219 1.0847842\n   14 0.63136476 0.5112992 -0.044588804 1.0694513\n   15 0.64067459 0.5025691 -0.082949996 1.0889540\n   16 0.63158119 0.5076009 -0.072470427 1.0855005\n   17 0.64699137 0.4930297 -0.060997248 1.0569655\n   18 0.65327001 0.4933165 -0.137813091 1.1249895\n</code></p>\n\n<p>There is a sudden bump of validation loss on epoch 6, while training loss continues to decrease smoothly.</p>\n\n<p>val loss does not look like ordinary overfitting U-curve. It just jumps around some high values.\nWhat can cause such effect? Did someone ever experience similar effect?</p>",
  "messages": [
    {
      "id": 872683,
      "postDate": "2020-06-03T12:23:39.350Z",
      "content": "<p>I think its being little overfit. You may try scale values  and smaller learning rate or control the features. maybe you can have useless features so use PCA and you may try regularization techniques(L1 regularization,L2regularization,Dropout etc.) or early-stopping.</p>",
      "rawMarkdown": "I think its being little overfit. You may try scale values  and smaller learning rate or control the features. maybe you can have useless features so use PCA and you may try regularization techniques(L1 regularization,L2regularization,Dropout etc.) or early-stopping.",
      "votes": 1,
      "replies": [
        {
          "id": 872822,
          "postDate": "2020-06-03T14:28:50.220Z",
          "content": "<p>I already have a dropout. But will try to add L2 regularization. Thanks.</p>\n\n<p>But still can't get why the val_loss jumps to high value so dramatically. At the beginning it evolves more smoothly.</p>",
          "rawMarkdown": "I already have a dropout. But will try to add L2 regularization. Thanks.\n\nBut still can't get why the val_loss jumps to high value so dramatically. At the beginning it evolves more smoothly."
        }
      ]
    },
    {
      "id": 872662,
      "postDate": "2020-06-03T12:02:51.297Z",
      "content": "<p>I experienced high fluctuations for my validation score when setting an inapropriate learning rate. I would firstly suggest to use a smaller learning rate and see what happens</p>",
      "rawMarkdown": "I experienced high fluctuations for my validation score when setting an inapropriate learning rate. I would firstly suggest to use a smaller learning rate and see what happens",
      "votes": 1,
      "replies": [
        {
          "id": 872819,
          "postDate": "2020-06-03T14:27:26.850Z",
          "content": "<p>Thanks. Will try.</p>\n\n<p>I just can't understand why it is that. It looks like the network switches to memorize some very peculiar features of training set. Why else can the the validation predictions can become so poor...</p>",
          "rawMarkdown": "Thanks. Will try.\n\nI just can't understand why it is that. It looks like the network switches to memorize some very peculiar features of training set. Why else can the the validation predictions can become so poor..."
        },
        {
          "id": 875133,
          "postDate": "2020-06-05T14:44:53.467Z",
          "content": "<p>Lower learning rate did not help. It just postponed (in terms of training epochs) the problem.</p>\n\n<p>Right now my working hypothesis is that my tiling approach generates tiles that sometimes miss relevant information. Thus the only approach for the network to learn is to memorize the training samples, not to extract relevant features. </p>",
          "rawMarkdown": "Lower learning rate did not help. It just postponed (in terms of training epochs) the problem.\n\nRight now my working hypothesis is that my tiling approach generates tiles that sometimes miss relevant information. Thus the only approach for the network to learn is to memorize the training samples, not to extract relevant features. "
        }
      ]
    },
    {
      "id": 872614,
      "postDate": "2020-06-03T11:09:40.577Z",
      "content": "<p>Hi all,</p>\n\n<p>I experience a strange phenomenon.\nStarting from some epoch the model performance on validation set becomes completely random.</p>\n\n<p>Consider the following log:</p>\n\n<p><code>\nepoch      kappa      loss    val_kappa  val_loss\n    0 0.07548171 0.9414767  0.048463047 1.2982789\n    1 0.28877169 0.8146502  0.184879363 0.9672309\n    2 0.35040712 0.7698837  0.153245032 0.8517976\n    3 0.40159923 0.7240623  0.208555162 0.9004576\n    4 0.44049644 0.6871418  0.127046525 0.8334766\n    5 0.48340845 0.6535359  0.168894529 0.8990936\n    6 0.51358300 0.6271099 -0.123218894 1.2090751\n    7 0.53834927 0.6048317 -0.116280079 1.1294516\n    8 0.55006683 0.5934765 -0.200695872 1.1731856\n    9 0.55658513 0.5901295 -0.151679993 1.1789567\n   10 0.57193851 0.5709416 -0.012553096 1.0003899\n   11 0.56137598 0.5726194 -0.006297827 1.0315917\n   12 0.61165726 0.5244491 -0.061643839 1.0894785\n   13 0.62888157 0.5081555 -0.052022219 1.0847842\n   14 0.63136476 0.5112992 -0.044588804 1.0694513\n   15 0.64067459 0.5025691 -0.082949996 1.0889540\n   16 0.63158119 0.5076009 -0.072470427 1.0855005\n   17 0.64699137 0.4930297 -0.060997248 1.0569655\n   18 0.65327001 0.4933165 -0.137813091 1.1249895\n</code></p>\n\n<p>There is a sudden bump of validation loss on epoch 6, while training loss continues to decrease smoothly.</p>\n\n<p>val loss does not look like ordinary overfitting U-curve. It just jumps around some high values.\nWhat can cause such effect? Did someone ever experience similar effect?</p>",
      "rawMarkdown": "Hi all,\n\nI experience a strange phenomenon.\nStarting from some epoch the model performance on validation set becomes completely random.\n\nConsider the following log:\n\n```\nepoch      kappa      loss    val_kappa  val_loss\n    0 0.07548171 0.9414767  0.048463047 1.2982789\n    1 0.28877169 0.8146502  0.184879363 0.9672309\n    2 0.35040712 0.7698837  0.153245032 0.8517976\n    3 0.40159923 0.7240623  0.208555162 0.9004576\n    4 0.44049644 0.6871418  0.127046525 0.8334766\n    5 0.48340845 0.6535359  0.168894529 0.8990936\n    6 0.51358300 0.6271099 -0.123218894 1.2090751\n    7 0.53834927 0.6048317 -0.116280079 1.1294516\n    8 0.55006683 0.5934765 -0.200695872 1.1731856\n    9 0.55658513 0.5901295 -0.151679993 1.1789567\n   10 0.57193851 0.5709416 -0.012553096 1.0003899\n   11 0.56137598 0.5726194 -0.006297827 1.0315917\n   12 0.61165726 0.5244491 -0.061643839 1.0894785\n   13 0.62888157 0.5081555 -0.052022219 1.0847842\n   14 0.63136476 0.5112992 -0.044588804 1.0694513\n   15 0.64067459 0.5025691 -0.082949996 1.0889540\n   16 0.63158119 0.5076009 -0.072470427 1.0855005\n   17 0.64699137 0.4930297 -0.060997248 1.0569655\n   18 0.65327001 0.4933165 -0.137813091 1.1249895\n```\n\nThere is a sudden bump of validation loss on epoch 6, while training loss continues to decrease smoothly.\n\nval loss does not look like ordinary overfitting U-curve. It just jumps around some high values.\nWhat can cause such effect? Did someone ever experience similar effect?"
    }
  ],
  "comments": [
    {
      "id": 872683,
      "author_name": "Cemhan Şenol",
      "author_url": "",
      "post_date": "2020-06-03T12:23:39.350000",
      "content": "<p>I think its being little overfit. You may try scale values  and smaller learning rate or control the features. maybe you can have useless features so use PCA and you may try regularization techniques(L1 regularization,L2regularization,Dropout etc.) or early-stopping.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 872822,
          "author_name": "Dmitry A. Grechka",
          "author_url": "",
          "post_date": "2020-06-03T14:28:50.220000",
          "content": "<p>I already have a dropout. But will try to add L2 regularization. Thanks.</p>\n\n<p>But still can't get why the val_loss jumps to high value so dramatically. At the beginning it evolves more smoothly.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 872662,
      "author_name": "Alex",
      "author_url": "",
      "post_date": "2020-06-03T12:02:51.297000",
      "content": "<p>I experienced high fluctuations for my validation score when setting an inapropriate learning rate. I would firstly suggest to use a smaller learning rate and see what happens</p>",
      "votes": 1,
      "replies": [
        {
          "id": 872819,
          "author_name": "Dmitry A. Grechka",
          "author_url": "",
          "post_date": "2020-06-03T14:27:26.850000",
          "content": "<p>Thanks. Will try.</p>\n\n<p>I just can't understand why it is that. It looks like the network switches to memorize some very peculiar features of training set. Why else can the the validation predictions can become so poor...</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 875133,
          "author_name": "Dmitry A. Grechka",
          "author_url": "",
          "post_date": "2020-06-05T14:44:53.467000",
          "content": "<p>Lower learning rate did not help. It just postponed (in terms of training epochs) the problem.</p>\n\n<p>Right now my working hypothesis is that my tiling approach generates tiles that sometimes miss relevant information. Thus the only approach for the network to learn is to memorize the training samples, not to extract relevant features. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "872683": "I think its being little overfit. You may try scale values  and smaller learning rate or control the features. maybe you can have useless features so use PCA and you may try regularization techniques(L1 regularization,L2regularization,Dropout etc.) or early-stopping.",
    "872662": "I experienced high fluctuations for my validation score when setting an inapropriate learning rate. I would firstly suggest to use a smaller learning rate and see what happens",
    "872614": "Hi all,\n\nI experience a strange phenomenon.\nStarting from some epoch the model performance on validation set becomes completely random.\n\nConsider the following log:\n\n```\nepoch      kappa      loss    val_kappa  val_loss\n    0 0.07548171 0.9414767  0.048463047 1.2982789\n    1 0.28877169 0.8146502  0.184879363 0.9672309\n    2 0.35040712 0.7698837  0.153245032 0.8517976\n    3 0.40159923 0.7240623  0.208555162 0.9004576\n    4 0.44049644 0.6871418  0.127046525 0.8334766\n    5 0.48340845 0.6535359  0.168894529 0.8990936\n    6 0.51358300 0.6271099 -0.123218894 1.2090751\n    7 0.53834927 0.6048317 -0.116280079 1.1294516\n    8 0.55006683 0.5934765 -0.200695872 1.1731856\n    9 0.55658513 0.5901295 -0.151679993 1.1789567\n   10 0.57193851 0.5709416 -0.012553096 1.0003899\n   11 0.56137598 0.5726194 -0.006297827 1.0315917\n   12 0.61165726 0.5244491 -0.061643839 1.0894785\n   13 0.62888157 0.5081555 -0.052022219 1.0847842\n   14 0.63136476 0.5112992 -0.044588804 1.0694513\n   15 0.64067459 0.5025691 -0.082949996 1.0889540\n   16 0.63158119 0.5076009 -0.072470427 1.0855005\n   17 0.64699137 0.4930297 -0.060997248 1.0569655\n   18 0.65327001 0.4933165 -0.137813091 1.1249895\n```\n\nThere is a sudden bump of validation loss on epoch 6, while training loss continues to decrease smoothly.\n\nval loss does not look like ordinary overfitting U-curve. It just jumps around some high values.\nWhat can cause such effect? Did someone ever experience similar effect?"
  }
}