{
  "id": 378175,
  "title": "Model Selection and Overfitting Question",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/378175",
  "author_name": "moth",
  "post_date": "2023-01-14T16:25:53.670000",
  "votes": 0,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I have a question regarding model selection. </p>\n<pre><code>---------- Fold: 1 ----------\n0:54:37 | Epoch: 1/10 | Train Loss: 166.3 | Valid Loss: 139.6 | Train ROC-AUC: 0.6011 | Valid ROC-AUC: 0.6678\n0:54:31 | Epoch: 2/10 | Train Loss: 160.9 | Valid Loss: 150.8 | Train ROC-AUC: 0.6539 | Valid ROC-AUC: 0.6878\n0:54:50 | Epoch: 3/10 | Train Loss: 155.8 | Valid Loss: 135.3 | Train ROC-AUC: 0.6949 | Valid ROC-AUC: 0.7148 &lt;---\n0:55:03 | Epoch: 4/10 | Train Loss: 150.4 | Valid Loss: 139.2 | Train ROC-AUC: 0.7327 | Valid ROC-AUC: 0.695\n0:55:42 | Epoch: 5/10 | Train Loss: 147.0 | Valid Loss: 139.2 | Train ROC-AUC: 0.7494 | Valid ROC-AUC: 0.7256\n0:55:35 | Epoch: 6/10 | Train Loss: 143.9 | Valid Loss: 143.8 | Train ROC-AUC: 0.7617 | Valid ROC-AUC: 0.7251\n0:55:36 | Epoch: 7/10 | Train Loss: 138.2 | Valid Loss: 135.6 | Train ROC-AUC: 0.7864 | Valid ROC-AUC: 0.7411 &lt;---\n0:55:23 | Epoch: 8/10 | Train Loss: 136.0 | Valid Loss: 131.2 | Train ROC-AUC: 0.7953 | Valid ROC-AUC: 0.7254\n0:55:27 | Epoch: 9/10 | Train Loss: 131.2 | Valid Loss: 147.3 | Train ROC-AUC: 0.8121 | Valid ROC-AUC: 0.7161\nEarly stopping (no improvement since 3 models) | Best Metric: 0.7411261998943077\n</code></pre>\n<p>On epoch 3 I get a model which has a greater Validation ROC-AUC than Train ROC-AUC. However, on epoch 7 I get a model which has the greatest Validation ROC-AUC of all epochs but its corresponding Train ROC-AUC is larger, suggesting overfitting. </p>\n<p>Which model should be selected?</p>",
  "messages": [
    {
      "id": 2100098,
      "postDate": "2023-01-14T23:46:27.217Z",
      "content": "<p>The best things to monitor with respect to model performance is the training loss and the validation loss. If the training loss is decreasing for each epoch look at the validation loss trend. The word 'trend' is very important. In any given epoch the validation loss might increase, However if the trend of the validation loss is to continually increase then you are over-fitting.   In your case the validation loss is inconsistent. You might get better performance by using a lower learning rate. It is always best to use and adjustable learning rate. So I recommend you include the keras callback ReduceLROnPlateau. Documentation is at <a href=\"https://keras.io/api/callbacks/reduce_lr_on_plateau/\" target=\"_blank\">https://keras.io/api/callbacks/reduce_lr_on_plateau/</a>.  Monitor the value loss, I use a factor of ,4 and a patience of 2. Set verbose=1 so you can see when the learning rate is reduced. In your EarlyStopping  callback set parameter restore_best_weights to True. That way your model is returned with the weights from the epoch with the lowest validation loss.</p>",
      "rawMarkdown": "The best things to monitor with respect to model performance is the training loss and the validation loss. If the training loss is decreasing for each epoch look at the validation loss trend. The word 'trend' is very important. In any given epoch the validation loss might increase, However if the trend of the validation loss is to continually increase then you are over-fitting.   In your case the validation loss is inconsistent. You might get better performance by using a lower learning rate. It is always best to use and adjustable learning rate. So I recommend you include the keras callback ReduceLROnPlateau. Documentation is at https://keras.io/api/callbacks/reduce_lr_on_plateau/.  Monitor the value loss, I use a factor of ,4 and a patience of 2. Set verbose=1 so you can see when the learning rate is reduced. In your EarlyStopping  callback set parameter restore_best_weights to True. That way your model is returned with the weights from the epoch with the lowest validation loss.",
      "votes": 2
    },
    {
      "id": 2099679,
      "postDate": "2023-01-14T16:25:53.670Z",
      "content": "<p>I have a question regarding model selection. </p>\n<pre><code>---------- Fold: 1 ----------\n0:54:37 | Epoch: 1/10 | Train Loss: 166.3 | Valid Loss: 139.6 | Train ROC-AUC: 0.6011 | Valid ROC-AUC: 0.6678\n0:54:31 | Epoch: 2/10 | Train Loss: 160.9 | Valid Loss: 150.8 | Train ROC-AUC: 0.6539 | Valid ROC-AUC: 0.6878\n0:54:50 | Epoch: 3/10 | Train Loss: 155.8 | Valid Loss: 135.3 | Train ROC-AUC: 0.6949 | Valid ROC-AUC: 0.7148 &lt;---\n0:55:03 | Epoch: 4/10 | Train Loss: 150.4 | Valid Loss: 139.2 | Train ROC-AUC: 0.7327 | Valid ROC-AUC: 0.695\n0:55:42 | Epoch: 5/10 | Train Loss: 147.0 | Valid Loss: 139.2 | Train ROC-AUC: 0.7494 | Valid ROC-AUC: 0.7256\n0:55:35 | Epoch: 6/10 | Train Loss: 143.9 | Valid Loss: 143.8 | Train ROC-AUC: 0.7617 | Valid ROC-AUC: 0.7251\n0:55:36 | Epoch: 7/10 | Train Loss: 138.2 | Valid Loss: 135.6 | Train ROC-AUC: 0.7864 | Valid ROC-AUC: 0.7411 &lt;---\n0:55:23 | Epoch: 8/10 | Train Loss: 136.0 | Valid Loss: 131.2 | Train ROC-AUC: 0.7953 | Valid ROC-AUC: 0.7254\n0:55:27 | Epoch: 9/10 | Train Loss: 131.2 | Valid Loss: 147.3 | Train ROC-AUC: 0.8121 | Valid ROC-AUC: 0.7161\nEarly stopping (no improvement since 3 models) | Best Metric: 0.7411261998943077\n</code></pre>\n<p>On epoch 3 I get a model which has a greater Validation ROC-AUC than Train ROC-AUC. However, on epoch 7 I get a model which has the greatest Validation ROC-AUC of all epochs but its corresponding Train ROC-AUC is larger, suggesting overfitting. </p>\n<p>Which model should be selected?</p>",
      "rawMarkdown": "I have a question regarding model selection. \n```\n---------- Fold: 1 ----------\n0:54:37 | Epoch: 1/10 | Train Loss: 166.3 | Valid Loss: 139.6 | Train ROC-AUC: 0.6011 | Valid ROC-AUC: 0.6678\n0:54:31 | Epoch: 2/10 | Train Loss: 160.9 | Valid Loss: 150.8 | Train ROC-AUC: 0.6539 | Valid ROC-AUC: 0.6878\n0:54:50 | Epoch: 3/10 | Train Loss: 155.8 | Valid Loss: 135.3 | Train ROC-AUC: 0.6949 | Valid ROC-AUC: 0.7148 <---\n0:55:03 | Epoch: 4/10 | Train Loss: 150.4 | Valid Loss: 139.2 | Train ROC-AUC: 0.7327 | Valid ROC-AUC: 0.695\n0:55:42 | Epoch: 5/10 | Train Loss: 147.0 | Valid Loss: 139.2 | Train ROC-AUC: 0.7494 | Valid ROC-AUC: 0.7256\n0:55:35 | Epoch: 6/10 | Train Loss: 143.9 | Valid Loss: 143.8 | Train ROC-AUC: 0.7617 | Valid ROC-AUC: 0.7251\n0:55:36 | Epoch: 7/10 | Train Loss: 138.2 | Valid Loss: 135.6 | Train ROC-AUC: 0.7864 | Valid ROC-AUC: 0.7411 <---\n0:55:23 | Epoch: 8/10 | Train Loss: 136.0 | Valid Loss: 131.2 | Train ROC-AUC: 0.7953 | Valid ROC-AUC: 0.7254\n0:55:27 | Epoch: 9/10 | Train Loss: 131.2 | Valid Loss: 147.3 | Train ROC-AUC: 0.8121 | Valid ROC-AUC: 0.7161\nEarly stopping (no improvement since 3 models) | Best Metric: 0.7411261998943077\n```\nOn epoch 3 I get a model which has a greater Validation ROC-AUC than Train ROC-AUC. However, on epoch 7 I get a model which has the greatest Validation ROC-AUC of all epochs but its corresponding Train ROC-AUC is larger, suggesting overfitting. \n\nWhich model should be selected?"
    }
  ],
  "comments": [
    {
      "id": 2100098,
      "author_name": "Gerry",
      "author_url": "",
      "post_date": "2023-01-14T23:46:27.217000",
      "content": "<p>The best things to monitor with respect to model performance is the training loss and the validation loss. If the training loss is decreasing for each epoch look at the validation loss trend. The word 'trend' is very important. In any given epoch the validation loss might increase, However if the trend of the validation loss is to continually increase then you are over-fitting.   In your case the validation loss is inconsistent. You might get better performance by using a lower learning rate. It is always best to use and adjustable learning rate. So I recommend you include the keras callback ReduceLROnPlateau. Documentation is at <a href=\"https://keras.io/api/callbacks/reduce_lr_on_plateau/\" target=\"_blank\">https://keras.io/api/callbacks/reduce_lr_on_plateau/</a>.  Monitor the value loss, I use a factor of ,4 and a patience of 2. Set verbose=1 so you can see when the learning rate is reduced. In your EarlyStopping  callback set parameter restore_best_weights to True. That way your model is returned with the weights from the epoch with the lowest validation loss.</p>",
      "votes": 2,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2100098": "The best things to monitor with respect to model performance is the training loss and the validation loss. If the training loss is decreasing for each epoch look at the validation loss trend. The word 'trend' is very important. In any given epoch the validation loss might increase, However if the trend of the validation loss is to continually increase then you are over-fitting.   In your case the validation loss is inconsistent. You might get better performance by using a lower learning rate. It is always best to use and adjustable learning rate. So I recommend you include the keras callback ReduceLROnPlateau. Documentation is at https://keras.io/api/callbacks/reduce_lr_on_plateau/.  Monitor the value loss, I use a factor of ,4 and a patience of 2. Set verbose=1 so you can see when the learning rate is reduced. In your EarlyStopping  callback set parameter restore_best_weights to True. That way your model is returned with the weights from the epoch with the lowest validation loss.",
    "2099679": "I have a question regarding model selection. \n```\n---------- Fold: 1 ----------\n0:54:37 | Epoch: 1/10 | Train Loss: 166.3 | Valid Loss: 139.6 | Train ROC-AUC: 0.6011 | Valid ROC-AUC: 0.6678\n0:54:31 | Epoch: 2/10 | Train Loss: 160.9 | Valid Loss: 150.8 | Train ROC-AUC: 0.6539 | Valid ROC-AUC: 0.6878\n0:54:50 | Epoch: 3/10 | Train Loss: 155.8 | Valid Loss: 135.3 | Train ROC-AUC: 0.6949 | Valid ROC-AUC: 0.7148 <---\n0:55:03 | Epoch: 4/10 | Train Loss: 150.4 | Valid Loss: 139.2 | Train ROC-AUC: 0.7327 | Valid ROC-AUC: 0.695\n0:55:42 | Epoch: 5/10 | Train Loss: 147.0 | Valid Loss: 139.2 | Train ROC-AUC: 0.7494 | Valid ROC-AUC: 0.7256\n0:55:35 | Epoch: 6/10 | Train Loss: 143.9 | Valid Loss: 143.8 | Train ROC-AUC: 0.7617 | Valid ROC-AUC: 0.7251\n0:55:36 | Epoch: 7/10 | Train Loss: 138.2 | Valid Loss: 135.6 | Train ROC-AUC: 0.7864 | Valid ROC-AUC: 0.7411 <---\n0:55:23 | Epoch: 8/10 | Train Loss: 136.0 | Valid Loss: 131.2 | Train ROC-AUC: 0.7953 | Valid ROC-AUC: 0.7254\n0:55:27 | Epoch: 9/10 | Train Loss: 131.2 | Valid Loss: 147.3 | Train ROC-AUC: 0.8121 | Valid ROC-AUC: 0.7161\nEarly stopping (no improvement since 3 models) | Best Metric: 0.7411261998943077\n```\nOn epoch 3 I get a model which has a greater Validation ROC-AUC than Train ROC-AUC. However, on epoch 7 I get a model which has the greatest Validation ROC-AUC of all epochs but its corresponding Train ROC-AUC is larger, suggesting overfitting. \n\nWhich model should be selected?"
  }
}