{
  "id": 379274,
  "title": "AUC and eval loss",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/379274",
  "author_name": "nicehzj",
  "post_date": "2023-01-19T00:43:07.201000",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>When I did the training, I found the AUC is decreasing with the eval loss.<br>\nI don't know what this means. And how can I improve this?</p>\n<p>Thanks for your help!<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5125958%2Ffad362b01b996ab31066a67e68c9d99e%2F_20230119014238.png?generation=1674088981356293&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": 2106083,
      "postDate": "2023-01-19T00:43:07.200Z",
      "content": "<p>When I did the training, I found the AUC is decreasing with the eval loss.<br>\nI don't know what this means. And how can I improve this?</p>\n<p>Thanks for your help!<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5125958%2Ffad362b01b996ab31066a67e68c9d99e%2F_20230119014238.png?generation=1674088981356293&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "When I did the training, I found the AUC is decreasing with the eval loss.\nI don't know what this means. And how can I improve this?\n\nThanks for your help!\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5125958%2Ffad362b01b996ab31066a67e68c9d99e%2F_20230119014238.png?generation=1674088981356293&alt=media)",
      "votes": 1
    },
    {
      "id": 2107559,
      "postDate": "2023-01-19T21:24:59.007Z",
      "content": "<p>I would interpret it the following:</p>\n<p>At the start of your training your model is getting big losses on few samples (higher AUC, high loss) after training a couple of epochs its getting tiny losses on a lot of samples leading to a decrease in AUC but also to a decrease in loss. You can find a nice general dicussion about it here: <a href=\"https://datascience.stackexchange.com/questions/42599/what-is-the-relationship-between-the-accuracy-and-the-loss-in-deep-learning/42611\" target=\"_blank\">https://datascience.stackexchange.com/questions/42599/what-is-the-relationship-between-the-accuracy-and-the-loss-in-deep-learning/42611</a></p>\n<p>Due to your decreasing eval_loss which does not seem like overfitting, you could try a more complex model e.g. EF-B1, EF-B2, etc. and see how these behave in terms of AUC. Also note that your loss dropped by more than 1 while your AUC dropped by \"only\" 0.06 which is a great trade-off.</p>",
      "rawMarkdown": "I would interpret it the following:\n\nAt the start of your training your model is getting big losses on few samples (higher AUC, high loss) after training a couple of epochs its getting tiny losses on a lot of samples leading to a decrease in AUC but also to a decrease in loss. You can find a nice general dicussion about it here: https://datascience.stackexchange.com/questions/42599/what-is-the-relationship-between-the-accuracy-and-the-loss-in-deep-learning/42611\n\nDue to your decreasing eval_loss which does not seem like overfitting, you could try a more complex model e.g. EF-B1, EF-B2, etc. and see how these behave in terms of AUC. Also note that your loss dropped by more than 1 while your AUC dropped by \"only\" 0.06 which is a great trade-off.",
      "votes": 2,
      "replies": [
        {
          "id": 2107660,
          "postDate": "2023-01-19T23:57:52.890Z",
          "content": "<p>Thank you so much for your help!</p>",
          "rawMarkdown": "Thank you so much for your help!"
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2107559,
      "author_name": "Ali Abdin",
      "author_url": "",
      "post_date": "2023-01-19T21:24:59.007000",
      "content": "<p>I would interpret it the following:</p>\n<p>At the start of your training your model is getting big losses on few samples (higher AUC, high loss) after training a couple of epochs its getting tiny losses on a lot of samples leading to a decrease in AUC but also to a decrease in loss. You can find a nice general dicussion about it here: <a href=\"https://datascience.stackexchange.com/questions/42599/what-is-the-relationship-between-the-accuracy-and-the-loss-in-deep-learning/42611\" target=\"_blank\">https://datascience.stackexchange.com/questions/42599/what-is-the-relationship-between-the-accuracy-and-the-loss-in-deep-learning/42611</a></p>\n<p>Due to your decreasing eval_loss which does not seem like overfitting, you could try a more complex model e.g. EF-B1, EF-B2, etc. and see how these behave in terms of AUC. Also note that your loss dropped by more than 1 while your AUC dropped by \"only\" 0.06 which is a great trade-off.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2107660,
          "author_name": "nicehzj",
          "author_url": "",
          "post_date": "2023-01-19T23:57:52.890000",
          "content": "<p>Thank you so much for your help!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2106083": "When I did the training, I found the AUC is decreasing with the eval loss.\nI don't know what this means. And how can I improve this?\n\nThanks for your help!\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5125958%2Ffad362b01b996ab31066a67e68c9d99e%2F_20230119014238.png?generation=1674088981356293&alt=media)",
    "2107559": "I would interpret it the following:\n\nAt the start of your training your model is getting big losses on few samples (higher AUC, high loss) after training a couple of epochs its getting tiny losses on a lot of samples leading to a decrease in AUC but also to a decrease in loss. You can find a nice general dicussion about it here: https://datascience.stackexchange.com/questions/42599/what-is-the-relationship-between-the-accuracy-and-the-loss-in-deep-learning/42611\n\nDue to your decreasing eval_loss which does not seem like overfitting, you could try a more complex model e.g. EF-B1, EF-B2, etc. and see how these behave in terms of AUC. Also note that your loss dropped by more than 1 while your AUC dropped by \"only\" 0.06 which is a great trade-off."
  }
}