{
  "id": 116073,
  "title": "How can we trust current public leaderboard in stage 2?",
  "url": "/competitions/rsna-intracranial-hemorrhage-detection/discussion/116073",
  "author_name": "Daniel",
  "post_date": "2019-11-06T23:50:13.020000",
  "votes": 2,
  "comment_count": 7,
  "views": 0,
  "content": "<p>For me, it is like, as the model is trained deeper, the public LB score turns worse. My CV looks as normal as in stage 1.</p>\n\n<p>After 1st epoch, Validation: 0.075, LB: 0.063,\nAfter 2nd epoch, Validation: 0.070, LB: 0.132,\nAfter 3rd epoch, Validation: 0.067, LB: 0.327.</p>\n\n<p>Just wondering if it occurs to everyone, or I introduced some mistakes while transferring the model into stage 2. </p>",
  "messages": [
    {
      "id": 667843,
      "postDate": "2019-11-07T18:31:02.940Z",
      "content": "<p>Thanks for your questions! The total number of samples in the public leaderboard is very small. I wouldn't draw any conclusions from it score-wise.</p>",
      "rawMarkdown": "Thanks for your questions! The total number of samples in the public leaderboard is very small. I wouldn't draw any conclusions from it score-wise.",
      "votes": 3,
      "replies": [
        {
          "id": 668784,
          "postDate": "2019-11-08T21:05:07.363Z",
          "content": "<p>Thanks, Phil. Glad to know that.</p>",
          "rawMarkdown": "Thanks, Phil. Glad to know that."
        }
      ]
    },
    {
      "id": 667373,
      "postDate": "2019-11-07T06:49:34.413Z",
      "content": "<p>IMHO, considering &lt;1% data, you may not rely on such statistics!</p>",
      "rawMarkdown": "IMHO, considering &lt;1% data, you may not rely on such statistics!",
      "votes": 3,
      "replies": [
        {
          "id": 667578,
          "postDate": "2019-11-07T12:04:24.590Z",
          "content": "<p>While that's true, now seeing what &lt;1% data does to the statistics, it is scary that many public policies are based on statistics obtained from polling less than 1% of the population.</p>",
          "rawMarkdown": "While that's true, now seeing what &lt;1% data does to the statistics, it is scary that many public policies are based on statistics obtained from polling less than 1% of the population.",
          "votes": 6
        }
      ]
    },
    {
      "id": 667446,
      "postDate": "2019-11-07T08:36:34.537Z",
      "content": "<p>don't fall in rat race trap,focus on your CV</p>",
      "rawMarkdown": "don't fall in rat race trap,focus on your CV",
      "votes": 1
    },
    {
      "id": 667435,
      "postDate": "2019-11-07T08:21:55.127Z",
      "content": "<p>As discussed in numerous places on the forum already, the purpose of stage 2 public LB is just to allow you to verify that your pipeline produces predictions in the right format so that you will actually get a score at the end on the private LB, where it matters. The score you get on the stage 2 public LB is irrelevant, as it represents an extremely small percentage of the global test set.</p>",
      "rawMarkdown": "As discussed in numerous places on the forum already, the purpose of stage 2 public LB is just to allow you to verify that your pipeline produces predictions in the right format so that you will actually get a score at the end on the private LB, where it matters. The score you get on the stage 2 public LB is irrelevant, as it represents an extremely small percentage of the global test set.",
      "votes": 1
    },
    {
      "id": 667210,
      "postDate": "2019-11-07T00:29:05.257Z",
      "content": "<p>dont trust it</p>",
      "rawMarkdown": "dont trust it",
      "votes": 1
    },
    {
      "id": 667198,
      "postDate": "2019-11-06T23:50:13.020Z",
      "content": "<p>For me, it is like, as the model is trained deeper, the public LB score turns worse. My CV looks as normal as in stage 1.</p>\n\n<p>After 1st epoch, Validation: 0.075, LB: 0.063,\nAfter 2nd epoch, Validation: 0.070, LB: 0.132,\nAfter 3rd epoch, Validation: 0.067, LB: 0.327.</p>\n\n<p>Just wondering if it occurs to everyone, or I introduced some mistakes while transferring the model into stage 2. </p>",
      "rawMarkdown": "For me, it is like, as the model is trained deeper, the public LB score turns worse. My CV looks as normal as in stage 1.\n\nAfter 1st epoch, Validation: 0.075, LB: 0.063,\nAfter 2nd epoch, Validation: 0.070, LB: 0.132,\nAfter 3rd epoch, Validation: 0.067, LB: 0.327.\n\nJust wondering if it occurs to everyone, or I introduced some mistakes while transferring the model into stage 2. ",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 667843,
      "author_name": "Phil Culliton",
      "author_url": "",
      "post_date": "2019-11-07T18:31:02.940000",
      "content": "<p>Thanks for your questions! The total number of samples in the public leaderboard is very small. I wouldn't draw any conclusions from it score-wise.</p>",
      "votes": 3,
      "replies": [
        {
          "id": 668784,
          "author_name": "Daniel",
          "author_url": "",
          "post_date": "2019-11-08T21:05:07.363000",
          "content": "<p>Thanks, Phil. Glad to know that.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 667373,
      "author_name": "Mohammad Azam Khan",
      "author_url": "",
      "post_date": "2019-11-07T06:49:34.413000",
      "content": "<p>IMHO, considering &lt;1% data, you may not rely on such statistics!</p>",
      "votes": 3,
      "replies": [
        {
          "id": 667578,
          "author_name": "Yee Ng",
          "author_url": "",
          "post_date": "2019-11-07T12:04:24.590000",
          "content": "<p>While that's true, now seeing what &lt;1% data does to the statistics, it is scary that many public policies are based on statistics obtained from polling less than 1% of the population.</p>",
          "votes": 6,
          "replies": []
        }
      ]
    },
    {
      "id": 667446,
      "author_name": "Mobassir",
      "author_url": "",
      "post_date": "2019-11-07T08:36:34.537000",
      "content": "<p>don't fall in rat race trap,focus on your CV</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 667435,
      "author_name": "juliencs",
      "author_url": "",
      "post_date": "2019-11-07T08:21:55.127000",
      "content": "<p>As discussed in numerous places on the forum already, the purpose of stage 2 public LB is just to allow you to verify that your pipeline produces predictions in the right format so that you will actually get a score at the end on the private LB, where it matters. The score you get on the stage 2 public LB is irrelevant, as it represents an extremely small percentage of the global test set.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 667210,
      "author_name": "DatNT",
      "author_url": "",
      "post_date": "2019-11-07T00:29:05.257000",
      "content": "<p>dont trust it</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "667843": "Thanks for your questions! The total number of samples in the public leaderboard is very small. I wouldn't draw any conclusions from it score-wise.",
    "667373": "IMHO, considering &lt;1% data, you may not rely on such statistics!",
    "667446": "don't fall in rat race trap,focus on your CV",
    "667435": "As discussed in numerous places on the forum already, the purpose of stage 2 public LB is just to allow you to verify that your pipeline produces predictions in the right format so that you will actually get a score at the end on the private LB, where it matters. The score you get on the stage 2 public LB is irrelevant, as it represents an extremely small percentage of the global test set.",
    "667210": "dont trust it",
    "667198": "For me, it is like, as the model is trained deeper, the public LB score turns worse. My CV looks as normal as in stage 1.\n\nAfter 1st epoch, Validation: 0.075, LB: 0.063,\nAfter 2nd epoch, Validation: 0.070, LB: 0.132,\nAfter 3rd epoch, Validation: 0.067, LB: 0.327.\n\nJust wondering if it occurs to everyone, or I introduced some mistakes while transferring the model into stage 2. "
  }
}