{
  "id": 155208,
  "title": "Notebook fails on submission run",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/155208",
  "author_name": "dymok",
  "post_date": "2020-05-31T18:30:13.960000",
  "votes": 0,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Guys, I'm facing the problem 'Submission file not found'. My notebook runs perfectly on training data and outputs the desired submission file, no memory overflow or other errors. However, when submitting, the notebook fails after a few hours. I'm saving preprocessed images to the working directory during training, and I clean it right before the prediction step. I tried to lower the batch size number in case of memory overflow in submission worker from original 64 to 32, 16, 8(although batch size of 8 produced terrible accuracy). PLEASE HELP, how can I fix it?</p>",
  "messages": [
    {
      "id": 869343,
      "postDate": "2020-06-01T00:38:21.953Z",
      "content": "<p>We can't say much from this alone, but many of us had this problem as well.\nIn my case, it was caused by a very silly issue, which you might check as well.</p>\n\n<p>Are you trying to access any of the following columns in the <strong>test.csv</strong> at inference time?\n isup_grade,     </p>\n\n<p>data_provider,    </p>\n\n<p>Gleason score?   </p>\n\n<p>If so, these are not available and that might be the issue. This would explain why you are able to run it on training data, but not on test data, as only image_id is available. </p>",
      "rawMarkdown": "We can't say much from this alone, but many of us had this problem as well.\nIn my case, it was caused by a very silly issue, which you might check as well.\n\nAre you trying to access any of the following columns in the **test.csv** at inference time?\n isup_grade,     \n\ndata_provider,    \n\nGleason score?   \n\n\nIf so, these are not available and that might be the issue. This would explain why you are able to run it on training data, but not on test data, as only image_id is available. ",
      "votes": 1,
      "replies": [
        {
          "id": 869565,
          "postDate": "2020-06-01T05:34:32.563Z",
          "content": "<p>Oh dear, I always read <strong>sample_submission.csv</strong> file with pandas to get ids. Do I have to read <strong>test.csv</strong> file on submission run? I thought these two are identical and I might use both..</p>",
          "rawMarkdown": "Oh dear, I always read **sample_submission.csv** file with pandas to get ids. Do I have to read **test.csv** file on submission run? I thought these two are identical and I might use both..",
          "votes": 1
        },
        {
          "id": 869830,
          "postDate": "2020-06-01T10:06:03.127Z",
          "content": "<p>You have to read in the test csv (named test. Csv)  during inference time, and create your own submission.csv from it.</p>\n\n<p>I admit it's really confusing because the test csv has only three rows when you run and commit the kernel. But when you make a submission, you will access the actual test csv which has more rows.</p>\n\n<p>Let me know if this solved it</p>",
          "rawMarkdown": "You have to read in the test csv (named test. Csv)  during inference time, and create your own submission.csv from it.\n\nI admit it's really confusing because the test csv has only three rows when you run and commit the kernel. But when you make a submission, you will access the actual test csv which has more rows.\n\nLet me know if this solved it",
          "votes": 1
        },
        {
          "id": 871124,
          "postDate": "2020-06-02T06:49:00.900Z",
          "content": "<p>Yes, thank you, it worked. The error message dissappeared.</p>",
          "rawMarkdown": "Yes, thank you, it worked. The error message dissappeared."
        }
      ]
    },
    {
      "id": 869099,
      "postDate": "2020-05-31T18:30:13.960Z",
      "content": "<p>Guys, I'm facing the problem 'Submission file not found'. My notebook runs perfectly on training data and outputs the desired submission file, no memory overflow or other errors. However, when submitting, the notebook fails after a few hours. I'm saving preprocessed images to the working directory during training, and I clean it right before the prediction step. I tried to lower the batch size number in case of memory overflow in submission worker from original 64 to 32, 16, 8(although batch size of 8 produced terrible accuracy). PLEASE HELP, how can I fix it?</p>",
      "rawMarkdown": "Guys, I'm facing the problem 'Submission file not found'. My notebook runs perfectly on training data and outputs the desired submission file, no memory overflow or other errors. However, when submitting, the notebook fails after a few hours. I'm saving preprocessed images to the working directory during training, and I clean it right before the prediction step. I tried to lower the batch size number in case of memory overflow in submission worker from original 64 to 32, 16, 8(although batch size of 8 produced terrible accuracy). PLEASE HELP, how can I fix it?"
    }
  ],
  "comments": [
    {
      "id": 869343,
      "author_name": "Stephan",
      "author_url": "",
      "post_date": "2020-06-01T00:38:21.953000",
      "content": "<p>We can't say much from this alone, but many of us had this problem as well.\nIn my case, it was caused by a very silly issue, which you might check as well.</p>\n\n<p>Are you trying to access any of the following columns in the <strong>test.csv</strong> at inference time?\n isup_grade,     </p>\n\n<p>data_provider,    </p>\n\n<p>Gleason score?   </p>\n\n<p>If so, these are not available and that might be the issue. This would explain why you are able to run it on training data, but not on test data, as only image_id is available. </p>",
      "votes": 1,
      "replies": [
        {
          "id": 869565,
          "author_name": "dymok",
          "author_url": "",
          "post_date": "2020-06-01T05:34:32.563000",
          "content": "<p>Oh dear, I always read <strong>sample_submission.csv</strong> file with pandas to get ids. Do I have to read <strong>test.csv</strong> file on submission run? I thought these two are identical and I might use both..</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 869830,
          "author_name": "Stephan",
          "author_url": "",
          "post_date": "2020-06-01T10:06:03.127000",
          "content": "<p>You have to read in the test csv (named test. Csv)  during inference time, and create your own submission.csv from it.</p>\n\n<p>I admit it's really confusing because the test csv has only three rows when you run and commit the kernel. But when you make a submission, you will access the actual test csv which has more rows.</p>\n\n<p>Let me know if this solved it</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 871124,
          "author_name": "dymok",
          "author_url": "",
          "post_date": "2020-06-02T06:49:00.900000",
          "content": "<p>Yes, thank you, it worked. The error message dissappeared.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "869343": "We can't say much from this alone, but many of us had this problem as well.\nIn my case, it was caused by a very silly issue, which you might check as well.\n\nAre you trying to access any of the following columns in the **test.csv** at inference time?\n isup_grade,     \n\ndata_provider,    \n\nGleason score?   \n\n\nIf so, these are not available and that might be the issue. This would explain why you are able to run it on training data, but not on test data, as only image_id is available. ",
    "869099": "Guys, I'm facing the problem 'Submission file not found'. My notebook runs perfectly on training data and outputs the desired submission file, no memory overflow or other errors. However, when submitting, the notebook fails after a few hours. I'm saving preprocessed images to the working directory during training, and I clean it right before the prediction step. I tried to lower the batch size number in case of memory overflow in submission worker from original 64 to 32, 16, 8(although batch size of 8 produced terrible accuracy). PLEASE HELP, how can I fix it?"
  }
}