{
  "id": 190798,
  "title": "[Solved -> OOM] - Submission CSV Not Found when setting the random seed?",
  "url": "/competitions/rsna-str-pulmonary-embolism-detection/discussion/190798",
  "author_name": "Kirderf",
  "post_date": "2020-10-13T11:49:40.297000",
  "votes": 2,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Today I submitted the exact same code as yesterday but today with setting the random seed.<br>\nSEED = 42<br>\ndef seed_everything(seed):<br>\n    random.seed(seed)<br>\n    os.environ['PYTHONHASHSEED'] = str(seed)<br>\n    np.random.seed(seed)<br>\n    torch.manual_seed(seed)<br>\n    torch.cuda.manual_seed(seed)<br>\n    torch.backends.cudnn.deterministic = True<br>\n    torch.backends.cudnn.benchmark = True<br>\nseed_everything(SEED)</p>\n<p>Got Submission CSV Not Found after 4-5 hour, around the same time it was finished successfully yesterday, for all my submissions, coincidence, underlying engine problem or due to the random seed?</p>",
  "messages": [
    {
      "id": 1048333,
      "postDate": "2020-10-13T11:49:40.297Z",
      "content": "<p>Today I submitted the exact same code as yesterday but today with setting the random seed.<br>\nSEED = 42<br>\ndef seed_everything(seed):<br>\n    random.seed(seed)<br>\n    os.environ['PYTHONHASHSEED'] = str(seed)<br>\n    np.random.seed(seed)<br>\n    torch.manual_seed(seed)<br>\n    torch.cuda.manual_seed(seed)<br>\n    torch.backends.cudnn.deterministic = True<br>\n    torch.backends.cudnn.benchmark = True<br>\nseed_everything(SEED)</p>\n<p>Got Submission CSV Not Found after 4-5 hour, around the same time it was finished successfully yesterday, for all my submissions, coincidence, underlying engine problem or due to the random seed?</p>",
      "rawMarkdown": "Today I submitted the exact same code as yesterday but today with setting the random seed.\nSEED = 42\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\nseed_everything(SEED)\n\nGot Submission CSV Not Found after 4-5 hour, around the same time it was finished successfully yesterday, for all my submissions, coincidence, underlying engine problem or due to the random seed?",
      "votes": 2
    },
    {
      "id": 1050464,
      "postDate": "2020-10-15T12:45:41.890Z",
      "content": "<p>Solved - seems to be OOM, thought \"Notebook Exceeded Allowed Compute\" is used for OOM, well well…now I know.</p>",
      "rawMarkdown": "Solved - seems to be OOM, thought \"Notebook Exceeded Allowed Compute\" is used for OOM, well well...now I know.",
      "votes": 1
    },
    {
      "id": 1048756,
      "postDate": "2020-10-13T19:09:24.677Z",
      "content": "<p>I think that the reason is that your notebook barely makes to 9 hours on the inference phase. Sometimes it finishes in time, sometimes not. You should try to speed your code up.</p>",
      "rawMarkdown": "I think that the reason is that your notebook barely makes to 9 hours on the inference phase. Sometimes it finishes in time, sometimes not. You should try to speed your code up.",
      "replies": [
        {
          "id": 1048759,
          "postDate": "2020-10-13T19:15:30.047Z",
          "content": "<p>Yes I know the limit, my kernels takes 4-5 hours.<br>\nLike I wrote:</p>\n<blockquote>\n  <p>Got Submission CSV Not Found after 4-5 hour, around the same time it was finished successfully yesterday</p>\n</blockquote>\n<p>Any other ideas?</p>",
          "rawMarkdown": "Yes I know the limit, my kernels takes 4-5 hours.\nLike I wrote:\n> Got Submission CSV Not Found after 4-5 hour, around the same time it was finished successfully yesterday\n\nAny other ideas?"
        },
        {
          "id": 1048763,
          "postDate": "2020-10-13T19:19:57.980Z",
          "content": "<p>I think 4-5 hours is too much. Remember that private test set is 3x larger than public.</p>",
          "rawMarkdown": "I think 4-5 hours is too much. Remember that private test set is 3x larger than public."
        },
        {
          "id": 1048798,
          "postDate": "2020-10-13T19:38:24.273Z",
          "content": "<p>It runs the whole set within that hours, so isn't the case, but thank you.</p>",
          "rawMarkdown": "It runs the whole set within that hours, so isn't the case, but thank you."
        },
        {
          "id": 1048810,
          "postDate": "2020-10-13T19:51:08.387Z",
          "content": "<p>I see. Have you installed GDCM in your notebook?</p>",
          "rawMarkdown": "I see. Have you installed GDCM in your notebook?"
        },
        {
          "id": 1048830,
          "postDate": "2020-10-13T20:20:21.257Z",
          "content": "<p>Yes and it have worked in many days, just today with random seed set it didn't</p>",
          "rawMarkdown": "Yes and it have worked in many days, just today with random seed set it didn't"
        }
      ]
    },
    {
      "id": 1048513,
      "postDate": "2020-10-13T15:05:32.270Z",
      "content": "<p>Same thing happened to me too, all the training time was wasted just because one data frame was missing.</p>",
      "rawMarkdown": "Same thing happened to me too, all the training time was wasted just because one data frame was missing."
    },
    {
      "id": 1048345,
      "postDate": "2020-10-13T12:05:13.847Z",
      "content": "<p>I am facing the same issue in one of my notebook. I was doing inference with some other model and now I changed my model and everything else is the same as before and now I am getting submission CSV not an error. I have no idea what is wrong now. LOL</p>",
      "rawMarkdown": "I am facing the same issue in one of my notebook. I was doing inference with some other model and now I changed my model and everything else is the same as before and now I am getting submission CSV not an error. I have no idea what is wrong now. LOL"
    }
  ],
  "comments": [
    {
      "id": 1050464,
      "author_name": "Kirderf",
      "author_url": "",
      "post_date": "2020-10-15T12:45:41.890000",
      "content": "<p>Solved - seems to be OOM, thought \"Notebook Exceeded Allowed Compute\" is used for OOM, well well…now I know.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1048756,
      "author_name": "Araik Tamazian",
      "author_url": "",
      "post_date": "2020-10-13T19:09:24.677000",
      "content": "<p>I think that the reason is that your notebook barely makes to 9 hours on the inference phase. Sometimes it finishes in time, sometimes not. You should try to speed your code up.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1048759,
          "author_name": "Kirderf",
          "author_url": "",
          "post_date": "2020-10-13T19:15:30.047000",
          "content": "<p>Yes I know the limit, my kernels takes 4-5 hours.<br>\nLike I wrote:</p>\n<blockquote>\n  <p>Got Submission CSV Not Found after 4-5 hour, around the same time it was finished successfully yesterday</p>\n</blockquote>\n<p>Any other ideas?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1048763,
          "author_name": "Araik Tamazian",
          "author_url": "",
          "post_date": "2020-10-13T19:19:57.980000",
          "content": "<p>I think 4-5 hours is too much. Remember that private test set is 3x larger than public.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1048798,
          "author_name": "Kirderf",
          "author_url": "",
          "post_date": "2020-10-13T19:38:24.273000",
          "content": "<p>It runs the whole set within that hours, so isn't the case, but thank you.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1048810,
          "author_name": "Araik Tamazian",
          "author_url": "",
          "post_date": "2020-10-13T19:51:08.387000",
          "content": "<p>I see. Have you installed GDCM in your notebook?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1048830,
          "author_name": "Kirderf",
          "author_url": "",
          "post_date": "2020-10-13T20:20:21.257000",
          "content": "<p>Yes and it have worked in many days, just today with random seed set it didn't</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1048513,
      "author_name": "ilovepotatoes",
      "author_url": "",
      "post_date": "2020-10-13T15:05:32.270000",
      "content": "<p>Same thing happened to me too, all the training time was wasted just because one data frame was missing.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1048345,
      "author_name": "Urvish",
      "author_url": "",
      "post_date": "2020-10-13T12:05:13.847000",
      "content": "<p>I am facing the same issue in one of my notebook. I was doing inference with some other model and now I changed my model and everything else is the same as before and now I am getting submission CSV not an error. I have no idea what is wrong now. LOL</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1048333": "Today I submitted the exact same code as yesterday but today with setting the random seed.\nSEED = 42\ndef seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\nseed_everything(SEED)\n\nGot Submission CSV Not Found after 4-5 hour, around the same time it was finished successfully yesterday, for all my submissions, coincidence, underlying engine problem or due to the random seed?",
    "1050464": "Solved - seems to be OOM, thought \"Notebook Exceeded Allowed Compute\" is used for OOM, well well...now I know.",
    "1048756": "I think that the reason is that your notebook barely makes to 9 hours on the inference phase. Sometimes it finishes in time, sometimes not. You should try to speed your code up.",
    "1048513": "Same thing happened to me too, all the training time was wasted just because one data frame was missing.",
    "1048345": "I am facing the same issue in one of my notebook. I was doing inference with some other model and now I changed my model and everything else is the same as before and now I am getting submission CSV not an error. I have no idea what is wrong now. LOL"
  }
}