{
  "id": 352354,
  "title": "Kaggle Notebook Memory Run-Out Error",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/352354",
  "author_name": "Collide_Conquer_19",
  "post_date": "2022-09-14T04:58:31.032000",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello all,</p>\n<p>I have been trying to experiment using different pre-trained models for training it's showing memory error and also i have seen many of the people are training in local setup and inferencing here. So are you people downloading the entire dataset locally if so, the data is huge and it's probable that it shows network error when downloading the data. so, how you people are managing these criterion and building models, help me to get the solution! </p>\n<p>Thanks,<br>\nConquer_19.</p>",
  "messages": [
    {
      "id": 1938297,
      "postDate": "2022-09-14T04:58:31.033Z",
      "content": "<p>Hello all,</p>\n<p>I have been trying to experiment using different pre-trained models for training it's showing memory error and also i have seen many of the people are training in local setup and inferencing here. So are you people downloading the entire dataset locally if so, the data is huge and it's probable that it shows network error when downloading the data. so, how you people are managing these criterion and building models, help me to get the solution! </p>\n<p>Thanks,<br>\nConquer_19.</p>",
      "rawMarkdown": "Hello all,\n\nI have been trying to experiment using different pre-trained models for training it's showing memory error and also i have seen many of the people are training in local setup and inferencing here. So are you people downloading the entire dataset locally if so, the data is huge and it's probable that it shows network error when downloading the data. so, how you people are managing these criterion and building models, help me to get the solution! \n\nThanks,\nConquer_19.",
      "votes": 2
    },
    {
      "id": 1939799,
      "postDate": "2022-09-15T02:35:21.303Z",
      "content": "<p>It is a very large data set.   It took probably over 10 hours to extract it from the zip.  I do have it locally on an external hard drive.  I tried to open the images on the kaggle notebook and it ran out of memory.  I am still working on my model, but what I can say is that you can only open a small batch of images at a time.  I am decoding those into tensors of pixel data. Hope this helps somewhat.  </p>\n<p>Here is a post that I got an answer on when I ran out of memory:<br>\n<a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350266\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350266</a></p>",
      "rawMarkdown": "It is a very large data set.   It took probably over 10 hours to extract it from the zip.  I do have it locally on an external hard drive.  I tried to open the images on the kaggle notebook and it ran out of memory.  I am still working on my model, but what I can say is that you can only open a small batch of images at a time.  I am decoding those into tensors of pixel data. Hope this helps somewhat.  \n\nHere is a post that I got an answer on when I ran out of memory:\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350266",
      "replies": [
        {
          "id": 1940242,
          "postDate": "2022-09-15T08:44:37.373Z",
          "content": "<p>Thanks for your help <a href=\"https://www.kaggle.com/cbarb15\" target=\"_blank\">@cbarb15</a> </p>",
          "rawMarkdown": "Thanks for your help @cbarb15 "
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1939799,
      "author_name": "cbarb15",
      "author_url": "",
      "post_date": "2022-09-15T02:35:21.303000",
      "content": "<p>It is a very large data set.   It took probably over 10 hours to extract it from the zip.  I do have it locally on an external hard drive.  I tried to open the images on the kaggle notebook and it ran out of memory.  I am still working on my model, but what I can say is that you can only open a small batch of images at a time.  I am decoding those into tensors of pixel data. Hope this helps somewhat.  </p>\n<p>Here is a post that I got an answer on when I ran out of memory:<br>\n<a href=\"https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350266\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350266</a></p>",
      "votes": 0,
      "replies": [
        {
          "id": 1940242,
          "author_name": "Collide_Conquer_19",
          "author_url": "",
          "post_date": "2022-09-15T08:44:37.373000",
          "content": "<p>Thanks for your help <a href=\"https://www.kaggle.com/cbarb15\" target=\"_blank\">@cbarb15</a> </p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1938297": "Hello all,\n\nI have been trying to experiment using different pre-trained models for training it's showing memory error and also i have seen many of the people are training in local setup and inferencing here. So are you people downloading the entire dataset locally if so, the data is huge and it's probable that it shows network error when downloading the data. so, how you people are managing these criterion and building models, help me to get the solution! \n\nThanks,\nConquer_19.",
    "1939799": "It is a very large data set.   It took probably over 10 hours to extract it from the zip.  I do have it locally on an external hard drive.  I tried to open the images on the kaggle notebook and it ran out of memory.  I am still working on my model, but what I can say is that you can only open a small batch of images at a time.  I am decoding those into tensors of pixel data. Hope this helps somewhat.  \n\nHere is a post that I got an answer on when I ran out of memory:\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350266"
  }
}