{
  "id": 148181,
  "title": "Memory Limit vs Nonetype",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/148181",
  "author_name": "BlueTurtle",
  "post_date": "2020-05-03T11:55:23.777000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I  originally got the following error on Version 7 of my PANDA EDA notebook when training the model:\nAttributeError: 'NoneType' object has no attribute 'size'</p>\n\n<p>I then found a package called NoneChucks (<a href=\"https://github.com/msamogh/nonechucks\">https://github.com/msamogh/nonechucks</a>) that replaces the Dataset and DataLoader with SafeDataset and SafeDataLoader that ignores corrupt/missing images in the dataloader.</p>\n\n<p>When using the Safe variants (Version 11) my script only runs to 260 steps (2600 images) before running out of CPU memory. I am running on the GPU - which doesn't hit the limit - so I am confused. It seems to be accumulating something into memory after every 10 steps whereas the original DataLoaders did not.</p>\n\n<p>I have tried to delete any data frames, tensors, etc to free up memory after they have been used but no luck.</p>\n\n<p>Thanks,</p>\n\n<p>BT</p>",
  "messages": [
    {
      "id": 831498,
      "postDate": "2020-05-03T11:55:23.777Z",
      "content": "<p>I  originally got the following error on Version 7 of my PANDA EDA notebook when training the model:\nAttributeError: 'NoneType' object has no attribute 'size'</p>\n\n<p>I then found a package called NoneChucks (<a href=\"https://github.com/msamogh/nonechucks\">https://github.com/msamogh/nonechucks</a>) that replaces the Dataset and DataLoader with SafeDataset and SafeDataLoader that ignores corrupt/missing images in the dataloader.</p>\n\n<p>When using the Safe variants (Version 11) my script only runs to 260 steps (2600 images) before running out of CPU memory. I am running on the GPU - which doesn't hit the limit - so I am confused. It seems to be accumulating something into memory after every 10 steps whereas the original DataLoaders did not.</p>\n\n<p>I have tried to delete any data frames, tensors, etc to free up memory after they have been used but no luck.</p>\n\n<p>Thanks,</p>\n\n<p>BT</p>",
      "rawMarkdown": "I  originally got the following error on Version 7 of my PANDA EDA notebook when training the model:\nAttributeError: 'NoneType' object has no attribute 'size'\n\nI then found a package called NoneChucks (https://github.com/msamogh/nonechucks) that replaces the Dataset and DataLoader with SafeDataset and SafeDataLoader that ignores corrupt/missing images in the dataloader.\n\nWhen using the Safe variants (Version 11) my script only runs to 260 steps (2600 images) before running out of CPU memory. I am running on the GPU - which doesn't hit the limit - so I am confused. It seems to be accumulating something into memory after every 10 steps whereas the original DataLoaders did not.\n\nI have tried to delete any data frames, tensors, etc to free up memory after they have been used but no luck.\n\nThanks,\n\nBT"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "831498": "I  originally got the following error on Version 7 of my PANDA EDA notebook when training the model:\nAttributeError: 'NoneType' object has no attribute 'size'\n\nI then found a package called NoneChucks (https://github.com/msamogh/nonechucks) that replaces the Dataset and DataLoader with SafeDataset and SafeDataLoader that ignores corrupt/missing images in the dataloader.\n\nWhen using the Safe variants (Version 11) my script only runs to 260 steps (2600 images) before running out of CPU memory. I am running on the GPU - which doesn't hit the limit - so I am confused. It seems to be accumulating something into memory after every 10 steps whereas the original DataLoaders did not.\n\nI have tried to delete any data frames, tensors, etc to free up memory after they have been used but no luck.\n\nThanks,\n\nBT"
  }
}