{
  "id": 430124,
  "title": "how to handle huge data set?",
  "url": "/competitions/rsna-2023-abdominal-trauma-detection/discussion/430124",
  "author_name": "Alok Kumar",
  "post_date": "2023-08-08T10:56:07.330000",
  "votes": 3,
  "comment_count": 1,
  "views": 0,
  "content": "<p>New to kaggle space. Anyone has an idea how to download/train/use data set of such a humongous size?</p>",
  "messages": [
    {
      "id": 2379924,
      "postDate": "2023-08-08T10:56:07.330Z",
      "content": "<p>New to kaggle space. Anyone has an idea how to download/train/use data set of such a humongous size?</p>",
      "rawMarkdown": "New to kaggle space. Anyone has an idea how to download/train/use data set of such a humongous size?",
      "votes": 3
    },
    {
      "id": 2380917,
      "postDate": "2023-08-08T20:44:54.623Z",
      "content": "<p>You can try to divide the dataset into different splits and use the splits to download and train the model. Using all of the data at stretch is always gonna be an issue of computational resources. There is no roundabout there. You can obviously use smaller batch sizes to fit them into memory however you should also keep in mind about the longer training time. </p>",
      "rawMarkdown": "You can try to divide the dataset into different splits and use the splits to download and train the model. Using all of the data at stretch is always gonna be an issue of computational resources. There is no roundabout there. You can obviously use smaller batch sizes to fit them into memory however you should also keep in mind about the longer training time. "
    }
  ],
  "comments": [
    {
      "id": 2380917,
      "author_name": "Harshanand",
      "author_url": "",
      "post_date": "2023-08-08T20:44:54.623000",
      "content": "<p>You can try to divide the dataset into different splits and use the splits to download and train the model. Using all of the data at stretch is always gonna be an issue of computational resources. There is no roundabout there. You can obviously use smaller batch sizes to fit them into memory however you should also keep in mind about the longer training time. </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2379924": "New to kaggle space. Anyone has an idea how to download/train/use data set of such a humongous size?",
    "2380917": "You can try to divide the dataset into different splits and use the splits to download and train the model. Using all of the data at stretch is always gonna be an issue of computational resources. There is no roundabout there. You can obviously use smaller batch sizes to fit them into memory however you should also keep in mind about the longer training time. "
  }
}