{
  "id": 448198,
  "title": "How to handle big data?",
  "url": "/competitions/UBC-OCEAN/discussion/448198",
  "author_name": "Intern_grind",
  "post_date": "2023-10-18T17:33:56.780000",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I am new to Kaggle and this is my first time working on a competition having dataset of 800gb.</p>\n<p>I usually work on my local enviornment or server, and both do not have the capacity of 800gb size. </p>\n<p>What are the alternative ways to procced in this??</p>\n<p>One solution can be to use Kaggle notebooks but they usually crash when applied large neural networks as they also have memory limitation?</p>\n<p>Can anyone guide me on how to tackle this situation??</p>",
  "messages": [
    {
      "id": 2491093,
      "postDate": "2023-10-21T10:23:12.893Z",
      "content": "<p>This notebook might help: <a href=\"https://www.kaggle.com/code/aditimutha10/ubc-eda\" target=\"_blank\">https://www.kaggle.com/code/aditimutha10/ubc-eda</a></p>",
      "rawMarkdown": "This notebook might help: https://www.kaggle.com/code/aditimutha10/ubc-eda",
      "votes": 1
    },
    {
      "id": 2487641,
      "postDate": "2023-10-18T17:33:56.780Z",
      "content": "<p>I am new to Kaggle and this is my first time working on a competition having dataset of 800gb.</p>\n<p>I usually work on my local enviornment or server, and both do not have the capacity of 800gb size. </p>\n<p>What are the alternative ways to procced in this??</p>\n<p>One solution can be to use Kaggle notebooks but they usually crash when applied large neural networks as they also have memory limitation?</p>\n<p>Can anyone guide me on how to tackle this situation??</p>",
      "rawMarkdown": "I am new to Kaggle and this is my first time working on a competition having dataset of 800gb.\n\nI usually work on my local enviornment or server, and both do not have the capacity of 800gb size. \n\nWhat are the alternative ways to procced in this??\n\nOne solution can be to use Kaggle notebooks but they usually crash when applied large neural networks as they also have memory limitation?\n\nCan anyone guide me on how to tackle this situation??",
      "votes": 2
    },
    {
      "id": 2488536,
      "postDate": "2023-10-19T10:07:48.070Z",
      "content": "<p>I am facing this problem too.This is my first time to join a competition in kaggle.<br>\nMy solution is renting a server with RTX4090 and a really large disk!</p>",
      "rawMarkdown": "I am facing this problem too.This is my first time to join a competition in kaggle.\nMy solution is renting a server with RTX4090 and a really large disk!"
    },
    {
      "id": 2488119,
      "postDate": "2023-10-19T03:40:10.573Z",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/shivyanshgarg\" target=\"_blank\">@shivyanshgarg</a> ,<br>\nIn such big-data tasks you usually don't have to load all the data at once. There are lots of ways to train models with batches loading from disk, not from ram.</p>",
      "rawMarkdown": "Hey @shivyanshgarg ,\nIn such big-data tasks you usually don't have to load all the data at once. There are lots of ways to train models with batches loading from disk, not from ram."
    }
  ],
  "comments": [
    {
      "id": 2491093,
      "author_name": "Aditi Mutha",
      "author_url": "",
      "post_date": "2023-10-21T10:23:12.893000",
      "content": "<p>This notebook might help: <a href=\"https://www.kaggle.com/code/aditimutha10/ubc-eda\" target=\"_blank\">https://www.kaggle.com/code/aditimutha10/ubc-eda</a></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2488536,
      "author_name": "LuoZiqian",
      "author_url": "",
      "post_date": "2023-10-19T10:07:48.070000",
      "content": "<p>I am facing this problem too.This is my first time to join a competition in kaggle.<br>\nMy solution is renting a server with RTX4090 and a really large disk!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2488119,
      "author_name": "georfed",
      "author_url": "",
      "post_date": "2023-10-19T03:40:10.573000",
      "content": "<p>Hey <a href=\"https://www.kaggle.com/shivyanshgarg\" target=\"_blank\">@shivyanshgarg</a> ,<br>\nIn such big-data tasks you usually don't have to load all the data at once. There are lots of ways to train models with batches loading from disk, not from ram.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2491093": "This notebook might help: https://www.kaggle.com/code/aditimutha10/ubc-eda",
    "2487641": "I am new to Kaggle and this is my first time working on a competition having dataset of 800gb.\n\nI usually work on my local enviornment or server, and both do not have the capacity of 800gb size. \n\nWhat are the alternative ways to procced in this??\n\nOne solution can be to use Kaggle notebooks but they usually crash when applied large neural networks as they also have memory limitation?\n\nCan anyone guide me on how to tackle this situation??",
    "2488536": "I am facing this problem too.This is my first time to join a competition in kaggle.\nMy solution is renting a server with RTX4090 and a really large disk!",
    "2488119": "Hey @shivyanshgarg ,\nIn such big-data tasks you usually don't have to load all the data at once. There are lots of ways to train models with batches loading from disk, not from ram."
  }
}