{
  "id": 165407,
  "title": "Colab Training - Issues with dataset size",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/165407",
  "author_name": "Vee",
  "post_date": "2020-07-09T16:01:42.255000",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I joined the competition just a few hours back.\nTried to fire up a Colab instance and got a shock with regard to the size of the dataset.</p>\n\n<p>Colab Pro doesn't allow me to clone this dataset because of memory issues.\nI see some mentions of people training on Colab - how are you doing it ? \nAm i doing something wrong ?</p>\n\n<p>Any help greatly appreciated. Thanks.</p>",
  "messages": [
    {
      "id": 922436,
      "postDate": "2020-07-10T05:25:27.203Z",
      "content": "<p>What I did was to create a bucket on GCS then with a Kaggle notebook extract data (images) in jpeg  and copy them to GCS. </p>",
      "rawMarkdown": "What I did was to create a bucket on GCS then with a Kaggle notebook extract data (images) in jpeg  and copy them to GCS. ",
      "votes": 1,
      "replies": [
        {
          "id": 922488,
          "postDate": "2020-07-10T06:08:49.177Z",
          "content": "<p>So you copied them to GCS and then trained on Colab - I am assuming that you didn't copy them onto Colab again.</p>\n\n<p>Didn't you face any slowdown ? Did you do anything to ensure that both Colab and GCS were in the same region.</p>\n\n<p>Thank you.</p>",
          "rawMarkdown": "So you copied them to GCS and then trained on Colab - I am assuming that you didn't copy them onto Colab again.\n\nDidn't you face any slowdown ? Did you do anything to ensure that both Colab and GCS were in the same region.\n\nThank you."
        },
        {
          "id": 923446,
          "postDate": "2020-07-10T21:21:43.390Z",
          "content": "<p>No I didn't copy them to the Colab VM. \nI train model with tensorflow - tpu, so the data need to be on GCS anyway.\nI create the bucket in the region US. No I don't know how to make sure that Colab and GCS are in the same region, but up till now it's been working out well for me.</p>",
          "rawMarkdown": "No I didn't copy them to the Colab VM. \nI train model with tensorflow - tpu, so the data need to be on GCS anyway.\nI create the bucket in the region US. No I don't know how to make sure that Colab and GCS are in the same region, but up till now it's been working out well for me.",
          "votes": 1
        },
        {
          "id": 928033,
          "postDate": "2020-07-13T17:47:40.393Z",
          "content": "<p>Thank you.\nI figured out a number of good solutions of working with Colab. gsutil and gcsfuse are good, but the size of the data set is a big problem. </p>\n\n<p>I eventually figured out that keeping the data on GCS and training with TPUs is a lot better.</p>\n\n<p>Hope this helps someone else too.</p>",
          "rawMarkdown": "Thank you.\nI figured out a number of good solutions of working with Colab. gsutil and gcsfuse are good, but the size of the data set is a big problem. \n\nI eventually figured out that keeping the data on GCS and training with TPUs is a lot better.\n\nHope this helps someone else too."
        }
      ]
    },
    {
      "id": 921867,
      "postDate": "2020-07-09T16:01:42.257Z",
      "content": "<p>I joined the competition just a few hours back.\nTried to fire up a Colab instance and got a shock with regard to the size of the dataset.</p>\n\n<p>Colab Pro doesn't allow me to clone this dataset because of memory issues.\nI see some mentions of people training on Colab - how are you doing it ? \nAm i doing something wrong ?</p>\n\n<p>Any help greatly appreciated. Thanks.</p>",
      "rawMarkdown": "I joined the competition just a few hours back.\nTried to fire up a Colab instance and got a shock with regard to the size of the dataset.\n\nColab Pro doesn't allow me to clone this dataset because of memory issues.\nI see some mentions of people training on Colab - how are you doing it ? \nAm i doing something wrong ?\n\nAny help greatly appreciated. Thanks.",
      "votes": 2
    },
    {
      "id": 931894,
      "postDate": "2020-07-16T14:34:59.517Z",
      "content": "<p>I did the image preprocessing on my desktop before uploading the preprocessed images (which take drastically less space) to my google drive, then I simply mounted my google drive into my colab instance.</p>",
      "rawMarkdown": "I did the image preprocessing on my desktop before uploading the preprocessed images (which take drastically less space) to my google drive, then I simply mounted my google drive into my colab instance."
    }
  ],
  "comments": [
    {
      "id": 922436,
      "author_name": "Duc-Kinh Le Tran",
      "author_url": "",
      "post_date": "2020-07-10T05:25:27.203000",
      "content": "<p>What I did was to create a bucket on GCS then with a Kaggle notebook extract data (images) in jpeg  and copy them to GCS. </p>",
      "votes": 1,
      "replies": [
        {
          "id": 922488,
          "author_name": "Vee",
          "author_url": "",
          "post_date": "2020-07-10T06:08:49.177000",
          "content": "<p>So you copied them to GCS and then trained on Colab - I am assuming that you didn't copy them onto Colab again.</p>\n\n<p>Didn't you face any slowdown ? Did you do anything to ensure that both Colab and GCS were in the same region.</p>\n\n<p>Thank you.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 923446,
          "author_name": "Duc-Kinh Le Tran",
          "author_url": "",
          "post_date": "2020-07-10T21:21:43.390000",
          "content": "<p>No I didn't copy them to the Colab VM. \nI train model with tensorflow - tpu, so the data need to be on GCS anyway.\nI create the bucket in the region US. No I don't know how to make sure that Colab and GCS are in the same region, but up till now it's been working out well for me.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 928033,
          "author_name": "Vee",
          "author_url": "",
          "post_date": "2020-07-13T17:47:40.393000",
          "content": "<p>Thank you.\nI figured out a number of good solutions of working with Colab. gsutil and gcsfuse are good, but the size of the data set is a big problem. </p>\n\n<p>I eventually figured out that keeping the data on GCS and training with TPUs is a lot better.</p>\n\n<p>Hope this helps someone else too.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 931894,
      "author_name": "Neil Kloper",
      "author_url": "",
      "post_date": "2020-07-16T14:34:59.517000",
      "content": "<p>I did the image preprocessing on my desktop before uploading the preprocessed images (which take drastically less space) to my google drive, then I simply mounted my google drive into my colab instance.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "922436": "What I did was to create a bucket on GCS then with a Kaggle notebook extract data (images) in jpeg  and copy them to GCS. ",
    "921867": "I joined the competition just a few hours back.\nTried to fire up a Colab instance and got a shock with regard to the size of the dataset.\n\nColab Pro doesn't allow me to clone this dataset because of memory issues.\nI see some mentions of people training on Colab - how are you doing it ? \nAm i doing something wrong ?\n\nAny help greatly appreciated. Thanks.",
    "931894": "I did the image preprocessing on my desktop before uploading the preprocessed images (which take drastically less space) to my google drive, then I simply mounted my google drive into my colab instance."
  }
}