![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8234844%2F5565efe329cb833af7dc07a4fa8e9e06%2FScreenshot%20from%202022-09-03%2018-07-07.png?generation=1662196390916631&alt=media\")
There are 400GB with train data, test data and other data. I would like to work with Colab but could not figure out how to do it. Fortunately, the following site explains how to mount (sorry for the Japanese).
\nhttps://slash-z.com/google-colab-mount-kaggle-competition-dataset/
\nThe outline is
\n①Open new notebokk in kaggle_notebook and display GCS path in kaggle_notebook.
\nfrom kaggle_datasets import KaggleDatasets
\nKaggleDatasets().get_gcs_path()
gs://{path-name}
\n②Start colab and allow authentication of Google Cloud SDK via web browser.
\nfrom google.colab import auth
\nauth.authenticate_user()
Paste the authentication code copied above into the colab blank and enter to authenticate.
\n③ GPG key acquisition / Google Cloud Strage FUSE installation in colab environment.
\n!echo \"deb http://packages.cloud.google.com/apt gcsfuse-lsb_release -c -s main\" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
\n!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
\n!apt-get -y -q update
\n!apt-get -y -q install gcsfuse
④ Create a directory for mount with any name you like / mount the GCS path with the directory you just created using the gscfuse command.
\n!mkdir -p tmp
\n!gcsfuse --implicit-dirs --limit-bytes-per-sec -1 --limit-ops-per-sec -1 \"｛①path-name｝\" tmp
There are 400GB with train data, test data and other data. I would like to work with Colab but could not figure out how to do it. Fortunately, the following site explains how to mount (sorry for the Japanese).
\nhttps://slash-z.com/google-colab-mount-kaggle-competition-dataset/
\nThe outline is
\n①Open new notebokk in kaggle_notebook and display GCS path in kaggle_notebook.
\nfrom kaggle_datasets import KaggleDatasets
\nKaggleDatasets().get_gcs_path()
gs://{path-name}
\n②Start colab and allow authentication of Google Cloud SDK via web browser.
\nfrom google.colab import auth
\nauth.authenticate_user()
Paste the authentication code copied above into the colab blank and enter to authenticate.
\n③ GPG key acquisition / Google Cloud Strage FUSE installation in colab environment.
\n!echo \"deb http://packages.cloud.google.com/apt gcsfuse-lsb_release -c -s main\" | sudo tee /etc/apt/sources.list.d/gcsfuse.list
\n!curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
\n!apt-get -y -q update
\n!apt-get -y -q install gcsfuse
④ Create a directory for mount with any name you like / mount the GCS path with the directory you just created using the gscfuse command.
\n!mkdir -p tmp
\n!gcsfuse --implicit-dirs --limit-bytes-per-sec -1 --limit-ops-per-sec -1 \"｛①path-name｝\" tmp
I wonder what could be the benefit of using Colab vs Kaggle?
\nDid you do any benchmarks on how fast is it loading images in colab vs kaggle? e.g. how long does it take to load one image in each environment.
As I know there is a restriction to use your GPU in Kaggle. You can use it some extent but if you use your own environment it will help you to use it as long as you want. If you wonder why GPU is matter, GPU makes you to train your models with better performance than your CPU (which is default in Kaggle).
", "rawMarkdown": "As I know there is a restriction to use your GPU in Kaggle. You can use it some extent but if you use your own environment it will help you to use it as long as you want. If you wonder why GPU is matter, GPU makes you to train your models with better performance than your CPU (which is default in Kaggle).", "votes": 1 }, { "id": 1922800, "postDate": "2022-09-01T18:03:16.273Z", "content": "the question is how fast do you load the image in both environment, cause for instance if it is 10x slower to load an image, the training will be just frustrating. Also even with Colab, there is a limitation you can't indefinitely use GPU unless they removed this restriction recently.
", "rawMarkdown": "the question is how fast do you load the image in both environment, cause for instance if it is 10x slower to load an image, the training will be just frustrating. Also even with Colab, there is a limitation you can't indefinitely use GPU unless they removed this restriction recently.", "votes": 1 }, { "id": 1923127, "postDate": "2022-09-02T02:41:26.450Z", "content": "Thanks for your question.
\nAs Ahmet Çalış said, the reason for using Colab is Kaggle's GPU usage time limit.
\nAs for loading images, the transfer speed between Colab and Kaggle is faster than I expected.
\nI used
\nhttps://www.kaggle.com/code/yasufuminakama/mayo-train-images-size-1024-n-16-1
\nnotebook with input files mounted from Kaggle and run in Colab.
\nIn Y.NAKAMA's run, the processing time was 1 hour and 13 minutes for 100 images, and when I ran it with Colab, the processing time was 42 minutes.
I tried this, but the tmp folder was empty by the end.
", "rawMarkdown": "I tried this, but the tmp folder was empty by the end.", "replies": [ { "id": 1924650, "postDate": "2022-09-03T09:14:17.597Z", "content": "GCS path may be updated.
\nPlease check the GCS path in the Kaggle notebook before mounting to Colab. In my case, it was different from the last time. Using the latest GCS path, I was able to mount as shown in the figure.
\n(I changed the mount destination to input/mayo-clinic-strip-ai instead of tmp)
I wonder what could be the benefit of using Colab vs Kaggle?
\nDid you do any benchmarks on how fast is it loading images in colab vs kaggle? e.g. how long does it take to load one image in each environment.
As I know there is a restriction to use your GPU in Kaggle. You can use it some extent but if you use your own environment it will help you to use it as long as you want. If you wonder why GPU is matter, GPU makes you to train your models with better performance than your CPU (which is default in Kaggle).
", "votes": 1, "replies": [] }, { "id": 1922800, "author_name": "bachir", "author_url": "", "post_date": "2022-09-01T18:03:16.273000", "content": "the question is how fast do you load the image in both environment, cause for instance if it is 10x slower to load an image, the training will be just frustrating. Also even with Colab, there is a limitation you can't indefinitely use GPU unless they removed this restriction recently.
", "votes": 1, "replies": [] }, { "id": 1923127, "author_name": "min fuka", "author_url": "", "post_date": "2022-09-02T02:41:26.450000", "content": "Thanks for your question.
\nAs Ahmet Çalış said, the reason for using Colab is Kaggle's GPU usage time limit.
\nAs for loading images, the transfer speed between Colab and Kaggle is faster than I expected.
\nI used
\nhttps://www.kaggle.com/code/yasufuminakama/mayo-train-images-size-1024-n-16-1
\nnotebook with input files mounted from Kaggle and run in Colab.
\nIn Y.NAKAMA's run, the processing time was 1 hour and 13 minutes for 100 images, and when I ran it with Colab, the processing time was 42 minutes.
I tried this, but the tmp folder was empty by the end.
", "votes": 0, "replies": [ { "id": 1924650, "author_name": "min fuka", "author_url": "", "post_date": "2022-09-03T09:14:17.597000", "content": "GCS path may be updated.
\nPlease check the GCS path in the Kaggle notebook before mounting to Colab. In my case, it was different from the last time. Using the latest GCS path, I was able to mount as shown in the figure.
\n(I changed the mount destination to input/mayo-clinic-strip-ai instead of tmp)