![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8305359%2F906147b6be1a1919d3ce16d496e2dcb8%2F_20230107205256.png?generation=1673177525858681&alt=media\")
I find an interesting method which was shared by Priyanshu at this. Kaggle has recently released TPU 1VM v3-8, which increases the RAM to 330 GB and the core to 96 when activated, which maybe helpful for you to train your models in kaggle! And @kaggleqrdl provided the code to decode all 54K images in train using dicomsdl which just only needs 13 minutes. https://www.kaggle.com/code/kaggleqrdl/13-minutes-decde-54k-dicom-using-new-96-core-arch. Thanks for your sharing and exploring @kaggleqrdl !!!
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F214989%2Fb5a3240e99831c774de5312e79840dbf%2Fflying.png?generation=1672919834462496&alt=media\")
tqdm\nstart_time = time.time()\n\nParallel(n_jobs=)(\n delayed(process)(f, size = , save_folder = image_dir_dicomsdl, dicom_process = )\n f tqdm.tqdm(train_images)\n)\n\n(time.time() - start_time)\nI find an interesting method which was shared by Priyanshu at this. Kaggle has recently released TPU 1VM v3-8, which increases the RAM to 330 GB and the core to 96 when activated, which maybe helpful for you to train your models in kaggle! And @kaggleqrdl provided the code to decode all 54K images in train using dicomsdl which just only needs 13 minutes. https://www.kaggle.com/code/kaggleqrdl/13-minutes-decde-54k-dicom-using-new-96-core-arch. Thanks for your sharing and exploring @kaggleqrdl !!!
\n
tqdm\nstart_time = time.time()\n\nParallel(n_jobs=)(\n delayed(process)(f, size = , save_folder = image_dir_dicomsdl, dicom_process = )\n f tqdm.tqdm(train_images)\n)\n\n(time.time() - start_time)\nfyi the tpu vm referenced can't be used for competition submission
", "rawMarkdown": "fyi the tpu vm referenced can't be used for competition submission", "votes": 5, "replies": [ { "id": 2091396, "postDate": "2023-01-08T11:24:48.323Z", "content": "But you can use this to train your pre-trained model.
", "rawMarkdown": "But you can use this to train your pre-trained model.", "votes": 1, "replies": [ { "id": 2091670, "postDate": "2023-01-08T16:33:12.790Z", "content": "Yeah, and for things like object detection this is very important as training can take a lot of time.
", "rawMarkdown": "Yeah, and for things like object detection this is very important as training can take a lot of time.", "votes": 1 } ] } ] }, { "id": 2091028, "postDate": "2023-01-07T23:42:26.977Z", "content": "That is some sexy stuff. Thanks for posting. Now off to learn PyTorch for TPU.
", "rawMarkdown": "That is some sexy stuff. Thanks for posting. Now off to learn PyTorch for TPU. ", "votes": 3, "replies": [ { "id": 2091030, "postDate": "2023-01-07T23:51:12.623Z", "content": "\nA general comment for folks new to Kaggle, upvoting when someone provides a resource you find very useful (and this is) is great cause it brings attention to it for other participants.
", "rawMarkdown": "@msthil \n\nA general comment for folks new to Kaggle, upvoting when someone provides a resource you find very useful (and this is) is great cause it brings attention to it for other participants.", "votes": 2 }, { "id": 2091417, "postDate": "2023-01-08T11:38:11.240Z", "content": "You're welcome! It's my honor to share something helpful for more kagglers!😄
", "rawMarkdown": "You're welcome! It's my honor to share something helpful for more kagglers!😄" } ] }, { "id": 2090966, "postDate": "2023-01-07T21:23:46.033Z", "content": "Also 96 CPU cores!
", "rawMarkdown": "Also 96 CPU cores! ", "votes": 3, "replies": [ { "id": 2091017, "postDate": "2023-01-07T23:01:01.567Z", "content": " tqdm\nstart_time = time.time()\n\nParallel(n_jobs=)(\n delayed(process)(f, size = , save_folder = image_dir_dicomsdl, dicom_process = )\n f tqdm.tqdm(train_images)\n)\n\n(time.time() - start_time)\n13 minutes to decode all 54K images in train using dicomsdl. Fun!
\n@songqizhou Can you edit your title to add the 96 cpu cores to it.
", "rawMarkdown": "```python\nimport tqdm\nstart_time = time.time()\n \nParallel(n_jobs=96)(\n delayed(process)(f, size = 512, save_folder = image_dir_dicomsdl, dicom_process = False)\n for f in tqdm.tqdm(train_images)\n)\n\nprint(time.time() - start_time)\n```\n\n**[13 minutes](https://www.kaggle.com/code/kaggleqrdl/13-minutes-decde-54k-dicom-using-new-96-core-arch)** to decode all 54K images in train using dicomsdl. Fun!\n\n@songqizhou Can you edit your title to add the 96 cpu cores to it. ", "votes": 4, "replies": [ { "id": 2091401, "postDate": "2023-01-08T11:26:32.843Z", "content": "Haha, that's fine! Thanks for your sharing!
", "rawMarkdown": "Haha, that's fine! Thanks for your sharing!", "votes": 3 }, { "id": 2094658, "postDate": "2023-01-10T23:29:28.797Z", "content": "\n\n13 minutes to decode all 54K images in train using dicomsdl. Fun!
\n
Did you actually run this in 13 mins ? seems unlikely - we are still going to be limited by IO. I tried with pydicom it didn't finish even in an hr. \"Fun!\" part I agree if it works ?
\nEdit: hope I'm doing this right, IO MB/s seems less than CPU
\n![\"Screen-Shot-2023-01-10-at-6-38-09-PM.png\"](\"https://i.postimg.cc/RFzHndF2/Screen-Shot-2023-01-10-at-6-38-09-PM.png\")
Yep, just ran it again, less than 13 minutes.
\nQ - did you try just forking and running the notebook? If so, what results did you see?
\nMaybe you saw different results with your own notebook because you aren't resizing to 512x512? pydicom is certainly slower than dicomsdl. Maybe it's memory usage interferes with multi core utilization, dunno.
\nMaybe there are different IO constraints depending on the VM you get. This could be a problem that folks might run into during submission.
\nIf you share your notebook I can try it and take a look.
\nAlso, not sure why @sohier upvoted your post. Perhaps he might explain his reasoning given that he's Kaggle staff.
\nIn theory the write to disk shouldn't be included in the 13 minutes, as this is about decoding and not also encoding. Everyone likely has their own caching scheme which may take more or less time. However, I wanted to verify the results.
", "rawMarkdown": "@rashmibanthia \n\nYep, just ran it again, less than 13 minutes. \n\n\nQ - did you try just forking and running the [notebook](https://www.kaggle.com/code/kaggleqrdl/13-minutes-decde-54k-dicom-using-new-96-core-arch)? If so, what results did you see?\n\nMaybe you saw different results with your own notebook because you aren't resizing to 512x512? pydicom is certainly slower than dicomsdl. Maybe it's memory usage interferes with multi core utilization, dunno.\n\nMaybe there are different IO constraints depending on the VM you get. This could be a problem that folks might run into during submission.\n\nIf you share your notebook I can try it and take a look.\n\nAlso, not sure why @sohier upvoted your post. Perhaps he might explain his reasoning given that he's Kaggle staff.\n\nIn theory the write to disk shouldn't be included in the 13 minutes, as this is about decoding and not also encoding. Everyone likely has their own caching scheme which may take more or less time. However, I wanted to verify the results.\n\n" }, { "id": 2094743, "postDate": "2023-01-11T02:09:40.887Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F8a3f76e77e31cea76476a465104315f4%2Frez.png?generation=1673402898008518&alt=media\")
I randomly sampled a bunch of the images, they all looked fine to me.
", "rawMarkdown": "\n\nI randomly sampled a bunch of the images, they all looked fine to me. " }, { "id": 2094760, "postDate": "2023-01-11T02:36:23.593Z", "content": "\n\nQ - did you try just forking and running the notebook? If so, what results did you see?
\n
This notebook isn't utilizing TPU VM v3-8 🤔 … what did I miss ?
", "rawMarkdown": ">Q - did you try just forking and running the [notebook](https://www.kaggle.com/code/kaggleqrdl/13-minutes-decde-54k-dicom-using-new-96-core-arch)? If so, what results did you see?\n\n\nThis notebook isn't utilizing TPU VM v3-8 🤔 ... what did I miss ? \n" }, { "id": 2094795, "postDate": "2023-01-11T03:10:31.727Z", "content": "Weird. Did you run copy and edit? tpu vm 3-8 is selected when I do. Could be some kaggle bug.
\nAnyways, just select the vm I guess and give it a shot!
", "rawMarkdown": "Weird. Did you run copy and edit? tpu vm 3-8 is selected when I do. Could be some kaggle bug. \n\nAnyways, just select the vm I guess and give it a shot!" }, { "id": 2094876, "postDate": "2023-01-11T04:15:11.900Z", "content": "my bad, pydicom seems to be working .. I had \"backend=threading\" copied over which was messing up timing.
\nI think by default Kaggle is saving TPU VM3-8 kernels in \"quick save\" mode so log on notebook says \"Accelerator None\"
", "rawMarkdown": "my bad, pydicom seems to be working .. I had \"backend=threading\" copied over which was messing up timing. \n\nI think by default Kaggle is saving TPU VM3-8 kernels in \"quick save\" mode so log on notebook says \"Accelerator None\" ", "votes": 1 }, { "id": 2094880, "postDate": "2023-01-11T04:18:11.400Z", "content": "Ah, great, glad it worked for you. This new VM could potentially level the playing field.
", "rawMarkdown": "Ah, great, glad it worked for you. This new VM could potentially level the playing field.", "votes": 1 } ] } ] }, { "id": 2100650, "postDate": "2023-01-15T10:31:20.567Z", "content": "giant FlexiViT on giant tpu would be interesting
", "rawMarkdown": "giant FlexiViT on giant tpu would be interesting" }, { "id": 2096297, "postDate": "2023-01-12T00:16:02.040Z", "content": "Just FYI, the original image on the TPU VM is cpu specific, while the latest image is TPU, as per here - https://www.kaggle.com/discussions/product-feedback/369338#2096029
", "rawMarkdown": "Just FYI, the original image on the TPU VM is cpu specific, while the latest image is TPU, as per here - https://www.kaggle.com/discussions/product-feedback/369338#2096029" }, { "id": 2095278, "postDate": "2023-01-11T09:06:11.483Z", "rawMarkdown": "", "votes": 4, "isDeleted": true }, { "id": 2100637, "postDate": "2023-01-15T10:19:07.723Z", "content": "informative info, thanks
", "rawMarkdown": "informative info, thanks" }, { "id": 2093797, "postDate": "2023-01-10T10:27:30.777Z", "content": "Very useful information! Thank you!
", "rawMarkdown": "Very useful information! Thank you!" } ], "comments": [ { "id": 2091052, "author_name": "David Austin", "author_url": "", "post_date": "2023-01-08T00:50:57.960000", "content": "fyi the tpu vm referenced can't be used for competition submission
", "votes": 5, "replies": [ { "id": 2091396, "author_name": "BarryZhou", "author_url": "", "post_date": "2023-01-08T11:24:48.323000", "content": "But you can use this to train your pre-trained model.
", "votes": 1, "replies": [ { "id": 2091670, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-01-08T16:33:12.790000", "content": "Yeah, and for things like object detection this is very important as training can take a lot of time.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2091028, "author_name": "Matt S.", "author_url": "", "post_date": "2023-01-07T23:42:26.977000", "content": "That is some sexy stuff. Thanks for posting. Now off to learn PyTorch for TPU.
", "votes": 3, "replies": [ { "id": 2091030, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-01-07T23:51:12.623000", "content": "\nA general comment for folks new to Kaggle, upvoting when someone provides a resource you find very useful (and this is) is great cause it brings attention to it for other participants.
", "votes": 2, "replies": [] }, { "id": 2091417, "author_name": "BarryZhou", "author_url": "", "post_date": "2023-01-08T11:38:11.240000", "content": "You're welcome! It's my honor to share something helpful for more kagglers!😄
", "votes": 0, "replies": [] } ] }, { "id": 2090966, "author_name": "The Devastator", "author_url": "", "post_date": "2023-01-07T21:23:46.033000", "content": "Also 96 CPU cores!
", "votes": 3, "replies": [ { "id": 2091017, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-01-07T23:01:01.567000", "content": " tqdm\nstart_time = time.time()\n\nParallel(n_jobs=)(\n delayed(process)(f, size = , save_folder = image_dir_dicomsdl, dicom_process = )\n f tqdm.tqdm(train_images)\n)\n\n(time.time() - start_time)\n13 minutes to decode all 54K images in train using dicomsdl. Fun!
\n@songqizhou Can you edit your title to add the 96 cpu cores to it.
", "votes": 4, "replies": [ { "id": 2091401, "author_name": "BarryZhou", "author_url": "", "post_date": "2023-01-08T11:26:32.843000", "content": "Haha, that's fine! Thanks for your sharing!
", "votes": 3, "replies": [] }, { "id": 2094658, "author_name": "RB", "author_url": "", "post_date": "2023-01-10T23:29:28.797000", "content": "\n\n13 minutes to decode all 54K images in train using dicomsdl. Fun!
\n
Did you actually run this in 13 mins ? seems unlikely - we are still going to be limited by IO. I tried with pydicom it didn't finish even in an hr. \"Fun!\" part I agree if it works ?
\nEdit: hope I'm doing this right, IO MB/s seems less than CPU
\n
Yep, just ran it again, less than 13 minutes.
\nQ - did you try just forking and running the notebook? If so, what results did you see?
\nMaybe you saw different results with your own notebook because you aren't resizing to 512x512? pydicom is certainly slower than dicomsdl. Maybe it's memory usage interferes with multi core utilization, dunno.
\nMaybe there are different IO constraints depending on the VM you get. This could be a problem that folks might run into during submission.
\nIf you share your notebook I can try it and take a look.
\nAlso, not sure why @sohier upvoted your post. Perhaps he might explain his reasoning given that he's Kaggle staff.
\nIn theory the write to disk shouldn't be included in the 13 minutes, as this is about decoding and not also encoding. Everyone likely has their own caching scheme which may take more or less time. However, I wanted to verify the results.
", "votes": 0, "replies": [] }, { "id": 2094743, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-01-11T02:09:40.887000", "content": "
I randomly sampled a bunch of the images, they all looked fine to me.
", "votes": 0, "replies": [] }, { "id": 2094760, "author_name": "RB", "author_url": "", "post_date": "2023-01-11T02:36:23.593000", "content": "\n\nQ - did you try just forking and running the notebook? If so, what results did you see?
\n
This notebook isn't utilizing TPU VM v3-8 🤔 … what did I miss ?
", "votes": 0, "replies": [] }, { "id": 2094795, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-01-11T03:10:31.727000", "content": "Weird. Did you run copy and edit? tpu vm 3-8 is selected when I do. Could be some kaggle bug.
\nAnyways, just select the vm I guess and give it a shot!
", "votes": 0, "replies": [] }, { "id": 2094876, "author_name": "RB", "author_url": "", "post_date": "2023-01-11T04:15:11.900000", "content": "my bad, pydicom seems to be working .. I had \"backend=threading\" copied over which was messing up timing.
\nI think by default Kaggle is saving TPU VM3-8 kernels in \"quick save\" mode so log on notebook says \"Accelerator None\"
", "votes": 1, "replies": [] }, { "id": 2094880, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-01-11T04:18:11.400000", "content": "Ah, great, glad it worked for you. This new VM could potentially level the playing field.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2100650, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-15T10:31:20.567000", "content": "giant FlexiViT on giant tpu would be interesting
", "votes": 0, "replies": [] }, { "id": 2096297, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2023-01-12T00:16:02.040000", "content": "Just FYI, the original image on the TPU VM is cpu specific, while the latest image is TPU, as per here - https://www.kaggle.com/discussions/product-feedback/369338#2096029
", "votes": 0, "replies": [] }, { "id": 2095278, "author_name": "", "author_url": "", "post_date": "2023-01-11T09:06:11.483000", "content": "", "votes": 4, "replies": [] }, { "id": 2100637, "author_name": "Gaju Ahmed", "author_url": "", "post_date": "2023-01-15T10:19:07.723000", "content": "informative info, thanks
", "votes": 0, "replies": [] }, { "id": 2093797, "author_name": "junseonglee11", "author_url": "", "post_date": "2023-01-10T10:27:30.777000", "content": "Very useful information! Thank you!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2090554": "I find an interesting method which was shared by **Priyanshu** at [this](https://www.kaggle.com/competitions/otto-recommender-system/discussion/375297). Kaggle has recently released TPU 1VM v3-8, which increases **the RAM to 330 GB** and the **core to 96** when activated, which maybe helpful for you to train your models in kaggle! And @kaggleqrdl provided the code to decode all 54K images in train using dicomsdl which just only needs 13 minutes. [https://www.kaggle.com/code/kaggleqrdl/13-minutes-decde-54k-dicom-using-new-96-core-arch](https://www.kaggle.com/code/kaggleqrdl/13-minutes-decde-54k-dicom-using-new-96-core-arch). Thanks for your sharing and exploring @kaggleqrdl !!!\n\n\n\n\n\n```python\nimport tqdm\nstart_time = time.time()\n\nParallel(n_jobs=96)(\n delayed(process)(f, size = 512, save_folder = image_dir_dicomsdl, dicom_process = False)\n for f in tqdm.tqdm(train_images)\n)\n\nprint(time.time() - start_time)\n```\n", "2091052": "fyi the tpu vm referenced can't be used for competition submission", "2091028": "That is some sexy stuff. Thanks for posting. Now off to learn PyTorch for TPU. ", "2090966": "Also 96 CPU cores! ", "2100650": "giant FlexiViT on giant tpu would be interesting", "2096297": "Just FYI, the original image on the TPU VM is cpu specific, while the latest image is TPU, as per here - https://www.kaggle.com/discussions/product-feedback/369338#2096029", "2095278": "", "2100637": "informative info, thanks", "2093797": "Very useful information! Thank you!" } }