{ "id": 381670, "title": "Preprocessing code running too slow on kaggle", "url": "/competitions/rsna-breast-cancer-detection/discussion/381670", "author_name": "Xiao-Su (Frank) Hu", "post_date": "2023-01-27T17:18:49.563000", "votes": 1, "comment_count": 2, "views": 0, "content": "

Hello Kagglers,

\n

Happy new year!

\n

I am new to Kaggle and am having an issue running DCM preprocessing code with notebooks on Kaggle.

\n

When I \"Save and commit\" my notebook, it will run beyond 12 hours and got automatically cancelled.

\n

The code I am using is posted by RADEK OSMULSKI @ https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs?scriptVersionId=113529850&cellId=1.

\n

He mentioned in his other post that using the 4 available cores on Kaggle the preprocessing should take ~3 hrs.

\n

However, this is not true in my case, my notebook is in this link https://www.kaggle.com/xiaosufrankhu/rsna-mammo-v3

\n

Any hint or guidance is appreciated.

\n

I understand that I can either download the data to convert using local machine, or using Google online computing services to speed up, but I am just wondering why my notebook is running so slow with Kaggle CPU.

\n

Also I am assuming the testing data set online will also need to be converted using the preprocessing code included in the submitted notebook and the running time for the notebook should be no more than 9 hrs (please let me know if this understanding is correct).

\n

Thank you so much!

", "messages": [ { "id": 2118289, "postDate": "2023-01-27T22:33:14.483Z", "content": "

You right you have to convert images before inference your model in submission notebook and no cache available between runs. Take a look at https://www.kaggle.com/code/theoviel/rsna-breast-baseline-faster-inference-with-dali this is proposal to use https://docs.nvidia.com/deeplearning/dali/user-guide/docs which is faster then pydicom based pipeline you've tried. DALI based pipeline is about 2h to process all test files (~30k) in kaggle env.

", "rawMarkdown": "You right you have to convert images before inference your model in submission notebook and no cache available between runs. Take a look at https://www.kaggle.com/code/theoviel/rsna-breast-baseline-faster-inference-with-dali this is proposal to use https://docs.nvidia.com/deeplearning/dali/user-guide/docs which is faster then pydicom based pipeline you've tried. DALI based pipeline is about 2h to process all test files (~30k) in kaggle env.", "votes": 1, "replies": [ { "id": 2119270, "postDate": "2023-01-28T16:25:08.970Z", "content": "

Thank you so much for your reply. I will definitely take a look at the links you suggested.

", "rawMarkdown": "Thank you so much for your reply. I will definitely take a look at the links you suggested." } ] }, { "id": 2117975, "postDate": "2023-01-27T17:18:49.563Z", "content": "

Hello Kagglers,

\n

Happy new year!

\n

I am new to Kaggle and am having an issue running DCM preprocessing code with notebooks on Kaggle.

\n

When I \"Save and commit\" my notebook, it will run beyond 12 hours and got automatically cancelled.

\n

The code I am using is posted by RADEK OSMULSKI @ https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs?scriptVersionId=113529850&cellId=1.

\n

He mentioned in his other post that using the 4 available cores on Kaggle the preprocessing should take ~3 hrs.

\n

However, this is not true in my case, my notebook is in this link https://www.kaggle.com/xiaosufrankhu/rsna-mammo-v3

\n

Any hint or guidance is appreciated.

\n

I understand that I can either download the data to convert using local machine, or using Google online computing services to speed up, but I am just wondering why my notebook is running so slow with Kaggle CPU.

\n

Also I am assuming the testing data set online will also need to be converted using the preprocessing code included in the submitted notebook and the running time for the notebook should be no more than 9 hrs (please let me know if this understanding is correct).

\n

Thank you so much!

", "rawMarkdown": "Hello Kagglers,\n\nHappy new year!\n\nI am new to Kaggle and am having an issue running DCM preprocessing code with notebooks on Kaggle.\n\nWhen I \"Save and commit\" my notebook, it will run beyond 12 hours and got automatically cancelled. \n\nThe code I am using is posted by RADEK OSMULSKI @ https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs?scriptVersionId=113529850&cellId=1.\n\nHe mentioned in his other post that using the 4 available cores on Kaggle the preprocessing should take ~3 hrs.\n\nHowever, this is not true in my case, my notebook is in this link https://www.kaggle.com/xiaosufrankhu/rsna-mammo-v3 \n\nAny hint or guidance is appreciated.\n\nI understand that I can either download the data to convert using local machine, or using Google online computing services to speed up, but I am just wondering why my notebook is running so slow with Kaggle CPU. \n\nAlso I am assuming the testing data set online will also need to be converted using the preprocessing code included in the submitted notebook and the running time for the notebook should be no more than 9 hrs (please let me know if this understanding is correct).\n\nThank you so much!", "votes": 1 } ], "comments": [ { "id": 2118289, "author_name": "A.P.", "author_url": "", "post_date": "2023-01-27T22:33:14.483000", "content": "

You right you have to convert images before inference your model in submission notebook and no cache available between runs. Take a look at https://www.kaggle.com/code/theoviel/rsna-breast-baseline-faster-inference-with-dali this is proposal to use https://docs.nvidia.com/deeplearning/dali/user-guide/docs which is faster then pydicom based pipeline you've tried. DALI based pipeline is about 2h to process all test files (~30k) in kaggle env.

", "votes": 1, "replies": [ { "id": 2119270, "author_name": "Xiao-Su (Frank) Hu", "author_url": "", "post_date": "2023-01-28T16:25:08.970000", "content": "

Thank you so much for your reply. I will definitely take a look at the links you suggested.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2118289": "You right you have to convert images before inference your model in submission notebook and no cache available between runs. Take a look at https://www.kaggle.com/code/theoviel/rsna-breast-baseline-faster-inference-with-dali this is proposal to use https://docs.nvidia.com/deeplearning/dali/user-guide/docs which is faster then pydicom based pipeline you've tried. DALI based pipeline is about 2h to process all test files (~30k) in kaggle env.", "2117975": "Hello Kagglers,\n\nHappy new year!\n\nI am new to Kaggle and am having an issue running DCM preprocessing code with notebooks on Kaggle.\n\nWhen I \"Save and commit\" my notebook, it will run beyond 12 hours and got automatically cancelled. \n\nThe code I am using is posted by RADEK OSMULSKI @ https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs?scriptVersionId=113529850&cellId=1.\n\nHe mentioned in his other post that using the 4 available cores on Kaggle the preprocessing should take ~3 hrs.\n\nHowever, this is not true in my case, my notebook is in this link https://www.kaggle.com/xiaosufrankhu/rsna-mammo-v3 \n\nAny hint or guidance is appreciated.\n\nI understand that I can either download the data to convert using local machine, or using Google online computing services to speed up, but I am just wondering why my notebook is running so slow with Kaggle CPU. \n\nAlso I am assuming the testing data set online will also need to be converted using the preprocessing code included in the submitted notebook and the running time for the notebook should be no more than 9 hrs (please let me know if this understanding is correct).\n\nThank you so much!" } }