{ "id": 372514, "title": "Long Submission Time", "url": "/competitions/rsna-breast-cancer-detection/discussion/372514", "author_name": "Arunim Basak", "post_date": "2022-12-16T12:28:24.328000", "votes": 1, "comment_count": 6, "views": 0, "content": "

I am submitting my notebook but it's taking a very long time to score like 4-5hrs but still not getting the score.
\nWhat is the time for everyone for submitting their notebooks ? Is it like 4-5hrs or less ?

", "messages": [ { "id": 2067579, "postDate": "2022-12-16T21:00:22.730Z", "content": "

Hi,

\n

I highly recommend to read this discussion https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371033. My precedent submissions tooks 6-7 hours and with dicomsdl it took around 4 hours. I also recommend using cpu notebook and be sure to use 4 cpu cores available.
\nThere are also methods using gpu with nvidia dali : https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371534. But maybe you can try the version with dicomsdl first since it it way easier to implement.

\n

Hope it helps !

", "rawMarkdown": "Hi,\n\nI highly recommend to read this discussion https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371033. My precedent submissions tooks 6-7 hours and with dicomsdl it took around 4 hours. I also recommend using cpu notebook and be sure to use 4 cpu cores available. \nThere are also methods using gpu with nvidia dali : https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371534. But maybe you can try the version with dicomsdl first since it it way easier to implement.\n\nHope it helps !", "replies": [ { "id": 2067661, "postDate": "2022-12-17T00:54:53.600Z", "content": "

Hmm, I've done benchmarking with cpu versus gpu, and cpu is simply too slow when using things like pytorch for this to be viable. Have you seen something different?

", "rawMarkdown": "Hmm, I've done benchmarking with cpu versus gpu, and cpu is simply too slow when using things like pytorch for this to be viable. Have you seen something different?", "votes": 1, "replies": [ { "id": 2067855, "postDate": "2022-12-17T08:46:14.070Z", "content": "

To clarify things, I have a notebook for training and one for submission. In my submission notebook, I import the model I trained and then use cpu for inference.
\nBeing able to use 4 core is really useful to parallelize dicom to png conversion which is taking most of the submission time. I tried it with different models like EfficientNet or ViT and with dicomsdl I achieved 4 hours to submit predictions.

\n

NB : I am also using PyTorch

", "rawMarkdown": "To clarify things, I have a notebook for training and one for submission. In my submission notebook, I import the model I trained and then use cpu for inference. \nBeing able to use 4 core is really useful to parallelize dicom to png conversion which is taking most of the submission time. I tried it with different models like EfficientNet or ViT and with dicomsdl I achieved 4 hours to submit predictions. \n\nNB : I am also using PyTorch" }, { "id": 2068230, "postDate": "2022-12-17T16:23:37Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2068234, "postDate": "2022-12-17T16:25:11.533Z", "content": "

Ok , thank you , I will surely see the links provided.

", "rawMarkdown": "Ok , thank you , I will surely see the links provided." } ] } ] }, { "id": 2067177, "postDate": "2022-12-16T13:35:23.870Z", "content": "

It takes close to 6 hours going from comments of other participants. I don't know if someone has done in lesser. Many are now working towards faster image conversion tools from .dcm to .jpg/.png formats for test data inference.

", "rawMarkdown": "It takes close to 6 hours going from comments of other participants. I don't know if someone has done in lesser. Many are now working towards faster image conversion tools from .dcm to .jpg/.png formats for test data inference." }, { "id": 2067123, "postDate": "2022-12-16T12:28:24.330Z", "content": "

I am submitting my notebook but it's taking a very long time to score like 4-5hrs but still not getting the score.
\nWhat is the time for everyone for submitting their notebooks ? Is it like 4-5hrs or less ?

", "rawMarkdown": "I am submitting my notebook but it's taking a very long time to score like 4-5hrs but still not getting the score. \nWhat is the time for everyone for submitting their notebooks ? Is it like 4-5hrs or less ?" } ], "comments": [ { "id": 2067579, "author_name": "Natyu", "author_url": "", "post_date": "2022-12-16T21:00:22.730000", "content": "

Hi,

\n

I highly recommend to read this discussion https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371033. My precedent submissions tooks 6-7 hours and with dicomsdl it took around 4 hours. I also recommend using cpu notebook and be sure to use 4 cpu cores available.
\nThere are also methods using gpu with nvidia dali : https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371534. But maybe you can try the version with dicomsdl first since it it way easier to implement.

\n

Hope it helps !

", "votes": 0, "replies": [ { "id": 2067661, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-17T00:54:53.600000", "content": "

Hmm, I've done benchmarking with cpu versus gpu, and cpu is simply too slow when using things like pytorch for this to be viable. Have you seen something different?

", "votes": 1, "replies": [ { "id": 2067855, "author_name": "Natyu", "author_url": "", "post_date": "2022-12-17T08:46:14.070000", "content": "

To clarify things, I have a notebook for training and one for submission. In my submission notebook, I import the model I trained and then use cpu for inference.
\nBeing able to use 4 core is really useful to parallelize dicom to png conversion which is taking most of the submission time. I tried it with different models like EfficientNet or ViT and with dicomsdl I achieved 4 hours to submit predictions.

\n

NB : I am also using PyTorch

", "votes": 0, "replies": [] }, { "id": 2068230, "author_name": "", "author_url": "", "post_date": "2022-12-17T16:23:37", "content": "", "votes": 0, "replies": [] }, { "id": 2068234, "author_name": "Arunim Basak", "author_url": "", "post_date": "2022-12-17T16:25:11.533000", "content": "

Ok , thank you , I will surely see the links provided.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2067177, "author_name": "Sandy", "author_url": "", "post_date": "2022-12-16T13:35:23.870000", "content": "

It takes close to 6 hours going from comments of other participants. I don't know if someone has done in lesser. Many are now working towards faster image conversion tools from .dcm to .jpg/.png formats for test data inference.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2067579": "Hi,\n\nI highly recommend to read this discussion https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371033. My precedent submissions tooks 6-7 hours and with dicomsdl it took around 4 hours. I also recommend using cpu notebook and be sure to use 4 cpu cores available. \nThere are also methods using gpu with nvidia dali : https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371534. But maybe you can try the version with dicomsdl first since it it way easier to implement.\n\nHope it helps !", "2067177": "It takes close to 6 hours going from comments of other participants. I don't know if someone has done in lesser. Many are now working towards faster image conversion tools from .dcm to .jpg/.png formats for test data inference.", "2067123": "I am submitting my notebook but it's taking a very long time to score like 4-5hrs but still not getting the score. \nWhat is the time for everyone for submitting their notebooks ? Is it like 4-5hrs or less ?" } }