This is not my dicovery but decided to create separate topic because this is probably important discovery. @kaggleqrdl described ( @alenic idea ) it here: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369684#2057282 I checked only speed of libraries and implemented notebook to show result and provide way to experiment.
\nIt seems that we got ~1,6x-2x improvemend in dicom processing speed.
I created quick implementation how to speed up dicom processing significantly.
\nGPU / 500 images (Parallel - 2 jobs):
\nMy estimation is that now processing 32.000 photos is about: 4h30min (but could be wrong)
\nhttps://www.kaggle.com/code/remekkinas/fast-dicom-processing
", "messages": [ { "id": 2058150, "postDate": "2022-12-07T16:32:10.117Z", "content": "This is not my dicovery but decided to create separate topic because this is probably important discovery. @kaggleqrdl described ( @alenic idea ) it here: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369684#2057282 I checked only speed of libraries and implemented notebook to show result and provide way to experiment.
\nIt seems that we got ~1,6x-2x improvemend in dicom processing speed.
I created quick implementation how to speed up dicom processing significantly.
\nGPU / 500 images (Parallel - 2 jobs):
\nMy estimation is that now processing 32.000 photos is about: 4h30min (but could be wrong)
\nhttps://www.kaggle.com/code/remekkinas/fast-dicom-processing
", "rawMarkdown": "**This is not my dicovery but decided to create separate topic because this is probably important discovery**. @kaggleqrdl described ( @alenic idea ) it here: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369684#2057282 I checked only speed of libraries and implemented notebook to show result and provide way to experiment.\nIt seems that we got ~1,6x-2x improvemend in dicom processing speed.\n\nI created quick implementation how to speed up dicom processing significantly.\n\nGPU / 500 images (Parallel - 2 jobs):\n- pydicom -> 396.74 sec\n- dicomsdl -> 243.39 sec \n\nMy estimation is that now processing 32.000 photos is about: 4h30min (but could be wrong)\n\nhttps://www.kaggle.com/code/remekkinas/fast-dicom-processing\n\n\n", "votes": 49 }, { "id": 2058426, "postDate": "2022-12-08T00:04:56.313Z", "content": "if you want to keep the previous code the same:
\n # https://github.com/tsangel/dicomsdl/blob/master/tutorials/timeit_test.ipynb\n dicom = pydicom.dcmread(f)\n image0 = dicom.pixel_array\n\n dicom1 = dicomsdl.open(f)\n image1 = dicom1.pixelData(storedvalue=True)\n\n #---- check\n diff = image0.astype(np.float32)-image1.astype(np.float32)\n print(diff.max(), diff.mean())\n\n plt.scatter(image1.reshape(-1), image0.reshape(-1))\n plt.show()\nto improve cpu to gpu transfer (e.g. for large 1024 or 2048) at inference, use 8bit byte and single channel.
\nconvert to float() and expand to 3 channel in your model forward function (which is in cuda)
refer to python/dicomsdl/__init__.py, you can modify
\n/python/dicomsdl/__init__.py\ndef __dataset__to_pil_image(self, index=0):\nor better still, use
\n__dataset__pixelData__\n\n storedvalue (bool): True for get stored values; pixel values before LUT\n transformation using RescaleSlope and RescaleIntercept.\n Returns:\n Numpy array containing pixel values of `index`'th image if dataset holds\n multiframe data. If `storedvalue` is False, RescaleSlope and\n RescaleIntercept are applied to pixel values.\n ( pixel values = stored values * RescsaleSlope + RescaleIntercept )\nfurther, the function below is c++ code and may be faster than numpy?
\nutil.convert_to_uint8(outarr, data8, xmin, xmax)\nThank you for your feedback - I will look into code and improve it.
\nNow (new version of notebook) it seems for me that images generated by dicomsdl are close to the pydicom - but this is observation based on model scoring only (test validation for both - pydicom and discomsdl). It increased speed significantly so now even 6 models (eg. resnet50d) on blend finished in 6h.
i verify that the difference is 1, -1 for uint8 (range = 0 to 255):
\nhttps://www.kaggle.com/code/hengck23/notebook8875e48da3
", "rawMarkdown": "i verify that the difference is 1, -1 for uint8 (range = 0 to 255):\n\nhttps://www.kaggle.com/code/hengck23/notebook8875e48da3", "votes": 1 }, { "id": 2060704, "postDate": "2022-12-10T10:28:43.143Z", "content": "I see (good comparision) - difference is tiny (if we can say such way in ML π). It should not influence on our data.
", "rawMarkdown": "I see (good comparision) - difference is tiny (if we can say such way in ML π). It should not influence on our data." } ] }, { "id": 2093811, "postDate": "2023-01-10T10:38:24.693Z", "content": "Hi there!
\nThank you @remekkinas for finding this awesome DicomSDL!
Now, I might be wrong, but I think that the only problem with DicomSDL is that it misses the apply_voi_lut (or in this competition's case), apply_windowing functions that Pydicom has.
\nI've dug into the source code of Pydicom to see what happens exactly, and I've posted some results here.
\nI think that if you use DicomSDL, you should check it out! :)
", "rawMarkdown": "Hi there!\nThank you @remekkinas for finding this awesome DicomSDL!\n\nNow, I might be wrong, but I think that the only problem with DicomSDL is that it misses the apply_voi_lut (or in this competition's case), apply_windowing functions that Pydicom has.\n\nI've dug into the source code of Pydicom to see what happens exactly, and I've posted some [results here](https://www.kaggle.com/code/bobdegraaf/dicomsdl-voi-lut).\n\nI think that if you use DicomSDL, you should check it out! :)", "votes": 3, "replies": [ { "id": 2093813, "postDate": "2023-01-10T10:42:02.333Z", "content": "Thank you for your contribution. Yes, you are right - I was trying to apply lut but with no effect. Thank you for providing solution. π
", "rawMarkdown": "Thank you for your contribution. Yes, you are right - I was trying to apply lut but with no effect. Thank you for providing solution. π" } ] }, { "id": 2058156, "postDate": "2022-12-07T16:35:24.787Z", "content": "All credit goes to @alenic .. nice call out! This is what makes Kaggle great.
\nIt's worth noting that dicomsdl decompresses into 2x the amount of bytes that pidicom does. Does this mean that dicomsdl is more accurate or more noisy? Or is there some redundancy in there?
\nst = time.time()\npa = dset.pixelData()\n(time.time() - st, pa.nbytes)\nst = time.time()\nimage_dataset = dcmread(f).pixel_array\n(time.time() - st, image_dataset.nbytes*)\n1.2884056568145752 105279300
\n1.5564007759094238 105279300
I understad. I changed in description.
\nBut β¦ your work is really awesome. This was nightmare to process files more then 60% of submission time.
\nGreat work guys!
Hi,
\nI just tried it for my submission (that failed because I saved the submission.csv in the wrong file π) and it took 4 hours with your script. With the previous script I used with pydicom it took 6-7 hours. So there indeed is a very nice improvement !
\nThanks !
", "rawMarkdown": "Hi, \n\nI just tried it for my submission (that failed because I saved the submission.csv in the wrong file π) and it took 4 hours with your script. With the previous script I used with pydicom it took 6-7 hours. So there indeed is a very nice improvement !\n\nThanks !", "votes": 4, "replies": [ { "id": 2060679, "postDate": "2022-12-10T09:40:51.373Z", "content": "Good to know! I managed to save a lot of time in my inference script as well. Look it works and we have a lot of time for next stages of pipeline.
", "rawMarkdown": "Good to know! I managed to save a lot of time in my inference script as well. Look it works and we have a lot of time for next stages of pipeline.", "votes": 1 } ] }, { "id": 2091919, "postDate": "2023-01-08T21:09:11.683Z", "content": "I tried my preprocessing with dicomsdl and the loading is really 1.6 times faster! Thank you @remekkinas
\nUnfortunately I also wanted to use the functions from pydicom apply_windowing/apply_voi_lut but they do not work with the files loaded with dicomsdl. The dicomsdl library doesn't contain the code for the similar functions. π
\nDid you manage to use the windowing function or did you write a custom one?
It does not support. I was experimenting with this but unfortunately have not managed to apply windowing. π¬
", "rawMarkdown": "It does not support. I was experimenting with this but unfortunately have not managed to apply windowing. π¬", "replies": [ { "id": 2091957, "postDate": "2023-01-08T22:10:18.107Z", "content": "\nLooking at the source code here it looks pretty easy to re-implement. You can use dicomsdl to grab the same attributes and do the calculation. I was able to do it within the same kernel I released before and build on top of it. I fed attributes through and did the calculation on the GPU. The great think about DALI is that I am able to write python functions using CuPy which is basically NumPy but GPU-accelerated so all the data remains on the GPU. I saw only a little minor decrease in throughput.
\nBest of luck!
", "rawMarkdown": "@paulbacher @remekkinas \n\nLooking at the [source code here](https://github.com/pydicom/pydicom/blob/54c5e493a299e4853bfb46de2e0ddfdd0d285e92/pydicom/pixel_data_handlers/util.py#L468) it looks pretty easy to re-implement. You can use dicomsdl to grab the same attributes and do the calculation. I was able to do it within the [same kernel](https://www.kaggle.com/code/outwrest/yolov5-roi-batch-dali-preprocessing-pipeline) I released before and build on top of it. I fed attributes through and did the calculation on the GPU. The great think about DALI is that I am able to write python functions using CuPy which is basically NumPy but GPU-accelerated so all the data remains on the GPU. I saw only a little minor decrease in throughput.\n\nBest of luck!", "votes": 3 }, { "id": 2091965, "postDate": "2023-01-08T22:25:56.473Z", "content": "# dicomsdl reader\ndef normalised_to_8bit(image, photometric_interpretation):\n xmin = image.min()\n xmax = image.max() \n norm = np.empty_like(image, dtype=np.uint8)\n dicomsdl.util.convert_to_uint8(image, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n return norm\n\ndef dicomsdl_to_numpy_image(ds, index=0): \n info = ds.getPixelDataInfo()\n if info['SamplesPerPixel'] != 1:\n raise RuntimeError('SamplesPerPixel != 1')\n shape = [info['Rows'], info['Cols']]\n dtype = info['dtype']\n outarr = np.empty(shape, dtype=dtype)\n ds.copyFrameData(index, outarr)\n return outarr\n\ndef dicomsdl_parallel_process(d, dcm_dir, image_dir, image_height, is_voi_lut):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = dicomsdl.open(dcm_file)\n image = dicomsdl_to_numpy_image(ds)\n if is_voi_lut:\n dc = pydicom.dcmread(dcm_file)\n image = apply_voi_lut(image, dc)\n image = image.astype(np.float32)\n image = normalised_to_8bit(image, ds.PhotometricInterpretation) # +1\n\n # resize and save as png\n ...\n\nParallel(n_jobs=n_jobs)(\n delayed(dicomsdl_parallel_process)(d, dcm_dir, image_dir, image_height, is_voi_lut)\n for t,d in tqdm(df.iterrows())\n )\nGreat! Thank you! π
", "rawMarkdown": "Great! Thank you! π" }, { "id": 2092475, "postDate": "2023-01-09T11:07:23.117Z", "content": "@outwrest Thank you, this is what I did and it's solved π
\n@hengck23 Thank you for this sharing π
You can look at my notebook having multiple packages (latest) for offline use. I have added \"dicomsdl\" to it as well.
\nHere is my notebook
", "rawMarkdown": "You can look at my notebook having multiple packages (latest) for offline use. I have added \"dicomsdl\" to it as well.\n\nHere is my [notebook](https://www.kaggle.com/code/hey24sheep/frozen-packages-for-offline-use)", "votes": 1 }, { "id": 2058198, "postDate": "2022-12-07T17:14:44.993Z", "content": "I think you also should include normalization in the dicomsdl function, so the images match per-pixel, but maybe it's unnecesary since the processing after JPEG decoding takes little time, nice!
", "rawMarkdown": "I think you also should include normalization in the dicomsdl function, so the images match per-pixel, but maybe it's unnecesary since the processing after JPEG decoding takes little time, nice!", "votes": 1, "replies": [ { "id": 2060705, "postDate": "2022-12-10T10:30:18.007Z", "content": "@martynoveduard than you for your commend. Exactly! I changed it in new notebook.
\nGood submission! Great score!
What do you think about further speeding it up by using an CPU notebook and therefore doubling the number of workers? The inference should be reasonably fast on a CPU tooβ¦.
", "rawMarkdown": "What do you think about further speeding it up by using an CPU notebook and therefore doubling the number of workers? The inference should be reasonably fast on a CPU too....", "votes": 1, "replies": [ { "id": 2060680, "postDate": "2022-12-10T09:43:18.430Z", "content": "I checked it - you are right CPU allow for 4 concurent processes. It speed up file processing twice but β¦ model blending and TTA will require a lot of GPU resources to process 32.000 images. I will stay with GPU.
", "rawMarkdown": "I checked it - you are right CPU allow for 4 concurent processes. It speed up file processing twice but ... model blending and TTA will require a lot of GPU resources to process 32.000 images. I will stay with GPU." } ] }, { "id": 2160767, "postDate": "2023-02-27T00:56:26.923Z", "content": "Hi there!
\nThank you @remekkinas for finding this awesome DicomSDL!
\nI have just one question and any help well be highly appreciated!
When I run '!pip list' in the cell, it shows that 'dicomsdl' was successfully downloaded and was one of my python site-packages; however, when I run the whole notebook by clicking either 'save version' or 'submit', it always raiseed: ModuleNotFoundError: No module named 'dicomsdl'.
\nI wonder how you solve this problem? Thank you so much!
", "rawMarkdown": "Hi there!\nThank you @remekkinas for finding this awesome DicomSDL!\nI have just one question and any help well be highly appreciated! \n\nWhen I run '!pip list' in the cell, it shows that 'dicomsdl' was successfully downloaded and was one of my python site-packages; however, when I run the whole notebook by clicking either 'save version' or 'submit', it always raiseed: ModuleNotFoundError: No module named 'dicomsdl'.\n\nI wonder how you solve this problem? Thank you so much!" }, { "id": 2058251, "postDate": "2022-12-07T18:27:39.133Z", "content": "Hey Remek, what do you think about a pipeline which utilizes both GPU and processors at the same time?
\nMy thinking is that with the current pipelines being used, we are idle for several hours on the GPU so we aren't optimizing our compute usage.
\neg:
\nhttps://www.kaggle.com/code/vslaykovsky/infer-effnetv2-aux-targets-weighted-loss-thres
\nThis hits the job lib and doesn't do anything else until all the images have been resized.
\nI think the architecture would be something like a job pool filled with conversion / augmentation / feeding images to the GPU.
\nI'm not a big GPU expert so it'd be great if folks can chime in here. I know there is a IO bottleneck with the pipeline between cpu and GPU, and there are also bottlenecks in CPU memory, but given the disk IO happening I don't think the latter is too much of an issue. Not sure what kind of bottleneck can occur between sending data to GPU / decoding at the same time. CPU caches can be problematic when it comes to performance.
\nAt the very least, if done right, this submission time should be <= CPU usage time, with all GPU time being free as its happening during CPU utilization.
\nHow much of a win that would be, I'm not sure.
", "rawMarkdown": "Hey Remek, what do you think about a pipeline which utilizes both GPU and processors at the same time?\n\nMy thinking is that with the current pipelines being used, we are idle for several hours on the GPU so we aren't optimizing our compute usage. \n\neg:\n\nhttps://www.kaggle.com/code/vslaykovsky/infer-effnetv2-aux-targets-weighted-loss-thres\n\nThis hits the job lib and doesn't do anything else until all the images have been resized.\n\n\nI think the architecture would be something like a job pool filled with conversion / augmentation / feeding images to the GPU. \n\nI'm not a big GPU expert so it'd be great if folks can chime in here. I know there is a IO bottleneck with the pipeline between cpu and GPU, and there are also bottlenecks in CPU memory, but given the disk IO happening I don't think the latter is too much of an issue. Not sure what kind of bottleneck can occur between sending data to GPU / decoding at the same time. CPU caches can be problematic when it comes to performance.\n\nAt the very least, if done right, this submission time should be <= CPU usage time, with all GPU time being free as its happening during CPU utilization.\n\nHow much of a win that would be, I'm not sure.\n\n", "replies": [ { "id": 2058269, "postDate": "2022-12-07T18:46:55.680Z", "content": "nvidia has dali, which seems to be along these lines.
\nhttps://developer.nvidia.com/blog/rapid-data-pre-processing-with-nvidia-dali/
\nmore about dali from our friends at tds:
\nhttps://towardsdatascience.com/overcoming-data-preprocessing-bottlenecks-with-tensorflow-data-service-nvidia-dali-and-other-d6321917f851
\n\nA CPU bottleneck occurs when the GPU resource is under utilized as a result of one, or more of the CPUs, having reached maximum utilization. In this situation, the GPU will be partially idle while it waits for the CPU to pass in training data. This is an undesired state. Being that the GPU is, typically, the most expensive resource in the system, your goal should always be to maximize its utilization.
\n
notebook here -
\nhttps://www.kaggle.com/code/hirune924/nvidia-dali-the-fastest-data-loading
I am working on it. First test is promising π
", "rawMarkdown": "I am working on it. First test is promising π", "votes": 1 }, { "id": 2058346, "postDate": "2022-12-07T20:34:24.263Z", "content": "That's really awesome. Should be cool. I know psi was concerned about this. Not sure if he's using dali.
", "rawMarkdown": "That's really awesome. Should be cool. I know psi was concerned about this. Not sure if he's using dali." }, { "id": 2058386, "postDate": "2022-12-07T21:51:23.860Z", "content": "Now I am prototyping using Ray.
\nI will look into Dali for sure.
if you want to keep the previous code the same:
\n # https://github.com/tsangel/dicomsdl/blob/master/tutorials/timeit_test.ipynb\n dicom = pydicom.dcmread(f)\n image0 = dicom.pixel_array\n\n dicom1 = dicomsdl.open(f)\n image1 = dicom1.pixelData(storedvalue=True)\n\n #---- check\n diff = image0.astype(np.float32)-image1.astype(np.float32)\n print(diff.max(), diff.mean())\n\n plt.scatter(image1.reshape(-1), image0.reshape(-1))\n plt.show()\nto improve cpu to gpu transfer (e.g. for large 1024 or 2048) at inference, use 8bit byte and single channel.
\nconvert to float() and expand to 3 channel in your model forward function (which is in cuda)
refer to python/dicomsdl/__init__.py, you can modify
\n/python/dicomsdl/__init__.py\ndef __dataset__to_pil_image(self, index=0):\nor better still, use
\n__dataset__pixelData__\n\n storedvalue (bool): True for get stored values; pixel values before LUT\n transformation using RescaleSlope and RescaleIntercept.\n Returns:\n Numpy array containing pixel values of `index`'th image if dataset holds\n multiframe data. If `storedvalue` is False, RescaleSlope and\n RescaleIntercept are applied to pixel values.\n ( pixel values = stored values * RescsaleSlope + RescaleIntercept )\nfurther, the function below is c++ code and may be faster than numpy?
\nutil.convert_to_uint8(outarr, data8, xmin, xmax)\nThank you for your feedback - I will look into code and improve it.
\nNow (new version of notebook) it seems for me that images generated by dicomsdl are close to the pydicom - but this is observation based on model scoring only (test validation for both - pydicom and discomsdl). It increased speed significantly so now even 6 models (eg. resnet50d) on blend finished in 6h.
i verify that the difference is 1, -1 for uint8 (range = 0 to 255):
\nhttps://www.kaggle.com/code/hengck23/notebook8875e48da3
", "votes": 1, "replies": [] }, { "id": 2060704, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-10T10:28:43.143000", "content": "I see (good comparision) - difference is tiny (if we can say such way in ML π). It should not influence on our data.
", "votes": 0, "replies": [] } ] }, { "id": 2093811, "author_name": "Bob de Graaf", "author_url": "", "post_date": "2023-01-10T10:38:24.693000", "content": "Hi there!
\nThank you @remekkinas for finding this awesome DicomSDL!
Now, I might be wrong, but I think that the only problem with DicomSDL is that it misses the apply_voi_lut (or in this competition's case), apply_windowing functions that Pydicom has.
\nI've dug into the source code of Pydicom to see what happens exactly, and I've posted some results here.
\nI think that if you use DicomSDL, you should check it out! :)
", "votes": 3, "replies": [ { "id": 2093813, "author_name": "Remek Kinas", "author_url": "", "post_date": "2023-01-10T10:42:02.333000", "content": "Thank you for your contribution. Yes, you are right - I was trying to apply lut but with no effect. Thank you for providing solution. π
", "votes": 0, "replies": [] } ] }, { "id": 2058156, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-07T16:35:24.787000", "content": "All credit goes to @alenic .. nice call out! This is what makes Kaggle great.
\nIt's worth noting that dicomsdl decompresses into 2x the amount of bytes that pidicom does. Does this mean that dicomsdl is more accurate or more noisy? Or is there some redundancy in there?
\nst = time.time()\npa = dset.pixelData()\n(time.time() - st, pa.nbytes)\nst = time.time()\nimage_dataset = dcmread(f).pixel_array\n(time.time() - st, image_dataset.nbytes*)\n1.2884056568145752 105279300
\n1.5564007759094238 105279300
I understad. I changed in description.
\nBut β¦ your work is really awesome. This was nightmare to process files more then 60% of submission time.
\nGreat work guys!
Hi,
\nI just tried it for my submission (that failed because I saved the submission.csv in the wrong file π) and it took 4 hours with your script. With the previous script I used with pydicom it took 6-7 hours. So there indeed is a very nice improvement !
\nThanks !
", "votes": 4, "replies": [ { "id": 2060679, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-10T09:40:51.373000", "content": "Good to know! I managed to save a lot of time in my inference script as well. Look it works and we have a lot of time for next stages of pipeline.
", "votes": 1, "replies": [] } ] }, { "id": 2091919, "author_name": "Paul Bacher", "author_url": "", "post_date": "2023-01-08T21:09:11.683000", "content": "I tried my preprocessing with dicomsdl and the loading is really 1.6 times faster! Thank you @remekkinas
\nUnfortunately I also wanted to use the functions from pydicom apply_windowing/apply_voi_lut but they do not work with the files loaded with dicomsdl. The dicomsdl library doesn't contain the code for the similar functions. π
\nDid you manage to use the windowing function or did you write a custom one?
It does not support. I was experimenting with this but unfortunately have not managed to apply windowing. π¬
", "votes": 0, "replies": [ { "id": 2091957, "author_name": "outwrest", "author_url": "", "post_date": "2023-01-08T22:10:18.107000", "content": "\nLooking at the source code here it looks pretty easy to re-implement. You can use dicomsdl to grab the same attributes and do the calculation. I was able to do it within the same kernel I released before and build on top of it. I fed attributes through and did the calculation on the GPU. The great think about DALI is that I am able to write python functions using CuPy which is basically NumPy but GPU-accelerated so all the data remains on the GPU. I saw only a little minor decrease in throughput.
\nBest of luck!
", "votes": 3, "replies": [] }, { "id": 2091965, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-08T22:25:56.473000", "content": "# dicomsdl reader\ndef normalised_to_8bit(image, photometric_interpretation):\n xmin = image.min()\n xmax = image.max() \n norm = np.empty_like(image, dtype=np.uint8)\n dicomsdl.util.convert_to_uint8(image, norm, xmin, xmax)\n if photometric_interpretation == 'MONOCHROME1':\n norm = 255 - norm\n return norm\n\ndef dicomsdl_to_numpy_image(ds, index=0): \n info = ds.getPixelDataInfo()\n if info['SamplesPerPixel'] != 1:\n raise RuntimeError('SamplesPerPixel != 1')\n shape = [info['Rows'], info['Cols']]\n dtype = info['dtype']\n outarr = np.empty(shape, dtype=dtype)\n ds.copyFrameData(index, outarr)\n return outarr\n\ndef dicomsdl_parallel_process(d, dcm_dir, image_dir, image_height, is_voi_lut):\n dcm_file = f'{dcm_dir}/{d.patient_id}/{d.image_id}.dcm'\n ds = dicomsdl.open(dcm_file)\n image = dicomsdl_to_numpy_image(ds)\n if is_voi_lut:\n dc = pydicom.dcmread(dcm_file)\n image = apply_voi_lut(image, dc)\n image = image.astype(np.float32)\n image = normalised_to_8bit(image, ds.PhotometricInterpretation) # +1\n\n # resize and save as png\n ...\n\nParallel(n_jobs=n_jobs)(\n delayed(dicomsdl_parallel_process)(d, dcm_dir, image_dir, image_height, is_voi_lut)\n for t,d in tqdm(df.iterrows())\n )\nGreat! Thank you! π
", "votes": 0, "replies": [] }, { "id": 2092475, "author_name": "Paul Bacher", "author_url": "", "post_date": "2023-01-09T11:07:23.117000", "content": "@outwrest Thank you, this is what I did and it's solved π
\n@hengck23 Thank you for this sharing π
You can look at my notebook having multiple packages (latest) for offline use. I have added \"dicomsdl\" to it as well.
\nHere is my notebook
", "votes": 1, "replies": [] }, { "id": 2058198, "author_name": "slime", "author_url": "", "post_date": "2022-12-07T17:14:44.993000", "content": "I think you also should include normalization in the dicomsdl function, so the images match per-pixel, but maybe it's unnecesary since the processing after JPEG decoding takes little time, nice!
", "votes": 1, "replies": [ { "id": 2060705, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-10T10:30:18.007000", "content": "@martynoveduard than you for your commend. Exactly! I changed it in new notebook.
\nGood submission! Great score!
What do you think about further speeding it up by using an CPU notebook and therefore doubling the number of workers? The inference should be reasonably fast on a CPU tooβ¦.
", "votes": 1, "replies": [ { "id": 2060680, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-10T09:43:18.430000", "content": "I checked it - you are right CPU allow for 4 concurent processes. It speed up file processing twice but β¦ model blending and TTA will require a lot of GPU resources to process 32.000 images. I will stay with GPU.
", "votes": 0, "replies": [] } ] }, { "id": 2160767, "author_name": "Eric006", "author_url": "", "post_date": "2023-02-27T00:56:26.923000", "content": "Hi there!
\nThank you @remekkinas for finding this awesome DicomSDL!
\nI have just one question and any help well be highly appreciated!
When I run '!pip list' in the cell, it shows that 'dicomsdl' was successfully downloaded and was one of my python site-packages; however, when I run the whole notebook by clicking either 'save version' or 'submit', it always raiseed: ModuleNotFoundError: No module named 'dicomsdl'.
\nI wonder how you solve this problem? Thank you so much!
", "votes": 0, "replies": [] }, { "id": 2058251, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-07T18:27:39.133000", "content": "Hey Remek, what do you think about a pipeline which utilizes both GPU and processors at the same time?
\nMy thinking is that with the current pipelines being used, we are idle for several hours on the GPU so we aren't optimizing our compute usage.
\neg:
\nhttps://www.kaggle.com/code/vslaykovsky/infer-effnetv2-aux-targets-weighted-loss-thres
\nThis hits the job lib and doesn't do anything else until all the images have been resized.
\nI think the architecture would be something like a job pool filled with conversion / augmentation / feeding images to the GPU.
\nI'm not a big GPU expert so it'd be great if folks can chime in here. I know there is a IO bottleneck with the pipeline between cpu and GPU, and there are also bottlenecks in CPU memory, but given the disk IO happening I don't think the latter is too much of an issue. Not sure what kind of bottleneck can occur between sending data to GPU / decoding at the same time. CPU caches can be problematic when it comes to performance.
\nAt the very least, if done right, this submission time should be <= CPU usage time, with all GPU time being free as its happening during CPU utilization.
\nHow much of a win that would be, I'm not sure.
", "votes": 0, "replies": [ { "id": 2058269, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-07T18:46:55.680000", "content": "nvidia has dali, which seems to be along these lines.
\nhttps://developer.nvidia.com/blog/rapid-data-pre-processing-with-nvidia-dali/
\nmore about dali from our friends at tds:
\nhttps://towardsdatascience.com/overcoming-data-preprocessing-bottlenecks-with-tensorflow-data-service-nvidia-dali-and-other-d6321917f851
\n\nA CPU bottleneck occurs when the GPU resource is under utilized as a result of one, or more of the CPUs, having reached maximum utilization. In this situation, the GPU will be partially idle while it waits for the CPU to pass in training data. This is an undesired state. Being that the GPU is, typically, the most expensive resource in the system, your goal should always be to maximize its utilization.
\n
notebook here -
\nhttps://www.kaggle.com/code/hirune924/nvidia-dali-the-fastest-data-loading
I am working on it. First test is promising π
", "votes": 1, "replies": [] }, { "id": 2058346, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-07T20:34:24.263000", "content": "That's really awesome. Should be cool. I know psi was concerned about this. Not sure if he's using dali.
", "votes": 0, "replies": [] }, { "id": 2058386, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-07T21:51:23.860000", "content": "Now I am prototyping using Ray.
\nI will look into Dali for sure.