![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2Fdbc2480e7f8863133c72cc37cc9f6b81%2Fgcp_vm_at_work.png?generation=1669731617602415&alt=media\")
Hey!
\nPlease find the DICOM files converted to PNG here: RSNA Mammography -- images as PNGs (256px x 256px)
\nEDIT: Based on a request by @remekkinas I created a new version of the dataset that also contains 512px x 512px images!
\nI tried converting the files on Kaggle, but the VM was just not enough π Even though the processing completed after more than ~3 hrs using the 4 available cores, I still didn't manage to commit the output.
\nOpted to go for a GCP VM to get the job done π
\n
On GCP the conversion took just under 17 minutes using 64 cores and multiprocessing. Please note, I converted the images in a way that accounts for dicom.PhotometricInterpretation == \"MONOCHROME1\" where the intensities need to be inverted.
I converted DICOM files to PNG images 256px by 256px. The folder structure is the same as of the DICOM files.
\nHere are a couple of examples:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F4fc6e1f8837c3be387dd45077bf7aa7f%2F2057295788.png?generation=1669735207946955&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2Ff0bd8462a9d27d8049185f6039d3536e%2F2046475482.png?generation=1669735227410985&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2Ff155f4f11b2e5d8e836423bf5f30c7a5%2F349510516.png?generation=1669735239725229&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2Ff1d5dd2efa0bbbd68d061f9aad6cd868%2F1365269360.png?generation=1669735252065645&alt=media\")
Happy Kaggling! π
\nHey!
\nPlease find the DICOM files converted to PNG here: RSNA Mammography -- images as PNGs (256px x 256px)
\nEDIT: Based on a request by @remekkinas I created a new version of the dataset that also contains 512px x 512px images!
\nI tried converting the files on Kaggle, but the VM was just not enough π Even though the processing completed after more than ~3 hrs using the 4 available cores, I still didn't manage to commit the output.
\nOpted to go for a GCP VM to get the job done π
\n
On GCP the conversion took just under 17 minutes using 64 cores and multiprocessing. Please note, I converted the images in a way that accounts for dicom.PhotometricInterpretation == \"MONOCHROME1\" where the intensities need to be inverted.
I converted DICOM files to PNG images 256px by 256px. The folder structure is the same as of the DICOM files.
\nHere are a couple of examples:
\n
Happy Kaggling! π
\nExercise caution when converting from DICOM to PNG. Mammograms are very high resolution (12MP on average) and downsampling heavily may render small findings invisible. 8-bit is sufficient for most cases but 12 and 16-bit should be considered as well. Run small experiments to understand the importance.
\nCropping the mamogram to remove background tissue is a good strategy but consider the implications of padding to achieve the same image size rather than distorting the aspect ratio. This may change the appearance of fine calcifications.
\nGood luck!
", "rawMarkdown": "Exercise caution when converting from DICOM to PNG. Mammograms are very high resolution (12MP on average) and downsampling heavily may render small findings invisible. 8-bit is sufficient for most cases but 12 and 16-bit should be considered as well. Run small experiments to understand the importance.\n\nCropping the mamogram to remove background tissue is a good strategy but consider the implications of padding to achieve the same image size rather than distorting the aspect ratio. This may change the appearance of fine calcifications.\n\nGood luck!", "votes": 18 }, { "id": 2056988, "postDate": "2022-12-06T16:33:12.897Z", "content": "As this information might get unnoticed, if you have been using this dataset, please be aware of:
\nhttps://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs/comments#2056978
", "rawMarkdown": "As this information might get unnoticed, if you have been using this dataset, please be aware of:\n\nhttps://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs/comments#2056978", "votes": 3 }, { "id": 2048805, "postDate": "2022-11-29T17:28:04.683Z", "content": "Nic work Radek! I will use your work.
\nI thnink that in case of 256x256 β¦ we should create more informative dataset - remove background (additional pixels). Look on second image and last one - we can crop it a little bit. Anyway β¦ it could be done in augumentation pipeline as well so do not need any work here.
Radek it would be cool if you provide 512x512 datset. Is any chance you create such dataset?
", "rawMarkdown": "Nic work Radek! I will use your work. \nI thnink that in case of 256x256 ... we should create more informative dataset - remove background (additional pixels). Look on second image and last one - we can crop it a little bit. Anyway ... it could be done in augumentation pipeline as well so do not need any work here.\n\nRadek it would be cool if you provide 512x512 datset. Is any chance you create such dataset?", "votes": 3, "replies": [ { "id": 2049028, "postDate": "2022-11-29T21:56:27.950Z", "content": "https://www.kaggle.com/datasets/theoviel/rsna-breast-cancer-512-pngs =)
", "rawMarkdown": "https://www.kaggle.com/datasets/theoviel/rsna-breast-cancer-512-pngs =)", "votes": 6 }, { "id": 2049084, "postDate": "2022-11-29T23:37:13.553Z", "content": "Cool @theoviel, thx for sharing!
\n@remekkinas, super glad you are finding this useful! π Just to make sure you have both sizes of images processed in the same way, I have now added images scaled to 512 x 512 to the dataset! π
", "rawMarkdown": "Cool @theoviel, thx for sharing!\n\n@remekkinas, super glad you are finding this useful! π Just to make sure you have both sizes of images processed in the same way, I have now added images scaled to 512 x 512 to the dataset! π" }, { "id": 2049882, "postDate": "2022-11-30T12:02:19.787Z", "content": "@theoviel thank you! You helped a lot.
\n@radek1 cool! Working on solution :)
", "rawMarkdown": "@theoviel thank you! You helped a lot. \n\n@radek1 cool! Working on solution :)", "votes": 1 }, { "id": 2050574, "postDate": "2022-11-30T21:03:17.587Z", "content": "Awesome, @remekkinas! π Crossing my fingers and curious about what you'll be able to cook up! Best of luck!!! π
", "rawMarkdown": "Awesome, @remekkinas! π Crossing my fingers and curious about what you'll be able to cook up! Best of luck!!! π ", "votes": 1 }, { "id": 2051560, "postDate": "2022-12-01T13:54:43.037Z", "content": "@remekkinas I was thinking about the augumentation as well. However, I just thought doing it by rotation / reflection, but i wonder whether the model was supposed to fit only the regular Mammography image. The same also apply to removing the background. Just curious for discussion.
\nLooking forward to your work.
", "rawMarkdown": "@remekkinas I was thinking about the augumentation as well. However, I just thought doing it by rotation / reflection, but i wonder whether the model was supposed to fit only the regular Mammography image. The same also apply to removing the background. Just curious for discussion. \n\nLooking forward to your work." }, { "id": 2051573, "postDate": "2022-12-01T13:58:26.443Z", "content": "Augumentation will be important here (as usual in CV task). Removing background is not important in my opinion (I will use ROI cropped images). We will see β¦ we have many possibilities in this competition.
", "rawMarkdown": "Augumentation will be important here (as usual in CV task). Removing background is not important in my opinion (I will use ROI cropped images). We will see ... we have many possibilities in this competition.", "votes": 3 }, { "id": 2051578, "postDate": "2022-12-01T14:01:18.290Z", "content": "The big component here is being able to train on more information with less compute⦠that is what the awesome work that you did @remekkinas helps a lot!
\nIf we go from 512x512 but removing the regions that carry no signal to 256x256 we decrease the input size not by 2x⦠but by 4x! And this gets compounded down the road.
", "rawMarkdown": "The big component here is being able to train on more information with less compute... that is what the awesome work that you did @remekkinas helps a lot!\n\nIf we go from 512x512 but removing the regions that carry no signal to 256x256 we decrease the input size not by 2x... but by 4x! And this gets compounded down the road. ", "votes": 1 } ] }, { "id": 2054262, "postDate": "2022-12-04T01:51:29.180Z", "content": "thanks so much bro!
", "rawMarkdown": "thanks so much bro!", "votes": 1, "replies": [ { "id": 2054290, "postDate": "2022-12-04T02:19:57.093Z", "content": "pleasure, @ayaan! π
", "rawMarkdown": "pleasure, @ayaan! π" } ] }, { "id": 2049845, "postDate": "2022-11-30T11:26:30.683Z", "content": "Thanks radek,
\nCan you pls share the code you've used to do this like any preprocessing is done by you or not.
Hey @level14taken! Sure thing, no problem!
\nPleas find the code here: π‘ how I processed DICOM images to PNGs
\nAll the best!
", "rawMarkdown": "Hey @level14taken! Sure thing, no problem!\n\nPleas find the code here: [π‘ how I processed DICOM images to PNGs](https://www.kaggle.com/code/radek1/how-i-processed-dicom-images-to-pngs?scriptVersionId=112592164)\n\nAll the best!", "votes": 2 }, { "id": 2051558, "postDate": "2022-12-01T13:51:11.070Z", "content": "@radek1 Thank you for your sharing. Sorry but the link is not working.
", "rawMarkdown": "@radek1 Thank you for your sharing. Sorry but the link is not working.", "votes": 1 }, { "id": 2051564, "postDate": "2022-12-01T13:56:40.103Z", "content": "oh no, I didn't make it publicβ¦
\nThank you @jackysywk for letting me know π Should now be fixed, can you confirm, please?
", "rawMarkdown": "oh no, I didn't make it public...\n\nThank you @jackysywk for letting me know π Should now be fixed, can you confirm, please?", "votes": 1 }, { "id": 2051592, "postDate": "2022-12-01T14:08:12.780Z", "content": "It now works perfectly. Thanks π―π @radek1
", "rawMarkdown": "It now works perfectly. Thanks π―π @radek1 ", "votes": 1 }, { "id": 2051599, "postDate": "2022-12-01T14:11:35.990Z", "content": "Wonderful to hear! Thank you for checking and letting me know, @jackysywk! π
", "rawMarkdown": "Wonderful to hear! Thank you for checking and letting me know, @jackysywk! π ", "votes": 1 } ] }, { "id": 2049359, "postDate": "2022-11-30T04:38:03.310Z", "content": "Nice work~~~ Thanks!
", "rawMarkdown": "Nice work~~~ Thanks!", "votes": 1, "replies": [ { "id": 2051567, "postDate": "2022-12-01T13:57:04.653Z", "content": "np @gavinxuai, my pleasure! π
", "rawMarkdown": "np @gavinxuai, my pleasure! π" } ] }, { "id": 2049287, "postDate": "2022-11-30T03:16:51.507Z", "content": "Would really help if you can make 768x768 and 1024x1024 as well. They can be used with certain transformer models
", "rawMarkdown": "Would really help if you can make 768x768 and 1024x1024 as well. They can be used with certain transformer models", "votes": 1, "replies": [ { "id": 2051562, "postDate": "2022-12-01T13:56:09.793Z", "content": "@nishantbhansali Wonderful idea
\nGrateful if you can check the link above and i can try doing other pixels, if this is not hardware demanding.
Hey! these have been added to the dataset! π happy scaling up to bigger inputs! π
", "rawMarkdown": "Hey! these have been added to the dataset! π happy scaling up to bigger inputs! π", "votes": 1 } ] }, { "id": 2159958, "postDate": "2023-02-26T07:31:31.203Z", "content": "Three days ago, I could run image.dicom.pixel_array to decompress a Dicom image. At this point, I get the error \"The pixel data with transfer syntax JPEG 2000 Image Compression (Lossless Only), cannot be read because Pillow lacks the JPEG 2000 plugin\". On my computer, it has been resolved by installing 'pip install -U pylibjpeg pylibjpeg-openjpeg pylibjpeg-libjpeg'.
\nIn Kaggle, it seems to install it in 'base environment' but it doesn't work, nor can I create another environment. Could someone help me please?
Hi @radek1 and all,
\nI have one naive question: Since we want to have the same distribution of both test and training data, don't we need to use the same data transformation pipeline during inference time? DICOM to PNGs, crop, etc. ?
", "rawMarkdown": "Hi @radek1 and all, \n\nI have one naive question: Since we want to have the same distribution of both test and training data, don't we need to use the same data transformation pipeline during inference time? DICOM to PNGs, crop, etc. ?" }, { "id": 2102074, "postDate": "2023-01-16T11:22:52.837Z", "content": "Wow! It will be very helpful, thanks bro :)
", "rawMarkdown": "Wow! It will be very helpful, thanks bro :)" }, { "id": 2057235, "postDate": "2022-12-06T21:52:43.297Z", "content": "This will not only help us to learn but to save space locally..π
", "rawMarkdown": "This will not only help us to learn but to save space locally..π " }, { "id": 2052541, "postDate": "2022-12-02T09:37:55.550Z", "content": "Hi there, don't get me wrong. I'm just new to these kind of competitions. is information loss when we convert DICOM images to PNGs ?
", "rawMarkdown": "Hi there, don't get me wrong. I'm just new to these kind of competitions. is information loss when we convert DICOM images to PNGs ?", "replies": [ { "id": 2054300, "postDate": "2022-12-04T02:26:36.843Z", "content": "hey! yes, it is always a question how to go from DICOM images to other formats π There has been some discussion on the forums, for a really good analysis please take a look here
", "rawMarkdown": "hey! yes, it is always a question how to go from DICOM images to other formats π There has been some discussion on the forums, for a really good analysis please take a look [here](https://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/114214?utm_source=pocket_saves)" }, { "id": 2054303, "postDate": "2022-12-04T02:32:17.147Z", "content": "okay bro, i will take a look. thank you very much.
", "rawMarkdown": "okay bro, i will take a look. thank you very much.\n" } ] }, { "id": 2048727, "postDate": "2022-11-29T16:29:50.213Z", "content": "Great work, one question is, what should we do when we need to submit prediction results?
", "rawMarkdown": "Great work, one question is, what should we do when we need to submit prediction results?", "replies": [ { "id": 2048801, "postDate": "2022-11-29T17:25:48.103Z", "content": "Nothing with images. You need to submit probability of cancer on particular photo.
\n256x256 is great for experimenting. Then we probably need more pixels and more GPU power.
Well, blending with lower resolution models might work. Need to experiment
", "rawMarkdown": "Well, blending with lower resolution models might work. Need to experiment" } ] }, { "id": 2070798, "postDate": "2022-12-20T10:53:29.970Z", "content": "thanks so much bro!
", "rawMarkdown": "thanks so much bro!" } ], "comments": [ { "id": 2056098, "author_name": "Hari T.", "author_url": "", "post_date": "2022-12-05T18:20:02.083000", "content": "Exercise caution when converting from DICOM to PNG. Mammograms are very high resolution (12MP on average) and downsampling heavily may render small findings invisible. 8-bit is sufficient for most cases but 12 and 16-bit should be considered as well. Run small experiments to understand the importance.
\nCropping the mamogram to remove background tissue is a good strategy but consider the implications of padding to achieve the same image size rather than distorting the aspect ratio. This may change the appearance of fine calcifications.
\nGood luck!
", "votes": 18, "replies": [] }, { "id": 2056988, "author_name": "raddar", "author_url": "", "post_date": "2022-12-06T16:33:12.897000", "content": "As this information might get unnoticed, if you have been using this dataset, please be aware of:
\nhttps://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs/comments#2056978
", "votes": 3, "replies": [] }, { "id": 2048805, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-11-29T17:28:04.683000", "content": "Nic work Radek! I will use your work.
\nI thnink that in case of 256x256 β¦ we should create more informative dataset - remove background (additional pixels). Look on second image and last one - we can crop it a little bit. Anyway β¦ it could be done in augumentation pipeline as well so do not need any work here.
Radek it would be cool if you provide 512x512 datset. Is any chance you create such dataset?
", "votes": 3, "replies": [ { "id": 2049028, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-11-29T21:56:27.950000", "content": "https://www.kaggle.com/datasets/theoviel/rsna-breast-cancer-512-pngs =)
", "votes": 6, "replies": [] }, { "id": 2049084, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-29T23:37:13.553000", "content": "Cool @theoviel, thx for sharing!
\n@remekkinas, super glad you are finding this useful! π Just to make sure you have both sizes of images processed in the same way, I have now added images scaled to 512 x 512 to the dataset! π
", "votes": 0, "replies": [] }, { "id": 2049882, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-11-30T12:02:19.787000", "content": "@theoviel thank you! You helped a lot.
\n@radek1 cool! Working on solution :)
", "votes": 1, "replies": [] }, { "id": 2050574, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-30T21:03:17.587000", "content": "Awesome, @remekkinas! π Crossing my fingers and curious about what you'll be able to cook up! Best of luck!!! π
", "votes": 1, "replies": [] }, { "id": 2051560, "author_name": "Jacky S", "author_url": "", "post_date": "2022-12-01T13:54:43.037000", "content": "@remekkinas I was thinking about the augumentation as well. However, I just thought doing it by rotation / reflection, but i wonder whether the model was supposed to fit only the regular Mammography image. The same also apply to removing the background. Just curious for discussion.
\nLooking forward to your work.
", "votes": 0, "replies": [] }, { "id": 2051573, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-01T13:58:26.443000", "content": "Augumentation will be important here (as usual in CV task). Removing background is not important in my opinion (I will use ROI cropped images). We will see β¦ we have many possibilities in this competition.
", "votes": 3, "replies": [] }, { "id": 2051578, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-01T14:01:18.290000", "content": "The big component here is being able to train on more information with less compute⦠that is what the awesome work that you did @remekkinas helps a lot!
\nIf we go from 512x512 but removing the regions that carry no signal to 256x256 we decrease the input size not by 2x⦠but by 4x! And this gets compounded down the road.
", "votes": 1, "replies": [] } ] }, { "id": 2054262, "author_name": "ayaan", "author_url": "", "post_date": "2022-12-04T01:51:29.180000", "content": "thanks so much bro!
", "votes": 1, "replies": [ { "id": 2054290, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-04T02:19:57.093000", "content": "pleasure, @ayaan! π
", "votes": 0, "replies": [] } ] }, { "id": 2049845, "author_name": "losingself", "author_url": "", "post_date": "2022-11-30T11:26:30.683000", "content": "Thanks radek,
\nCan you pls share the code you've used to do this like any preprocessing is done by you or not.
Hey @level14taken! Sure thing, no problem!
\nPleas find the code here: π‘ how I processed DICOM images to PNGs
\nAll the best!
", "votes": 2, "replies": [] }, { "id": 2051558, "author_name": "Jacky S", "author_url": "", "post_date": "2022-12-01T13:51:11.070000", "content": "@radek1 Thank you for your sharing. Sorry but the link is not working.
", "votes": 1, "replies": [] }, { "id": 2051564, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-01T13:56:40.103000", "content": "oh no, I didn't make it publicβ¦
\nThank you @jackysywk for letting me know π Should now be fixed, can you confirm, please?
", "votes": 1, "replies": [] }, { "id": 2051592, "author_name": "Jacky S", "author_url": "", "post_date": "2022-12-01T14:08:12.780000", "content": "It now works perfectly. Thanks π―π @radek1
", "votes": 1, "replies": [] }, { "id": 2051599, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-01T14:11:35.990000", "content": "Wonderful to hear! Thank you for checking and letting me know, @jackysywk! π
", "votes": 1, "replies": [] } ] }, { "id": 2049359, "author_name": "zhenghuoer", "author_url": "", "post_date": "2022-11-30T04:38:03.310000", "content": "Nice work~~~ Thanks!
", "votes": 1, "replies": [ { "id": 2051567, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-01T13:57:04.653000", "content": "np @gavinxuai, my pleasure! π
", "votes": 0, "replies": [] } ] }, { "id": 2049287, "author_name": "Nishant Bhansali", "author_url": "", "post_date": "2022-11-30T03:16:51.507000", "content": "Would really help if you can make 768x768 and 1024x1024 as well. They can be used with certain transformer models
", "votes": 1, "replies": [ { "id": 2051562, "author_name": "Jacky S", "author_url": "", "post_date": "2022-12-01T13:56:09.793000", "content": "@nishantbhansali Wonderful idea
\nGrateful if you can check the link above and i can try doing other pixels, if this is not hardware demanding.
Hey! these have been added to the dataset! π happy scaling up to bigger inputs! π
", "votes": 1, "replies": [] } ] }, { "id": 2159958, "author_name": "Eduardo Gasca Alvarez", "author_url": "", "post_date": "2023-02-26T07:31:31.203000", "content": "Three days ago, I could run image.dicom.pixel_array to decompress a Dicom image. At this point, I get the error \"The pixel data with transfer syntax JPEG 2000 Image Compression (Lossless Only), cannot be read because Pillow lacks the JPEG 2000 plugin\". On my computer, it has been resolved by installing 'pip install -U pylibjpeg pylibjpeg-openjpeg pylibjpeg-libjpeg'.
\nIn Kaggle, it seems to install it in 'base environment' but it doesn't work, nor can I create another environment. Could someone help me please?
Hi @radek1 and all,
\nI have one naive question: Since we want to have the same distribution of both test and training data, don't we need to use the same data transformation pipeline during inference time? DICOM to PNGs, crop, etc. ?
", "votes": 0, "replies": [] }, { "id": 2102074, "author_name": "Ncolmann", "author_url": "", "post_date": "2023-01-16T11:22:52.837000", "content": "Wow! It will be very helpful, thanks bro :)
", "votes": 0, "replies": [] }, { "id": 2057235, "author_name": "Saurav Solanki", "author_url": "", "post_date": "2022-12-06T21:52:43.297000", "content": "This will not only help us to learn but to save space locally..π
", "votes": 0, "replies": [] }, { "id": 2052541, "author_name": "nadhir hasan", "author_url": "", "post_date": "2022-12-02T09:37:55.550000", "content": "Hi there, don't get me wrong. I'm just new to these kind of competitions. is information loss when we convert DICOM images to PNGs ?
", "votes": 0, "replies": [ { "id": 2054300, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-04T02:26:36.843000", "content": "hey! yes, it is always a question how to go from DICOM images to other formats π There has been some discussion on the forums, for a really good analysis please take a look here
", "votes": 0, "replies": [] }, { "id": 2054303, "author_name": "nadhir hasan", "author_url": "", "post_date": "2022-12-04T02:32:17.147000", "content": "okay bro, i will take a look. thank you very much.
", "votes": 0, "replies": [] } ] }, { "id": 2048727, "author_name": "yanqiangmiffy", "author_url": "", "post_date": "2022-11-29T16:29:50.213000", "content": "Great work, one question is, what should we do when we need to submit prediction results?
", "votes": 0, "replies": [ { "id": 2048801, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-11-29T17:25:48.103000", "content": "Nothing with images. You need to submit probability of cancer on particular photo.
\n256x256 is great for experimenting. Then we probably need more pixels and more GPU power.
Well, blending with lower resolution models might work. Need to experiment
", "votes": 0, "replies": [] } ] }, { "id": 2070798, "author_name": "1! 5!", "author_url": "", "post_date": "2022-12-20T10:53:29.970000", "content": "thanks so much bro!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2048609": "Hey!\n\nPlease find the DICOM files converted to PNG here: [RSNA Mammography -- images as PNGs (256px x 256px)](https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs/settings)\n\n**EDIT: Based on a request by @remekkinas I created a new version of the dataset that also contains 512px x 512px images!**\n\n\nI tried converting the files on Kaggle, but the VM was just not enough π Even though the processing completed after more than ~3 hrs using the 4 available cores, I still didn't manage to commit the output.\n\nOpted to go for a GCP VM to get the job done π\n\n\n\nOn GCP the conversion took just under 17 minutes using 64 cores and multiprocessing. Please note, I converted the images in a way that accounts for `dicom.PhotometricInterpretation == \"MONOCHROME1\"` where the intensities need to be inverted.\n\nI converted DICOM files to PNG images 256px by 256px. The folder structure is the same as of the DICOM files.\n\nHere are a couple of examples:\n\n\n\n\n\n\n\n\n\nHappy Kaggling! π\n\n### Other resources you might find useful:\n\n* [π‘ how to process DICOM images to PNGs](https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs)\n* [π EDA + training a fast.ai model + submission π](https://www.kaggle.com/code/radek1/eda-training-a-fast-ai-model-submission)\n* [π€ [fast.ai starter pack] train + inference π](https://www.kaggle.com/code/radek1/fast-ai-starter-pack-train-inference)\n* [πΈ Over 56GB of processed data, 5 different methods π₯³](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/371268)\n* [3 resources to get started with Computer Vision in this competition πππ](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369706)\n* [π‘ 6 Computer Vision tricks for faster training and better models πππ](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369155)\n\n", "2056098": "Exercise caution when converting from DICOM to PNG. Mammograms are very high resolution (12MP on average) and downsampling heavily may render small findings invisible. 8-bit is sufficient for most cases but 12 and 16-bit should be considered as well. Run small experiments to understand the importance.\n\nCropping the mamogram to remove background tissue is a good strategy but consider the implications of padding to achieve the same image size rather than distorting the aspect ratio. This may change the appearance of fine calcifications.\n\nGood luck!", "2056988": "As this information might get unnoticed, if you have been using this dataset, please be aware of:\n\nhttps://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs/comments#2056978", "2048805": "Nic work Radek! I will use your work. \nI thnink that in case of 256x256 ... we should create more informative dataset - remove background (additional pixels). Look on second image and last one - we can crop it a little bit. Anyway ... it could be done in augumentation pipeline as well so do not need any work here.\n\nRadek it would be cool if you provide 512x512 datset. Is any chance you create such dataset?", "2054262": "thanks so much bro!", "2049845": "Thanks radek, \nCan you pls share the code you've used to do this like any preprocessing is done by you or not.", "2049359": "Nice work~~~ Thanks!", "2049287": "Would really help if you can make 768x768 and 1024x1024 as well. They can be used with certain transformer models", "2159958": "Three days ago, I could run image.dicom.pixel_array to decompress a Dicom image. At this point, I get the error \"The pixel data with transfer syntax JPEG 2000 Image Compression (Lossless Only), cannot be read because Pillow lacks the JPEG 2000 plugin\". On my computer, it has been resolved by installing 'pip install -U pylibjpeg pylibjpeg-openjpeg pylibjpeg-libjpeg'. \nIn Kaggle, it seems to install it in 'base environment' but it doesn't work, nor can I create another environment. Could someone help me please?", "2112596": "Hi @radek1 and all, \n\nI have one naive question: Since we want to have the same distribution of both test and training data, don't we need to use the same data transformation pipeline during inference time? DICOM to PNGs, crop, etc. ?", "2102074": "Wow! It will be very helpful, thanks bro :)", "2057235": "This will not only help us to learn but to save space locally..π ", "2052541": "Hi there, don't get me wrong. I'm just new to these kind of competitions. is information loss when we convert DICOM images to PNGs ?", "2048727": "Great work, one question is, what should we do when we need to submit prediction results?", "2070798": "thanks so much bro!" } }