Hey!
\nI wanted to give you an update on the RSNA Mammo PNGs 256px, 384px, 512px, 768px, 1024px that I know many people have been using!
\n(can also be useful to people who just joined the competition)
\nI kept adding to the dataset so it now has a decent collection of sizes processed in various ways 🙂
\nAll processing is documented in 💡 how to process DICOM images to PNGs so you can grab it and straight away implement it in your inference notebook!
\nThe four processing methods are as follows:
\nPIL - PIL is a great choice but turned out for me to be 10%-20% slower than opencv, so switched to thatopencv - like above, but faster, using opencv (these methods normalize images to between 0 and 1 and account fo the MONOCHROME1 status)opencv with VOI LUT transformation (thank you @raddar 🙏)opencv without VOI LUT transformation AND with reading the data in by dicomsdl (2x faster then pydicom!) (thank you @remekkinas! 🙏) 🔥🔥🔥 opencv with keeping the aspect ratio, resizing the longer side to 1024 (please see discussion in this thread on this Hope you find this useful in working on the solution! 🙂
\nHey!
\nI wanted to give you an update on the RSNA Mammo PNGs 256px, 384px, 512px, 768px, 1024px that I know many people have been using!
\n(can also be useful to people who just joined the competition)
\nI kept adding to the dataset so it now has a decent collection of sizes processed in various ways 🙂
\nAll processing is documented in 💡 how to process DICOM images to PNGs so you can grab it and straight away implement it in your inference notebook!
\nThe four processing methods are as follows:
\nPIL - PIL is a great choice but turned out for me to be 10%-20% slower than opencv, so switched to thatopencv - like above, but faster, using opencv (these methods normalize images to between 0 and 1 and account fo the MONOCHROME1 status)opencv with VOI LUT transformation (thank you @raddar 🙏)opencv without VOI LUT transformation AND with reading the data in by dicomsdl (2x faster then pydicom!) (thank you @remekkinas! 🙏) 🔥🔥🔥 opencv with keeping the aspect ratio, resizing the longer side to 1024 (please see discussion in this thread on this Hope you find this useful in working on the solution! 🙂
\nThank you very much for mentioning. Great work!
\nI thnink that one more dataset is required 😁 - images with preserved aspect ratio.
def image_resize(image, width = None, height = None, inter = cv2.INTER_LINEAR):\n\n dim = None\n (h, w) = image.shape[:2]\n\n if width is None and height is None:\n return image\n\n if width is None:\n r = height / float(h)\n dim = (int(w * r), height)\n else:\n r = width / float(w)\n dim = (width, int(h * r))\n resized = cv2.resize(image, dim, interpolation = inter)\n\n return resized\n\nh, w = tmp_im.shape\n\nif w > h:\n img = image_resize(tmp_im, width = size)\nelse:\n img = image_resize(tmp_im, height = size)\nWhat do you think?
", "rawMarkdown": "Thank you very much for mentioning. Great work!\nI thnink that one more dataset is required 😁 - images with preserved aspect ratio.\n\n```\ndef image_resize(image, width = None, height = None, inter = cv2.INTER_LINEAR):\n\n dim = None\n (h, w) = image.shape[:2]\n\n if width is None and height is None:\n return image\n \n if width is None:\n r = height / float(h)\n dim = (int(w * r), height)\n else:\n r = width / float(w)\n dim = (width, int(h * r))\n resized = cv2.resize(image, dim, interpolation = inter)\n \n return resized\n\nh, w = tmp_im.shape\n\nif w > h:\n img = image_resize(tmp_im, width = size)\nelse:\n img = image_resize(tmp_im, height = size)\n```\nWhat do you think?", "votes": 1, "replies": [ { "id": 2059768, "postDate": "2022-12-09T07:51:23.997Z", "content": "Working on it! Especially for you, @remekkinas! 🙂
\nWas about to delete that instance, but guess will squeeze in processing the dataset this way as well really quickly 🙂
\nIt will have the full thing (VOI LUT processing as well) and will be resized to the longer side being 1024 -- this way if someone needs the images smaller, they should be able to resize again without too much loss of quality.
\nShould be ready in probably a couple of hours given how slow on my end 🙂
", "rawMarkdown": "Working on it! Especially for you, @remekkinas! 🙂 \n\nWas about to delete that instance, but guess will squeeze in processing the dataset this way as well really quickly 🙂\n\nIt will have the full thing (VOI LUT processing as well) and will be resized to the longer side being 1024 -- this way if someone needs the images smaller, they should be able to resize again without too much loss of quality.\n\nShould be ready in probably a couple of hours given how slow on my end 🙂", "votes": 2 }, { "id": 2060036, "postDate": "2022-12-09T13:53:49.957Z", "content": "Files added, processing now 🙂 Updated the notebook as well to show the processing done 🙂 Hope this can be of use!
\nWill update the original post I guess once the processing finishes, the dataset grew again! 😄
", "rawMarkdown": "Files added, processing now 🙂 Updated the notebook as well to show the processing done 🙂 Hope this can be of use!\n\nWill update the original post I guess once the processing finishes, the dataset grew again! 😄", "votes": 1 }, { "id": 2060041, "postDate": "2022-12-09T14:03:56.350Z", "content": "Great work and dataset! 👍👍👍
", "rawMarkdown": "Great work and dataset! 👍👍👍", "votes": 1 }, { "id": 2060046, "postDate": "2022-12-09T14:10:46.293Z", "content": "Thanks, @remekkinas! 😊 Team effort! Appreciate your suggestions and the awesome things you've found! (like dicomsdl!).
Here is the fun bit on dicomsdl BTW. On a two core VM it is impossible to process the dataset with the VOI LUT transform AFAICT! (and this is the only thing we really need pydicom for). So I am thinking that quite likely people will start shifting to dicomsdl. Because of the speed.
You can possibly ensemble a good amount of models if you just use pydicom and opencv (without VOI LUT) but dicomsdl opens up a completely new sea of possibility 🙂 Just gotta verify the images are really good, but glancing over at the results they seem okay.
Anyhow, those are just some random musings, but maybe someone will find them useful! 🙂
\nThank you again, @remekkinas, for your kind words and the awesome things you shared! 🙏
", "rawMarkdown": "Thanks, @remekkinas! 😊 Team effort! Appreciate your suggestions and the awesome things you've found! (like `dicomsdl`!).\n\nHere is the fun bit on `dicomsdl` BTW. On a two core VM it is impossible to process the dataset with the VOI LUT transform AFAICT! (and this is the only thing we really need `pydicom` for). So I am thinking that quite likely people will start shifting to `dicomsdl`. Because of the speed.\n\nYou can possibly ensemble a good amount of models if you just use `pydicom` and `opencv` (without VOI LUT) but `dicomsdl` opens up a completely new sea of possibility 🙂 Just gotta verify the images are really good, but glancing over at the results they seem okay.\n\nAnyhow, those are just some random musings, but maybe someone will find them useful! 🙂\n\nThank you again, @remekkinas, for your kind words and the awesome things you shared! 🙏", "votes": 3 }, { "id": 2060113, "postDate": "2022-12-09T14:57:23.827Z", "content": "Thanks!
\nThis is not my dicovery - @kaggleqrdl described ( @alenic idea ) but I implemented it only in notebook 🙂 Speed is here important. We have more room for models/blending/TTA … etc.
\nOK. We have first stage (data processing) more or less under controll. We can do next step :)
Thank you very much for mentioning. Great work!
\nI thnink that one more dataset is required 😁 - images with preserved aspect ratio.
def image_resize(image, width = None, height = None, inter = cv2.INTER_LINEAR):\n\n dim = None\n (h, w) = image.shape[:2]\n\n if width is None and height is None:\n return image\n\n if width is None:\n r = height / float(h)\n dim = (int(w * r), height)\n else:\n r = width / float(w)\n dim = (width, int(h * r))\n resized = cv2.resize(image, dim, interpolation = inter)\n\n return resized\n\nh, w = tmp_im.shape\n\nif w > h:\n img = image_resize(tmp_im, width = size)\nelse:\n img = image_resize(tmp_im, height = size)\nWhat do you think?
", "votes": 1, "replies": [ { "id": 2059768, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-09T07:51:23.997000", "content": "Working on it! Especially for you, @remekkinas! 🙂
\nWas about to delete that instance, but guess will squeeze in processing the dataset this way as well really quickly 🙂
\nIt will have the full thing (VOI LUT processing as well) and will be resized to the longer side being 1024 -- this way if someone needs the images smaller, they should be able to resize again without too much loss of quality.
\nShould be ready in probably a couple of hours given how slow on my end 🙂
", "votes": 2, "replies": [] }, { "id": 2060036, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-09T13:53:49.957000", "content": "Files added, processing now 🙂 Updated the notebook as well to show the processing done 🙂 Hope this can be of use!
\nWill update the original post I guess once the processing finishes, the dataset grew again! 😄
", "votes": 1, "replies": [] }, { "id": 2060041, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-09T14:03:56.350000", "content": "Great work and dataset! 👍👍👍
", "votes": 1, "replies": [] }, { "id": 2060046, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-09T14:10:46.293000", "content": "Thanks, @remekkinas! 😊 Team effort! Appreciate your suggestions and the awesome things you've found! (like dicomsdl!).
Here is the fun bit on dicomsdl BTW. On a two core VM it is impossible to process the dataset with the VOI LUT transform AFAICT! (and this is the only thing we really need pydicom for). So I am thinking that quite likely people will start shifting to dicomsdl. Because of the speed.
You can possibly ensemble a good amount of models if you just use pydicom and opencv (without VOI LUT) but dicomsdl opens up a completely new sea of possibility 🙂 Just gotta verify the images are really good, but glancing over at the results they seem okay.
Anyhow, those are just some random musings, but maybe someone will find them useful! 🙂
\nThank you again, @remekkinas, for your kind words and the awesome things you shared! 🙏
", "votes": 3, "replies": [] }, { "id": 2060113, "author_name": "Remek Kinas", "author_url": "", "post_date": "2022-12-09T14:57:23.827000", "content": "Thanks!
\nThis is not my dicovery - @kaggleqrdl described ( @alenic idea ) but I implemented it only in notebook 🙂 Speed is here important. We have more room for models/blending/TTA … etc.
\nOK. We have first stage (data processing) more or less under controll. We can do next step :)