![\"1\"](\"https://i.ibb.co/9h6nLf6/Screenshot-from-2023-08-11-10-52-15.png\")
The RSNA 2023 Abdominal Trauma Detection Challenge showcases an exceptionally heterogeneous DICOM data structure due to the large number of institutions that donated data.
\nCertain DICOM pixel arrays will appear unusual when opened using the pydicom library.
\nEvery DICOM file contains a field called Pixel Representation indicating if the pixel datatype is unsigned(0) or signed (1).
\nSome DICOM files with PixelRepresentation == 1 exhibit visually unusual characteristics. You can use the [train/test]_dicom_tags.parquet files to identify images likely to be affected.
\nThe following function is a potential solution to the described problem.
\n standardize_pixel_array(dcm: pydicom.dataset.) -> np.ndarray:\n # pixel_array == \n pixel_array = dcm.pixel_array\n dcm. == :\n bit_shift = dcm. - dcm.\n d = pixel_array.dtype \n pixel_array = (pixel_array << bit_shift).(dtype) >> bit_shift\n return pixel_array\nPlease see this notebook for example implementation. We did not apply this processing step to the competition dataset as we did not have time to confirm that it properly resolves the issue in all cases and does not have any unintended side effects.
", "messages": [ { "id": 2360696, "postDate": "2023-07-26T23:25:03.827Z", "content": "The RSNA 2023 Abdominal Trauma Detection Challenge showcases an exceptionally heterogeneous DICOM data structure due to the large number of institutions that donated data.
\nCertain DICOM pixel arrays will appear unusual when opened using the pydicom library.
\nEvery DICOM file contains a field called Pixel Representation indicating if the pixel datatype is unsigned(0) or signed (1).
\nSome DICOM files with PixelRepresentation == 1 exhibit visually unusual characteristics. You can use the [train/test]_dicom_tags.parquet files to identify images likely to be affected.
\nThe following function is a potential solution to the described problem.
\n standardize_pixel_array(dcm: pydicom.dataset.) -> np.ndarray:\n # pixel_array == \n pixel_array = dcm.pixel_array\n dcm. == :\n bit_shift = dcm. - dcm.\n d = pixel_array.dtype \n pixel_array = (pixel_array << bit_shift).(dtype) >> bit_shift\n return pixel_array\nPlease see this notebook for example implementation. We did not apply this processing step to the competition dataset as we did not have time to confirm that it properly resolves the issue in all cases and does not have any unintended side effects.
", "rawMarkdown": "# Unusual DICOM Files\n\nThe RSNA 2023 Abdominal Trauma Detection Challenge showcases an exceptionally heterogeneous DICOM data structure due to the large number of institutions that donated data.\n\nCertain DICOM pixel arrays will appear unusual when opened using the pydicom library.\n\nEvery DICOM file contains a field called [Pixel Representation](https://dicom.innolitics.com/ciods/rt-dose/image-pixel/00280103) indicating if the pixel datatype is unsigned(0) or signed (1).\n\nSome DICOM files with PixelRepresentation == 1 exhibit visually unusual characteristics. You can use the [train/test]_dicom_tags.parquet files to identify images likely to be affected.\n\n\n# Potential Solution\n\nThe following function is a potential solution to the described problem.\n\n```\ndef standardize_pixel_array(dcm: pydicom.dataset.FileDataset) -> np.ndarray:\n # Correct DICOM pixel_array if PixelRepresentation == 1.\n pixel_array = dcm.pixel_array\n if dcm.PixelRepresentation == 1:\n bit_shift = dcm.BitsAllocated - dcm.BitsStored\n dtype = pixel_array.dtype \n pixel_array = (pixel_array << bit_shift).astype(dtype) >> bit_shift\n return pixel_array\n```\n\n\nPlease see this [notebook](https://www.kaggle.com/code/huiminglin/rsna-2023-abdomen-dicom-fix) for example implementation. We did not apply this processing step to the competition dataset as we did not have time to confirm that it properly resolves the issue in all cases and does not have any unintended side effects.", "votes": 45 }, { "id": 2370495, "postDate": "2023-08-02T13:08:33.580Z", "content": "Why the apply_modality_lut is applied only when dcm.PixelRepresentation == 1?
\nI think this should be applied to both, otherwise, neither.
if dcm.PixelRepresentation == 1:\n bit_shift = dcm.BitsAllocated - dcm.BitsStored\n dtype = pixel_array.dtype \n\n\n\n return pixel_array\npixel_array
", "rawMarkdown": "Why the `apply_modality_lut` is applied only when `dcm.PixelRepresentation == 1`?\nI think this should be applied to both, otherwise, neither.\n\n```diff\n if dcm.PixelRepresentation == 1:\n bit_shift = dcm.BitsAllocated - dcm.BitsStored\n dtype = pixel_array.dtype \n! pixel_array = (pixel_array << bit_shift).astype(dtype) >> bit_shift\n- pixel_array = pydicom.pixel_data_handlers.util.apply_modality_lut(new_array, dcm)\n+ pixel_array = pydicom.pixel_data_handlers.util.apply_modality_lut(pixel_array, dcm)\n return pixel_array\n```\npixel_array", "votes": 6, "replies": [ { "id": 2370520, "postDate": "2023-08-02T13:23:55.530Z", "content": "i think so, too. apply_modality_lut function is used to convert pixel array to Hounsfield Unit (HU). it has nothing to do with PixelRepresentation (signed or unsigned data)
", "rawMarkdown": "i think so, too. apply_modality_lut function is used to convert pixel array to Hounsfield Unit (HU). it has nothing to do with PixelRepresentation (signed or unsigned data)", "votes": 1 }, { "id": 2370672, "postDate": "2023-08-02T15:06:50.460Z", "content": "You are absolutely correct, thank you for pointing this out. I have removed apply_modality_lut from the function. The notebook has also been updated.
", "rawMarkdown": "You are absolutely correct, thank you for pointing this out. I have removed apply_modality_lut from the function. The notebook has also been updated." } ] }, { "id": 2373646, "postDate": "2023-08-04T11:09:42.310Z", "content": "I noticed a particular quirk of the dataset. DICOMs where PixelRepresentation == 1 always returns pixel_array as a np.int16 and other DICOMs always returns pixel_array as a np.uint16. Does this affect normalization in any way?
Edit: By normalization, I meant the final output of the standardize_pixel_array function. Because the function will still return np.int16 for PixelRepresentation == 1 and np.uint16 otherwise.
What do you mean by the visually unusual characteristics?
", "rawMarkdown": "What do you mean by the visually unusual characteristics?", "votes": 1 }, { "id": 2383574, "postDate": "2023-08-10T13:00:09.300Z", "content": "please how to view dataset images
", "rawMarkdown": " please how to view dataset images\n" }, { "id": 2465637, "postDate": "2023-10-03T06:50:21.100Z", "content": "As a beginner in such competitions, I've taken a look at multiple notebooks. I'm questioning whether this will directly influence my results or if it's more about comprehending pixel representations.
", "rawMarkdown": "As a beginner in such competitions, I've taken a look at multiple notebooks. I'm questioning whether this will directly influence my results or if it's more about comprehending pixel representations." }, { "id": 2383052, "postDate": "2023-08-10T06:47:55.457Z", "content": "I don't understand how shifting bits would change the output. You left shift bits by some amount and then you right shift with the same amount which returns the same exact array. What am I missing?
", "rawMarkdown": "I don't understand how shifting bits would change the output. You left shift bits by some amount and then you right shift with the same amount which returns the same exact array. What am I missing?", "replies": [ { "id": 2384908, "postDate": "2023-08-11T05:51:26.940Z", "content": "Does this example help you?
\n numpy np\n\nBitsAllocated=\nBitsStored=\nbit_shift = BitsAllocated - BitsStored\n\nx = np.array().astype(np.int8) \nx_new = (x << bit_shift).astype(np.int8) >> bit_shift\n\n(np.binary_repr(x, BitsAllocated), )\n(np.binary_repr(x_new, BitsAllocated), )\n\n\n\nThank you. I think I understood the problem. Even though those images are labeled as signed 16-bit integers (PixelRepresentation == 1), their pixel values are starting from 0. If the values are starting from 0, then we are not using some of the allocated bits.
\n
When we left shift the bits, we make more bits used for a signed data type.
\n![\"2\"](\"https://i.ibb.co/r6HXxxm/Screenshot-from-2023-08-11-10-52-35.png\")
When we right shift with the same amount again, we truncate the last unused bits on the right.
\n![\"3\"](\"https://i.ibb.co/qmGBHyj/Screenshot-from-2023-08-11-10-52-44.png\")
Need to check the BitsAllocated and BitsStored DICOM tags. Most CTs are actually 10-12 bits deep. Since there isn't a 10bit datatype, we must use 16 bit dtypes, then calculate the actual pixel width of the image.
", "rawMarkdown": "Need to check the BitsAllocated and BitsStored DICOM tags. Most CTs are actually 10-12 bits deep. Since there isn't a 10bit datatype, we must use 16 bit dtypes, then calculate the actual pixel width of the image.", "votes": 4 } ] } ] }, { "id": 2465634, "postDate": "2023-10-03T06:47:41.610Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2370495, "author_name": "i-aki.y", "author_url": "", "post_date": "2023-08-02T13:08:33.580000", "content": "Why the apply_modality_lut is applied only when dcm.PixelRepresentation == 1?
\nI think this should be applied to both, otherwise, neither.
if dcm.PixelRepresentation == 1:\n bit_shift = dcm.BitsAllocated - dcm.BitsStored\n dtype = pixel_array.dtype \n\n\n\n return pixel_array\npixel_array
", "votes": 6, "replies": [ { "id": 2370520, "author_name": "Vietnamese-NHNAM", "author_url": "", "post_date": "2023-08-02T13:23:55.530000", "content": "i think so, too. apply_modality_lut function is used to convert pixel array to Hounsfield Unit (HU). it has nothing to do with PixelRepresentation (signed or unsigned data)
", "votes": 1, "replies": [] }, { "id": 2370672, "author_name": "Hui Ming Lin", "author_url": "", "post_date": "2023-08-02T15:06:50.460000", "content": "You are absolutely correct, thank you for pointing this out. I have removed apply_modality_lut from the function. The notebook has also been updated.
", "votes": 0, "replies": [] } ] }, { "id": 2373646, "author_name": "coderRKJ", "author_url": "", "post_date": "2023-08-04T11:09:42.310000", "content": "I noticed a particular quirk of the dataset. DICOMs where PixelRepresentation == 1 always returns pixel_array as a np.int16 and other DICOMs always returns pixel_array as a np.uint16. Does this affect normalization in any way?
Edit: By normalization, I meant the final output of the standardize_pixel_array function. Because the function will still return np.int16 for PixelRepresentation == 1 and np.uint16 otherwise.
What do you mean by the visually unusual characteristics?
", "votes": 1, "replies": [] }, { "id": 2383574, "author_name": "martial taga", "author_url": "", "post_date": "2023-08-10T13:00:09.300000", "content": "please how to view dataset images
", "votes": 0, "replies": [] }, { "id": 2465637, "author_name": "USMAN SAFDAR", "author_url": "", "post_date": "2023-10-03T06:50:21.100000", "content": "As a beginner in such competitions, I've taken a look at multiple notebooks. I'm questioning whether this will directly influence my results or if it's more about comprehending pixel representations.
", "votes": 0, "replies": [] }, { "id": 2383052, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2023-08-10T06:47:55.457000", "content": "I don't understand how shifting bits would change the output. You left shift bits by some amount and then you right shift with the same amount which returns the same exact array. What am I missing?
", "votes": 0, "replies": [ { "id": 2384908, "author_name": "i-aki.y", "author_url": "", "post_date": "2023-08-11T05:51:26.940000", "content": "Does this example help you?
\n numpy np\n\nBitsAllocated=\nBitsStored=\nbit_shift = BitsAllocated - BitsStored\n\nx = np.array().astype(np.int8) \nx_new = (x << bit_shift).astype(np.int8) >> bit_shift\n\n(np.binary_repr(x, BitsAllocated), )\n(np.binary_repr(x_new, BitsAllocated), )\n\n\n\nThank you. I think I understood the problem. Even though those images are labeled as signed 16-bit integers (PixelRepresentation == 1), their pixel values are starting from 0. If the values are starting from 0, then we are not using some of the allocated bits.
\n
When we left shift the bits, we make more bits used for a signed data type.
\n
When we right shift with the same amount again, we truncate the last unused bits on the right.
\n
Need to check the BitsAllocated and BitsStored DICOM tags. Most CTs are actually 10-12 bits deep. Since there isn't a 10bit datatype, we must use 16 bit dtypes, then calculate the actual pixel width of the image.
", "votes": 4, "replies": [] } ] } ] }, { "id": 2465634, "author_name": "", "author_url": "", "post_date": "2023-10-03T06:47:41.610000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2360696": "# Unusual DICOM Files\n\nThe RSNA 2023 Abdominal Trauma Detection Challenge showcases an exceptionally heterogeneous DICOM data structure due to the large number of institutions that donated data.\n\nCertain DICOM pixel arrays will appear unusual when opened using the pydicom library.\n\nEvery DICOM file contains a field called [Pixel Representation](https://dicom.innolitics.com/ciods/rt-dose/image-pixel/00280103) indicating if the pixel datatype is unsigned(0) or signed (1).\n\nSome DICOM files with PixelRepresentation == 1 exhibit visually unusual characteristics. You can use the [train/test]_dicom_tags.parquet files to identify images likely to be affected.\n\n\n# Potential Solution\n\nThe following function is a potential solution to the described problem.\n\n```\ndef standardize_pixel_array(dcm: pydicom.dataset.FileDataset) -> np.ndarray:\n # Correct DICOM pixel_array if PixelRepresentation == 1.\n pixel_array = dcm.pixel_array\n if dcm.PixelRepresentation == 1:\n bit_shift = dcm.BitsAllocated - dcm.BitsStored\n dtype = pixel_array.dtype \n pixel_array = (pixel_array << bit_shift).astype(dtype) >> bit_shift\n return pixel_array\n```\n\n\nPlease see this [notebook](https://www.kaggle.com/code/huiminglin/rsna-2023-abdomen-dicom-fix) for example implementation. We did not apply this processing step to the competition dataset as we did not have time to confirm that it properly resolves the issue in all cases and does not have any unintended side effects.", "2370495": "Why the `apply_modality_lut` is applied only when `dcm.PixelRepresentation == 1`?\nI think this should be applied to both, otherwise, neither.\n\n```diff\n if dcm.PixelRepresentation == 1:\n bit_shift = dcm.BitsAllocated - dcm.BitsStored\n dtype = pixel_array.dtype \n! pixel_array = (pixel_array << bit_shift).astype(dtype) >> bit_shift\n- pixel_array = pydicom.pixel_data_handlers.util.apply_modality_lut(new_array, dcm)\n+ pixel_array = pydicom.pixel_data_handlers.util.apply_modality_lut(pixel_array, dcm)\n return pixel_array\n```\npixel_array", "2373646": "I noticed a particular quirk of the dataset. DICOMs where `PixelRepresentation == 1` always returns `pixel_array` as a `np.int16` and other DICOMs always returns `pixel_array` as a `np.uint16`. Does this affect normalization in any way?\n\nEdit: By normalization, I meant the final output of the `standardize_pixel_array` function. Because the function will still return `np.int16` for `PixelRepresentation == 1` and `np.uint16` otherwise.", "2426792": "What do you mean by the visually unusual characteristics?", "2383574": " please how to view dataset images\n", "2465637": "As a beginner in such competitions, I've taken a look at multiple notebooks. I'm questioning whether this will directly influence my results or if it's more about comprehending pixel representations.", "2383052": "I don't understand how shifting bits would change the output. You left shift bits by some amount and then you right shift with the same amount which returns the same exact array. What am I missing?", "2465634": "" } }