\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6625861%2Fcd3aba50e58dba90320a7a26b0139ecf%2F8a804f858ca96c5b773fef8e2b1f0f1.png?generation=1754487190516082&alt=media\")
For a significant fraction of CTAs, RescaleSlope and RescaleIntercept are present in the DICOM metadata, however when applying them to pixel_array, the output values are incorrect. In these cases, using pixel_array without rescaling actually returns the expected Hounsfield unit output.
\nI was able to work around this by only rescaling if:
\nI only checked the CTAs which were positive for aneurysm, so I'm not sure if the above is applicable to the negative CTAs.
\nI have attached a CSV file containing these instances.
", "messages": [ { "id": 3255855, "postDate": "2025-07-29T14:51:10.113Z", "content": "For a significant fraction of CTAs, RescaleSlope and RescaleIntercept are present in the DICOM metadata, however when applying them to pixel_array, the output values are incorrect. In these cases, using pixel_array without rescaling actually returns the expected Hounsfield unit output.
\nI was able to work around this by only rescaling if:
\nI only checked the CTAs which were positive for aneurysm, so I'm not sure if the above is applicable to the negative CTAs.
\nI have attached a CSV file containing these instances.
", "rawMarkdown": "For a significant fraction of CTAs, RescaleSlope and RescaleIntercept are present in the DICOM metadata, however when applying them to pixel_array, the output values are incorrect. In these cases, using pixel_array **without** rescaling actually returns the expected Hounsfield unit output. \n\nI was able to work around this by only rescaling if:\n- RescaleSlope and RescaleIntercept were present (some do not have these values, these look OK as is)\n- Minimum pixel value in pixel_array is greater than or equal to -100 OR equal to -2000 (few edge cases)\n\nI only checked the CTAs which were positive for aneurysm, so I'm not sure if the above is applicable to the negative CTAs. \n\nI have attached a CSV file containing these instances.", "votes": 27 }, { "id": 3264324, "postDate": "2025-08-06T13:33:14.177Z", "content": "Thats how they look like in my dicom viewer.
\n
I wonder if this is a result of the face removal (anonymization) step, which manipulated the pixel arrays without changing header variables. If we remove the slope/intercept values, will this solve the issue? Sounds like there may be some edge cases…
", "rawMarkdown": "I wonder if this is a result of the face removal (anonymization) step, which manipulated the pixel arrays without changing header variables. If we remove the slope/intercept values, will this solve the issue? Sounds like there may be some edge cases…", "replies": [ { "id": 3255915, "postDate": "2025-07-29T16:25:21.850Z", "content": "Certainly possible, but I find it odd that there are still a fair number of CTAs that do require rescaling, assuming that all of them underwent the same anonymization pipeline, in which case just removing the slope and intercept values from all studies wouldn't work. Presumably facial anonymization occurred after the annotation phase? Otherwise I would expect the annotators to notice the images were not being displayed properly if annotation was being done in a DICOM viewer.
\nThe easiest thing to do might be to just confirm that the heuristic I described above to determine which CTAs should be rescaled works for the CTAs in the test set. I verified this visually by rescaling (if necessary), windowing, and then saving a slice (ideally with relevant anatomy) as a PNG to confirm that the windowing looks as expected.
", "rawMarkdown": "Certainly possible, but I find it odd that there are still a fair number of CTAs that **do** require rescaling, assuming that all of them underwent the same anonymization pipeline, in which case just removing the slope and intercept values from all studies wouldn't work. Presumably facial anonymization occurred after the annotation phase? Otherwise I would expect the annotators to notice the images were not being displayed properly if annotation was being done in a DICOM viewer. \n\nThe easiest thing to do might be to just confirm that the heuristic I described above to determine which CTAs should be rescaled works for the CTAs in the test set. I verified this visually by rescaling (if necessary), windowing, and then saving a slice (ideally with relevant anatomy) as a PNG to confirm that the windowing looks as expected. ", "replies": [ { "id": 3255919, "postDate": "2025-07-29T16:42:29.943Z", "content": "Yes, definitely some more investigation to be done. Thanks for the heads up!
", "rawMarkdown": "Yes, definitely some more investigation to be done. Thanks for the heads up!", "votes": 1 }, { "id": 3255931, "postDate": "2025-07-29T17:07:10.377Z", "content": "I am new to this field and may not have a full understanding of the current situation. Based on the ongoing discussion, it seems that the dataset might require some corrections. Does this mean I would need to re-download the entire 340GB dataset? If corrections are indeed needed, could you kindly provide the specific portions of the data that require updates (I assume it is not the entire dataset), along with a replacement or patching script?
", "rawMarkdown": "I am new to this field and may not have a full understanding of the current situation. Based on the ongoing discussion, it seems that the dataset might require some corrections. Does this mean I would need to re-download the entire 340GB dataset? If corrections are indeed needed, could you kindly provide the specific portions of the data that require updates (I assume it is not the entire dataset), along with a replacement or patching script?" }, { "id": 3255947, "postDate": "2025-07-29T17:34:53.490Z", "content": "Yes, it seems that some of the CT data (a subset of the whole dataset) may need some updates. This may or may not be relevant to downstream algorithm development. Specifically, this affects the absolute values of CT pixels (not relative values) so if you are planning to do normalization (as many likely will) then it’s probably not relevant. That said, we will attempt to address the issue and provide a fix. I am hoping we can provide a fix script as you suggested, which would not require a re-download.
", "rawMarkdown": "Yes, it seems that some of the CT data (a subset of the whole dataset) may need some updates. This may or may not be relevant to downstream algorithm development. Specifically, this affects the absolute values of CT pixels (not relative values) so if you are planning to do normalization (as many likely will) then it’s probably not relevant. That said, we will attempt to address the issue and provide a fix. I am hoping we can provide a fix script as you suggested, which would not require a re-download. ", "votes": 2 }, { "id": 3255948, "postDate": "2025-07-29T17:37:41.047Z", "content": "Certainly, thank you very much! Given the large size of the dataset, re-downloading it would indeed be somewhat inconvenient.
", "rawMarkdown": "Certainly, thank you very much! Given the large size of the dataset, re-downloading it would indeed be somewhat inconvenient." } ] }, { "id": 3265218, "postDate": "2025-08-07T07:12:58.067Z", "content": "If the dataset is updated, could you please ensure consistency between the train and test sets? Also, would it be possible to notify us clearly when updates happen, so people who already downloaded the data don't miss the changes? Thanks!
", "rawMarkdown": "If the dataset is updated, could you please ensure consistency between the train and test sets? Also, would it be possible to notify us clearly when updates happen, so people who already downloaded the data don't miss the changes? Thanks!", "replies": [ { "id": 3265641, "postDate": "2025-08-07T19:47:33.760Z", "content": "Absolutely. We are currently working on an update that will fix multiple small issues including this one, and once its ready we will clearly communicate this. Please look out for a pinned post in the next few days (hoping for Monday). Unfortunately we cannot fix minor issues one by once since it requires re-running extensive tests on the entire train/test dataset to ensure consistency.
", "rawMarkdown": "Absolutely. We are currently working on an update that will fix multiple small issues including this one, and once its ready we will clearly communicate this. Please look out for a pinned post in the next few days (hoping for Monday). Unfortunately we cannot fix minor issues one by once since it requires re-running extensive tests on the entire train/test dataset to ensure consistency.", "votes": 3 } ] } ] } ], "comments": [ { "id": 3264324, "author_name": "Seeing Times", "author_url": "", "post_date": "2025-08-06T13:33:14.177000", "content": "Thats how they look like in my dicom viewer.
\n
I wonder if this is a result of the face removal (anonymization) step, which manipulated the pixel arrays without changing header variables. If we remove the slope/intercept values, will this solve the issue? Sounds like there may be some edge cases…
", "votes": 0, "replies": [ { "id": 3255915, "author_name": "Ian Pan", "author_url": "", "post_date": "2025-07-29T16:25:21.850000", "content": "Certainly possible, but I find it odd that there are still a fair number of CTAs that do require rescaling, assuming that all of them underwent the same anonymization pipeline, in which case just removing the slope and intercept values from all studies wouldn't work. Presumably facial anonymization occurred after the annotation phase? Otherwise I would expect the annotators to notice the images were not being displayed properly if annotation was being done in a DICOM viewer.
\nThe easiest thing to do might be to just confirm that the heuristic I described above to determine which CTAs should be rescaled works for the CTAs in the test set. I verified this visually by rescaling (if necessary), windowing, and then saving a slice (ideally with relevant anatomy) as a PNG to confirm that the windowing looks as expected.
", "votes": 0, "replies": [ { "id": 3255919, "author_name": "Evan Calabrese", "author_url": "", "post_date": "2025-07-29T16:42:29.943000", "content": "Yes, definitely some more investigation to be done. Thanks for the heads up!
", "votes": 1, "replies": [] }, { "id": 3255931, "author_name": "predictnewbie", "author_url": "", "post_date": "2025-07-29T17:07:10.377000", "content": "I am new to this field and may not have a full understanding of the current situation. Based on the ongoing discussion, it seems that the dataset might require some corrections. Does this mean I would need to re-download the entire 340GB dataset? If corrections are indeed needed, could you kindly provide the specific portions of the data that require updates (I assume it is not the entire dataset), along with a replacement or patching script?
", "votes": 0, "replies": [] }, { "id": 3255947, "author_name": "Evan Calabrese", "author_url": "", "post_date": "2025-07-29T17:34:53.490000", "content": "Yes, it seems that some of the CT data (a subset of the whole dataset) may need some updates. This may or may not be relevant to downstream algorithm development. Specifically, this affects the absolute values of CT pixels (not relative values) so if you are planning to do normalization (as many likely will) then it’s probably not relevant. That said, we will attempt to address the issue and provide a fix. I am hoping we can provide a fix script as you suggested, which would not require a re-download.
", "votes": 2, "replies": [] }, { "id": 3255948, "author_name": "predictnewbie", "author_url": "", "post_date": "2025-07-29T17:37:41.047000", "content": "Certainly, thank you very much! Given the large size of the dataset, re-downloading it would indeed be somewhat inconvenient.
", "votes": 0, "replies": [] } ] }, { "id": 3265218, "author_name": "YiZ277", "author_url": "", "post_date": "2025-08-07T07:12:58.067000", "content": "If the dataset is updated, could you please ensure consistency between the train and test sets? Also, would it be possible to notify us clearly when updates happen, so people who already downloaded the data don't miss the changes? Thanks!
", "votes": 0, "replies": [ { "id": 3265641, "author_name": "Evan Calabrese", "author_url": "", "post_date": "2025-08-07T19:47:33.760000", "content": "Absolutely. We are currently working on an update that will fix multiple small issues including this one, and once its ready we will clearly communicate this. Please look out for a pinned post in the next few days (hoping for Monday). Unfortunately we cannot fix minor issues one by once since it requires re-running extensive tests on the entire train/test dataset to ensure consistency.
", "votes": 3, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3255855": "For a significant fraction of CTAs, RescaleSlope and RescaleIntercept are present in the DICOM metadata, however when applying them to pixel_array, the output values are incorrect. In these cases, using pixel_array **without** rescaling actually returns the expected Hounsfield unit output. \n\nI was able to work around this by only rescaling if:\n- RescaleSlope and RescaleIntercept were present (some do not have these values, these look OK as is)\n- Minimum pixel value in pixel_array is greater than or equal to -100 OR equal to -2000 (few edge cases)\n\nI only checked the CTAs which were positive for aneurysm, so I'm not sure if the above is applicable to the negative CTAs. \n\nI have attached a CSV file containing these instances.", "3264324": "Thats how they look like in my dicom viewer.\n", "3255863": "I wonder if this is a result of the face removal (anonymization) step, which manipulated the pixel arrays without changing header variables. If we remove the slope/intercept values, will this solve the issue? Sounds like there may be some edge cases…" } }