Hi organizers and teams,
\ndespite lots of similar discussions about the hidden test set, we don't see the responce from competition organizers recently after the last update, thus we want to raise some discussion on such issue again.
\nTo specify, we found there is more than one attributes missing in the test set. We try-except a lot and find that there must be something different in the multi-frame DICOMs in the test set. Such issue is not in the public training set, so this can't be prevented during local run. It is SOOO frustrating that we can't apply our model on a proportion of test set, that is just because of a file processing failure, while we have no idea what's missing.
\nMoreover, we don't really understand why the training set is different from the test set. Obviously the split of training and hidden test set is not independent and random, otherwise we should encounter such problem in the training data. Therefore we have 0 confidence if there would be another problem in another 68% test set after the submission is closed.
\nIs there any successful experience on using try-except or set default value, in order to skip the problematic multi-frame attributes? Also we would like to hear from the organizers to see if such issue is awared.
\nCheers,
", "messages": [ { "id": 3294734, "postDate": "2025-09-26T15:48:05.123Z", "content": "Hi @wenxiaozhan,
\nThe hidden test set should now have the PerFrameFunctionalGroupsSequence tag included. It is not present in every DICOM, but only a subset.
As discussed on the Data page, the tags in the test set DICOMs have been filtered to include only those in an allowlist. This is to help prevent leakage and other spurious correlations from affecting prediction quality.
\nAlso, submissions will not be run on a different 68% test set after submissions are closed. Your submissions are already being run on the private test set whose scores will populate the final leaderboard, but we withhold those scores until after close.
", "rawMarkdown": "Hi @wenxiaozhan,\n\nThe hidden test set should now have the `PerFrameFunctionalGroupsSequence` tag included. It is not present in every DICOM, but only a subset.\n\nAs discussed on the [Data](https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/data) page, the tags in the test set DICOMs have been filtered to include only those in an allowlist. This is to help prevent leakage and other spurious correlations from affecting prediction quality.\n\nAlso, submissions will not be run on a different 68% test set after submissions are closed. Your submissions are already being run on the private test set whose scores will populate the final leaderboard, but we withhold those scores until after close.", "votes": 3, "replies": [ { "id": 3294736, "postDate": "2025-09-26T15:56:08.220Z", "content": "Ok, so not all of the allowed tags are necessarily present in the test set? I thought the list means that these will be availlable?
", "rawMarkdown": "Ok, so not all of the *allowed* tags are necessarily present in the test set? I thought the list means that these will be availlable?", "replies": [ { "id": 3294781, "postDate": "2025-09-26T17:45:06.907Z", "content": "The DICOM series represent a diverse set of imaging studies. Not every kind of study records every kind of tag. Only multi-frame DICOMs include the PerFrameFunctionalGroupsSequence tag, for instance. So the list of tags on the Data page is a list of tags allowed, not necessarily present, in the test set. If the tag wasn't present in the original data, it won't be present in the competition test set, either.
Ok, thanks. The I‘ll try to make it work without it.
", "rawMarkdown": "Ok, thanks. The I‘ll try to make it work without it." }, { "id": 3295447, "postDate": "2025-09-28T19:27:14.797Z", "content": "Hi @ryanholbrook @jeffrudie , thanks for reply. I looked into one multi-frame DICOM in the training set, and it looks like:
\n(,) Shared Functional item(s) \n (,) Plane Orientation item(s) \n (,) Image Orientation (Patient) DS: [, , , , , ]\n \n (,) Pixel Measures item(s) \n (,) Thickness DS: \n (,) Spacing Slices DS: \n (,) Pixel Spacing DS: [, ]\n \n \n(,) Per-Frame Functional item(s) \n (,) Plane Position item(s) \n (,) Image Position (Patient) DS: [, , ]\n \n \n (,) Plane Position item(s) \n (,) Image Position (Patient) DS: [, , ]\nTherefore, yes, PerFrameFunctionalGroupsSequence is available, but only in reading Spacing Between Slices, which should be also available in Shared Functional Groups Sequence as well.
However, I think most people are stuck in reading Pixel Spacing from SharedFunctionalGroupsSequence, as, if it's true, SharedFunctionalGroupsSequence doesn't exist in test set at all. It is impossible to read it from PerFrameFunctionalGroupsSequence, as x and y are always 0 as you see in the block.
So all in all, the thing is, there's a big discrepency between hidden test set and training set, while we have no idea what they are. I think the best solution is at least to add all types of data form into training set. I'm really frustrated if we should replace any metadata of a patient by a default value, where we have no idea what it could be
\nSo could you please confirm if there is a big difference in the hidden test set? And could you make an update in the discussion to tell us what should be used as the correct tags in our submission?
", "rawMarkdown": "Hi @ryanholbrook @jeffrudie , thanks for reply. I looked into one multi-frame DICOM in the training set, and it looks like:\n\n```\n(5200,9229) Shared Functional Groups Sequence 1 item(s) ---- \n (0020,9116) Plane Orientation Sequence 1 item(s) ---- \n (0020,0037) Image Orientation (Patient) DS: [1, 0, 0, 0, 1, 0]\n ---------\n (0028,9110) Pixel Measures Sequence 1 item(s) ---- \n (0018,0050) Slice Thickness DS: '5.0'\n (0018,0088) Spacing Between Slices DS: '5.0'\n (0028,0030) Pixel Spacing DS: [0.5, 0.5]\n ---------\n ---------\n(5200,9230) Per-Frame Functional Groups Sequence 23 item(s) ---- \n (0020,9113) Plane Position Sequence 1 item(s) ---- \n (0020,0032) Image Position (Patient) DS: [0, 0, 0.0]\n ---------\n ---------\n (0020,9113) Plane Position Sequence 1 item(s) ---- \n (0020,0032) Image Position (Patient) DS: [0, 0, 5.0]\n```\n\nTherefore, yes, `PerFrameFunctionalGroupsSequence` is available, but only in reading `Spacing Between Slices`, which should be also available in `Shared Functional Groups Sequence` as well.\n\nHowever, I think most people are stuck in reading `Pixel Spacing` from `SharedFunctionalGroupsSequence`, as, if it's true, `SharedFunctionalGroupsSequence` doesn't exist in test set at all. It is impossible to read it from `PerFrameFunctionalGroupsSequence`, as x and y are always 0 as you see in the block.\n\nSo all in all, the thing is, there's a big discrepency between hidden test set and training set, while we have no idea what they are. I think the best solution is at least to add all types of data form into training set. I'm really frustrated if we should replace any metadata of a patient by a default value, where we have no idea what it could be\n\nSo could you please confirm if there is a big difference in the hidden test set? And could you make an update in the discussion to tell us what should be used as the correct tags in our submission?", "votes": 1 } ] }, { "id": 3294769, "postDate": "2025-09-26T17:11:50.983Z", "content": "\nApologies, just found the answer here.
", "rawMarkdown": "~~Thanks for the reply @ryanholbrook. Is the`PerFrameFunctionalGroup` change as of today? ~~\n\nApologies, just found the answer [here](https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/discussion/606917#3294727).\n\n", "replies": [ { "id": 3295051, "postDate": "2025-09-27T14:08:06.300Z", "content": "@ryanholbrook is Shared Functional Groups Sequence present in multiframe? can you confirm this?
", "rawMarkdown": "@ryanholbrook is Shared Functional Groups Sequence present in multiframe? can you confirm this?" } ] } ] }, { "id": 3294725, "postDate": "2025-09-26T15:37:12.403Z", "content": "Apologies for the delay and oversight on this issue.
\n@ryanholbrook can comment when a pending update to the test set to restore the 'PerFrameFunctionalGroupsSequence' tag is complete. It was originally restored in the second update, but the dataset was reverted when other issues were identified with the update.
\nIn general, we restricted the number of DICOM tags available in the test set so that models would utilize the images rather than DICOM metadata, but did want to share more DICOM tags as part of the public training set for future evaluation.
", "rawMarkdown": "Apologies for the delay and oversight on this issue. \n\n@ryanholbrook can comment when a pending update to the test set to restore the 'PerFrameFunctionalGroupsSequence' tag is complete. It was originally restored in the second update, but the dataset was reverted when other issues were identified with the update. \n\nIn general, we restricted the number of DICOM tags available in the test set so that models would utilize the images rather than DICOM metadata, but did want to share more DICOM tags as part of the public training set for future evaluation.", "votes": 3 }, { "id": 3294714, "postDate": "2025-09-26T15:14:44.557Z", "content": "Hi organizers and teams,
\ndespite lots of similar discussions about the hidden test set, we don't see the responce from competition organizers recently after the last update, thus we want to raise some discussion on such issue again.
\nTo specify, we found there is more than one attributes missing in the test set. We try-except a lot and find that there must be something different in the multi-frame DICOMs in the test set. Such issue is not in the public training set, so this can't be prevented during local run. It is SOOO frustrating that we can't apply our model on a proportion of test set, that is just because of a file processing failure, while we have no idea what's missing.
\nMoreover, we don't really understand why the training set is different from the test set. Obviously the split of training and hidden test set is not independent and random, otherwise we should encounter such problem in the training data. Therefore we have 0 confidence if there would be another problem in another 68% test set after the submission is closed.
\nIs there any successful experience on using try-except or set default value, in order to skip the problematic multi-frame attributes? Also we would like to hear from the organizers to see if such issue is awared.
\nCheers,
", "rawMarkdown": "Hi organizers and teams,\n\ndespite lots of similar discussions about the hidden test set, we don't see the responce from competition organizers recently after the last update, thus we want to raise some discussion on such issue again.\n\nTo specify, we found there is more than one attributes missing in the test set. We try-except a lot and find that there must be something different in the multi-frame DICOMs in the test set. Such issue is not in the public training set, so this can't be prevented during local run. It is SOOO frustrating that we can't apply our model on a proportion of test set, that is just because of a file processing failure, while we have no idea what's missing.\n\nMoreover, we don't really understand why the training set is different from the test set. Obviously the split of training and hidden test set is not independent and random, otherwise we should encounter such problem in the training data. Therefore we have 0 confidence if there would be another problem in another 68% test set after the submission is closed.\n\nIs there any successful experience on using try-except or set default value, in order to skip the problematic multi-frame attributes? Also we would like to hear from the organizers to see if such issue is awared.\n\nCheers,", "votes": 3 }, { "id": 3294886, "postDate": "2025-09-27T03:29:12.120Z", "content": "I think there is another multi-frame DICOM issue. Extracting the ImageOrientationPatient on the LB fails.
tmp = dicoms\niops = ((tmp))]\nLooks like SharedFunctionalGroupsSequence and/or PlaneOrientationSequence might not be allowed tags, or ImageOrientationPatient could be missing in some of the series.
Did you try PerFrameFunctionalGroupsSequence? I guess SharedFunctionalGroupsSequence doesn't necessarily exist. I notice the competition hosts confirm PerFrameFunctionalGroupsSequence exists, but never see them confirming SharedFunctionalGroupsSequence are stored in every dcm 🤔
Thanks for the comment @wenxiaozhan. I just tried probing PerFrameFunctionalGroupsSequence and it works on the LB.
My concern is that we are unable to access multi-frame DICOM tags that would otherwise be available in non-multi-frame equivalents. For example, PixelSpacing, ImageOrientationPatient, etc.
Personally I think reading errors are caused for very uncommon outliers just present in test. And just about DICOM metadata standards. In my opinion you are safe to use try/except with safe returning values such axial if not IOP, trust instance number if not IPP, and so on.
", "rawMarkdown": "Personally I think reading errors are caused for very uncommon outliers just present in test. And just about DICOM metadata standards. In my opinion you are safe to use try/except with safe returning values such axial if not IOP, trust instance number if not IPP, and so on." } ], "comments": [ { "id": 3294734, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2025-09-26T15:48:05.123000", "content": "Hi @wenxiaozhan,
\nThe hidden test set should now have the PerFrameFunctionalGroupsSequence tag included. It is not present in every DICOM, but only a subset.
As discussed on the Data page, the tags in the test set DICOMs have been filtered to include only those in an allowlist. This is to help prevent leakage and other spurious correlations from affecting prediction quality.
\nAlso, submissions will not be run on a different 68% test set after submissions are closed. Your submissions are already being run on the private test set whose scores will populate the final leaderboard, but we withhold those scores until after close.
", "votes": 3, "replies": [ { "id": 3294736, "author_name": "thomas rost", "author_url": "", "post_date": "2025-09-26T15:56:08.220000", "content": "Ok, so not all of the allowed tags are necessarily present in the test set? I thought the list means that these will be availlable?
", "votes": 0, "replies": [ { "id": 3294781, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2025-09-26T17:45:06.907000", "content": "The DICOM series represent a diverse set of imaging studies. Not every kind of study records every kind of tag. Only multi-frame DICOMs include the PerFrameFunctionalGroupsSequence tag, for instance. So the list of tags on the Data page is a list of tags allowed, not necessarily present, in the test set. If the tag wasn't present in the original data, it won't be present in the competition test set, either.
Ok, thanks. The I‘ll try to make it work without it.
", "votes": 0, "replies": [] }, { "id": 3295447, "author_name": "xmh", "author_url": "", "post_date": "2025-09-28T19:27:14.797000", "content": "Hi @ryanholbrook @jeffrudie , thanks for reply. I looked into one multi-frame DICOM in the training set, and it looks like:
\n(,) Shared Functional item(s) \n (,) Plane Orientation item(s) \n (,) Image Orientation (Patient) DS: [, , , , , ]\n \n (,) Pixel Measures item(s) \n (,) Thickness DS: \n (,) Spacing Slices DS: \n (,) Pixel Spacing DS: [, ]\n \n \n(,) Per-Frame Functional item(s) \n (,) Plane Position item(s) \n (,) Image Position (Patient) DS: [, , ]\n \n \n (,) Plane Position item(s) \n (,) Image Position (Patient) DS: [, , ]\nTherefore, yes, PerFrameFunctionalGroupsSequence is available, but only in reading Spacing Between Slices, which should be also available in Shared Functional Groups Sequence as well.
However, I think most people are stuck in reading Pixel Spacing from SharedFunctionalGroupsSequence, as, if it's true, SharedFunctionalGroupsSequence doesn't exist in test set at all. It is impossible to read it from PerFrameFunctionalGroupsSequence, as x and y are always 0 as you see in the block.
So all in all, the thing is, there's a big discrepency between hidden test set and training set, while we have no idea what they are. I think the best solution is at least to add all types of data form into training set. I'm really frustrated if we should replace any metadata of a patient by a default value, where we have no idea what it could be
\nSo could you please confirm if there is a big difference in the hidden test set? And could you make an update in the discussion to tell us what should be used as the correct tags in our submission?
", "votes": 1, "replies": [] } ] }, { "id": 3294769, "author_name": "Bartley", "author_url": "", "post_date": "2025-09-26T17:11:50.983000", "content": "\nApologies, just found the answer here.
", "votes": 0, "replies": [ { "id": 3295051, "author_name": "Arunodhayan", "author_url": "", "post_date": "2025-09-27T14:08:06.300000", "content": "@ryanholbrook is Shared Functional Groups Sequence present in multiframe? can you confirm this?
", "votes": 0, "replies": [] } ] } ] }, { "id": 3294725, "author_name": "JeffRudie", "author_url": "", "post_date": "2025-09-26T15:37:12.403000", "content": "Apologies for the delay and oversight on this issue.
\n@ryanholbrook can comment when a pending update to the test set to restore the 'PerFrameFunctionalGroupsSequence' tag is complete. It was originally restored in the second update, but the dataset was reverted when other issues were identified with the update.
\nIn general, we restricted the number of DICOM tags available in the test set so that models would utilize the images rather than DICOM metadata, but did want to share more DICOM tags as part of the public training set for future evaluation.
", "votes": 3, "replies": [] }, { "id": 3294886, "author_name": "Bartley", "author_url": "", "post_date": "2025-09-27T03:29:12.120000", "content": "I think there is another multi-frame DICOM issue. Extracting the ImageOrientationPatient on the LB fails.
tmp = dicoms\niops = ((tmp))]\nLooks like SharedFunctionalGroupsSequence and/or PlaneOrientationSequence might not be allowed tags, or ImageOrientationPatient could be missing in some of the series.
Did you try PerFrameFunctionalGroupsSequence? I guess SharedFunctionalGroupsSequence doesn't necessarily exist. I notice the competition hosts confirm PerFrameFunctionalGroupsSequence exists, but never see them confirming SharedFunctionalGroupsSequence are stored in every dcm 🤔
Thanks for the comment @wenxiaozhan. I just tried probing PerFrameFunctionalGroupsSequence and it works on the LB.
My concern is that we are unable to access multi-frame DICOM tags that would otherwise be available in non-multi-frame equivalents. For example, PixelSpacing, ImageOrientationPatient, etc.
Personally I think reading errors are caused for very uncommon outliers just present in test. And just about DICOM metadata standards. In my opinion you are safe to use try/except with safe returning values such axial if not IOP, trust instance number if not IPP, and so on.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3294734": "Hi @wenxiaozhan,\n\nThe hidden test set should now have the `PerFrameFunctionalGroupsSequence` tag included. It is not present in every DICOM, but only a subset.\n\nAs discussed on the [Data](https://www.kaggle.com/competitions/rsna-intracranial-aneurysm-detection/data) page, the tags in the test set DICOMs have been filtered to include only those in an allowlist. This is to help prevent leakage and other spurious correlations from affecting prediction quality.\n\nAlso, submissions will not be run on a different 68% test set after submissions are closed. Your submissions are already being run on the private test set whose scores will populate the final leaderboard, but we withhold those scores until after close.", "3294725": "Apologies for the delay and oversight on this issue. \n\n@ryanholbrook can comment when a pending update to the test set to restore the 'PerFrameFunctionalGroupsSequence' tag is complete. It was originally restored in the second update, but the dataset was reverted when other issues were identified with the update. \n\nIn general, we restricted the number of DICOM tags available in the test set so that models would utilize the images rather than DICOM metadata, but did want to share more DICOM tags as part of the public training set for future evaluation.", "3294714": "Hi organizers and teams,\n\ndespite lots of similar discussions about the hidden test set, we don't see the responce from competition organizers recently after the last update, thus we want to raise some discussion on such issue again.\n\nTo specify, we found there is more than one attributes missing in the test set. We try-except a lot and find that there must be something different in the multi-frame DICOMs in the test set. Such issue is not in the public training set, so this can't be prevented during local run. It is SOOO frustrating that we can't apply our model on a proportion of test set, that is just because of a file processing failure, while we have no idea what's missing.\n\nMoreover, we don't really understand why the training set is different from the test set. Obviously the split of training and hidden test set is not independent and random, otherwise we should encounter such problem in the training data. Therefore we have 0 confidence if there would be another problem in another 68% test set after the submission is closed.\n\nIs there any successful experience on using try-except or set default value, in order to skip the problematic multi-frame attributes? Also we would like to hear from the organizers to see if such issue is awared.\n\nCheers,", "3294886": "I think there is another multi-frame DICOM issue. Extracting the `ImageOrientationPatient` on the LB fails.\n\n```\ntmp = dicoms[0].SharedFunctionalGroupsSequence[0].PlaneOrientationSequence\niops = [_pos[i].ImageOrientationPatient for i in range(len(tmp))]\n```\n\nLooks like `SharedFunctionalGroupsSequence` and/or `PlaneOrientationSequence` might not be allowed tags, or `ImageOrientationPatient` could be missing in some of the series.", "3294728": "Personally I think reading errors are caused for very uncommon outliers just present in test. And just about DICOM metadata standards. In my opinion you are safe to use try/except with safe returning values such axial if not IOP, trust instance number if not IPP, and so on." } }