![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7480915%2F15ae0ef55b0e4dfb581d51b40f2b7ccc%2Fsample.png?generation=1671107884722133&alt=media\")
I read about the other discussions but none of them seemed to solve my problem. Whenever I submit the output, it just fails. The format of the csv file seems okay. But I always get Notebook Threw Exception\nYour notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset. error.
\nDo you have any idea why is it like that? :(
\n
Read this discussion
", "rawMarkdown": "Read this [discussion](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370156#2053540)", "votes": 1 }, { "id": 2066415, "postDate": "2022-12-15T17:07:26.190Z", "content": "I get your point. :-)
\nOne suggestion that may or may not be useful (it helped me); I notice that you are using converted images rather than converting everything yourself. You may want to consider running your conversion on the entire training set. If your processing has problems with certain files you may see those problems in the interactive run rather than getting a hidden error that results in \"Threw exception\".
Great idea, thank you!
", "rawMarkdown": "Great idea, thank you!", "votes": 1 } ] }, { "id": 2066142, "postDate": "2022-12-15T12:46:38.230Z", "content": "I have struggled with this as well (and \"Out of memory\" and \"Timeout\").
\nThe \"threw exception\" turned out to be the way some of the images in the hidden dataset are compressed which caused pydicom.dcmread to fail. I'm now using dicomsdl.open instead and that works (using https://www.kaggle.com/code/hey24sheep/frozen-packages-for-offline-use).
Thank you for your reply! But I'm already using that function to read dicom files:
\n\n", "rawMarkdown": "Thank you for your reply! But I'm already using that function to read dicom files:\n\n>def read_dicom(path, fix_monochrome = True):\n dicom = dicomsdl.open(path)\n data = dicom.pixelData(storedvalue=False) # storedvalue = True for int16 return otherwise float32\n data = data - np.min(data)\n data = data / np.max(data)\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":\n data = 1.0 - data\n return data" } ] }, { "id": 2066135, "postDate": "2022-12-15T12:38:09.420Z", "content": "def read_dicom(path, fix_monochrome = True):
\n
\n dicom = dicomsdl.open(path)
\n data = dicom.pixelData(storedvalue=False) # storedvalue = True for int16 return otherwise float32
\n data = data - np.min(data)
\n data = data / np.max(data)
\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":
\n data = 1.0 - data
\n return data
I read about the other discussions but none of them seemed to solve my problem. Whenever I submit the output, it just fails. The format of the csv file seems okay. But I always get Notebook Threw Exception\nYour notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset. error.
\nDo you have any idea why is it like that? :(
\n
That's interesting. Maybe PhotometricInterpretation is missing from some of the images in the test set? I just have:
\n dicom1 = dicomsdl.open(VAL_PATH + str(row['patient_id']) + '/' + str(row['image_id']) + '.dcm')
\n image1 = dicom1.pixelData(storedvalue=True)
\n img_max = np.amax(image1)
\n img_min = np.amin(image1)
\n image1 = ((image1 - img_min) / (img_max - img_min))
\n if image1.mean() > 128: # Inverted image detected. Invert again
\n image1 = 255 - image1
\n img = crop_image(Image.fromarray(image1))
Thank you for sharing the function that worked out for you but I don't think it fails because of the PhotometricInterpretation in test set since this function is used by many else too. I set my notebook as public. If you have time, I would be really happy to know my noob mistake.
\nhttps://www.kaggle.com/code/truthisneverlinear/rsna-efficientnetv2-xl-pytorch
", "rawMarkdown": "Thank you for sharing the function that worked out for you but I don't think it fails because of the PhotometricInterpretation in test set since this function is used by many else too. I set my notebook as public. If you have time, I would be really happy to know my noob mistake.\n\nhttps://www.kaggle.com/code/truthisneverlinear/rsna-efficientnetv2-xl-pytorch" } ] } ], "comments": [ { "id": 2070551, "author_name": "Hey24sheep", "author_url": "", "post_date": "2022-12-20T06:22:59.087000", "content": "Read this discussion
", "votes": 1, "replies": [] }, { "id": 2066415, "author_name": "lmadsen", "author_url": "", "post_date": "2022-12-15T17:07:26.190000", "content": "I get your point. :-)
\nOne suggestion that may or may not be useful (it helped me); I notice that you are using converted images rather than converting everything yourself. You may want to consider running your conversion on the entire training set. If your processing has problems with certain files you may see those problems in the interactive run rather than getting a hidden error that results in \"Threw exception\".
Great idea, thank you!
", "votes": 1, "replies": [] } ] }, { "id": 2066142, "author_name": "lmadsen", "author_url": "", "post_date": "2022-12-15T12:46:38.230000", "content": "I have struggled with this as well (and \"Out of memory\" and \"Timeout\").
\nThe \"threw exception\" turned out to be the way some of the images in the hidden dataset are compressed which caused pydicom.dcmread to fail. I'm now using dicomsdl.open instead and that works (using https://www.kaggle.com/code/hey24sheep/frozen-packages-for-offline-use).
Thank you for your reply! But I'm already using that function to read dicom files:
\n\n", "votes": 0, "replies": [] } ] }, { "id": 2066254, "author_name": "lmadsen", "author_url": "", "post_date": "2022-12-15T14:15:02.130000", "content": "def read_dicom(path, fix_monochrome = True):
\n
\n dicom = dicomsdl.open(path)
\n data = dicom.pixelData(storedvalue=False) # storedvalue = True for int16 return otherwise float32
\n data = data - np.min(data)
\n data = data / np.max(data)
\n if fix_monochrome and dicom.PhotometricInterpretation == \"MONOCHROME1\":
\n data = 1.0 - data
\n return data
That's interesting. Maybe PhotometricInterpretation is missing from some of the images in the test set? I just have:
\n dicom1 = dicomsdl.open(VAL_PATH + str(row['patient_id']) + '/' + str(row['image_id']) + '.dcm')
\n image1 = dicom1.pixelData(storedvalue=True)
\n img_max = np.amax(image1)
\n img_min = np.amin(image1)
\n image1 = ((image1 - img_min) / (img_max - img_min))
\n if image1.mean() > 128: # Inverted image detected. Invert again
\n image1 = 255 - image1
\n img = crop_image(Image.fromarray(image1))
Thank you for sharing the function that worked out for you but I don't think it fails because of the PhotometricInterpretation in test set since this function is used by many else too. I set my notebook as public. If you have time, I would be really happy to know my noob mistake.
\nhttps://www.kaggle.com/code/truthisneverlinear/rsna-efficientnetv2-xl-pytorch
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2070551": "Read this [discussion](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370156#2053540)", "2066415": "I get your point. :-)\nOne suggestion that may or may not be useful (it helped me); I notice that you are using converted images rather than converting everything yourself. You may want to consider running your conversion on the entire training set. If your processing has problems with certain files you may see those problems in the interactive run rather than getting a hidden error that results in \"Threw exception\".", "2066142": "I have struggled with this as well (and \"Out of memory\" and \"Timeout\").\nThe \"threw exception\" turned out to be the way some of the images in the hidden dataset are compressed which caused pydicom.dcmread to fail. I'm now using dicomsdl.open instead and that works (using https://www.kaggle.com/code/hey24sheep/frozen-packages-for-offline-use).", "2066135": "I read about the other discussions but none of them seemed to solve my problem. Whenever I submit the output, it just fails. The format of the csv file seems okay. But I always get *Notebook Threw Exception\nYour notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset.* error.\n\nDo you have any idea why is it like that? :(\n\n", "2066254": "That's interesting. Maybe PhotometricInterpretation is missing from some of the images in the test set? I just have:\n\t\tdicom1 = dicomsdl.open(VAL_PATH + str(row['patient_id']) + '/' + str(row['image_id']) + '.dcm')\n\t\timage1 = dicom1.pixelData(storedvalue=True)\n\t\timg_max = np.amax(image1)\n\t\timg_min = np.amin(image1)\n\t\timage1 = ((image1 - img_min) / (img_max - img_min))\n\t\tif image1.mean() > 128: # Inverted image detected. Invert again\n\t\t\timage1 = 255 - image1\n\t\timg = crop_image(Image.fromarray(image1))\n" } }