👋 Greetings!
\nI have question about StudyInstanceUID and patient_id.
\nUsually, in DICOM files the ds.StudyInstanceUID gives an unique identifier for each examination. Patients on the other hand have a ds.PatientID which distinguishes them from other patients.
So, StudyInstanceUID allows to distinguish one examination from another. (And then some time label will tell where the examination is positioned on the time axis.)
If I am not mistaken, in this RSNA dataset for screening mammography, and especially the metadata, patient_id seems to equal ds.StudyInstanceUID? @sohier
How are we able to stratify the data properly for deep learning, if there is no anonymous ID to tell if the same patient has several examinations in the dataset? Or should we conclude that each patient has only one examination (with several images) in the dataset?
\nEDIT: Here is my investigation regarding the StudyInstanceUID: 🎗️ [16-bit post-processing] RSNA BC Detection
Best,
\nAntti 👍
", "messages": [ { "id": 2145446, "postDate": "2023-02-15T04:45:08.480Z", "content": "\"How are we able to stratify the data properly for deep learning, if there is no anonymous ID to tell if the same patient has several examinations in the dataset? Or should we conclude that each patient has only one examination (with several images) in the dataset?\"
\nFrom my perspective, I assume that there are some duplicate DICOM images, but fewer. In addition, I normally distinguish them by implementing Pandas library (e.g. Series.unique). And match the number to the number of images in the competition dataset (54k).
\nAs you know, the process of training a Neural Network is a repeated job, not just a single process. That means, in each epoch, the patients' data is used to train our model repeatedly.
", "rawMarkdown": "\"How are we able to stratify the data properly for deep learning, if there is no anonymous ID to tell if the same patient has several examinations in the dataset? Or should we conclude that each patient has only one examination (with several images) in the dataset?\"\n\nFrom my perspective, I assume that there are some duplicate DICOM images, but fewer. In addition, I normally distinguish them by implementing Pandas library (e.g. Series.unique). And match the number to the number of images in the competition dataset (54k).\n\nAs you know, the process of training a Neural Network is a repeated job, not just a single process. That means, in each epoch, the patients' data is used to train our model repeatedly.", "replies": [ { "id": 2146433, "postDate": "2023-02-15T21:36:16.557Z", "content": "\n\nThat means, in each epoch, the patients' data is used to train our model repeatedly.
\n
Perhaps all the more reason to make sure that there is no patient-wise leakage between train and validation subsets? So that we don't accidentally learn (i.e., fit to) the anatomy of some particular patient. Or that we don't get a model, which we assume to generalize well based on our validation set, but in truth is only good for breast cancer patterns distinctive (again) to some particular patient?
", "rawMarkdown": ">That means, in each epoch, the patients' data is used to train our model repeatedly.\n\nPerhaps all the more reason to make sure that there is no patient-wise leakage between train and validation subsets? So that we don't accidentally learn (*i.e.*, fit to) **the anatomy of some particular patient**. Or that we don't get a model, which we assume to generalize well based on our validation set, but in truth is only good for breast cancer patterns distinctive (again) **to some particular patient**?", "replies": [ { "id": 2146638, "postDate": "2023-02-16T02:58:54.433Z", "content": "Yes, the probability of a bias can occur in terms of the quality of a dataset. Assume that the 10,000 paitent images are distinguished well, but 2000 don't. It might follow the cases:
\nPreprosessing is a good option to overcome the shortage of our data. Because it gives us a wide range of new data derived from the orginial dataset. It will be a good resource in terms of quantity and quality.
Our model is difficult to train the medical images of DICOM. Because the patterns of tumers might be challenged to the model. (In this case, we have 2 options)
\nOp1 : change the model.
\nOp2 : have to think about making standard of factors, especially labels.
Include outer dataset in order to fill the gap between biased data and non-biased one.
👋 Greetings!
\nI have question about StudyInstanceUID and patient_id.
\nUsually, in DICOM files the ds.StudyInstanceUID gives an unique identifier for each examination. Patients on the other hand have a ds.PatientID which distinguishes them from other patients.
So, StudyInstanceUID allows to distinguish one examination from another. (And then some time label will tell where the examination is positioned on the time axis.)
If I am not mistaken, in this RSNA dataset for screening mammography, and especially the metadata, patient_id seems to equal ds.StudyInstanceUID? @sohier
How are we able to stratify the data properly for deep learning, if there is no anonymous ID to tell if the same patient has several examinations in the dataset? Or should we conclude that each patient has only one examination (with several images) in the dataset?
\nEDIT: Here is my investigation regarding the StudyInstanceUID: 🎗️ [16-bit post-processing] RSNA BC Detection
Best,
\nAntti 👍
", "rawMarkdown": "👋 Greetings!\n\nI have question about StudyInstanceUID and patient_id.\n\nUsually, in `DICOM` files the `ds.StudyInstanceUID` gives an unique identifier for each examination. Patients on the other hand have a `ds.PatientID` which distinguishes them from other patients.\n\nSo, `StudyInstanceUID` allows to distinguish one examination from another. (And then some time label will tell where the examination is positioned on the time axis.)\n\nIf I am not mistaken, in this RSNA dataset for screening mammography, and especially the metadata, `patient_id` seems to equal `ds.StudyInstanceUID`? @sohier \n\n**How are we able to stratify** the data properly for deep learning, if there is no anonymous ID to tell if the same patient has several **examinations** in the dataset? Or should we conclude that each patient has only one examination (with several images) in the dataset?\n\n**EDIT**: Here is my investigation regarding the `StudyInstanceUID`: [🎗️ [16-bit post-processing] RSNA BC Detection](https://www.kaggle.com/code/anttiisosalo/16-bit-post-processing-rsna-bc-detection?scriptVersionId=119182142&cellId=28)\n\nBest,\n\nAntti 👍" } ], "comments": [ { "id": 2145446, "author_name": "Hyunsoo Lee 1010", "author_url": "", "post_date": "2023-02-15T04:45:08.480000", "content": "\"How are we able to stratify the data properly for deep learning, if there is no anonymous ID to tell if the same patient has several examinations in the dataset? Or should we conclude that each patient has only one examination (with several images) in the dataset?\"
\nFrom my perspective, I assume that there are some duplicate DICOM images, but fewer. In addition, I normally distinguish them by implementing Pandas library (e.g. Series.unique). And match the number to the number of images in the competition dataset (54k).
\nAs you know, the process of training a Neural Network is a repeated job, not just a single process. That means, in each epoch, the patients' data is used to train our model repeatedly.
", "votes": 0, "replies": [ { "id": 2146433, "author_name": "Antti Isosalo", "author_url": "", "post_date": "2023-02-15T21:36:16.557000", "content": "\n\nThat means, in each epoch, the patients' data is used to train our model repeatedly.
\n
Perhaps all the more reason to make sure that there is no patient-wise leakage between train and validation subsets? So that we don't accidentally learn (i.e., fit to) the anatomy of some particular patient. Or that we don't get a model, which we assume to generalize well based on our validation set, but in truth is only good for breast cancer patterns distinctive (again) to some particular patient?
", "votes": 0, "replies": [ { "id": 2146638, "author_name": "Hyunsoo Lee 1010", "author_url": "", "post_date": "2023-02-16T02:58:54.433000", "content": "Yes, the probability of a bias can occur in terms of the quality of a dataset. Assume that the 10,000 paitent images are distinguished well, but 2000 don't. It might follow the cases:
\nPreprosessing is a good option to overcome the shortage of our data. Because it gives us a wide range of new data derived from the orginial dataset. It will be a good resource in terms of quantity and quality.
Our model is difficult to train the medical images of DICOM. Because the patterns of tumers might be challenged to the model. (In this case, we have 2 options)
\nOp1 : change the model.
\nOp2 : have to think about making standard of factors, especially labels.
Include outer dataset in order to fill the gap between biased data and non-biased one.