\nhttps://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_model_training_guide.md#1-cache-io-and-transforms-data-to-accelerate-training
\n
![\"cache-dataset\"](\"https://i.ibb.co/1fZ6Qjj/cache-dataset.png\")
i usually write my own libraries for data loading and augmentation.
\nbut for 3d medical data, i think it might be faster to use opensource.
i tried monai. but it is slow (but i could be my bug since i am a new user)
\ni wonder if there are other faster lib for pytorch 3d medical data?
how about torchIO, kornia or other data augmentation that uses gpu?
\n#my monai augmentation:\n\nimport monai.transforms as MT\n\nclass SpineDataset(Dataset):\n def __init__(self, df,augment=None):\n self.df=df\n self.augment=augment\n self.length=len(df)\n\n def __len__(self):\n return self.length\n\n def __str__(self):\n string = ''\n string += '\\tlen = %d\\n' % len(self)\n\n name = label_to_name.values()\n d = self.df[name].sum(0)\n for k in name:\n string += '%24s %3d (%0.3f) \\n'%(k, d.get(k,0), d.get(k,0)/len(self.df))\n return string\n\n def __getitem__(self, index):\n d = self.df.iloc[index]\n\n nii_file = data_dir +'/fixed_nii/%s.nii.gz'%d.StudyInstanceUID\n seg_file = data_dir +'/fixed_nii/%s.seg.nii.gz'%d.StudyInstanceUID\n r = {\n 'index': index,\n 'study_instance': d.StudyInstanceUID,\n 'image': nii_file,\n 'cervical': seg_file,\n }\n if self.augment is not None:\n r=self.augment(r)\n return r\n\nmt_train_transform = dotdict(\n do_load = MT.LoadImaged(['image', 'cervical']),\n #do_to_tensor = MT.ToTensord(['image', 'cervical']),\n do_cast = MT.CastToTyped(['image', 'cervical'], dtype=torch.float32),\n do_unsqueeze = MT.EnsureChannelFirstd(['image', 'cervical']),\n do_space = MT.Spacingd( ['image', 'cervical'], pixdim=(0.6, 0.6, 0.6), mode=('bilinear', 'nearest'),),\n do_random_crop = MT.RandSpatialCropd(['image', 'cervical'], roi_size=crop_size, random_size=False),\n do_normalise_image = MT.ScaleIntensityd(['image'], minv=0, maxv=1),\n do_normalise_cervical = MT.Lambdad(['cervical'], func=clean_cervical_label),\n do_pad = MT.SpatialPadd(['image', 'cervical'], spatial_size=crop_size),\n do_cast1 = MT.CastToTyped(['cervical'], dtype=[torch.long]),\n do_to_tensor1 = MT.FromMetaTensord(['image','cervical']),\n)\ntrain_transform = MT.Compose(list(mt_train_transform.values()))\ni usually write my own libraries for data loading and augmentation.
\nbut for 3d medical data, i think it might be faster to use opensource.
i tried monai. but it is slow (but i could be my bug since i am a new user)
\ni wonder if there are other faster lib for pytorch 3d medical data?
how about torchIO, kornia or other data augmentation that uses gpu?
\n#my monai augmentation:\n\nimport monai.transforms as MT\n\nclass SpineDataset(Dataset):\n def __init__(self, df,augment=None):\n self.df=df\n self.augment=augment\n self.length=len(df)\n\n def __len__(self):\n return self.length\n\n def __str__(self):\n string = ''\n string += '\\tlen = %d\\n' % len(self)\n\n name = label_to_name.values()\n d = self.df[name].sum(0)\n for k in name:\n string += '%24s %3d (%0.3f) \\n'%(k, d.get(k,0), d.get(k,0)/len(self.df))\n return string\n\n def __getitem__(self, index):\n d = self.df.iloc[index]\n\n nii_file = data_dir +'/fixed_nii/%s.nii.gz'%d.StudyInstanceUID\n seg_file = data_dir +'/fixed_nii/%s.seg.nii.gz'%d.StudyInstanceUID\n r = {\n 'index': index,\n 'study_instance': d.StudyInstanceUID,\n 'image': nii_file,\n 'cervical': seg_file,\n }\n if self.augment is not None:\n r=self.augment(r)\n return r\n\nmt_train_transform = dotdict(\n do_load = MT.LoadImaged(['image', 'cervical']),\n #do_to_tensor = MT.ToTensord(['image', 'cervical']),\n do_cast = MT.CastToTyped(['image', 'cervical'], dtype=torch.float32),\n do_unsqueeze = MT.EnsureChannelFirstd(['image', 'cervical']),\n do_space = MT.Spacingd( ['image', 'cervical'], pixdim=(0.6, 0.6, 0.6), mode=('bilinear', 'nearest'),),\n do_random_crop = MT.RandSpatialCropd(['image', 'cervical'], roi_size=crop_size, random_size=False),\n do_normalise_image = MT.ScaleIntensityd(['image'], minv=0, maxv=1),\n do_normalise_cervical = MT.Lambdad(['cervical'], func=clean_cervical_label),\n do_pad = MT.SpatialPadd(['image', 'cervical'], spatial_size=crop_size),\n do_cast1 = MT.CastToTyped(['cervical'], dtype=[torch.long]),\n do_to_tensor1 = MT.FromMetaTensord(['image','cervical']),\n)\ntrain_transform = MT.Compose(list(mt_train_transform.values()))\nTo solve your I/O, just prepare the dicom data in .jpg or if you have enough gpu power like me that even loading jpg is an I/O bottleneck for you, you can just save every single slice as a numpy array (.npy), jpg data is like ~45 gb, and numpy data would be ~187 gb on disk…
\nOnce you are done with I/O problem, you can just code your own data loading libraries and augmentation and it will probably be much faster than monai or torchio or something like that.
\nIf you are doing something which would need you to load multiple images from the same patients in one batch (for example, 4 patients 40 slices, for MIL or 3D modeling), then saving the dicoms as a 3d numpy array in shape (slices, height, width) would be best, loading time will be much better when loading the whole volume in a numpy array then loading 20 different numpy arrays, but if you are not interested in loading multiple images from a patient at once, then just go with the slice saving apporach mentioned earlier.
\", you can just save every single slice as a numpy array (.npy), jpg data is like ~45 gb, and numpy data would be ~187 gb on disk…\"
\nactually monai has cache data that stores sample in memory
\nhttps://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_model_training_guide.md#1-cache-io-and-transforms-data-to-accelerate-training
\n
\"Even with CacheDataset, we usually need to copy the same data to GPU memory for GPU random transforms or network computation in every epoch. An efficient approach is to cache the data to GPU memory directly, then every epoch can start from GPU computation immediately.\"
\nhttps://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb
\n![\"nsys-epoch-short\"](\"https://i.ibb.co/fngVfqw/nsys-epoch-short.png\")
\n\n\"Even with CacheDataset, we usually need to copy the same data to GPU memory for GPU random transforms or network computation in every epoch. An efficient approach is to cache the data to GPU memory directly, then every epoch can start from GPU computation immediately.\"\n\n\nhttps://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb \n"
},
{
"id": 1933188,
"postDate": "2022-09-10T10:29:17.540Z",
"content": "Looks good, although I don't have I/O bottleneck if I don't train small networks which does not give good result in my pipeline, small models scores are not worth talking about in my experience of this comp…
", "rawMarkdown": "Looks good, although I don't have I/O bottleneck if I don't train small networks which does not give good result in my pipeline, small models scores are not worth talking about in my experience of this comp...", "votes": 2 }, { "id": 1933322, "postDate": "2022-09-10T12:39:51.447Z", "content": "I think MONAI's PersistentDataset class is more useful for large datasets (compared to other Dataset interfaces it proposes).
\nThe I/O bottleneck would essentially eliminate after 1st epoch. But unfortunately, these are not useful during inference.
thanks all, i am now going over the MONAI tutorials now. so far things are working good.
", "rawMarkdown": "thanks all, i am now going over the MONAI tutorials now. so far things are working good." } ] }, { "id": 1932856, "postDate": "2022-09-10T04:16:21.340Z", "content": "Actually I did a quick benchmarking task a while back and both got similar results. In this competition as each slice is given as a DCM file, the I/O is the bottleneck.
\nBoth these libraries utilize itk in the background for dicom parsing, yielding similar results. IMHO, itk is slower than other dicom parsing libraries, but it comes with some additional perks, like orientation correction and automatic slice sorting.
To solve your I/O, just prepare the dicom data in .jpg or if you have enough gpu power like me that even loading jpg is an I/O bottleneck for you, you can just save every single slice as a numpy array (.npy), jpg data is like ~45 gb, and numpy data would be ~187 gb on disk…
\nOnce you are done with I/O problem, you can just code your own data loading libraries and augmentation and it will probably be much faster than monai or torchio or something like that.
\nIf you are doing something which would need you to load multiple images from the same patients in one batch (for example, 4 patients 40 slices, for MIL or 3D modeling), then saving the dicoms as a 3d numpy array in shape (slices, height, width) would be best, loading time will be much better when loading the whole volume in a numpy array then loading 20 different numpy arrays, but if you are not interested in loading multiple images from a patient at once, then just go with the slice saving apporach mentioned earlier.
\", you can just save every single slice as a numpy array (.npy), jpg data is like ~45 gb, and numpy data would be ~187 gb on disk…\"
\nactually monai has cache data that stores sample in memory
\nhttps://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_model_training_guide.md#1-cache-io-and-transforms-data-to-accelerate-training
\n
\"Even with CacheDataset, we usually need to copy the same data to GPU memory for GPU random transforms or network computation in every epoch. An efficient approach is to cache the data to GPU memory directly, then every epoch can start from GPU computation immediately.\"
\nhttps://github.com/Project-MONAI/tutorials/blob/main/acceleration/fast_training_tutorial.ipynb
\n
Looks good, although I don't have I/O bottleneck if I don't train small networks which does not give good result in my pipeline, small models scores are not worth talking about in my experience of this comp…
", "votes": 2, "replies": [] }, { "id": 1933322, "author_name": "Bardia Khosravi", "author_url": "", "post_date": "2022-09-10T12:39:51.447000", "content": "I think MONAI's PersistentDataset class is more useful for large datasets (compared to other Dataset interfaces it proposes).
\nThe I/O bottleneck would essentially eliminate after 1st epoch. But unfortunately, these are not useful during inference.
thanks all, i am now going over the MONAI tutorials now. so far things are working good.
", "votes": 0, "replies": [] } ] }, { "id": 1932856, "author_name": "Bardia Khosravi", "author_url": "", "post_date": "2022-09-10T04:16:21.340000", "content": "Actually I did a quick benchmarking task a while back and both got similar results. In this competition as each slice is given as a DCM file, the I/O is the bottleneck.
\nBoth these libraries utilize itk in the background for dicom parsing, yielding similar results. IMHO, itk is slower than other dicom parsing libraries, but it comes with some additional perks, like orientation correction and automatic slice sorting.