{"cells":[{"metadata":{},"cell_type":"markdown","source":"This notebook is the first in a series of notebooks which use the [fast.ai](https://fast.ai) Medical Imaging API built on top of Pytorch. It is modeled after [Jeremy Howard's notebooks](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/114214) from the 2019 RSNA Intracranial Hemorrhage Detection Kaggle Challenge.\n\nFirst, we need to upgrade the `fastai` library to `version 2.0.x` and import the relevant libraries."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install fastai --upgrade >/dev/null","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# order of importing the fastai libraries matters here, possibly due to a namespace conflict\nfrom fastai.medical.imaging import *\nfrom fastai.basics import *\n\nimport glob","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Exploring the file tree\nWe know from the data overview that the data are split into `train` and _public_ `test` groups of 7,279 and 650 studies, respectively. Each study is organized in standard DICOM format with the top-level directory labeled with the `StudyInstanceUID` and the images organized under a `SeriesInstanceUID` sub-directory with each individual file labeled by the `SOPInstanceUID` in the following path format `<StudyInstanceUID>/<SeriesInstanceUID>/<SOPInstanceUID>.dcm`."},{"metadata":{"trusted":true},"cell_type":"code","source":"path = Path('../input/rsna-str-pulmonary-embolism-detection')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls {path}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path_trn = path/'train'\ndirs_trn = path_trn.ls()\ndirs_trn[:5].attrgot('name')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path_tst = path/'test'\ndirs_tst = path_tst.ls()\nprint(f'Number of training studies: {len(dirs_trn)}')\nprint(f'Number of test studies: {len(dirs_tst)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Creating a DataFrame of DICOM metadata\nNow we'll proceed to extract the metadata from the DICOM files and put it into a `pandas.DataFrame`, which we'll save in `feather` format for later use."},{"metadata":{"trusted":true},"cell_type":"code","source":"fns_trn = L(glob.glob(f'{path_trn}/**/*.dcm', recursive=True))\nfns_trn = fns_trn.map(Path)\nprint(len(fns_trn))\nfns_trn[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since there are ~1.8 million images in the training dataset, it's impractical to extract metadata for every image...\n\nSo, we'll select one image from each study for inclusion in our DICOM metadata Data Frame."},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc, os\ndel(fns_trn)\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fns_trn = L()\nfor r, d, f in os.walk(path_trn):\n    if f:\n        fn = Path(f'{r}/{f[0]}')\n        fns_trn.append(fn)\nprint(len(fns_trn))\nfns_trn[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fn = fns_trn[0]\ndcm = fn.dcmread()\ndcm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_trn = pd.DataFrame.from_dicoms(fns_trn, px_summ=False)\ndf_trn.to_feather('df_trn.fth')\ndf_trn.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll clean up here before proceeding."},{"metadata":{"trusted":true},"cell_type":"code","source":"del(df_trn, fns_trn)\ngc.collect();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Creating a DataFrame of labels\nHere we'll extract the labels from `train.csv` and save them in `feather` format for future use."},{"metadata":{"trusted":true},"cell_type":"code","source":"path_lbls = path/'train.csv'\nlbls = pd.read_csv(path_lbls)\nprint(lbls.shape)\nlbls.drop_duplicates(['StudyInstanceUID', 'SOPInstanceUID'], inplace=True)\nprint(lbls.shape)\nlbls.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks like the labels are in a nice, readable format, so we'll save them in `feather` format."},{"metadata":{"trusted":true},"cell_type":"code","source":"lbls.to_feather('lbls.fth')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The [second notebook in the series](https://www.kaggle.com/wfwiggins203/exploring-the-dicom-metadata-images-with-fast-ai) explores the DICOM metadata a little further and looks at a sampling of the images. I try inject some extra domain knowledge from my day job as a radiologist."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}