{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import os\nimport random\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport pydicom\n\nfrom tqdm import tqdm_notebook","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# Get directory names/locations\ndata_root = os.path.abspath(\"../input/rsna-intracranial-hemorrhage-detection/\")\n\ntrain_img_root = data_root + \"/stage_1_train_images/\"\ntest_img_root  = data_root + \"/stage_1_test_images/\"\n\ntrain_labels_path = data_root + \"/stage_1_train.csv\"\ntest_labels_path  = data_root + \"/stage_1_test.csv\"\n\n# Create list of paths to actual training data\ntrain_img_paths = os.listdir(train_img_root)\ntest_img_paths  = os.listdir(test_img_root)\n\n# Dataset size\nnum_train = len(train_img_paths)\nnum_test  = len(test_img_paths)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"def create_efficient_df(data_path):\n    \n    # Define the datatypes we're going to use\n    final_types = {\n        \"ID\": \"str\",\n        \"Label\": \"float16\"\n    }\n    features = list(final_types.keys())\n    \n    # Use chunks to import the data so that less efficient machines can only use a \n    # specific amount of chunks on import\n    df_list = []\n\n    chunksize = 1_000_000\n\n    for df_chunk in pd.read_csv(data_path, dtype=final_types, chunksize=chunksize): \n        df_list.append(df_chunk)\n        \n    df = pd.concat(df_list)\n    df = df[~df.isin([np.nan, np.inf, -np.inf]).any(1)]\n\n    del df_list\n\n    return df\n\ntrain_labels_df = create_efficient_df(train_labels_path)\ntrain_labels_df[train_labels_df[\"Label\"] > 0].head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"hem_types = [\n    \"epidural\",\n    \"intraparenchymal\",\n    \"intraventricular\",\n    \"subarachnoid\",\n    \"subdural\",\n    \"any\"\n]\n\nnew_cols = [\n    \"id\",\n    \"type_0\",\n    \"type_1\",\n    \"type_2\",\n    \"type_3\",\n    \"type_4\",\n    \"type_5\"\n]\n\nnum_ids = int(train_labels_df.shape[0] / len(hem_types))\nprint(\"Number of unique patient IDs: {}\".format(num_ids))\n\nempty_array = np.ones((num_ids, len(new_cols)))\nhem_df = pd.DataFrame(data=empty_array, columns=new_cols)\n\n# Fill in the ID of each image\nhem_df[\"id\"] = list(train_labels_df.iloc[::len(hem_types)][\"ID\"].str.split(pat=\"_\").str[1])\n    \n# Fill in the categorical columns of each image\nfor hem_ix, hem_col in enumerate(list(hem_df)[1:]):\n    hem_df[hem_col] = list(train_labels_df.iloc[hem_ix::len(hem_types), 1])\n    \nhem_df.info()\nhem_df[hem_df[\"type_5\"] > 0].head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nHello there! My name is Mick and I'm continuing my analysis of the RSNA Intracranial Hemorrhage dataset. I'll be analyzing the dataset of DICOM images in conjunction with Pydicom, a Python package specifically for parsing .dcm files. This notebook is going to be all about the DICOM file format and trying to parse out clinical, diagnostic, and locational information from DICOM files.\n\nIf you've read [my EDA notebook for this dataset](https://www.kaggle.com/smit2300/hemorrhage-eda-encoding-dicom-introduction) you can see how I encoded the dataset labels to be a bit more analysis friendly. If you dont want to look through that then the methods I used are recreated above in this notebook.\n\nSo with that bit of preprocessing knowledge out of the way let's get into some DICOM analysis!"},{"metadata":{"trusted":true},"cell_type":"code","source":"hem_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DICOM Refresher\n\nThis is explained in a bunch of other notebooks throughout this competition, but briefly: a DICOM image is one created specifically for sending medical images between clinicians that contain contextual information for the clinician. The purpose of the format is to send an image that contains some of its own labeling information directly in the file. Below is an example of the info in a DICOM image for this dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"random_ix = random.randint(0, len(train_img_paths))\nrandom_path = train_img_root + train_img_paths[random_ix]\n\ndcm_info = pydicom.dcmread(random_path)\nprint(\"===IMAGE MEDICAL INFO===\")\nprint(dcm_info)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# DICOM Information DataFrame\n\nNow that we know how to extract information from a .dcm file we can choose which of these data fields is going to be most useful for our prediction of intracranial hemorrhage. We can enrich our exploration of this dataset by splitting out the dataset by patient, rather than just individual image ID as well. If we get enough images of a patient then we can try to recreate a clinical trial where a patient is being imaged for this study. Then based on the positional information in the .dcm files we can possible do some scan recreation. Let's expand our `hem_df` dataframe to include the following columns in addition to our hemorrhage labels:\n * Image ID\n * Patient ID\n * Position of scan image in patient\n * Orientation of scan image in patient"},{"metadata":{"trusted":true},"cell_type":"code","source":"DEV_RUN = True\n\nif DEV_RUN:\n    SET_SIZE = 50_000\n    print(\"Creating {} element subset of hemorrhage dataset\".format(SET_SIZE))\n    hem_df = hem_df.iloc[:SET_SIZE, :]\n    \npatient_ids  = np.zeros((hem_df.shape[0],))\npositions    = np.zeros((hem_df.shape[0]))\norientations = np.zeros((hem_df.shape[0]))\n\nhem_df[\"patient_id\"]    = patient_ids\nhem_df[\"position_0\"]    = positions\nhem_df[\"position_1\"]    = positions\nhem_df[\"position_2\"]    = positions\nhem_df[\"orientation_0\"] = orientations\nhem_df[\"orientation_1\"] = orientations\nhem_df[\"orientation_2\"] = orientations\nhem_df[\"orientation_3\"] = orientations\nhem_df[\"orientation_4\"] = orientations\nhem_df[\"orientation_5\"] = orientations\n\ndel patient_ids\ndel positions\ndel orientations\n\nfor row_ix, row in tqdm_notebook(hem_df.iterrows()):\n    \n    full_path = train_img_root + \"ID_\" + row[\"id\"] + \".dcm\"\n    dcm_info  = pydicom.dcmread(full_path)\n    \n    patient_id  = dcm_info.PatientID.split(\"_\")[1]\n    position    = dcm_info.ImagePositionPatient\n    orientation = dcm_info.ImageOrientationPatient\n        \n    hem_df[\"patient_id\"].iloc[row_ix]  = patient_id\n    \n    hem_df[\"position_0\"].iloc[row_ix]    = position[0]\n    hem_df[\"position_1\"].iloc[row_ix]    = position[1]\n    hem_df[\"position_2\"].iloc[row_ix]    = position[2]\n    \n    hem_df[\"orientation_0\"].iloc[row_ix] = orientation[0]\n    hem_df[\"orientation_1\"].iloc[row_ix] = orientation[1]\n    hem_df[\"orientation_2\"].iloc[row_ix] = orientation[2]\n    hem_df[\"orientation_3\"].iloc[row_ix] = orientation[3]\n    hem_df[\"orientation_4\"].iloc[row_ix] = orientation[4]\n    hem_df[\"orientation_5\"].iloc[row_ix] = orientation[5]\n        \nhem_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analyzing Duplicates\n\nNow that we have more metadata loaded into our dataframe, we can start to analyze based on each patient. Since I'm using a subsample of the data and not all patients are guaranteed to have the same amount of CT images, I'm going to find the patient that has the *most* repeats in this dataset and show all of the images associated with that patient."},{"metadata":{"trusted":true},"cell_type":"code","source":"dup_df = hem_df.pivot_table(index=['patient_id'], aggfunc='size')\ndup_df = dup_df[dup_df > 1]\n\npatient_df = hem_df[hem_df[\"patient_id\"] == dup_df.idxmax()]\npatient_df = patient_df.sort_values(\"id\")\n\nprint(\"=======PATIENT ID: {}=======\".format(patient_df[\"patient_id\"].iloc[0]))\n\ndef show_patient_frames(df):\n    \n    id_list = list(df[\"id\"])\n\n    # Used for subplots but that's been deprecated for larger subset sizes\n    num_cols = 3\n    num_rows = int(len(id_list) / num_cols)\n    \n    id_ix = 0\n    for row in range(num_rows):\n        for col in range(num_cols):\n            \n            fig = plt.figure(figsize=(8,8))\n    \n            current_id = id_list[id_ix]\n            full_path = train_img_root + \"ID_\" + current_id + \".dcm\"\n            dcm_info = pydicom.dcmread(full_path)\n            pixel_data = dcm_info.pixel_array\n\n            plt.imshow(pixel_data)\n\n            plt.grid(\"off\")\n            plt.axis(\"off\")\n#             axes[row, col].set_title(\"Image ID: {}\\nEpidural: {}\\nIntraparenchymal: {}\\nIntraventricular: {}\\nSubdural: {}\\nSubarachnoid: {}\"\n#                  .format(current_id, df.iloc[id_ix, 1], df.iloc[id_ix, 2], df.iloc[id_ix, 3], df.iloc[id_ix, 4], df.iloc[id_ix, 5]))\n\n            plt.title(\"Image ID: {}\\nx: {} y: {} z: {}\"\n                    .format(current_id, df.iloc[id_ix, 8], df.iloc[id_ix, 9], df.iloc[id_ix, 10]))\n\n            id_ix += 1\n\n    plt.show()\n    \nshow_patient_frames(patient_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Conclusion\nFor now I'll leave it there. We can do further analysis like trying to group by location within the patient but the lack of access to the full dataset makes further analysis a bit difficult. Until I can find a way to work with all of the images in the dataset and split things out by StudyInstanceUID I'll call it here."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}