{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n \n \n \n \n \n
\n\nHey there! I'm going to try to get up and running with a potentially new image format and try to explore some of this dataset. The RSNA Intracranial Hemorrhage dataset was provided to allow Kaggle users to perform classification on images and determine whether or not they contain hemorrhages. Since my background is more medically oriented I'll just be working on some basic analysis and medical image processing techniques, while doing my best to explain any medical background that might be interesting or pertinent as I go through this dataset.\n\nIf you're at all curious about the medical background for this dataset and want to learn more about hemorrhages, feel free to check out [the medical introduction kernel I've created for this competition](https://www.kaggle.com/smit2300/hemorrhage-medical-introduction).\n\nI'll be focusing more on the actual dataset in this notebook and will assume a little bit of user understanding of the medical background as I move through my exploratory analysis here. Let's jump right in!"},{"metadata":{},"cell_type":"markdown","source":"# BASH Dataset Exploration\n\nThe first thing I like to do when I come across a new dataset is get a bit of familiarity with the data directory structure and what sorts of files we'll be dealing with. In a clinical setting this is an extremely important step and has to be done in a scalable way. If someone prepares a dataset and doesn't include navigation tips or instructions to get a clinician started, then a lot of time can be wasted trying to just understand what sort of data is being sent over. These are some of the basic UNIX commands that I run when I come across new medical or other styles of dataset.\n\nIf you're already comfortable with the dataset and with DICOM images then you can go ahead and skip through this section."},{"metadata":{},"cell_type":"markdown","source":"Let's get familiar with the directory layout. I just want to see which files are available\nto us in this dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"!printf \"==Data Root==\\n\"\n!ls ../input\n!printf \"\\n==Data Structure==\\n\"\n!ls ../input/rsna-intracranial-hemorrhage-detection\n\n# Alternatively if you've downloaded the dataset locally you could run something like:\n# $tree -L 2 ../input","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Rather than use Python's `os` module functions, we can simply use UNIX commands to get a \nquicker assessment of how many image files are in our training dataset. \n\nRemember, the output of the `wc` command is: number of lines | number of words | number of bytes. \n\nFor our purposes (finding number of files in `ls` output) we're only concerned with the first output."},{"metadata":{"trusted":true},"cell_type":"code","source":"!printf \"\\n==Profiling Training Dataset==\\n\"\n!ls -U ../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ | wc","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see the first 5 image filenames in the training directory. This will be important for\nany sort of regular expressions that might become necessary later in our analysis when\ndealing with filenames that have a specific format."},{"metadata":{"trusted":true},"cell_type":"code","source":"!printf \"\\n==Training Image Filename Examples==\\n\"\n!ls -U ../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ | head -5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see what the first 5 rows of one of the label csv files looks like as well."},{"metadata":{"trusted":true},"cell_type":"code","source":"!printf \"\\n==Labels CSV Syntax==\\n\"\n!head -5 ../input/rsna-intracranial-hemorrhage-detection/stage_1_train.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a bit messy. Our CSV file contains 6 rows for each CT image. We'll probably have to cut that down a bit later with some encoding techniques."},{"metadata":{},"cell_type":"markdown","source":"And lastly I want to see what the heck a .dcm file is. I'm assuming a lot of people\nworking through this dataset will be seeing this format for the first time so I think\nit'll be a good idea to give a solid foundation to people trying to understand it."},{"metadata":{"trusted":true},"cell_type":"code","source":"!printf \"\\n==DCM File==\\n\"\n!file ../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_b48228b12.dcm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So this image format is very specifically tuned to the display and transfer of medical images--even UNIX knows it! It turns out that Python has a wonderful module called `Pydicom` that makes it possible to go through your whole life without knowing anything about the DICOM image format while still analyzing medical images. If you want a spoiler, you can simply get image data in a numpy array without observing the image data at all. \n\nHowever, with some careful understanding of the file format, it's possible to open up a whole new world of medical data just contained within the images if you are analyzing DICOM images. But more on that later since we can't use the metadata as features for our submission to this competition. For now let's get familiar with this labeling system."},{"metadata":{},"cell_type":"markdown","source":"# EDA\n\n## Python Libary Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport random\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport pydicom\nimport PIL\n\nfrom tqdm import tqdm, tqdm_notebook","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Path Exploration"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Get directory names/locations\ndata_root = os.path.abspath(\"../input/rsna-intracranial-hemorrhage-detection/\")\n\ntrain_img_root = data_root + \"/stage_1_train_images/\"\ntest_img_root = data_root + \"/stage_1_test_images/\"\n\ntrain_labels_path = data_root + \"/stage_1_train.csv\"\ntest_labels_path = data_root + \"/stage_1_test.csv\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Image Path Variables"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create list of paths to actual training data\ntrain_img_paths = os.listdir(train_img_root)\ntest_img_paths = os.listdir(test_img_root)\n\n# Dataset size\nnum_train = len(train_img_paths)\nnum_test = len(test_img_paths)\n\nprint(\"Train dataset consists of {} images\".format(num_train))\nprint(\"Test dataset consists of {} images\".format(num_test))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Large Dataset Importing\n\nSince we're looking at a relatively large dataset I'm going to use a method to load in our CSV to memory that might be a bit of overkill. I'm using a method described by Kaggle user [szelee](https://www.kaggle.com/szelee) in his excellent [notebook on NYC Taxi rides](https://www.kaggle.com/szelee/how-to-import-a-csv-file-of-55-million-rows). \n\nThe main method I'm going to be borrowing from that notebook is the changing of dataset import datatypes. Since we're only importing the labels this might not be necessary, but even datasets with only two columns can end up using up a significant amount of RAM."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_tmp = pd.read_csv(\n train_labels_path,\n nrows=5\n)\n\ncolumns = list(df_tmp.columns)\n\nprint(\"\\nFeatures in training labels:\")\nfor column in columns:\n print(column)\n\nprint(\"\\nDataFrame Datatype Information:\")\nprint(df_tmp.info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndef create_efficient_df(data_path):\n \n # Define the datatypes we're going to use\n final_types = {\n \"ID\": \"str\",\n \"Label\": \"float32\"\n }\n features = list(final_types.keys())\n \n # Use chunks to import the data so that less efficient machines can only use a \n # specific amount of chunks on import\n df_list = []\n\n chunksize = 1_000_000\n\n for df_chunk in pd.read_csv(data_path, dtype=final_types, chunksize=chunksize): \n df_list.append(df_chunk)\n \n df = pd.concat(df_list)\n df = df[~df.isin([np.nan, np.inf, -np.inf]).any(1)]\n\n del df_list\n\n return df\n\ntrain_labels_df = create_efficient_df(train_labels_path)\n# test_labels_df = create_efficient_df(test_labels_path)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels_df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"About 80MB of RAM used even on our optimized import! I really like the attention to detail that this process brings regardless of whether or not it's necessary. As an engineer it's an important element of the job being able to know the exact certainty that's necessary for a given task and for data scientists this certainty comes in the form of datatype precision. \n\nPandas is a wonderful tool, but if you're not careful it can be a bit bloated. It only takes a couple of minutes and some technical care and know-how to really prune down an otherwise unwieldy data structure in your machine's memory. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Syntax = Which image + hemorrhage type, Probability image contains that hemorrhage type\ntrain_labels_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Label Encoding\nThis labeling dataset is pretty large--I would argue about 6x larger than it needs to be. Having a new row for each prediction seems like a waste of vertical scaling space to me. Let's try to clean it up in a way that will be easier for future machine learning/deep learning models to work with futher down the line.\n\nI would like the dataframe to have one row per image and within that row contain the certainty the image contains each type of hemorrhage as a column of that row. \n\nWe'll encode a new dataframe to use the following names corresponding to each hemorrhage type:\n * type_0: epidural\n * type_1: intraparenchymal\n * type_2: intraventricular\n * type_3: subarachnoid\n * type_4: subdural\n * type_5: any"},{"metadata":{"trusted":true},"cell_type":"code","source":"hem_types = [\n \"epidural\",\n \"intraparenchymal\",\n \"intraventricular\",\n \"subarachnoid\",\n \"subdural\",\n \"any\"\n]\n\nnew_cols = [\n \"id\",\n \"type_0\",\n \"type_1\",\n \"type_2\",\n \"type_3\",\n \"type_4\",\n \"type_5\"\n]\n\nnum_ids = int(train_labels_df.shape[0] / len(hem_types))\nprint(\"Number of unique patient IDs: {}\".format(num_ids))\n\nempty_array = np.ones((num_ids, len(new_cols)))\nnew_labels_df = pd.DataFrame(data=empty_array, columns=new_cols)\n\n# Fill in the ID of each image\nnew_labels_df[\"id\"] = list(train_labels_df.iloc[::len(hem_types)][\"ID\"].str.split(pat=\"_\").str[1])\n \n# Fill in the categorical columns of each image\nfor hem_ix, hem_col in enumerate(list(new_labels_df)[1:]):\n new_labels_df[hem_col] = list(train_labels_df.iloc[hem_ix::len(hem_types), 1])\n \nnew_labels_df.sample(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's much better. Now instead of a dataset containing 4 million rows, we have one that's down to 674K and each row has a richer set of information."},{"metadata":{},"cell_type":"markdown","source":"# DICOM Introduction\n\n\n\nThe DICOM image format is a beautiful thing once you get to know it. As a data scientist or imaging scientist, it can combine information from two worlds very nicely. This format gives context to images in a way that is crucial for physicians to know where within a scan they are looking as well as data for an engineer to know where in a scan they are predicting."},{"metadata":{"trusted":true},"cell_type":"code","source":"random_ix = random.randint(0, len(train_img_paths))\nrandom_path = train_img_root + train_img_paths[random_ix]\n\ndcm_info = pydicom.dcmread(random_path)\nprint(\"===DICOM MEDICAL INFO===\")\nprint(dcm_info)\n\npixel_data = dcm_info.pixel_array\nprint(\"\\n===IMAGE PIXEL INFO===\")\nprint(\"Image dimensions: {}\".format(pixel_data.shape))\nprint(np.max(pixel_data))\nprint(np.min(pixel_data))\nprint(np.median(pixel_data))\n\nplt.figure(figsize=(10,10))\nsns.distplot(pixel_data.flatten())\nplt.title(\"Pixel Brightness Distribution for DICOM Image\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Information DICOM Tells Us\n\nOne assumption we might have made without knowing how DICOM works is that each image ID in the .dcm filename tells us which patient the image belongs to. However, when we inspect the DICOM file, we get a unique Patient ID field that is different than the image ID. We can use this information to build out a full profile for a given patient. This can set up data structures that contain entire scans for a patient and allow for things like 3D reconstruction of a scan, more accurate and localized hemorrhage diagnoses, and generally a more contextualized image. "},{"metadata":{},"cell_type":"markdown","source":"# Viewing Some DICOM Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to show a random image containing a specific type of hemorrhage\n# We can set the threshold to be lower or higher as well. \ndef show_random_sample(hem_choice, thresh):\n \n types = new_labels_df.columns[1:]\n chosen_type = types[hem_choice]\n \n print(\"Displaying image with >= %.2f%% chance of containing an _%s_ hemmorhage...\" % (thresh*100, chosen_type))\n\n filtered_df = new_labels_df[new_labels_df[chosen_type] > thresh]\n \n random_ix = random.randint(0, filtered_df.shape[0])\n \n target_record = filtered_df.iloc[random_ix, :]\n target_id = target_record[0]\n image_path = train_img_root + \"ID_\" + target_id + \".dcm\"\n \n print(\"Opening {}...\".format(image_path))\n \n print(target_record)\n \n dcm_img = pydicom.dcmread(image_path)\n plt.imshow(dcm_img.pixel_array)\n \n plt.grid(\"off\")\n plt.axis(\"off\")\n plt.title(\"Image of Patient with {} Hemorrhage\".format(hem_types[hem_choice].title()))\n \n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see a couple of different types of hemorrhages with high probabilities."},{"metadata":{"trusted":true},"cell_type":"code","source":"for type in range(6):\n show_random_sample(type, 0.8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_by_id(patient_id):\n\n image_path = train_img_root + \"ID_\" + patient_id + \".dcm\"\n \n print(\"Opening {}...\".format(image_path))\n \n dcm_img = pydicom.dcmread(image_path)\n plt.imshow(dcm_img.pixel_array)\n \n plt.grid(\"off\")\n plt.axis(\"off\")\n plt.title(\"Image of Patient {}\".format(patient_id))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display_by_id(\"4e16848f1\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nThat's it for now! My next notebook is going to explore splitting out DICOM images by patient and evaluating hemorrhages on a per-patient basis. I hope this was a useful introduction to this dataset as well as to the DICOM image format!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}