{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Undertand the goal of this competition: \n\nFrom a set of medical images we need to give a diagnosis of hymorrage. To do that for each image we need to give a probability of the 5 subtypes of hymorrage if there is any.\nSo the target column is Label whitch will contain the probability.\n\nThe 5 types of hymorrages:\n* 1_epidural\n* intraparenchymal\n* intraventricular\n* subarachnoid\n* subdural\n* any\n\nIn this stage 1 we have a crossover of patient IDs between test images and train images. We'll explore this specifity below.\nOne of the weird rule of this competition is not use metadata of scan images to developp our model. So we will stick to the pixel array as much as possible but what the metadata hiding? :)\n\nI hope this kernel will be helpful. Please give it an upvote if you like it"},{"metadata":{},"cell_type":"markdown","source":"\n# Import libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom os import listdir\nfrom os.path import isfile, join\nimport matplotlib.pylab as plt\n%matplotlib inline\nimport os\nimport seaborn as sns\nfrom keras.models import Sequential\nfrom keras.layers import Convolution2D\nfrom keras.layers import MaxPooling2D\nfrom keras.layers import Flatten\nfrom keras.layers import Dense\nimport pydicom\nfrom glob import glob\nfrom mpl_toolkits.mplot3d.art3d import Poly3DCollection\nimport scipy.ndimage\nfrom skimage import morphology\nfrom skimage import measure\nfrom skimage.transform import resize\nfrom sklearn.cluster import KMeans\nfrom plotly import __version__\nfrom plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot\nfrom plotly.tools import FigureFactory as FF\nfrom plotly.graph_objs import *\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load Data\n\nWe have 4 files. \n1. stage_1_train.csv : Csv with the columns ID and target\n2. stage_1_train_images: contains images of the training set (dcm files) with the Id\n3. stage_1_test_images: contains images of the test set (dcm files) with the Id\n4. stage_1_sample_submission.csv: a sample submission file in the correct format. Contains Ids for the test set.\n\nFor the DICOM images files, we're going to use the pydicom library. "},{"metadata":{},"cell_type":"markdown","source":"### Load the csv files"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/rsna-intracranial-hemorrhage-detection/stage_1_train.csv')\nsample_submission = pd.read_csv('../input/rsna-intracranial-hemorrhage-detection/stage_1_sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's print the rows for the 2 first images"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train.Label)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, we have 674262 images( x6 = 445572)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['filename'] = train['ID'].apply(lambda st: \"ID_\" + st.split('_')[1] + \".dcm\")\ntrain['Subtype'] = train['ID'].apply(lambda st: st.split('_')[2])\ntrain['image_id'] = train['ID'].apply(lambda st:\"ID_\" + st.split('_')[1])\ntrain.head(12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(train.image_id.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('Subtype').sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Clearly the epidural cases are not sufficient"},{"metadata":{},"cell_type":"markdown","source":"I find it a little confusing to have all the images data in rows. We should have for each image 6 values of labels by hymorrages subtype. So let's pivot our dataframe."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train[['Label', 'image_id', 'Subtype']].drop_duplicates().pivot(index='image_id', columns='Subtype', values='Label').reset_index()\ntrain_df.head(12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.shape","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(10, 8))\nncount = len(train_df['any'])\nax = sns.countplot(train_df['any'])\nplt.title(\"Positive vs Negative results\")\nplt.xlabel(\"Number of Hymorrages\")\nax2=ax.twinx()\n#ax2.set_yticks(train_df['any'].count()*len(train_df)/100)\n# Switch so count axis is on right, frequency on left\nax2.yaxis.tick_left()\nax.yaxis.tick_right()\n# Also switch the labels over\nax.yaxis.set_label_position('right')\nax2.yaxis.set_label_position('left')\nax2.set_ylabel('Results [%]')\n\nfor p in ax.patches:\n    x=p.get_bbox().get_points()[:,0]\n    y=p.get_bbox().get_points()[1,1]\n    ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y), \n            ha='center', va='bottom') # set the alignment of the text\n    \nax2.set_ylim(0,100)\nax.set_ylim(0,ncount)\n\nax2.grid(None)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(10, 8))\nncount = len(train_df['epidural'])\nax = sns.countplot(train_df['epidural'])\nplt.title(\"Positive vs Negative epidural results\")\nplt.xlabel(\"Number of epidural Hymorrages\")\nax2=ax.twinx()\n#ax2.set_yticks(train_df['any'].count()*len(train_df)/100)\n# Switch so count axis is on right, frequency on left\nax2.yaxis.tick_left()\nax.yaxis.tick_right()\n# Also switch the labels over\nax.yaxis.set_label_position('right')\nax2.yaxis.set_label_position('left')\nax2.set_ylabel('Results [%]')\n\nfor p in ax.patches:\n    x=p.get_bbox().get_points()[:,0]\n    y=p.get_bbox().get_points()[1,1]\n    ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y), \n            ha='center', va='bottom') # set the alignment of the text\n    \nax2.set_ylim(0,100)\nax.set_ylim(0,ncount)\n\nax2.grid(None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(train_df[train_df['epidural']==1]))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we have only 2761 cases of epidural hymorrage on 674258 images. So we're gonna overfit for sure."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(10, 8))\nPositive_df = train[train['Label']==1]\nncount = len(Positive_df['image_id'])\n\nax = sns.countplot(x=\"Subtype\", hue=\"Label\", data=Positive_df.loc[Positive_df['Subtype']!=\"any\"])\n\nplt.title(\"Hymorrages by type\")\n\nax2=ax.twinx()\n#ax2.set_yticks(train_df['any'].count()*len(train_df)/100)\n# Switch so count axis is on right, frequency on left\nax2.yaxis.tick_left()\nax.yaxis.tick_right()\n# Also switch the labels over\nax.yaxis.set_label_position('right')\nax2.yaxis.set_label_position('left')\nax2.set_ylabel('Results [%]')\n\n\nfor p in ax.patches:\n    x=p.get_bbox().get_points()[:,0]\n    y=p.get_bbox().get_points()[1,1]\n    ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y), \n            ha='center', va='bottom') # set the alignment of the text\n    \nax2.set_ylim(0,100)\nax.set_ylim(0,ncount)\n\nax2.grid(None)\nax.legend_.remove()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load the images files using Pydicom\nPydicom library enables us to read a dicom image on python. So we have to store the directory paths on a list and so we can read them one by one using pydicom.read_file(imagename). I don't know for now if we can avoid the for loop. Let's see the first images"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pydicom\nfrom pydicom.data import get_testdata_files\n\n\ntrain_images_dir = '../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/'\ntest_images_dir = '../input/rsna-intracranial-hemorrhage-detection/stage_1_test_images/'\n\ntrain_images_id = [f.split(\"/\")[-1] for f in listdir(train_images_dir)]\ntest_images_id = [f.split(\"/\")[-1] for f in listdir(test_images_dir)]\n\nprint(train_images_id[0:5])\nprint(test_images_id[0:5])\n\ntrain_images = glob('../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/*.dcm')\nprint(train_images[0])\ntest_images = glob('../input/rsna-intracranial-hemorrhage-detection/stage_1_test_images/*.dcm')\nprint(test_images[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = \"../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/\"\n\ng = glob(data_path + '/*.dcm')\n\n# Print out the first 5 file names to verify we're in the right folder.\nprint (\"Total of %d DICOM images.\\nFirst 5 filenames:\" % len(g))\nprint ('\\n'.join(g[:5]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Print the metadata of the first image\n\nds = pydicom.filereader.dcmread(train_images[12])\nprint(ds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ds.pixel_array.shape\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plot the image using matplotlib\nimg = pydicom.read_file(train_images[0]).pixel_array\nplt.imshow(img,cmap = plt.cm.bone) # colormap here is the colors of medical cm.bone\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig=plt.figure(figsize=(20, 20))\ncolumns = 10; rows = 10\nfor i in range(1, columns*rows +1):\n    ds = pydicom.dcmread(train_images[i])\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(ds.pixel_array, cmap=plt.cm.bone)\n    fig.add_subplot","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"    So here the size of heads is really confusing. Maybe there are some scans for kids or even babies. Maybe there are juste some scans that are empty. We can see only the section of the table for example image 15"},{"metadata":{"trusted":true},"cell_type":"code","source":"img = pydicom.read_file(train_images[15]).pixel_array\nplt.imshow(img,cmap = plt.cm.bone) # colormap here is the colors of medical cm.bone\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's visualize scans with hymorrage"},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train['Label']==1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_hymo_path = '../input/rsna-intracranial-hemorrhage-detection/stage_1_train_images/ID_5c8b5d701.dcm'\nimg = pydicom.read_file(img_hymo_path).pixel_array\nplt.imshow(img,cmap = plt.cm.bone) # colormap here is the colors of medical cm.bone\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I took the code from https://www.kaggle.com/allunia/rsna-ih-detection-eda-baseline to put the window and level manually"},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_manual_window(hu_image, custom_center, custom_width):\n    min_value = custom_center - (custom_width/2)\n    max_value = custom_center + (custom_width/2)\n    hu_image[hu_image < min_value] = min_value\n    hu_image[hu_image > max_value] = max_value\n    return hu_image\ndef rescale_pixelarray(dataset):\n    image = dataset.pixel_array\n    rescaled_image = image * dataset.RescaleSlope + dataset.RescaleIntercept\n    rescaled_image[rescaled_image < -1024] = -1024\n    return rescaled_image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = pydicom.filereader.dcmread(img_hymo_path)\npixelarray = ds.pixel_array\nplt.imshow(pixelarray, cmap=plt.cm.bone)\nplt.grid(False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rescaled_image = rescale_pixelarray(ds)\nplt.imshow(rescaled_image, cmap=plt.cm.bone)\nplt.grid(False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"org_windowed_image = set_manual_window(rescaled_image, 30, 80)\nplt.imshow(org_windowed_image, cmap=plt.cm.bone)\nplt.grid(False)\n\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The images in the train set and the test set are so bad we can't see anything and sometimes I'm not even able to see that its a section of a human brain. Why is that? Maybe we can improve the brightness but let's explore the data first"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"dataset = pydicom.dcmread(img_hymo_path)\n#image = get_LUT_value(dataset.pixel_array, dataset.WindowWidth,\n                             # dataset.WindowCenter)\narray1 = dataset.pixel_array.copy()\narray1[array1 < 30] = 0\narray1[array1 > 70] = 0\nplt.imshow(array1, cmap=plt.cm.bone)\nplt.show()\nplt.imshow(dataset.pixel_array, cmap=plt.cm.bone)\n\nfarray = array1.flatten()\nplt.figure(figsize=(10, 10))\nplt.hist(farray,bins=100)\nplt.xlim((0,500))\nplt.ylim((0,2000))\nplt.show()\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll be back with my analyse of dicom images :) "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}