{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Table of contents\n\n* [Loading packages](#packages)\n* [Exploring the meta data](#eda)\n    * [Missing values](#missing_vals)\n    * [Understanding patients](#patients)\n    * [Image features](#image_features)\n    * [Inspecting target features](#target_features)\n    * [Machine Ids](#machine_ids)\n* [Exploring the images](#images)\n    * [Inspecting dicom files](#dicom)\n    * [Pixelarray distributions](#raw_values)\n    * [Background values and image sizes](#mid_dependence)\n    * [Exploring dicom images of machine 49](#mid_49)\n    * [Exploring images showing cancer](#cancer)\n* [Conclusion](#conclusion)","metadata":{}},{"cell_type":"code","source":"!pip install -U pylibjpeg pylibjpeg-openjpeg pylibjpeg-libjpeg pydicom python-gdcm","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:00.096043Z","iopub.execute_input":"2022-12-29T17:41:00.097015Z","iopub.status.idle":"2022-12-29T17:41:20.678124Z","shell.execute_reply.started":"2022-12-29T17:41:00.096881Z","shell.execute_reply":"2022-12-29T17:41:20.676931Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading packages <a class=\"anchor\" id=\"packages\"></a>","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport pydicom\nfrom os import listdir\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_style('darkgrid')\nsns.set_color_codes('bright')\nimport plotly.express as px\n\nfrom PIL import Image\n\nfrom scipy.stats import mode, skew\n\nfrom sklearn.mixture import GaussianMixture\nfrom sklearn.preprocessing import StandardScaler\n\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:20.680426Z","iopub.execute_input":"2022-12-29T17:41:20.680785Z","iopub.status.idle":"2022-12-29T17:41:23.071687Z","shell.execute_reply.started":"2022-12-29T17:41:20.680749Z","shell.execute_reply":"2022-12-29T17:41:23.070834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Exploring at the meta data <a class=\"anchor\" id=\"eda\"></a>","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/train.csv')\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:23.073296Z","iopub.execute_input":"2022-12-29T17:41:23.073717Z","iopub.status.idle":"2022-12-29T17:41:23.215702Z","shell.execute_reply.started":"2022-12-29T17:41:23.073676Z","shell.execute_reply":"2022-12-29T17:41:23.214519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:23.218005Z","iopub.execute_input":"2022-12-29T17:41:23.218355Z","iopub.status.idle":"2022-12-29T17:41:23.225598Z","shell.execute_reply.started":"2022-12-29T17:41:23.218325Z","shell.execute_reply":"2022-12-29T17:41:23.224488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, 54706 rows and 14 columns. Do we have missing values?","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv('/kaggle/input/rsna-breast-cancer-detection/test.csv')\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:23.226887Z","iopub.execute_input":"2022-12-29T17:41:23.227301Z","iopub.status.idle":"2022-12-29T17:41:23.253485Z","shell.execute_reply.started":"2022-12-29T17:41:23.227267Z","shell.execute_reply":"2022-12-29T17:41:23.252354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:23.255036Z","iopub.execute_input":"2022-12-29T17:41:23.255891Z","iopub.status.idle":"2022-12-29T17:41:23.262858Z","shell.execute_reply.started":"2022-12-29T17:41:23.255845Z","shell.execute_reply":"2022-12-29T17:41:23.261591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aha! There are a lot of columns in train that are not present in test. Some of them are related to the target feature **cancer** and it's obvious that they are not given (biopsy, invasive, BIRADS and difficult_negative_case). But what makes me wonder is that the density feature is also not given in test... ","metadata":{}},{"cell_type":"markdown","source":"## Missing values <a class=\"anchor\" id=\"missing_vals\"></a>","metadata":{}},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:23.264197Z","iopub.execute_input":"2022-12-29T17:41:23.264621Z","iopub.status.idle":"2022-12-29T17:41:23.301126Z","shell.execute_reply.started":"2022-12-29T17:41:23.26459Z","shell.execute_reply":"2022-12-29T17:41:23.30004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, we do have missing values for the age, BIRADS and density. ","metadata":{}},{"cell_type":"markdown","source":"## Understanding patients <a class=\"anchor\" id=\"patients\"></a>","metadata":{}},{"cell_type":"markdown","source":"How many patients do we have?","metadata":{}},{"cell_type":"code","source":"train.patient_id.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:23.302773Z","iopub.execute_input":"2022-12-29T17:41:23.303212Z","iopub.status.idle":"2022-12-29T17:41:23.313069Z","shell.execute_reply.started":"2022-12-29T17:41:23.303104Z","shell.execute_reply":"2022-12-29T17:41:23.311911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"How old are the patients?","metadata":{}},{"cell_type":"code","source":"train.groupby('patient_id').age.nunique().unique()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:23.314795Z","iopub.execute_input":"2022-12-29T17:41:23.315281Z","iopub.status.idle":"2022-12-29T17:41:23.334278Z","shell.execute_reply.started":"2022-12-29T17:41:23.315247Z","shell.execute_reply":"2022-12-29T17:41:23.333292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, that's good! We only have one age value per patient or none at all. ","metadata":{}},{"cell_type":"code","source":"ages = train[train.age.isnull() == False].groupby(\n    'patient_id').age.apply(lambda l: np.unique(l)[0])\n\nfig, ax = plt.subplots(1,2,figsize=(20,5))\n\nsns.histplot(ages, color='orange', bins=60, ax=ax[0])\nax[0].set_title('Age distribution of patients');\nsns.violinplot(train.cancer, train.age, ax=ax[1], palette='Set2');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-29T18:50:22.800288Z","iopub.execute_input":"2022-12-29T18:50:22.800674Z","iopub.status.idle":"2022-12-29T18:50:23.976385Z","shell.execute_reply.started":"2022-12-29T18:50:22.800644Z","shell.execute_reply":"2022-12-29T18:50:23.975135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* Most of the patients are older than 40 years. \n* It seems that we have two peaks around the age of 50 and close to 70. \n* There is a drop of patients counts after the age of 70. \n* For patients with cancer it's more likely that they are older and above the age of 50. ","metadata":{}},{"cell_type":"markdown","source":"How many patients do have cancer and how many images were taken per patient?","metadata":{}},{"cell_type":"code","source":"def has_cancer(l):\n    if len(l) == 1:\n        if l[0] == 0:\n            return False\n        elif l[0] == 1:\n            return True\n        else:\n            raise Exception\n    elif len(l) == 2:\n        return True\n    else:\n        raise Exception\n\npatient_cancer_map = train.groupby('patient_id').cancer.unique().apply(lambda l: has_cancer(l))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-29T17:41:24.234055Z","iopub.execute_input":"2022-12-29T17:41:24.234421Z","iopub.status.idle":"2022-12-29T17:41:24.841838Z","shell.execute_reply.started":"2022-12-29T17:41:24.234391Z","shell.execute_reply":"2022-12-29T17:41:24.840534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\n\nsns.countplot(patient_cancer_map, palette='Paired', ax=ax[0]);\nax[0].set_title('Number of patients with cancer');\n\nsns.countplot(train.groupby('patient_id').size(), ax=ax[1])\nax[1].set_title('Number of images per patient')\nax[1].set_xlabel('Number of images')\nax[1].set_ylabel('Counts of patients');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-29T17:41:24.843895Z","iopub.execute_input":"2022-12-29T17:41:24.844862Z","iopub.status.idle":"2022-12-29T17:41:25.297289Z","shell.execute_reply.started":"2022-12-29T17:41:24.844812Z","shell.execute_reply":"2022-12-29T17:41:25.29603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* It's an imbalanced classification problem! \n* In most cases we have given 4 images per patient but there are a few cases with more than 10 images as well! Why?","metadata":{}},{"cell_type":"markdown","source":"## Image features <a class=\"anchor\" id=\"image_features\"></a>","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,10))\nsns.countplot(train.laterality, ax=ax[0,0], palette='Greens_r')\nsns.countplot(train.view, ax=ax[0,1], palette='Reds_r')\nsns.countplot(train.implant, ax=ax[1,0], palette='Blues_r')\nsns.countplot(train.density, ax=ax[1,1], palette='Purples_r');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-29T17:41:25.298462Z","iopub.execute_input":"2022-12-29T17:41:25.298748Z","iopub.status.idle":"2022-12-29T17:41:26.025Z","shell.execute_reply.started":"2022-12-29T17:41:25.298721Z","shell.execute_reply":"2022-12-29T17:41:26.023701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* The laterality is quite balanced. \n* In the data description we can find that there are usually two views per breast. The most common views are CC and MLO and given these 2 views for the right and left breast we end up with 4 images that most of the patients show.  \n* Only a very few images show implants. \n* Most of the images show medium dense images of category B and C. Nonetheless there are also cases that are very dense (D) or less dense (A). Given the information that it could be more difficult to identify cancer in dense tissues this could be an interesting feature when thinking about validation strategies. ","metadata":{}},{"cell_type":"markdown","source":"## Inspecting target features <a class=\"anchor\" id=\"target_features\"></a>\n\nWe have already seen that the number of patients with cancer is quiet low compared to the patients without. Let's see how it looks like on the image-level.","metadata":{}},{"cell_type":"code","source":"biopsy_counts = train.groupby('cancer').biopsy.value_counts().unstack().fillna(0) \nbiopsy_perc = biopsy_counts.transpose() / biopsy_counts.sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:26.026838Z","iopub.execute_input":"2022-12-29T17:41:26.027311Z","iopub.status.idle":"2022-12-29T17:41:26.044905Z","shell.execute_reply.started":"2022-12-29T17:41:26.027269Z","shell.execute_reply":"2022-12-29T17:41:26.043788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.countplot(train.cancer, palette='Reds', ax=ax[0])\nax[0].set_title('Number of images displaying cancer');\nsns.heatmap(biopsy_perc.transpose(), ax=ax[1], annot=True, cmap='Oranges');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-29T17:41:26.046221Z","iopub.execute_input":"2022-12-29T17:41:26.04654Z","iopub.status.idle":"2022-12-29T17:41:26.469912Z","shell.execute_reply.started":"2022-12-29T17:41:26.046506Z","shell.execute_reply":"2022-12-29T17:41:26.468782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* The number of images displaying cancer is very low. It should be even of lower percentage than the number of patients with cancer. \n* Looking at the biopsy feature we can say that all patients with cancer had a biopsy. But only around 3 % of images without cancer had resulted in a follow-up biopsy. Maybe we should better have a look at this feature on the patient-level. ","metadata":{}},{"cell_type":"code","source":"train.cancer.value_counts()/train.shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:26.471314Z","iopub.execute_input":"2022-12-29T17:41:26.471638Z","iopub.status.idle":"2022-12-29T17:41:26.480772Z","shell.execute_reply.started":"2022-12-29T17:41:26.47161Z","shell.execute_reply":"2022-12-29T17:41:26.48003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patient_cancer_map.value_counts() / patient_cancer_map.shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:26.481914Z","iopub.execute_input":"2022-12-29T17:41:26.482881Z","iopub.status.idle":"2022-12-29T17:41:26.496514Z","shell.execute_reply.started":"2022-12-29T17:41:26.482846Z","shell.execute_reply":"2022-12-29T17:41:26.495433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As expected, the percentage of patients with cancer is higher than the percentage of images displaying cancer. How many images show invasive cancer that has spread beyond the layer of tissue in which it developed?","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(2,2,figsize=(20,10))\nsns.countplot(train[train.cancer==1].invasive, ax=ax[0,0], palette='Reds')\nsns.countplot(train.BIRADS, order=[0., 1., 2.], ax=ax[0,1], palette='Blues')\nax[0,0].set_title('Images showing invasive cancer');\n\nsns.countplot(train[train.cancer==0].BIRADS, order=[0., 1., 2.], ax=ax[1,0], palette='Purples')\nsns.countplot(train[train.cancer==1].BIRADS, order=[0., 1., 2.], ax=ax[1,1], palette='Purples')","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:26.498131Z","iopub.execute_input":"2022-12-29T17:41:26.498484Z","iopub.status.idle":"2022-12-29T17:41:27.153275Z","shell.execute_reply.started":"2022-12-29T17:41:26.498455Z","shell.execute_reply":"2022-12-29T17:41:27.152164Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[train.cancer==1].invasive.value_counts() / train[train.cancer==1].shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:27.154776Z","iopub.execute_input":"2022-12-29T17:41:27.15521Z","iopub.status.idle":"2022-12-29T17:41:27.169618Z","shell.execute_reply.started":"2022-12-29T17:41:27.155167Z","shell.execute_reply":"2022-12-29T17:41:27.168379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* Roughly 70 % of all images displaying cancer are showing invasive cancer thus the one that spreads into other tissues as well. :(\n* For BIRADS we need to remember that its value is\n    * 0 if the breast required follow-up\n    * 1 if the breast was rated as negative for cancer \n    * 2 if the breast was rated as normal\n* Given that information, we can see that most images were rated as negative which suites to the fact that most images are not positive for cancer. But nonetheless there is still a high number of images that lead into a follow-up. I'm a bit confused... what's the difference between 1 and 2? From the ordering I would suspect that 2 is better than 1. \n* Going one level deeper and showing the BIRADS counts for 'no cancer'-images and 'cancer'-images, we can at least say that all images showing cancer needed a follow up (what we expected! ;))","metadata":{}},{"cell_type":"markdown","source":"## Machine Ids <a class=\"anchor\" id=\"machine_ids\"></a>","metadata":{}},{"cell_type":"markdown","source":"How many different imaging devices were used?","metadata":{}},{"cell_type":"code","source":"train.machine_id.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:27.171046Z","iopub.execute_input":"2022-12-29T17:41:27.171428Z","iopub.status.idle":"2022-12-29T17:41:27.179022Z","shell.execute_reply.started":"2022-12-29T17:41:27.171397Z","shell.execute_reply":"2022-12-29T17:41:27.177766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,1,figsize=(20,5))\nsns.countplot(train.machine_id, palette='tab10', ax=ax);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-29T17:41:27.180441Z","iopub.execute_input":"2022-12-29T17:41:27.180759Z","iopub.status.idle":"2022-12-29T17:41:27.474966Z","shell.execute_reply.started":"2022-12-29T17:41:27.18073Z","shell.execute_reply":"2022-12-29T17:41:27.474206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* 10 different machines but most of the images were take with device of id 49, 21, 29 and 48. \n* Personally I'm not sure whether this feature has some importance for us. To be honest, I think that the patient_id, the age and the density will be much more important than the machine. ","metadata":{}},{"cell_type":"markdown","source":"# Exploring the images <a class=\"anchor\" id=\"images\"></a>\n\nBrowsing through the train and test folder of the images, we can see that the data is given as dicom files. Long time ago I wrote a tutorial notebook on dicom files while I was learning more about it myself. Please take a look at it if dicom is unknown to you. I hope that it will help to get started with the data structure. ;) \n\nhttps://www.kaggle.com/code/allunia/pulmonary-dicom-preprocessing\n","metadata":{}},{"cell_type":"code","source":"train_path = '/kaggle/input/rsna-breast-cancer-detection/train_images/'\ntest_path = '/kaggle/input/rsna-breast-cancer-detection/test_images/'","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:27.475992Z","iopub.execute_input":"2022-12-29T17:41:27.477Z","iopub.status.idle":"2022-12-29T17:41:27.481493Z","shell.execute_reply.started":"2022-12-29T17:41:27.476967Z","shell.execute_reply":"2022-12-29T17:41:27.480428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First of all, we need a method to get all scans of one patient:","metadata":{}},{"cell_type":"code","source":"def load_scans(path, patient_id):\n    dcm_path = path + str(patient_id)\n    slices = [pydicom.dcmread(dcm_path + '/' + file, force=True) for file in listdir(dcm_path)]\n    return slices","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:27.482986Z","iopub.execute_input":"2022-12-29T17:41:27.483566Z","iopub.status.idle":"2022-12-29T17:41:27.492509Z","shell.execute_reply.started":"2022-12-29T17:41:27.483534Z","shell.execute_reply":"2022-12-29T17:41:27.49149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pixelarray distributions <a class=\"anchor\" id=\"raw values\"></a>","metadata":{}},{"cell_type":"markdown","source":"* Usually we need to transform the raw pixelarray distributions to Hounsfield Units. The raw values depend on the measurement settings like acquisition parameters and tube voltage of the scanner. \n* By normalizing to values of water and air (water has HU 0 and air -1000) the images of different measurements are becoming comparable. \n* We also need to understand how background values were treated! In previous competitions it was often the case that background values were set to values smaller than -1000, but we need to check whether this was done in this competition as well!","metadata":{}},{"cell_type":"markdown","source":"Let's load an example:","metadata":{}},{"cell_type":"code","source":"patient_ids = train.patient_id.unique()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:27.493858Z","iopub.execute_input":"2022-12-29T17:41:27.494759Z","iopub.status.idle":"2022-12-29T17:41:27.50429Z","shell.execute_reply.started":"2022-12-29T17:41:27.494707Z","shell.execute_reply":"2022-12-29T17:41:27.503052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scans = load_scans(train_path, patient_ids[0])","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:27.505592Z","iopub.execute_input":"2022-12-29T17:41:27.506566Z","iopub.status.idle":"2022-12-29T17:41:27.806899Z","shell.execute_reply.started":"2022-12-29T17:41:27.506517Z","shell.execute_reply":"2022-12-29T17:41:27.805804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And let's plot the raw pixelarray distribution for this example and display the related image:","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.histplot(scans[0].pixel_array.flatten(), ax=ax[0])\nax[1].imshow(scans[0].pixel_array)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:27.808276Z","iopub.execute_input":"2022-12-29T17:41:27.808793Z","iopub.status.idle":"2022-12-29T17:41:50.171283Z","shell.execute_reply.started":"2022-12-29T17:41:27.80876Z","shell.execute_reply":"2022-12-29T17:41:50.170206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* Uhh! That's interesting! The background seems to be above 3000 this time! So it's competely different to previous competitions.\n* As the raw values depend on the machine, it could be worth it to explore background values dependent on the machine id. ","metadata":{}},{"cell_type":"markdown","source":"Let's get an impression by extracting the mode per scan for a few patients (let's say 50) for each machine id.","metadata":{}},{"cell_type":"code","source":"machine_ids = train.machine_id.unique()\nmachine_ids","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:50.172853Z","iopub.execute_input":"2022-12-29T17:41:50.173334Z","iopub.status.idle":"2022-12-29T17:41:50.181716Z","shell.execute_reply.started":"2022-12-29T17:41:50.173291Z","shell.execute_reply":"2022-12-29T17:41:50.180551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scans[0].Columns","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:50.191888Z","iopub.execute_input":"2022-12-29T17:41:50.19275Z","iopub.status.idle":"2022-12-29T17:41:50.293661Z","shell.execute_reply.started":"2022-12-29T17:41:50.192705Z","shell.execute_reply":"2022-12-29T17:41:50.292316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Background values and image sizes <a class=\"anchor\" id=\"mid_dependence\"></a>","metadata":{}},{"cell_type":"code","source":"background_values = []\nrows = []\ncolumns = []\nfor mid in train.machine_id.unique():\n    mid_background_values = []\n    mid_rows = []\n    mid_columns = []\n    print(f'machine id {mid} in progress')\n    for n in range(50):\n        try:\n            scans = load_scans(train_path, train[train.machine_id==mid].patient_id.unique()[n])\n            mid_background_values.append(mode(scans[0].pixel_array.flatten())[0][0])\n            mid_rows.append(scans[0].Rows)\n            mid_columns.append(scans[0].Columns)\n        except IndexError:\n            break\n    background_values.append(mid_background_values)\n    rows.append(mid_rows)\n    columns.append(mid_columns)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:41:50.295365Z","iopub.execute_input":"2022-12-29T17:41:50.295788Z","iopub.status.idle":"2022-12-29T17:51:35.922091Z","shell.execute_reply.started":"2022-12-29T17:41:50.295731Z","shell.execute_reply":"2022-12-29T17:51:35.920548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for n in range(len(background_values)):\n    print(machine_ids[n], np.median(background_values[n]), np.std(background_values[n]))","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:35.924423Z","iopub.execute_input":"2022-12-29T17:51:35.92481Z","iopub.status.idle":"2022-12-29T17:51:35.937234Z","shell.execute_reply.started":"2022-12-29T17:51:35.924771Z","shell.execute_reply":"2022-12-29T17:51:35.93609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* It depends on the machine id, how background values are treated. In most of the cases they seem to be 0. But for id 29 it's above 3000 and for id 210 it seems to be 1017. \n* Id 197 looks interesting. Currently I don't know, what it means. Let's take a look at the values","metadata":{}},{"cell_type":"code","source":"background_values[-1]","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:35.938779Z","iopub.execute_input":"2022-12-29T17:51:35.939969Z","iopub.status.idle":"2022-12-29T17:51:35.948722Z","shell.execute_reply.started":"2022-12-29T17:51:35.939922Z","shell.execute_reply":"2022-12-29T17:51:35.947735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ah, ok, only 4 patients and it could be that the background values for patient 2 and 4 are not the same as the mode.","metadata":{}},{"cell_type":"markdown","source":"What about the image sizes? Are they also dependent on the machine id? Let's have a look at the columns and the rows:","metadata":{}},{"cell_type":"code","source":"for l in range(len(columns)):\n    print(f\"machine id {train.machine_id.unique()[l]}, unique columns: {np.unique(columns[l])}\")","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:35.950728Z","iopub.execute_input":"2022-12-29T17:51:35.951171Z","iopub.status.idle":"2022-12-29T17:51:35.964759Z","shell.execute_reply.started":"2022-12-29T17:51:35.95111Z","shell.execute_reply":"2022-12-29T17:51:35.963498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for l in range(len(rows)):\n    print(f\"machine id {train.machine_id.unique()[l]}, unique rows: {np.unique(rows[l])}\")","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:35.966311Z","iopub.execute_input":"2022-12-29T17:51:35.967458Z","iopub.status.idle":"2022-12-29T17:51:35.980446Z","shell.execute_reply.started":"2022-12-29T17:51:35.967417Z","shell.execute_reply":"2022-12-29T17:51:35.979163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"IHH! :-o The image size depends on the machine id as well! Ohoh! What else will depend on it? ... Time to take a look at a single dicom file to search for further features:","metadata":{}},{"cell_type":"code","source":"mid = train.machine_id.value_counts().index.values[8]\nmid","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:35.981891Z","iopub.execute_input":"2022-12-29T17:51:35.982239Z","iopub.status.idle":"2022-12-29T17:51:35.990907Z","shell.execute_reply.started":"2022-12-29T17:51:35.982208Z","shell.execute_reply":"2022-12-29T17:51:35.990033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mid_patients = train[train.machine_id == mid].patient_id.unique()\nscans = load_scans(train_path, mid_patients[0])\nscans[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:35.991945Z","iopub.execute_input":"2022-12-29T17:51:35.992526Z","iopub.status.idle":"2022-12-29T17:51:36.022524Z","shell.execute_reply.started":"2022-12-29T17:51:35.992493Z","shell.execute_reply":"2022-12-29T17:51:36.021523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What else could be interesting or worth it to explore? After browsing through some examples per mid, I found a few more interesting features that might be relevant or interesting to understand:\n\n\n* Body Part Thickness (what's that?) - not always present\n* Compression Force (what's that?) - not always present\n* Pixel Spacing - not always present\n* Pixel Padding Value (often 0, but not always & sometimes not given)\n* Photometric Interpretation: Changes sometimes\n* Pixel Intensity Relationship Sign (seems to be 1 or -1)\n* window width and window center (do we need that?)","metadata":{}},{"cell_type":"markdown","source":"### Insights\n\n* The sizes of the dicom images as well as the background values depend on the machine id.\n* Besides that there are further features that may also have this dependency but we don't know it yet. We have to investigate this even further.\n* We also found a few machine ids that have some variability inside their group (like images with two different sizes or a few possible background values).\n\nOne idea could be to do preprocessing of dicom files dependent on the machine id or another idea could be to explore this structure even further and to uncover all different patterns we can find in train... hmm :-)","metadata":{}},{"cell_type":"markdown","source":"## Exploring dicom images of machine 49 <a class=\"anchor\" id=\"mid_49\"></a>\n\nLet's keep it simple for now by making the assumption that all files with the same machine id share similar or same features. Then we could try to explore more (probably visual) patterns within one machine id group. The largest one belongs to id 49. ","metadata":{}},{"cell_type":"code","source":"train[train.machine_id==49].shape","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:36.023859Z","iopub.execute_input":"2022-12-29T17:51:36.024287Z","iopub.status.idle":"2022-12-29T17:51:36.035576Z","shell.execute_reply.started":"2022-12-29T17:51:36.024256Z","shell.execute_reply":"2022-12-29T17:51:36.034455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[train.machine_id==49].patient_id.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:36.037241Z","iopub.execute_input":"2022-12-29T17:51:36.037524Z","iopub.status.idle":"2022-12-29T17:51:36.047895Z","shell.execute_reply.started":"2022-12-29T17:51:36.037498Z","shell.execute_reply":"2022-12-29T17:51:36.047164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"4589 patients in that group! Maybe it's sufficient to explore the images given a resized and transformed dataset like that one of Theo Viel with png-images of size 256x256:\n\n* https://www.kaggle.com/datasets/theoviel/rsna-breast-cancer-256-pngs","metadata":{}},{"cell_type":"code","source":"train['png_256_path'] = \"/kaggle/input/rsna-breast-cancer-256-pngs/\" + train.patient_id.astype(str) + \"_\" + train.image_id.astype(str) + \".png\"","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:51:36.049423Z","iopub.execute_input":"2022-12-29T17:51:36.050446Z","iopub.status.idle":"2022-12-29T17:51:36.176761Z","shell.execute_reply.started":"2022-12-29T17:51:36.050401Z","shell.execute_reply":"2022-12-29T17:51:36.175469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"means = []\nmedians = []\nstds = []\nskews = []\npaths = []\ntargets = []\n\npatients_49 = train[train.machine_id==49].patient_id.unique()\n\nfor n in range(len(patients_49)):\n    pid = patients_49[n]\n    path = train[(train.machine_id==49) & (train.patient_id==pid)].png_256_path.values[0]\n    targets = train[(train.machine_id==49) & (train.patient_id==pid)].cancer.values[0]\n    paths.append(path)\n    image = Image.open(path)\n    means.append(np.mean(image))\n    medians.append(np.median(image))\n    stds.append(np.std(image))\n    skews.append(skew(image, axis=None))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-29T17:51:36.17841Z","iopub.execute_input":"2022-12-29T17:51:36.179565Z","iopub.status.idle":"2022-12-29T17:52:33.28568Z","shell.execute_reply.started":"2022-12-29T17:51:36.179521Z","shell.execute_reply":"2022-12-29T17:52:33.284487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stats_49 = pd.DataFrame(means, index=patients_49, columns =[\"means\"])\nstats_49['stds'] = stds\nstats_49['medians'] = medians\nstats_49['skews'] = skews\nstats_49['paths'] = paths\nstats_49['cancer'] = targets ","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:52:33.28733Z","iopub.execute_input":"2022-12-29T17:52:33.288227Z","iopub.status.idle":"2022-12-29T17:52:33.302062Z","shell.execute_reply.started":"2022-12-29T17:52:33.288183Z","shell.execute_reply":"2022-12-29T17:52:33.300797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.scatter_3d(stats_49, x='means', y='stds', z='skews',\n              color='medians')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:52:33.303668Z","iopub.execute_input":"2022-12-29T17:52:33.304103Z","iopub.status.idle":"2022-12-29T17:52:34.613253Z","shell.execute_reply.started":"2022-12-29T17:52:33.30406Z","shell.execute_reply":"2022-12-29T17:52:34.612445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* Given the image statistics I would say that there are at least four groups. \n* One group has large median values, small stds, high mean values and low skewness. \n* Second groupd is very dense with low means and stds but low to high skewnesses. \n* The thrid group shows high mean values and stds, but low skewness. \n* The last group has low values for all features.\n\nLet's cluster that a bit to explore different visual patterns:","metadata":{}},{"cell_type":"code","source":"scaler = StandardScaler()\nX = stats_49.drop(['paths', 'cancer'], axis=1).values\nX = scaler.fit_transform(X)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:52:34.61451Z","iopub.execute_input":"2022-12-29T17:52:34.615394Z","iopub.status.idle":"2022-12-29T17:52:34.623101Z","shell.execute_reply.started":"2022-12-29T17:52:34.615358Z","shell.execute_reply":"2022-12-29T17:52:34.622314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gmm = GaussianMixture(n_components=4, random_state=0)\nstats_49[\"cluster_label\"] = gmm.fit_predict(X)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:59:49.036415Z","iopub.execute_input":"2022-12-29T17:59:49.036843Z","iopub.status.idle":"2022-12-29T17:59:49.337906Z","shell.execute_reply.started":"2022-12-29T17:59:49.036812Z","shell.execute_reply":"2022-12-29T17:59:49.336228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.scatter_3d(stats_49, x='means', y='stds', z='skews',\n              color='cluster_label')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-29T17:59:52.657702Z","iopub.execute_input":"2022-12-29T17:59:52.658791Z","iopub.status.idle":"2022-12-29T17:59:53.296106Z","shell.execute_reply.started":"2022-12-29T17:59:52.658749Z","shell.execute_reply":"2022-12-29T17:59:53.294937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok let's take a look at the different images per cluster label:","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(10,4,figsize=(20,40))\nfor l in range(4):\n    img_paths = stats_49[stats_49.cluster_label==l].paths.values[0:10]\n    for n in range(10):\n        try:\n            image = Image.open(img_paths[n])\n            ax[n,l].imshow(image, cmap='jet')\n            ax[n,l].set_title(f\"label {l}\")\n        except:\n            continue\n        ax[n,l].axis('off')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-29T17:59:53.659405Z","iopub.execute_input":"2022-12-29T17:59:53.65978Z","iopub.status.idle":"2022-12-29T18:00:01.658527Z","shell.execute_reply.started":"2022-12-29T17:59:53.659751Z","shell.execute_reply":"2022-12-29T18:00:01.657616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.unique(stats_49.cancer)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:00:01.660048Z","iopub.execute_input":"2022-12-29T18:00:01.660952Z","iopub.status.idle":"2022-12-29T18:00:01.667885Z","shell.execute_reply.started":"2022-12-29T18:00:01.660906Z","shell.execute_reply":"2022-12-29T18:00:01.667168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Insights\n\n* The group with high median values stands out most. We can see that there are large regions within the breast with high intensity values. And it's not cancer!\n* In contrast the other images vary mainly in the amount of background values and still look very similar. ","metadata":{}},{"cell_type":"code","source":"high_intensity_patients = stats_49[stats_49.cluster_label==2].index.values","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:00:49.365467Z","iopub.execute_input":"2022-12-29T18:00:49.365851Z","iopub.status.idle":"2022-12-29T18:00:49.372599Z","shell.execute_reply.started":"2022-12-29T18:00:49.365821Z","shell.execute_reply":"2022-12-29T18:00:49.371747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"selection = train[train.patient_id.isin(high_intensity_patients)]\n\n\nfig, ax = plt.subplots(1,2,figsize=(20,5))\nsns.countplot(selection.implant, ax=ax[0], palette='Reds')\nsns.countplot(selection.density, ax=ax[1], palette='Purples_r')\nax[0].set_title('Cluster label 2 patients \\n with implants');\nax[1].set_title('Cluster label 2 patients \\n density category');","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:05:42.921853Z","iopub.execute_input":"2022-12-29T18:05:42.922259Z","iopub.status.idle":"2022-12-29T18:05:43.299511Z","shell.execute_reply.started":"2022-12-29T18:05:42.922223Z","shell.execute_reply":"2022-12-29T18:05:43.298687Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ahh! Most of these images show implants! :-)","metadata":{}},{"cell_type":"code","source":"train[train.implant==1].difficult_negative_case.value_counts() / train[train.implant==1].shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:07:18.619221Z","iopub.execute_input":"2022-12-29T18:07:18.620013Z","iopub.status.idle":"2022-12-29T18:07:18.640084Z","shell.execute_reply.started":"2022-12-29T18:07:18.619961Z","shell.execute_reply":"2022-12-29T18:07:18.638189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[train.implant==0].difficult_negative_case.value_counts() / train[train.implant==0].shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:07:31.372329Z","iopub.execute_input":"2022-12-29T18:07:31.372767Z","iopub.status.idle":"2022-12-29T18:07:31.397068Z","shell.execute_reply.started":"2022-12-29T18:07:31.372732Z","shell.execute_reply":"2022-12-29T18:07:31.395803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hmm, I'm not sure if this is just noise. ","metadata":{}},{"cell_type":"markdown","source":"## Exploring some outliers of machine 49 <a class=\"anchor\" id=\"outliers\"></a>","metadata":{}},{"cell_type":"code","source":"stats_49['logL'] = gmm.score_samples(X)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:31:55.106483Z","iopub.execute_input":"2022-12-29T18:31:55.106904Z","iopub.status.idle":"2022-12-29T18:31:55.116208Z","shell.execute_reply.started":"2022-12-29T18:31:55.10687Z","shell.execute_reply":"2022-12-29T18:31:55.115176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"stats_49.logL.quantile(q=0.01)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:37:20.910588Z","iopub.execute_input":"2022-12-29T18:37:20.911137Z","iopub.status.idle":"2022-12-29T18:37:20.92202Z","shell.execute_reply.started":"2022-12-29T18:37:20.911076Z","shell.execute_reply":"2022-12-29T18:37:20.920616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"outliers = stats_49[stats_49.logL < -10].sort_values(by='logL')\nlen(outliers)","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:37:24.662542Z","iopub.execute_input":"2022-12-29T18:37:24.662918Z","iopub.status.idle":"2022-12-29T18:37:24.671966Z","shell.execute_reply.started":"2022-12-29T18:37:24.662888Z","shell.execute_reply":"2022-12-29T18:37:24.670842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(4,8, figsize=(20,10))\nfor n in range(4):\n    for m in range(8):\n        image = Image.open(outliers.paths.values[8*n+m])\n        ax[n,m].imshow(image, cmap='jet')\n        ax[n,m].axis('off')","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:38:35.273741Z","iopub.execute_input":"2022-12-29T18:38:35.274133Z","iopub.status.idle":"2022-12-29T18:38:38.448259Z","shell.execute_reply.started":"2022-12-29T18:38:35.274086Z","shell.execute_reply":"2022-12-29T18:38:38.447177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ah, ok! We should expect both - cases with the whole image covered with breast and images with almost no tissue at all. Furthermore we will have some cases with some kind of artifacts like the thrid one in the first row that shows different background values separated by a sharp edge. Most likely there will be other outliers as well that we will not be able to find with the simple method of using image statistics. ","metadata":{}},{"cell_type":"markdown","source":"## Exploring images with cancer <a class=\"anchor\" id=\"cancer\"></a>\n\nLast but not least I would like to explore some dicom images with cancer and to ensure that dicom files are similar I will continue with machine id 49. ","metadata":{}},{"cell_type":"code","source":"selection = train[(train.machine_id==49) & (train.cancer==1)]","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:21:04.885634Z","iopub.execute_input":"2022-12-29T18:21:04.886028Z","iopub.status.idle":"2022-12-29T18:21:04.893761Z","shell.execute_reply.started":"2022-12-29T18:21:04.885998Z","shell.execute_reply":"2022-12-29T18:21:04.892653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"N = 4\nM = 6\nfig, ax = plt.subplots(M,N,figsize=(20,20))\nfor m in range(M):\n    for n in range(N):\n        image = Image.open(selection.png_256_path.values[N*m+n])\n        ax[m,n].imshow(image, cmap='jet')\n        ax[m,n].axis(\"off\")","metadata":{"execution":{"iopub.status.busy":"2022-12-29T18:27:38.511422Z","iopub.execute_input":"2022-12-29T18:27:38.511852Z","iopub.status.idle":"2022-12-29T18:27:41.085308Z","shell.execute_reply.started":"2022-12-29T18:27:38.511817Z","shell.execute_reply":"2022-12-29T18:27:41.084384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hmm, difficult! I can't really say that I see a difference to those above.","metadata":{}},{"cell_type":"markdown","source":"# Conclusion <a class=\"anchor\" id=\"conclusion\"></a>\n\nMy simplified data exploration has come to an end. What have I found so far?\n\n* We are asked to solve an imbalanced classification problem with only ~ 2% of images showing cancer. \n* For patients with cancer it's more likely that they are already very old.\n* Most of the patients have 4 images with left and right breast in combination with the two default views CC and MLO. But there are a few exceptions from this rule with more images and more views. \n* How the raw dicom pixelarray values need to be preprocessed depends on the machine id! Background values were treated differently and it seems that are more relevant dicom features we should try to understand before generating a png/jpeg dataset. \n* Most images (~70%) with cancer are showing invasive cancer. \n* The image sizes depend on the machine id as well!\n* The clustering of image statistics has shown that implants show high intensity values and that we should expect artifacts and different levels of tissue covering the image area. \n\n\nHow would I like to continue?\n\n* First of all, I would try to find a good way to preprocess the raw dicom pixelarray values given the machine id (or better features that can be found in the dicom file as they will be availabe for test data as well). \n* Then I would think about strategies for validation and I think some of the findings above could be interesting to take into account.\n","metadata":{}}]}