{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---\n## 1 | Introduction\n---\n\n## Goal of the Competition\n\nThe goal of this competition is to identify cases of breast cancer in mammograms from screening exams. It is important to identify cases of cancer for obvious reasons, but false positives also have downsides for patients. As millions of women get mammograms each year, a useful machine learning tool could help a great many people.\n\n## Competition's Metric\n\nSubmissions are evaluated using the **probabilistic F1 score** (pF1). This extension of the traditional F score accepts probabilities instead of binary classifications. Our model should output the likelihood of cancer in the corresponding image. You can find a Python implementation [here](https://www.kaggle.com/code/sohier/probabilistic-f-score).\n\nWith $p_X$ as the probabilistic version of X:\n\n$$p_{F_1} = 2 \\cdot \\frac{p_{precision} \\cdot p_{recall}}{p_{precision} + p_{recall}}$$\n\nwhere: \n\n$$p_{precision} = \\frac{p_{TP}}{p_{TP} + p_{FP}}$$\n\n$$p_{recall} = \\frac{p_{TP}}{TP + FN}$$\n\n## Images file format\n\n> Images are given in dicom format. Here you'll find a tutorial to get started -> [Pulmonary Dicom Preprocessing](https://www.kaggle.com/code/allunia/pulmonary-dicom-preprocessing)\n\nDICOM or digital imaging and communications in medicine are image files sourced from different modalities and it is the international standard to transmit, store, retrieve, print, process, and display medical imaging information. However, DICOM groups information into the data set, and that means that the image file contains the patient information ID, date of birth, age, sex, and other information about the diagnosis all this within the image, as shown in the figure the main components of the medical image.\n\n![](https://miro.medium.com/max/1400/1*BvVR-348gg0qRmVmm8gtxw.webp)\n\n* **Pixel Depth**: is the number of bits used to encode the information of each pixel. For example, an 8-bit raster can have 256 unique values that range from 0 to 255.\n\n* **Photometric Interpretation**: specifies how the pixel data should be interpreted for the correct image display as a monochrome or color image. To specify if the color information is or is not stored in the image pixel values, we introduce the concept of samples per pixel, also known as (number of channels).\n\n* **Metadata**: is the information that describes the image (i.e. patients ID, date of the image).\n\n* **Pixel Data**: is the section where the numerical values of the pixels are stored. All the components are essential but in our scope the pixel depth and pixel data. To my knowledge that ultrasound images are not an issue with converting the image to another format, but we have to look into consideration the depth of the image since we cannot convert 16-bit DICOM image to JPEG or PNG with 8-bit that might corrupt the image quality and image features. Pixel data the data that we are going to feed it to the network.","metadata":{}},{"cell_type":"code","source":"!pip install -U pylibjpeg pylibjpeg-openjpeg pylibjpeg-libjpeg pydicom python-gdcm","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:05:40.883757Z","iopub.execute_input":"2022-12-31T21:05:40.884777Z","iopub.status.idle":"2022-12-31T21:05:58.390503Z","shell.execute_reply.started":"2022-12-31T21:05:40.884645Z","shell.execute_reply":"2022-12-31T21:05:58.389047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.display import clear_output, display_html\nimport os\nimport warnings\nfrom pathlib import Path\n\n# Basic libraries\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom tqdm import tqdm\n\n# Set Color Palettes for the notebook\n'''Inspired by: https://www.kaggle.com/code/andradaolteanu/rsna-fracture-detection-dicom-images-explore'''\ncustom_colors = ['#74a09e','#86c1b2','#98e2c6','#f3c969','#f2a553', '#d96548', '#c14953']\nprint('Custom Colors Palette: ')\nsns.palplot(sns.color_palette(custom_colors))\n\nimport scipy as sc\nfrom scipy import stats\n\n# Train Test Split\nfrom sklearn.model_selection import train_test_split\n\n# Cross Validation\nfrom sklearn.model_selection import KFold, cross_val_score, StratifiedKFold, learning_curve, train_test_split\n\n# Tensorflow\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\n# Plotly\nimport plotly.express as px\nfrom plotly.subplots import make_subplots\nimport plotly.figure_factory as ff\nimport plotly.offline as offline\nimport plotly.graph_objs as go\n\nwarnings.filterwarnings('ignore')\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:05:58.39368Z","iopub.execute_input":"2022-12-31T21:05:58.394189Z","iopub.status.idle":"2022-12-31T21:06:07.893297Z","shell.execute_reply.started":"2022-12-31T21:05:58.394137Z","shell.execute_reply":"2022-12-31T21:06:07.89155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## 2 | Exploratory Data Analysis\n---\n\nIn this subsection we'll focus on examining both the metadata and the images that we're given. Let's start by loading the metadata datasets. ","metadata":{}},{"cell_type":"code","source":"def load_data():\n    '''Load each of the datasets we are given.'''\n    \n    data_dir = Path(\"../input/rsna-breast-cancer-detection\")\n    train = pd.read_csv(data_dir / \"train.csv\")\n    test = pd.read_csv(data_dir / \"test.csv\")\n    sample_submission = pd.read_csv(data_dir / 'sample_submission.csv')\n    return train, test, sample_submission\n\nfrom termcolor import colored\ndef data_info(csv, name=\"Train\"):\n    '''Prints basic information about the datasets we are given.'''\n    '''Inspired by: https://www.kaggle.com/code/andradaolteanu/rsna-fracture-detection-dicom-images-explore'''\n    \n    print(colored('==== {} ===='.format(name), 'cyan', attrs=['bold']))\n    print(colored('Shape: ', 'cyan', attrs=['bold']), csv.shape)\n    print(colored('NaN Values: ', 'cyan', attrs=['bold']), csv.isnull().sum().sum(), '\\n')\n    #print(colored('Columns: ', 'blue', attrs=['bold']), list(csv.columns))\n    \n    display_html(csv.head())\n    if name != 'Sample Submission': print(\"\\n\")\n\ntrain, test, sample_submission = load_data()\nclear_output()\n\nnames = [\"Train\", \"Test\", \"Sample Submission\"]\nfor i, df in enumerate([train, test, sample_submission]): \n    data_info(df, names[i])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:06:07.896175Z","iopub.execute_input":"2022-12-31T21:06:07.897474Z","iopub.status.idle":"2022-12-31T21:06:08.124645Z","shell.execute_reply.started":"2022-12-31T21:06:07.897394Z","shell.execute_reply":"2022-12-31T21:06:08.123515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📌 **Early insights:** \n* In the metadata training file we have plenty of missing values. \n* Repeated values for `patient_id`. It seems that for each patient, 4 images have been taken. \n* Some features from the training set do not appear in the testing one. \n\nIn order to make a proper analysis, we're gonna load every metadata from the images into a dataframe. Some of this data may be useful afterwards for the model training and splitting strategies. Below, you have a quick example of .dcm file metadata. ","metadata":{}},{"cell_type":"code","source":"import pydicom\nfrom os import listdir\n\ndcm_path = \"/kaggle/input/rsna-breast-cancer-detection/train_images/10006/1459541791.dcm\"\nimg = pydicom.dcmread(dcm_path)\nimg","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:06:08.127298Z","iopub.execute_input":"2022-12-31T21:06:08.127697Z","iopub.status.idle":"2022-12-31T21:06:08.406855Z","shell.execute_reply.started":"2022-12-31T21:06:08.127663Z","shell.execute_reply":"2022-12-31T21:06:08.405926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The image data is stored in `Pixel Data`. Everything else is metadata.\n\n* The `Rows` and `Columns` values tell us the image size.\n* The `Pixel Spacing` and `Slice Thickness` tell us the pixel size and thickness.\n* The `Window Center` and `Window Width` give information about the brightness and contrast of the image respectively.\n* The `Rescale Intercept` and `Rescale Slope` determine the range of pixel values. (ref).\n* `ImagePositionPatient` tells us the x, y, and z coordinates of the top left corner of each image in mm\n* `InstanceNumber` is the slice number.","metadata":{}},{"cell_type":"code","source":"dcms = []\nfor root, dirs, fnames in os.walk('/kaggle/input/rsna-breast-cancer-detection/train_images/'):\n    dcms += list(os.path.join(root, f) for f in fnames if f.endswith('.dcm'))\nprint(f'There are {len(dcms)} images')\n\nattrs = set()\nfor fname in tqdm(dcms[:5000]):\n    with pydicom.dcmread(fname) as obj:\n        attrs.update(obj.dir())\n\ndcm_keys = list(attrs)\ndcm_keys.remove('PixelData') # The actual array of pixels, this is not metadata\ndcm_keys","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:06:08.408029Z","iopub.execute_input":"2022-12-31T21:06:08.409148Z","iopub.status.idle":"2022-12-31T21:11:00.03879Z","shell.execute_reply.started":"2022-12-31T21:06:08.409101Z","shell.execute_reply":"2022-12-31T21:11:00.036953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta = []\ntypemap = {\n    pydicom.uid.UID: str,\n    pydicom.multival.MultiValue: list\n}\ndef cast(x):\n    return typemap.get(type(x), lambda x: x)(x)\n\nfor i, fname in enumerate(tqdm(dcms[:5000])):\n    with pydicom.dcmread(fname) as obj:\n        meta.append([cast(obj.get(key, np.nan)) for key in dcm_keys])\n\ndfmeta = pd.DataFrame(meta, columns=dcm_keys)\ndfmeta.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:11:00.041315Z","iopub.execute_input":"2022-12-31T21:11:00.041759Z","iopub.status.idle":"2022-12-31T21:12:36.232705Z","shell.execute_reply.started":"2022-12-31T21:11:00.041708Z","shell.execute_reply":"2022-12-31T21:12:36.231427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Values for Photometric Interpretation: {}'.format(dfmeta['PhotometricInterpretation'].unique()))\nprint('Values for VOILUTFunction: {}\\n'.format(dfmeta['VOILUTFunction'].unique()))\n\nplt.figure(figsize = (22,16))\nfor i, col in enumerate(dfmeta.select_dtypes([int, float]).columns):\n    plt.subplot(4,4, i+1)\n    sns.distplot(dfmeta[col], color = custom_colors[0])","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:12:36.234312Z","iopub.execute_input":"2022-12-31T21:12:36.234714Z","iopub.status.idle":"2022-12-31T21:12:39.856891Z","shell.execute_reply.started":"2022-12-31T21:12:36.234677Z","shell.execute_reply":"2022-12-31T21:12:39.856051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📌 **Early Insights**:\n* Big images sizes. Rows' peak values are near 4k. For columns this value is 3k. Moreover, we observe that we have images with different sizes and resolutions. Afterwards, we'll determine whether padding is going to be needed. ","metadata":{}},{"cell_type":"code","source":"dfmeta[['Rows','Columns']].describe().T.style.background_gradient(cmap='GnBu_r')","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:12:39.858084Z","iopub.execute_input":"2022-12-31T21:12:39.858466Z","iopub.status.idle":"2022-12-31T21:12:39.962367Z","shell.execute_reply.started":"2022-12-31T21:12:39.858424Z","shell.execute_reply":"2022-12-31T21:12:39.961273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* `Photometric Interpretation` is set to **MONOCHROME1** and **MONOCHROME2**. We have to be careful about that as image interpretation could vary from one type to the other. The same happens to`VOILUTFunction`, different values are given. \n\n* The dataset contains **compressed Pixel Data**. By itself pydicom can only handle Pixel Data that hasn't been compressed, but if you install [one or more optional libraries](https://pydicom.github.io/pydicom/stable/tutorials/installation.html#install-the-optional-libraries) then it can handle various compressions. [This table](https://pydicom.github.io/pydicom/stable/old/image_data_handlers.html#supported-transfer-syntaxes) tells you which package is required.\n\n* `BodyPartThickness` refers to the average thickness in mm of the body part examined when compressed, if compression has been applied during exposure.","metadata":{}},{"cell_type":"code","source":"dfmeta[['CompressionForce','BodyPartThickness']].describe().T.style.background_gradient(cmap='GnBu_r')","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:12:39.964043Z","iopub.execute_input":"2022-12-31T21:12:39.964427Z","iopub.status.idle":"2022-12-31T21:12:40.002546Z","shell.execute_reply.started":"2022-12-31T21:12:39.964367Z","shell.execute_reply":"2022-12-31T21:12:40.001251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's now show some breast images. ","metadata":{}},{"cell_type":"code","source":"dcm_path = \"/kaggle/input/rsna-breast-cancer-detection/train_images/\"\n\ndef patient_images(p_id): \n    ''' Shows all the images that are associated with the patient for whom the ID is given. '''\n    \n    figure = plt.figure(figsize = (22,5))\n    for i, file in enumerate(listdir(dcm_path + str(p_id) + '/')):\n        plt.subplot(1, 4, i+1)\n        dataset = pydicom.dcmread(dcm_path + str(p_id) + '/' + file)\n        plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)\n        plt.axis('off');\n        \npatient_images(train['patient_id'].unique()[0])    ","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:12:40.003984Z","iopub.execute_input":"2022-12-31T21:12:40.004341Z","iopub.status.idle":"2022-12-31T21:12:56.56726Z","shell.execute_reply.started":"2022-12-31T21:12:40.004308Z","shell.execute_reply":"2022-12-31T21:12:56.565977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Breasts are shown in a **small portion** of the image. So it'd be nice to **crop out** those sections of the images that not contain any useful information. As you may have observed, it seems that in some images we're given a vertical line. It could be useful to consider them to do the cropping.\n\n## Site and Patient\n\nStarting with the hospital, we can observe that we only have two of them in our dataset. Apart from that, we can observe that **background colors depend on the site** where the image was taken. For site nº 2, background color is blank. However, for site nº 1 this color is almost black","metadata":{}},{"cell_type":"code","source":"dcm_path = \"/kaggle/input/rsna-breast-cancer-detection/train_images/\"\n\ndef images_site(site_id):\n    ids = train[train.site_id == site_id]['patient_id'].unique()\n    for i, id_ in enumerate(ids[[0,3]]):\n        patient_path = dcm_path + str(id_) +'/'\n        fig = plt.figure(figsize = (22,5))\n        for j, file in enumerate(listdir(patient_path)):\n            plt.subplot(1, 4, j+1)\n            dataset = pydicom.dcmread(patient_path + file)\n            p = plt.imshow(dataset.pixel_array, cmap=plt.cm.bone)\n            plt.axis('off');\n\nprint('There are {} different hospitals in the dataset.\\n'.format(len(train.site_id.unique())))            \nfor val in train.site_id.unique(): \n    images_site(val)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:12:56.572747Z","iopub.execute_input":"2022-12-31T21:12:56.573212Z","iopub.status.idle":"2022-12-31T21:13:41.089181Z","shell.execute_reply.started":"2022-12-31T21:12:56.573151Z","shell.execute_reply":"2022-12-31T21:13:41.087747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('There are {} unique patients in the Train Set.'.format(len(train['patient_id'].unique())))\n\ndata = train.groupby(by=\"patient_id\")['laterality'].count().reset_index(drop=False)\ndata = data.sort_values(['laterality']).reset_index(drop=True)\n\nprint(\"\\nMinimum number of entries are: {}\".format(data[\"laterality\"].min()), \"\\n\" +\n      \"Maximum number of entries are: {}\\n\".format(data[\"laterality\"].max()))\n\nplt.figure(figsize = (16, 4))\nimg = sns.barplot(data.index, data['laterality'], color=custom_colors[2])\nplt.title(\"Number of Entries per Patient\", fontsize = 17)\nplt.ylabel('Frequency', fontsize=14)\nimg.axes.get_xaxis().set_visible(False);","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:13:41.09092Z","iopub.execute_input":"2022-12-31T21:13:41.09129Z","iopub.status.idle":"2022-12-31T21:14:40.724647Z","shell.execute_reply.started":"2022-12-31T21:13:41.091258Z","shell.execute_reply":"2022-12-31T21:14:40.723318Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Most common frequency is **4 images per pacient**. However, we observe that there is a big amount of them having between 5-6 of them. Rarely, a pacient has more images asociated. \n\n## Laterality, View and Age \n\n* We almost have the same amount of left breast images than right ones. \n* Ver few values under 40 years old for `Age`. Some peaks between 50 and 70 yo. \n* Six different values for `view` feature. Quite imbalanced (**CC** and **MLO** are the most common ones). \n\n`Laterality` feature indicates whether the image is of the left or right breast. This issue can be fixed quite fast with OpenCV tools, for example. We'll focus on it later. `View` instead, refers to the orientation of the image. The default for a screening exam is to capture two views per breast. That's the reason for having almost the same amount of left and right breast images.  ","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows = 1, ncols = 3, figsize = (16,4))\nsns.countplot(train.laterality, label = ['Left','Right'], ax = axes[0], palette = custom_colors)\naxes[0].set_title('Laterality Count')\nsns.countplot(train.view, ax = axes[1], palette = custom_colors[3:])\naxes[1].set_title('View Count')\nsns.distplot(train.age, ax = axes[2], color = custom_colors[5])\naxes[2].set_title('Age Dsitribution')","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:14:40.726706Z","iopub.execute_input":"2022-12-31T21:14:40.727475Z","iopub.status.idle":"2022-12-31T21:14:41.635879Z","shell.execute_reply.started":"2022-12-31T21:14:40.727426Z","shell.execute_reply":"2022-12-31T21:14:41.63466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ❗❗❗View feature is more important than you think❗❗❗\n\nNow, **let's explain more in detail the `View` feature**. Recently, [@hengck23](https://www.kaggle.com/hengck23) posted a discussion talking about the fact that ADMANI dataset[1] is part of this competition's data (see discussion https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370333#2076911, and one notebook explaining the details of [one paper] that shows good results with the ADMANI dataset. Let's examine better the image attached to the explanation: \n\n![](https://i.ibb.co/4S9BcxG/Selection-318.png)\n\nYou can observe that images are clasified in **main** and **auxiliary** images. Let's now observe an example of an image for each of the different types of view we're given. ","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows = 2, ncols = 3, figsize = (22,10))\nfor i, val in enumerate(train.view.unique()):\n    ids = train[train.view == val]['patient_id'].unique()\n    image_id = train[(train.patient_id == ids[0]) & (train.view == val)]['image_id']\n    img_id = dcm_path + str(ids[0]) + '/' + str(image_id.values[0]) + '.dcm'\n    dataset = pydicom.dcmread(img_id)\n    axes[i // 3, i % 3].imshow(dataset.pixel_array, cmap=plt.cm.bone)\n    axes[i // 3, i % 3].axis('off')\n    axes[i // 3, i % 3].set_title('View: {}'.format(val))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:14:41.637604Z","iopub.execute_input":"2022-12-31T21:14:41.638496Z","iopub.status.idle":"2022-12-31T21:14:58.004905Z","shell.execute_reply.started":"2022-12-31T21:14:41.638445Z","shell.execute_reply":"2022-12-31T21:14:58.003736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Actually **CC and MLO correspond to the main and auxiliary types**, respectively. To find out it by yourselves please head to the following [article](https://radiopaedia.org/articles/craniocaudal-view). Therefore, trying some different approaches when training (such as making a distinction) our models could make a difference and have a ridiculously significant effect in LB. Thus, it's gonna be time-worthy to do some research about it. \n\nMoreover, let's examine whether there is any relationship between these values and the diagnosis of cancer. Just to remind, except from CC and MLO values the rest have very few samples. We must take this into account to analyse properly these plots. \n\n> As it can be appreciated, values are the same for CC and MLO types. This makes sense regarding what I told above. ","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows = 2, ncols = 3, figsize = (22, 8))\nfor i, val in enumerate(train.view.unique()):\n    dt = [train[(train.view == val) & (train.cancer == c)].shape[0] for c in [0,1]]\n    axes[i // 3, i % 3].pie(dt, labels = ['No Cancer','Cancer'], colors=[custom_colors[0], \n                            custom_colors[5]], autopct='%.2f%%')\n    axes[i // 3, i % 3].set_title('View: {}'.format(val))","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:14:58.007045Z","iopub.execute_input":"2022-12-31T21:14:58.007931Z","iopub.status.idle":"2022-12-31T21:14:58.630613Z","shell.execute_reply.started":"2022-12-31T21:14:58.007883Z","shell.execute_reply":"2022-12-31T21:14:58.629394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Cancer, Biopsy, Invasive and BIRADS\n\nThese features are only provided for training. First of all, let's show images with both a negative and a positive diagnosis. Aparently, I can't notice any significant difference between these images. \n\n","metadata":{"execution":{"iopub.status.busy":"2022-12-24T00:11:31.158532Z","iopub.execute_input":"2022-12-24T00:11:31.158935Z","iopub.status.idle":"2022-12-24T00:11:31.170369Z","shell.execute_reply.started":"2022-12-24T00:11:31.158903Z","shell.execute_reply":"2022-12-24T00:11:31.169066Z"}}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows = 2, ncols = 4, figsize = (22,8))\nfor i, val in enumerate(train.cancer.unique()):\n    ids = train[train.cancer == val]['patient_id'].unique()\n    patient_path = dcm_path + str(ids[i]) +'/'\n    for j, file in enumerate(listdir(patient_path)[:4]): \n        dataset = pydicom.dcmread(patient_path + file)\n        axes[i,j].imshow(dataset.pixel_array, cmap=plt.cm.bone)\n        axes[i,j].axis('off')\n        axes[i,j].set_title('Cancer: {}'.format(val))","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:14:58.632348Z","iopub.execute_input":"2022-12-31T21:14:58.633638Z","iopub.status.idle":"2022-12-31T21:15:18.465753Z","shell.execute_reply.started":"2022-12-31T21:14:58.633585Z","shell.execute_reply":"2022-12-31T21:15:18.46451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Cancer, biopsy and invasive distributions are very imbalanced. Seems that there could be a relationship between them.\n* In `BIRADS` plot, we observe that there are lots of negative ratings for cancer. Rating a breast as normal is the least common one.","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows = 1, ncols = 4, figsize = (22,5))\nsns.countplot(train.cancer, ax = axes[0], palette = custom_colors)\naxes[0].set_title('Cancer')\nsns.countplot(train.biopsy, ax = axes[1], palette = custom_colors[2:])\naxes[1].set_title('Biopsy')\nsns.countplot(train.invasive, ax = axes[2], palette = custom_colors[3:])\naxes[2].set_title('Invasive')\nsns.countplot(train.BIRADS, ax = axes[3], palette = custom_colors[4:])\naxes[3].set_title('BIRADS')","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:15:18.467588Z","iopub.execute_input":"2022-12-31T21:15:18.468072Z","iopub.status.idle":"2022-12-31T21:15:18.975337Z","shell.execute_reply.started":"2022-12-31T21:15:18.468037Z","shell.execute_reply":"2022-12-31T21:15:18.973901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Any relationship between them ? Let's explore it below. \n\n* `Invasive` and `cancer` are **very correlated** (this makes sense if we observe features' definitions).   \n* Negative correlation between `BIRADS` and the rest of the features. ","metadata":{}},{"cell_type":"code","source":"corr= train[['cancer','biopsy','invasive','BIRADS']].corr()\n# Getting the Upper Triangle of the co-relation matrix\nmatrix = np.triu(corr)\n\nfig, axes = plt.subplots(nrows = 1, ncols = 2, figsize = (22,8))\n# Heatmap without absolute values\nsns.heatmap(corr, mask=matrix, center = 0, cmap = 'vlag', ax = axes[0]).set_title('Without absolute values')\n# Heatmap with absolute values\nsns.heatmap(abs(corr), mask=matrix, center = 0, cmap = 'vlag', ax = axes[1]).set_title('With absolute values')\n\nfig.tight_layout(h_pad=1.0, w_pad=0.5)","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:15:18.97695Z","iopub.execute_input":"2022-12-31T21:15:18.977323Z","iopub.status.idle":"2022-12-31T21:15:19.675402Z","shell.execute_reply.started":"2022-12-31T21:15:18.97729Z","shell.execute_reply":"2022-12-31T21:15:19.673951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`Biopsy` feature determines whether or not a follow-up biopsy was performed on the breast. Thus, it could be interesting to analyse its relation with `Cancer` and `View` features.","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows = 2, ncols = 3, figsize = (22, 8))\nfig.suptitle('Biopsy vs View', fontsize = 15)\nfor i, val in enumerate(train.view.unique()):\n    dt = [train[(train.view == val) & (train.biopsy == c)].shape[0] for c in [0,1]]\n    axes[i // 3, i % 3].pie(dt, labels = ['No Biopsy','Biopsy'], colors=[custom_colors[0], \n                            custom_colors[5]], autopct='%.2f%%')\n    axes[i // 3, i % 3].set_title('View: {}'.format(val))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:15:19.677009Z","iopub.execute_input":"2022-12-31T21:15:19.677564Z","iopub.status.idle":"2022-12-31T21:15:20.309636Z","shell.execute_reply.started":"2022-12-31T21:15:19.677527Z","shell.execute_reply":"2022-12-31T21:15:20.308155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we focus on its relation with cancer diagnosis. It's appreciatable that when there is no need to make a biopsy, cancer diagnosis is discarded. However, we can notice that when it's done we have the same amount of positive and negative results. ","metadata":{}},{"cell_type":"code","source":"for b in [0,1]: \n    print('Biopsy: {}'.format('Not Performed' if b == 0 else 'Performed'))\n    for c in [0,1]: \n        dt = [train[(train.biopsy == b) & (train.cancer == c)].shape[0] for c in [0,1]]\n        print('\\tPatients with{}diagnosed cancer: {}'.format(' no ' if c == 0 else ' ', dt[c]))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:15:20.311195Z","iopub.execute_input":"2022-12-31T21:15:20.311617Z","iopub.status.idle":"2022-12-31T21:15:20.341332Z","shell.execute_reply.started":"2022-12-31T21:15:20.311581Z","shell.execute_reply":"2022-12-31T21:15:20.339853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we head into `Invasive` feature analysis. No clear difference between one type of image and another.","metadata":{}},{"cell_type":"code","source":"def invasive_images(val): \n    fig, axes = plt.subplots(nrows = 3, ncols = 4, figsize = (22,15))\n    fig.suptitle('Invasive: {}'.format(val), fontsize = 10)\n    ids = train[train.invasive == val]['patient_id'].unique()\n    for i in range(3): \n        patient_path = dcm_path + str(ids[i]) +'/'\n        for j, file in enumerate(listdir(patient_path)[:4]): \n            dataset = pydicom.dcmread(patient_path + file)\n            axes[i,j].imshow(dataset.pixel_array, cmap=plt.cm.bone)\n            axes[i,j].axis('off')\n        \nfor val in train.invasive.unique():\n    invasive_images(val)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:15:20.342614Z","iopub.execute_input":"2022-12-31T21:15:20.342975Z","iopub.status.idle":"2022-12-31T21:16:23.097017Z","shell.execute_reply.started":"2022-12-31T21:15:20.342943Z","shell.execute_reply":"2022-12-31T21:16:23.095476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"s1 = len(train[train.cancer == 1].axes[0])\ns2 = len(train[(train.cancer == 1) & (train.invasive == 1)])\nprint('Percentage of breasts with an invasive cancer: {}%'.format(round(s2/s1 * 100, 2)))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:16:23.09907Z","iopub.execute_input":"2022-12-31T21:16:23.100158Z","iopub.status.idle":"2022-12-31T21:16:23.11455Z","shell.execute_reply.started":"2022-12-31T21:16:23.10011Z","shell.execute_reply":"2022-12-31T21:16:23.113465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Implant and Density\n\nLet's explore now the other two image features that we're given in this dataset. ","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows = 1, ncols = 2, figsize = (16,5))\nsns.countplot(train.implant, label = ['Left','Right'], ax = axes[0], palette = custom_colors)\naxes[0].set_title('Implant Count')\nsns.countplot(train.density, ax = axes[1], palette = custom_colors[2:])\naxes[1].set_title('Density Count')","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:16:23.116419Z","iopub.execute_input":"2022-12-31T21:16:23.117734Z","iopub.status.idle":"2022-12-31T21:16:23.517258Z","shell.execute_reply.started":"2022-12-31T21:16:23.117671Z","shell.execute_reply":"2022-12-31T21:16:23.516427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Almost every breast have no implants. We have just a few images from breasts with implants. \n* B and C values for `density` are the most usual ones. Just to remind, highly dense tissue can make diagnosis more difficult (case D). So this is something that we'll need to take into account in validation strategies. \n\nIn the following chart we can appreciate the four different types of density that we have. We can notice that the closer the value is to D, we can observe that there are more white spots on the end of the breast. On the other hand, for A density type images, colour is quite uniform.","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows = 1, ncols = 4, figsize = (22,5))\nfig.suptitle('Exploring Density Types', fontsize = 12)\nfor i, val in enumerate(train.density.unique()[1:]):\n    ids = train[train.density == val]['patient_id'].unique()\n    patient_path = dcm_path + str(ids[9]) +'/'\n    for j, file in enumerate([listdir(patient_path)[0]]): \n        dataset = pydicom.dcmread(patient_path + file)\n        axes[i].imshow(dataset.pixel_array, cmap=plt.cm.bone)\n        axes[i].axis('off')\n        axes[i].set_title('Density: {}'.format(val))","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:16:23.518619Z","iopub.execute_input":"2022-12-31T21:16:23.519174Z","iopub.status.idle":"2022-12-31T21:16:32.431044Z","shell.execute_reply.started":"2022-12-31T21:16:23.519142Z","shell.execute_reply":"2022-12-31T21:16:32.429824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Machine ID\n\nThis feature does not seem that will have a significant effect. However, we realise that most of the pictures have been taken with machine 49. ","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize = (16,5))\nsns.countplot(train.machine_id, palette = custom_colors)","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:16:32.433006Z","iopub.execute_input":"2022-12-31T21:16:32.43377Z","iopub.status.idle":"2022-12-31T21:16:32.701581Z","shell.execute_reply.started":"2022-12-31T21:16:32.43371Z","shell.execute_reply":"2022-12-31T21:16:32.700679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## 3 External Data\n---\n\nIn this subsection we'll explore a bit the Chinese Mammography Database. This is one the publicly available mammogram datasets. It contains 5,202 mammograms from 1,775 patients with normal, benign or malignant biopsy-confirmed tumours. \n\n> This dataset could be quite useful for several things. We can use it as a dev set, with the aim of extending our dataset, to treat the class imbalance we have for cancer, etc. ","metadata":{}},{"cell_type":"code","source":"def load_data():\n    '''Load each of the datasets we are given.'''\n    \n    data_dir = Path(\"../input/cmmd2022/TheChineseMammographyDatabase\")\n    metadata = pd.read_csv(data_dir / \"metadata.csv\")\n    sheet = pd.read_excel(data_dir / \"CMMD_clinicaldata_revision.xlsx\")\n    return metadata, sheet\n\nmetadata, sheet = load_data()\nclear_output()\n\nnames = [\"Metadata\", \"Sheet\"]\nfor i, df in enumerate([metadata, sheet]): \n    data_info(df, names[i])","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:16:40.877723Z","iopub.execute_input":"2022-12-31T21:16:40.878253Z","iopub.status.idle":"2022-12-31T21:16:41.184523Z","shell.execute_reply.started":"2022-12-31T21:16:40.878216Z","shell.execute_reply":"2022-12-31T21:16:41.183147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's examine how `classification` feature's values are distributed. ","metadata":{}},{"cell_type":"code","source":"sns.countplot(sheet.classification)","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:17:30.38692Z","iopub.execute_input":"2022-12-31T21:17:30.387436Z","iopub.status.idle":"2022-12-31T21:17:30.587882Z","shell.execute_reply.started":"2022-12-31T21:17:30.387394Z","shell.execute_reply":"2022-12-31T21:17:30.586502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import glob\n\nbenign_ids = sheet[sheet.classification == 'Benign'].ID1.values\nmalignant_ids = sheet[sheet.classification == 'Malignant'].ID1.values\n\n# =============== Code by stpete_ishii https://www.kaggle.com/stpeteishii/siim-acr-dicom-slide-show/notebook?select=stage_2_images\n\nfiles=os.listdir('../input/cmmd2022/TheChineseMammographyDatabase/CMMD')\nprint(len(files))\nn=len(files)\nN=[]\nfor i in range(n//10):\n    N+=[i*10]\n    \n# ============== Code by: https://www.kaggle.com/code/mpwolke/mammography-dcm-slide-show\n\ndef load_dicom(path):\n    dicom=pydicom.read_file(path,force=True)\n    data=dicom.pixel_array\n    data=data-np.min(data)\n    if np.max(data) != 0:\n        data=data/np.max(data)\n    data=(data*255).astype(np.uint8)\n    return data\n\ndef load_dicom_line(path):\n    t_paths = sorted(\n        glob.glob(os.path.join(path,\"*/**/***/*.dcm\")), #Original (os.path.join(path,\"*\")),\n        key=lambda x: x[:-4].split(\"_\")[-1],\n    )\n    \n    images = []\n    for filename in tqdm(np.array(t_paths)[N]):\n        data = load_dicom(filename)\n        if data.max() == 0:\n            continue\n        images.append(data)\n        \n    return t_paths, images\n\n# ==============\n\nscans = !ls ../input/cmmd2022/TheChineseMammographyDatabase/CMMD/ -d\nall_paths, all_images = load_dicom_line(scans[0])\n\nbenign_imgs_path = []\nmalignant_imgs_path = []\n\nfor img_path in all_paths: \n    for idx in benign_ids:\n        if img_path.find(idx) != -1: \n            benign_imgs_path.append(img_path)\n            break\n            \nfor img_path in all_paths: \n    for idx in malignant_ids:\n        if img_path.find(idx) != -1: \n            malignant_imgs_path.append(img_path)\n            break\n            \nbenign_images = []\nmalignant_images = []\n\nfor path in tqdm(benign_imgs_path): \n    benign_images.append(load_dicom(path))\n    \nfor path in tqdm(malignant_imgs_path): \n    malignant_images.append(load_dicom(path))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-31T21:18:12.553399Z","iopub.execute_input":"2022-12-31T21:18:12.554972Z","iopub.status.idle":"2022-12-31T21:32:06.526965Z","shell.execute_reply.started":"2022-12-31T21:18:12.554894Z","shell.execute_reply":"2022-12-31T21:32:06.524167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(benign_images[2], cmap = 'gray')\nplt.axis('off')\nplt.title('Benign')","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:36:15.808902Z","iopub.execute_input":"2022-12-31T21:36:15.809959Z","iopub.status.idle":"2022-12-31T21:36:16.634387Z","shell.execute_reply.started":"2022-12-31T21:36:15.809897Z","shell.execute_reply":"2022-12-31T21:36:16.633071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To finish this part, we're gonna save this images in the following directory: ","metadata":{}},{"cell_type":"code","source":"import cv2\n\nfor i, img in tqdm(enumerate(malignant_images)):\n    os.makedirs(\"./Malignant/{}\".format(malignant_imgs_path[i][53:60]), exist_ok=True)\n    cv2.imwrite(\"./Malignant/{}/{}.png\".format(malignant_imgs_path[i][53:60], \\\n                                                       malignant_imgs_path[i][102:105]), img)\n    \nfor i, img in tqdm(enumerate(benign_images)):\n    os.makedirs(\"./Benign/{}\".format(benign_imgs_path[i][53:60]), exist_ok=True)\n    cv2.imwrite(\"./Benign/{}/{}.png\".format(benign_imgs_path[i][53:60], \\\n                                                       benign_imgs_path[i][102:105]), img)","metadata":{"execution":{"iopub.status.busy":"2022-12-31T21:36:39.936Z","iopub.execute_input":"2022-12-31T21:36:39.936466Z","iopub.status.idle":"2022-12-31T21:40:27.321414Z","shell.execute_reply.started":"2022-12-31T21:36:39.936431Z","shell.execute_reply":"2022-12-31T21:40:27.319533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## 4 | References\n\n---\n\n* [DICOM Metadata Extracting Attributes to DataFrame. Author: @anarthal](https://www.kaggle.com/code/anarthal/dicom-metadata-extracting-attributes-to-dataframe/notebook)\n\n* [RSNA Fracture Detection: DICOM & Images Explore. Author: @andradaolteanu](https://www.kaggle.com/code/andradaolteanu/rsna-fracture-detection-dicom-images-explore)\n\n* [MVCCL Model for Admani dataset, Author: @hengck23](https://www.kaggle.com/code/hengck23/mvccl-model-for-admani-dataset)\n\n* [Mammography DCM Slide show](https://www.kaggle.com/code/mpwolke/mammography-dcm-slide-show)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}