{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div align=\"center\"><p style=\"font-family: 'Mochiy Pop P One';font-size:32px;color:black\">RSNA Screening Mammography Breast Cancer Detection - EDA and Modeling</p></div>\n","metadata":{}},{"cell_type":"code","source":"!pip install -qU python-gdcm pydicom pylibjpeg","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:03:33.895725Z","iopub.execute_input":"2022-11-29T18:03:33.896432Z","iopub.status.idle":"2022-11-29T18:03:54.865878Z","shell.execute_reply.started":"2022-11-29T18:03:33.896329Z","shell.execute_reply":"2022-11-29T18:03:54.864326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport cv2\nimport glob\nimport gdcm\nimport pydicom\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom tqdm.notebook import tqdm\nfrom joblib import Parallel, delayed","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:03:54.869328Z","iopub.execute_input":"2022-11-29T18:03:54.869785Z","iopub.status.idle":"2022-11-29T18:03:56.063152Z","shell.execute_reply.started":"2022-11-29T18:03:54.869745Z","shell.execute_reply":"2022-11-29T18:03:56.061683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:22px;color:black\">Data Description</p>\n\n<p><strong>[train/test]_images/[patient_id]/[image_id].dcm</strong> The mammograms, in dicom format. You can expect roughly 8,000 patients in the hidden test set. There are usually but not always 4 images per patient. Note that many of the images use the jpeg 2000 format which may you may need special libraries to load.<br><br>\n<strong>sample_submission.csv</strong> A valid sample submission. Only the first few rows are available for download.<br><br>\n<strong>[train/test].csv</strong> Metadata for each patient and image. Only the first few rows of the test set are available for download.</p>\n\n<ul>\n<li><code>site_id</code> - ID code for the source hospital.</li>\n<li><code>patient_id</code> - ID code for the patient.</li>\n<li><code>image_id</code> - ID code for the image.</li>\n<li><code>laterality</code> - Whether the image is of the left or right breast.</li>\n<li><code>view</code> - The orientation of the image. The default for a screening exam is to capture two views per breast.</li>\n<li><code>age</code> - The patient's age in years.</li>\n<li><code>implant</code> - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.</li>\n<li><code>density</code> - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult.</li>\n<li><code>machine_id</code> - An ID code for the imaging device.</li>\n<li><code>cancer</code> - The target value. Only provided for train.</li>\n<li><code>biopsy</code> - Whether or not a follow-up biopsy was performed on the breast. Only provided for train.</li>\n<li><code>invasive</code> - If the breast is positive for cancer, whether or not the cancer proved to be invasive. Only provided for train.</li>\n<li><code>BIRADS</code> - 0 if the breast required follow-up, 1 if the breast was rated as negative for cancer, and 2 if the breast was rated as normal. Only provided for train.</li>\n<li><code>prediction_id</code> - The ID for the matching submission row. Multiple images will share the same prediction ID. Test only.</li>\n<li><code>difficult_negative_case</code> - True if the case was unusually difficult. Only provided for train.</li>\n</ul>","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\")\ntrain_df.head(3)","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:04:05.15396Z","iopub.execute_input":"2022-11-29T18:04:05.154369Z","iopub.status.idle":"2022-11-29T18:04:05.322582Z","shell.execute_reply.started":"2022-11-29T18:04:05.154338Z","shell.execute_reply":"2022-11-29T18:04:05.321337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:20px;\">Target variable</p>","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(5, 5))\nsns.countplot(data=train_df, x=\"cancer\");","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:04:07.518417Z","iopub.execute_input":"2022-11-29T18:04:07.518839Z","iopub.status.idle":"2022-11-29T18:04:07.794504Z","shell.execute_reply.started":"2022-11-29T18:04:07.518806Z","shell.execute_reply":"2022-11-29T18:04:07.793149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:20px;\">density - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult.</p>","metadata":{}},{"cell_type":"code","source":"train_df[\"density\"].hist()","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:23:44.970455Z","iopub.execute_input":"2022-11-29T18:23:44.970908Z","iopub.status.idle":"2022-11-29T18:23:45.226268Z","shell.execute_reply.started":"2022-11-29T18:23:44.970846Z","shell.execute_reply":"2022-11-29T18:23:45.225245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:20px;\">Age</p>","metadata":{}},{"cell_type":"code","source":"train_df[\"age\"].hist()","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:24:44.02042Z","iopub.execute_input":"2022-11-29T18:24:44.020846Z","iopub.status.idle":"2022-11-29T18:24:44.214813Z","shell.execute_reply.started":"2022-11-29T18:24:44.020813Z","shell.execute_reply":"2022-11-29T18:24:44.21345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:20px;\">Display the images</p>","metadata":{}},{"cell_type":"code","source":"def load_img(img_path):\n    #img_path=\"/kaggle/input/rsna-breast-cancer-detection/train_images/10038/1967300488.dcm\"\n    dicom = pydicom.dcmread(img_path)\n    img = dicom.pixel_array\n\n    img = (img - img.min()) / (img.max() - img.min())\n\n    if dicom.PhotometricInterpretation == \"MONOCHROME1\":\n        img = 1 - img\n        \n    return img","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:06:15.462527Z","iopub.execute_input":"2022-11-29T18:06:15.463063Z","iopub.status.idle":"2022-11-29T18:06:15.470016Z","shell.execute_reply.started":"2022-11-29T18:06:15.463024Z","shell.execute_reply":"2022-11-29T18:06:15.469085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_dicom(path):\n    dicom = pydicom.read_file(path)\n    #dicom = pydicom.dcmread(path)\n    data = dicom.pixel_array\n    data = data - np.min(data)\n    if np.max(data) != 0:\n        data = data / np.max(data)\n    data = (data * 255).astype(np.uint8)\n    return data\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize_sample(\n    base_path, \n    data_df\n):\n    fig, axs = plt.subplots(1, data_df.shape[0], figsize=(18, 6), sharey=False, sharex=False)\n    plt.style.use('seaborn-whitegrid')\n    i = 0\n    for _, row in data_df.iterrows():\n        patient_id = row['patient_id']\n        image_id = row['image_id']\n        img_path =  f'{base_path}/{patient_id}/{image_id}.dcm'\n        #data = load_dicom(img_path)\n        data = load_img(img_path)\n        axs[i].imshow(data, cmap=\"gray\")\n        axs[i].set_title(f'{image_id}-view:{row[\"view\"]}, side: {row[\"laterality\"]}\\ncancer positive: {row[\"cancer\"]}, age: {row[\"age\"]}', \n                         fontsize=13, loc='left')\n        i = i + 1\n\n    plt.suptitle(f\"Patient: {patient_id}\", fontsize=15,)\n    plt.xticks(fontsize=10)\n    plt.yticks(fontsize=10)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:31:12.886399Z","iopub.execute_input":"2022-11-29T18:31:12.886875Z","iopub.status.idle":"2022-11-29T18:31:12.897746Z","shell.execute_reply.started":"2022-11-29T18:31:12.886824Z","shell.execute_reply":"2022-11-29T18:31:12.895834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_path=\"/kaggle/input/rsna-breast-cancer-detection/train_images\"","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:31:13.264437Z","iopub.execute_input":"2022-11-29T18:31:13.265145Z","iopub.status.idle":"2022-11-29T18:31:13.270093Z","shell.execute_reply.started":"2022-11-29T18:31:13.265108Z","shell.execute_reply":"2022-11-29T18:31:13.268917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_ids = np.random.choice(train_df['patient_id'].unique(), 10, replace=False)\nsample_ids","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:31:13.777466Z","iopub.execute_input":"2022-11-29T18:31:13.778319Z","iopub.status.idle":"2022-11-29T18:31:13.79025Z","shell.execute_reply.started":"2022-11-29T18:31:13.778275Z","shell.execute_reply":"2022-11-29T18:31:13.788954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#sample_ids = [10006, 10011, 10025]","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:31:14.77688Z","iopub.execute_input":"2022-11-29T18:31:14.778655Z","iopub.status.idle":"2022-11-29T18:31:14.784288Z","shell.execute_reply.started":"2022-11-29T18:31:14.778566Z","shell.execute_reply":"2022-11-29T18:31:14.783009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for patient_id in sample_ids:\n    sub = train_df[train_df['patient_id']==patient_id]\n    \n    visualize_sample(base_path=base_path, data_df=sub)","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:31:15.264224Z","iopub.execute_input":"2022-11-29T18:31:15.26499Z","iopub.status.idle":"2022-11-29T18:32:19.684796Z","shell.execute_reply.started":"2022-11-29T18:31:15.264947Z","shell.execute_reply":"2022-11-29T18:32:19.683886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:22px;color:black\">unusually difficult cases</p>","metadata":{}},{"cell_type":"code","source":"sample_ids = train_df[train_df['difficult_negative_case']]['patient_id'].unique()[:5]\nfor patient_id in sample_ids:\n    sub = train_df[train_df['patient_id']==patient_id]\n    \n    visualize_sample(base_path=base_path, data_df=sub)","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:32:19.686753Z","iopub.execute_input":"2022-11-29T18:32:19.687403Z","iopub.status.idle":"2022-11-29T18:32:30.358902Z","shell.execute_reply.started":"2022-11-29T18:32:19.687369Z","shell.execute_reply":"2022-11-29T18:32:30.357638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:22px;color:black\">Cancer Positive</p>","metadata":{}},{"cell_type":"code","source":"def visualize_sample2(\n    base_path, \n    data_df\n):\n    fig, axs = plt.subplots(1, data_df.shape[0], figsize=(18, 6), sharey=False, sharex=False)\n    plt.style.use('seaborn-whitegrid')\n    i = 0\n    for _, row in data_df.iterrows():\n        patient_id = row['patient_id']\n        image_id = row['image_id']\n        img_path =  f'{base_path}/{patient_id}/{image_id}.dcm'\n        #data = load_dicom(img_path)\n        data = load_img(img_path)\n        axs[i].imshow(data, cmap=\"gray\")\n        axs[i].set_title(f'patient: {patient_id}\\n{image_id}-view:{row[\"view\"]}, side: {row[\"laterality\"]}\\ncancer positive: {row[\"cancer\"]}, age: {row[\"age\"]}', \n                         fontsize=13, loc='left')\n        i = i + 1\n\n    plt.suptitle(f\"\", fontsize=15,)\n    plt.xticks(fontsize=10)\n    plt.yticks(fontsize=10)\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-29T18:34:03.528731Z","iopub.execute_input":"2022-11-29T18:34:03.529233Z","iopub.status.idle":"2022-11-29T18:34:03.539443Z","shell.execute_reply.started":"2022-11-29T18:34:03.529198Z","shell.execute_reply":"2022-11-29T18:34:03.538086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = train_df[(train_df['cancer']==1)].head(4)\n    \nvisualize_sample2(base_path=base_path, data_df=sub)","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:34:04.542443Z","iopub.execute_input":"2022-11-29T18:34:04.543568Z","iopub.status.idle":"2022-11-29T18:34:12.260277Z","shell.execute_reply.started":"2022-11-29T18:34:04.543522Z","shell.execute_reply":"2022-11-29T18:34:12.258997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:22px;color:black\">Cancer Negative</p>","metadata":{}},{"cell_type":"code","source":"sub = train_df[(train_df['cancer']==0)].head(4)\n    \nvisualize_sample2(base_path=base_path, data_df=sub)","metadata":{"execution":{"iopub.status.busy":"2022-11-29T18:34:12.262975Z","iopub.execute_input":"2022-11-29T18:34:12.263374Z","iopub.status.idle":"2022-11-29T18:34:25.057929Z","shell.execute_reply.started":"2022-11-29T18:34:12.263339Z","shell.execute_reply":"2022-11-29T18:34:25.056383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p style=\"font-family: 'Mochiy Pop P One';font-size:22px;color:black\">Modeling</p>","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n<p style=\"font-family: 'Mochiy Pop P One';font-size:20px;\">Reference:</p>\n\n* https://www.kaggle.com/code/ihelon/brain-tumor-eda-with-animations-and-modeling\n* https://www.kaggle.com/code/theoviel/dicom-resized-png-jpg/notebook","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}