{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# RSNA Metadata Exploratory Analysis \n\n\n\n### [**1. Introduction**](#introduction)\n\n [Breast Cancer and Mammography](#introsubsection1)\n \n [Competition Objective and Success Metrics](#introsubsection2)\n \n [Provided Data](#introsubsection3)\n \n [What this notebook is all about](#introsubsection4)\n \n---\n \n### [**2. Load Data**](#loaddata)\n\n---\n\n### [**3. Metadata Structure**](#meta)\n\n [Column Descriptions](#meta1)\n\n [Metadata Interaction](#meta2)\n \n---\n\n### [**4. Profiling Report**](#profile)\n\n [Early Observations](#profile1)\n\n [Patient Level Profiles](#profile2)\n\n [Missing Treatment](#profile3)\n\n [Bivariate Analysis](#profile4)\n \n - [Age](#profile5)\n - [Site](#profile6)\n - [Machine](#profile7)\n - [View](#profile8)\n - [BIRADS & Density](#profile9)\n \n---\n\n### [**5. Insights**](#insights)\n\n [Data Insights](#insights1)\n \n [Patient Level Insights](#insights2)\n \n [Age Related Insights](#insights3)\n \n### [**6. Conclusion**](#conclusion)\n\n---\n---","metadata":{"_kg_hide-input":false}},{"cell_type":"markdown","source":"\n# 1. Introduction \n\n[Main Menu](#top) | [< prev](#top) | [next >](#introsubsection1)\n\n\n#### Breast Cancer and Mammography \n[Main Menu](#top) | [< prev](#introduction) | [next >](#introsubsection2)\n\nBreast cancer is a common type of cancer that affects women and, less commonly, men. According to the World Health Organization (WHO), breast cancer is the most common cancer among women worldwide, accounting for approximately 25% of all cases. In the United States, breast cancer is the second most common type of cancer in women, after skin cancer.\n\nMammography is a type of imaging test that uses low-dose x-rays to examine the breasts. It is the most common test used to detect breast cancer. Mammograms can be used to screen for breast cancer in women who have no symptoms and to diagnose breast cancer in women who do have symptoms, such as a lump in the breast or abnormal discharge from the nipple.\n\nMammograms are generally considered to be a safe and effective way to detect breast cancer, but they are not perfect. They can sometimes miss breast cancers, especially in women with dense breast tissue. Additionally, they can sometimes produce false-positive results, which can lead to unnecessary anxiety and further testing.\n\n\n\n---\n\n#### Competition Objective & Success Metrics \n[Main Menu](#top) | [< prev](#introsubsection1) | [next >](#introsubsection3)\n\nCurrently, early detection of breast cancer requires the expertise of highly-trained human observers, making screening mammography programs expensive to conduct. A looming shortage of radiologists in several countries will likely worsen this problem.\n\nEarly detection and treatment are critical to reducing cancer fatalities, and an accurate inference model that uses mammography images to predict cancer will help reduce the burden on human observers by streamlining the screening process.\n\nThe model will be evaluated on the probablistic F1 Score. F1 Score is the harmonic mean of precision and recall.\n\n---\n\n#### Provided Data \n[Main Menu](#top) | [< prev](#introsubsection2) | [next >](#introsubsection4)\n\nThe provided data and test sets has multiple components - \n\n1. train.csv - Information about each mammography image. A patient has multiple images. 'cancer' variable is the target or the dependent variable\n\n2. test.csv - Information about each mammography image of test patients. The available information in train and test is different\n> Note : Only a single patient information is available for download. The entire test data will be made available to the notebook upon submission\n\n3. sample_submission.csv - A format in which the predicted probability of cancer is provided for each breast (Left and Right) for a test patients\n\n4. train_images - A corresponding image for each observation in train.csv. This folder is structured with subfolders for every patient and each patient has multiple images.\n\n5. test_images - In the same structure as train_images, a subfolder for a single test patient is provided with four images\n\n\n\n---\n#### What this notebook is all about \n[Main Menu](#top) | [< prev](#introsubsection3) | [next >](#loaddata)\n\nIn this notebook, we'll be looking at the metadata information provided in the train.csv and test.csv files along with a look at the sample_submission.csv file\n\n---\n---","metadata":{}},{"cell_type":"markdown","source":"#### We'll begin by importing a few libraries \n- Pandas and Numpy for regular evaluation\n- Seaborn, Matplotlib and Pyplot for visualisation\n- Pandas Profiling for EDA report\n- Warnings for suppressing warnings in the notebook","metadata":{}},{"cell_type":"code","source":"%matplotlib inline\n#magic command to show the matplotlib plots in the output cell of the notebook\n\n#basic libraries\nimport pandas as pd\nimport numpy as np\nimport os\n\n#visualisation libraries\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom pandas_profiling import ProfileReport\n'''\nfrom sklearn.model_selection import KFold, cross_val_score, StratifiedKFold, learning_curve, train_test_split\nfrom sklearn.impute import SimpleImputer\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\n#plotly\nimport plotly.express as px\nfrom plotly.subplots import make_subplots\nimport plotly.figure_factory as ff\nimport plotly.offline as offline\nimport plotly.graph_objects as go\n'''\nimport warnings\nwarnings.filterwarnings('ignore')\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=UserWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:08.244574Z","iopub.execute_input":"2023-01-02T17:40:08.245035Z","iopub.status.idle":"2023-01-02T17:40:08.257222Z","shell.execute_reply.started":"2023-01-02T17:40:08.244996Z","shell.execute_reply":"2023-01-02T17:40:08.255591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Load Data \n[Main Menu](#top) | [< prev](#introsubsection4) | [next >](#meta)\n\nLoading the three csv files using 'read_csv' API of the pandas library","metadata":{}},{"cell_type":"code","source":"train=pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\")\ntest=pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/test.csv\")\nsample_submission=pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:08.389093Z","iopub.execute_input":"2023-01-02T17:40:08.389525Z","iopub.status.idle":"2023-01-02T17:40:08.476218Z","shell.execute_reply.started":"2023-01-02T17:40:08.389485Z","shell.execute_reply":"2023-01-02T17:40:08.475216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n---","metadata":{}},{"cell_type":"markdown","source":"# 3. Metadata Structure \n[Main Menu](#top) | [< prev](#loaddata) | [next >](#meta1)\n\nA cursory look at each of the files","metadata":{}},{"cell_type":"code","source":"print(\"\\nColumns in each of the datasets\")\nprint(\"\\nData: train.csv\")\nprint(train.columns)\nprint(\"\\nData: test.csv\")\nprint(test.columns)\nprint(\"\\nData: sample_submission.csv\")\nprint(sample_submission.columns)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:08.477921Z","iopub.execute_input":"2023-01-02T17:40:08.478464Z","iopub.status.idle":"2023-01-02T17:40:08.486181Z","shell.execute_reply.started":"2023-01-02T17:40:08.47843Z","shell.execute_reply":"2023-01-02T17:40:08.485139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\nA few rows of train.csv\\n\")\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:08.487874Z","iopub.execute_input":"2023-01-02T17:40:08.488553Z","iopub.status.idle":"2023-01-02T17:40:08.511778Z","shell.execute_reply.started":"2023-01-02T17:40:08.488461Z","shell.execute_reply":"2023-01-02T17:40:08.510471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\nRows of train.csv\\n\")\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:08.514716Z","iopub.execute_input":"2023-01-02T17:40:08.515254Z","iopub.status.idle":"2023-01-02T17:40:08.532371Z","shell.execute_reply.started":"2023-01-02T17:40:08.515201Z","shell.execute_reply":"2023-01-02T17:40:08.531128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\nRows of sample_submission.csv\\n\")\nsample_submission.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:08.533709Z","iopub.execute_input":"2023-01-02T17:40:08.534041Z","iopub.status.idle":"2023-01-02T17:40:08.546221Z","shell.execute_reply.started":"2023-01-02T17:40:08.534012Z","shell.execute_reply":"2023-01-02T17:40:08.544824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n### Column Descriptions (as provided in the problem statement) \n[Main Menu](#top) | [< prev](#meta) | [next >](#meta2)\n\n- site_id - ID code for the source hospital.\n- patient_id - ID code for the patient.\n- image_id - ID code for the image.\n- laterality - Whether the image is of the left or right breast.\n- view - The orientation of the image. The default for a screening exam is to capture two views per breast.\n- age - The patient's age in years.\n- implant - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.\n- density - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely - dense tissue can make diagnosis more difficult. Only provided for train.\n- machine_id - An ID code for the imaging device.\n- cancer - Whether or not the breast was positive for malignant cancer. The target value. Only provided for train.\n- biopsy - Whether or not a follow-up biopsy was performed on the breast. Only provided for train.\n- invasive - If the breast is positive for cancer, whether or not the cancer proved to be invasive. Only provided for train.\n- BIRADS - 0 if the breast required follow-up, 1 if the breast was rated as negative for cancer, and 2 if the breast was rated as normal. Only provided for train.\n- prediction_id - The ID for the matching submission row. Multiple images will share the same prediction ID. Test only.\n- difficult_negative_case - True if the case was unusually difficult. Only provided for train.\n \n \n [Main Menu](#top) | [< prev](#meta1) | [next >](#profile)\n \n \n\n\n\n---\n---","metadata":{}},{"cell_type":"markdown","source":"# 4. Profiling Report \n[Main Menu](#top) | [< prev](#meta2) | [next >](#profile1)\n\n\nPandas Profiling is a powerful library to access all the exploratory information about the data","metadata":{}},{"cell_type":"code","source":"profile=ProfileReport(train,title=\"RSNA Train data profiling\", explorative=True)\nprofile.to_notebook_iframe()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:08.702562Z","iopub.execute_input":"2023-01-02T17:40:08.70299Z","iopub.status.idle":"2023-01-02T17:40:27.388437Z","shell.execute_reply.started":"2023-01-02T17:40:08.702953Z","shell.execute_reply":"2023-01-02T17:40:27.386996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n### Early Observations \n[Main Menu](#top) | [< prev](#profile) | [next >](#profile2)\n\n- 'image_id' is the unique key present in the training data\n- The training data has observations for 54,706 images (no duplicates present)\n- This information is for 11,913 patients\n- The training data has observations from two sites\n- Images are of 6 views. However, MLO and CC views account for >99% images\n- 'age' ranges from 26 years to 89 years. For 0.1% (37) images Age is missing\n- 'cancer', the dependent or target variable has a rate of 2.1% amongst 54,706 images\n- 5.4% of the images warranted a biopsy\n- 1.5% of the images showed an invasive cancer. For all positive cancer images 70.6% showed invasive cancer\n- BIRADS information is missing for a majority of images\n- Less than 2.7% images were done on breasts with implants\n- A large proportion of density information is missing\n- 10 different machines were used across the two sites\n- 15% of all image scans were difficult to infer as negative for cancer\n---","metadata":{}},{"cell_type":"markdown","source":"### Patient Level Profiles \n[Main Menu](#top) | [< prev](#profile1) | [next >](#profile3)\n\nWhile the entire training data is at an image level, it is intriguing how this translates at a patient level\nWe will try to look at the following -\n- How many patients out of the 11,913 were diagnosed with cancer?\n- How many patients were advised for a biopsy?\n- How many patients had invasive cancer?\n- How many patients had implants and how did that correspond to having cancer?","metadata":{}},{"cell_type":"markdown","source":"---\n#### **Unique cancer patients**\n\nWe'll group the data by 'patient_id' and if the sum of 'cancer' indicator is not zero, we will count them as cancer patients","metadata":{}},{"cell_type":"code","source":"total_patients=train['patient_id'].nunique()\nunique_cancer_patients=np.where(train.groupby(['patient_id'])['cancer'].sum().reset_index()['cancer']>0,1,0).sum()\nprint(\"\\nThe total number of patients diagnosed with cancer is \" + str(unique_cancer_patients))\nprint(\"This indicates a prevalence rate of \" + str(round(unique_cancer_patients/total_patients*100,2)) + \"%\" + \" in \" + str(total_patients)+\" patients.\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.390571Z","iopub.execute_input":"2023-01-02T17:40:27.390942Z","iopub.status.idle":"2023-01-02T17:40:27.406699Z","shell.execute_reply.started":"2023-01-02T17:40:27.390903Z","shell.execute_reply":"2023-01-02T17:40:27.405516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Unique breasts with cancer**\n\n\nWe'll group the data by 'patient_id' and 'laterality' and if the sum of 'cancer' indicator is not zero, we will count them as breasts with cancer","metadata":{}},{"cell_type":"code","source":"total_breasts_scanned=train.groupby(['patient_id','laterality'])['cancer'].sum().reset_index().shape[0]\nunique_cancer_breasts=np.where(train.groupby(['patient_id','laterality'])['cancer'].sum().reset_index()['cancer']>0,1,0).sum()\nprint(\"\\nThe total number of breasts diagnosed with cancer is \" + str(unique_cancer_breasts))\nprint(\"This indicates a prevalence rate of \" + str(round(unique_cancer_breasts/total_breasts_scanned*100,2)) + \"%\" + \" in \" + str(total_breasts_scanned)+\" breasts.\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.408156Z","iopub.execute_input":"2023-01-02T17:40:27.408607Z","iopub.status.idle":"2023-01-02T17:40:27.448885Z","shell.execute_reply.started":"2023-01-02T17:40:27.408571Z","shell.execute_reply":"2023-01-02T17:40:27.447309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Patients with cancer in both breasts**","metadata":{}},{"cell_type":"code","source":"print(\"\\n\"+ str(unique_cancer_breasts-unique_cancer_patients) + \" patients were diagnosed with cancer in both breasts \")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.451858Z","iopub.execute_input":"2023-01-02T17:40:27.45224Z","iopub.status.idle":"2023-01-02T17:40:27.458456Z","shell.execute_reply.started":"2023-01-02T17:40:27.452205Z","shell.execute_reply":"2023-01-02T17:40:27.457306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\nWe'll now create a **patient level dataset** with necessary information","metadata":{}},{"cell_type":"code","source":"cancer_patient=train.groupby(['patient_id'])['biopsy','cancer','implant','invasive','difficult_negative_case'].sum().reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.460587Z","iopub.execute_input":"2023-01-02T17:40:27.460998Z","iopub.status.idle":"2023-01-02T17:40:27.481118Z","shell.execute_reply.started":"2023-01-02T17:40:27.46095Z","shell.execute_reply":"2023-01-02T17:40:27.47956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#changing sum to indicators\ncancer_patient['biopsy']=np.where(cancer_patient['biopsy']>0,1,0)\ncancer_patient['implant']=np.where(cancer_patient['implant']>0,1,0)\ncancer_patient['invasive']=np.where(cancer_patient['invasive']>0,1,0)\ncancer_patient['cancer']=np.where(cancer_patient['cancer']>0,1,0)\ncancer_patient['difficult_negative_case']=np.where(cancer_patient['difficult_negative_case']>0,1,0)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.482888Z","iopub.execute_input":"2023-01-02T17:40:27.483222Z","iopub.status.idle":"2023-01-02T17:40:27.494376Z","shell.execute_reply.started":"2023-01-02T17:40:27.483192Z","shell.execute_reply":"2023-01-02T17:40:27.493119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Patients advised for biopsy**","metadata":{}},{"cell_type":"code","source":"unique_biopsy_patients=cancer_patient['biopsy'].sum()\nprint(\"\\nThe total number of patients advised for biopsy is \" + str(unique_biopsy_patients))\nprint(\"This indicates that \" + str(round(unique_biopsy_patients/total_patients*100,2)) + \"%\" + \" of the total \" + str(total_patients)+\" patients were advised to get a biopsy\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.496345Z","iopub.execute_input":"2023-01-02T17:40:27.49684Z","iopub.status.idle":"2023-01-02T17:40:27.506307Z","shell.execute_reply.started":"2023-01-02T17:40:27.496777Z","shell.execute_reply":"2023-01-02T17:40:27.504961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Patients with invasive cancer**\n\nWe'll find out how many patients who were diagnosed with cancer had an invasive one","metadata":{}},{"cell_type":"code","source":"unique_invasive_patients=cancer_patient['invasive'].sum()\nprint(\"\\nThe total number of patients detected with invasive cancer is \" + str(unique_invasive_patients))\nprint(\"This indicates that \" + str(round(unique_invasive_patients/total_patients*100,2)) + \"%\" + \" of the total \" + str(total_patients)+\" patients and \"+str(round(unique_invasive_patients/unique_cancer_patients*100,2))+\"% of the \"+str(unique_cancer_patients)+\" patients detected with cancer had an invasive cancer\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.50803Z","iopub.execute_input":"2023-01-02T17:40:27.508573Z","iopub.status.idle":"2023-01-02T17:40:27.518246Z","shell.execute_reply.started":"2023-01-02T17:40:27.508539Z","shell.execute_reply":"2023-01-02T17:40:27.516914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Patients with implants**","metadata":{}},{"cell_type":"code","source":"unique_implant_patients=cancer_patient['implant'].sum()\nprint(\"\\nThe total number of patients with implants is \" + str(unique_implant_patients))\nprint(\"This indicates that \" + str(round(unique_invasive_patients/total_patients*100,2)) + \"%\" + \" of the total \" + str(total_patients)+\" patients had implants\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.520147Z","iopub.execute_input":"2023-01-02T17:40:27.520938Z","iopub.status.idle":"2023-01-02T17:40:27.533634Z","shell.execute_reply.started":"2023-01-02T17:40:27.520881Z","shell.execute_reply":"2023-01-02T17:40:27.532363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"implant_patients_cancer=cancer_patient[(cancer_patient.cancer==1)&(cancer_patient.implant==1)]['implant'].sum()\nprint(\"\\nOut of the \"+str(unique_implant_patients)+\" patient with implants \"+str(implant_patients_cancer)+\" were diagnosed with cancer\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.539193Z","iopub.execute_input":"2023-01-02T17:40:27.539586Z","iopub.status.idle":"2023-01-02T17:40:27.549378Z","shell.execute_reply.started":"2023-01-02T17:40:27.539552Z","shell.execute_reply":"2023-01-02T17:40:27.547887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Patients with a difficult negative diagnosis**","metadata":{}},{"cell_type":"code","source":"unique_neg_patients=cancer_patient['difficult_negative_case'].sum()\nprint(\"\\nThe total number of patients with a difficult negative diagnosis was \" + str(unique_neg_patients))\nprint(\"This indicates that \" + str(round(unique_neg_patients/total_patients*100,2)) + \"%\" + \" of the total \" + str(total_patients)+\" patients had a difficult negative diagnosis\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.55097Z","iopub.execute_input":"2023-01-02T17:40:27.551378Z","iopub.status.idle":"2023-01-02T17:40:27.558904Z","shell.execute_reply.started":"2023-01-02T17:40:27.551342Z","shell.execute_reply":"2023-01-02T17:40:27.557701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n### Missing Treatment \n[Main Menu](#top) | [< prev](#profile2) | [next >](#profile4)\n","metadata":{}},{"cell_type":"markdown","source":"Let's also take a look at **patient age**\n\nWe know from the profile report that some age data is missing","metadata":{}},{"cell_type":"code","source":"print(\"\\n Age value is missing for \"+str(train.groupby(['patient_id'])['age'].mean().isna().sum()) +\" out of the \"+str(total_patients)+\" patients\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.560156Z","iopub.execute_input":"2023-01-02T17:40:27.56065Z","iopub.status.idle":"2023-01-02T17:40:27.573848Z","shell.execute_reply.started":"2023-01-02T17:40:27.560616Z","shell.execute_reply":"2023-01-02T17:40:27.572306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's now do a median imputation for the missing values. The same can be imputed at the image level in the train.csv data","metadata":{}},{"cell_type":"code","source":"patient_age_data=train.groupby(['patient_id'])['age'].mean().reset_index()\npatient_age_data['age'].fillna(patient_age_data['age'].median(), inplace=True)\ncancer_patient=cancer_patient.merge(patient_age_data, on='patient_id', how='inner')","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.57539Z","iopub.execute_input":"2023-01-02T17:40:27.575766Z","iopub.status.idle":"2023-01-02T17:40:27.594436Z","shell.execute_reply.started":"2023-01-02T17:40:27.575734Z","shell.execute_reply":"2023-01-02T17:40:27.592984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n#### **Distribution of Age**","metadata":{}},{"cell_type":"code","source":"# We can also look at the distributions of continuous variables and check for skew\ndef skew_cat(skew_val):\n if skew_val>0.5:\n return 'skewed towards the tail'\n elif skew_val<-.5:\n return 'skewed towards the head'\n else:\n return 'fairly symmetric'","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.596016Z","iopub.execute_input":"2023-01-02T17:40:27.596368Z","iopub.status.idle":"2023-01-02T17:40:27.602333Z","shell.execute_reply.started":"2023-01-02T17:40:27.596334Z","shell.execute_reply":"2023-01-02T17:40:27.601144Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(data=cancer_patient, x='age', fill=True, height=5, aspect=1.5)\nplt.show()\nprint(\"The skew of age is \"+str(cancer_patient.age.skew()) + \" which indicates that it is \" + skew_cat(cancer_patient.age.skew()))","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.603599Z","iopub.execute_input":"2023-01-02T17:40:27.603957Z","iopub.status.idle":"2023-01-02T17:40:27.923485Z","shell.execute_reply.started":"2023-01-02T17:40:27.603922Z","shell.execute_reply":"2023-01-02T17:40:27.922352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bins=[0,50,60,70,100]\nlabels=['1. 0-50 yrs', '2. 50-60 yrs', '3. 60-70 yrs', '4. 70+ years']\ncancer_patient['age_group'] = pd.cut(x = cancer_patient['age'], bins = bins, labels = labels, include_lowest = True)\n","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.925179Z","iopub.execute_input":"2023-01-02T17:40:27.925537Z","iopub.status.idle":"2023-01-02T17:40:27.935439Z","shell.execute_reply.started":"2023-01-02T17:40:27.925502Z","shell.execute_reply":"2023-01-02T17:40:27.933821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can now analyze the **age** variable for **propensity of cancer**, **invasive cancer** or a **biopsy recommendation**","metadata":{}},{"cell_type":"markdown","source":"---\n### Bi-variate Analysis \n[Main Menu](#top) | [< prev](#profile3) | [next >](#profile5)\n","metadata":{}},{"cell_type":"markdown","source":"#### Analysis of Age Variable \n[Main Menu](#top) | [< prev](#profile4) | [next >](#profile6)\n","metadata":{}},{"cell_type":"markdown","source":"#### Age and Cancer diagnosis","metadata":{}},{"cell_type":"code","source":"sns.displot(data=cancer_patient, x='age', hue='cancer', kind='kde', fill=True, height=5, aspect=1.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:27.93742Z","iopub.execute_input":"2023-01-02T17:40:27.938037Z","iopub.status.idle":"2023-01-02T17:40:28.369412Z","shell.execute_reply.started":"2023-01-02T17:40:27.937997Z","shell.execute_reply":"2023-01-02T17:40:28.368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is a slight skew towards higher age in the detection of cancer","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient, x='age_group', hue='cancer')\nplt.show()\ncan_by_age=cancer_patient.groupby(['age_group'])['cancer'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()\ncan_by_age.rename(columns={'cancer':'cancer rate'},inplace=True)\nsns.lineplot(data=can_by_age, x=\"age_group\", y=\"cancer rate\")\nplt.show()\n\nprint(can_by_age)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:28.370959Z","iopub.execute_input":"2023-01-02T17:40:28.371311Z","iopub.status.idle":"2023-01-02T17:40:28.684823Z","shell.execute_reply.started":"2023-01-02T17:40:28.371277Z","shell.execute_reply":"2023-01-02T17:40:28.683652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- We can observe that **age** has a ***significant positive correlation*** with the diagnosis of breast cancer.\n\n- The propensity of having cancer is **4.5 times higher in patients with age more than 70 years** when compared to those less than 50 years\n- The propensity **almost doubles in patients over 70 years** when compared to those between 60 and 70\n---","metadata":{}},{"cell_type":"markdown","source":"#### Age and Invasive Cancer diagnosis","metadata":{}},{"cell_type":"code","source":"sns.displot(data=cancer_patient[cancer_patient.cancer==1], x='age', hue='invasive', kind='kde', fill=True, height=5, aspect=1.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:28.686791Z","iopub.execute_input":"2023-01-02T17:40:28.687748Z","iopub.status.idle":"2023-01-02T17:40:28.992912Z","shell.execute_reply.started":"2023-01-02T17:40:28.6877Z","shell.execute_reply":"2023-01-02T17:40:28.99205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Age does not seem to have an effect on cancer being invasive\n\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Age and Biopsy recommendation","metadata":{}},{"cell_type":"code","source":"sns.displot(data=cancer_patient, x='age', hue='biopsy', kind='kde', fill=True, height=5, aspect=1.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:28.994548Z","iopub.execute_input":"2023-01-02T17:40:28.995668Z","iopub.status.idle":"2023-01-02T17:40:29.418746Z","shell.execute_reply.started":"2023-01-02T17:40:28.995621Z","shell.execute_reply":"2023-01-02T17:40:29.41748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Age also does not play a significant role with respect to the recommendations for biopsy\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Age and difficult negative cases","metadata":{}},{"cell_type":"code","source":"sns.displot(data=cancer_patient, x='age', hue='difficult_negative_case', kind='kde', fill=True, height=5, aspect=1.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:29.42033Z","iopub.execute_input":"2023-01-02T17:40:29.420784Z","iopub.status.idle":"2023-01-02T17:40:29.770005Z","shell.execute_reply.started":"2023-01-02T17:40:29.420738Z","shell.execute_reply":"2023-01-02T17:40:29.768864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Age of patients with a difficult negative case seems to be skewed towards younger patients","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient, x='age_group', hue='difficult_negative_case')\nplt.show()\nneg_by_age=cancer_patient.groupby(['age_group'])['difficult_negative_case'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()\nneg_by_age.rename(columns={'difficult_negative_case':'difficult negative case rate'},inplace=True)\nsns.lineplot(data=neg_by_age, x=\"age_group\", y=\"difficult negative case rate\")\nplt.show()\n\nprint(neg_by_age)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:29.771431Z","iopub.execute_input":"2023-01-02T17:40:29.771779Z","iopub.status.idle":"2023-01-02T17:40:30.069729Z","shell.execute_reply.started":"2023-01-02T17:40:29.771747Z","shell.execute_reply":"2023-01-02T17:40:30.068383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can observe that difficulty in a negative diagnosis is **more prevalent amongst younger patients**\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Other Bivariates","metadata":{}},{"cell_type":"markdown","source":"For the analysis of **site_id** and **machine_id**, we'll create a patient level dataset with site and machine information","metadata":{}},{"cell_type":"code","source":"cancer_patient_loc=train.groupby(['patient_id','site_id','machine_id'])['biopsy','cancer','implant','invasive','difficult_negative_case'].sum().reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:30.071514Z","iopub.execute_input":"2023-01-02T17:40:30.072321Z","iopub.status.idle":"2023-01-02T17:40:30.098134Z","shell.execute_reply.started":"2023-01-02T17:40:30.072272Z","shell.execute_reply":"2023-01-02T17:40:30.096968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#changing sum to indicators\ncancer_patient_loc['biopsy']=np.where(cancer_patient_loc['biopsy']>0,1,0)\ncancer_patient_loc['implant']=np.where(cancer_patient_loc['implant']>0,1,0)\ncancer_patient_loc['invasive']=np.where(cancer_patient_loc['invasive']>0,1,0)\ncancer_patient_loc['cancer']=np.where(cancer_patient_loc['cancer']>0,1,0)\ncancer_patient_loc['difficult_negative_case']=np.where(cancer_patient_loc['difficult_negative_case']>0,1,0)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:30.099875Z","iopub.execute_input":"2023-01-02T17:40:30.100611Z","iopub.status.idle":"2023-01-02T17:40:30.112775Z","shell.execute_reply.started":"2023-01-02T17:40:30.100562Z","shell.execute_reply":"2023-01-02T17:40:30.111759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cancer_patient_loc.shape","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:30.114269Z","iopub.execute_input":"2023-01-02T17:40:30.115338Z","iopub.status.idle":"2023-01-02T17:40:30.124365Z","shell.execute_reply.started":"2023-01-02T17:40:30.115301Z","shell.execute_reply":"2023-01-02T17:40:30.123073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\nThere is one patient with **multiple entries**. This means that atleast one patient has had the **diagnosis done on two separate machines**\n\n---","metadata":{}},{"cell_type":"code","source":"print(\"\\npatient_id with multiple entry\\n\")\nprint(cancer_patient_loc[cancer_patient_loc['patient_id'].duplicated()]['patient_id'])","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:30.126022Z","iopub.execute_input":"2023-01-02T17:40:30.126979Z","iopub.status.idle":"2023-01-02T17:40:30.138523Z","shell.execute_reply.started":"2023-01-02T17:40:30.126934Z","shell.execute_reply":"2023-01-02T17:40:30.137135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cancer_patient_loc[cancer_patient_loc['patient_id']==22637]","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:30.140488Z","iopub.execute_input":"2023-01-02T17:40:30.140916Z","iopub.status.idle":"2023-01-02T17:40:30.153891Z","shell.execute_reply.started":"2023-01-02T17:40:30.140848Z","shell.execute_reply":"2023-01-02T17:40:30.152822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can see that patient with id **222637** had scanning done at **machine 93** and **machine 190**.\n\n---","metadata":{}},{"cell_type":"markdown","source":"### Site Analysis \n[Main Menu](#top) | [< prev](#profile5) | [next >](#profile7)\n","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient_loc, x='site_id')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:30.160386Z","iopub.execute_input":"2023-01-02T17:40:30.161007Z","iopub.status.idle":"2023-01-02T17:40:30.338638Z","shell.execute_reply.started":"2023-01-02T17:40:30.16097Z","shell.execute_reply":"2023-01-02T17:40:30.337495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There's almost an **equal distribution** of patients between site 1 and site 2\n\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Site by Cancer diagnosis","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient_loc, x='site_id', hue='cancer')\nplt.show()\ncan_by_site=cancer_patient_loc.groupby(['site_id'])['cancer'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()\ncan_by_site.rename(columns={'cancer':'cancer rate'},inplace=True)\nsns.lineplot(data=can_by_site, x=\"site_id\", y=\"cancer rate\")\nplt.show()\n\nprint(can_by_site)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:30.340298Z","iopub.execute_input":"2023-01-02T17:40:30.340708Z","iopub.status.idle":"2023-01-02T17:40:30.756962Z","shell.execute_reply.started":"2023-01-02T17:40:30.340669Z","shell.execute_reply":"2023-01-02T17:40:30.755849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There **doesn't seem a significant difference** between the diagnosis rates of site 1 and site 2\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Site by biopsy","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient_loc, x='site_id', hue='biopsy')\nplt.show()\nbio_by_site=cancer_patient_loc.groupby(['site_id'])['biopsy'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()\nbio_by_site.rename(columns={'biopsy':'biopsy rate'},inplace=True)\nsns.lineplot(data=bio_by_site, x=\"site_id\", y=\"biopsy rate\")\nplt.show()\n\nprint(bio_by_site)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:30.758713Z","iopub.execute_input":"2023-01-02T17:40:30.759495Z","iopub.status.idle":"2023-01-02T17:40:31.18806Z","shell.execute_reply.started":"2023-01-02T17:40:30.759455Z","shell.execute_reply":"2023-01-02T17:40:31.186892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The biopsy recommendations rate at **site 1 is three times higher** than in site 2\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Site by difficult negative cases","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient_loc, x='site_id', hue='difficult_negative_case')\nplt.show()\nneg_by_site=cancer_patient_loc.groupby(['site_id'])['difficult_negative_case'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()\nneg_by_site.rename(columns={'difficult_negative_case':'difficult negative case rate'},inplace=True)\nsns.lineplot(data=neg_by_site, x=\"site_id\", y=\"difficult negative case rate\")\nplt.show()\n\nprint(neg_by_site)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:31.189721Z","iopub.execute_input":"2023-01-02T17:40:31.190642Z","iopub.status.idle":"2023-01-02T17:40:31.602175Z","shell.execute_reply.started":"2023-01-02T17:40:31.190605Z","shell.execute_reply":"2023-01-02T17:40:31.601159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The difficulty in diagnosis of negative cases is **higher in site 1** than site 2\n\n---","metadata":{}},{"cell_type":"markdown","source":"### Machine Analysis \n[Main Menu](#top) | [< prev](#profile6) | [next >](#profile8) ","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient_loc, x='machine_id', hue='site_id')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:31.603876Z","iopub.execute_input":"2023-01-02T17:40:31.60435Z","iopub.status.idle":"2023-01-02T17:40:31.880305Z","shell.execute_reply.started":"2023-01-02T17:40:31.604307Z","shell.execute_reply":"2023-01-02T17:40:31.879052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Machines most used are \n- #49 at site 1\n- #21 at site 2\n- #29 at site 2\n- #48 at site 2\n- other machines have low usage\n\nIt can be observed that while **site 2 has balanced load on all three machines** it has, **site 1 relies heavily on a single machine**\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Machine by Cancer Diagnosis","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient_loc, x='machine_id', hue='cancer')\nplt.show()\ncan_by_mac=cancer_patient_loc.groupby(['machine_id'])['cancer'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()\ncan_by_mac.rename(columns={'cancer':'cancer rate'},inplace=True)\nsns.lineplot(data=can_by_mac, x=\"machine_id\", y=\"cancer rate\")\nplt.show()\n\nprint(can_by_mac)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:31.882926Z","iopub.execute_input":"2023-01-02T17:40:31.883432Z","iopub.status.idle":"2023-01-02T17:40:32.376953Z","shell.execute_reply.started":"2023-01-02T17:40:31.883384Z","shell.execute_reply":"2023-01-02T17:40:32.375812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Amongst the four most used machines, patients diagnosed at machine #49 at site 1 have a higher cancer diagnosis rate\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Machine by biopsy recommendation","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient_loc, x='machine_id', hue='biopsy')\nplt.show()\ncan_by_mac=cancer_patient_loc.groupby(['machine_id'])['biopsy'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()\ncan_by_mac.rename(columns={'biopsy':'biopsy rate'},inplace=True)\nsns.lineplot(data=can_by_mac, x=\"machine_id\", y=\"biopsy rate\")\nplt.show()\n\nprint(can_by_mac)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:32.378541Z","iopub.execute_input":"2023-01-02T17:40:32.379184Z","iopub.status.idle":"2023-01-02T17:40:32.889772Z","shell.execute_reply.started":"2023-01-02T17:40:32.379149Z","shell.execute_reply":"2023-01-02T17:40:32.888363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Patients scanned at machine #49 at site 1 have a **1 in 5 chance** of being recommended for biopsy\n\nScans at other frequently used machines (21,29 & 48) have less than **1/4th** of that rate\n\n---","metadata":{}},{"cell_type":"markdown","source":"#### Machine by difficult negative cases","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=cancer_patient_loc, x='machine_id', hue='difficult_negative_case')\nplt.show()\ncan_by_mac=cancer_patient_loc.groupby(['machine_id'])['difficult_negative_case'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()\ncan_by_mac.rename(columns={'difficult_negative_case':'difficult_negative_case rate'},inplace=True)\nsns.lineplot(data=can_by_mac, x=\"machine_id\", y=\"difficult_negative_case rate\")\nplt.show()\n\nprint(can_by_mac)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:32.891348Z","iopub.execute_input":"2023-01-02T17:40:32.891688Z","iopub.status.idle":"2023-01-02T17:40:33.398838Z","shell.execute_reply.started":"2023-01-02T17:40:32.891658Z","shell.execute_reply":"2023-01-02T17:40:33.397496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The patients scanned by machine #49 have **50% (33% vs ~21%) higher chances in being difficult to diagnose negatively** compared to those scanned by machines 21,29 and 48\n\n---","metadata":{}},{"cell_type":"markdown","source":"### Bivariates with View \n\n[Main Menu](#top) | [< prev](#profile7) | [next >](#profile9)\n\nFor the analysis of **view**, we'll create a patient + view level dataset","metadata":{}},{"cell_type":"code","source":"cancer_patient_view=train.groupby(['patient_id','view'])['biopsy','cancer','implant','invasive','difficult_negative_case'].sum().reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:33.40068Z","iopub.execute_input":"2023-01-02T17:40:33.401212Z","iopub.status.idle":"2023-01-02T17:40:33.432892Z","shell.execute_reply.started":"2023-01-02T17:40:33.401168Z","shell.execute_reply":"2023-01-02T17:40:33.431615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#changing sum to indicators\ncancer_patient_view['biopsy']=np.where(cancer_patient_view['biopsy']>0,1,0)\ncancer_patient_view['implant']=np.where(cancer_patient_view['implant']>0,1,0)\ncancer_patient_view['invasive']=np.where(cancer_patient_view['invasive']>0,1,0)\ncancer_patient_view['cancer']=np.where(cancer_patient_view['cancer']>0,1,0)\ncancer_patient_view['difficult_negative_case']=np.where(cancer_patient_view['difficult_negative_case']>0,1,0)","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:33.434365Z","iopub.execute_input":"2023-01-02T17:40:33.434676Z","iopub.status.idle":"2023-01-02T17:40:33.445053Z","shell.execute_reply.started":"2023-01-02T17:40:33.434646Z","shell.execute_reply":"2023-01-02T17:40:33.443857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Patient distribution by view","metadata":{}},{"cell_type":"code","source":"cancer_patient_view.groupby(['view'])['patient_id'].count().reset_index()\nsns.countplot(data=cancer_patient_view, x='view')\nplt.show()\nprint(cancer_patient_view.groupby(['view'])['patient_id'].count().reset_index())","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:33.446522Z","iopub.execute_input":"2023-01-02T17:40:33.447142Z","iopub.status.idle":"2023-01-02T17:40:33.668345Z","shell.execute_reply.started":"2023-01-02T17:40:33.447106Z","shell.execute_reply":"2023-01-02T17:40:33.667152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**All patients** have be scanned with views **CC** and **MLO**. Other views are **negligible**\n\n*Patient level stats will remain the same for CC & MLO views since all patients have both the views*\n___","metadata":{}},{"cell_type":"code","source":"cancer_view=train.groupby(['view'])['biopsy','cancer','implant','invasive','difficult_negative_case'].apply(lambda x:round(x.sum()/x.count()*100,2)).reset_index()","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:33.669934Z","iopub.execute_input":"2023-01-02T17:40:33.670323Z","iopub.status.idle":"2023-01-02T17:40:33.704919Z","shell.execute_reply.started":"2023-01-02T17:40:33.670289Z","shell.execute_reply":"2023-01-02T17:40:33.703637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cancer_view[(cancer_view['view'] =='MLO') | (cancer_view['view']=='CC')]","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:33.706421Z","iopub.execute_input":"2023-01-02T17:40:33.706753Z","iopub.status.idle":"2023-01-02T17:40:33.723069Z","shell.execute_reply.started":"2023-01-02T17:40:33.706722Z","shell.execute_reply":"2023-01-02T17:40:33.721863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**All rates are similar** for both CC and MLO view\n\n---","metadata":{}},{"cell_type":"markdown","source":"### BIRADS & Density\n\n**BIRADS** and **Density** have **information missing in almost half of the cases**. \n\n[Main Menu](#top) | [< prev](#profile8) | [next >](#insights)\n","metadata":{}},{"cell_type":"code","source":"den_pres=train[train['density'].isna()==False]['patient_id'].nunique()\nprint(\"\\nDensity Information : Only \"+ str(den_pres)+ \"(\"+str(round(den_pres/total_patients*100,2))+\"%) patients have density information available\")\nden_pres=train[train['BIRADS'].isna()==False]['patient_id'].nunique()\nprint(\"\\nBIRADS Information : Only \"+ str(den_pres)+ \"(\"+str(round(den_pres/total_patients*100,2))+\"%) patients have BIRADS information available\")","metadata":{"execution":{"iopub.status.busy":"2023-01-02T17:40:33.725166Z","iopub.execute_input":"2023-01-02T17:40:33.725596Z","iopub.status.idle":"2023-01-02T17:40:33.746075Z","shell.execute_reply.started":"2023-01-02T17:40:33.725528Z","shell.execute_reply":"2023-01-02T17:40:33.74478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It does not make sense to do any analysis on them. It may lead to spurious insights\n\n---\n---","metadata":{}},{"cell_type":"markdown","source":"# 5. Insights \n[Main Menu](#top) | [< prev](#profile9) | [next >](#insights1)","metadata":{}},{"cell_type":"markdown","source":"#### There are three aspects in which we can draw our insight\n- **Data Insights** : The data being at image level warrants us to assess the process at an image level. These insights can lead to bettering of the diagnostic process\n- **Patient Level Insights** : The data might be at image level, the impact, however, should be assessed at the patient level. Patient level insights will help in improving the care logistics\n- **Impact of Age** : Age is a significant metric in the diagnosis and prevalence of cancer. Age is also important to understand from an elderly care perspective","metadata":{}},{"cell_type":"markdown","source":"### Data Insights \n[Main Menu](#top) | [< prev](#insights) | [next >](#insights2)\n\n- Training data is present at a scan **image level**. Each patient has a **minimum of four scan** images.\n- Close to **1 in 20 scans** warrant a biopsy.\n- About **2 in 5 biopsies** confirm the presence of breast cancer.\n- Close to **1 in 50 scans** are diagnosed as positive for cancer.\n- **70%** of all detected cancers are **invasive** which is concerning.\n- In about **15%** of the images, negative diagnosis is difficult\n\n\n\n---","metadata":{}},{"cell_type":"markdown","source":"### Patient Level Insights \n[Main Menu](#top) | [< prev](#insights1) | [next >](#insights3)\n\n- On average, each patient has **4.6 scan images**.\n- Almost **one in every ten** scanned patients are recommended a **biopsy**.\n- There is over a four in ten chance that a biopsy will confirm a cancer. In **more than five out ten cases**, the biopsy results are negative.\n- With every **one in four patients** the negative diagnosis becomes **difficult**.\n- Every **one in twenty-five** scanned patients are diagnosed with cancer.\n- **70%** of patients diagnosed with cancer, are diagnosed with an **invasive** one.\n\n\n\n---","metadata":{}},{"cell_type":"markdown","source":"### Age Related Insights \n[Main Menu](#top) | [< prev](#insights2) | [next >](#conclusion)\n\n- The chances of getting a positive cancer diagnosis for patients above the age of 70 are **4.5X** the chances for patients less than 50 years of age\n- The rate of cancer **almost doubles** in age group above 70 years when compared to the age group between 50 and 60 years\n\n\n\n---","metadata":{}},{"cell_type":"markdown","source":"# 6. Conclusion \n[Main Menu](#top) | [< prev](#insights3)\n\nEarly detection is key to successful treatment of breast cancer, and women are encouraged to get regular mammograms as part of their preventive healthcare. \n\nAll through the analysis, there were two themes that emerged -\n1. In more than half of the cases, the biopsies are not required.\n2. For almost 25% of the patients, though they are negative for cancer, this diagnosis becomes difficult.\n\nCancer is a deadly disease and to wait for a diagnosis result can be excruciating. A model that will predict cancer with a higher degree of accuracy will ease both the patients' and the radiologists' stress. \n\nThis analysis is, in no way, a big step towards building the model. I will continue the analysis of the scans and through computer vision techniques be able to build a good model that will help the society.\n\nIf you have reached here, I thank you for your patience. Please share your feedback.\n\n---","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}