{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"As part of this notebook, we are trying to detect breast cancer using screening images.\n\nThe metadata we have has the following columns:\n\nsite_id - ID code for the source hospital.\n\npatient_id - ID code for the patient.\n\nimage_id - ID code for the image.\n\nlaterality - Whether the image is of the left or right breast.\n\nview - The orientation of the image. The default for a screening exam is to capture two views per breast.\n\nage - The patient's age in years.\n\nimplant - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.\n\ndensity - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult. Only provided for train.\n\nmachine_id - An ID code for the imaging device.\n\ncancer - Whether or not the breast was positive for cancer. The target value. Only provided for train.\n\nbiopsy - Whether or not a follow-up biopsy was performed on the breast. Only provided for train.\n\ninvasive - If the breast is positive for cancer, whether or not the cancer proved to be invasive. Only provided for train.\n\nBIRADS - 0 if the breast required follow-up, 1 if the breast was rated as negative for cancer, and 2 if the breast was rated as normal. Only provided for train.\nprediction_id - The ID for the matching submission row. Multiple images will share the same prediction ID. Test only.\n\ndifficult_negative_case - True if the case was unusually difficult. Only provided for train.","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport pydicom as dicom\nimport random\n!pip install pylibjpeg pylibjpeg-libjpeg pydicom\nimport pylibjpeg\nplt.style.use('fivethirtyeight')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-12-13T04:47:19.955715Z","iopub.execute_input":"2022-12-13T04:47:19.956162Z","iopub.status.idle":"2022-12-13T04:47:33.7859Z","shell.execute_reply.started":"2022-12-13T04:47:19.956125Z","shell.execute_reply":"2022-12-13T04:47:33.784619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\")\ntest_metadata = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/test.csv\")\nsample_sub = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:33.363387Z","iopub.execute_input":"2022-12-13T04:06:33.363727Z","iopub.status.idle":"2022-12-13T04:06:33.504002Z","shell.execute_reply.started":"2022-12-13T04:06:33.363697Z","shell.execute_reply":"2022-12-13T04:06:33.502606Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.columns","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:33.505559Z","iopub.execute_input":"2022-12-13T04:06:33.506015Z","iopub.status.idle":"2022-12-13T04:06:33.51663Z","shell.execute_reply.started":"2022-12-13T04:06:33.505971Z","shell.execute_reply":"2022-12-13T04:06:33.515383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.describe()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:33.520467Z","iopub.execute_input":"2022-12-13T04:06:33.520967Z","iopub.status.idle":"2022-12-13T04:06:33.594525Z","shell.execute_reply.started":"2022-12-13T04:06:33.520891Z","shell.execute_reply":"2022-12-13T04:06:33.593297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Insights:\n\nWe have data for women from 26-89 years of age.\n\ncancer(target column), biopsy, invasive, BIRADS, density and implant are categorical columns.","metadata":{}},{"cell_type":"code","source":"plt.title('Target distribution in train')\nsns.countplot(x=train_metadata.cancer)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:33.596254Z","iopub.execute_input":"2022-12-13T04:06:33.596674Z","iopub.status.idle":"2022-12-13T04:06:33.822941Z","shell.execute_reply.started":"2022-12-13T04:06:33.596638Z","shell.execute_reply":"2022-12-13T04:06:33.821793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Insight:\n\nData is highly imbalanced. This would need to be accounted while modelling.","metadata":{}},{"cell_type":"code","source":"plt.title('Relation of the age of a patient with the target')\nsns.scatterplot(x=train_metadata.cancer,y=train_metadata.age)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:33.824056Z","iopub.execute_input":"2022-12-13T04:06:33.824905Z","iopub.status.idle":"2022-12-13T04:06:34.170458Z","shell.execute_reply.started":"2022-12-13T04:06:33.824869Z","shell.execute_reply":"2022-12-13T04:06:34.169265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.title('Distribution of age of patients')\nsns.countplot(x=train_metadata.age)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:34.171977Z","iopub.execute_input":"2022-12-13T04:06:34.17374Z","iopub.status.idle":"2022-12-13T04:06:34.961208Z","shell.execute_reply.started":"2022-12-13T04:06:34.173662Z","shell.execute_reply":"2022-12-13T04:06:34.959589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Insight:\n\nLooks like women above the age of 35 are more probable to be affected by breast cancer.\n\nThis could also be because we have a smaller sample of women below 35 in our train data.","metadata":{}},{"cell_type":"code","source":"train_metadata.laterality.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:34.964878Z","iopub.execute_input":"2022-12-13T04:06:34.96528Z","iopub.status.idle":"2022-12-13T04:06:34.977943Z","shell.execute_reply.started":"2022-12-13T04:06:34.965242Z","shell.execute_reply":"2022-12-13T04:06:34.976695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_metadata.view.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:34.980274Z","iopub.execute_input":"2022-12-13T04:06:34.981109Z","iopub.status.idle":"2022-12-13T04:06:34.99523Z","shell.execute_reply.started":"2022-12-13T04:06:34.981071Z","shell.execute_reply":"2022-12-13T04:06:34.993979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.title('Relation of the feature density with the target')\nsns.countplot(hue=train_metadata.cancer,x=train_metadata.density)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:34.998024Z","iopub.execute_input":"2022-12-13T04:06:34.998389Z","iopub.status.idle":"2022-12-13T04:06:35.267625Z","shell.execute_reply.started":"2022-12-13T04:06:34.998357Z","shell.execute_reply":"2022-12-13T04:06:35.266293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Insights:\n\nThe data distribution of field density is imbalanced, we have more samples with density B and C than with A and D.","metadata":{}},{"cell_type":"code","source":"plt.title('Does having an implant contribute towards breast cancer?')\nsns.countplot(hue=train_metadata.cancer,x=train_metadata.implant)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:35.269214Z","iopub.execute_input":"2022-12-13T04:06:35.269596Z","iopub.status.idle":"2022-12-13T04:06:35.496472Z","shell.execute_reply.started":"2022-12-13T04:06:35.269564Z","shell.execute_reply":"2022-12-13T04:06:35.495328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the data we have do not have implants.","metadata":{}},{"cell_type":"code","source":"corr = train_metadata.corr()\nmask = np.triu(np.ones_like(corr, dtype=bool))\nplt.figure(figsize=(11, 9))\nsns.heatmap(corr,mask=mask,annot=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:06:35.498257Z","iopub.execute_input":"2022-12-13T04:06:35.499047Z","iopub.status.idle":"2022-12-13T04:06:36.140554Z","shell.execute_reply.started":"2022-12-13T04:06:35.499002Z","shell.execute_reply":"2022-12-13T04:06:36.139563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Both biopsy and invasive have a high correlation with the target.\n\nBut these features are only present in train.","metadata":{}},{"cell_type":"markdown","source":"# Viewing Images","metadata":{}},{"cell_type":"markdown","source":"I have written the below function which uses the random library from python to pick a random number whose corresponding index images would be printed.\n\nFor reading and displaying the dicom images I have used the pydicom library\n\nThe os library is used in the below function to work around with the paths of images to be displayed.","metadata":{}},{"cell_type":"code","source":"def plot_images(pid):\n    path = \"/kaggle/input/rsna-breast-cancer-detection/train_images\"\n    \n    patient_id = train_metadata.patient_id[pid]\n    is_cancer = train_metadata.cancer[pid]\n    lat_list = list(train_metadata.query('patient_id==patient_id').laterality)\n    view_list = list(train_metadata.query('patient_id==patient_id').view)\n    path = os.path.join(path,str(patient_id))\n    x = os.listdir(path)\n    count = 1\n    fig = plt.figure(figsize=(10, 7))\n    plt.title('The scan images are of target type, {}'.format(is_cancer))\n    for i in x:\n        \n        final_path = os.path.join(path,str(i))\n        fig.add_subplot(2,2,count)\n        ds = dicom.dcmread(final_path)\n        plt.imshow(ds.pixel_array)\n        plt.xlabel('lat: '+lat_list[count]+' view:'+view_list[count])\n        \n        count+=1\n\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:52:50.147442Z","iopub.execute_input":"2022-12-13T04:52:50.14784Z","iopub.status.idle":"2022-12-13T04:52:50.157999Z","shell.execute_reply.started":"2022-12-13T04:52:50.147809Z","shell.execute_reply":"2022-12-13T04:52:50.156592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_images(10025)","metadata":{"execution":{"iopub.status.busy":"2022-12-13T04:53:01.425869Z","iopub.execute_input":"2022-12-13T04:53:01.426277Z","iopub.status.idle":"2022-12-13T04:53:17.5558Z","shell.execute_reply.started":"2022-12-13T04:53:01.426245Z","shell.execute_reply":"2022-12-13T04:53:17.554648Z"},"trusted":true},"execution_count":null,"outputs":[]}]}