{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Context\n\"According to the WHO, breast cancer is the most commonly occurring cancer worldwide. In 2020 alone, there were 2.3 million new breast cancer diagnoses and 685,000 deaths. Yet breast cancer mortality in high-income countries has dropped by 40% since the 1980s when health authorities implemented regular mammography screening in age groups considered at risk. Early detection and treatment are critical to reducing cancer fatalities, and your machine learning skills could help streamline the process radiologists use to evaluate screening mammograms.\n\n\"Currently, early detection of breast cancer requires the expertise of highly-trained human observers, making screening mammography programs expensive to conduct. A looming shortage of radiologists in several countries will likely worsen this problem. Mammography screening also leads to a high incidence of false positive results. This can result in unnecessary anxiety, inconvenient follow-up care, extra imaging tests, and sometimes a need for tissue sampling (often a needle biopsy).\n\n\"The competition host, the Radiological Society of North America (RSNA) is a non-profit organization that represents 31 radiologic subspecialties from 145 countries around the world. RSNA promotes excellence in patient care and health care delivery through education, research, and technological innovation.\n\n\"Your efforts in this competition could help extend the benefits of early detection to a broader population. Greater access could further reduce breast cancer mortality worldwide.\"\n\n### Goal of the Competition\nThe goal of this competition is to identify breast cancer. You'll train your model with screening mammograms obtained from regular screening.\n\nYour work improving the automation of detection in screening mammography may enable radiologists to be more accurate and efficient, improving the quality and safety of patient care. It could also help reduce costs and unnecessary medical procedures.","metadata":{}},{"cell_type":"markdown","source":"### Data Dictionary\n\nThe colum description for the train and test datset is given below.\n<ul>\n<li><code>site_id</code> - ID code for the source hospital.</li>\n<li><code>patient_id</code> - ID code for the patient.</li>\n<li><code>image_id</code> - ID code for the image.</li>\n<li><code>laterality</code> - Whether the image is of the left or right breast.</li>\n<li><code>view</code> - The orientation of the image. The default for a screening exam is to capture two views per breast.</li>\n<li><code>age</code> - The patient's age in years.</li>\n<li><code>implant</code> - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.</li>\n<li><code>density</code> - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult.</li>\n<li><code>machine_id</code> - An ID code for the imaging device.</li>\n<li><code>cancer</code> - The target value. Only provided for train.</li>\n<li><code>biopsy</code> - Whether or not a follow-up biopsy was performed on the breast. Only provided for train.</li>\n<li><code>invasive</code> - If the breast is positive for cancer, whether or not the cancer proved to be invasive. Only provided for train.</li>\n<li><code>BIRADS</code> - 0 if the breast required follow-up, 1 if the breast was rated as negative for cancer, and 2 if the breast was rated as normal. Only provided for train.</li>\n<li><code>prediction_id</code> - The ID for the matching submission row. Multiple images will share the same prediction ID. Test only.</li>\n<li><code>difficult_negative_case</code> - True if the case was unusually difficult. Only provided for train.</li>\n</ul>","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pylab as plt\nimport pydicom\nfrom pydicom import dcmread\nfrom pathlib import Path","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-11-29T15:39:21.788975Z","iopub.execute_input":"2022-11-29T15:39:21.789436Z","iopub.status.idle":"2022-11-29T15:39:21.795234Z","shell.execute_reply.started":"2022-11-29T15:39:21.789399Z","shell.execute_reply":"2022-11-29T15:39:21.794133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\")\ntest_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:05:03.527995Z","iopub.execute_input":"2022-11-29T15:05:03.529028Z","iopub.status.idle":"2022-11-29T15:05:03.694192Z","shell.execute_reply.started":"2022-11-29T15:05:03.528977Z","shell.execute_reply":"2022-11-29T15:05:03.692971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.shape","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:05:03.695643Z","iopub.execute_input":"2022-11-29T15:05:03.696004Z","iopub.status.idle":"2022-11-29T15:05:03.705854Z","shell.execute_reply.started":"2022-11-29T15:05:03.695971Z","shell.execute_reply":"2022-11-29T15:05:03.70459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:05:03.710034Z","iopub.execute_input":"2022-11-29T15:05:03.710797Z","iopub.status.idle":"2022-11-29T15:05:03.738492Z","shell.execute_reply.started":"2022-11-29T15:05:03.710755Z","shell.execute_reply":"2022-11-29T15:05:03.737378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in train_df.columns:\n    print(col + \":\", train_df[col].nunique())","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:05:03.857499Z","iopub.execute_input":"2022-11-29T15:05:03.858117Z","iopub.status.idle":"2022-11-29T15:05:03.894553Z","shell.execute_reply.started":"2022-11-29T15:05:03.858068Z","shell.execute_reply":"2022-11-29T15:05:03.89354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.columns","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:05:04.153917Z","iopub.execute_input":"2022-11-29T15:05:04.154367Z","iopub.status.idle":"2022-11-29T15:05:04.161343Z","shell.execute_reply.started":"2022-11-29T15:05:04.154336Z","shell.execute_reply":"2022-11-29T15:05:04.160205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Features distribution","metadata":{}},{"cell_type":"code","source":"cols_to_plot = ['site_id', 'laterality', 'view', 'biopsy', 'invasive', 'BIRADS', 'implant', 'density','machine_id', 'difficult_negative_case']","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:05:04.431707Z","iopub.execute_input":"2022-11-29T15:05:04.432621Z","iopub.status.idle":"2022-11-29T15:05:04.43788Z","shell.execute_reply.started":"2022-11-29T15:05:04.432581Z","shell.execute_reply":"2022-11-29T15:05:04.436553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pal= ['#3FFEBA', '#F50EF5']\nfig, ax = plt.subplots(4, 3,figsize=(18, 16))\nfig.subplots_adjust(top=0.94)\nitr = 1\nfor col in cols_to_plot:\n    plt.subplot(4, 3, itr)\n    sns.countplot(x=train_df[col], hue=train_df['cancer'], palette=pal)\n    itr += 1\nplt.suptitle('Distibution of features', fontsize=20);\nplt.show();","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:32:58.286618Z","iopub.execute_input":"2022-11-29T15:32:58.287134Z","iopub.status.idle":"2022-11-29T15:33:00.05348Z","shell.execute_reply.started":"2022-11-29T15:32:58.287064Z","shell.execute_reply":"2022-11-29T15:33:00.05249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Age distribution","metadata":{}},{"cell_type":"code","source":"sns.kdeplot(data=train_df, x=\"age\", hue='cancer', palette=pal);","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:33:05.619018Z","iopub.execute_input":"2022-11-29T15:33:05.619745Z","iopub.status.idle":"2022-11-29T15:33:06.156001Z","shell.execute_reply.started":"2022-11-29T15:33:05.61971Z","shell.execute_reply":"2022-11-29T15:33:06.154572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_df[(train_df['cancer'] == 1) & (train_df['biopsy'] == 1)])","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:33:06.157896Z","iopub.execute_input":"2022-11-29T15:33:06.158288Z","iopub.status.idle":"2022-11-29T15:33:06.170105Z","shell.execute_reply.started":"2022-11-29T15:33:06.158256Z","shell.execute_reply":"2022-11-29T15:33:06.168742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Few observation in the train_dataset\n- No cancerous samples when `difficult_negative_case = 1` > (the case was unusually difficult)\n- No cancerous samples when `biopsy = 0`\n- 13 cancerous samples when `implant = 1`\n- All of the cancerous samples are from `BIRADS=0` > (breast required follow-up)","metadata":{}},{"cell_type":"markdown","source":"### Target Distribustion\n- The traget is highly imbalanced.","metadata":{}},{"cell_type":"code","source":"sns.countplot(x=train_df[\"cancer\"], palette=pal)\nplt.suptitle('Target distribution', fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:33:11.406241Z","iopub.execute_input":"2022-11-29T15:33:11.406733Z","iopub.status.idle":"2022-11-29T15:33:11.602829Z","shell.execute_reply.started":"2022-11-29T15:33:11.406689Z","shell.execute_reply":"2022-11-29T15:33:11.601785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Select one image `without` cancer and display","metadata":{}},{"cell_type":"code","source":"train_df[train_df['cancer'] == 0].head()","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:36:05.467688Z","iopub.execute_input":"2022-11-29T15:36:05.468229Z","iopub.status.idle":"2022-11-29T15:36:05.49546Z","shell.execute_reply.started":"2022-11-29T15:36:05.468187Z","shell.execute_reply":"2022-11-29T15:36:05.493894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patient_id = 10006\n\npath = '/kaggle/input/rsna-breast-cancer-detection/train_images/'\n\nfig, ax = plt.subplots(2, 2, figsize=(16,10))\nax = ax.flatten()\nfor i, img_path in enumerate(list(Path(path+f'{patient_id}').iterdir())):\n    ds = pydicom.dcmread(img_path)\n    arr = ds.pixel_array\n    ax[i].imshow(arr, cmap=\"gray\")\nplt.suptitle(\"Images from patient ID \" + f'{patient_id}' + \" (Without cancer!)\", fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:30:47.68042Z","iopub.execute_input":"2022-11-29T15:30:47.680782Z","iopub.status.idle":"2022-11-29T15:31:01.266683Z","shell.execute_reply.started":"2022-11-29T15:30:47.680749Z","shell.execute_reply":"2022-11-29T15:31:01.265329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Select one image `with` cancer and display","metadata":{}},{"cell_type":"code","source":"train_df[train_df['cancer'] == 1].head()","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:36:13.2477Z","iopub.execute_input":"2022-11-29T15:36:13.248109Z","iopub.status.idle":"2022-11-29T15:36:13.275062Z","shell.execute_reply.started":"2022-11-29T15:36:13.248062Z","shell.execute_reply":"2022-11-29T15:36:13.273524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patient_id = 9840\npath = '/kaggle/input/rsna-breast-cancer-detection/train_images/'\nfig, ax = plt.subplots(2, 2, figsize=(16,10))\nax = ax.flatten()\nfor i, img_path in enumerate(list(Path(path+f'{patient_id}').iterdir())):\n    ds = pydicom.dcmread(img_path)\n    arr = ds.pixel_array\n    ax[i].imshow(arr, cmap=\"gray\")\nplt.suptitle(\"Images from patient ID \" + f'{patient_id}' + \" (With cancer!)\", fontsize=20);","metadata":{"execution":{"iopub.status.busy":"2022-11-29T15:29:45.495721Z","iopub.execute_input":"2022-11-29T15:29:45.496236Z","iopub.status.idle":"2022-11-29T15:29:51.838715Z","shell.execute_reply.started":"2022-11-29T15:29:45.496196Z","shell.execute_reply":"2022-11-29T15:29:51.837597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reference\n- https://www.kaggle.com/competitions/rsna-breast-cancer-detection/overview\n\nTBC...","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}