Introduction
","metadata":{}},{"cell_type":"markdown","source":"Hi, I'm Vadym and this is my EDA for RSNA Screening Mammographies Competition. I did this notebook for myself a few months ago and now I decided to make it public. Hope, here you will find something interesting for you too😊","metadata":{}},{"cell_type":"markdown","source":"\n##Load libraries
","metadata":{}},{"cell_type":"markdown","source":"Download this package to be able to plot mammograms","metadata":{}},{"cell_type":"code","source":"!pip install pylibjpeg\n!pip install python-gdcm\n!pip install plotly==5.11.0","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-08T23:43:34.330685Z","iopub.execute_input":"2023-02-08T23:43:34.331126Z","iopub.status.idle":"2023-02-08T23:44:48.599863Z","shell.execute_reply.started":"2023-02-08T23:43:34.33104Z","shell.execute_reply":"2023-02-08T23:44:48.598098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nfrom collections import Counter\nfrom pathlib import Path\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nfrom mpl_toolkits.axes_grid1 import ImageGrid\nimport pydicom\nimport pylibjpeg\n\nsns.set_style(\"darkgrid\")","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:48.602364Z","iopub.execute_input":"2023-02-08T23:44:48.602791Z","iopub.status.idle":"2023-02-08T23:44:50.457039Z","shell.execute_reply.started":"2023-02-08T23:44:48.60275Z","shell.execute_reply":"2023-02-08T23:44:50.455615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##Discover data
","metadata":{}},{"cell_type":"markdown","source":"Let's load our tabular data first","metadata":{}},{"cell_type":"code","source":"train_images = Path(\"/kaggle/input/rsna-breast-cancer-detection/train_images\")\ntest_images = Path(\"/kaggle/input/rsna-breast-cancer-detection/test_images\")","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:50.458812Z","iopub.execute_input":"2023-02-08T23:44:50.459158Z","iopub.status.idle":"2023-02-08T23:44:50.464643Z","shell.execute_reply.started":"2023-02-08T23:44:50.459126Z","shell.execute_reply":"2023-02-08T23:44:50.463012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\")\ntest_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/test.csv\")\nsample_submission_df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/sample_submission.csv\")","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:50.466971Z","iopub.execute_input":"2023-02-08T23:44:50.467368Z","iopub.status.idle":"2023-02-08T23:44:50.614665Z","shell.execute_reply.started":"2023-02-08T23:44:50.467321Z","shell.execute_reply":"2023-02-08T23:44:50.613521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:50.616159Z","iopub.execute_input":"2023-02-08T23:44:50.61739Z","iopub.status.idle":"2023-02-08T23:44:50.651524Z","shell.execute_reply.started":"2023-02-08T23:44:50.617343Z","shell.execute_reply":"2023-02-08T23:44:50.650323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:50.653174Z","iopub.execute_input":"2023-02-08T23:44:50.653513Z","iopub.status.idle":"2023-02-08T23:44:50.666403Z","shell.execute_reply.started":"2023-02-08T23:44:50.653485Z","shell.execute_reply":"2023-02-08T23:44:50.665301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission_df","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:50.668315Z","iopub.execute_input":"2023-02-08T23:44:50.668761Z","iopub.status.idle":"2023-02-08T23:44:50.679395Z","shell.execute_reply.started":"2023-02-08T23:44:50.668719Z","shell.execute_reply":"2023-02-08T23:44:50.677994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, here we have the next features:\n- site_id - ID code for the source hospital\n \n- patient_id - ID code for the patient.\n \n- image_id - ID code for the image.\n \n- laterality - Whether the image is of the left or right breast.\n \n- view - The orientation of the image. The default for a screening exam is to capture two views per breast.\n \n- age - The patient's age in years.\n \n- implant - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.\n \n- density - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult. **Only provided for train.**\n \n- machine_id - An ID code for the imaging device.\n \n- cancer - Whether or not the breast was positive for cancer. **Only provided for train.**\n \n- biopsy - Whether or not a follow-up biopsy was performed on the breast. **Only provided for train.**\n \n- invasive - If the breast is positive for cancer, whether or not the cancer proved to be invasive. **Only provided for train.**\n \n- BIRADS - 0 if the breast required follow-up, 1 if the breast was rated as negative for cancer, and 2 if the breast was rated as normal. **Only provided for train.**\n \n- prediction_id - The ID for the matching submission row. Multiple images will share the same prediction ID. **Only provided for train.**\n \n- difficult_negative_case - True if the case was unusually difficult. **Only provided for train.**","metadata":{}},{"cell_type":"markdown","source":"In submission, we have to give the probability of cancer for a particular breast of the patient.","metadata":{}},{"cell_type":"markdown","source":"Given this data, I have some questions. I'll try to answer them in this EDA.\n\n - What distribution of images has patients?\n - Why do we have density, biopsy, invasive, and BIRADS features just in training data, because it's not gonna be used for prediction?\n - Do all patients have both images of the left and right breast? (laterality)\n - Do all patients have images of different views of all breasts?\n - What distribution of age is in this dataset?\n - Does higher age mean more cancer cases?\n - Does the presence of a breast implant increase the risk of cancer?","metadata":{}},{"cell_type":"markdown","source":"So, seems like in the training data we have more features, than in the test. Let's discover them.","metadata":{}},{"cell_type":"markdown","source":"\n##Number of images per patient
","metadata":{}},{"cell_type":"code","source":"images_counter = train_df[\"patient_id\"].value_counts().sort_index()\n\nfig = px.histogram(images_counter, text_auto=True, title=\"Number of images per patient\")\nfig.update_layout(bargap=0.2)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:50.680761Z","iopub.execute_input":"2023-02-08T23:44:50.681072Z","iopub.status.idle":"2023-02-08T23:44:51.122835Z","shell.execute_reply.started":"2023-02-08T23:44:50.681044Z","shell.execute_reply":"2023-02-08T23:44:51.121193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the patients have 4 images. I have a guess, that 4 images mostly have patients without problems. Let's figure that out.","metadata":{}},{"cell_type":"markdown","source":"\n##Correlation between the number of images and cancer
","metadata":{}},{"cell_type":"markdown","source":"Each person has multiple images of different breasts, and it's possible, that cancer is detected in only one. So we need to know if on any image of the person, cancer is detected","metadata":{}},{"cell_type":"code","source":"is_cancer_person = train_df.groupby(\"patient_id\")['cancer'].max().sort_index()","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:51.124652Z","iopub.execute_input":"2023-02-08T23:44:51.125125Z","iopub.status.idle":"2023-02-08T23:44:51.140238Z","shell.execute_reply.started":"2023-02-08T23:44:51.125064Z","shell.execute_reply":"2023-02-08T23:44:51.138765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's check out, is indices in 'is_cancer_person' and 'images_counter' have the same indices order","metadata":{}},{"cell_type":"code","source":"images_counter.index.tolist() == is_cancer_person.index.tolist()","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:51.1457Z","iopub.execute_input":"2023-02-08T23:44:51.146125Z","iopub.status.idle":"2023-02-08T23:44:51.191347Z","shell.execute_reply.started":"2023-02-08T23:44:51.146068Z","shell.execute_reply":"2023-02-08T23:44:51.189972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can count the correlation coefficient","metadata":{}},{"cell_type":"code","source":"np.corrcoef(images_counter.values, is_cancer_person.values)","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:51.192604Z","iopub.execute_input":"2023-02-08T23:44:51.192895Z","iopub.status.idle":"2023-02-08T23:44:51.270358Z","shell.execute_reply.started":"2023-02-08T23:44:51.192868Z","shell.execute_reply":"2023-02-08T23:44:51.269126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, there is no correlation🙃","metadata":{}},{"cell_type":"markdown","source":"\n##Distribution of patients' age
","metadata":{}},{"cell_type":"code","source":"person_age = train_df.groupby(\"patient_id\")['age'].max().sort_index().fillna(0).astype('int64')\n\nfig = px.histogram(person_age, title=\"Distribution of patients' age\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:44:51.271679Z","iopub.execute_input":"2023-02-08T23:44:51.272581Z","iopub.status.idle":"2023-02-08T23:44:51.370806Z","shell.execute_reply.started":"2023-02-08T23:44:51.272549Z","shell.execute_reply":"2023-02-08T23:44:51.369751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From this plot, I see, that at age of 40, more women go to mammography, and after the age of 68 begins a noticeable decrease in patients. Maybe, fewer people stay alive at this age.","metadata":{}},{"cell_type":"markdown","source":"\n##Correlation matrix
","metadata":{}},{"cell_type":"code","source":"def plot_correlation_heatmap(df):\n df = pd.get_dummies(df, columns=['laterality', 'view', 'density'])\n df['difficult_negative_case'] = df['difficult_negative_case'].astype(int)\n\n corr = df.corr()\n mask = np.triu(np.ones_like(corr, dtype=bool))\n fig, ax = plt.subplots(figsize=(11, 9))\n cmap = sns.diverging_palette(230, 20, as_cmap=True)\n sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0, square=True, linewidths=.5, cbar_kws={\"shrink\": .5})\n","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:45:05.288929Z","iopub.execute_input":"2023-02-08T23:45:05.289367Z","iopub.status.idle":"2023-02-08T23:45:05.297708Z","shell.execute_reply.started":"2023-02-08T23:45:05.289333Z","shell.execute_reply":"2023-02-08T23:45:05.295361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_correlation_heatmap(train_df)","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:45:05.408571Z","iopub.execute_input":"2023-02-08T23:45:05.409128Z","iopub.status.idle":"2023-02-08T23:45:06.322101Z","shell.execute_reply.started":"2023-02-08T23:45:05.409097Z","shell.execute_reply":"2023-02-08T23:45:06.320736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n##Plot images
","metadata":{}},{"cell_type":"markdown","source":"Now let's load sample image in dicom format","metadata":{}},{"cell_type":"code","source":"sample_image = \"/kaggle/input/rsna-breast-cancer-detection/train_images/10006/1459541791.dcm\"\npydicom.dcmread(sample_image)","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:45:16.376338Z","iopub.execute_input":"2023-02-08T23:45:16.376725Z","iopub.status.idle":"2023-02-08T23:45:16.427227Z","shell.execute_reply.started":"2023-02-08T23:45:16.376694Z","shell.execute_reply":"2023-02-08T23:45:16.426479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And now I can create function for plotting images of some patients","metadata":{}},{"cell_type":"code","source":"test_df[\"patient_id\"]","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:45:16.871452Z","iopub.execute_input":"2023-02-08T23:45:16.871851Z","iopub.status.idle":"2023-02-08T23:45:16.879954Z","shell.execute_reply.started":"2023-02-08T23:45:16.871821Z","shell.execute_reply":"2023-02-08T23:45:16.878891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_images_of_patient(patient_id):\n patient_df = test_df[test_df[\"patient_id\"] == patient_id] if patient_id in test_df[\"patient_id\"].values else train_df[train_df[\"patient_id\"] == patient_id] if patient_id in train_df[\"patient_id\"].values else None\n if patient_df is None:\n print(f\"No patient with id {patient_id}\")\n return\n \n patient_folder = test_images / f\"{patient_id}\" if patient_id in test_df[\"patient_id\"].values else train_images / f\"{patient_id}\"\n num_images = patient_df.shape[0]\n n_rows, n_cols = (2, 2) if num_images <= 4 else (4, 4) if num_images <= 8 else (8, 8)\n \n fig, ax = plt.subplots(figsize=(24., 24.), nrows=n_rows, ncols=n_cols, gridspec_kw={'wspace': 0.6, 'hspace': 0.6})\n fig.suptitle(f\"{patient_id} - {num_images} images\", fontsize=16)\n \n for idx, curr_image in enumerate(patient_folder.iterdir()):\n curr_image = Path(curr_image)\n image_data = patient_df[patient_df[\"image_id\"] == int(curr_image.stem)].squeeze()\n ds = pydicom.dcmread(curr_image)\n image_as_np = ds.pixel_array.astype(np.float32)\n curr_ax = ax[idx // n_cols, idx % n_cols]\n curr_ax.imshow(image_as_np)\n curr_ax.set_title(\"image_id: {}\\n laterality: {}\\n view {}\\n machine_id {}\".format(image_data[\"image_id\"], image_data[\"laterality\"], image_data[\"view\"], image_data[\"machine_id\"]), fontdict={\"fontsize\": 9})\n","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:45:17.210825Z","iopub.execute_input":"2023-02-08T23:45:17.211227Z","iopub.status.idle":"2023-02-08T23:45:17.22261Z","shell.execute_reply.started":"2023-02-08T23:45:17.211196Z","shell.execute_reply":"2023-02-08T23:45:17.221427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And here I can visualize images of few patients","metadata":{}},{"cell_type":"code","source":"plot_images_of_patient(32751)","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:45:17.91971Z","iopub.execute_input":"2023-02-08T23:45:17.920101Z","iopub.status.idle":"2023-02-08T23:45:25.315135Z","shell.execute_reply.started":"2023-02-08T23:45:17.920071Z","shell.execute_reply":"2023-02-08T23:45:25.314325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_images_of_patient(2518)","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:45:25.316487Z","iopub.execute_input":"2023-02-08T23:45:25.317003Z","iopub.status.idle":"2023-02-08T23:45:47.48074Z","shell.execute_reply.started":"2023-02-08T23:45:25.316973Z","shell.execute_reply":"2023-02-08T23:45:47.47953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_images_of_patient(10179)","metadata":{"execution":{"iopub.status.busy":"2023-02-08T23:45:47.481869Z","iopub.execute_input":"2023-02-08T23:45:47.482181Z","iopub.status.idle":"2023-02-08T23:45:53.440063Z","shell.execute_reply.started":"2023-02-08T23:45:47.482144Z","shell.execute_reply":"2023-02-08T23:45:53.439073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Images look very different - they have different sizes, background color, and precision. It's gonna be a serious task of data preprocessing.","metadata":{}},{"cell_type":"markdown","source":"That's all for now. This notebook is still under development, so put your likes and watch the development of this EDA.\n\nThanks for your attention😊","metadata":{}}]}