{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"👋 In this kernel I will investicate the relationship between `cancer` and `BIRADS` variables in the competition metadata.","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-warning\">\n<b>Tip:</b>\n    BIRADS assesment categories: <b>Category 0</b>: Incomplete, need additional imaging evaluation and/or prior mammograms for comparison; <b>Category 1</b>: Negative; <b>Category 2</b>: Benign; <b>Category 3</b>: Probably benign; <b>Category 4</b>: Suspicious; <b>Category 4A</b>: Low suspicion for malignancy; <b>Category 4B</b>: Moderate suspicion for malignancy; <b>Category 4C</b>: High suspicion for malignancy; <b>Category 5</b>: Highly suggestive of malignancy; <b>Category 6</b>: Known biopsy-proven malignancy.\n</div>","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-success\">\n<b>Data source:</b> \nRadiological Society of North America. (2022, Nov 29). RSNA Screening Mammography Breast Cancer Detection, Version 1. Retrieved 2023 Feb 9 from [https://www.kaggle.com/competitions/rsna-breast-cancer-detection/data].\n</div>","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:22.508099Z","iopub.execute_input":"2023-02-21T00:18:22.50878Z","iopub.status.idle":"2023-02-21T00:18:23.468402Z","shell.execute_reply.started":"2023-02-21T00:18:22.508685Z","shell.execute_reply":"2023-02-21T00:18:23.467147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%matplotlib inline","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:23.470004Z","iopub.execute_input":"2023-02-21T00:18:23.471092Z","iopub.status.idle":"2023-02-21T00:18:23.482482Z","shell.execute_reply.started":"2023-02-21T00:18:23.471055Z","shell.execute_reply":"2023-02-21T00:18:23.481285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/rsna-breast-cancer-detection/train.csv\", dtype=object)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:23.483893Z","iopub.execute_input":"2023-02-21T00:18:23.485151Z","iopub.status.idle":"2023-02-21T00:18:23.641113Z","shell.execute_reply.started":"2023-02-21T00:18:23.485097Z","shell.execute_reply":"2023-02-21T00:18:23.640041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(df.index)","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:23.643109Z","iopub.execute_input":"2023-02-21T00:18:23.643942Z","iopub.status.idle":"2023-02-21T00:18:23.651658Z","shell.execute_reply.started":"2023-02-21T00:18:23.643902Z","shell.execute_reply":"2023-02-21T00:18:23.650513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['projection'] = df['laterality'].astype(str) + '-' + df['view'].astype(str)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:23.65337Z","iopub.execute_input":"2023-02-21T00:18:23.653765Z","iopub.status.idle":"2023-02-21T00:18:23.689557Z","shell.execute_reply.started":"2023-02-21T00:18:23.653727Z","shell.execute_reply":"2023-02-21T00:18:23.688479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.heatmap(df.isna(), yticklabels=False, cbar=False, cmap='viridis')  # https://www.kaggle.com/code/mashithaa/breast-cancer-detection-beginner","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:23.692954Z","iopub.execute_input":"2023-02-21T00:18:23.69332Z","iopub.status.idle":"2023-02-21T00:18:24.815551Z","shell.execute_reply.started":"2023-02-21T00:18:23.693294Z","shell.execute_reply":"2023-02-21T00:18:24.814015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Examination level cancer class\nLet's try to find out examination-level cancer class.","metadata":{}},{"cell_type":"code","source":"df['group'] = df.groupby(['patient_id',]).cumcount()","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:24.817269Z","iopub.execute_input":"2023-02-21T00:18:24.817588Z","iopub.status.idle":"2023-02-21T00:18:24.860929Z","shell.execute_reply.started":"2023-02-21T00:18:24.81756Z","shell.execute_reply":"2023-02-21T00:18:24.859915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's organize our data a bit differently. There will be only one row for each `patient_id` and all information related to individual files with that same `patient_id` will rearanged into columns with a running number.","metadata":{}},{"cell_type":"code","source":"df_pivot = df.pivot(['patient_id',], 'group').fillna('')\ndf_pivot.columns = [f'{a}_{b + 1}' if b > 0 else a for a, b in df_pivot.columns]\ndf_pivot = df_pivot.reset_index().rename_axis(None, axis=1)\ndf_pivot.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:24.86233Z","iopub.execute_input":"2023-02-21T00:18:24.862825Z","iopub.status.idle":"2023-02-21T00:18:25.281566Z","shell.execute_reply.started":"2023-02-21T00:18:24.862782Z","shell.execute_reply":"2023-02-21T00:18:25.280691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some exams have many files.","metadata":{}},{"cell_type":"markdown","source":"Let's then find out the examination-level `cancer` class.","metadata":{}},{"cell_type":"code","source":"sel_cols_cancer = [col for col in df_pivot.columns if 'cancer' in col]","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:25.28262Z","iopub.execute_input":"2023-02-21T00:18:25.282885Z","iopub.status.idle":"2023-02-21T00:18:25.287644Z","shell.execute_reply.started":"2023-02-21T00:18:25.282862Z","shell.execute_reply":"2023-02-21T00:18:25.28625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_pivot['cancer_study_level'] = df_pivot[sel_cols_cancer].apply(pd.to_numeric).max(axis=1, numeric_only=True)\ndf_pivot[['patient_id', 'cancer_study_level']].head()","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:25.290605Z","iopub.execute_input":"2023-02-21T00:18:25.291415Z","iopub.status.idle":"2023-02-21T00:18:25.352429Z","shell.execute_reply.started":"2023-02-21T00:18:25.291379Z","shell.execute_reply":"2023-02-21T00:18:25.351314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's also try to come up with an examination level BIRADS class. Mainly we would like to know if the class is `0` or `nan`.","metadata":{}},{"cell_type":"code","source":"sel_cols_birads = [col for col in df_pivot.columns if 'BIRADS' in col]","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:25.353872Z","iopub.execute_input":"2023-02-21T00:18:25.354477Z","iopub.status.idle":"2023-02-21T00:18:25.360539Z","shell.execute_reply.started":"2023-02-21T00:18:25.354449Z","shell.execute_reply":"2023-02-21T00:18:25.359018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_pivot['BIRADS_study_level'] = df_pivot[sel_cols_birads].apply(pd.to_numeric).min(axis=1, numeric_only=True)\ndf_pivot[['patient_id', 'cancer_study_level', 'BIRADS_study_level']].head()","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:25.361985Z","iopub.execute_input":"2023-02-21T00:18:25.362341Z","iopub.status.idle":"2023-02-21T00:18:25.426219Z","shell.execute_reply.started":"2023-02-21T00:18:25.362308Z","shell.execute_reply":"2023-02-21T00:18:25.425222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here are all our cancerous cases in the examination-level.","metadata":{}},{"cell_type":"code","source":"df_pivot.loc[(df_pivot['cancer_study_level'] == 1)][['patient_id', 'cancer_study_level', 'BIRADS_study_level']]","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:25.427381Z","iopub.execute_input":"2023-02-21T00:18:25.428187Z","iopub.status.idle":"2023-02-21T00:18:25.45223Z","shell.execute_reply.started":"2023-02-21T00:18:25.428152Z","shell.execute_reply":"2023-02-21T00:18:25.451083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here are all our cancerous cases in the examination-level having BIRADS class `0`.","metadata":{}},{"cell_type":"code","source":"df_pivot.loc[(df_pivot['cancer_study_level'] == 1) & (df_pivot['BIRADS_study_level'] == 0)][['patient_id', 'cancer_study_level', 'BIRADS_study_level']]","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:25.453748Z","iopub.execute_input":"2023-02-21T00:18:25.454381Z","iopub.status.idle":"2023-02-21T00:18:25.47866Z","shell.execute_reply.started":"2023-02-21T00:18:25.454345Z","shell.execute_reply":"2023-02-21T00:18:25.47724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see, all cancerous cases are not present.","metadata":{}},{"cell_type":"markdown","source":"Here are all our cancerous cases in the examination-level having BIRADS class `1`.","metadata":{}},{"cell_type":"code","source":"df_pivot.loc[(df_pivot['cancer_study_level'] == 1) & (df_pivot['BIRADS_study_level'] == 1)][['patient_id', 'cancer_study_level', 'BIRADS_study_level']]","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:25.480389Z","iopub.execute_input":"2023-02-21T00:18:25.480801Z","iopub.status.idle":"2023-02-21T00:18:25.498888Z","shell.execute_reply.started":"2023-02-21T00:18:25.480767Z","shell.execute_reply":"2023-02-21T00:18:25.497116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here are all our cancerous cases in the examination-level having BIRADS class `2`.","metadata":{}},{"cell_type":"code","source":"df_pivot.loc[(df_pivot['cancer_study_level'] == 1) & (df_pivot['BIRADS_study_level'] == 2)][['patient_id', 'cancer_study_level', 'BIRADS_study_level']]","metadata":{"execution":{"iopub.status.busy":"2023-02-21T00:18:25.500952Z","iopub.execute_input":"2023-02-21T00:18:25.50135Z","iopub.status.idle":"2023-02-21T00:18:25.514178Z","shell.execute_reply.started":"2023-02-21T00:18:25.501313Z","shell.execute_reply":"2023-02-21T00:18:25.513261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here are all our cancerous cases in the examination-level having BIRADS class `nan`.","metadata":{}},{"cell_type":"code","source":"df_pivot.loc[(df_pivot['cancer_study_level'] == 1) & (df_pivot['BIRADS_study_level'] != 0) & (\n              df_pivot['BIRADS_study_level'] != 1) & (df_pivot['BIRADS_study_level'] != 2)][['patient_id', 'cancer_study_level', 'BIRADS_study_level']]","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-02-21T00:18:25.515784Z","iopub.execute_input":"2023-02-21T00:18:25.516097Z","iopub.status.idle":"2023-02-21T00:18:25.542982Z","shell.execute_reply.started":"2023-02-21T00:18:25.516065Z","shell.execute_reply":"2023-02-21T00:18:25.542084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have found our missing cancerous cases.","metadata":{}},{"cell_type":"markdown","source":"# Conclusions\n\nNow we can better split our data using BIRADS category for our experiments (especially in a [multi-view](https://www.kaggle.com/code/anttiisosalo/birads-classification-rsna-bc-detection) setting). Strafying according to birads class when using K-Fold Cross-Validation might not be a bad idea either. 👍","metadata":{}},{"cell_type":"markdown","source":"# References\n\nD’Orsi, C., Bassett, L., & Feig, S. (2018). Breast imaging reporting and data system (BI-RADS). Breast imaging atlas, 4th edn. American College of Radiology, Reston.","metadata":{}}]}