{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n \n \n
","metadata":{}},{"cell_type":"markdown","source":"# RSNA Screening Mammography Breast Cancer Detection\n## Find breast cancers in screening mammograms\n\n### Context\n* The RSNA International Conference on Artificial Intelligence in Radiology (RSNA-AI) is a new conference that will be held in conjunction with the 2019 RSNA Annual Meeting in Chicago, Illinois, USA.\n* The goal of the conference is to bring together radiologists, radiology trainees, and radiology researchers to discuss the latest advances in artificial intelligence (AI) and machine learning (ML) in radiology.\n* The RSNA and the American College of Radiology (ACR) provide the RSNA-AI Challenge to promote the development of AI and ML algorithms for the detection of breast cancer in mammography.\n* The goal of the challenge is to develop an algorithm that can automatically detect breast cancer in screening mammograms.\n* The work of improving the automation of detection in screening mammography may enable radiologists to be more accurate and efficient, improving the quality and safety of patient care. It could also help reduce costs and unnecessary medical procedures.\n \nLink to the competition: [Here](https://www.kaggle.com/competitions/rsna-breast-cancer-detection/overview)
\nLink to the dataset: [Here](https://www.kaggle.com/c/rsna-breast-cancer-detection/data)","metadata":{}},{"cell_type":"markdown","source":"### Notebooks\n* [RSNA-BCD | DICOM ➜ ROI-PNG](https://www.kaggle.com/code/matteoperfidio/rsna-bcd-dicom-roi-png): This notebook converts the DICOM images to PNG images with the ROI (Region of Interest).\n* [RSNA-BCD | EfficientNetB3 [Train]](https://www.kaggle.com/code/matteoperfidio/rsna-bcd-efficientnetb3-train): This notebook uses the EfficientNetB3 model to train the provided dataset.\n* [RSNA-BCD | EfficientNetB3 [Test]](https://www.kaggle.com/code/matteoperfidio/rsna-bcd-efficientnetb3-test): This notebook uses the EfficientNetB3 model to test the provided dataset and submit the results to the competition.\n\n**Note**: *The test notebook is required because the model is trained on the TPU but the submission must be done via the GPU.*","metadata":{}},{"cell_type":"markdown","source":"### Dataset\n* [RSNA-BCD | DICOM ➜ ROI-PNG [Dataset]](https://www.kaggle.com/datasets/nottyche/rsnabcdroipng): This dataset contains the DICOM images converted to PNG images with the ROI (Region of Interest) obtained via the Notebook [RSNA BCD | DICOM ➜ ROI-PNG](https://www.kaggle.com/code/matteoperfidio/rsna-bcd-dicom-roi-png).\n\n**Note**: *The dataset is required because by using the TPU is not possible to load the output of the Notebook [RSNA BCD | DICOM ➜ ROI-PNG](https://www.kaggle.com/code/matteoperfidio/rsna-bcd-dicom-roi-png).*","metadata":{}},{"cell_type":"markdown","source":"### Overview\n#### Goal\n* In this competition, the goal is to develop a model that can predict the presence of breast cancer in mammography screening images.\n\n#### TPU v3-8\n* In this notebook, we will use the TPU v3-8 accelerator to train the model. \n* There is the possibility to use the GPU, but since the time required to train the model is very long, we will use the TPU to speed up the training process.\n\n#### ROI Images\n* In this notebook, we will use the ROI (Region of Interest) images in order to improve the recognition of the breast cancer. \n* By observing the images, we can see that there is a problem of pixel imbalance, in fact we have that the breast cancer occupies a small part of the image, so we can use the ROI images to focus on the breast cancer and ignore the rest of the image.\n\n#### Oversampling\n* There is problem of class imbalance beetween cancer and non-cancer classes. \n* In order to solve this problem, we will use oversampling technique to increase the number of cancer images, this will help to reduce the class imbalance effect on the loss function.\n\n#### Comet.ml","metadata":{}},{"cell_type":"markdown","source":"
\n \n
","metadata":{}},{"cell_type":"markdown","source":"* In this project we will use [Comet.ml](https://www.comet.ml/) which is a platform for tracking machine learning experiments.\n* It is used to track metrics and hyperparameters in machine learning experiments. In this way, we can track the training process and we can compare the different models.\n\n#### Stratified K-Fold Cross Validation\n* In order to avoid overfitting, we will use the Stratified K-Fold technique to split the dataset into K folds and train the model on K-1 folds and validate it on the remaining fold. \n* In particular, we will create 2 models, one with the fold 0 as validation set and one with the fold 1 as validation set. In this way, we will have an ensamble of 2 models that will be trained on different folds. \n* Then we will create an out-of-fold (OOF) dataset and we use the 2 models to predict on this dataset and obtain the final metrics. \n* Finally, we export the 2 best models and we use them to predict on the test dataset (in the test notebook).\n\n#### Augmentation\n* In order to avoid overfitting, we will use the augmentation technique to create new images from the original ones. \n* In particular, we will use the horizontal flip, vertical flip, brightness, contrast, saturation, and hue augmentation techniques.\n\n#### Transfer Learning\n* In this notebook, we will use the EfficientNetB3 model to train the dataset. \n* In particular, we will use the pretrained weights of the model on the ImageNet dataset and we will train the model on the provided dataset. \n* In addition, we use another input layer for the informations about the patients and concatenate it with the output of the model. \n\n#### Hyperparameters\n* Since the training process is very long, we not have time to perform a hyperparameter tuning. \n* We decide to explore the hyperparameters of the EfficientNetB3 model used in the literature and we use them in this notebook.\n\n#### Learning Rate Scheduler\n* In order to improve the convergence of the model, we will use an user-defined learning rate scheduler. \n* With this scheduler, we start with a low learning rate and we increase it exponentially until we reach the maximum learning rate.\n* Then, we decrease the learning rate more slowly until we reach the minimum learning rate.\n\n#### Metrics\n* In this notebook, as the competition metric, we will use the P-F1 score. This is an extension of the traditional F score that accepts probabilities instead of binary classifications. \n* Furthermore, we will use the ROC AUC score and the accuracy as additional metrics.","metadata":{}},{"cell_type":"markdown","source":"### Outline\n* [1. Install Libraries](#1)\n* [2. Import Libraries](#2)\n* [3. Configuration](#3)\n* [4. Seed](#4)\n* [5. Device Configuration](#5)\n* [6. Load Data](#6)\n* [7. Exploratory Data Analysis (EDA)](#7)\n* [8. Missing Values](#8)\n* [9. Fix Data Types](#9)\n* [10. Numerical Categorization](#10)\n* [11. Dummy Variables](#11)\n* [12. Stratified K-Fold Cross Validation](#12)\n* [13. Build Dataset](#13)\n* [14. Define Input Features](#14)\n* [15. Augumentation](#15)\n* [16. Normalization](#16)\n* [17. Custom Metrics](#17)\n* [18. Build Model](#18)\n* [19. Learning Rate Scheduler](#19)\n* [20. Callbacks](#20)\n* [21. Train Model](#21)","metadata":{}},{"cell_type":"markdown","source":"### 1. Install Libraries ","metadata":{}},{"cell_type":"code","source":"from IPython.display import clear_output","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:11:34.740924Z","iopub.execute_input":"2023-01-18T16:11:34.741293Z","iopub.status.idle":"2023-01-18T16:11:34.746351Z","shell.execute_reply.started":"2023-01-18T16:11:34.741257Z","shell.execute_reply":"2023-01-18T16:11:34.745538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -qU comet_ml\n!pip install -qU scikit-learn\nclear_output()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:11:34.749316Z","iopub.execute_input":"2023-01-18T16:11:34.749699Z","iopub.status.idle":"2023-01-18T16:11:52.682165Z","shell.execute_reply.started":"2023-01-18T16:11:34.749652Z","shell.execute_reply":"2023-01-18T16:11:52.681039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2. Import Libraries ","metadata":{}},{"cell_type":"code","source":"import os, re, math, random, shutil, warnings, gc\nimport numpy as np\nimport pandas as pd\nimport comet_ml\nimport sklearn\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nimport plotly.express as px\nimport plotly.offline as py\nimport plotly.figure_factory as ff\nfrom plotly.subplots import make_subplots\nimport tensorflow as tf\n\nfrom comet_ml import Experiment\nfrom tensorflow import keras\nfrom tensorflow.python.client import device_lib\nfrom sklearn.model_selection import StratifiedGroupKFold\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.metrics import roc_curve, auc\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom matplotlib.ticker import StrMethodFormatter\nfrom mpl_toolkits.axes_grid1 import ImageGrid\nfrom kaggle_datasets import KaggleDatasets\nfrom kaggle_secrets import UserSecretsClient\n\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\npd.options.mode.chained_assignment = None\ntf.get_logger().setLevel('ERROR')\npy.init_notebook_mode(connected=True)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:11:52.684673Z","iopub.execute_input":"2023-01-18T16:11:52.685101Z","iopub.status.idle":"2023-01-18T16:11:52.700957Z","shell.execute_reply.started":"2023-01-18T16:11:52.685049Z","shell.execute_reply":"2023-01-18T16:11:52.699837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3. Configuration ","metadata":{}},{"cell_type":"code","source":"devices = ['TPU', 'GPU']\nimg_size = [(1024,512), (512,256)]\nimg_ext = ['png', 'jpg', 'jpeg']\n\nclass Config:\n \n def __init__(self):\n \n self.comet = True\n \n self.project_name = \"rsna-bcd-train-model\"\n self.api_key = UserSecretsClient().get_secret(\"api_key\")\n self.workspace = \"nottyche\"\n \n self.debug = False\n \n self.device = devices[0]\n self.num_devices = 1\n self.model_name = 'EfficientNetB3'\n self.seed = 97\n\n self.path = '/kaggle/input/rsnabcdroipng/'\n self.train_path = self.path + 'train_images/'\n self.train_csv = self.path + 'train.csv'\n \n self.weights = \"/kaggle/input/efficientnetb3-notop/efficientnetb3_notop.h5\"\n self.folds = 5\n self.validation_folds = [0,1]\n self.oversampling = True\n self.oversampling_factor = 10\n \n self.threshold = 0.6\n\n self.parameters = {\n 'batch_size': 32,\n 'epochs': 12,\n 'dropout': 0.05,\n 'optimizer': 'adam',\n 'loss': 'binary_crossentropy',\n 'patience': 5\n }\n\n self.img_size = img_size[1]\n self.img_ext = img_ext[0]\n\nconfig = Config()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:11:52.703392Z","iopub.execute_input":"2023-01-18T16:11:52.704193Z","iopub.status.idle":"2023-01-18T16:11:52.853289Z","shell.execute_reply.started":"2023-01-18T16:11:52.704156Z","shell.execute_reply":"2023-01-18T16:11:52.852229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note**: *In order to execute the code in this notebook, you have to import the [RSNA BCD | DICOM ➜ ROI-PNG](https://www.kaggle.com/code/matteoperfidio/rsna-bcd-dicom-roi-png) notebook and use the output of the notebook as input of this notebook.*","metadata":{}},{"cell_type":"markdown","source":"**Note**: *If you want to use [Comet.ml](https://www.comet.ml/) to track the training process, you have to create an account on the platform and you have to insert your API key in Kaggle Secrets. See [this](https://www.kaggle.com/docs/kernels#secrets) for more information.*","metadata":{}},{"cell_type":"markdown","source":"### 4. Seed ","metadata":{}},{"cell_type":"code","source":"def seed_everything(seed):\n os.environ['PYTHONHASHSEED'] = str(seed)\n os.environ['TF_DETERMINISTIC_OPS'] = str(seed)\n os.environ['TF_CUDNN_DETERMINISTIC'] = str(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n random.seed(seed)\n\nseed_everything(config.seed)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:11:52.85492Z","iopub.execute_input":"2023-01-18T16:11:52.855207Z","iopub.status.idle":"2023-01-18T16:11:52.862386Z","shell.execute_reply.started":"2023-01-18T16:11:52.855173Z","shell.execute_reply":"2023-01-18T16:11:52.861232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 5. Device Configuration ","metadata":{}},{"cell_type":"code","source":"if config.device == 'TPU':\n\n print('Connecting to TPU...')\n try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect() \n except ValueError:\n raise BaseException('ERROR: Not connected to a TPU runtime; please see the previous cell in this notebook for instructions!')\n\n tf.keras.mixed_precision.set_global_policy(\"mixed_bfloat16\")\n tf.config.set_soft_device_placement(True)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n\n config.path = KaggleDatasets().get_gcs_path(config.path.split('/')[3])\n config.train_path = config.path + '/' + 'train_images/'\n config.num_devices = strategy.num_replicas_in_sync\n config.parameters['batch_size'] = config.parameters['batch_size'] * config.num_devices\n\n print(f'Running on {config.num_devices} TPU devices')\n\n\nif config.device == 'GPU':\n \n num_devices = len(tf.config.list_physical_devices('GPU'))\n\n if num_devices > 1:\n config.num_devices = num_devices\n strategy = tf.distribute.MirroredStrategy()\n config.parameters['batch_size'] = config.parameters['batch_size'] * config.num_devices\n print(f'Running on {num_devices} GPU devices')\n elif num_devices == 1:\n strategy = tf.distribute.get_strategy()\n print(f'Running on {num_devices} GPU device')\n else:\n strategy = tf.distribute.get_strategy()\n config.device = 'CPU'\n print(f'Running on CPU')\n \n tf.config.optimizer.set_jit(True)\n tf.keras.mixed_precision.set_global_policy(\"mixed_float16\")","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:11:52.864005Z","iopub.execute_input":"2023-01-18T16:11:52.865294Z","iopub.status.idle":"2023-01-18T16:11:59.790518Z","shell.execute_reply.started":"2023-01-18T16:11:52.865239Z","shell.execute_reply":"2023-01-18T16:11:59.789332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 6. Load Data ","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(config.train_csv)\n\nif config.debug:\n train_df = train_df.sample(1000).reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:11:59.791845Z","iopub.execute_input":"2023-01-18T16:11:59.792106Z","iopub.status.idle":"2023-01-18T16:11:59.870854Z","shell.execute_reply.started":"2023-01-18T16:11:59.792075Z","shell.execute_reply":"2023-01-18T16:11:59.869625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['image_path'] = config.train_path + train_df['patient_id'].astype(str) + '/' + train_df['image_id'].astype(str) + '.png'","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:11:59.87225Z","iopub.execute_input":"2023-01-18T16:11:59.872517Z","iopub.status.idle":"2023-01-18T16:12:00.019167Z","shell.execute_reply.started":"2023-01-18T16:11:59.872483Z","shell.execute_reply":"2023-01-18T16:12:00.018115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.io.gfile.exists(train_df.image_path.iloc[0])","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:00.021909Z","iopub.execute_input":"2023-01-18T16:12:00.022511Z","iopub.status.idle":"2023-01-18T16:12:00.111447Z","shell.execute_reply.started":"2023-01-18T16:12:00.022468Z","shell.execute_reply":"2023-01-18T16:12:00.110397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:00.113132Z","iopub.execute_input":"2023-01-18T16:12:00.113668Z","iopub.status.idle":"2023-01-18T16:12:00.138427Z","shell.execute_reply.started":"2023-01-18T16:12:00.113619Z","shell.execute_reply":"2023-01-18T16:12:00.137505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:00.13966Z","iopub.execute_input":"2023-01-18T16:12:00.140013Z","iopub.status.idle":"2023-01-18T16:12:00.170262Z","shell.execute_reply.started":"2023-01-18T16:12:00.13998Z","shell.execute_reply":"2023-01-18T16:12:00.169265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7. Exploratory Data Analysis (EDA) ","metadata":{}},{"cell_type":"code","source":"fig = go.Figure(data=[go.Pie(labels=['No Cancer', 'Cancer'], values=train_df['cancer'].value_counts(), hole=.3)])\nfig.update_traces(textposition='outside', textinfo='percent+label')\nfig.update_layout(title_text='Number of patients with cancer', title_x=0.5)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:00.171429Z","iopub.execute_input":"2023-01-18T16:12:00.171671Z","iopub.status.idle":"2023-01-18T16:12:00.194199Z","shell.execute_reply.started":"2023-01-18T16:12:00.171642Z","shell.execute_reply":"2023-01-18T16:12:00.193029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(train_df, x=\"age\", title='Age distribution of the patients')\nfig.update_traces(marker_line_color='rgb(255,255,255)', marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_x=0.5)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:00.195941Z","iopub.execute_input":"2023-01-18T16:12:00.196293Z","iopub.status.idle":"2023-01-18T16:12:00.721716Z","shell.execute_reply.started":"2023-01-18T16:12:00.196247Z","shell.execute_reply":"2023-01-18T16:12:00.720436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.violin(train_df.sample(frac=0.1), y=\"age\", x=\"cancer\", color=\"cancer\", box=True, points=\"all\", title='Age distribution with respect to the cancer')\nfig.update_layout(title_x=0.5)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:00.723027Z","iopub.execute_input":"2023-01-18T16:12:00.72328Z","iopub.status.idle":"2023-01-18T16:12:00.868404Z","shell.execute_reply.started":"2023-01-18T16:12:00.723248Z","shell.execute_reply":"2023-01-18T16:12:00.867486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(train_df.groupby('patient_id').count().reset_index(), x=\"image_id\", color='image_id', title='Number of images per patient')\nfig.update_traces(marker_line_color='rgb(255,255,255)', marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_x=0.5)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:00.869617Z","iopub.execute_input":"2023-01-18T16:12:00.870165Z","iopub.status.idle":"2023-01-18T16:12:01.123452Z","shell.execute_reply.started":"2023-01-18T16:12:00.870114Z","shell.execute_reply":"2023-01-18T16:12:01.122484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = make_subplots(rows=2, cols=2, subplot_titles=('Laterality', 'Density', 'View', 'Implant'))\n\nfig.add_trace(go.Bar(x=train_df['laterality'].value_counts().index, y=train_df['laterality'].value_counts().values), row=1, col=1)\nfig.add_trace(go.Bar(x=train_df['density'].value_counts().index, y=train_df['density'].value_counts().values), row=1, col=2)\nfig.add_trace(go.Bar(x=train_df['view'].value_counts().index, y=train_df['view'].value_counts().values), row=2, col=1)\nfig.add_trace(go.Bar(x=train_df['implant'].value_counts().index, y=train_df['implant'].value_counts().values), row=2, col=2)\n\nfig.update_traces(marker_line_color='rgb(255,255,255)', marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_text='Count of laterality, density, view and implant', title_x=0.5)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.124686Z","iopub.execute_input":"2023-01-18T16:12:01.124948Z","iopub.status.idle":"2023-01-18T16:12:01.229889Z","shell.execute_reply.started":"2023-01-18T16:12:01.124917Z","shell.execute_reply":"2023-01-18T16:12:01.229005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = make_subplots(rows=2, cols=2, subplot_titles=('Laterality', 'Density', 'View', 'Implant'))\n\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 1]['laterality'].value_counts().index, y=train_df[train_df['cancer'] == 1]['laterality'].value_counts().values, name='Cancer'), row=1, col=1)\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 0]['laterality'].value_counts().index, y=train_df[train_df['cancer'] == 0]['laterality'].value_counts().values, name='No Cancer'), row=1, col=1)\n\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 1]['density'].value_counts().index, y=train_df[train_df['cancer'] == 1]['density'].value_counts().values, name='Cancer'), row=1, col=2)\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 0]['density'].value_counts().index, y=train_df[train_df['cancer'] == 0]['density'].value_counts().values, name='No Cancer'), row=1, col=2)\n\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 0]['view'].value_counts().index, y=train_df[train_df['cancer'] == 0]['view'].value_counts().values, name='No Cancer'), row=2, col=1)\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 1]['view'].value_counts().index, y=train_df[train_df['cancer'] == 1]['view'].value_counts().values, name='Cancer'), row=2, col=1)\n\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 0]['implant'].value_counts().index, y=train_df[train_df['cancer'] == 0]['implant'].value_counts().values, name='No Cancer'), row=2, col=2)\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 1]['implant'].value_counts().index, y=train_df[train_df['cancer'] == 1]['implant'].value_counts().values, name='Cancer'), row=2, col=2)\n\nfig.update_traces(marker_line_color='rgb(255,255,255)', marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_text='Count of laterality, density, view and implant with respect to the cancer', title_x=0.5)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.231178Z","iopub.execute_input":"2023-01-18T16:12:01.231419Z","iopub.status.idle":"2023-01-18T16:12:01.429067Z","shell.execute_reply.started":"2023-01-18T16:12:01.23139Z","shell.execute_reply":"2023-01-18T16:12:01.428062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 8. Missing Values ","metadata":{}},{"cell_type":"code","source":"def find_missing_data(data):\n total = data.isnull().sum().sort_values(ascending = False)\n percentage = (data.isnull().sum()/data.isnull().count()).sort_values(ascending = False)\n return pd.concat([total,percentage] , axis = 1 , keys = ['Total' , 'Percent'])","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.430424Z","iopub.execute_input":"2023-01-18T16:12:01.430737Z","iopub.status.idle":"2023-01-18T16:12:01.436947Z","shell.execute_reply.started":"2023-01-18T16:12:01.43069Z","shell.execute_reply":"2023-01-18T16:12:01.43608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"find_missing_data(train_df)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.438068Z","iopub.execute_input":"2023-01-18T16:12:01.438863Z","iopub.status.idle":"2023-01-18T16:12:01.509198Z","shell.execute_reply.started":"2023-01-18T16:12:01.438816Z","shell.execute_reply":"2023-01-18T16:12:01.508212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = train_df.drop(['BIRADS'], axis=1)\ntrain_df['age'] = train_df['age'].fillna(train_df['age'].mean())\ntrain_df['density'] = train_df['density'].fillna('E')","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.510541Z","iopub.execute_input":"2023-01-18T16:12:01.510958Z","iopub.status.idle":"2023-01-18T16:12:01.529363Z","shell.execute_reply.started":"2023-01-18T16:12:01.510922Z","shell.execute_reply":"2023-01-18T16:12:01.528506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"find_missing_data(train_df)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.530482Z","iopub.execute_input":"2023-01-18T16:12:01.531265Z","iopub.status.idle":"2023-01-18T16:12:01.591669Z","shell.execute_reply.started":"2023-01-18T16:12:01.531223Z","shell.execute_reply":"2023-01-18T16:12:01.590518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 9. Fix Data Types ","metadata":{}},{"cell_type":"code","source":"train_df['laterality'] = train_df['laterality'].astype('category')\ntrain_df['view'] = train_df['view'].astype('category')\ntrain_df['age'] = train_df['age'].astype('int64')\ntrain_df['image_path'] = train_df['image_path'].astype('string')","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.593169Z","iopub.execute_input":"2023-01-18T16:12:01.593755Z","iopub.status.idle":"2023-01-18T16:12:01.628332Z","shell.execute_reply.started":"2023-01-18T16:12:01.593694Z","shell.execute_reply":"2023-01-18T16:12:01.626924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.629877Z","iopub.execute_input":"2023-01-18T16:12:01.630171Z","iopub.status.idle":"2023-01-18T16:12:01.656272Z","shell.execute_reply.started":"2023-01-18T16:12:01.630136Z","shell.execute_reply":"2023-01-18T16:12:01.655487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 10. Numerical Categorization ","metadata":{}},{"cell_type":"code","source":"train_df[\"age_bin\"] = pd.cut(train_df['age'].values.reshape(-1), bins=5, labels=False)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.659988Z","iopub.execute_input":"2023-01-18T16:12:01.660412Z","iopub.status.idle":"2023-01-18T16:12:01.667263Z","shell.execute_reply.started":"2023-01-18T16:12:01.660377Z","shell.execute_reply":"2023-01-18T16:12:01.666253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 11. Dummy Variables ","metadata":{}},{"cell_type":"code","source":"cat_cols = ['laterality', 'view', 'density', 'difficult_negative_case']\ntrain_df = pd.get_dummies(train_df, columns=cat_cols)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.668616Z","iopub.execute_input":"2023-01-18T16:12:01.669102Z","iopub.status.idle":"2023-01-18T16:12:01.702742Z","shell.execute_reply.started":"2023-01-18T16:12:01.669067Z","shell.execute_reply":"2023-01-18T16:12:01.701861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 12. Stratified K-Fold Cross Validation ","metadata":{}},{"cell_type":"code","source":"stratified_columns = train_df.columns[~train_df.columns.isin(['patient_id', 'image_id', \n 'image_path', 'target', \n 'site_id', 'width', 'height', \n 'age', 'invasive', \n 'implant', 'machine_id'])]\n\ntrain_df['stratify'] = ''\n\nfor col in stratified_columns:\n train_df['stratify'] += train_df[col].astype(str)\n\ntrain_df['stratify'] = train_df['stratify'].astype('string')\n\n\nk_fold = StratifiedGroupKFold(n_splits=config.folds, shuffle=True, random_state=config.seed)\n\n\nfor fold, (train_idx, val_idx) in enumerate(k_fold.split(train_df, train_df['stratify'], train_df['patient_id'])):\n train_df.loc[val_idx, 'fold'] = fold\n\ntrain_df['fold'] = train_df['fold'].astype('int32')","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:01.704031Z","iopub.execute_input":"2023-01-18T16:12:01.704443Z","iopub.status.idle":"2023-01-18T16:12:08.401238Z","shell.execute_reply.started":"2023-01-18T16:12:01.704409Z","shell.execute_reply":"2023-01-18T16:12:08.400126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:08.402386Z","iopub.execute_input":"2023-01-18T16:12:08.402652Z","iopub.status.idle":"2023-01-18T16:12:08.431261Z","shell.execute_reply.started":"2023-01-18T16:12:08.402624Z","shell.execute_reply":"2023-01-18T16:12:08.43023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df.groupby(['fold', \"cancer\"]).size())","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:08.432532Z","iopub.execute_input":"2023-01-18T16:12:08.432817Z","iopub.status.idle":"2023-01-18T16:12:08.445282Z","shell.execute_reply.started":"2023-01-18T16:12:08.43278Z","shell.execute_reply":"2023-01-18T16:12:08.444142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 1]['fold'].value_counts().index, y=train_df[train_df['cancer'] == 1]['fold'].value_counts().values, name='Cancer'))\nfig.add_trace(go.Bar(x=train_df[train_df['cancer'] == 0]['fold'].value_counts().index, y=train_df[train_df['cancer'] == 0]['fold'].value_counts().values, name='No Cancer'))\nfig.update_traces(marker_line_color='rgb(255,255,255)', marker_line_width=1.5, opacity=0.6)\nfig.update_layout(title_text='Count of folds with respect to the cancer', title_x=0.5)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:08.446885Z","iopub.execute_input":"2023-01-18T16:12:08.447917Z","iopub.status.idle":"2023-01-18T16:12:08.51406Z","shell.execute_reply.started":"2023-01-18T16:12:08.447831Z","shell.execute_reply":"2023-01-18T16:12:08.512984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 13. Build Dataset ","metadata":{}},{"cell_type":"code","source":"def decode_image(label=True, img_size=config.img_size, ext=config.img_ext):\n\n def _decode_image(input):\n image = tf.io.read_file(input['input_image'])\n if ext == 'png':\n image = tf.image.decode_png(image, channels=3)\n elif ext in ['jpg', 'jpeg']:\n image = tf.image.decode_jpeg(image, channels=3)\n else:\n raise ValueError(\"Image extension not supported\")\n image = tf.image.resize(image, img_size)\n image = tf.reshape(image, [*img_size, 3])\n image = tf.cast(image, tf.float32) / 255.0\n input['input_image'] = image\n return input\n \n def _decode_labeled_image(input, label):\n input = _decode_image(input)\n return input, label\n\n return _decode_labeled_image if label else _decode_image\n\n\ndef data_augment(label=True):\n def _data_augment(input):\n image = input['input_image']\n image = tf.image.random_flip_left_right(image)\n image = tf.image.random_hue(image, 0.05)\n image = tf.image.random_saturation(image, 0.5, 1)\n image = tf.image.random_contrast(image, 1, 2)\n image = tf.image.random_brightness(image, 0.15)\n input['input_image'] = image\n return input\n \n def _data_augment_labeled(input, label):\n input = _data_augment(input)\n return input, label\n \n return _data_augment_labeled if label else _data_augment\n\n\ndef build_dataset(df, input_features, image_size=config.img_size, batch_size=config.parameters['batch_size'], \n label=True, shuffle=True, augment=False, repeat=False, cache=False, ext=config.img_ext):\n \n decode = decode_image(label, img_size=image_size, ext=ext)\n\n if label:\n dataset = tf.data.Dataset.from_tensor_slices(({'input_image': df['image_path'].values, \n 'input_features' : df[input_features].values}, \n df['cancer'].values))\n else:\n dataset = tf.data.Dataset.from_tensor_slices({'input_image': df['image_path'].values, \n 'input_features' : df[input_features].values})\n\n dataset = dataset.map(decode, num_parallel_calls=tf.data.AUTOTUNE)\n \n if augment:\n dataset = dataset.map(data_augment(label), num_parallel_calls=tf.data.AUTOTUNE)\n if shuffle:\n dataset = dataset.shuffle(batch_size, reshuffle_each_iteration=True)\n if repeat:\n dataset = dataset.repeat()\n if cache:\n dataset = dataset.cache()\n \n dataset = dataset.batch(batch_size, drop_remainder=shuffle)\n dataset = dataset.prefetch(tf.data.AUTOTUNE)\n return dataset","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:08.516131Z","iopub.execute_input":"2023-01-18T16:12:08.516501Z","iopub.status.idle":"2023-01-18T16:12:08.535322Z","shell.execute_reply.started":"2023-01-18T16:12:08.516455Z","shell.execute_reply":"2023-01-18T16:12:08.534308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 14. Define Input Features ","metadata":{}},{"cell_type":"code","source":"input_features = train_df.columns[~train_df.columns.isin(['patient_id', 'image_id', 'site_id', 'machine_id',\n 'width', 'height', 'cancer', 'age',\n 'stratify', 'image_path', 'fold'])]","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:08.536899Z","iopub.execute_input":"2023-01-18T16:12:08.537177Z","iopub.status.idle":"2023-01-18T16:12:08.555965Z","shell.execute_reply.started":"2023-01-18T16:12:08.537146Z","shell.execute_reply":"2023-01-18T16:12:08.554814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 15. Augumentation ","metadata":{}},{"cell_type":"code","source":"def show_images(dataset, title=\"\", rows=1, cols=5, figsize=(20, 8)):\n fig = plt.figure(figsize=figsize)\n grid = ImageGrid(fig, 111, nrows_ncols=(rows, cols), axes_pad=0.2)\n for i, (images, labels) in enumerate(dataset.take(1)):\n for j in range(rows*cols):\n grid[j].imshow(images['input_image'][j].numpy())\n grid[j].set_title(f\"Label: {labels[j].numpy()}\")\n plt.suptitle(title)\n plt.show()\n\nsample_with_augument = build_dataset(train_df, input_features, batch_size=config.parameters['batch_size'], label=True, shuffle=False, augment=True, repeat=False, cache=False, ext=config.img_ext)\nsample_without_augument = build_dataset(train_df, input_features, batch_size=config.parameters['batch_size'], label=True, shuffle=False, augment=False, repeat=False, cache=False, ext=config.img_ext)\n\nshow_images(sample_without_augument, title=\"Without Augumentation\")\nshow_images(sample_with_augument, title=\"With Augumentation\")","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:08.55743Z","iopub.execute_input":"2023-01-18T16:12:08.558023Z","iopub.status.idle":"2023-01-18T16:12:34.428886Z","shell.execute_reply.started":"2023-01-18T16:12:08.55792Z","shell.execute_reply":"2023-01-18T16:12:34.428016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 16. Normalization ","metadata":{}},{"cell_type":"code","source":"train_df.to_csv('train_df.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:34.429916Z","iopub.execute_input":"2023-01-18T16:12:34.430188Z","iopub.status.idle":"2023-01-18T16:12:34.878034Z","shell.execute_reply.started":"2023-01-18T16:12:34.430152Z","shell.execute_reply":"2023-01-18T16:12:34.877185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df[input_features] = (train_df[input_features] - train_df[input_features].mean()) / train_df[input_features].std()\ntrain_df[input_features] = train_df[input_features].astype('float32')","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:34.879306Z","iopub.execute_input":"2023-01-18T16:12:34.880291Z","iopub.status.idle":"2023-01-18T16:12:34.950255Z","shell.execute_reply.started":"2023-01-18T16:12:34.880244Z","shell.execute_reply":"2023-01-18T16:12:34.949445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 17. Custom Metrics ","metadata":{}},{"cell_type":"code","source":"def p_f1(y_true, y_pred):\n y_true = tf.cast(y_true, tf.float32)\n y_pred = tf.cast(y_pred, tf.float32)\n \n tp = tf.reduce_sum(y_true * y_pred)\n tn = tf.reduce_sum((1 - y_true) * (1 - y_pred))\n fp = tf.reduce_sum((1 - y_true) * y_pred)\n fn = tf.reduce_sum(y_true * (1 - y_pred))\n \n p = tp / (tp + fp + tf.keras.backend.epsilon())\n r = tp / (tp + fn + tf.keras.backend.epsilon())\n \n f1 = 2 * p * r / (p + r + tf.keras.backend.epsilon())\n f1 = tf.where(tf.math.is_nan(f1), tf.zeros_like(f1), f1)\n\n return tf.reduce_mean(f1)\n\ndef p_f1_np(y_true, y_pred):\n y_true = y_true.astype(np.float32)\n y_pred = y_pred.astype(np.float32)\n \n tp = np.sum(y_true * y_pred)\n tn = np.sum((1 - y_true) * (1 - y_pred))\n fp = np.sum((1 - y_true) * y_pred)\n fn = np.sum(y_true * (1 - y_pred))\n \n p = tp / (tp + fp + np.finfo(np.float32).eps)\n r = tp / (tp + fn + np.finfo(np.float32).eps)\n \n f1 = 2 * p * r / (p + r + np.finfo(np.float32).eps)\n f1 = np.where(np.isnan(f1), np.zeros_like(f1), f1)\n\n return np.mean(f1)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:34.951303Z","iopub.execute_input":"2023-01-18T16:12:34.952139Z","iopub.status.idle":"2023-01-18T16:12:34.963713Z","shell.execute_reply.started":"2023-01-18T16:12:34.952102Z","shell.execute_reply":"2023-01-18T16:12:34.962718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 18. Build Model ","metadata":{}},{"cell_type":"code","source":"def build_model(input_features, loss=config.parameters['loss'], dropout=config.parameters['dropout'], \n optimizer=config.parameters['optimizer'], img_size=config.img_size):\n with strategy.scope():\n inputs = tf.keras.layers.Input(shape=img_size+(3,), name='input_image')\n features = tf.keras.layers.Input(shape=[len(input_features)], name='input_features')\n x = tf.keras.applications.EfficientNetB3(input_shape=img_size+(3,), include_top=False, \n drop_connect_rate=0.4, weights=config.weights)(inputs)\n x = tf.keras.layers.GlobalAveragePooling2D()(x)\n x = tf.keras.layers.Dropout(dropout)(x)\n x = tf.keras.layers.Dense(32, activation=\"relu\")(x)\n x = tf.keras.layers.BatchNormalization()(x)\n x = tf.keras.layers.Dropout(dropout)(x)\n x = tf.keras.layers.Concatenate()([x, features])\n x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n model = tf.keras.Model(inputs=[inputs, features], outputs=x)\n model.compile(optimizer=optimizer, loss=loss, metrics=['accuracy', p_f1, tf.keras.metrics.AUC(name='auc')])\n return model\n\nmodel = build_model(input_features)\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:34.96491Z","iopub.execute_input":"2023-01-18T16:12:34.965374Z","iopub.status.idle":"2023-01-18T16:12:57.179847Z","shell.execute_reply.started":"2023-01-18T16:12:34.96534Z","shell.execute_reply":"2023-01-18T16:12:57.176054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.utils.plot_model(model, show_shapes=True, dpi=64)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:57.183697Z","iopub.execute_input":"2023-01-18T16:12:57.184064Z","iopub.status.idle":"2023-01-18T16:12:57.436961Z","shell.execute_reply.started":"2023-01-18T16:12:57.184029Z","shell.execute_reply":"2023-01-18T16:12:57.435936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 19. Learning Rate Scheduler ","metadata":{}},{"cell_type":"code","source":"def lr_scheduler(batch_size=config.parameters['batch_size'], plot=False):\n \n lr_start = 0.000002\n lr_max = 0.00000105 * batch_size\n lr_min = 0.000001\n lr_ramp_ep = 6\n lr_sus_ep = 0\n lr_decay = 0.8\n \n def lr_fn(epoch):\n if epoch < lr_ramp_ep:\n lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n elif epoch < lr_ramp_ep + lr_sus_ep:\n lr = lr_max\n else:\n lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n return lr\n \n if plot:\n plt.figure(figsize=(10, 5))\n rng = [i for i in range(config.parameters['epochs']+1)]\n y = [lr_fn(x) for x in rng]\n fig = go.Figure()\n fig.add_trace(go.Scatter(x=rng, y=y, mode='lines+markers', name='lr'))\n fig.update_yaxes(tickformat=\".5f\")\n fig.update_layout(title='Learning Rate Schedule', xaxis_title='Epoch', yaxis_title='Learning Rate')\n fig.show()\n\n \n lr_callback = tf.keras.callbacks.LearningRateScheduler(lr_fn, verbose=False)\n return lr_callback\n\n\n_ = lr_scheduler(batch_size=config.parameters['batch_size'], plot=True)","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:57.439241Z","iopub.execute_input":"2023-01-18T16:12:57.439534Z","iopub.status.idle":"2023-01-18T16:12:57.475415Z","shell.execute_reply.started":"2023-01-18T16:12:57.439499Z","shell.execute_reply":"2023-01-18T16:12:57.474623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 20. Callbacks ","metadata":{}},{"cell_type":"code","source":"def get_callbacks(batch_size, fold, patience=config.parameters['patience']):\n early = tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\", mode=\"min\", patience=patience, verbose=1, restore_best_weights=True)\n best_model = tf.keras.callbacks.ModelCheckpoint(f'/kaggle/working/models/model_{fold}.h5', \n monitor='val_p_f1', mode='max', save_freq='epoch', \n save_best_only=True, save_weights_only=False, verbose=1)\n callbacks_list = [early, best_model, lr_scheduler(batch_size)]\n return callbacks_list","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:57.476924Z","iopub.execute_input":"2023-01-18T16:12:57.47786Z","iopub.status.idle":"2023-01-18T16:12:57.485476Z","shell.execute_reply.started":"2023-01-18T16:12:57.477812Z","shell.execute_reply":"2023-01-18T16:12:57.484686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 21. Train Model ","metadata":{}},{"cell_type":"code","source":"results = {}\n\nfor fold in range(config.folds):\n \n if fold not in config.validation_folds:\n continue\n \n if config.comet:\n experiment = Experiment(api_key = config.api_key,\n project_name = f\"{config.project_name}-{fold}\",\n workspace = config.workspace)\n \n print(f'\\n- Fold [{fold}] as validation set, and folds {[i for i in range(config.folds) if i != fold]} as training set\\n')\n \n train = train_df.query(\"fold != @fold\")\n validation = train_df.query(\"fold == @fold\")\n\n if config.oversampling:\n positive = train.query('cancer == 1').sample(frac=config.oversampling_factor, replace=True, random_state=config.seed)\n negative = train.query('cancer == 0')\n train = pd.concat([positive, negative], axis=0).reset_index(drop=True)\n \n train_dataset = build_dataset(train, input_features, batch_size=config.parameters['batch_size'], \n shuffle=True, augment=True, repeat=False, cache=True)\n \n val_dataset = build_dataset(validation, input_features, batch_size=config.parameters['batch_size'], \n shuffle=False, augment=False, repeat=False, cache=True)\n \n class_weight = compute_class_weight(class_weight='balanced',\n classes=train[\"cancer\"].unique(),\n y=train[\"cancer\"].values)\n \n model = build_model(input_features)\n \n history = model.fit(train_dataset, \n validation_data = val_dataset, \n epochs = config.parameters['epochs'],\n callbacks = get_callbacks(config.parameters['batch_size'], fold),\n class_weight = dict(zip(train[\"cancer\"].unique(), class_weight)),\n steps_per_epoch = len(train) // config.parameters['batch_size'])\n \n print('- Performing out-of-fold predictions...')\n print('- Loading best model...')\n model.load_weights(f'/kaggle/working/models/model_{fold}.h5')\n\n print('- Predicting OOF validation set...')\n b_val_dataset = build_dataset(validation, input_features, batch_size=config.parameters['batch_size'], \n shuffle=False, augment=True, repeat=False, cache=True)\n \n validation.loc[:, 'pred'] = model.predict(b_val_dataset, verbose=1).astype('float32')\n validation_bin = validation.loc[:, 'pred'].apply(lambda x: 1 if x > config.threshold else 0)\n validation.loc[:, 'pred'] = validation_bin\n\n if config.comet:\n print('- Log Confusion Matrix...')\n experiment.log_confusion_matrix(y_true=validation['cancer'].values, y_predicted=validation['pred'].values, \n title=f'confusion_matrix')\n\n if config.comet:\n print('- Log ROC Curve...')\n for i in range(len(validation['cancer'].unique())):\n fpr, tpr, _ = roc_curve(validation['cancer'], validation['pred'])\n experiment.log_curve(f'roc_curve_{i}', fpr, tpr, step=0)\n\n\n print('- Compute metrics...')\n pf = p_f1_np(validation['cancer'], validation['pred'])\n auc = roc_auc_score(validation['cancer'], validation['pred'])\n accuracy = accuracy_score(validation['cancer'], validation['pred'])\n\n if config.comet:\n print('- Log final metrics...')\n experiment.log_metric(f'p_f1_oof', pf)\n experiment.log_metric(f'auc_oof', auc)\n experiment.log_metric(f'accuracy_oof', accuracy)\n\n results[fold] = {'p_f1': pf, 'auc': auc, 'accuracy': accuracy}\n print(f'- P-F1: {pf}, AUC: {auc}, Accuracy: {accuracy}\\n')\n \n del model\n gc.collect()\n tf.keras.backend.clear_session()\n \n experiment.end()\n ","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:12:57.48717Z","iopub.execute_input":"2023-01-18T16:12:57.4875Z","iopub.status.idle":"2023-01-18T16:55:53.543549Z","shell.execute_reply.started":"2023-01-18T16:12:57.487467Z","shell.execute_reply":"2023-01-18T16:55:53.54163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('\\nFinal results:')\nprint(f'- P-F1: {np.mean([results[i][\"p_f1\"] for i in results])}')\nprint(f'- AUC: {np.mean([results[i][\"auc\"] for i in results])}')\nprint(f'- Accuracy: {np.mean([results[i][\"accuracy\"] for i in results])}')","metadata":{"execution":{"iopub.status.busy":"2023-01-18T16:55:53.54505Z","iopub.status.idle":"2023-01-18T16:55:53.545587Z","shell.execute_reply.started":"2023-01-18T16:55:53.545295Z","shell.execute_reply":"2023-01-18T16:55:53.545323Z"},"trusted":true},"execution_count":null,"outputs":[]}]}