{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🎗️[Training] RSNA BCD - Keras Model\n---\nIn this notebook, I will provide a comprehensive overview of the process of creating a baseline model for the [**RSNA Breast Cancer competition**](https://www.kaggle.com/competitions/rsna-breast-cancer-detection). The images used in this notebook have already undergone preprocessing, and we will now use them to train our model. They are TFRecords, this format is faster than PNG to read. This notebook will serve as a valuable resource for anyone interested in learning about the process of creating a baseline model for a medical imaging competition.\n\n### TPU\nTo perform the heavy computations, we will be using the TPU v3-8 device provided by Kaggle. Check the [Kaggle documentation](https://www.kaggle.com/docs/tpu) for the use of TPUs.\n\n### Steps\n- Configure the **operating device**\n- Define the **configuration parameters**\n- Split the data into **train** and **validation** sets\n- Create the TensorFlow **datasets**\n- Define the **optimizer**, **loss** and **metrics**\n- **Build** and **compile** the model\n- Define the **learning rate scheduler**\n- **Fit** the model\n- **Save** the model\n\n### Previous notebooks\nYou can also have a look at my previous work on the same topic:\n- [🎗️[EDA] RSNA Breast Cancer Detection](https://www.kaggle.com/code/paulbacher/eda-rsna-breast-cancer-detection)\n- [🎗️[Custom Preprocessor] RSNA Breast Cancer](https://www.kaggle.com/code/paulbacher/custom-preprocessor-rsna-breast-cancer)\n\n### Versions\n(The scores are obtained with 80% of the data for training)\n\n**V1: Score: 0.37**\n- 1024x512 images\n- Random brightness, contrast, JPEG quality, cropping\n- LR scheduler callback (cosine)\n- 15 epochs, 80% training\n\n**V2: Score: 0.48**\n- 1456x728 TFRecords (20GB JPEG with 97% quality)\n- Early stopping callback\n- Weight decay callback\n\n**V3: Score 0.44 (no improvement)**\n- Run 20 epochs\n- Early stopping waiting 3 -> 5 epochs\n- Change filter 66% -> 75% of classe 0\n- Change class weights to balanced point\n\n**V4: Score 0.39 (no improvement) Less gap between val/test**\n- Back to V2 and add coarse dropout size: 70, nb: 10\n- 20 epochs\n- LR from 5e-6 to 3.75e-6\n\n**V5:**\n- Tunned augmentation parameters\n- Reduced coarse dropout\n\n### References\n- Mark Wijkhuizen - [RSNA ConvNextV2 Training Tensorflow TPU](https://www.kaggle.com/code/markwijkhuizen/rsna-efficientnetv2-training-tensorflow-tpu/notebook)\n- Awsaf - [RSNA-BCD: EfficientNet [TF][TPU-1VM][Train]](https://www.kaggle.com/code/awsaf49/rsna-bcd-efficientnet-tf-tpu-1vm-train)","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"# Install the attention models package\n!pip install -qq /kaggle/input/keras-cv-attention-models/keras_cv_attention_models-1.3.9-py3-none-any.whl\n\n# Standard libraries\nimport os\nimport sys\nimport time\n\n# Data manipulation and visualization libraries\nimport numpy as np\nimport pandas as pd\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\n\n# Deep learning libraries\nimport tensorflow as tf\nimport tensorflow_addons as tfa\n\n# Support libraries\nimport pickle\nimport math\nimport random\nfrom sklearn.model_selection import train_test_split\nfrom kaggle_datasets import KaggleDatasets\nfrom keras_cv_attention_models import convnext\nfrom keras_cv_attention_models import resnet_family\nfrom multiprocessing import cpu_count\nfrom tqdm.notebook import tqdm\n\n# Image processing libraries\nimport cv2\n\n# Miscellaneous libraries\nimport gc\nimport datetime\n\n# IPython display library\nfrom IPython.display import clear_output\nclear_output()\n\nprint(f'Tensorflow Version: {tf.version}')\nprint(f'Python Version: {sys.version}')","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:47:15.907149Z","iopub.execute_input":"2023-02-09T11:47:15.907663Z","iopub.status.idle":"2023-02-09T11:47:24.184958Z","shell.execute_reply.started":"2023-02-09T11:47:15.907618Z","shell.execute_reply":"2023-02-09T11:47:24.183775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install keras-tuner --upgrade","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:23:30.598433Z","iopub.execute_input":"2023-02-09T11:23:30.5989Z","iopub.status.idle":"2023-02-09T11:23:39.604819Z","shell.execute_reply.started":"2023-02-09T11:23:30.598862Z","shell.execute_reply":"2023-02-09T11:23:39.603822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import keras_tuner\nfrom tensorflow import keras","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:24:49.150825Z","iopub.execute_input":"2023-02-09T11:24:49.151177Z","iopub.status.idle":"2023-02-09T11:24:49.208154Z","shell.execute_reply.started":"2023-02-09T11:24:49.151141Z","shell.execute_reply":"2023-02-09T11:24:49.207401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Seed\nA seed allows you to reproduce the same results.","metadata":{}},{"cell_type":"code","source":"SEED = 42\ndef seeding(seed=SEED):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    random.seed(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T11:22:53.766336Z","iopub.execute_input":"2023-02-09T11:22:53.766641Z","iopub.status.idle":"2023-02-09T11:22:53.772262Z","shell.execute_reply.started":"2023-02-09T11:22:53.766608Z","shell.execute_reply":"2023-02-09T11:22:53.771279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TPU Config","metadata":{}},{"cell_type":"code","source":"# Detect hardware, return appropriate distribution strategy\ntry:\n    TPU = tf.distribute.cluster_resolver.TPUClusterResolver()  # TPU detection. No parameters necessary if TPU_NAME environment variable is set. On Kaggle this is always the case.\n    print('Running on TPU ', TPU.master())\nexcept ValueError:\n    print('Running on CPU or GPU')\n    TPU = None\n\nif TPU:\n    IS_TPU = True\n    tf.config.experimental_connect_to_cluster(TPU)\n    tf.tpu.experimental.initialize_tpu_system(TPU)\n    STRATEGY = tf.distribute.experimental.TPUStrategy(TPU)\nelse:\n    IS_TPU = False\n    STRATEGY = tf.distribute.get_strategy() # default distribution strategy in Tensorflow. Works on CPU and single GPU.\n\nclear_output()\nN_REPLICAS = STRATEGY.num_replicas_in_sync\nprint(f'N_REPLICAS: {N_REPLICAS}, IS_TPU: {IS_TPU}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T11:22:55.812205Z","iopub.execute_input":"2023-02-09T11:22:55.812513Z","iopub.status.idle":"2023-02-09T11:23:00.5621Z","shell.execute_reply.started":"2023-02-09T11:22:55.812479Z","shell.execute_reply":"2023-02-09T11:23:00.561248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# GCS Path\n- The **GCS path** is required if using **remote TPU** (TPU v3-8)\n- It is not required if using a **local TPU** (TPU VM v3-8)\n\nWe will need to get the GCS path in this notebook","metadata":{}},{"cell_type":"code","source":"SOURCE_PATH = KaggleDatasets().get_gcs_path(\n    'dataset-rsna-bcd-1456x728-final-tfrecords')","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:23:01.920852Z","iopub.execute_input":"2023-02-09T11:23:01.921173Z","iopub.status.idle":"2023-02-09T11:23:02.444202Z","shell.execute_reply.started":"2023-02-09T11:23:01.921143Z","shell.execute_reply":"2023-02-09T11:23:02.443288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Mixed precision policy\n- Speed up training time by using `float16` as computation dtype with CPU and GPU\n- TPU use `bfloat16` if mixed precision is used\n- The mixed precision could slightly affect the accuracy of the model\n- If not used, computation and variables have default `float32` dtypes.\n\nIn this notebook, we do not use mixed precision. TPU are powerful enough to deal with float32 dtypes.","metadata":{}},{"cell_type":"code","source":"MIXED_PRECISION = False\nDEVICE = 'TPU'\n\nif MIXED_PRECISION:\n    if 'TPU' in DEVICE:\n        policy_type = 'mixed_bfloat16'\n    else:\n        policy_type = 'mixed_float16'\nelse:\n    policy_type = 'float32'\npolicy = tf.keras.mixed_precision.Policy(policy_type)\ntf.keras.mixed_precision.set_global_policy(policy)\nprint(f'Computation dtype: {tf.keras.mixed_precision.global_policy().compute_dtype}')\nprint(f'Variable dtype: {tf.keras.mixed_precision.global_policy().variable_dtype}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T11:23:06.409428Z","iopub.execute_input":"2023-02-09T11:23:06.409722Z","iopub.status.idle":"2023-02-09T11:23:06.417192Z","shell.execute_reply.started":"2023-02-09T11:23:06.409694Z","shell.execute_reply":"2023-02-09T11:23:06.416066Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Parameters","metadata":{}},{"cell_type":"code","source":"DEBUG = False\n\n# Image dimensions\nIMG_HEIGHT = 1456\nIMG_WIDTH = 728\nN_CHANNELS = 1\nINPUT_SHAPE = (IMG_HEIGHT, IMG_WIDTH, 1)\nN_SAMPLES_TFRECORDS = 548\n\n# Peak Learning Rate\nEPOCHS = 15\nWARMUP = 0\nLR_MAX = 5e-6 * N_REPLICAS\nCYCLE = 0.5\nWD_RATIO = 0.01\n\n# Batch size\nBATCH_SIZE = 8 * N_REPLICAS\n\n# Augmentation\nBRIGHTNESS = 0.10\nCONTRAST = (0.90, 1.10)\nJPEG_QUALITY = (90, 100)\nCROP_RATIO = (0.80, 1.00)\n\n# Coarse dropout\nCOARSE_DROPOUT = False\nCHUNK_SIZE = 35\nNB_CHUNKS = 20\n\nIS_INTERACTIVE = os.environ['KAGGLE_KERNEL_RUN_TYPE'] == 'Interactive'\nVERBOSE = 1 if IS_INTERACTIVE else 2\n\n# Tensorflow AUTO flag\nAUTO = tf.data.experimental.AUTOTUNE\n\nprint(f'BATCH_SIZE: {BATCH_SIZE}')","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:23:11.518536Z","iopub.execute_input":"2023-02-09T11:23:11.518873Z","iopub.status.idle":"2023-02-09T11:23:11.529011Z","shell.execute_reply.started":"2023-02-09T11:23:11.51884Z","shell.execute_reply":"2023-02-09T11:23:11.528083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train-validation sets\nIt is possible to specify to use cross validation, a single train/val split or the training on the full data.\nCross validation will fit 5 models and average them for the evaluation, which will need more training time.\n\n**Note:** For the final training of the model, it is better to reuse all the data.","metadata":{}},{"cell_type":"code","source":"# TFRecord file paths\ntfrecords = sorted(tf.io.gfile.glob(f'{SOURCE_PATH}/*.tfrecords'))\nprint(f'Found {len(tfrecords)} TFRecords')\n\n# Train Test Split\ntrain_tfrecords, valid_tfrecords = train_test_split(\n    tfrecords,\n    train_size=0.80,\n    random_state=SEED,\n    shuffle=True)\nprint(f'# Train TFRecords: {len(train_tfrecords)}, # Valid TFRecords: {len(valid_tfrecords)}')","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:24:56.089982Z","iopub.execute_input":"2023-02-09T11:24:56.090803Z","iopub.status.idle":"2023-02-09T11:24:56.190015Z","shell.execute_reply.started":"2023-02-09T11:24:56.090767Z","shell.execute_reply":"2023-02-09T11:24:56.188978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utility Function","metadata":{}},{"cell_type":"code","source":"def tf_rand_int(minval, maxval, dtype=tf.int64):\n    minval = tf.cast(minval, dtype)\n    maxval = tf.cast(maxval, dtype)\n    return tf.random.uniform(shape=(), minval=minval, maxval=maxval, dtype=dtype)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:24:59.072277Z","iopub.execute_input":"2023-02-09T11:24:59.072627Z","iopub.status.idle":"2023-02-09T11:24:59.077946Z","shell.execute_reply.started":"2023-02-09T11:24:59.072589Z","shell.execute_reply":"2023-02-09T11:24:59.076815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Datasets\nThe TFRecords files are composed of 547 or 548 samples. Each sample is an encoded string containing the features and labels. When creating the dataset, this string has to be decoded to recover the image and the label of each sample. Then, for the train set, the images are filtered in order to undersample the negative examples. The model will see more positive examples. The images are also augmented and suffled.\n\nAugmentation consists in applying:\n- Random brightness\n- Random contrast\n- Random JPEG quality\n- Random cropping","metadata":{}},{"cell_type":"code","source":"def create_dataset(tfrecords, batch_size=BATCH_SIZE,\n                   valid=False, debug=DEBUG):\n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = False\n    # Create dataset path/label\n    dataset = tf.data.TFRecordDataset(tfrecords,\n                                      num_parallel_reads=AUTO,\n                                      compression_type='GZIP')\n    # Decode mapping\n    dataset = dataset.map(decoder, num_parallel_calls=AUTO)\n    # Val/Debug cases\n    if not valid:\n        dataset = dataset.filter(undersample_majority)\n        if COARSE_DROPOUT:\n            dataset = dataset.map(coarse_dropout, num_parallel_calls=AUTO)\n        dataset = dataset.map(augmenter, num_parallel_calls=AUTO)\n        dataset = dataset.with_options(ignore_order)\n        if not debug:\n            dataset = dataset.shuffle(1024)\n        dataset = dataset.repeat()\n    dataset = dataset.batch(BATCH_SIZE, drop_remainder=not valid)\n    dataset = dataset.prefetch(AUTO)\n    return dataset\n\ndef decoder(record_bytes):\n    features = tf.io.parse_single_example(record_bytes, {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'target': tf.io.FixedLenFeature([], tf.int64),\n        'patient_id': tf.io.FixedLenFeature([], tf.int64)})\n    img = tf.io.decode_png(features['image'], channels=N_CHANNELS)\n    img = tf.reshape(img, [IMG_HEIGHT, IMG_WIDTH, N_CHANNELS])\n    label = features['target']\n    return img, label\n\ndef undersample_majority(img, label):\n    return label == 1 or tf.random.uniform([]) > 2/3\n\ndef coarse_dropout(img, label):\n    for k in range(NB_CHUNKS):\n        x = tf.cast(tf.random.uniform([], 0, IMG_WIDTH), tf.int32)\n        y = tf.cast(tf.random.uniform([], 0, IMG_HEIGHT), tf.int32)\n        size = tf.cast(CHUNK_SIZE, tf.int32)\n        xa = tf.math.maximum(0, x - size // 2)\n        ya = tf.math.maximum(0, y - size // 2)\n        xb = tf.math.minimum(x + size // 2, IMG_WIDTH)\n        yb = tf.math.minimum(y + size // 2, IMG_HEIGHT)\n        one = img[ya : yb, 0 : xa, :]\n        two = tf.zeros([yb - ya, xb - xa, 1], dtype=img.dtype) \n        three = img[ya : yb, xb : IMG_WIDTH, :]\n        middle = tf.concat([one, two, three], axis=1)\n        img = tf.concat([img[0 : ya, :, :], middle, img[yb : IMG_HEIGHT, :, :]], axis=0)\n        img = tf.reshape(img, [IMG_HEIGHT, IMG_WIDTH, 1])\n    return img, label\n\ndef augmenter(img, label):\n    # Pixels\n    img = tf.image.random_brightness(img, BRIGHTNESS)\n    img = tf.image.random_contrast(img, *CONTRAST)\n    img = tf.image.random_jpeg_quality(img, *JPEG_QUALITY)\n    # Crop\n    ratio = tf.random.uniform([], *CROP_RATIO)\n    img_height_crop = tf.cast(ratio * IMG_HEIGHT, tf.int32)\n    img_width_crop = tf.cast(ratio * IMG_WIDTH, tf.int32)\n    img_height_offset = tf_rand_int(0, IMG_HEIGHT - img_height_crop)\n    img_width_offset = 0\n    img = tf.slice(img, [img_height_offset, img_width_offset, 0],\n                   [img_height_crop, img_width_crop, N_CHANNELS])\n    img = tf.image.resize(img, [IMG_HEIGHT, IMG_WIDTH])\n    # Clip\n    img = tf.clip_by_value(img, 0, 255)\n    img = tf.cast(img, tf.uint8)\n    return img, label","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:25:03.685286Z","iopub.execute_input":"2023-02-09T11:25:03.685643Z","iopub.status.idle":"2023-02-09T11:25:03.707206Z","shell.execute_reply.started":"2023-02-09T11:25:03.685611Z","shell.execute_reply":"2023-02-09T11:25:03.706555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The samples in each dataset will be fed into the model in batches correponding to one epoch step. We must calculate the number of steps per epochs for both train and validation sets.","metadata":{}},{"cell_type":"code","source":"# Get Train/Validation datasets\ntrain_dataset = create_dataset(train_tfrecords, valid=False, debug=DEBUG)\nvalid_dataset = create_dataset(valid_tfrecords, valid=True, debug=DEBUG)\nTRAIN_STEPS_PER_EPOCH = len(train_tfrecords) * N_SAMPLES_TFRECORDS // BATCH_SIZE\nVAL_STEPS_PER_EPOCH = len(valid_tfrecords) * N_SAMPLES_TFRECORDS // BATCH_SIZE\nprint(f'TRAIN_STEPS_PER_EPOCH: {TRAIN_STEPS_PER_EPOCH}, VAL_STEPS_PER_EPOCH: {VAL_STEPS_PER_EPOCH}')","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:25:06.525225Z","iopub.execute_input":"2023-02-09T11:25:06.525569Z","iopub.status.idle":"2023-02-09T11:25:06.954264Z","shell.execute_reply.started":"2023-02-09T11:25:06.525528Z","shell.execute_reply":"2023-02-09T11:25:06.953459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we check that the inputs and outputs have good sizes and data types.","metadata":{}},{"cell_type":"code","source":"# Sanity checking\ndef check_dataset(dataset):\n    image, label = next(iter(dataset))\n    image = image.numpy()\n    clear_output()\n    print(f\"X_batch shape: {image.shape}, y_batch shape: {label.shape}\")\n    print(f\"X_batch dtype: {image.dtype}, y_batch dtype: {label.dtype}\")\n    print(f\"X_batch min: {image.min():.2f}, max: {image.max():.2f}\")\n\ncheck_dataset(train_dataset)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:25:11.626199Z","iopub.execute_input":"2023-02-09T11:25:11.626492Z","iopub.status.idle":"2023-02-09T11:25:18.488557Z","shell.execute_reply.started":"2023-02-09T11:25:11.626463Z","shell.execute_reply":"2023-02-09T11:25:18.487563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Display batch\nLet's look at a batch example of the train set.","metadata":{}},{"cell_type":"code","source":"def display_batch(dataset):\n    images, labels = next(iter(dataset))\n    clear_output()\n    plt.figure(figsize=(20, BATCH_SIZE / 2))\n    for i in range(BATCH_SIZE):\n        plt.subplot(int(BATCH_SIZE / 8), 8, i + 1)\n        plt.title(f\"Cancer: {int(labels[i])}\", fontsize=12)\n        plt.xticks([])\n        plt.yticks([])\n        plt.imshow(images[i], cmap='jet')\n    plt.show()\n\ndisplay_batch(train_dataset)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:25:26.783861Z","iopub.execute_input":"2023-02-09T11:25:26.784199Z","iopub.status.idle":"2023-02-09T11:25:41.097513Z","shell.execute_reply.started":"2023-02-09T11:25:26.784165Z","shell.execute_reply":"2023-02-09T11:25:41.096542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Competition metric\nThe evaluation metric used in this competition is the **probabilistic F1 score (pF1)**, which is a variation of the traditional F1 score that takes into account the probabilities of predictions rather than binary classifications.\n\nAdditionally, the pF1 score also includes a version that allows for a greater emphasis on precision over recall by adjusting the beta parameter. The general formula for this version is as follows:\n\n$$ pF_{\\beta} = (1 + \\beta^2) \\cdot \\frac{precision \\cdot recall}{(\\beta^2 \\cdot precision) + recall} $$\nWhere,\n$$ precision = \\frac{pTP}{pTP + pFP} \\hspace{1cm} recall = \\frac{pTP}{TP + FN} $$\n\n**Note:**\n\nWhen the beta parameter is set to a value less than 1, the model is rewarded for having a higher recall even if the precision is lower. On the other hand, when the beta parameter is greater than 1, the model is rewarded for having a higher precision even if the recall is lower. In this competition, the beta parameter is set to 1, which results in the probabilistic F1 score (pF1) which gives equal importance to precision and recall. The model will be rewarded for achieving a balance between the two.","metadata":{}},{"cell_type":"code","source":"class pFBeta(tf.keras.metrics.Metric):\n    def __init__(self, beta=1, name='pF1', **kwargs):\n        super().__init__(name=name, **kwargs)\n        self.beta = beta\n        self.epsilon = 1e-10\n        self.pos = self.add_weight(name='pos', initializer='zeros')\n        self.ctp = self.add_weight(name='ctp', initializer='zeros')\n        self.cfp = self.add_weight(name='cfp', initializer='zeros')\n\n    def update_state(self, y_true, y_pred, sample_weight=None):\n        y_true = tf.cast(y_true, tf.float32)\n        y_pred = tf.clip_by_value(y_pred, 0, 1)\n        pos = tf.cast(tf.reduce_sum(y_true), tf.float32)\n        ctp = tf.cast(tf.reduce_sum(y_pred[y_true == 1]), tf.float32)\n        cfp = tf.cast(tf.reduce_sum(y_pred[y_true == 0]), tf.float32)\n        self.pos.assign_add(pos)\n        self.ctp.assign_add(ctp)\n        self.cfp.assign_add(cfp)\n\n    def result(self):\n        beta2 = self.beta * self.beta\n        prec = self.ctp / (self.ctp + self.cfp + self.epsilon)\n        reca = self.ctp / (self.pos + self.epsilon)\n        return (1 + beta2) * prec * reca / (beta2 * prec + reca)\n\n    def reset_state(self):\n        self.pos.assign(0.)\n        self.ctp.assign(0.)\n        self.cfp.assign(0.)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:25:41.099142Z","iopub.execute_input":"2023-02-09T11:25:41.099589Z","iopub.status.idle":"2023-02-09T11:25:41.110103Z","shell.execute_reply.started":"2023-02-09T11:25:41.099555Z","shell.execute_reply":"2023-02-09T11:25:41.109385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\nIn the normalizer function, the images are:\n- **Duplicated** to get 3 channels required for the pretrained model input\n- **Converted** from `uint8` to `float32`\n- **Scaled** between 0 and 1 and then normalized with respect to the ImageNet dataset.","metadata":{}},{"cell_type":"code","source":"def normalizer(img):\n    img = tf.repeat(img, repeats=3, axis=3)\n    img = tf.cast(img, tf.float32)\n    img = tf.keras.applications.imagenet_utils.preprocess_input(img, mode='torch')\n    return img","metadata":{"execution":{"iopub.status.busy":"2023-02-09T11:25:50.283966Z","iopub.execute_input":"2023-02-09T11:25:50.285009Z","iopub.status.idle":"2023-02-09T11:25:50.291116Z","shell.execute_reply.started":"2023-02-09T11:25:50.284962Z","shell.execute_reply":"2023-02-09T11:25:50.290207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_model():\n    # Mixed Policy Settings\n    print(f'Compute dtype: {tf.keras.mixed_precision.global_policy().compute_dtype}')\n    print(f'Variable dtype: {tf.keras.mixed_precision.global_policy().variable_dtype}')\n\n    with STRATEGY.scope():\n        seeding()\n\n        # Architecture\n        inputs = tf.keras.Input(INPUT_SHAPE, name='inputs', dtype=tf.uint8)\n        x = normalizer(inputs)\n        x = resnet_family.ResNet50D(\n            input_shape=(IMG_HEIGHT, IMG_WIDTH, 3),\n            pretrained='imagenet',\n            num_classes=0)(x)\n        x = tf.keras.layers.GlobalAveragePooling2D()(x)\n        x = tf.keras.layers.Dropout(0.30)(x)\n        outputs = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n        model = tf.keras.Model(inputs, outputs)\n        \n        # Optimizer\n        optimizer = tfa.optimizers.AdamW(\n            learning_rate=LR_MAX,\n            weight_decay=LR_MAX * WD_RATIO,\n            epsilon=1e-6)\n        \n        # Loss\n        loss = tf.keras.losses.BinaryCrossentropy(from_logits=False)\n        \n        # Metrics\n        metrics = [pFBeta(beta=1, name='pF1'),\n                   tfa.metrics.F1Score(num_classes=1, threshold=0.50, name='F1'),\n                   tf.metrics.Precision(name='Prec'),\n                   tf.metrics.Recall(name='Reca'),\n                   tf.metrics.AUC(name='AUC'),\n                   tf.metrics.BinaryAccuracy(name='BinAcc')]\n        \n        # Compile\n        model.compile(optimizer=optimizer,\n                      loss=loss,\n                      metrics=metrics)\n        \n        #model.load_weights('/kaggle/input/rsna-weights-convnextv2/model.h5')\n        model.load_weights('/kaggle/working/first_model_resnet50d.h5')\n        return model\n\n    \ntf.keras.backend.clear_session()\ntf.config.optimizer.set_jit(True)\n\nmodel = create_model()","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:51:47.788241Z","iopub.execute_input":"2023-02-09T13:51:47.78921Z","iopub.status.idle":"2023-02-09T13:52:01.185302Z","shell.execute_reply.started":"2023-02-09T13:51:47.789158Z","shell.execute_reply":"2023-02-09T13:52:01.184419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now the model is created, let's see how it looks.\n\n**Note**: The biggest block is the `conv_v2_tiny` and it can be displayed by setting `expand_nested=True`.","metadata":{}},{"cell_type":"code","source":"tf.keras.utils.plot_model(model, show_shapes=True, show_dtype=True,\n                          show_layer_names=True, expand_nested=False)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:52:01.187806Z","iopub.execute_input":"2023-02-09T13:52:01.188145Z","iopub.status.idle":"2023-02-09T13:52:01.439632Z","shell.execute_reply.started":"2023-02-09T13:52:01.188102Z","shell.execute_reply":"2023-02-09T13:52:01.438527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Callbacks","metadata":{}},{"cell_type":"markdown","source":"## Define early stop to val_pF1 as it is the competition evaluation","metadata":{}},{"cell_type":"code","source":"es_callback = tf.keras.callbacks.EarlyStopping(\n    monitor='val_pF1',\n    patience=3,\n    restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T12:17:44.198276Z","iopub.execute_input":"2023-02-09T12:17:44.198651Z","iopub.status.idle":"2023-02-09T12:17:44.203972Z","shell.execute_reply.started":"2023-02-09T12:17:44.19861Z","shell.execute_reply":"2023-02-09T12:17:44.20305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## - Learning Rate Scheduler\nIt is common to control the learning rate during training and decrease it along epochs. Let's use a cosine function and display it.","metadata":{}},{"cell_type":"code","source":"def lr_fn(step, epochs=EPOCHS, warmup=WARMUP,\n          lr_max=LR_MAX, cycle=CYCLE):\n    if step < warmup:\n        lr = lr_max * 0.10 ** (warmup - step)\n    else:\n        progress = float(step - warmup) / float(max(1, epochs - warmup))\n        lr = max(0.0, 0.5 * (1.0 + np.cos(2.0 * np.pi * float(cycle) * progress))) * lr_max\n    return lr\n\n\ndef plot_lr_schedule(epochs=EPOCHS, warmup=WARMUP,\n                     lr_max=LR_MAX, cycle=CYCLE):\n    steps = np.array(range(epochs))\n    lrs = np.vectorize(lr_fn)(steps, epochs, warmup, lr_max, cycle)\n    plt.figure(figsize=(16, 8))\n    plt.plot(steps + 1, lrs, '-', color='green', lw=3)\n    plt.plot(steps + 1, lrs, 'o', color='black')\n    offset_y = (max(lrs) - min(lrs)) * 0.02\n    for x, y in zip(steps, lrs):\n        plt.annotate('%.1E' % y, xy=(x+1.1, y+offset_y))\n    plt.xticks(range(1, epochs + 1))\n    plt.grid(True)\n    plt.title(f\"LR schedule:\\nEpochs: {epochs} | Warmup: {warmup} | LR max: {lr_max} | Cycle: {cycle}\")\n    plt.show()\n\n\nplot_lr_schedule()","metadata":{"execution":{"iopub.status.busy":"2023-02-09T12:17:46.384634Z","iopub.execute_input":"2023-02-09T12:17:46.38498Z","iopub.status.idle":"2023-02-09T12:17:46.779282Z","shell.execute_reply.started":"2023-02-09T12:17:46.384948Z","shell.execute_reply":"2023-02-09T12:17:46.778332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lr_schedule = [lr_fn(step) for step in range(EPOCHS)]\nlr_callback = tf.keras.callbacks.LearningRateScheduler(\n    lambda step: lr_schedule[step], verbose=0)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T12:17:49.334388Z","iopub.execute_input":"2023-02-09T12:17:49.334668Z","iopub.status.idle":"2023-02-09T12:17:49.339971Z","shell.execute_reply.started":"2023-02-09T12:17:49.33464Z","shell.execute_reply":"2023-02-09T12:17:49.338997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## - Weight decay","metadata":{}},{"cell_type":"code","source":"class WeightDecayCallback(tf.keras.callbacks.Callback):\n    def __init__(self, wd_ratio=WD_RATIO):\n        self.step_counter = 0\n        self.wd_ratio = wd_ratio\n    \n    def on_epoch_begin(self, epoch, logs=None):\n        model.optimizer.weight_decay = model.optimizer.learning_rate * self.wd_ratio\n        print(f\"Learning rate: {model.optimizer.learning_rate.numpy():.2e}\")\n        print(f\"Weight decay: {model.optimizer.weight_decay.numpy():.2e}\")\n        \nwd_callback = WeightDecayCallback()","metadata":{"execution":{"iopub.status.busy":"2023-02-09T12:17:51.314066Z","iopub.execute_input":"2023-02-09T12:17:51.314404Z","iopub.status.idle":"2023-02-09T12:17:51.321095Z","shell.execute_reply.started":"2023-02-09T12:17:51.314367Z","shell.execute_reply":"2023-02-09T12:17:51.320148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{}},{"cell_type":"code","source":"MODEL_TYPE = \"resnet50d\"\ncheckpoint_filepath = f'./model_{MODEL_TYPE}_' + '{epoch:04d}.h5'\nbest_filepath = f'./model_{MODEL_TYPE}_' + 'best_pf1.h5'","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:58:01.045886Z","iopub.execute_input":"2023-02-09T13:58:01.046191Z","iopub.status.idle":"2023-02-09T13:58:01.050842Z","shell.execute_reply.started":"2023-02-09T13:58:01.046162Z","shell.execute_reply":"2023-02-09T13:58:01.050008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n                            filepath=checkpoint_filepath,\n                            save_weights_only=True,\n                            save_freq='epoch')\n","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:57:35.519392Z","iopub.execute_input":"2023-02-09T13:57:35.519723Z","iopub.status.idle":"2023-02-09T13:57:35.524923Z","shell.execute_reply.started":"2023-02-09T13:57:35.519689Z","shell.execute_reply":"2023-02-09T13:57:35.52386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_checkpoint_callback_pf1_best = tf.keras.callbacks.ModelCheckpoint(\n                                        filepath = best_filepath,\n                                        save_weights_only=True,\n                                        monitor='val_pF1',\n                                        mode='max',\n                                        save_best_only=True)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:58:32.249937Z","iopub.execute_input":"2023-02-09T13:58:32.250247Z","iopub.status.idle":"2023-02-09T13:58:32.256022Z","shell.execute_reply.started":"2023-02-09T13:58:32.250216Z","shell.execute_reply":"2023-02-09T13:58:32.254925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(\n    x=train_dataset,\n    epochs=EPOCHS,\n    verbose=VERBOSE,\n    callbacks=[es_callback, lr_callback, wd_callback, model_checkpoint_callback,model_checkpoint_callback_pf1_best],\n    validation_data=valid_dataset,\n    class_weight={0: 1, 1: 5},\n    steps_per_epoch=TRAIN_STEPS_PER_EPOCH)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:59:20.810493Z","iopub.execute_input":"2023-02-09T13:59:20.811176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save model weights for inference\nmodel.save_weights('first_model_resnet50d.h5')","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:39:17.547528Z","iopub.execute_input":"2023-02-09T13:39:17.547844Z","iopub.status.idle":"2023-02-09T13:39:19.251396Z","shell.execute_reply.started":"2023-02-09T13:39:17.547816Z","shell.execute_reply":"2023-02-09T13:39:19.250423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **F1 By Threshold Plot**","metadata":{}},{"cell_type":"code","source":"# Get true labels and predictions for validation set\ny_true_val = []\ny_pred_val = []\nfor X_batch, y_batch in tqdm(create_dataset(valid_tfrecords), total=VAL_STEPS_PER_EPOCH):\n    y_true_val += y_batch.numpy().tolist()\n    y_pred_val += model.predict_on_batch(X_batch).squeeze().tolist()","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:42:01.608936Z","iopub.execute_input":"2023-02-09T13:42:01.609304Z","iopub.status.idle":"2023-02-09T13:50:02.502654Z","shell.execute_reply.started":"2023-02-09T13:42:01.609266Z","shell.execute_reply":"2023-02-09T13:50:02.501213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# source: https://www.kaggle.com/code/sohier/probabilistic-f-score\n# Competition Leaderboard Metric\ndef pfbeta(labels, predictions, beta=1):\n    y_true_count = 0\n    ctp = 0\n    cfp = 0\n\n    for idx in range(len(labels)):\n        prediction = min(max(predictions[idx], 0), 1)\n        if (labels[idx]):\n            y_true_count += 1\n            ctp += prediction\n        else:\n            cfp += prediction\n\n    beta_squared = beta * beta\n    c_precision = ctp / (ctp + cfp)\n    c_recall = ctp / y_true_count\n    if (c_precision > 0 and c_recall > 0):\n        result = (1 + beta_squared) * (c_precision * c_recall) / (beta_squared * c_precision + c_recall)\n        return result\n    else:\n        return 0","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:50:05.920553Z","iopub.execute_input":"2023-02-09T13:50:05.920857Z","iopub.status.idle":"2023-02-09T13:50:05.928102Z","shell.execute_reply.started":"2023-02-09T13:50:05.920827Z","shell.execute_reply":"2023-02-09T13:50:05.927224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot pF1 by threshold plot to find best threshold\npf1_by_threshold = []\nthresholds = np.arange(0, 1.01, 0.01)\nfor t in tqdm(thresholds):\n    # Compute pF1 for each threshold\n    pf1_by_threshold.append(\n        pfbeta(y_true_val, y_pred_val > t)\n    )\n    \nplt.figure(figsize=(15,8))\nplt.title('F1 By Threshold', size=24)\nplt.plot(pf1_by_threshold, label='F1 Score')\n\n# Best threshold and pF1 score\narg_max = np.argmax(pf1_by_threshold)\nval_max = np.max(pf1_by_threshold)\nthreshold_best = thresholds[arg_max]\nplt.scatter(arg_max, val_max, color='red', label=f'Best Threshold {threshold_best:.2f}, pF1 Score: {val_max:.2f}')\n\n# Plot pF1 by Threshold\nplt.xticks(np.arange(0, 110, 10), [f'{t:.2f}' for t in np.arange(0, 1.1, 0.1)])\nplt.yticks(np.arange(0, 1.1, 0.1))\nplt.xlim(0, 100)\nplt.ylim(0, 1)\nplt.xlabel('Threshold')\nplt.ylabel('pF1 Score')\nplt.legend(fontsize=12)\nplt.grid()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:50:07.867041Z","iopub.execute_input":"2023-02-09T13:50:07.867396Z","iopub.status.idle":"2023-02-09T13:50:08.216948Z","shell.execute_reply.started":"2023-02-09T13:50:07.867349Z","shell.execute_reply":"2023-02-09T13:50:08.215743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plots","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/code/markwijkhuizen/rsna-convnextv2-training-tensorflow-tpu?scriptVersionId=116484001\ndef plot_history_metric(metric, f_best=np.argmax, ylim=None, yscale=None, yticks=None):\n    plt.figure(figsize=(20, 10))\n    \n    values = history.history[metric]\n    N_EPOCHS = len(values)\n    val = 'val' in ''.join(history.history.keys())\n    # Epoch Ticks\n    if N_EPOCHS <= 20:\n        x = np.arange(1, N_EPOCHS + 1)\n    else:\n        x = [1, 5] + [10 + 5 * idx for idx in range((N_EPOCHS - 10) // 5 + 1)]\n\n    x_ticks = np.arange(1, N_EPOCHS+1)\n\n    # Validation\n    if val:\n        val_values = history.history[f'val_{metric}']\n        val_argmin = f_best(val_values)\n        plt.plot(x_ticks, val_values, label=f'val')\n\n    # summarize history for accuracy\n    plt.plot(x_ticks, values, label=f'train')\n    argmin = f_best(values)\n    plt.scatter(argmin + 1, values[argmin], color='red', s=75, marker='o', label=f'train_best')\n    if val:\n        plt.scatter(val_argmin + 1, val_values[val_argmin], color='purple', s=75, marker='o', label=f'val_best')\n\n    plt.title(f'Model {metric}', fontsize=24, pad=10)\n    plt.ylabel(metric, fontsize=20, labelpad=10)\n\n    if ylim:\n        plt.ylim(ylim)\n\n    if yscale is not None:\n        plt.yscale(yscale)\n        \n    if yticks is not None:\n        plt.yticks(yticks, fontsize=16)\n\n    plt.xlabel('epoch', fontsize=20, labelpad=10)        \n    plt.tick_params(axis='x', labelsize=8)\n    plt.xticks(x, fontsize=16) # set tick step to 1 and let x axis start at 1\n    plt.yticks(fontsize=16)\n    \n    plt.legend(prop={'size': 10})\n    plt.grid()\n    plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T13:50:15.377866Z","iopub.execute_input":"2023-02-09T13:50:15.378693Z","iopub.status.idle":"2023-02-09T13:50:15.391471Z","shell.execute_reply.started":"2023-02-09T13:50:15.378652Z","shell.execute_reply":"2023-02-09T13:50:15.390548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history_metric('loss', f_best=np.argmin)","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:50:19.078567Z","iopub.execute_input":"2023-02-09T13:50:19.078907Z","iopub.status.idle":"2023-02-09T13:50:19.488758Z","shell.execute_reply.started":"2023-02-09T13:50:19.078869Z","shell.execute_reply":"2023-02-09T13:50:19.487747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history_metric('pF1', ylim=[0,1], yticks=np.arange(0.0, 1.1, 0.1))","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:50:25.955389Z","iopub.execute_input":"2023-02-09T13:50:25.955677Z","iopub.status.idle":"2023-02-09T13:50:26.281965Z","shell.execute_reply.started":"2023-02-09T13:50:25.955648Z","shell.execute_reply":"2023-02-09T13:50:26.280954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history_metric('F1', ylim=[0,1], yticks=np.arange(0.0, 1.1, 0.1))","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:50:35.309488Z","iopub.execute_input":"2023-02-09T13:50:35.309806Z","iopub.status.idle":"2023-02-09T13:50:35.645467Z","shell.execute_reply.started":"2023-02-09T13:50:35.309772Z","shell.execute_reply":"2023-02-09T13:50:35.644702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history_metric('Prec', ylim=[0,1], yticks=np.arange(0.0, 1.1, 0.1))","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:50:41.321564Z","iopub.execute_input":"2023-02-09T13:50:41.322518Z","iopub.status.idle":"2023-02-09T13:50:41.664814Z","shell.execute_reply.started":"2023-02-09T13:50:41.322471Z","shell.execute_reply":"2023-02-09T13:50:41.663696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history_metric('Reca', ylim=[0,1], yticks=np.arange(0.0, 1.1, 0.1))","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:50:56.86549Z","iopub.execute_input":"2023-02-09T13:50:56.86629Z","iopub.status.idle":"2023-02-09T13:50:57.229653Z","shell.execute_reply.started":"2023-02-09T13:50:56.866244Z","shell.execute_reply":"2023-02-09T13:50:57.228666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history_metric('AUC', ylim=[0,1], yticks=np.arange(0.0, 1.1, 0.1))","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:51:03.176096Z","iopub.execute_input":"2023-02-09T13:51:03.176433Z","iopub.status.idle":"2023-02-09T13:51:03.532828Z","shell.execute_reply.started":"2023-02-09T13:51:03.1764Z","shell.execute_reply":"2023-02-09T13:51:03.531494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history_metric('BinAcc', ylim=[0,1], yticks=np.arange(0.0, 1.1, 0.1))","metadata":{"execution":{"iopub.status.busy":"2023-02-09T13:51:07.960818Z","iopub.execute_input":"2023-02-09T13:51:07.961158Z","iopub.status.idle":"2023-02-09T13:51:08.304444Z","shell.execute_reply.started":"2023-02-09T13:51:07.961122Z","shell.execute_reply":"2023-02-09T13:51:08.303431Z"},"trusted":true},"execution_count":null,"outputs":[]}]}