{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n\nI will be following [this notebook](https://www.kaggle.com/code/aritrag/kerascv-starter-notebook-train) to understand how to train a **Convolutional Neural Network (CNN)** model using **Keras**.\n\n> Keras is a deep learning API, developed by Google, for implementing neural networks.\n\nTraditionally it runs on top of *TensorFlow*, but by using *KerasCV* (which is built on top of *Keras Core*), we can choose either *TensorFlow*, *JAX* or *PyTorch* as our backend. We choose **TensorFlow** in this notebook.\n\nThe context of this competition is detecting various trauma related injuries from CT scans. This is a Computer Vision task, and so *KerasCV* is a good choice.","metadata":{}},{"cell_type":"markdown","source":"## Ways to improve model\n\n- different augmentation layers\n- look up ways to improve CNN\n- take coursera courses on [Neural Networks & Deep Learning](https://www.coursera.org/learn/neural-networks-deep-learning) as well as [Convolutional Neural Networks](https://www.coursera.org/learn/convolutional-neural-networks#outcomes)\n","metadata":{"execution":{"iopub.status.busy":"2023-09-13T18:32:15.590305Z","iopub.execute_input":"2023-09-13T18:32:15.590767Z","iopub.status.idle":"2023-09-13T18:32:15.598422Z","shell.execute_reply.started":"2023-09-13T18:32:15.590735Z","shell.execute_reply":"2023-09-13T18:32:15.596592Z"}}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"! pip install -q git+https://github.com/keras-team/keras-cv","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:31:18.652677Z","iopub.execute_input":"2023-10-12T12:31:18.653047Z","iopub.status.idle":"2023-10-12T12:31:44.439349Z","shell.execute_reply.started":"2023-10-12T12:31:18.65301Z","shell.execute_reply":"2023-10-12T12:31:44.437997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nos.environ[\"KERAS_BACKEND\"] = \"tensorflow\"\n\nimport keras_cv\nimport keras_core as keras\nfrom keras_core import layers\n\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:31:44.442043Z","iopub.execute_input":"2023-10-12T12:31:44.442448Z","iopub.status.idle":"2023-10-12T12:31:57.267769Z","shell.execute_reply.started":"2023-10-12T12:31:44.442409Z","shell.execute_reply":"2023-10-12T12:31:57.266786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Configurations & Reproducability\n\nIt is good practice to have a configuration class in notebooks.","metadata":{}},{"cell_type":"code","source":"class Config:\n    SEED = 0\n    IMAGE_SIZE = [256, 256]\n    BATCH_SIZE = 64\n    EPOCHS = 10\n    TARGET_COLS = [\n        \"bowel_injury\", \"extravasation_injury\",\n        \"kidney_healthy\", \"kidney_low\", \"kidney_high\",\n        \"liver_healthy\", \"liver_low\", \"liver_high\",\n        \"spleen_healthy\", \"spleen_low\", \"spleen_high\",\n    ]\n    AUTOTUNE = tf.data.AUTOTUNE\n\nconfig = Config()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:31:57.269297Z","iopub.execute_input":"2023-10-12T12:31:57.270204Z","iopub.status.idle":"2023-10-12T12:31:57.276587Z","shell.execute_reply.started":"2023-10-12T12:31:57.270167Z","shell.execute_reply":"2023-10-12T12:31:57.275661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keras.utils.set_random_seed(seed=config.SEED)","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:31:57.279017Z","iopub.execute_input":"2023-10-12T12:31:57.279739Z","iopub.status.idle":"2023-10-12T12:31:57.306185Z","shell.execute_reply.started":"2023-10-12T12:31:57.279699Z","shell.execute_reply":"2023-10-12T12:31:57.305187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataset\n\nWe will train on PNG images instead of the DICOM format the images come in. Because `KerasCV` cannot read DICOM images.\n\nThe below dataset is from https://www.kaggle.com/awsaf49\n","metadata":{}},{"cell_type":"code","source":"BASE_PATH = f\"/kaggle/input/rsna-atd-512x512-png-v2-dataset\"","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:31:57.307464Z","iopub.execute_input":"2023-10-12T12:31:57.308399Z","iopub.status.idle":"2023-10-12T12:31:57.318382Z","shell.execute_reply.started":"2023-10-12T12:31:57.308367Z","shell.execute_reply":"2023-10-12T12:31:57.317494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataframe = pd.read_csv(f\"{BASE_PATH}/train.csv\")\ndataframe[\"image_path\"] = f\"{BASE_PATH}/train_images\"\\\n                    + \"/\" + dataframe.patient_id.astype(str)\\\n                    + \"/\" + dataframe.series_id.astype(str)\\\n                    + \"/\" + dataframe.instance_number.astype(str) +\".png\"\ndataframe = dataframe.drop_duplicates()\n\ndataframe.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:31:57.319526Z","iopub.execute_input":"2023-10-12T12:31:57.320382Z","iopub.status.idle":"2023-10-12T12:31:57.457507Z","shell.execute_reply.started":"2023-10-12T12:31:57.320353Z","shell.execute_reply":"2023-10-12T12:31:57.456448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataframe.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:31:57.458972Z","iopub.execute_input":"2023-10-12T12:31:57.459623Z","iopub.status.idle":"2023-10-12T12:31:57.466036Z","shell.execute_reply.started":"2023-10-12T12:31:57.45959Z","shell.execute_reply":"2023-10-12T12:31:57.465074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Splitting into train and validation\n\nFor a binary classification we would ensure that the representation of both classes would be the same in both the training and validation sets. We could do this by using `stratify=[target]` in the `train_test_split` function.\n\nHere, with multiple targets, we must use a more nuanced approach.\n\n1. We divide the whole dataset into groups based on the values in the target columns. I.e. all entries within a group will have identical target columns\n\n2. Then perform `train_test_split` within these groups, and concatenate results to train and validation datasets.","metadata":{}},{"cell_type":"code","source":"# Funcion to handle the split for each group\ndef split_group(group, test_size=0.2):\n    if len(group) == 1:\n        if np.random.rand() < test_size:\n            return (group, pd.DataFrame())\n        else:\n            return (pd.DataFrame(), group)\n    else:\n        return train_test_split(group, test_size=test_size, random_state=config.SEED)\n    \n# intialize the train and validation datasets\ntrain_data = pd.DataFrame()\nval_data = pd.DataFrame()\n\n# Iterate through groups and split them, handling single-sample groups\n\nfor _, group in dataframe.groupby(config.TARGET_COLS):\n    train_group, val_group = split_group(group)\n    train_data = pd.concat([train_data, train_group], ignore_index=True)\n    val_data = pd.concat([val_data, val_group], ignore_index=True)","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:33:25.627011Z","iopub.execute_input":"2023-10-12T12:33:25.627413Z","iopub.status.idle":"2023-10-12T12:33:25.764546Z","shell.execute_reply.started":"2023-10-12T12:33:25.627377Z","shell.execute_reply":"2023-10-12T12:33:25.763474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.shape, val_data.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:33:25.798709Z","iopub.execute_input":"2023-10-12T12:33:25.800955Z","iopub.status.idle":"2023-10-12T12:33:25.810811Z","shell.execute_reply.started":"2023-10-12T12:33:25.800918Z","shell.execute_reply":"2023-10-12T12:33:25.809833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"path = \"/kaggle/input/rsna-atd-512x512-png-v2-dataset/train_images/10004/21057/362.png\"","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:33:25.997861Z","iopub.execute_input":"2023-10-12T12:33:25.998281Z","iopub.status.idle":"2023-10-12T12:33:26.005418Z","shell.execute_reply.started":"2023-10-12T12:33:25.998241Z","shell.execute_reply":"2023-10-12T12:33:26.004491Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Pipeline w/ tf.data\n\nUp to now has just been regular data preparation, now we will use the `tf.data` API to build an input pipeline for the neural network\n\n### `decode_image_and_label`\n\n- Convert PNG into a normalized TensorFlow tensor, with dtype=float\n- Resizing from 512x512 to 256x256 done using bilinear interpolation\n\nhttps://en.wikipedia.org/wiki/Bilinear_interpolation\n","metadata":{}},{"cell_type":"code","source":"def decode_image_and_label(image_path, label):\n    \n    # get the bytes from a png image\n    file_bytes = tf.io.read_file(image_path)\n    \n    # decode png image into a uint8 tensor, channels=3 means 3 colors RGB\n    image = tf.io.decode_png(file_bytes, channels=3, dtype=tf.uint8)\n    \n    # resize to 256x256 using bilinear interpolation\n    image = tf.image.resize(image, config.IMAGE_SIZE, method=\"bilinear\")\n\n    # change to float dype as we wish to normalize, max value is 254\n    image = tf.cast(image, tf.float32) / 255.0\n    \n    # change labels to floats too\n    label = tf.cast(label, tf.float32)\n    \n    #         bowel       fluid       kidney      liver       spleen\n    labels = (label[0:1], label[1:2], label[2:5], label[5:8], label[8:11])\n    \n    return (image, labels)","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:33:27.13602Z","iopub.execute_input":"2023-10-12T12:33:27.136687Z","iopub.status.idle":"2023-10-12T12:33:27.142598Z","shell.execute_reply.started":"2023-10-12T12:33:27.136657Z","shell.execute_reply":"2023-10-12T12:33:27.141702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `apply_augmentation`\n\n- Use a `RandomFlip` layer that randomly flips a training image horizontally and/or vertically\n- Use a `RandomCutout` layer that randomly removes a square of dimension 0.2x0.2 full image size\n- Purpose of these is to improve the robustness of the model to handle variations of images that are not in the training dataset I.e. protect from overfitting\n- Note, the original input images are used during inference.","metadata":{}},{"cell_type":"code","source":"def apply_augmentation(images, labels):\n    augmenter = keras.Sequential(\n        [\n            keras_cv.layers.RandomFlip(mode=\"horizontal_and_vertical\"),\n            keras_cv.layers.RandomCutout(height_factor=0.2, width_factor=0.2),\n            keras_cv.layers.RandomRotation(factor=0.3),\n            keras_cv.layers.RandomContrast(factor=0.2,value_range=[0, 255])\n        ])\n    \n    return (augmenter(images), labels)\n\n","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:40:07.739714Z","iopub.execute_input":"2023-10-12T12:40:07.74039Z","iopub.status.idle":"2023-10-12T12:40:07.745893Z","shell.execute_reply.started":"2023-10-12T12:40:07.740359Z","shell.execute_reply":"2023-10-12T12:40:07.744863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `build_dataset`\n\n- First slice the images up\n- Then apply the `decode_image_and_label` function to each slice\n- shuffle them with a buffer\n- batch them up\n- Apply the augmentations in the `apply_augmentation` function","metadata":{}},{"cell_type":"code","source":"def build_dataset(image_paths, labels):\n    ds = (\n        tf.data.Dataset.from_tensor_slices((image_paths, labels))\n        .map(decode_image_and_label, num_parallel_calls=config.AUTOTUNE)\n        .shuffle(config.BATCH_SIZE * 10)\n        .batch(config.BATCH_SIZE)\n        .map(apply_augmentation, num_parallel_calls=config.AUTOTUNE)\n        .prefetch(config.AUTOTUNE)\n    )\n    return ds\n    ","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:40:08.477671Z","iopub.execute_input":"2023-10-12T12:40:08.477996Z","iopub.status.idle":"2023-10-12T12:40:08.482823Z","shell.execute_reply.started":"2023-10-12T12:40:08.477964Z","shell.execute_reply":"2023-10-12T12:40:08.481898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prepare our inputs and apply `build_dataset`\nunpack the result using an iterator.","metadata":{}},{"cell_type":"code","source":"paths = train_data.image_path.tolist()\nlabels = train_data[config.TARGET_COLS].values\n\nds = build_dataset(image_paths=paths, labels=labels)\n\nimages, labels = next(iter(ds))\nimages.shape, [label.shape for label in labels]","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:40:09.020651Z","iopub.execute_input":"2023-10-12T12:40:09.02172Z","iopub.status.idle":"2023-10-12T12:40:15.833985Z","shell.execute_reply.started":"2023-10-12T12:40:09.021682Z","shell.execute_reply":"2023-10-12T12:40:15.83306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now handle the viusalization settings","metadata":{}},{"cell_type":"code","source":"keras_cv.visualization.plot_image_gallery(\n    images=images,\n    value_range=(0, 1),\n    rows=2,\n    cols=2\n    );","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:40:15.835835Z","iopub.execute_input":"2023-10-12T12:40:15.836171Z","iopub.status.idle":"2023-10-12T12:40:16.209987Z","shell.execute_reply.started":"2023-10-12T12:40:15.836138Z","shell.execute_reply":"2023-10-12T12:40:16.208589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:40:23.089612Z","iopub.execute_input":"2023-10-12T12:40:23.089928Z","iopub.status.idle":"2023-10-12T12:40:23.095863Z","shell.execute_reply.started":"2023-10-12T12:40:23.089905Z","shell.execute_reply":"2023-10-12T12:40:23.094992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Build the Model\n\nWe will use a pretrained model from the list of available backbones in KerasCV. We will use the `ResNetBackbone`, which is a CNN.\n\nWe must define the necks and heads for each of the targets, see below diagram.\n\n<img src=\"https://www.researchgate.net/publication/356638292/figure/fig1/AS:1096930679623686@1638540062632/A-detection-model-contains-a-backbone-neck-head-module-The-backbone-module-exploits.png\" alt=\"NN Structure\"/>","metadata":{}},{"cell_type":"code","source":"def build_model(warmup_steps, decay_steps):\n    \n    # define input\n    inputs = keras.Input(shape=config.IMAGE_SIZE + [3,], batch_size=config.BATCH_SIZE)\n    \n    # define backbone\n    backbone = keras_cv.models.ResNetBackbone.from_preset(\"resnet50_imagenet\")\n    backbone.include_rescaling = False\n    x = backbone(inputs)\n    \n    # GAP to get the activation maps\n    gap = keras.layers.GlobalAveragePooling2D()\n    x = gap(x)\n    \n    # Define 'necks' for each head\n    x_bowel = keras.layers.Dense(32, activation='relu')(x)\n    x_extra = keras.layers.Dense(32, activation='relu')(x)\n    x_liver = keras.layers.Dense(32, activation='relu')(x)\n    x_kidney = keras.layers.Dense(32, activation='relu')(x)\n    x_spleen = keras.layers.Dense(32, activation='relu')(x)\n    \n    # Define heads\n    out_bowel = keras.layers.Dense(1, name='bowel', activation='sigmoid')(x_bowel) # use sigmoid to convert predictions to [0-1]\n    out_extra = keras.layers.Dense(1, name='extra', activation='sigmoid')(x_extra) # use sigmoid to convert predictions to [0-1]\n    out_liver = keras.layers.Dense(3, name='liver', activation='softmax')(x_liver) # use softmax for the liver head\n    out_kidney = keras.layers.Dense(3, name='kidney', activation='softmax')(x_kidney) # use softmax for the kidney head\n    out_spleen = keras.layers.Dense(3, name='spleen', activation='softmax')(x_spleen) # use softmax for the spleen head\n    \n    # concatenate the outputs\n    outputs = [out_bowel, out_extra, out_liver, out_kidney, out_spleen]\n    \n    # create model\n    print(\"[INFO] Building the model...\")\n    model = keras.Model(inputs=inputs, outputs=outputs)\n    \n    # cosine decay\n    cosine_decay = keras.optimizers.schedules.CosineDecay(\n        initial_learning_rate=1e-3,\n        decay_steps=decay_steps,\n        alpha=0.0,\n        warmup_target=1e-3,\n        warmup_steps=warmup_steps)\n    \n    # compile the model, AdamW is a SGD algorithm\n    optimizer = keras.optimizers.AdamW(learning_rate=cosine_decay)\n    loss = {\n        \"bowel\":keras.losses.BinaryCrossentropy(),\n        \"extra\":keras.losses.BinaryCrossentropy(),\n        \"liver\":keras.losses.CategoricalCrossentropy(),\n        \"kidney\":keras.losses.CategoricalCrossentropy(),\n        \"spleen\":keras.losses.CategoricalCrossentropy(),\n    }\n    metrics = {\n        \"bowel\":[\"accuracy\"],\n        \"extra\":[\"accuracy\"],\n        \"liver\":[\"accuracy\"],\n        \"kidney\":[\"accuracy\"],\n        \"spleen\":[\"accuracy\"],\n    }\n    print(\"[INFO] Compiling the model...\")\n    model.compile(\n        optimizer=optimizer,\n        loss=loss,\n        metrics=metrics)\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:45:57.833258Z","iopub.execute_input":"2023-10-12T12:45:57.833619Z","iopub.status.idle":"2023-10-12T12:45:57.843974Z","shell.execute_reply.started":"2023-10-12T12:45:57.833584Z","shell.execute_reply":"2023-10-12T12:45:57.842604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train the model with \"model.fit\"","metadata":{"execution":{"iopub.status.busy":"2023-09-13T23:47:50.629918Z","iopub.execute_input":"2023-09-13T23:47:50.630308Z","iopub.status.idle":"2023-09-13T23:47:52.094863Z","shell.execute_reply.started":"2023-09-13T23:47:50.630279Z","shell.execute_reply":"2023-09-13T23:47:52.093695Z"}}},{"cell_type":"code","source":"print(\"[INFO] Building the dataset...\")\ntrain_paths = train_data.image_path.values\ntrain_labels = train_data[config.TARGET_COLS].values.astype(np.float32)\nvalid_paths = val_data.image_path.values\nvalid_labels = val_data[config.TARGET_COLS].values.astype(np.float32)\n\n# train and valid dataset\ntrain_ds = build_dataset(image_paths=train_paths, labels=train_labels)\nval_ds = build_dataset(image_paths=valid_paths, labels=valid_labels)\n\ntotal_train_steps = train_ds.cardinality().numpy() * config.BATCH_SIZE * config.EPOCHS\nwarmup_steps = int(total_train_steps * 0.1)\ndecay_steps = total_train_steps - warmup_steps\n\nprint(f\"{total_train_steps=}\")\nprint(f\"{warmup_steps=}\")\nprint(f\"{decay_steps=}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:45:58.72919Z","iopub.execute_input":"2023-10-12T12:45:58.729512Z","iopub.status.idle":"2023-10-12T12:46:03.159946Z","shell.execute_reply.started":"2023-10-12T12:45:58.729488Z","shell.execute_reply":"2023-10-12T12:46:03.15893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# build the model\nprint(\"[INFO] Building the model...\")\nmodel = build_model(warmup_steps, decay_steps)\n\n# train\nprint(\"[INFO] Training...\")\nhistory = model.fit(\n    train_ds,\n    epochs=config.EPOCHS,\n    validation_data=val_ds\n)","metadata":{"execution":{"iopub.status.busy":"2023-10-12T12:46:03.161918Z","iopub.execute_input":"2023-10-12T12:46:03.162612Z","iopub.status.idle":"2023-10-12T13:25:34.054428Z","shell.execute_reply.started":"2023-10-12T12:46:03.16256Z","shell.execute_reply":"2023-10-12T13:25:34.05331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualise the training plots","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(5, 1, figsize=(5, 15))\n\n# flatten axes to iterate through them\naxes = axes.flatten()\n\nfor i, name in enumerate([\"bowel\", \"extra\", \"kidney\", \"liver\", \"spleen\"]):\n    # plot training accuracy\n    axes[i].plot(history.history[name + '_accuracy'], label='Training ' + name)\n    \n    # Plot validation accuracy\n    axes[i].plot(history.history['val_' + name + '_accuracy'], label='Validation ' + name)\n    \n    axes[i].set_title(name)\n    axes[i].set_xlabel('Epoch')\n    axes[i].set_ylabel('Accuracy')\n    axes[i].legend()\n    \nplt.tight_layout()\nplt.show()\n    ","metadata":{"execution":{"iopub.status.busy":"2023-10-12T13:40:27.292047Z","iopub.execute_input":"2023-10-12T13:40:27.292403Z","iopub.status.idle":"2023-10-12T13:40:28.441152Z","shell.execute_reply.started":"2023-10-12T13:40:27.292374Z","shell.execute_reply":"2023-10-12T13:40:28.440218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history.history[\"loss\"], label=\"loss\")\nplt.plot(history.history[\"val_loss\"], label=\"val loss\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T13:40:28.442921Z","iopub.execute_input":"2023-10-12T13:40:28.443768Z","iopub.status.idle":"2023-10-12T13:40:28.693491Z","shell.execute_reply.started":"2023-10-12T13:40:28.443724Z","shell.execute_reply":"2023-10-12T13:40:28.692366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# store best results\nbest_epoch = np.argmin(history.history['val_loss'])\nbest_loss = history.history['val_loss'][best_epoch]\nbest_acc_bowel = history.history['val_bowel_accuracy'][best_epoch]\nbest_acc_extra = history.history['val_extra_accuracy'][best_epoch]\nbest_acc_liver = history.history['val_liver_accuracy'][best_epoch]\nbest_acc_kidney = history.history['val_kidney_accuracy'][best_epoch]\nbest_acc_spleen = history.history['val_spleen_accuracy'][best_epoch]\n\n# Find mean accuracy\nbest_acc = np.mean(\n    [best_acc_bowel,\n     best_acc_extra,\n     best_acc_liver,\n     best_acc_kidney,\n     best_acc_spleen\n])\n\nprint(f'>>>> BEST Loss  : {best_loss:.3f}\\n>>>> BEST Acc   : {best_acc:.3f}\\n>>>> BEST Epoch : {best_epoch}\\n')\nprint('ORGAN Acc:')\nprint(f'  >>>> {\"Bowel\".ljust(15)} : {best_acc_bowel:.3f}')\nprint(f'  >>>> {\"Extravasation\".ljust(15)} : {best_acc_extra:.3f}')\nprint(f'  >>>> {\"Liver\".ljust(15)} : {best_acc_liver:.3f}')\nprint(f'  >>>> {\"Kidney\".ljust(15)} : {best_acc_kidney:.3f}')\nprint(f'  >>>> {\"Spleen\".ljust(15)} : {best_acc_spleen:.3f}')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T13:40:28.695367Z","iopub.execute_input":"2023-10-12T13:40:28.696411Z","iopub.status.idle":"2023-10-12T13:40:28.707137Z","shell.execute_reply.started":"2023-10-12T13:40:28.696373Z","shell.execute_reply":"2023-10-12T13:40:28.705763Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Store the model for inference","metadata":{}},{"cell_type":"code","source":"# save the model\nmodel.save(\"rsna-atd-mas-2.keras\")","metadata":{"execution":{"iopub.status.busy":"2023-10-12T13:41:50.154006Z","iopub.execute_input":"2023-10-12T13:41:50.154381Z","iopub.status.idle":"2023-10-12T13:41:51.582044Z","shell.execute_reply.started":"2023-10-12T13:41:50.154353Z","shell.execute_reply":"2023-10-12T13:41:51.580976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nos.listdir(\"/kaggle/working/\")","metadata":{"execution":{"iopub.status.busy":"2023-10-12T13:41:51.583808Z","iopub.execute_input":"2023-10-12T13:41:51.58418Z","iopub.status.idle":"2023-10-12T13:41:51.591961Z","shell.execute_reply.started":"2023-10-12T13:41:51.584147Z","shell.execute_reply":"2023-10-12T13:41:51.590708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}