{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---\n\n[J2Letters comment] Beware I am not part of the Keras Team ! This is the original [KerasCV starter notebook [Train]](https://www.kaggle.com/code/aritrag/kerascv-starter-notebook-train) with additionnal content.  As for now, I changed the way to do the train/val split, mitigated overfitting with data augmentation and appended a [Grad-CAM part with Keras](https://keras.io/examples/vision/grad_cam/) to visualize what the ResNet50 is seeing.\n\n---\n\n<center><img src=\"https://keras.io/img/logo-small.png\" alt=\"Keras logo\" width=\"100\"><br/>\nThis starter notebook is provided by the Keras team.</center>\n","metadata":{}},{"cell_type":"markdown","source":"# Training Notebook\n\n# RSNA 2023 Abdominal Trauma Detection with [KerasCV](https://github.com/keras-team/keras-cv) and [KerasCore](https://github.com/keras-team/keras-core)\n\nThis notebook walks you through how to train a **Convolutional Neural Network (CNN)** model using Keras (Core and CV) on the RSNA 2023 Abdominal Trauma Detection dataset made available for this competition.\n\nFun fact: This notebook is backend (tensorflow, pytorch, jax) agnostic. Using KerasCV and KerasCore we can choose a backend of our choise! Feel free to read [Keras Core](https://keras.io/keras_core/announcement/) announcement to know more about Keras.\n\nIn this notebook you will learn:\n\n* Loading the data using [`tf.data`](https://www.tensorflow.org/guide/data).\n* Applying augmentations inside the data pipeline.\n* Create the model using KerasCV presets.\n* Train the model.\n* Visualize the training plots.\n\n## Notebooks\n\nFor this competition we have two starter notebook. This notebook (you are reading) trains the model on the dataset, while there lies another notebook that performs inference and submits to the competition.\n\n1. [**Training Kernel**](https://www.kaggle.com/code/aritrag/kerascv-starter-notebook-train)\n2. [**Inference Kernel**](https://www.kaggle.com/code/aritrag/kerascv-starter-notebook-infer)\n\n**Note**: [KerasCV guides](https://keras.io/guides/keras_cv/) is the place to go for a deeper understanding of KerasCV individually.","metadata":{}},{"cell_type":"markdown","source":"# Setup and Imports\n\nWe will need KerasCV for this notebook.\n\nFeel free to use `pip install keras-cv` instead of the installation from github.","metadata":{}},{"cell_type":"code","source":"! pip install -q git+https://github.com/keras-team/keras-cv","metadata":{"_kg_hide-output":true,"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-09-15T12:43:21.228696Z","iopub.execute_input":"2023-09-15T12:43:21.229041Z","iopub.status.idle":"2023-09-15T12:43:51.366215Z","shell.execute_reply.started":"2023-09-15T12:43:21.229011Z","shell.execute_reply":"2023-09-15T12:43:51.364842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\n# You can use `tensorflow`, `pytorch`, `jax` here\n# KerasCore makes the notebook backend agnostic :)\nos.environ[\"KERAS_BACKEND\"] = \"tensorflow\"\n\nimport keras_cv\nimport keras_core as keras\nfrom keras_core import layers\n\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom matplotlib import pyplot as plt\nimport matplotlib.cm as cm\nfrom sklearn.model_selection import train_test_split\n\n\n# Display\nfrom IPython.display import Image, display","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:43:51.370194Z","iopub.execute_input":"2023-09-15T12:43:51.370587Z","iopub.status.idle":"2023-09-15T12:44:05.483549Z","shell.execute_reply.started":"2023-09-15T12:43:51.370559Z","shell.execute_reply":"2023-09-15T12:44:05.482488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Configuration\n\nA particularly good practise is to have a configuration class for your notebooks. This not only keeps your configurations all at a single place but also becomes handy to map the configs to the performance of the model.\n\nPlease play around with the configurations and see how the performance of the model changes.","metadata":{}},{"cell_type":"markdown","source":"## Note on some observations\n\nReference Notebook: https://www.kaggle.com/code/aritrag/eda-train-csv\n\n1. Class Dependencies: Refers to inherent relationships between classes in the analysis.\n2. Complementarity: `bowel_injury` and `bowel_healthy`, as well as `extravasation_injury` and `extravasation_healthy`, are perfectly complementary, with their sum always equal to 1.0.\n3. Simplification: For the model, only `{bowel/extravasation}_injury` will be included, and the corresponding healthy status can be calculated using a sigmoid function.\n4. Softmax: `{kidney/liver/spleen}_{healthy/low/high}` classifications are softmaxed, ensuring their combined probabilities sum up to 1.0 for each organ, simplifying the model while preserving essential information.","metadata":{}},{"cell_type":"code","source":"class Config:\n    SEED = 42\n    IMAGE_SIZE = [256, 256]\n    BATCH_SIZE = 64\n    EPOCHS = 10\n    TARGET_COLS  = [\n        \"bowel_injury\", \"extravasation_injury\",\n        \"kidney_healthy\", \"kidney_low\", \"kidney_high\",\n        \"liver_healthy\", \"liver_low\", \"liver_high\",\n        \"spleen_healthy\", \"spleen_low\", \"spleen_high\",\n    ]\n    AUTOTUNE = tf.data.AUTOTUNE\n\nconfig = Config()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.485189Z","iopub.execute_input":"2023-09-15T12:44:05.48587Z","iopub.status.idle":"2023-09-15T12:44:05.496193Z","shell.execute_reply.started":"2023-09-15T12:44:05.485834Z","shell.execute_reply":"2023-09-15T12:44:05.493565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reproducibility\n\nWe would want this notebook to have reproducible results. Here we set the seed for all the random algorithms so that we can reproduce the experiments each time exactly the same way.","metadata":{}},{"cell_type":"code","source":"keras.utils.set_random_seed(seed=config.SEED)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.500845Z","iopub.execute_input":"2023-09-15T12:44:05.501423Z","iopub.status.idle":"2023-09-15T12:44:05.516741Z","shell.execute_reply.started":"2023-09-15T12:44:05.501389Z","shell.execute_reply":"2023-09-15T12:44:05.515795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset\n\nThe dataset provided in the competition consists of DICOM images. We will not be training on the DICOM images, rather would work on PNG image which are extracted from the DICOM format.\n\n[A helpful resource on the conversion of DICOM to PNG](https://www.kaggle.com/code/radek1/how-to-process-dicom-images-to-pngs)","metadata":{}},{"cell_type":"code","source":"BASE_PATH = f\"/kaggle/input/rsna-atd-512x512-png-v2-dataset\"","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-15T12:44:05.520234Z","iopub.execute_input":"2023-09-15T12:44:05.520559Z","iopub.status.idle":"2023-09-15T12:44:05.527753Z","shell.execute_reply.started":"2023-09-15T12:44:05.520526Z","shell.execute_reply":"2023-09-15T12:44:05.526696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Meta Data\n\nThe `train.csv` file contains the following meta information:\n\n- `patient_id`: A unique ID code for each patient.\n- `series_id`: A unique ID code for each scan.\n- `instance_number`: The image number within the scan. The lowest instance number for many series is above zero as the original scans were cropped to the abdomen.\n- `[bowel/extravasation]_[healthy/injury]`: The two injury types with binary targets.\n- `[kidney/liver/spleen]_[healthy/low/high]`: The three injury types with three target levels.\n- `any_injury`: Whether the patient had any injury at all.\n","metadata":{}},{"cell_type":"code","source":"# train\ndataframe = pd.read_csv(f\"{BASE_PATH}/train.csv\")\ndataframe[\"image_path\"] = f\"{BASE_PATH}/train_images\"\\\n                    + \"/\" + dataframe.patient_id.astype(str)\\\n                    + \"/\" + dataframe.series_id.astype(str)\\\n                    + \"/\" + dataframe.instance_number.astype(str) +\".png\"\ndataframe = dataframe.drop_duplicates()\n\ndataframe.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.529253Z","iopub.execute_input":"2023-09-15T12:44:05.529931Z","iopub.status.idle":"2023-09-15T12:44:05.709464Z","shell.execute_reply.started":"2023-09-15T12:44:05.529898Z","shell.execute_reply":"2023-09-15T12:44:05.708422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We split the training dataset into train and validation. This is a common practise in the Machine Learning pipelines. We not only want to train our model, but also want to validate it's training.\n\nA small catch here is that the training and validation data should have an aligned data distribution. Here we handle that by grouping the lables and then splitting the dataset. This ensures an aligned data distribution between the training and the validation splits.","metadata":{}},{"cell_type":"code","source":"# Function to handle the split for each group\ndef split_group(group, test_size=0.2):\n    if len(group) == 1:\n        return (group, pd.DataFrame()) if np.random.rand() < test_size else (pd.DataFrame(), group)\n    else:\n        return train_test_split(group, test_size=test_size, random_state=42)\n\n# Initialize the train and validation datasets\ntrain_data = pd.DataFrame()\nval_data = pd.DataFrame()\n\n# Iterate through the groups and split them, handling single-sample groups\nfor _, group in dataframe.groupby(config.TARGET_COLS):\n    train_group, val_group = split_group(group)\n    train_data = pd.concat([train_data, train_group], ignore_index=True)\n    val_data = pd.concat([val_data, val_group], ignore_index=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.710891Z","iopub.execute_input":"2023-09-15T12:44:05.711576Z","iopub.status.idle":"2023-09-15T12:44:05.811537Z","shell.execute_reply.started":"2023-09-15T12:44:05.711539Z","shell.execute_reply":"2023-09-15T12:44:05.810412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.shape, val_data.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.813057Z","iopub.execute_input":"2023-09-15T12:44:05.813401Z","iopub.status.idle":"2023-09-15T12:44:05.819717Z","shell.execute_reply.started":"2023-09-15T12:44:05.813367Z","shell.execute_reply":"2023-09-15T12:44:05.818808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n\n[J2Letters comment] : I find bothering the duplication of a patient in train and validation. One patient should not be part of both to avoid polluting the validation set. All its CT scans should intervene in the decision. I assert that the issue is really there in the following cells.","metadata":{}},{"cell_type":"code","source":"type(train_data), type(val_data)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.820985Z","iopub.execute_input":"2023-09-15T12:44:05.821804Z","iopub.status.idle":"2023-09-15T12:44:05.832068Z","shell.execute_reply.started":"2023-09-15T12:44:05.82177Z","shell.execute_reply":"2023-09-15T12:44:05.830959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# in the end we shall have the same value\nlen(train_data) + len(val_data)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.833419Z","iopub.execute_input":"2023-09-15T12:44:05.83389Z","iopub.status.idle":"2023-09-15T12:44:05.845012Z","shell.execute_reply.started":"2023-09-15T12:44:05.833857Z","shell.execute_reply":"2023-09-15T12:44:05.844046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.846421Z","iopub.execute_input":"2023-09-15T12:44:05.846845Z","iopub.status.idle":"2023-09-15T12:44:05.867172Z","shell.execute_reply.started":"2023-09-15T12:44:05.846813Z","shell.execute_reply":"2023-09-15T12:44:05.866182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.columns","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.868679Z","iopub.execute_input":"2023-09-15T12:44:05.869026Z","iopub.status.idle":"2023-09-15T12:44:05.875727Z","shell.execute_reply.started":"2023-09-15T12:44:05.868995Z","shell.execute_reply":"2023-09-15T12:44:05.874681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_patient_series = train_data[[\"patient_id\", \"series_id\"]].drop_duplicates()\nval_patient_series = val_data[[\"patient_id\", \"series_id\"]].drop_duplicates()\n\ntrain_patient_series.head(2), val_patient_series.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.877582Z","iopub.execute_input":"2023-09-15T12:44:05.877984Z","iopub.status.idle":"2023-09-15T12:44:05.89516Z","shell.execute_reply.started":"2023-09-15T12:44:05.877951Z","shell.execute_reply":"2023-09-15T12:44:05.894101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patient_series_check = train_patient_series.merge(val_patient_series, how=\"outer\", on=[\"patient_id\", \"series_id\"], indicator=True)\npatient_series_check.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.9008Z","iopub.execute_input":"2023-09-15T12:44:05.90107Z","iopub.status.idle":"2023-09-15T12:44:05.926049Z","shell.execute_reply.started":"2023-09-15T12:44:05.901047Z","shell.execute_reply":"2023-09-15T12:44:05.925138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"patient_series_check.groupby(\"_merge\").count()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.929048Z","iopub.execute_input":"2023-09-15T12:44:05.929322Z","iopub.status.idle":"2023-09-15T12:44:05.942587Z","shell.execute_reply.started":"2023-09-15T12:44:05.929299Z","shell.execute_reply":"2023-09-15T12:44:05.941619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[J2Letters comment] : A vast majority of the patients are in both train and val, we can split the data on a patient level.","metadata":{}},{"cell_type":"code","source":"# Prepare a dataframe with each patient and its target columns\npatient_level = dataframe[[\"patient_id\"]+config.TARGET_COLS].drop_duplicates()\nprint(f\"Number of patients : {len(patient_level)}\")\npatient_level.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.944447Z","iopub.execute_input":"2023-09-15T12:44:05.945589Z","iopub.status.idle":"2023-09-15T12:44:05.965541Z","shell.execute_reply.started":"2023-09-15T12:44:05.945552Z","shell.execute_reply":"2023-09-15T12:44:05.96449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize the train and validation datasets\ntrain_patient_data = pd.DataFrame()\nval_patient_data = pd.DataFrame()\n\n# Iterate through the groups and split them, handling single-sample groups\nfor _, patient_group in patient_level.groupby(config.TARGET_COLS):\n    train_patient_group, val_patient_group = split_group(patient_group)\n    train_patient_data = pd.concat([train_patient_data, train_patient_group], ignore_index=True)\n    val_patient_data = pd.concat([val_patient_data, val_patient_group], ignore_index=True)\n    \nprint(f\"Number of patients in the train set : {len(train_patient_data)}, and in val set : {len(val_patient_data)}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:05.967098Z","iopub.execute_input":"2023-09-15T12:44:05.967643Z","iopub.status.idle":"2023-09-15T12:44:06.017472Z","shell.execute_reply.started":"2023-09-15T12:44:05.967608Z","shell.execute_reply":"2023-09-15T12:44:06.016574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[J2Letters comment] : Once patients are dispatched in the train and val set, we collect the associated data.","metadata":{}},{"cell_type":"code","source":"# Re-Initialize the train and validation datasets\ntrain_data = pd.DataFrame()\nval_data = pd.DataFrame()\n\ntrain_data = dataframe.loc[dataframe[\"patient_id\"].isin(train_patient_data[\"patient_id\"])]\nval_data = dataframe.loc[dataframe[\"patient_id\"].isin(val_patient_data[\"patient_id\"])]\ntrain_data.reset_index(drop=True, inplace=True)\nval_data.reset_index(drop=True, inplace=True)\n\ntrain_data.shape, val_data.shape, len(train_data)+len(val_data)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:06.019002Z","iopub.execute_input":"2023-09-15T12:44:06.019357Z","iopub.status.idle":"2023-09-15T12:44:06.032877Z","shell.execute_reply.started":"2023-09-15T12:44:06.019324Z","shell.execute_reply":"2023-09-15T12:44:06.031708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[J2Letters comment] : Seems correct, the shape is of the same magnitude as the previous split originally in the notebook.","metadata":{}},{"cell_type":"code","source":"train_data.head(2)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:06.034681Z","iopub.execute_input":"2023-09-15T12:44:06.035022Z","iopub.status.idle":"2023-09-15T12:44:06.050665Z","shell.execute_reply.started":"2023-09-15T12:44:06.034991Z","shell.execute_reply":"2023-09-15T12:44:06.049633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[J2letters comment] Our custom split is done, we resume with the original notebook.\n\n---","metadata":{}},{"cell_type":"markdown","source":"## Data Pipeline /w tf.data\n\nHere we build the data pipeline using `tf.data`. Using `tf.data` we can map out data to an augmentation pipeline simple by using the ` map` API.\n\nAdding augmentations to the data pipeline is as simple as adding a layer into the list of layers that the `Augmenter` processes.\n\nReference: https://keras.io/api/keras_cv/layers/augmentation/","metadata":{}},{"cell_type":"markdown","source":"[J2Letters comment] : as mentionned in the original notebook comment section, the Augmenter is now replaced with Keras.Sequential. There are other augmentations that can be done. It becomes important to review them in case of overfitting. ","metadata":{}},{"cell_type":"code","source":"def decode_image_and_label(image_path, label):\n    file_bytes = tf.io.read_file(image_path)\n    image = tf.io.decode_png(file_bytes, channels=3, dtype=tf.uint8)\n    image = tf.image.resize(image, config.IMAGE_SIZE, method=\"bilinear\")\n    image = tf.cast(image, tf.float32) / 255.0\n    \n    label = tf.cast(label, tf.float32)\n    #         bowel       fluid       kidney      liver       spleen\n    #labels = (label[0:1], label[1:2], label[2:5], label[5:8], label[8:11])\n    \n    return (image, label)#labels)\n\ndef split_label(images, labels):\n    # nothing is done on the images here\n    splitted_labels = (labels[:,0:1], labels[:,1:2], labels[:,2:5], labels[:,5:8], labels[:,8:11])\n    print(labels.shape)\n    print(f\"Labels : {labels} | Splitted : {splitted_labels}\")\n    return (images, splitted_labels)\n\n\ndef apply_augmentation(images, labels):\n    image_label_augmenter = keras_cv.layers.CutMix(alpha=1, seed=config.SEED) # [J2Letters comment] Try with and without CutMix, not sure it does any good here.\n    image_augmenter = keras_cv.layers.Augmenter(\n        [\n            keras_cv.layers.RandAugment(value_range=(0, 1), augmentations_per_image=3, magnitude=0.5, magnitude_stddev=0.15), # [J2Letters comment] Attempt to mitigate overfitting\n            #keras_cv.layers.RandomFlip(mode=\"horizontal_and_vertical\"),\n            #keras_cv.layers.RandomCutout(height_factor=0.2, width_factor=0.2),\n        ]\n    )\n    updated_images_labels = image_label_augmenter({\"images\": images, \"labels\": labels})\n    return (image_augmenter(updated_images_labels[\"images\"]), updated_images_labels[\"labels\"])\n\n\ndef build_dataset(image_paths, labels):\n    ds = (\n        tf.data.Dataset.from_tensor_slices((image_paths, labels))\n        .map(decode_image_and_label, num_parallel_calls=config.AUTOTUNE)\n        .shuffle(config.BATCH_SIZE * 10)\n        .batch(config.BATCH_SIZE)\n        .map(apply_augmentation, num_parallel_calls=config.AUTOTUNE)\n        .map(split_label, num_parallel_calls=config.AUTOTUNE)\n        .prefetch(config.AUTOTUNE)\n    )\n    return ds","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:06.052639Z","iopub.execute_input":"2023-09-15T12:44:06.052979Z","iopub.status.idle":"2023-09-15T12:44:06.066669Z","shell.execute_reply.started":"2023-09-15T12:44:06.052948Z","shell.execute_reply":"2023-09-15T12:44:06.066006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"paths  = train_data.image_path.tolist()\nlabels = train_data[config.TARGET_COLS].values\n\nds = build_dataset(image_paths=paths, labels=labels)\nimages, labels = next(iter(ds))\nimages.shape, [label.shape for label in labels]","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:06.067864Z","iopub.execute_input":"2023-09-15T12:44:06.068579Z","iopub.status.idle":"2023-09-15T12:44:42.182074Z","shell.execute_reply.started":"2023-09-15T12:44:06.068547Z","shell.execute_reply":"2023-09-15T12:44:42.181155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# No more customizing your plots by hand, KerasCV has your back ;)\nkeras_cv.visualization.plot_image_gallery(\n    images=images,\n    value_range=(0, 1),\n    rows=2,\n    cols=2,\n)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:42.183628Z","iopub.execute_input":"2023-09-15T12:44:42.184177Z","iopub.status.idle":"2023-09-15T12:44:42.9453Z","shell.execute_reply.started":"2023-09-15T12:44:42.184144Z","shell.execute_reply":"2023-09-15T12:44:42.94443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build Model\n\nWe are going to load a pretrained model from the [list of avaiable backbones in KerasCV](https://keras.io/api/keras_cv/models/backbones/). We are using the `ResNetBackbone` as our backbone. The practise of using a pretrained model and finetuning it to a specific dataset is prevalent in the DL community.\n\nWe use the [Functional API](https://keras.io/guides/functional_api/) of Keras to build the model. The design of the model would be such that we input a single image and we get different heads for the various predictions we need (kidney, spleen...).\n\nWe have also added a Learning Rate scheduler for you to work with. When an athlete trains, the first step is always to warm up. We take a similar approach to training our models. We warm up with model where the learning rate increses from the initial LR to a higher LR. After the warmup stage we provide a decay algorithm (cosine here). A list of all the learning rate scheduler can be found [here](https://keras.io/api/optimizers/learning_rate_schedules/).","metadata":{}},{"cell_type":"code","source":"def build_model(warmup_steps, decay_steps):\n    # Define Input\n    inputs = keras.Input(shape=config.IMAGE_SIZE + [3,], batch_size=config.BATCH_SIZE)\n    \n    # Define Backbone\n    backbone = keras_cv.models.ResNetBackbone.from_preset(\"resnet50_imagenet\")\n    backbone.include_rescaling = False\n    x = backbone(inputs)\n    \n    # GAP to get the activation maps\n    gap = keras.layers.GlobalAveragePooling2D()\n    x = gap(x)\n\n    # Define 'necks' for each head\n    x_bowel = keras.layers.Dense(32, activation='silu')(x)\n    x_extra = keras.layers.Dense(32, activation='silu')(x)\n    x_liver = keras.layers.Dense(32, activation='silu')(x)\n    x_kidney = keras.layers.Dense(32, activation='silu')(x)\n    x_spleen = keras.layers.Dense(32, activation='silu')(x)\n\n    # Define heads\n    out_bowel = keras.layers.Dense(1, name='bowel', activation='sigmoid')(x_bowel) # use sigmoid to convert predictions to [0-1]\n    out_extra = keras.layers.Dense(1, name='extra', activation='sigmoid')(x_extra) # use sigmoid to convert predictions to [0-1]\n    out_liver = keras.layers.Dense(3, name='liver', activation='softmax')(x_liver) # use softmax for the liver head\n    out_kidney = keras.layers.Dense(3, name='kidney', activation='softmax')(x_kidney) # use softmax for the kidney head\n    out_spleen = keras.layers.Dense(3, name='spleen', activation='softmax')(x_spleen) # use softmax for the spleen head\n    \n    # Concatenate the outputs\n    outputs = [out_bowel, out_extra, out_liver, out_kidney, out_spleen]\n\n    # Create model\n    print(\"[INFO] Building the model...\")\n    model = keras.Model(inputs=inputs, outputs=outputs)\n    \n    # Cosine Decay\n    cosine_decay = keras.optimizers.schedules.CosineDecay(\n        initial_learning_rate=1e-4,\n        decay_steps=decay_steps,\n        alpha=0.0,\n        warmup_target=1e-3,\n        warmup_steps=warmup_steps,\n    )\n\n    # Compile the model\n    optimizer = keras.optimizers.Adam(learning_rate=cosine_decay)\n    loss = {\n        \"bowel\":keras.losses.BinaryCrossentropy(),\n        \"extra\":keras.losses.BinaryCrossentropy(),\n        \"liver\":keras.losses.CategoricalCrossentropy(),\n        \"kidney\":keras.losses.CategoricalCrossentropy(),\n        \"spleen\":keras.losses.CategoricalCrossentropy(),\n    }\n    metrics = {\n        \"bowel\":[\"accuracy\"],\n        \"extra\":[\"accuracy\"],\n        \"liver\":[\"accuracy\"],\n        \"kidney\":[\"accuracy\"],\n        \"spleen\":[\"accuracy\"],\n    }\n    print(\"[INFO] Compiling the model...\")\n    model.compile(\n        optimizer=optimizer,\n      loss=loss,\n      metrics=metrics\n    )\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:42.947037Z","iopub.execute_input":"2023-09-15T12:44:42.94765Z","iopub.status.idle":"2023-09-15T12:44:42.963588Z","shell.execute_reply.started":"2023-09-15T12:44:42.947617Z","shell.execute_reply":"2023-09-15T12:44:42.962639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train the model with \"model.fit\"","metadata":{}},{"cell_type":"code","source":"# get image_paths and labels\nprint(\"[INFO] Building the dataset...\")\ntrain_paths = train_data.image_path.values; train_labels = train_data[config.TARGET_COLS].values.astype(np.float32)\nvalid_paths = val_data.image_path.values; valid_labels = val_data[config.TARGET_COLS].values.astype(np.float32)\n\n# train and valid dataset\ntrain_ds = build_dataset(image_paths=train_paths, labels=train_labels)\nval_ds = build_dataset(image_paths=valid_paths, labels=valid_labels)\n\ntotal_train_steps = train_ds.cardinality().numpy() * config.BATCH_SIZE * config.EPOCHS\nwarmup_steps = int(total_train_steps * 0.10)\ndecay_steps = total_train_steps - warmup_steps\n\nprint(f\"{total_train_steps=}\")\nprint(f\"{warmup_steps=}\")\nprint(f\"{decay_steps=}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:44:42.967342Z","iopub.execute_input":"2023-09-15T12:44:42.967655Z","iopub.status.idle":"2023-09-15T12:45:02.728146Z","shell.execute_reply.started":"2023-09-15T12:44:42.96763Z","shell.execute_reply":"2023-09-15T12:45:02.727163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# build model\nprint(\"[INFO Building...]\")\nmodel = build_model(warmup_steps, decay_steps)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:45:02.730365Z","iopub.execute_input":"2023-09-15T12:45:02.731385Z","iopub.status.idle":"2023-09-15T12:45:04.542129Z","shell.execute_reply.started":"2023-09-15T12:45:02.731341Z","shell.execute_reply":"2023-09-15T12:45:04.54121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train\nprint(\"[INFO] Training...\")\nhistory = model.fit(\n    train_ds,\n    epochs=config.EPOCHS,\n    validation_data=val_ds,\n)","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-09-15T12:45:04.543482Z","iopub.execute_input":"2023-09-15T12:45:04.543936Z","iopub.status.idle":"2023-09-15T13:30:27.286474Z","shell.execute_reply.started":"2023-09-15T12:45:04.543898Z","shell.execute_reply":"2023-09-15T13:30:27.285427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualize the training plots","metadata":{}},{"cell_type":"code","source":"# Create a 3x2 grid for the subplots\nfig, axes = plt.subplots(5, 1, figsize=(5, 15))\n\n# Flatten axes to iterate through them\naxes = axes.flatten()\n\n# Iterate through the metrics and plot them\nfor i, name in enumerate([\"bowel\", \"extra\", \"kidney\", \"liver\", \"spleen\"]):\n    # Plot training accuracy\n    axes[i].plot(history.history[name + '_accuracy'], label='Training ' + name)\n    # Plot validation accuracy\n    axes[i].plot(history.history['val_' + name + '_accuracy'], label='Validation ' + name)\n    axes[i].set_title(name)\n    axes[i].set_xlabel('Epoch')\n    axes[i].set_ylabel('Accuracy')\n    axes[i].legend()\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:27.290139Z","iopub.execute_input":"2023-09-15T13:30:27.290552Z","iopub.status.idle":"2023-09-15T13:30:28.495642Z","shell.execute_reply.started":"2023-09-15T13:30:27.290516Z","shell.execute_reply":"2023-09-15T13:30:28.494489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history.history[\"loss\"], label=\"loss\")\nplt.plot(history.history[\"val_loss\"], label=\"val loss\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:28.497274Z","iopub.execute_input":"2023-09-15T13:30:28.497632Z","iopub.status.idle":"2023-09-15T13:30:28.754431Z","shell.execute_reply.started":"2023-09-15T13:30:28.4976Z","shell.execute_reply":"2023-09-15T13:30:28.753426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# store best results\nbest_epoch = np.argmin(history.history['val_loss'])\nbest_loss = history.history['val_loss'][best_epoch]\nbest_acc_bowel = history.history['val_bowel_accuracy'][best_epoch]\nbest_acc_extra = history.history['val_extra_accuracy'][best_epoch]\nbest_acc_liver = history.history['val_liver_accuracy'][best_epoch]\nbest_acc_kidney = history.history['val_kidney_accuracy'][best_epoch]\nbest_acc_spleen = history.history['val_spleen_accuracy'][best_epoch]\n\n# Find mean accuracy\nbest_acc = np.mean(\n    [best_acc_bowel,\n     best_acc_extra,\n     best_acc_liver,\n     best_acc_kidney,\n     best_acc_spleen\n])\n\n\nprint(f'>>>> BEST Loss  : {best_loss:.3f}\\n>>>> BEST Acc   : {best_acc:.3f}\\n>>>> BEST Epoch : {best_epoch}\\n')\nprint('ORGAN Acc:')\nprint(f'  >>>> {\"Bowel\".ljust(15)} : {best_acc_bowel:.3f}')\nprint(f'  >>>> {\"Extravasation\".ljust(15)} : {best_acc_extra:.3f}')\nprint(f'  >>>> {\"Liver\".ljust(15)} : {best_acc_liver:.3f}')\nprint(f'  >>>> {\"Kidney\".ljust(15)} : {best_acc_kidney:.3f}')\nprint(f'  >>>> {\"Spleen\".ljust(15)} : {best_acc_spleen:.3f}')","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:28.755833Z","iopub.execute_input":"2023-09-15T13:30:28.756267Z","iopub.status.idle":"2023-09-15T13:30:28.766197Z","shell.execute_reply.started":"2023-09-15T13:30:28.756233Z","shell.execute_reply":"2023-09-15T13:30:28.76523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Store the model for inference","metadata":{}},{"cell_type":"code","source":"# Save the model\nmodel.save(\"rsna-atd.keras\")","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:28.767634Z","iopub.execute_input":"2023-09-15T13:30:28.768237Z","iopub.status.idle":"2023-09-15T13:30:30.090633Z","shell.execute_reply.started":"2023-09-15T13:30:28.768202Z","shell.execute_reply":"2023-09-15T13:30:30.089455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n\n[J2Letters comment] : Wouldn't it be nice to see what the CNN sees ? [Grad-CAM](https://keras.io/examples/vision/grad_cam/) does exactly that. Let's try to do it with our trained ResNet.\n\n# XAI\n\n## Grad-CAM\n\nThere is a [very nice package](https://github.com/jacobgil/pytorch-grad-cam) based on pytorch that does the job but I would like to scratch the surface by following [Keras example on Grad-CAM](https://keras.io/examples/vision/grad_cam/).\n\nWe need the last_conv_layer_name.","metadata":{}},{"cell_type":"code","source":"model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:30.092154Z","iopub.execute_input":"2023-09-15T13:30:30.092559Z","iopub.status.idle":"2023-09-15T13:30:30.134832Z","shell.execute_reply.started":"2023-09-15T13:30:30.092524Z","shell.execute_reply":"2023-09-15T13:30:30.133896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I might have to look for one output after another due to the ListWrapper format of the output. Let's start with bowel.","metadata":{}},{"cell_type":"code","source":"model.output[0]","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:30.136439Z","iopub.execute_input":"2023-09-15T13:30:30.137116Z","iopub.status.idle":"2023-09-15T13:30:30.144337Z","shell.execute_reply.started":"2023-09-15T13:30:30.137082Z","shell.execute_reply":"2023-09-15T13:30:30.143371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#preprocess_input = tf.keras.applications.resnet50.preprocess_input\n#decode_predictions = tf.keras.applications.resnet50.decode_predictions\nimg_size = config.IMAGE_SIZE\noutput_int = 2 # 0 to see the bowel, 1 extravasation, 2 liver, 3 kidney, 4 spleen\nafter_last_conv_layer_index = 2","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:30.145622Z","iopub.execute_input":"2023-09-15T13:30:30.146382Z","iopub.status.idle":"2023-09-15T13:30:30.154663Z","shell.execute_reply.started":"2023-09-15T13:30:30.146348Z","shell.execute_reply":"2023-09-15T13:30:30.153869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_images, train_labels = next(iter(train_ds))\ntrain_images.shape, [train_label.shape for train_label in train_labels]","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:30.156168Z","iopub.execute_input":"2023-09-15T13:30:30.156579Z","iopub.status.idle":"2023-09-15T13:30:35.039383Z","shell.execute_reply.started":"2023-09-15T13:30:30.156547Z","shell.execute_reply":"2023-09-15T13:30:35.038436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_gradcam_heatmap(img_array, model, output_int, after_last_conv_layer_index, pred_index=None):\n    # First, we create a model that maps the input image to the activations\n    # of the last conv layer as well as the output predictions\n    # [J2letters comment] in case of the ResNetBackbone it is the very last activation layer \n    grad_model = keras.models.Model(\n        [model.input], [model.layers[after_last_conv_layer_index].input, model.output[output_int]] # [J2letters comment] does not work with model.layers[1].output but I do not know why\n    )\n\n    # Then, we compute the gradient of the top predicted class for our input image\n    # with respect to the activations of the last conv layer\n    with tf.GradientTape() as tape:\n        last_conv_layer_output, preds = grad_model(img_array) # [J2letters comment] important to call directly and not with predict as it would return a np.array hence without the computation graph. Here we get 2 tensorflow.python.framework.ops.EagerTensor\n        if pred_index is None:\n            pred_index = tf.argmax(preds[0])\n        class_channel = preds[:, pred_index]\n\n    # This is the gradient of the output neuron (top predicted or chosen)\n    # with regard to the output feature map of the last conv layer\n    grads = tape.gradient(class_channel, last_conv_layer_output) # [J2letters comment] does not work with np.array needs the computation graph\n\n    # This is a vector where each entry is the mean intensity of the gradient\n    # over a specific feature map channel\n    pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2))\n\n    # We multiply each channel in the feature map array\n    # by \"how important this channel is\" with regard to the top predicted class\n    # then sum all the channels to obtain the heatmap class activation\n    last_conv_layer_output = last_conv_layer_output[0]\n    heatmap = last_conv_layer_output @ pooled_grads[..., tf.newaxis]\n    heatmap = tf.squeeze(heatmap)\n\n    # For visualization purpose, we will also normalize the heatmap between 0 & 1\n    heatmap = tf.maximum(heatmap, 0) / tf.math.reduce_max(heatmap)\n    return heatmap.numpy()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:35.040883Z","iopub.execute_input":"2023-09-15T13:30:35.041216Z","iopub.status.idle":"2023-09-15T13:30:35.051233Z","shell.execute_reply.started":"2023-09-15T13:30:35.041184Z","shell.execute_reply":"2023-09-15T13:30:35.050057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Remove last layers sigmoid and softmax\nfor name in [\"bowel\", \"extra\", \"liver\", \"kidney\", \"spleen\"]: \n    model.get_layer(name).activation = None","metadata":{"execution":{"iopub.status.busy":"2023-09-15T13:30:35.052869Z","iopub.execute_input":"2023-09-15T13:30:35.053254Z","iopub.status.idle":"2023-09-15T13:30:35.064818Z","shell.execute_reply.started":"2023-09-15T13:30:35.053221Z","shell.execute_reply":"2023-09-15T13:30:35.063781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#img_array = preprocess_input(train_images[:1,...])\nimg_array = train_images[1:2,...] \nimg_array.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-15T15:01:18.397253Z","iopub.execute_input":"2023-09-15T15:01:18.398066Z","iopub.status.idle":"2023-09-15T15:01:18.405846Z","shell.execute_reply.started":"2023-09-15T15:01:18.398034Z","shell.execute_reply":"2023-09-15T15:01:18.404818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compute grad cam\nheatmap = make_gradcam_heatmap(img_array, model, output_int, after_last_conv_layer_index)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T15:01:18.862761Z","iopub.execute_input":"2023-09-15T15:01:18.863667Z","iopub.status.idle":"2023-09-15T15:01:19.221972Z","shell.execute_reply.started":"2023-09-15T15:01:18.863626Z","shell.execute_reply":"2023-09-15T15:01:19.220961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display heatmap\nplt.matshow(heatmap)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T15:01:19.377903Z","iopub.execute_input":"2023-09-15T15:01:19.378252Z","iopub.status.idle":"2023-09-15T15:01:19.752411Z","shell.execute_reply.started":"2023-09-15T15:01:19.378223Z","shell.execute_reply":"2023-09-15T15:01:19.751319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_gradcam(img_array, heatmap, alpha=0.4):\n    # Squeeze the first dimension\n    img = img_array[0]\n    # Rescale B&W image\n    img = np.uint8(255 * img)\n    \n    # Rescale heatmap to a range 0-255\n    heatmap = np.uint8(255 * heatmap)\n\n    # Use jet colormap to colorize heatmap\n    jet = plt.colormaps['jet']\n\n    # Use RGB values of the colormap\n    jet_colors = jet(np.arange(256))[:, :3]\n    jet_heatmap = jet_colors[heatmap]\n\n    # Create an image with RGB colorized heatmap\n    jet_heatmap = keras.utils.array_to_img(jet_heatmap)\n    jet_heatmap = jet_heatmap.resize((img.shape[1], img.shape[0]))\n    jet_heatmap = keras.utils.img_to_array(jet_heatmap)\n\n    # Superimpose the heatmap on original image\n    superimposed_img = jet_heatmap * alpha + img\n    superimposed_img = keras.utils.array_to_img(superimposed_img)\n\n    return superimposed_img","metadata":{"execution":{"iopub.status.busy":"2023-09-15T15:01:20.620399Z","iopub.execute_input":"2023-09-15T15:01:20.621079Z","iopub.status.idle":"2023-09-15T15:01:20.628484Z","shell.execute_reply.started":"2023-09-15T15:01:20.621044Z","shell.execute_reply":"2023-09-15T15:01:20.62748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_gradcam(img_array, heatmap)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T15:01:21.315535Z","iopub.execute_input":"2023-09-15T15:01:21.316637Z","iopub.status.idle":"2023-09-15T15:01:21.363009Z","shell.execute_reply.started":"2023-09-15T15:01:21.316594Z","shell.execute_reply":"2023-09-15T15:01:21.362003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following image shows the liver visibly /kaggle/input/rsna-atd-512x512-png-v2-dataset/train_images/10004/21057/386.png. Does the model pay attention to that area ?\n\n","metadata":{}},{"cell_type":"code","source":"example = train_data.loc[train_data[\"image_path\"] == \"/kaggle/input/rsna-atd-512x512-png-v2-dataset/train_images/10065/37324/48.png\"]\nexample.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-15T15:01:23.211409Z","iopub.execute_input":"2023-09-15T15:01:23.212096Z","iopub.status.idle":"2023-09-15T15:01:23.221412Z","shell.execute_reply.started":"2023-09-15T15:01:23.212063Z","shell.execute_reply":"2023-09-15T15:01:23.220441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_array, label = decode_image_and_label(example.image_path[0], example[config.TARGET_COLS].values.astype(np.float32))\nimg_array = tf.expand_dims(img_array, axis=0)\nimg_array.shape, label.shape","metadata":{"execution":{"iopub.status.busy":"2023-09-15T15:01:24.065223Z","iopub.execute_input":"2023-09-15T15:01:24.065916Z","iopub.status.idle":"2023-09-15T15:01:24.087449Z","shell.execute_reply.started":"2023-09-15T15:01:24.065883Z","shell.execute_reply":"2023-09-15T15:01:24.086549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output_int = 2 # as the model seems okay on the liver, and it is on the image\nheatmap = make_gradcam_heatmap(img_array, model, output_int, after_last_conv_layer_index)\ndisplay_gradcam(img_array, heatmap)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T15:01:25.548093Z","iopub.execute_input":"2023-09-15T15:01:25.548786Z","iopub.status.idle":"2023-09-15T15:01:25.951348Z","shell.execute_reply.started":"2023-09-15T15:01:25.548743Z","shell.execute_reply":"2023-09-15T15:01:25.950415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I expected the heatmap to be on the liver which is unfortunately not the case here. ","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"# Next Steps\n\n1. Please refer to the [Inference Notebook](https://www.kaggle.com/code/aritrag/kerascv-starter-notebook-infer) to learn about submitting to the competition\n2. Dive deep into [KerasCV](https://github.com/keras-team/keras-cv) and [KerasCore](https://github.com/keras-team/keras-core)\n\n# Credits\n\nThis notebook was forked from https://www.kaggle.com/code/awsaf49/rsna-atd-cnn-tpu-train","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}