{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## implementation of ResNet-50 which is a convolutional neural network that is 50 layers deep on RSNA intracranial hemorrhage detection using Keras tensorflow by Samane Monfared\n\n## Table of contents\n\n1. [Prepare to start](#prepare)\n2. [Building up the bruteforce model](#baseline)\n    * [Image preprocessing](#preprocessor)\n    * [Custom Dataloader](#dataloader)\n    * [Preparing the dataframe](#dataframeprep)\n    * [Exploring train and test images](#traintestdiff)\n    * [Building up the brutforce model](#buildbruteforce)\n    * [Speeding up & dealing with class imbalance](#speedup1)\n    * [Validation strategy](#validation)\n    * [Let it run! :-)](#letitrun)\n3. [Building up the any-subtype network](#anysubtype)\n    * [The custom loss](#customloss)\n    * [A two output layer network](#twooutputs)","metadata":{}},{"cell_type":"markdown","source":"## Updates\n\nA lot of stuff is going on at the moment! Probably in a few days I will post an update what has changed. ;-)","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-02T11:40:14.634166Z","iopub.execute_input":"2022-12-02T11:40:14.634496Z","iopub.status.idle":"2022-12-02T11:40:14.639556Z","shell.execute_reply.started":"2022-12-02T11:40:14.634421Z","shell.execute_reply":"2022-12-02T11:40:14.638718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare to start <a class=\"anchor\" id=\"prepare\"></a>","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nsns.set()\n\nimport pydicom\n\nfrom os import listdir\n\nfrom skimage.transform import resize\nfrom imgaug import augmenters as iaa\n\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\n\nfrom keras.applications import ResNet50, VGG16\nfrom keras.applications.resnet50 import preprocess_input as preprocess_resnet_50\nfrom keras.applications.vgg16 import preprocess_input as preprocess_vgg_16\nfrom keras.layers import GlobalAveragePooling2D, Dense, Activation, concatenate, Dropout\nfrom keras.initializers import glorot_normal, he_normal\nfrom keras.regularizers import l2\nfrom keras.models import Model, load_model\nfrom keras.utils import Sequence\nfrom keras.callbacks import ModelCheckpoint, ReduceLROnPlateau, EarlyStopping\nfrom keras.optimizers import Adam\n\n\nfrom tensorflow.nn import sigmoid_cross_entropy_with_logits\nimport tensorflow as tf\n\nlistdir(\"../input/\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-12-02T11:40:14.650398Z","iopub.execute_input":"2022-12-02T11:40:14.650981Z","iopub.status.idle":"2022-12-02T11:40:14.67022Z","shell.execute_reply.started":"2022-12-02T11:40:14.650905Z","shell.execute_reply":"2022-12-02T11:40:14.668974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#listdir(\"../input/rsna-ih-detection-baseline-models\")","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:14.672336Z","iopub.execute_input":"2022-12-02T11:40:14.672824Z","iopub.status.idle":"2022-12-02T11:40:14.680471Z","shell.execute_reply.started":"2022-12-02T11:40:14.672625Z","shell.execute_reply":"2022-12-02T11:40:14.67953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Settings","metadata":{}},{"cell_type":"code","source":"train_brute_force = True\ntrain_anysubtype_network = True","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:14.68189Z","iopub.execute_input":"2022-12-02T11:40:14.68246Z","iopub.status.idle":"2022-12-02T11:40:14.689535Z","shell.execute_reply.started":"2022-12-02T11:40:14.682148Z","shell.execute_reply":"2022-12-02T11:40:14.688673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MODELOUTPUT_PATH_BRUTE_FORCE = \"bruteforce_best_model.hdf5\"\nMODELOUTPUT_PATH_ANYSUBTYPE = \"anysubtype_best_model.hdf5\"\nbrute_force_model_input = \"../input/rsna-ih-detection-baseline-models/_bruteforce_best_model.hdf5\"\nbrute_force_losses_path = \"../input/rsna-ih-detection-baseline-models/brute_force_losses.csv\"\nany_subtype_model_input = \"../input/rsna-ih-detection-baseline-models/anysubtype_best_model.hdf5\"","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:14.691038Z","iopub.execute_input":"2022-12-02T11:40:14.691512Z","iopub.status.idle":"2022-12-02T11:40:14.698618Z","shell.execute_reply.started":"2022-12-02T11:40:14.691296Z","shell.execute_reply":"2022-12-02T11:40:14.697876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building up the baseline <a class=\"anchor\" id=\"baseline\"></a>\n\nOk, enough about dicom! ;-) \n\n## Image Preprocessing <a class=\"anchor\" id=\"preprocessor\"></a>\n\nOk, before we can build up a deep learning model we need to setup our dataloader and image preprocessor. I decided to use Keras as I feel most familiar with it when it comes to custom losses and in this case this could be of great benefit. Originally I wanted to use Keras ImageDataGenerator but as we have to do some preprocessing for our dicom images I like to write custom methods instead and use the [imaug](https://imgaug.readthedocs.io/en/latest/) package. My workflow is as follows:\n\n1. Read the dicom dataset of an raw dicom image\n2. Focus on a specific hounsfield window. This way we can get rid of information in the image we don't need. This could include signals from the CT-tube, towels or perhaps also some bones or further tissue we are not interested in. \n3. We need to make sure that all images have the same width and height. So far I haven't seen another shape than 512x512 but we don't know if this could change! In addition we need to make sure that the shape of the image is what is expected by our model.","metadata":{}},{"cell_type":"code","source":"def rescale_pixelarray(dataset):\n    image = dataset.pixel_array\n    rescaled_image = image * dataset.RescaleSlope + dataset.RescaleIntercept\n    rescaled_image[rescaled_image < -1024] = -1024\n    return rescaled_image\n\ndef set_manual_window(hu_image, min_value, max_value):\n    hu_image[hu_image < min_value] = min_value\n    hu_image[hu_image > max_value] = min_value #max_value\n    return hu_image\n\nclass Preprocessor:    \n    \n    def __init__(self, path, backbone, hu_min_value, hu_max_value, augment=False):\n        self.path = path\n        self.backbone = backbone\n        self.nn_input_shape = backbone[\"nn_input_shape\"]\n        self.hu_min_value = hu_min_value\n        self.hu_max_value = hu_max_value\n        self.augment = augment\n        \n    # 1. We need to load the dicom dataset\n    def load_dicom_dataset(self, filename):\n        dataset = pydicom.dcmread(self.path + filename)\n        return dataset\n    \n    # 2. We need to rescale the pixelarray to Hounsfield units\n    #    and we need to focus on our custom window:\n    def get_hounsfield_window(self, dataset, min_value, max_value):\n        try:\n            hu_image = rescale_pixelarray(dataset)\n            windowed_image = set_manual_window(hu_image, min_value, max_value)\n        except ValueError:\n            # set to level \n            windowed_image = min_value * np.ones((self.nn_input_shape[0], self.nn_input_shape[1]))\n        return windowed_image\n        \n    \n    # 3. Resize the image to the input shape of our CNN\n    def resize(self, image):\n        image = resize(image, self.nn_input_shape)\n        return image\n    \n    # 4. If we like to augment our image, let's do it:\n    def augment_img(self, image): \n        augment_img = iaa.Sequential([\n            #iaa.Crop(keep_size=True, percent=(0.01, 0.05), sample_independently=False),\n            #iaa.Affine(rotate=(-10, 10)),\n            iaa.Fliplr(0.5)])\n        image_aug = augment_img.augment_image(image)\n        return image_aug\n    \n    def fill_channels(self, image):\n        filled_image = np.stack((image,)*3, axis=-1)\n        return filled_image\n    \n    def preprocess(self, identifier):\n        filename = identifier +  \".dcm\"\n        dataset = self.load_dicom_dataset(filename)\n        windowed_image = self.get_hounsfield_window(dataset, self.hu_min_value, self.hu_max_value)\n        image = self.resize(windowed_image)\n        if self.augment:\n            image = self.augment_img(image)\n        image = self.fill_channels(image)\n        return image\n    \n    def normalize(self, image):\n        return (image - self.hu_min_value)/(self.hu_max_value-self.hu_min_value) * 0.5","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2022-12-02T11:40:14.700964Z","iopub.execute_input":"2022-12-02T11:40:14.701427Z","iopub.status.idle":"2022-12-02T11:40:14.716607Z","shell.execute_reply.started":"2022-12-02T11:40:14.701229Z","shell.execute_reply":"2022-12-02T11:40:14.716033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Custom dataloader <a class=\"anchor\" id=\"dataloader\"></a>\n\nAs we have custom preprocessing we need to write a custom dataloader as well. For this purpose we will extend the keras Sequence class that allows us to use multiprocessing. Taking a look at the [docs](https://keras.io/utils/#sequence) you can see that we have to:\n\n* implement the getitem method\n* and the len method\n\nI like to pass unique image ids of train, dev or test data to the dataloader and load and preprocess images batchwise. ","metadata":{}},{"cell_type":"code","source":"class DataLoader(Sequence):\n    \n    def __init__(self, dataframe,\n                 preprocessor,\n                 batch_size,\n                 shuffle,\n                 num_classes=6,\n                 steps=None):\n        \n        self.preprocessor = preprocessor\n        self.data_ids = dataframe.index.values\n        self.dataframe = dataframe\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.input_shape = self.preprocessor.backbone[\"nn_input_shape\"]\n        self.preprocess_fun = self.preprocessor.backbone[\"preprocess_fun\"]\n        self.num_classes = num_classes\n        self.current_epoch=0\n        \n        self.steps=steps\n        if self.steps is not None:\n            self.steps = np.round(self.steps/3) * 3\n            self.undersample()\n        \n    def undersample(self):\n        part = np.int(self.steps/3 * self.batch_size)\n        zero_ids = np.random.choice(self.dataframe.loc[self.dataframe[\"any\"] == 0].index.values, size=2*part, replace=False)\n        hot_ids = np.random.choice(self.dataframe.loc[self.dataframe[\"any\"] == 1].index.values, size=1*part, replace=False)\n        self.data_ids = list(set(zero_ids).union(hot_ids))\n        np.random.shuffle(self.data_ids)\n        \n    # defines the number of steps per epoch\n    def __len__(self):\n        if self.steps is None:\n            return np.int(np.ceil(len(self.data_ids) / np.float(self.batch_size)))\n        else:\n            return 3*np.int(self.steps/3) \n    \n    # at the end of an epoch: \n    def on_epoch_end(self):\n        # if steps is None and shuffle is true:\n        if self.steps is None:\n            self.data_ids = self.dataframe.index.values\n            if self.shuffle:\n                np.random.shuffle(self.data_ids)\n        else:\n            self.undersample()\n        self.current_epoch += 1\n    \n    # should return a batch of images\n    def __getitem__(self, item):\n        # select the ids of the current batch\n        current_ids = self.data_ids[item*self.batch_size:(item+1)*self.batch_size]\n        X, y = self.__generate_batch(current_ids)\n        return X, y\n    \n    # collect the preprocessed images and targets of one batch\n    def __generate_batch(self, current_ids):\n        X = np.empty((self.batch_size, *self.input_shape, 3))\n        y = np.empty((self.batch_size, self.num_classes))\n        for idx, ident in enumerate(current_ids):\n            # Store sample\n            image = self.preprocessor.preprocess(ident)\n            X[idx] = self.preprocessor.normalize(image)\n            # Store class\n            y[idx] = self.__get_target(ident)\n        return X, y\n    \n    # extract the targets of one image id:\n    def __get_target(self, ident):\n        targets = self.dataframe.loc[ident].values\n        return targets","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:14.9214Z","iopub.execute_input":"2022-12-02T11:40:14.921749Z","iopub.status.idle":"2022-12-02T11:40:14.943296Z","shell.execute_reply.started":"2022-12-02T11:40:14.921685Z","shell.execute_reply":"2022-12-02T11:40:14.94234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparing the data <a class=\"anchor\" id=\"dataframeprep\"></a>","metadata":{}},{"cell_type":"code","source":"# INPUT_PATH = \"../input/rsna-intracranial-hemorrhage-detection/rsna-intracranial-hemorrhage-detection\"\ntrain_dir = \"/kaggle/input/rsna-intracranial-hemorrhage-detection/rsna-intracranial-hemorrhage-detection/stage_2_train/\"\ntest_dir = \"/kaggle/input/rsna-intracranial-hemorrhage-detection/rsna-intracranial-hemorrhage-detection/stage_2_test/\"","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:14.945309Z","iopub.execute_input":"2022-12-02T11:40:14.945695Z","iopub.status.idle":"2022-12-02T11:40:14.955617Z","shell.execute_reply.started":"2022-12-02T11:40:14.945584Z","shell.execute_reply":"2022-12-02T11:40:14.954783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv(\"/kaggle/input/rsna-intracranial-hemorrhage-detection/rsna-intracranial-hemorrhage-detection/stage_2_sample_submission.csv\")\nsubmission.head(7)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:14.958357Z","iopub.execute_input":"2022-12-02T11:40:14.958847Z","iopub.status.idle":"2022-12-02T11:40:15.430888Z","shell.execute_reply.started":"2022-12-02T11:40:14.958772Z","shell.execute_reply":"2022-12-02T11:40:15.430168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"traindf = pd.read_csv(\"/kaggle/input/rsna-intracranial-hemorrhage-detection/rsna-intracranial-hemorrhage-detection/stage_2_train.csv\")\ntraindf.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:15.432589Z","iopub.execute_input":"2022-12-02T11:40:15.432936Z","iopub.status.idle":"2022-12-02T11:40:18.380096Z","shell.execute_reply.started":"2022-12-02T11:40:15.432877Z","shell.execute_reply":"2022-12-02T11:40:18.379083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label = traindf.Label.values\ntraindf = traindf.ID.str.rsplit(\"_\", n=1, expand=True)\ntraindf.loc[:, \"label\"] = label\ntraindf = traindf.rename({0: \"id\", 1: \"subtype\"}, axis=1)\ntraindf.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:18.383275Z","iopub.execute_input":"2022-12-02T11:40:18.38357Z","iopub.status.idle":"2022-12-02T11:40:38.685881Z","shell.execute_reply.started":"2022-12-02T11:40:18.383515Z","shell.execute_reply":"2022-12-02T11:40:38.684952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"testdf = submission.ID.str.rsplit(\"_\", n=1, expand=True)\ntestdf = testdf.rename({0: \"id\", 1: \"subtype\"}, axis=1)\ntestdf.loc[:, \"label\"] = 0\ntestdf.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:38.688724Z","iopub.execute_input":"2022-12-02T11:40:38.68904Z","iopub.status.idle":"2022-12-02T11:40:40.966789Z","shell.execute_reply.started":"2022-12-02T11:40:38.688988Z","shell.execute_reply":"2022-12-02T11:40:40.965972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To use our dataloader we need to turn our traindf into something more useful. For this purpose we need to have the image id in one column and the subtypes of hemorrhage in further individual columns. ","metadata":{}},{"cell_type":"code","source":"traindf = pd.pivot_table(traindf, index=\"id\", columns=\"subtype\", values=\"label\")\ntraindf.head()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:40.968232Z","iopub.execute_input":"2022-12-02T11:40:40.968532Z","iopub.status.idle":"2022-12-02T11:40:49.314787Z","shell.execute_reply.started":"2022-12-02T11:40:40.968482Z","shell.execute_reply":"2022-12-02T11:40:49.314017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"testdf = pd.pivot_table(testdf, index=\"id\", columns=\"subtype\", values=\"label\")\ntestdf.head(1)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:49.316308Z","iopub.execute_input":"2022-12-02T11:40:49.316601Z","iopub.status.idle":"2022-12-02T11:40:50.231869Z","shell.execute_reply.started":"2022-12-02T11:40:49.316553Z","shell.execute_reply":"2022-12-02T11:40:50.230935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building up the brute force model <a class=\"anchor\" id=\"buildbruteforce\"></a>\n\nI have added some pretrained models as datasets to this kernel. Let's check their folder:","metadata":{}},{"cell_type":"code","source":"pretrained_models_path = \"../input/keras-pretrained-models/\"\nlistdir(\"../input/keras-pretrained-models/\")","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.233367Z","iopub.execute_input":"2022-12-02T11:40:50.23376Z","iopub.status.idle":"2022-12-02T11:40:50.242388Z","shell.execute_reply.started":"2022-12-02T11:40:50.23366Z","shell.execute_reply":"2022-12-02T11:40:50.241552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Just add more if you like:","metadata":{}},{"cell_type":"code","source":"pretrained_models = {\n    \"resnet_50\": {\"weights\": \"resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5\",\n                  \"nn_input_shape\": (224,224),\n                  \"preprocess_fun\": preprocess_resnet_50},\n    \"vgg16\": {\"weights\": \"vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5\",\n              \"nn_input_shape\": (224,224),\n              \"preprocess_fun\": preprocess_vgg_16},\n}","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.244177Z","iopub.execute_input":"2022-12-02T11:40:50.244675Z","iopub.status.idle":"2022-12-02T11:40:50.249944Z","shell.execute_reply.started":"2022-12-02T11:40:50.24444Z","shell.execute_reply":"2022-12-02T11:40:50.24921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I like to experiment with different models and for this purpose I will write separate model functions that we have to pass to the constructor of our network class. ","metadata":{}},{"cell_type":"code","source":"def resnet_50():\n    weights_path = pretrained_models_path + pretrained_models[\"resnet_50\"][\"weights\"]\n    net = ResNet50(include_top=False, weights=weights_path)\n    for layer in net.layers:\n        layer.trainable = False\n    return net\n\ndef vgg_16():\n    weights_path = pretrained_models_path + pretrained_models[\"vgg_16\"][\"weights\"]\n    net = VGG16(include_top=False, weights=weights_path)\n    for layer in net.layers:\n        layer.trainable = False\n    return net","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.251164Z","iopub.execute_input":"2022-12-02T11:40:50.251605Z","iopub.status.idle":"2022-12-02T11:40:50.259045Z","shell.execute_reply.started":"2022-12-02T11:40:50.251551Z","shell.execute_reply":"2022-12-02T11:40:50.258268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I really love that python does not need classes, but I often end up with a network class when training neural networks that can easily be fed with new losses, metrics, hyperparameters without searching all parts where they occur. Now we can pass a model of our choice as well as custom losses and metrics without worries. ","metadata":{}},{"cell_type":"code","source":"class MyNetwork:\n    \n    def __init__(self,\n                 model_fun,\n                 loss_fun,\n                 metrics_list,\n                 train_generator,\n                 dev_generator,\n                 epochs,\n                 num_classes=6,\n                 checkpoint_path=MODELOUTPUT_PATH_BRUTE_FORCE):\n        self.model_fun = model_fun\n        self.loss_fun = loss_fun\n        self.metrics_list = metrics_list\n        self.train_generator = train_generator\n        self.dev_generator = dev_generator\n        self.epochs = epochs\n        self.num_classes = num_classes\n        self.checkpoint_path = checkpoint_path \n        self.checkpoint = ModelCheckpoint(filepath=self.checkpoint_path,\n                                          mode=\"min\",\n                                          verbose=1,\n                                          save_best_only=True,\n                                          save_weights_only=True,\n                                          period=1)\n        self.reduce_lr = ReduceLROnPlateau(monitor='val_loss',\n                                           factor=0.5,\n                                           patience=2,\n                                           min_lr=1e-8,\n                                           mode=\"min\")\n        self.e_stopping = EarlyStopping(monitor=\"val_loss\",\n                                        min_delta=0.01,\n                                        patience=5,\n                                        mode=\"min\",\n                                        restore_best_weights=True)\n        \n    def build_model(self):\n        base_model = self.model_fun()\n        x = base_model.output\n        x = GlobalAveragePooling2D()(x)\n        x = Dropout(0.3)(x)\n        x = Dense(100, activation=\"relu\")(x)\n        x = Dropout(0.3)(x)\n        pred = Dense(self.num_classes,\n                     kernel_initializer=he_normal(seed=11),\n                     kernel_regularizer=l2(0.05),\n                     bias_regularizer=l2(0.05), activation=\"sigmoid\")(x)\n        self.model = Model(inputs=base_model.input, outputs=pred)\n    \n    def compile_model(self):\n        self.model.compile(optimizer=Adam(learning_rate=LR),\n                           loss=self.loss_fun, \n                           metrics=self.metrics_list)\n    \n    def learn(self):\n        return self.model.fit_generator(generator=self.train_generator,\n                    validation_data=self.dev_generator,\n                    epochs=self.epochs,\n                    callbacks=[self.checkpoint, self.reduce_lr, self.e_stopping],\n                    #use_multiprocessing=False,\n                    workers=8)\n    \n    def load_weights(self, path):\n        self.model.load_weights(path)\n    \n    def predict(self, test_generator):\n        predictions = self.model.predict_generator(test_generator, workers=8)\n        return predictions","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.260389Z","iopub.execute_input":"2022-12-02T11:40:50.260905Z","iopub.status.idle":"2022-12-02T11:40:50.275976Z","shell.execute_reply.started":"2022-12-02T11:40:50.260643Z","shell.execute_reply":"2022-12-02T11:40:50.275272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Validation\n","metadata":{}},{"cell_type":"code","source":"split_seed = 1\nkfold = StratifiedKFold(n_splits=5, random_state=split_seed).split(np.arange(traindf.shape[0]), traindf[\"any\"].values)\n\ntrain_idx, dev_idx = next(kfold)\n\ntrain_data = traindf.iloc[train_idx]\ndev_data = traindf.iloc[dev_idx]\n\n#train_data, dev_data = train_test_split(traindf, test_size=0.1, stratify=traindf.values, random_state=split_seed)\nprint(train_data.shape)\nprint(dev_data.shape)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.277149Z","iopub.execute_input":"2022-12-02T11:40:50.277598Z","iopub.status.idle":"2022-12-02T11:40:50.494596Z","shell.execute_reply.started":"2022-12-02T11:40:50.277549Z","shell.execute_reply":"2022-12-02T11:40:50.493725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pos_perc_train = train_data.sum() / train_data.shape[0] * 100\npos_perc_dev = dev_data.sum() / dev_data.shape[0] * 100\n\nfig, ax = plt.subplots(2,1,figsize=(20,14))\nsns.barplot(x=pos_perc_train.index, y=pos_perc_train.values, palette=\"Set2\", ax=ax[0]);\nax[0].set_title(\"Target distribution used for training data\")\nsns.barplot(x=pos_perc_dev.index, y=pos_perc_dev.values, palette=\"Set2\", ax=ax[1]);\nax[1].set_title(\"Target distribution used for dev data\");","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.49595Z","iopub.execute_input":"2022-12-02T11:40:50.496236Z","iopub.status.idle":"2022-12-02T11:40:50.897738Z","shell.execute_reply.started":"2022-12-02T11:40:50.496187Z","shell.execute_reply":"2022-12-02T11:40:50.896979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loss functions\n\nBy writing this kernel I'm going to play with different loss functions. \n\n\n### Multilabel loss\n\nThe first one is just a simple binary cross entropy per class summed up. We can use tensorflows sigmoid_cross_entropy for this purpose as it assumes mutually exclusive targets for this task. The weights $w_{s}$ can be used to set more importance to specific classes like the any-part in the brute force approach that contributes more to the evaluation loss.\n\n$$L = - \\frac{1}{N} \\sum_{n=1}^{N} \\sum_{s=1}^{S} w_{s} \\cdot [t_{n,s} \\cdot \\ln(y_{n,s}) + (1-t_{n,s}) \\cdot \\ln(1-y_{n,s})] $$  \n\n### Multilabel Focal loss\n\nTo tackle the class imbalance we could also introduce class weights for each multilabel class:\n\n$$l_{n,s} = \\alpha_{s} \\cdot t_{n,s} \\cdot \\ln(y_{n,s}) + (1-\\alpha_{s}) \\cdot (1-t_{n,s}) \\cdot \\ln(1-y_{n,s}) $$  \n\nFurthermore we could also try to improve by using a [focal loss ](https://arxiv.org/abs/1708.02002)per multilabel class:\n\n$$ L = - \\frac{1}{N} \\sum_{n=1}^{N} \\sum_{s=1}^{S} w_{s} \\cdot [(1-\\alpha) \\cdot (1-y_{n,s})^{\\gamma} \\cdot t_{n,s} \\cdot  \\ln(y_{n,s}) + \\alpha \\cdot y_{n,s}^{\\gamma} \\cdot (1-t_{n,s}) \\cdot \\ln(1-y_{n,s})]$$\n\n$$ L = -\\frac{1}{N} \\sum_{n=1}^{N} \\sum_{s=1}^{S} w_{s} \\cdot [(1-\\alpha_{t})(1-y_{n,s,t})^{\\gamma} \\cdot \\ln(y_{n,s,t})] $$\n\nWith $\\gamma$ between 0 and 5. With $\\gamma=0$ we have no effect but with increasing its value we enforce higher contributions to the loss for misclassified samples whereas the contributions for already well classified samples are down-weighted. \n","metadata":{}},{"cell_type":"code","source":"def np_multilabel_loss(y_true, y_pred, class_weights=None):\n    y_pred = np.where(y_pred > 1-(1e-07), 1-1e-07, y_pred)\n    y_pred = np.where(y_pred < 1e-07, 1e-07, y_pred)\n    single_class_cross_entropies = - np.mean(y_true * np.log(y_pred) + (1-y_true) * np.log(1-y_pred), axis=0)\n    \n    print(single_class_cross_entropies)\n    if class_weights is None:\n        loss = np.mean(single_class_cross_entropies)\n    else:\n        loss = np.sum(class_weights*single_class_cross_entropies)\n    return loss\n\ndef get_raw_xentropies(y_true, y_pred):\n    y_pred = tf.clip_by_value(y_pred, 1e-7, 1-1e-7)\n    xentropies = y_true * tf.log(y_pred) + (1-y_true) * tf.log(1-y_pred)\n    return -xentropies\n\n# multilabel focal loss equals multilabel loss in case of alpha=0.5 and gamma=0 \ndef multilabel_focal_loss(class_weights=None, alpha=0.5, gamma=2):\n    def mutlilabel_focal_loss_inner(y_true, y_pred):\n        y_true = tf.cast(y_true, tf.float32)\n        y_pred = tf.cast(y_pred, tf.float32)\n        \n        xentropies = get_raw_xentropies(y_true, y_pred)\n\n        # compute pred_t:\n        y_t = tf.where(tf.equal(y_true,1), y_pred, 1.-y_pred)\n        alpha_t = tf.where(tf.equal(y_true, 1), alpha * tf.ones_like(y_true), (1-alpha) * tf.ones_like(y_true))\n\n        # compute focal loss contributions\n        focal_loss_contributions =  tf.multiply(tf.multiply(tf.pow(1-y_t, gamma), xentropies), alpha_t) \n\n        # our focal loss contributions have shape (n_samples, s_classes), we need to reduce with mean over samples:\n        focal_loss_per_class = tf.reduce_mean(focal_loss_contributions, axis=0)\n\n        # compute the overall loss if class weights are None (equally weighted):\n        if class_weights is None:\n            focal_loss_result = tf.reduce_mean(focal_loss_per_class)\n        else:\n            # weight the single class losses and compute the overall loss\n            weights = tf.constant(class_weights, dtype=tf.float32)\n            focal_loss_result = tf.reduce_sum(tf.multiply(weights, focal_loss_per_class))\n            \n        return focal_loss_result\n    return mutlilabel_focal_loss_inner","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.899204Z","iopub.execute_input":"2022-12-02T11:40:50.899714Z","iopub.status.idle":"2022-12-02T11:40:50.913102Z","shell.execute_reply.started":"2022-12-02T11:40:50.899652Z","shell.execute_reply":"2022-12-02T11:40:50.912034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Let it run! :-)","metadata":{}},{"cell_type":"code","source":"BACKBONE = \"resnet_50\"\nBATCH_SIZE = 16\nTEST_BATCH_SIZE = 5\nMIN_VALUE = 0\nMAX_VALUE = 90\nSTEPS = 50\nEPOCHS = 20\n\nLR = 0.0001","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.914625Z","iopub.execute_input":"2022-12-02T11:40:50.914953Z","iopub.status.idle":"2022-12-02T11:40:50.926161Z","shell.execute_reply.started":"2022-12-02T11:40:50.914881Z","shell.execute_reply":"2022-12-02T11:40:50.925496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_preprocessor = Preprocessor(path=train_dir,\n                                  backbone=pretrained_models[BACKBONE],\n                                  hu_min_value=MIN_VALUE,\n                                  hu_max_value=MAX_VALUE,\n                                  augment=True)\n\ndev_preprocessor = Preprocessor(path=train_dir,\n                                backbone=pretrained_models[BACKBONE],\n                                hu_min_value=MIN_VALUE,\n                                hu_max_value=MAX_VALUE,\n                                augment=False)\n\ntest_preprocessor = Preprocessor(path=test_dir,\n                                backbone=pretrained_models[BACKBONE],\n                                hu_min_value=MIN_VALUE,\n                                hu_max_value=MAX_VALUE,\n                                augment=False)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.92872Z","iopub.execute_input":"2022-12-02T11:40:50.92929Z","iopub.status.idle":"2022-12-02T11:40:50.938803Z","shell.execute_reply.started":"2022-12-02T11:40:50.929241Z","shell.execute_reply":"2022-12-02T11:40:50.938033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(1,4,figsize=(20,20))\n\n\nfor m in range(4):\n    example = train_data.index.values[m]\n    title = [col for col in train_data.loc[example,:].index if train_data.loc[example, col]==1]\n    if len(title) == 0:\n        title=\"Healthy\"\n    preprocess_example = train_preprocessor.preprocess(example)\n    ax[m].imshow(preprocess_example[:,:,2], cmap=\"Spectral\")\n    ax[m].grid(False)\n    ax[m].set_title(title);","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:50.940118Z","iopub.execute_input":"2022-12-02T11:40:50.940732Z","iopub.status.idle":"2022-12-02T11:40:51.799213Z","shell.execute_reply.started":"2022-12-02T11:40:50.94037Z","shell.execute_reply":"2022-12-02T11:40:51.79846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(2,1,figsize=(20,10))\nsns.distplot(preprocess_example[:,:,2].flatten(), kde=False, ax=ax[0])\nsns.distplot(train_preprocessor.normalize(preprocess_example)[:,:,2].flatten(), kde=False)\nplt.title(\"Image distribution after normalisation\");","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:51.800478Z","iopub.execute_input":"2022-12-02T11:40:51.800859Z","iopub.status.idle":"2022-12-02T11:40:52.368423Z","shell.execute_reply.started":"2022-12-02T11:40:51.800818Z","shell.execute_reply":"2022-12-02T11:40:52.367458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_preprocessor.normalize(preprocess_example[:,:,2]).min()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.370163Z","iopub.execute_input":"2022-12-02T11:40:52.370612Z","iopub.status.idle":"2022-12-02T11:40:52.378867Z","shell.execute_reply.started":"2022-12-02T11:40:52.370427Z","shell.execute_reply":"2022-12-02T11:40:52.377985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_preprocessor.normalize(preprocess_example[:,:,2]).max()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.380191Z","iopub.execute_input":"2022-12-02T11:40:52.380736Z","iopub.status.idle":"2022-12-02T11:40:52.388244Z","shell.execute_reply.started":"2022-12-02T11:40:52.380687Z","shell.execute_reply":"2022-12-02T11:40:52.387507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataloader = DataLoader(train_data,\n                              train_preprocessor,\n                              BATCH_SIZE,\n                              shuffle=True,\n                              steps=STEPS)\n\ndev_dataloader = DataLoader(dev_data, \n                            dev_preprocessor,\n                            BATCH_SIZE,\n                            shuffle=True,\n                            steps=STEPS)\n\ntest_dataloader = DataLoader(testdf, \n                             test_preprocessor,\n                             TEST_BATCH_SIZE,\n                             shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.389625Z","iopub.execute_input":"2022-12-02T11:40:52.390249Z","iopub.status.idle":"2022-12-02T11:40:52.571502Z","shell.execute_reply.started":"2022-12-02T11:40:52.390199Z","shell.execute_reply":"2022-12-02T11:40:52.570726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataloader.__len__()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.57397Z","iopub.execute_input":"2022-12-02T11:40:52.574422Z","iopub.status.idle":"2022-12-02T11:40:52.580193Z","shell.execute_reply.started":"2022-12-02T11:40:52.574372Z","shell.execute_reply":"2022-12-02T11:40:52.579306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_dataloader.data_ids)/BATCH_SIZE","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.581636Z","iopub.execute_input":"2022-12-02T11:40:52.582161Z","iopub.status.idle":"2022-12-02T11:40:52.588977Z","shell.execute_reply.started":"2022-12-02T11:40:52.58211Z","shell.execute_reply":"2022-12-02T11:40:52.588161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.loc[train_dataloader.data_ids][\"any\"].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.590413Z","iopub.execute_input":"2022-12-02T11:40:52.591014Z","iopub.status.idle":"2022-12-02T11:40:52.657739Z","shell.execute_reply.started":"2022-12-02T11:40:52.590856Z","shell.execute_reply":"2022-12-02T11:40:52.657139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataloader.__len__()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.659074Z","iopub.execute_input":"2022-12-02T11:40:52.659331Z","iopub.status.idle":"2022-12-02T11:40:52.666735Z","shell.execute_reply.started":"2022-12-02T11:40:52.659287Z","shell.execute_reply":"2022-12-02T11:40:52.665902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dev_dataloader.__len__()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.668036Z","iopub.execute_input":"2022-12-02T11:40:52.668504Z","iopub.status.idle":"2022-12-02T11:40:52.674568Z","shell.execute_reply.started":"2022-12-02T11:40:52.668445Z","shell.execute_reply":"2022-12-02T11:40:52.673806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_class_weights = [0.5, 0.1, 0.1, 0.1, 0.1, 0.1]","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.676126Z","iopub.execute_input":"2022-12-02T11:40:52.676718Z","iopub.status.idle":"2022-12-02T11:40:52.681293Z","shell.execute_reply.started":"2022-12-02T11:40:52.676648Z","shell.execute_reply":"2022-12-02T11:40:52.680279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def turn_pred_to_dataframe(data_df, pred):\n    df = pd.DataFrame(pred, columns=data_df.columns, index=data_df.index)\n    df = df.stack().reset_index()\n    df.loc[:, \"ID\"] = df.id.str.cat(df.subtype, sep=\"_\")\n    df = df.drop([\"id\", \"subtype\"], axis=1)\n    df = df.rename({0: \"Label\"}, axis=1)\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:40:52.682628Z","iopub.execute_input":"2022-12-02T11:40:52.683253Z","iopub.status.idle":"2022-12-02T11:40:52.691025Z","shell.execute_reply.started":"2022-12-02T11:40:52.682981Z","shell.execute_reply":"2022-12-02T11:40:52.690014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if train_brute_force:\n    model = MyNetwork(model_fun=resnet_50, #ResNet-50 is a convolutional neural network that is 50 layers deep\n                      loss_fun=\"binary_crossentropy\", #multilabel_focal_loss(class_weights=my_class_weights, alpha=0.5, gamma=0),\n                      metrics_list=[multilabel_focal_loss(alpha=0.5, gamma=0)],\n                      train_generator=train_dataloader,\n                      dev_generator=dev_dataloader,\n                      epochs=EPOCHS,\n                      num_classes=6)\n    model.build_model()\n    model.compile_model()\n    history = model.learn()\n    \n    print(history.history.keys())\n    \n    fig, ax = plt.subplots(2,1,figsize=(20,5))\n    ax[0].plot(history.history[\"loss\"], 'o-')\n    ax[0].plot(history.history[\"val_loss\"], 'o-')\n    ax[1].plot(history.history[\"lr\"], 'o-')\n    \n    #test_pred = model.predict(test_dataloader)[0:testdf.shape[0]]\n    #dev_pred = model.predict(dev_dataloader)\n    \n    #test_pred_df = turn_pred_to_dataframe(testdf, test_pred)\n    #dev_pred_df = turn_pred_to_dataframe(dev_data, dev_pred)\n    \n    #test_pred_df.to_csv(\"brute_force_test_pred.csv\", index=False)\n    #dev_pred_df.to_csv(\"brute_force_dev_pred.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:42:30.19949Z","iopub.execute_input":"2022-12-02T11:42:30.199818Z","iopub.status.idle":"2022-12-02T11:49:17.459648Z","shell.execute_reply.started":"2022-12-02T11:42:30.199743Z","shell.execute_reply":"2022-12-02T11:49:17.4589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dev_proba = model.predict(dev_dataloader) \ndev_proba.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:36.581064Z","iopub.execute_input":"2022-12-02T11:50:36.581372Z","iopub.status.idle":"2022-12-02T11:50:46.78268Z","shell.execute_reply.started":"2022-12-02T11:50:36.581319Z","shell.execute_reply":"2022-12-02T11:50:46.781993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20,5))\nfor n in range(6):\n    sns.distplot(dev_proba[:,n])","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:46.784743Z","iopub.execute_input":"2022-12-02T11:50:46.785095Z","iopub.status.idle":"2022-12-02T11:50:47.400264Z","shell.execute_reply.started":"2022-12-02T11:50:46.785038Z","shell.execute_reply":"2022-12-02T11:50:47.399386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building up the any-subtype network","metadata":{}},{"cell_type":"markdown","source":"## The custom loss <a class=\"anchor\" id=\"customloss\"></a>\n\nIn my opinion the first important part is to reconstruct the **weighted multilabel logarithmic loss**. I have already touched it a bit during the [evaluation part](#evaluation) in the beginning of this kernel:\n\n$$l_{n,s} = t_{n,s} \\cdot \\ln(y_{n,s}) + (1-t_{n,s}) \\cdot \\ln(1-y_{n,s}) $$  \n\nThen they seem to be added, whereas the any-subtype obtains a higher weight than the others:\n\n$$ l_{n} = \\sum_{s} w_{s} \\cdot l_{n,s} $$\n\nAnd finally this loss is averaged over all samples:\n\n$$ Loss = -\\frac{1}{N} \\cdot \\sum_{n} l_{n} $$\n\nBut there was one part I haven't mentioned: **The any part depends on the subtype decisions. In contrast we can assume that the subtypes themselves are independent from each other to make the problem simpler**. Hence if you have epidural hemorrhage this might not mean that there is a higher probability of intraparenchymal as well. By assuming independece we can decouple the subtypes and use the binary xentropy loss as described above. But this does not include the any-part. In this case we need to take care of the dependence of any (A) and subtype (S) and could write:\n\n$$P(A,S) = P(A) \\cdot P(S|A) = P(S) \\cdot P(A|S) $$\n\nConsequently we could either let the network answer the questions: \n* Do you have any hemorrhage? And dependent on this answer we could make subtype decisions. \n* Or we could do it the other way round: Do you have subtype s? And dependent on the answers of each subtype - is there any kind of hemorrhage?\n\n**I feel more comfortable with the first way**. Answering the any-part shows lower class imbalance than all the others (no wonders). By yielding a better fit for this part, we have additional fruitful information (the probability of any) to answer the second more difficutl part of the subtypes which shows very high class imbalance. Would love to discuss about it. If you like, let me know if you find this important or not in the comments.","metadata":{}},{"cell_type":"markdown","source":"## A two output layer network <a class=\"anchor\" id=\"twooutputs\"></a>\n\nTo realize the idea of making the subtype decision based on the any-decision, we need to rework a bit the network architecture:\n\n* We need an output for the any decision based on one layer (i) of our choice.\n* We need an output for the subtype decision that is based on the layer (i) **and** the output of the any-decision.\n\nLet's extend our MyNetwork class. This way we can switch between brute force approach (decouple all targets) and our \"keep any-subtype dependence\"-approach: ","metadata":{}},{"cell_type":"code","source":"alpha_subtypes = 0.25 \ngamma_subtypes = 2","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:47.401692Z","iopub.execute_input":"2022-12-02T11:50:47.402234Z","iopub.status.idle":"2022-12-02T11:50:47.406501Z","shell.execute_reply.started":"2022-12-02T11:50:47.402183Z","shell.execute_reply":"2022-12-02T11:50:47.405696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class AnySubtypeNetwork(MyNetwork):\n    \n    def __init__(self,\n                 model_fun,\n                 loss_fun,\n                 metrics_list,\n                 train_generator,\n                 dev_generator,\n                 epochs,\n                 num_subtype_classes=5,\n                 checkpoint_path=MODELOUTPUT_PATH_ANYSUBTYPE):\n        MyNetwork.__init__(self, \n                           model_fun=model_fun,\n                           loss_fun=loss_fun,\n                           metrics_list=metrics_list,\n                           train_generator=train_generator,\n                           dev_generator=dev_generator,\n                           epochs=epochs,\n                           num_classes=num_subtype_classes)\n    \n    def build_model(self):\n        base_model = self.model_fun()\n        x = base_model.output\n        x = GlobalAveragePooling2D()(x)\n        x = Dropout(0.5)(x)\n        any_logits = Dense(1, kernel_initializer=he_normal(seed=11))(x)\n        any_pred = Activation(\"sigmoid\", name=\"any_predictions\")(any_logits)\n        x = concatenate([any_pred, x])\n        sub_pred = Dense(self.num_classes,\n                         name=\"subtype_pred\",\n                         kernel_initializer=he_normal(seed=12),\n                         activation=\"sigmoid\")(x) \n        self.model = Model(inputs=base_model.input, outputs=[any_pred, sub_pred])\n    \n    def compile_model(self):\n        self.model.compile(optimizer=Adam(LR),\n                           loss=['binary_crossentropy', multilabel_focal_loss(alpha=alpha_subtypes, gamma=gamma_subtypes)],\n                           loss_weights = [1., 0.],\n                           metrics=self.metrics_list)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:47.408034Z","iopub.execute_input":"2022-12-02T11:50:47.408668Z","iopub.status.idle":"2022-12-02T11:50:47.420929Z","shell.execute_reply.started":"2022-12-02T11:50:47.408604Z","shell.execute_reply":"2022-12-02T11:50:47.420232Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now you can see that the any loss and the subtype loss are weighted the same. Doing so the any-part has comparable higher weight than any of the single subtypes. This is still a simplification. We don't know it neither for the any nor for single subtypes. **But we have taken the first advice \"weight any higher\" into account**.","metadata":{}},{"cell_type":"markdown","source":"## Multioutput DataLoader <a class=\"anchor\" id=\"twooutputsdataloader\"></a>","metadata":{}},{"cell_type":"code","source":"class AnySubtypeDataLoader(Sequence):\n    \n    def __init__(self, dataframe,\n                 preprocessor,\n                 batch_size,\n                 num_classes=5,\n                 shuffle=False,\n                 steps=None):\n        self.preprocessor = preprocessor\n        self.data_ids = dataframe.index.values\n        self.dataframe = dataframe\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.input_shape = self.preprocessor.backbone[\"nn_input_shape\"]\n        self.preprocess_fun = self.preprocessor.backbone[\"preprocess_fun\"]\n        self.num_classes = num_classes\n        self.current_epoch=0\n        \n        self.steps=steps\n        if self.steps is not None:\n            self.steps = np.round(self.steps/2) * 2\n            self.undersample()\n        \n    def undersample(self):\n        part = np.int(self.steps/2 * self.batch_size)\n        zero_ids = np.random.choice(self.dataframe.loc[self.dataframe[\"any\"] == 0].index.values, size=1*part, replace=False)\n        hot_ids = np.random.choice(self.dataframe.loc[self.dataframe[\"any\"] == 1].index.values, size=1*part, replace=False)\n        self.data_ids = list(set(zero_ids).union(hot_ids))\n        np.random.shuffle(self.data_ids)\n    \n    # defines the number of steps per epoch\n    def __len__(self):\n        if self.steps is None:\n            return np.int(np.ceil(len(self.data_ids) / np.float(self.batch_size)))\n        else:\n            return 2*np.int(self.steps/2) \n    \n    # at the end of an epoch: \n    def on_epoch_end(self):\n        # if steps is None and shuffle is true:\n        if self.steps is None:\n            self.data_ids = self.dataframe.index.values\n            if self.shuffle:\n                np.random.shuffle(self.data_ids)\n        else:\n            self.undersample()\n        self.current_epoch += 1\n    \n    # should return a batch of images\n    def __getitem__(self, item):\n        # select the ids of the current batch\n        current_ids = self.data_ids[item*self.batch_size:(item+1)*self.batch_size]\n        X, y_any, y_subtype = self.__generate_batch(current_ids)\n        return X, [y_any, y_subtype]\n    \n    # collect the preprocessed images and targets of one batch\n    def __generate_batch(self, current_ids):\n        X = np.empty((self.batch_size, *self.input_shape, 3))\n        y_subtype = np.empty((self.batch_size, self.num_classes))\n        y_any = np.empty((self.batch_size, 1))\n        for idx, ident in enumerate(current_ids):\n            # Store sample\n            image = self.preprocessor.preprocess(ident)\n            X[idx] = self.preprocessor.normalize(image)\n            # Store class\n            y_any[idx], y_subtype[idx] = self.__get_target(ident)\n        return X, y_any, y_subtype\n    \n    # extract the targets of one image id:\n    def __get_target(self, ident):\n        y_any = self.dataframe.loc[ident, \"any\"]\n        y_subtype = self.dataframe.drop(\"any\", axis=1).loc[ident].values\n        return y_any, y_subtype","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:47.425854Z","iopub.execute_input":"2022-12-02T11:50:47.426278Z","iopub.status.idle":"2022-12-02T11:50:47.445351Z","shell.execute_reply.started":"2022-12-02T11:50:47.426213Z","shell.execute_reply":"2022-12-02T11:50:47.444374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataloader = AnySubtypeDataLoader(train_data,\n                                        train_preprocessor,\n                                        BATCH_SIZE,\n                                        steps=STEPS)\ndev_dataloader = AnySubtypeDataLoader(dev_data, \n                                      dev_preprocessor,\n                                      BATCH_SIZE,\n                                      steps=STEPS,\n                                      shuffle=False)\n\ntest_dataloader = AnySubtypeDataLoader(testdf, \n                                       test_preprocessor,\n                                       TEST_BATCH_SIZE,\n                                       shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:47.447692Z","iopub.execute_input":"2022-12-02T11:50:47.448208Z","iopub.status.idle":"2022-12-02T11:50:47.586289Z","shell.execute_reply.started":"2022-12-02T11:50:47.448143Z","shell.execute_reply":"2022-12-02T11:50:47.585436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X, [y1, y2] = train_dataloader.__getitem__(0)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:47.587878Z","iopub.execute_input":"2022-12-02T11:50:47.588393Z","iopub.status.idle":"2022-12-02T11:50:48.053112Z","shell.execute_reply.started":"2022-12-02T11:50:47.588152Z","shell.execute_reply":"2022-12-02T11:50:48.052267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y1.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.054756Z","iopub.execute_input":"2022-12-02T11:50:48.055091Z","iopub.status.idle":"2022-12-02T11:50:48.062069Z","shell.execute_reply.started":"2022-12-02T11:50:48.05504Z","shell.execute_reply":"2022-12-02T11:50:48.061164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y2.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.063662Z","iopub.execute_input":"2022-12-02T11:50:48.064201Z","iopub.status.idle":"2022-12-02T11:50:48.071069Z","shell.execute_reply.started":"2022-12-02T11:50:48.064149Z","shell.execute_reply":"2022-12-02T11:50:48.070138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.072494Z","iopub.execute_input":"2022-12-02T11:50:48.073177Z","iopub.status.idle":"2022-12-02T11:50:48.079495Z","shell.execute_reply.started":"2022-12-02T11:50:48.073121Z","shell.execute_reply":"2022-12-02T11:50:48.078484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y1[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.080984Z","iopub.execute_input":"2022-12-02T11:50:48.081611Z","iopub.status.idle":"2022-12-02T11:50:48.08785Z","shell.execute_reply.started":"2022-12-02T11:50:48.081505Z","shell.execute_reply":"2022-12-02T11:50:48.087024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y2[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.089478Z","iopub.execute_input":"2022-12-02T11:50:48.090175Z","iopub.status.idle":"2022-12-02T11:50:48.096103Z","shell.execute_reply.started":"2022-12-02T11:50:48.090124Z","shell.execute_reply":"2022-12-02T11:50:48.095251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(X[0,:,:,0])","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.09768Z","iopub.execute_input":"2022-12-02T11:50:48.098195Z","iopub.status.idle":"2022-12-02T11:50:48.26346Z","shell.execute_reply.started":"2022-12-02T11:50:48.098144Z","shell.execute_reply":"2022-12-02T11:50:48.262742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataloader.__len__()","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.26474Z","iopub.execute_input":"2022-12-02T11:50:48.265184Z","iopub.status.idle":"2022-12-02T11:50:48.271006Z","shell.execute_reply.started":"2022-12-02T11:50:48.265125Z","shell.execute_reply":"2022-12-02T11:50:48.270284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(train_dataloader.data_ids)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.272554Z","iopub.execute_input":"2022-12-02T11:50:48.273102Z","iopub.status.idle":"2022-12-02T11:50:48.279824Z","shell.execute_reply.started":"2022-12-02T11:50:48.273052Z","shell.execute_reply":"2022-12-02T11:50:48.278875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data.loc[train_dataloader.data_ids].sum() / train_data.loc[train_dataloader.data_ids].shape[0]","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.281216Z","iopub.execute_input":"2022-12-02T11:50:48.281698Z","iopub.status.idle":"2022-12-02T11:50:48.298167Z","shell.execute_reply.started":"2022-12-02T11:50:48.281648Z","shell.execute_reply":"2022-12-02T11:50:48.297419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if train_anysubtype_network:\n    model = AnySubtypeNetwork(model_fun=resnet_50,\n                              loss_fun=None,\n                              metrics_list={\"any_predictions\":\"binary_crossentropy\",\n                                            \"subtype_pred\": multilabel_focal_loss(alpha=0.5, gamma=0)},\n                              train_generator=train_dataloader,\n                              dev_generator=dev_dataloader,\n                              epochs=50) \n    model.build_model()\n    model.compile_model()\n    history = model.learn()\n    \n    print(history.history.keys())\n    \n    fig, ax = plt.subplots(2,1,figsize=(20,10))\n    ax[0].plot(history.history[\"loss\"], 'o-')\n    ax[0].plot(history.history[\"val_loss\"], 'o-')\n    ax[1].plot(history.history[\"lr\"], 'o-')\nelse:\n    model = AnySubtypeNetwork(model_fun=resnet_50,\n                              loss_fun=None,\n                              metrics_list={\"any_predictions\":\"binary_crossentropy\",\n                                            \"subtype_pred\": multilabel_focal_loss(alpha=0.5, gamma=0)},\n                              train_generator=train_dataloader,\n                              dev_generator=dev_dataloader,\n                              epochs=50) \n    model.build_model()\n    #model.load_weights(any_subtype_model_input)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:48.299659Z","iopub.execute_input":"2022-12-02T11:50:48.300112Z","iopub.status.idle":"2022-12-02T11:50:57.226186Z","shell.execute_reply.started":"2022-12-02T11:50:48.300064Z","shell.execute_reply":"2022-12-02T11:50:57.225317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dev_proba_any, dev_proba_subtype = model.predict(dev_dataloader)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:50:57.227531Z","iopub.execute_input":"2022-12-02T11:50:57.227828Z","iopub.status.idle":"2022-12-02T11:51:10.810388Z","shell.execute_reply.started":"2022-12-02T11:50:57.227781Z","shell.execute_reply":"2022-12-02T11:51:10.809497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_proba_any, train_proba_subtype = model.predict(train_dataloader)","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:51:10.812891Z","iopub.execute_input":"2022-12-02T11:51:10.813227Z","iopub.status.idle":"2022-12-02T11:51:27.709921Z","shell.execute_reply.started":"2022-12-02T11:51:10.813165Z","shell.execute_reply":"2022-12-02T11:51:27.708818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dev_proba_any.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:51:27.711394Z","iopub.execute_input":"2022-12-02T11:51:27.711664Z","iopub.status.idle":"2022-12-02T11:51:27.717793Z","shell.execute_reply.started":"2022-12-02T11:51:27.711618Z","shell.execute_reply":"2022-12-02T11:51:27.716905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dev_proba_subtype.shape","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:51:27.719165Z","iopub.execute_input":"2022-12-02T11:51:27.719693Z","iopub.status.idle":"2022-12-02T11:51:27.727972Z","shell.execute_reply.started":"2022-12-02T11:51:27.719634Z","shell.execute_reply":"2022-12-02T11:51:27.727047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if train_anysubtype_network:\n    fig, ax = plt.subplots(2,1, figsize=(20,10))\n    sns.distplot(dev_proba_any[:,0], ax=ax[0], color=\"Purple\")\n    sns.distplot(dev_proba_subtype[:,0], ax=ax[1])\n    sns.distplot(dev_proba_subtype[:,1], ax=ax[1])\n    sns.distplot(dev_proba_subtype[:,2], ax=ax[1])\n    sns.distplot(dev_proba_subtype[:,3], ax=ax[1])\n    sns.distplot(dev_proba_subtype[:,4], ax=ax[1])\n    ax[0].set_title(\"Predicted probability of hemorrhage occurence in dev batch\")\n    ax[1].set_title(\"Predicted probability of hemorrhage subtypes in dev batch\")\n    \n    fig, ax = plt.subplots(2,1, figsize=(20,10))\n    sns.distplot(train_proba_any[:,0], ax=ax[0], color=\"Purple\")\n    sns.distplot(train_proba_subtype[:,0], ax=ax[1])\n    sns.distplot(train_proba_subtype[:,1], ax=ax[1])\n    sns.distplot(train_proba_subtype[:,2], ax=ax[1])\n    sns.distplot(train_proba_subtype[:,3], ax=ax[1])\n    sns.distplot(train_proba_subtype[:,4], ax=ax[1])\n    ax[0].set_title(\"Predicted probability of hemorrhage occurence in train\")\n    ax[1].set_title(\"Predicted probability of hemorrhage subtypes in train\")","metadata":{"execution":{"iopub.status.busy":"2022-12-02T11:51:43.894605Z","iopub.execute_input":"2022-12-02T11:51:43.894993Z","iopub.status.idle":"2022-12-02T11:51:45.307389Z","shell.execute_reply.started":"2022-12-02T11:51:43.894929Z","shell.execute_reply":"2022-12-02T11:51:45.30664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}