{"cells":[{"metadata":{},"cell_type":"markdown","source":"This competition provides an exciting and challenging task of doing multi-label classification on a dataset with well over half a million images. There are multiple very nice notebooks which perform only 2 or 3 epochs with all the training data. In this notebook I will try out and see what the effect is of using more epochs but less steps per epoch. By averaging the predictions made during the last few epochs we should be able to achieve a nice LB score. This also should provide some alternative ways to experiment for the Kagglers that don't have the adequate computing resources available and are dependent on Kaggle Kernels.\n\nAs model I will be using the EfficientNet B2 model. It should be able to provide highly accurate predictions while still being able to run within the kernel limits. With 9 hours max time for a GPU kernel you have to make some trade-offs ;-)\n\nI hope this kernel will be usefull and may'be will provide you with some new and alternative ideas to try out. If you like it..then please upvote it ;-)\nAny feedback or remarks are appreciated.\n\nLets start by importing all the necessary modules.\n\nNote!! This kernel is now updated for Stage2 Training and Test data..altough with less epochs because of the increase in train and test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport pydicom\nimport os\nimport collections\nimport sys\nimport glob\nimport random\nimport cv2\nimport tensorflow as tf\nimport multiprocessing\n\nfrom math import ceil, floor\nfrom copy import deepcopy\nfrom tqdm import tqdm_notebook as tqdm\nfrom imgaug import augmenters as iaa\n\nimport keras\nimport keras.backend as K\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.layers import Dense, Flatten, Dropout, GlobalAveragePooling2D\nfrom keras.models import Model, load_model\nfrom keras.utils import Sequence\nfrom keras.losses import binary_crossentropy\nfrom keras.optimizers import Adam","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Install and import the efficientnet and iterative-stratification packages from the internet. The iterative-stratification package provides a very nice implementation of multi-label stratification. I've used it in a few competitions now with good results. There are offcourse more packages that provide implementations for it."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Install Modules from internet\n!pip install efficientnet\n!pip install iterative-stratification\n!pip install image-classifiers==1.0.0b1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Import Custom Modules\nimport efficientnet.keras as efn \nfrom iterstrat.ml_stratifiers import MultilabelStratifiedShuffleSplit\nfrom classification_models.keras import Classifiers","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we will set the random_state, some constants and folders that will be used later on. I've specified a rather small test size as I want to maximize the training time available and minimize the time used for validation. I'am not using methods like early stopping...when the kernel time limit is approaching we could still increase the results on the LB if we were allowed to continue."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Seed\nSEED = 12345\nnp.random.seed(SEED)\ntf.set_random_seed(SEED)\n\n# Constants\nTEST_SIZE = 0.02\nHEIGHT = 256\nWIDTH = 256\nCHANNELS = 3\nTRAIN_BATCH_SIZE = 16\nVALID_BATCH_SIZE = 16\nSHAPE = (HEIGHT, WIDTH, CHANNELS)\n\n# Folders\nDATA_DIR = '/kaggle/input/rsna-intracranial-hemorrhage-detection/rsna-intracranial-hemorrhage-detection/'\nTEST_IMAGES_DIR = DATA_DIR + 'stage_2_test/'\nTRAIN_IMAGES_DIR = DATA_DIR + 'stage_2_train/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next the code for the DICOM windowing and the Data Generators. After seeing the effect of different versions of windowing as presented in this very nice [kernel](https://www.kaggle.com/akensert/inceptionv3-prev-resnet50-keras-baseline-model) I decided to also update my kernel with it. Lets see what the effect will be."},{"metadata":{"trusted":true},"cell_type":"code","source":"def correct_dcm(dcm):\n    x = dcm.pixel_array + 1000\n    px_mode = 4096\n    x[x>=px_mode] = x[x>=px_mode] - px_mode\n    dcm.PixelData = x.tobytes()\n    dcm.RescaleIntercept = -1000\n\ndef window_image(dcm, window_center, window_width):    \n    if (dcm.BitsStored == 12) and (dcm.PixelRepresentation == 0) and (int(dcm.RescaleIntercept) > -100):\n        correct_dcm(dcm)\n    img = dcm.pixel_array * dcm.RescaleSlope + dcm.RescaleIntercept\n    \n    # Resize\n    img = cv2.resize(img, SHAPE[:2], interpolation = cv2.INTER_LINEAR)\n   \n    img_min = window_center - window_width // 2\n    img_max = window_center + window_width // 2\n    img = np.clip(img, img_min, img_max)\n    return img\n\ndef bsb_window(dcm):\n    brain_img = window_image(dcm, 40, 80)\n    subdural_img = window_image(dcm, 80, 200)\n    soft_img = window_image(dcm, 40, 380)\n    \n    brain_img = (brain_img - 0) / 80\n    subdural_img = (subdural_img - (-20)) / 200\n    soft_img = (soft_img - (-150)) / 380\n    bsb_img = np.array([brain_img, subdural_img, soft_img]).transpose(1,2,0)\n    return bsb_img\n\ndef _read(path, SHAPE):\n    dcm = pydicom.dcmread(path)\n    try:\n        img = bsb_window(dcm)\n    except:\n        img = np.zeros(SHAPE)\n    return img","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll specify some light image augmentation. Some horizontal and vertical flipping and some cropping. I haven't yet tried out more augmentation but will do so in future versions of the kernel. Also the code for Data Generators for train and test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Image Augmentation\nsometimes = lambda aug: iaa.Sometimes(0.25, aug)\naugmentation = iaa.Sequential([ iaa.Fliplr(0.25),\n                                iaa.Flipud(0.10),\n                                sometimes(iaa.Crop(px=(0, 25), keep_size = True, sample_independently = False))   \n                            ], random_order = True)       \n        \n# Generators\nclass TrainDataGenerator(keras.utils.Sequence):\n    def __init__(self, dataset, labels, batch_size = 16, img_size = SHAPE, img_dir = TRAIN_IMAGES_DIR, augment = False, *args, **kwargs):\n        self.dataset = dataset\n        self.ids = dataset.index\n        self.labels = labels\n        self.batch_size = batch_size\n        self.img_size = img_size\n        self.img_dir = img_dir\n        self.augment = augment\n        self.on_epoch_end()\n\n    def __len__(self):\n        return int(ceil(len(self.ids) / self.batch_size))\n\n    def __getitem__(self, index):\n        indices = self.indices[index*self.batch_size:(index+1)*self.batch_size]\n        X, Y = self.__data_generation(indices)\n        return X, Y\n\n    def augmentor(self, image):\n        augment_img = augmentation        \n        image_aug = augment_img.augment_image(image)\n        return image_aug\n\n    def on_epoch_end(self):\n        self.indices = np.arange(len(self.ids))\n        np.random.shuffle(self.indices)\n\n    def __data_generation(self, indices):\n        X = np.empty((self.batch_size, *self.img_size))\n        Y = np.empty((self.batch_size, 6), dtype=np.float32)\n        \n        for i, index in enumerate(indices):\n            ID = self.ids[index]\n            image = _read(self.img_dir+ID+\".dcm\", self.img_size)\n            if self.augment:\n                X[i,] = self.augmentor(image)\n            else:\n                X[i,] = image\n            Y[i,] = self.labels.iloc[index].values        \n        return X, Y\n    \nclass TestDataGenerator(keras.utils.Sequence):\n    def __init__(self, dataset, labels, batch_size = 16, img_size = SHAPE, img_dir = TEST_IMAGES_DIR, *args, **kwargs):\n        self.dataset = dataset\n        self.ids = dataset.index\n        self.labels = labels\n        self.batch_size = batch_size\n        self.img_size = img_size\n        self.img_dir = img_dir\n        self.on_epoch_end()\n\n    def __len__(self):\n        return int(ceil(len(self.ids) / self.batch_size))\n\n    def __getitem__(self, index):\n        indices = self.indices[index*self.batch_size:(index+1)*self.batch_size]\n        X = self.__data_generation(indices)\n        return X\n\n    def on_epoch_end(self):\n        self.indices = np.arange(len(self.ids))\n    \n    def __data_generation(self, indices):\n        X = np.empty((self.batch_size, *self.img_size))\n        \n        for i, index in enumerate(indices):\n            ID = self.ids[index]\n            image = _read(self.img_dir+ID+\".dcm\", self.img_size)\n            X[i,] = image              \n        return X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Import the training and test datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_testset(filename = DATA_DIR + \"stage_2_sample_submission.csv\"):\n    df = pd.read_csv(filename)\n    df[\"Image\"] = df[\"ID\"].str.slice(stop=12)\n    df[\"Diagnosis\"] = df[\"ID\"].str.slice(start=13)\n    df = df.loc[:, [\"Label\", \"Diagnosis\", \"Image\"]]\n    df = df.set_index(['Image', 'Diagnosis']).unstack(level=-1)\n    return df\n\ndef read_trainset(filename = DATA_DIR + \"stage_2_train.csv\"):\n    df = pd.read_csv(filename)\n    df[\"Image\"] = df[\"ID\"].str.slice(stop=12)\n    df[\"Diagnosis\"] = df[\"ID\"].str.slice(start=13)\n    duplicates_to_remove = [56346, 56347, 56348, 56349,\n                            56350, 56351, 1171830, 1171831,\n                            1171832, 1171833, 1171834, 1171835,\n                            3705312, 3705313, 3705314, 3705315,\n                            3705316, 3705317, 3842478, 3842479,\n                            3842480, 3842481, 3842482, 3842483 ]\n    df = df.drop(index = duplicates_to_remove)\n    df = df.reset_index(drop = True)    \n    df = df.loc[:, [\"Label\", \"Diagnosis\", \"Image\"]]\n    df = df.set_index(['Image', 'Diagnosis']).unstack(level=-1)\n    return df\n\n# Read Train and Test Datasets\ntest_df = read_testset()\ntrain_df = read_trainset()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training data contains some class inbalance. Multiple kernels explored the use of undersampling..so let's try the opposite and oversample the minority class 'epidural' one additional time."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Oversampling\nepidural_df = train_df[train_df.Label['epidural'] == 1]\ntrain_oversample_df = pd.concat([train_df, epidural_df])\ntrain_df = train_oversample_df\n\n# Summary\nprint('Train Shape: {}'.format(train_df.shape))\nprint('Test Shape: {}'.format(test_df.shape))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some methods for predictions on the test data, a callback method and a method to create the EfficientNet B2 model. For the EfficientNet we use the pretrained imagenet weights. Also a Dropout layer is added with a small value to prevent some overfitting. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def predictions(test_df, model):    \n    test_preds = model.predict_generator(TestDataGenerator(test_df, None, 8, SHAPE, TEST_IMAGES_DIR), verbose = 1)\n    return test_preds[:test_df.iloc[range(test_df.shape[0])].shape[0]]\n\ndef ModelCheckpointFull(model_name):\n    return ModelCheckpoint(model_name, \n                            monitor = 'val_loss', \n                            verbose = 1, \n                            save_best_only = False, \n                            save_weights_only = True, \n                            mode = 'min', \n                            period = 1)\n\n# Create Model\ndef create_model():\n    K.clear_session()\n    seresnext50, _ = Classifiers.get('seresnext50')\n    base_model =  seresnext50(weights = 'imagenet', include_top = False, input_shape = SHAPE)\n    x = base_model.output\n    x = GlobalAveragePooling2D()(x)\n    x = Dropout(0.15)(x)\n    y_pred = Dense(6, activation = 'sigmoid')(x)\n\n    return Model(inputs = base_model.input, outputs = y_pred)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we setup the multi label stratification. I've specified multiple splits but only using the first one for train data and validation data. Optionally you can also loop through the different splits and use a different train and validation set for each epoch. "},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Multi Label Stratified Split stuff...\nmsss = MultilabelStratifiedShuffleSplit(n_splits = 10, test_size = TEST_SIZE, random_state = SEED)\nX = train_df.index\nY = train_df.Label.values\n\n# Get train and test index\nmsss_splits = next(msss.split(X, Y))\ntrain_idx = msss_splits[0]\nvalid_idx = msss_splits[1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can train the model for a number of epochs. All epochs we train the full model but each time on only 1/6 of the train data. With each epoch only a subset of the train data will allow us to make more epochs and allows todo averaging over more then just 1 or 2 epochs (compared to using all data every epoch).\n\nNote that I recreate the data generators and model on each epoch. This is only necessary when using the different Multi-label stratified splits since the data generators will get a totally different set of data on each epoch then. I left it in so that you can try it out.\n\nStarting with the 4th epoch a prediction for the test set is made on each epoch. In total predictions from the last 5 epochs will be averaged this way for the final submission."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loop through Folds of Multi Label Stratified Split\n#for epoch, msss_splits in zip(range(0, 9), msss.split(X, Y)): \n#    # Get train and test index\n#    train_idx = msss_splits[0]\n#    valid_idx = msss_splits[1]\n\n# Shuffle Train data\nnp.random.shuffle(train_idx)\nprint(train_idx[:5])    \nprint(valid_idx[:5])\n\n# Create Data Generators for Train and Valid\ndata_generator_train = TrainDataGenerator(train_df.iloc[train_idx], \n                                            train_df.iloc[train_idx], \n                                            TRAIN_BATCH_SIZE, \n                                            SHAPE,\n                                            augment = True)\ndata_generator_val = TrainDataGenerator(train_df.iloc[valid_idx], \n                                        train_df.iloc[valid_idx], \n                                        VALID_BATCH_SIZE, \n                                        SHAPE,\n                                        augment = False)\n\n# Create Model\nmodel = create_model()\n\n# Full Training Model\nfor base_layer in model.layers[:-1]:\n    base_layer.trainable = True\n    \nTRAIN_STEPS = int(len(data_generator_train) / 6)\nLR = 0.000125\n \n\nmodel.compile(optimizer = Adam(learning_rate = LR), \n              loss = 'binary_crossentropy',\n              metrics = ['acc', tf.keras.metrics.AUC()])\n\n# Train Model\nhistory = model.fit_generator(generator = data_generator_train,\n                    validation_data = data_generator_val,\n                    steps_per_epoch = TRAIN_STEPS,\n                    epochs = 7,\n                    callbacks = [ModelCheckpointFull('seresnext50_holdout_model.h5')],\n                    verbose = 1)\n    \nplt.plot(history.history['loss'], label='train')\nplt.plot(history.history['val_loss'], label='valid')\nplt.title(\"model loss\")\nplt.ylabel(\"loss\")\nplt.xlabel(\"epoch\")\nplt.legend([\"train\", \"valid\"], loc=\"upper right\")\nplt.savefig('seresnext50_loss_performance.png')\nplt.clf()\nplt.plot(history.history['acc'], label='train')\nplt.plot(history.history['val_acc'], label='valid')\nplt.title(\"model accuracy\")\nplt.ylabel(\"accuracy\")\nplt.xlabel(\"epoch\")\nplt.legend([\"train\", \"valid\"], loc=\"lower right\")\nplt.savefig('seresnext50_acc_performance.png')\nplt.clf()\n\npreds = predictions(test_df, model)\nnp.save('seresnext50_holdout_saved_preds.npy', preds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally we create the submission file by averaging all submission_predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.iloc[:, :] = preds\ntest_df = test_df.stack().reset_index()\ntest_df.insert(loc = 0, column = 'ID', value = test_df['Image'].astype(str) + \"_\" + test_df['Diagnosis'])\ntest_df = test_df.drop([\"Image\", \"Diagnosis\"], axis=1)\ntest_df.to_csv('seresnext50_holdout_submission.csv', index = False)\nprint(test_df.head(12))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!kaggle competitions submit -c rsna-intracranial-hemorrhage-detection -f seresnext50_holdout_submission.csv -m \"holdout\"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}