{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":37333,"databundleVersionId":3949526,"sourceType":"competition"},{"sourceId":3915695,"sourceType":"datasetVersion","datasetId":2325530},{"sourceId":3919495,"sourceType":"datasetVersion","datasetId":2327615}],"dockerImageVersionId":30203,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n

Background vs. Clot classifier

\n

Work in progress βš οΈπŸ‘·πŸΌ

\n\nCheckout the [dataset](https://www.kaggle.com/datasets/alejopaullier/strip-ai-background-clot) that comes along with this notebook.\n\nWelcome to this competition! πŸ‘‹πŸ‘‹πŸ‘‹\n\nThe goal of this competition is to classify the blood clot origins in ischemic stroke. Using whole slide digital pathology images, you'll build a model that differentiates between the two major acute ischemic stroke (AIS) etiology subtypes: cardiac and large artery atherosclerosis.\n\nImage classification is one of the simplest tasks on Computer Vision with tonnes of examples and models available on Kaggle. However, in this competition you will have to **deal with very large images!** Most CNN architectures handle much lower resolutions than the images provided.\n\nFor example, [EfficientNet's resolutions](https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/) go from 224 to 600, while our images sizes on this competition's dataset are way higher than these, being some images **tens of thousands of pixels in height and width!**\n\n### How can we tackle this problem? πŸ€”\n\nOne way to tackle this problem is to **divide these enormous images into smaller crops**, like 1024x1024 and individually classify them. This has already been done by @robikscube for you [here](https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/335755)!\n\nBut our problems do not end here ☹️. Most of our crops consist of background! This is a huge problem in two ways:\n1. First, background introduces noise and adds little to no information.\n2. Second and most important, training with background images is very computationally expensive, leading to worse results with larger training times.\n\n### How can we get rid of backgrounds?\n\nFortunately, we can simply deal with backgrounds by training a binary classifier that distinguishes backgrounds from blood clots 🩸. But to do so we need labeled data😒! Luckily for you, I have manually labeled 20000 images from Rob's crops, 10000 backgrounds and 10000 clots, so our classifier can be trained with this balanced dataset! πŸŽ‰πŸŽŠ\n\n### What should we do after we classified backgrounds and clots?\n\nThe binary CNN classifier trained on this notebook will end up being used in a **pre-processing stage prior to classifing our final images.**\n\nOnce we get a distilled dataset consisted of only clots we can use it to train our final classification model! \n\n### Inspiration\n\nThis notebook's train function and model are a modified version of @andradaolteanu [excellent notebook](https://www.kaggle.com/code/andradaolteanu/melanoma-competiton-aug-resnet-effnet-lb-0-91).","metadata":{}},{"cell_type":"markdown","source":"### Import libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport torch\n\n# System\nimport datetime\nimport gc\nimport numpy as np\nimport os, os.path\nimport time\nfrom tqdm import tqdm\n\n# Sklearn\nfrom sklearn.metrics import accuracy_score, auc, roc_auc_score\n\n# PyTorch\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import DataLoader\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\n\n!mkdir logs\n!mkdir saved_models","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:56:41.943746Z","iopub.execute_input":"2022-07-09T15:56:41.944105Z","iopub.status.idle":"2022-07-09T15:56:45.616247Z","shell.execute_reply.started":"2022-07-09T15:56:41.94407Z","shell.execute_reply":"2022-07-09T15:56:45.614977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Install EfficientNet Model","metadata":{}},{"cell_type":"code","source":"!pip install efficientnet_pytorch","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:56:45.618795Z","iopub.execute_input":"2022-07-09T15:56:45.619658Z","iopub.status.idle":"2022-07-09T15:56:57.959859Z","shell.execute_reply.started":"2022-07-09T15:56:45.619617Z","shell.execute_reply":"2022-07-09T15:56:57.95875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Settings\n\nThe next cell provides general settings, paths to data, model hyperparameters, etc.","metadata":{}},{"cell_type":"code","source":"class config:\n BATCH_SIZE_TRAIN = 32\n BATCH_SIZE_VALIDATION = 16\n BATCH_SIZE_TEST = 32\n EPOCHS = 1\n FOLDS = 5\n LEARNING_RATE = 0.0005\n LR_FACTOR = 0.4 # BY HOW MUCH THE LR IS DECREASING\n LR_PATIENCE = 1 # 1 MODEL NOT IMPROVING UNTIL LR IS DECREASING\n NUM_WORKERS = 1\n OUTPUT_SIZE = 1\n PATIENCE = 3\n TTA = 3\n WEIGHT_DECAY = 0.0\n\n \nclass paths:\n TRAIN_CSV = \"../input/strip-ai-background-clot/metadata.csv\"\n TRAIN_JPG_FOLDER = \"../input/mayo-clinic-1024-jpg-part1/train\"\n \n \nclass model_hp:\n EFFNET_B4_HIDDEN_LAYER_SIZE = 1792\n EFFNET_B2_HIDDEN_LAYER_SIZE = 1408\n EFFNET_B7_HIDDEN_LAYER_SIZE = 2560\n FFNN_HIDDEN_LAYER_SIZE = 250\n DROPOUT = 0.2","metadata":{"execution":{"iopub.status.busy":"2022-07-09T16:07:54.472507Z","iopub.execute_input":"2022-07-09T16:07:54.473369Z","iopub.status.idle":"2022-07-09T16:07:54.481662Z","shell.execute_reply.started":"2022-07-09T16:07:54.473326Z","shell.execute_reply":"2022-07-09T16:07:54.479917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Check device","metadata":{}},{"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint('Device available now: ', device)","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:56:57.974333Z","iopub.execute_input":"2022-07-09T15:56:57.974586Z","iopub.status.idle":"2022-07-09T15:56:58.046729Z","shell.execute_reply.started":"2022-07-09T15:56:57.974564Z","shell.execute_reply":"2022-07-09T15:56:58.045499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load dataframe","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(paths.TRAIN_CSV, sep = ',')\ndf.reset_index(drop = True, inplace=True)\nprint(f\"Dataframe shape is: {df.shape}\")\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:56:59.452309Z","iopub.execute_input":"2022-07-09T15:56:59.452654Z","iopub.status.idle":"2022-07-09T15:56:59.613184Z","shell.execute_reply.started":"2022-07-09T15:56:59.452626Z","shell.execute_reply":"2022-07-09T15:56:59.611845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's add a column which contains the path to the image","metadata":{}},{"cell_type":"code","source":"def add_image_path(slice_id, folder_path):\n image_path = folder_path + \"/\" + slice_id + \".jpg\"\n return image_path\n\ndf[\"image_path\"] = df[\"slice_id\"].apply(lambda x: add_image_path(x, paths.TRAIN_JPG_FOLDER))\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:57:00.257435Z","iopub.execute_input":"2022-07-09T15:57:00.257922Z","iopub.status.idle":"2022-07-09T15:57:00.317876Z","shell.execute_reply.started":"2022-07-09T15:57:00.257863Z","shell.execute_reply":"2022-07-09T15:57:00.316986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### How many background and clot images are labeled?","metadata":{}},{"cell_type":"code","source":"print(f\"There are {len(df[df['class']==1])} clot images\")\nprint(f\"There are {len(df[df['class']==0])} background images\")\nprint(f\"There are {len(df[df['class']==-1])} unlabeled images\")","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:57:01.119548Z","iopub.execute_input":"2022-07-09T15:57:01.119889Z","iopub.status.idle":"2022-07-09T15:57:01.141717Z","shell.execute_reply.started":"2022-07-09T15:57:01.119861Z","shell.execute_reply":"2022-07-09T15:57:01.140756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's get rid of the unlabeled images","metadata":{}},{"cell_type":"code","source":"df = df[df[\"class\"]!=-1]\ndf.reset_index(drop = True, inplace=True)\nprint(f\"Dataframe shape is: {df.shape}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:57:02.017303Z","iopub.execute_input":"2022-07-09T15:57:02.018305Z","iopub.status.idle":"2022-07-09T15:57:02.029417Z","shell.execute_reply.started":"2022-07-09T15:57:02.018257Z","shell.execute_reply":"2022-07-09T15:57:02.028274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Validation\n\nFirst we will split our dataset into a `train` and a `test` dataframes (90/10 split).\n\nThen we will split our `train` dataset into five folds, where we will get our final `train` train (80%) dataframe and validation (20%) dataframe. Our cross-validator is a [StratifiedKFold()](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html) validator, this means that our classes will be roughly equally represented in each split. Anyways, our dataset is perfectly balanced so you could theoretically use uniform sampling and achieve the same results.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain, df_test = train_test_split(df, test_size=0.1)\ntrain.reset_index(drop = True, inplace=True)\ndf_test.reset_index(drop = True, inplace=True)\npredictions = torch.zeros(size = (len(df_test), 1), dtype=torch.float32, device=device)\nprint(f\"Train dataframe shape is: {train.shape}\")\nprint(f\"Test dataframe shape is: {df_test.shape}\")","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:57:03.580433Z","iopub.execute_input":"2022-07-09T15:57:03.5811Z","iopub.status.idle":"2022-07-09T15:57:06.44455Z","shell.execute_reply.started":"2022-07-09T15:57:03.581042Z","shell.execute_reply":"2022-07-09T15:57:06.443522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold\n\n\nskf = StratifiedKFold(n_splits=config.FOLDS,\n shuffle=True,\n random_state=27)\nX = train.loc[:, df.columns != \"class\"]\ny = train.loc[:, df.columns == \"class\"]","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:57:08.41393Z","iopub.execute_input":"2022-07-09T15:57:08.414608Z","iopub.status.idle":"2022-07-09T15:57:08.423857Z","shell.execute_reply.started":"2022-07-09T15:57:08.414569Z","shell.execute_reply":"2022-07-09T15:57:08.422881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's check everything is correct for one split","metadata":{}},{"cell_type":"code","source":"for fold, (train_index, valid_index) in enumerate(skf.split(X, y)): \n df_train = train.loc[train_index,:]\n df_valid = train.loc[valid_index,:]\n print(f\"There are {df_train['class'].value_counts()[0]} positive labels and {df_train['class'].value_counts()[1]} negative labels in train dataframe\")\n print(f\"There are {df_valid['class'].value_counts()[1]} positive labels and {df_valid['class'].value_counts()[1]} negative labels in validation dataframe\")\n break","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:57:21.488022Z","iopub.execute_input":"2022-07-09T15:57:21.488998Z","iopub.status.idle":"2022-07-09T15:57:21.513313Z","shell.execute_reply.started":"2022-07-09T15:57:21.488951Z","shell.execute_reply":"2022-07-09T15:57:21.512352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Splits are balanced!","metadata":{}},{"cell_type":"markdown","source":"### Dataset\n\nLet's build our dataset class. It needs to have an `__init__`, `__len__` and `__getitem__` methods.","metadata":{}},{"cell_type":"code","source":"import cv2\nimport numpy as np\n\n# Data Augmentation for Image Preprocessing\nfrom albumentations import (Normalize, VerticalFlip, HorizontalFlip, Compose,\n RandomBrightnessContrast, HueSaturationValue,\n RandomResizedCrop, ShiftScaleRotate)\nfrom albumentations.pytorch import ToTensorV2\nfrom torch.utils.data import Dataset\n\n\nclass StripAIDataset(Dataset):\n\n def __init__(self, dataframe, is_train=True, is_valid=False, is_test=False):\n\n self.dataframe = dataframe\n self.is_train = is_train\n self.is_valid = is_valid\n self.is_test = is_test\n\n # Data Augmentation (custom for each dataset type)\n # Crop dimensions are because of EfficientNet input layer. Checkout each EfficientNet variant resolution here:\n # https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/\n if is_train or is_test:\n self.transform = Compose([RandomResizedCrop(height=224,\n width=224,\n scale=(0.4, 1.0)),\n ShiftScaleRotate(rotate_limit=90,\n scale_limit=[0.8, 1.2]),\n HorizontalFlip(p=0.5),\n VerticalFlip(p=0.5),\n HueSaturationValue(sat_shift_limit=[0.7, 1.3],\n hue_shift_limit=[-0.1, 0.1]),\n RandomBrightnessContrast(brightness_limit=[0.7, 1.3],\n contrast_limit=[0.7, 1.3]),\n Normalize(),\n ToTensorV2()])\n else:\n self.transform = Compose([RandomResizedCrop(height=224,\n width=224,\n scale=(0.4, 1.0)),\n Normalize(),\n ToTensorV2()])\n\n def __len__(self):\n return len(self.dataframe)\n\n def __getitem__(self, index):\n # Select path and read image\n image_path = self.dataframe['image_path'][index]\n image = cv2.imread(image_path)\n # Apply transforms\n image = self.transform(image=image)\n # Extract image from dictionary\n image = image['image']\n\n # If train/valid: image + class | If test: only image\n if self.is_train or self.is_valid:\n return image, self.dataframe['class'][index]\n else:\n return image\n","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:57:34.052424Z","iopub.execute_input":"2022-07-09T15:57:34.052775Z","iopub.status.idle":"2022-07-09T15:57:35.279533Z","shell.execute_reply.started":"2022-07-09T15:57:34.052745Z","shell.execute_reply":"2022-07-09T15:57:35.2784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create Model","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\n\nfrom efficientnet_pytorch import EfficientNet\n\nclass EfficientNetwork(nn.Module):\n def __init__(self, output_size, b4=False, b2=False):\n super().__init__()\n\n self.b4 = b4\n self.b2 = b2\n\n # Define Feature part (IMAGE)\n if b4:\n self.features = EfficientNet.from_pretrained('efficientnet-b4')\n elif b2:\n self.features = EfficientNet.from_pretrained('efficientnet-b2')\n else:\n self.features = EfficientNet.from_pretrained('efficientnet-b7')\n\n # Define Classification part\n if b4:\n self.classification = nn.Sequential(nn.Linear(model_hp.EFFNET_B4_HIDDEN_LAYER_SIZE,\n model_hp.FFNN_HIDDEN_LAYER_SIZE),\n nn.Linear(model_hp.FFNN_HIDDEN_LAYER_SIZE,\n output_size))\n elif b2:\n self.classification = nn.Sequential(nn.Linear(model_hp.EFFNET_B2_HIDDEN_LAYER_SIZE,\n model_hp.FFNN_HIDDEN_LAYER_SIZE),\n nn.Linear(model_hp.FFNN_HIDDEN_LAYER_SIZE,\n output_size))\n else:\n self.classification = nn.Sequential(nn.Linear(model_hp.EFFNET_B7_HIDDEN_LAYER_SIZE,\n model_hp.FFNN_HIDDEN_LAYER_SIZE),\n nn.Linear(model_hp.FFNN_HIDDEN_LAYER_SIZE,\n output_size))\n\n def forward(self, image, verbose=False):\n\n if verbose: print('Input Image shape:', image.shape)\n\n # IMAGE CNN\n image = self.features.extract_features(image)\n if verbose: print('Features Image shape:', image.shape)\n\n if self.b4:\n image = F.avg_pool2d(image, image.size()[2:]).reshape(-1, model_hp.EFFNET_B4_HIDDEN_LAYER_SIZE)\n elif self.b2:\n image = F.avg_pool2d(image, image.size()[2:]).reshape(-1, model_hp.EFFNET_B2_HIDDEN_LAYER_SIZE)\n else:\n image = F.avg_pool2d(image, image.size()[2:]).reshape(-1, model_hp.EFFNET_B7_HIDDEN_LAYER_SIZE)\n if verbose: print('Image Reshaped shape:', image.shape)\n\n # CLASSIF\n out = self.classification(image)\n if verbose: print('Out shape:', out.shape)\n\n return out\n \n \nmodel = EfficientNetwork(output_size=config.OUTPUT_SIZE,\n b4=False, b2=True).to(device)","metadata":{"execution":{"iopub.status.busy":"2022-07-09T15:57:35.281247Z","iopub.execute_input":"2022-07-09T15:57:35.281808Z","iopub.status.idle":"2022-07-09T15:57:37.206184Z","shell.execute_reply.started":"2022-07-09T15:57:35.281778Z","shell.execute_reply":"2022-07-09T15:57:37.205084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train function","metadata":{}},{"cell_type":"code","source":"def train_function(predictions, train_df, test_df, model, StripAIDataset, device, version='v1'):\n \"\"\"\n This function iterates over folds. On each fold, the original train dataset is split into a new train subset and a\n validation dataset. Test dataset is static through fold iterations. For each fold it trains the model for the\n specified epochs. Hence, training as well as evaluation on validation dataset is performed at an epoch level.\n The amount of iterations is therefore FOLDS*EPOCHS. After training is complete, the model is evaluated on the\n validation dataset and the model artifacts are saved if the metric is improved. Finally, at the last of every fold\n iteration the evaluation metric is computed for the test dataset. Please note that model selection is performed\n with respect to the validation metric and that no retraining on the original train dataset is performed.\n\n :param predictions: predictions for the test set.\n :param model: model architecture. \n :param StripAIDataset: a custom dataset for this problem. \n :param version: model version. Each time we train a new model we must create a new version.\n :return oof: Out of Fold predictions. In each fold we predict the Validation set, in consequence, as validation\n sets are non overlapping we end up with predictions for the whole train set.\n :return predictions: predictions for the test set\n \"\"\"\n # Creates a .txt file that will contain the logs\n f = open(f\"logs/logs_{version}.txt\", \"w+\")\n\n # Out of Fold Predictions\n oof = np.zeros(shape=(len(train_df), 1))\n print(f\"OOF shape: {oof.shape}\")\n\n # Iterate over folds\n for fold, (train_index, valid_index) in enumerate(skf.split(X, y)):\n # Append to .txt\n with open(f\"logs/logs_{version}.txt\", 'a+') as f:\n print('-' * 10, 'Fold:', fold + 1, '-' * 10, file=f)\n print('-' * 10, 'Fold:', fold + 1, '-' * 10)\n\n # --- Create Instances ---\n # Best ROC score in this fold\n best_roc = None\n # Reset patience before every fold.\n patience_f = config.PATIENCE\n\n # Initiate the model\n model = model\n\n # Create optimizer.\n optimizer = torch.optim.Adam(model.parameters(),\n lr=config.LEARNING_RATE,\n weight_decay=config.WEIGHT_DECAY)\n\n # Create scheduler.\n scheduler = ReduceLROnPlateau(optimizer=optimizer,\n mode='max',\n patience=config.LR_PATIENCE,\n verbose=True,\n factor=config.LR_FACTOR)\n\n # Create Loss. \n criterion = nn.BCEWithLogitsLoss()\n\n # --- Read in Data ---\n train_data = train_df.iloc[train_index].reset_index(drop=True)\n valid_data = train_df.iloc[valid_index].reset_index(drop=True)\n\n # Create Data instances. \n train = StripAIDataset(train_data,\n is_train=True, is_valid=False, is_test=False)\n valid = StripAIDataset(valid_data,\n is_train=False, is_valid=True, is_test=False)\n # Read in test data | Remember! We're using data augmentation like we use for Train data.\n test = StripAIDataset(test_df,\n is_train=False, is_valid=False, is_test=True)\n\n # Create Dataloaders\n train_loader = DataLoader(train,\n batch_size=config.BATCH_SIZE_TRAIN,\n shuffle=True,\n num_workers=config.NUM_WORKERS)\n # shuffle=False! Otherwise function won't work!!!\n valid_loader = DataLoader(valid,\n batch_size=config.BATCH_SIZE_VALIDATION,\n shuffle=False,\n num_workers=config.NUM_WORKERS)\n test_loader = DataLoader(test,\n batch_size=config.BATCH_SIZE_TEST,\n shuffle=False,\n num_workers=config.NUM_WORKERS)\n\n # === EPOCHS ===\n epochs = config.EPOCHS\n for epoch in range(epochs):\n start_time = time.time()\n correct = 0\n train_losses = 0\n\n # === TRAIN ===\n # Sets the module in training mode.\n model.train()\n\n # === Iterate over batches ===\n with tqdm(train_loader, unit=\"train_batch\") as tqdm_train_loader:\n for images, labels in tqdm_train_loader:\n # Save them to device\n images = torch.tensor(images, device=device, dtype=torch.float32)\n labels = torch.tensor(labels, device=device, dtype=torch.float32)\n\n # Clear gradients first; very important, usually done BEFORE prediction\n optimizer.zero_grad()\n\n # Log Probabilities & Backpropagation\n out = model(images, verbose=False)\n loss = criterion(out, labels.unsqueeze(1))\n loss.backward()\n optimizer.step()\n\n # --- Save information after this batch ---\n # Save loss\n train_losses += loss.item()\n # From log probabilities to actual probabilities\n train_preds = torch.round(torch.sigmoid(out)) # 0 and 1\n # Number of correct predictions\n correct += (train_preds.cpu() == labels.cpu().unsqueeze(1)).sum().item()\n # Compute Train Accuracy\n train_acc = correct / len(train_index)\n\n # === EVAL ===\n # Sets the model in evaluation mode\n model.eval()\n\n # Create matrix to store evaluation predictions (for accuracy)\n valid_preds = torch.zeros(size=(len(valid_index), 1), device=device, dtype=torch.float32)\n\n # Disables gradients (we need to be sure no optimization happens)\n with torch.no_grad():\n for k, (images, labels) in enumerate(tqdm(valid_loader, unit=\"valid_batch\")):\n images = torch.tensor(images, device=device, dtype=torch.float32)\n out = model(images)\n pred = torch.sigmoid(out)\n valid_preds[k * images.shape[0]: k * images.shape[0] + images.shape[0]] = pred\n\n # Compute accuracy\n valid_acc = accuracy_score(valid_data['class'].values,\n torch.round(valid_preds.cpu()))\n # Compute ROC\n valid_roc = roc_auc_score(valid_data['class'].values,\n valid_preds.cpu())\n\n # Compute time on Train + Eval\n duration = str(datetime.timedelta(seconds=time.time() - start_time))[:7]\n\n # PRINT INFO\n # Append to .txt file\n with open(f\"logs/logs_{version}.txt\", 'a+') as f:\n print('{} | Epoch: {}/{} | Loss: {:.4} | Train Acc: {:.3} | Valid Acc: {:.3} | ROC: {:.3}'. \\\n format(duration, epoch + 1, epochs, train_losses, train_acc, valid_acc, valid_roc), file=f)\n # Print to console\n print('{} | Epoch: {}/{} | Loss: {:.4} | Train Acc: {:.3} | Valid Acc: {:.3} | ROC: {:.3}'. \\\n format(duration, epoch + 1, epochs, train_losses, train_acc, valid_acc, valid_roc))\n\n # === SAVE MODEL ===\n\n # Update scheduler (for learning_rate)\n scheduler.step(valid_roc)\n\n # Update best_roc\n if not best_roc: # If best_roc = None\n best_roc = valid_roc\n torch.save(model.state_dict(),\n f\"saved_models/Fold{fold + 1}_Epoch{epoch + 1}_ValidAcc_{valid_acc:.3f}_ROC_{valid_roc:.3f}.pth\")\n continue\n\n if valid_roc > best_roc:\n best_roc = valid_roc\n # Reset patience (because we have improvement)\n patience_f = config.PATIENCE\n torch.save(model.state_dict(),\n f\"saved_models/Fold{fold + 1}_Epoch{epoch + 1}_ValidAcc_{valid_acc:.3f}_ROC_{valid_roc:.3f}.pth\")\n else:\n # Decrease patience (no improvement in ROC)\n patience_f = patience_f - 1\n if patience_f == 0:\n with open(f\"logs/logs_{version}.txt\", 'a+') as f:\n print('Early stopping (no improvement since 3 models) | Best ROC: {}'. \\\n format(best_roc), file=f)\n print('Early stopping (no improvement since 3 models) | Best ROC: {}'. \\\n format(best_roc))\n break\n\n # === INFERENCE ===\n # Choose model with best_roc in this fold\n best_model_path = 'saved_models/' + [file for file in os.listdir('saved_models') if\n str(round(best_roc, 3)) in file and 'Fold' + str(fold + 1) in file][0]\n # Using best model from Epoch Train\n model = EfficientNetwork(output_size=config.OUTPUT_SIZE,b4=False, b2=True).to(device)\n model.load_state_dict(torch.load(best_model_path))\n # Set the model in evaluation mode\n model.eval()\n\n with torch.no_grad():\n # --- EVAL ---\n # Predicting again on Validation data to get preds for OOF\n valid_preds = torch.zeros(size=(len(valid_index), 1), device=device, dtype=torch.float32)\n\n for k, (images, _) in enumerate(tqdm(valid_loader, unit=\"oof_batch\")):\n images = torch.tensor(images, device=device, dtype=torch.float32)\n out = model(images)\n pred = torch.sigmoid(out)\n valid_preds[k * images.shape[0]: k * images.shape[0] + images.shape[0]] = pred\n\n # Save info to OOF\n oof[valid_index] = valid_preds.cpu().numpy()\n \n # --- TEST ---\n # Now (Finally) prediction for our TEST data\n for i in range(config.TTA):\n for k, images in enumerate(tqdm(test_loader, unit=f\"test_loader_TTA_{i}\")):\n images = torch.tensor(images, device=device, dtype=torch.float32)\n out = model(images)\n # Convert to probablities\n out = torch.sigmoid(out)\n # ADDS! the prediction to the matrix we already created\n predictions[k * images.shape[0]: k * images.shape[0] + images.shape[0]] += out\n\n # Divide Predictions by TTA (to average the results during TTA)\n predictions /= config.TTA\n\n # === CLEANING ===\n # Clear memory\n del train, valid, train_loader, valid_loader, images, labels\n # Garbage collector\n gc.collect()\n\n return oof, predictions","metadata":{"execution":{"iopub.status.busy":"2022-07-09T16:16:57.798803Z","iopub.execute_input":"2022-07-09T16:16:57.799185Z","iopub.status.idle":"2022-07-09T16:16:57.838635Z","shell.execute_reply.started":"2022-07-09T16:16:57.799152Z","shell.execute_reply":"2022-07-09T16:16:57.837503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train","metadata":{}},{"cell_type":"code","source":"# Train\noof, predictions = train_function(predictions,\n train,\n df_test,\n model,\n StripAIDataset,\n device)","metadata":{"execution":{"iopub.status.busy":"2022-07-09T16:16:59.379368Z","iopub.execute_input":"2022-07-09T16:16:59.379934Z","iopub.status.idle":"2022-07-09T16:55:43.561404Z","shell.execute_reply.started":"2022-07-09T16:16:59.379897Z","shell.execute_reply":"2022-07-09T16:55:43.560328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Save OOF and Predictions","metadata":{}},{"cell_type":"code","source":"oof = pd.DataFrame(oof)\npredictions = pd.DataFrame(predictions.cpu().numpy())\noof.to_csv(\"oof.csv\", index=False)\npredictions.to_csv(\"predictions.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-09T16:57:42.597001Z","iopub.execute_input":"2022-07-09T16:57:42.598139Z","iopub.status.idle":"2022-07-09T16:57:42.657464Z","shell.execute_reply.started":"2022-07-09T16:57:42.598081Z","shell.execute_reply":"2022-07-09T16:57:42.656543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load Out of Fold","metadata":{}},{"cell_type":"code","source":"oof = pd.read_csv(\"oof.csv\", sep = ',')\noof.columns = [\"oof\"]\nprint(f\"Out of fold dataframe shape is: {oof.shape}\")\noof.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-09T16:58:15.627517Z","iopub.execute_input":"2022-07-09T16:58:15.627878Z","iopub.status.idle":"2022-07-09T16:58:15.649023Z","shell.execute_reply.started":"2022-07-09T16:58:15.627845Z","shell.execute_reply":"2022-07-09T16:58:15.648039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plot Out of Fold predictions","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.hist(oof[\"oof\"])\nplt.xlabel(\"Model prediction\")\nplt.ylabel(\"Frequency\")\nplt.title(\"Model predictions histogram\")","metadata":{"execution":{"iopub.status.busy":"2022-07-09T16:58:18.9801Z","iopub.execute_input":"2022-07-09T16:58:18.981162Z","iopub.status.idle":"2022-07-09T16:58:19.224573Z","shell.execute_reply.started":"2022-07-09T16:58:18.981108Z","shell.execute_reply":"2022-07-09T16:58:19.223592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Confusion matrix","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nfrom sklearn.metrics import confusion_matrix\n\n# Make OOF Binary\noof.loc[oof.oof >= 0.5, 'oof'] = 1\noof.loc[oof.oof < 0.5, 'oof'] = 0\n\n# Create Confusion Matrix\ncf_matrix = confusion_matrix(train['class'], oof)\n\n# Pretty CM:\ngroup_names = ['True Neg','False Pos','False Neg','True Pos']\n# Format of the absolute numbers\ngroup_counts = ['{:,}'.format(value) for value in cf_matrix.flatten()]\n# Format for relative numbers\ngroup_percentages = ['{0:.1%}'.format(value) for value in cf_matrix.flatten()/np.sum(cf_matrix)]\n\nlabels = [f'{v1}\\n{v2}\\n{v3}' for v1, v2, v3 in zip(group_names,group_counts,group_percentages)]\nlabels = np.asarray(labels).reshape(2,2)\n\n# --- The figure ---\nplt.figure(figsize=(16, 5))\nsns.heatmap(cf_matrix, annot=labels, fmt='', cmap='Oranges',xticklabels=['background', 'clot'], \n yticklabels=['background', 'clot'], cbar=False)\nplt.hist(oof[\"oof\"])","metadata":{"execution":{"iopub.status.busy":"2022-07-09T16:58:45.765184Z","iopub.execute_input":"2022-07-09T16:58:45.765822Z","iopub.status.idle":"2022-07-09T16:58:46.010942Z","shell.execute_reply.started":"2022-07-09T16:58:45.765784Z","shell.execute_reply":"2022-07-09T16:58:46.009953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### What to do next?\n\nOur classifier has achieve a ROC of 0.99 so its already a very good model. However, to achieve even better results, you can:\n- Train more epochs.\n- Train more folds.\n- Tune hyperparameters.\n- Try different architectures!\n- Try Pseudolabelling","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}