{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this Notebook\n\n* This kernel is for kagglers like me who believe in trying a lot of things and don't fear of failing as failing is the part of learning. \n\n* Starting today, i am going to log my progress in this kernel, this kernel will be work in progress and will contain hint about a lot of things which i tried or will try in upcoming days.\n\n* This is an inference kernel where CV and LB score of various models and various image related tricks will be tracked, so that it will be easier to keep track of things which worked and which didn't work for this competition. By default i will not share any training code and keep models private, if you want the training kernel or models to be public feel free to mention in the comments section.\n\n\n**<span style=\"color:Red\">Please upvote this kernel if you like it . It motivates me to produce more quality content :)**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Versions History\n\n* [version3](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=32756285): \n    * `model_tried`: se_resnext50_32x4d\n    * `folds`: 3 folds ensemble\n    * `image`: full image resized (224x224)\n    * `LB`: 0.66\n\n\n* [version4](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=36118508):\n    * `model_tried`: se_resnext50_32x4d\n    * `folds`: submitted on fold 0\n    * `image`: tiles of size (16x256x256)\n    * `CV`: 0.7177\n    * `LB`: 0.67\n\n\n* [version5](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=36122740):\n    * `model_tried`: se_resnext50_32x4d\n    * `folds`: 5 folds ensemble\n    * `image`: tiles of size (16x256x256)\n    * `CV`: [ fold0: 0.717, fold1: 0.674, fold2: 0.692, fold3: 0.704, fold4: 0.695  ]\n    * `LB`: 0.70\n\n* [version6](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=36131890): \n    * `model_tried`: efficientnet-B3\n    * `folds`: 2 folds ensemble (fold 0 and fold 1)\n    * `image`: tiles of size (16x256x256)\n    * `CV`: [ fold0: 0.719, fold1: 0.726 ]\n    * `LB`: 0.74   \n\n* [version7](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=36150737):\n    * `model_tried`: efficientnet-B3\n    * `folds`: 5 folds ensemble\n    * `image`: tiles of size (16x256x256)\n    * `CV`: [ fold0: 0.719, fold1: 0.726, fold2: 0.729, fold3: 0.719, fold4: 0.732 ]\n    * `LB`: 0.73\n\n\n* [version8](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=36230754): \n    * `model_tried`: efficientnet-B3\n    * `folds`: fold 0\n    * `image`: tiles of size (16x256x256) [image resized to 512x512]\n    * `CV`: [ fold0: 0.749]\n    * `LB`: 0.81\n\n* [version9](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=36242789): \n    * `model_tried`: efficientnet-B3\n    * `folds`: ensemble of fold 0 and fold 1\n    * `image`: tiles of size (16x256x256) [image resized to 512x512]\n    * `CV`: [ fold0: 0.749, fold1: 0.7509]\n    * `LB`: 0.81\n\n* [version10](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=36282650): \n    * `model_tried`: efficientnet-B3\n    * `folds`: ensemble of all 5 folds\n    * `image`: tiles of size (16x256x256) [image resized to 512x512]\n    * `CV`: [ fold0: 0.749, fold1: 0.7509, fold2: 0.7592, fold3: 0.7362, fold4: 0.7676]\n    * `LB`: 0.84\n\n* [version11](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=36741869): \n    * `model_tried`: efficientnet-B3\n    * `folds`: ensemble of all folds [0,1,2,4]\n    * `image`: tiles of size (16x256x256) [image resized to 512x512]\n    * `CV`: [ fold0: 0.749, fold1: 0.7509, fold2: 0.7592,fold4: 0.7676]\n    * `LB`: 0.85\n    \n* [version12](https://www.kaggle.com/rohitsingh9990/panda-inference-ensemble-trying-various-models?scriptVersionId=37080102): \n    * `model_tried`: efficientnet-B0\n    * `folds`: fold1\n    * `image`: tiles of size (9x256x256) [image resized to 512x512]\n    * `CV`: [fold1: 0.7751]\n    * `LB`: 0.69\n\n* version13: \n    * `model_tried`: efficientnet-B3\n    * `folds`: fold1\n    * `image`: tiles of size (16x256x256) [image resized to 768x768]\n    * `CV`: [fold1: 0.7549]\n    * `LB`: ??\n\n## Best LB score (before current version)\n\n* Best LB score is from version11 which is 0.85\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport sys\nsys.path = [\n    '../input/efficientnet-pytorch/EfficientNet-PyTorch/EfficientNet-PyTorch-master',\n] + sys.path","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport torch.nn.functional as F\nimport os\n\n# Any results you write to the current directory are saved as output.\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset,DataLoader\nfrom torchvision import transforms,models\nfrom tqdm import tqdm_notebook as tqdm\nimport math\nimport torch.utils.model_zoo as model_zoo\nimport cv2\nfrom sklearn.metrics import cohen_kappa_score\n\nimport openslide\nimport skimage.io\nimport random\nfrom sklearn.metrics import cohen_kappa_score\nimport albumentations\n# General packages\n\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* For EDA and visualizations, Please visit https://www.kaggle.com/rohitsingh9990/panda-eda-better-visualization/comments\n\n* For Simple inference using Resnext50 please visit https://www.kaggle.com/rohitsingh9990/panda-resnext-inference","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#https://www.kaggle.com/c/tweet-sentiment-extraction/discussion/158726","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%bash\napt-get install -y -q xarchiver || true\n# SAVED_MODEL_PATH=\"../input/eff-b0-16-768/efficientnetB0_0.pth\"\n\nSAVED_MODEL_PATH=\"../input/eff-b0-16-768\"\nNEW_MODEL_PATH=\"/kaggle/working\"\nfor model_file in $(ls $SAVED_MODEL_PATH/*.pth)\ndo\n    just_filename=$(basename \"${model_file%.*}\")\n    if [[ ! -e \"$NEW_MODEL_PATH/$just_filename.pth\" ]]; then\n        echo \"Copying $model_file to $NEW_MODEL_PATH\"\n        cp $model_file /tmp\n        cd /tmp/\n        mkdir -p $just_filename\n        echo 3 > $just_filename/version\n        echo \"\"\n        zip -u  $just_filename.pth $just_filename/version\n        echo \"Moving model file $just_filename.pth to $NEW_MODEL_PATH\"\n        mv $just_filename.pth $NEW_MODEL_PATH/$just_filename.pth\n    fi\n    echo \"(After) Contents of '$just_filename/version' in the model file '$model_file'\"\n    unzip -p $NEW_MODEL_PATH/$(basename $model_file) $just_filename/version\ndone","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Config","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class config:\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    IMG_WIDTH = 512\n    IMG_HEIGHT = 512\n    TEST_BATCH_SIZE = 16\n    CLASSES = 6\n    # In order to check weather your submission will work or not on test data simply set DEBUG = True\n    DEBUG = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Loading Data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"BASE_PATH = '../input/prostate-cancer-grade-assessment'\n\ndata_dir = f'{BASE_PATH}/test_images'\ntest = pd.read_csv(f'{BASE_PATH}/test.csv')\nsubmission = pd.read_csv(f'{BASE_PATH}/sample_submission.csv')\n\nif config.DEBUG:\n    data_dir = f'{BASE_PATH}/train_images'\n    test = pd.read_csv(f'{BASE_PATH}/train.csv').head(200)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_torch(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nseed_torch(seed=42)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## EfficientNet-B3 Model","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from efficientnet_pytorch import EfficientNet ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class EfficientNetB3(nn.Module):\n    def __init__(self, pretrained):\n        super(EfficientNetB3, self).__init__()\n        if pretrained == True:\n            self.model = EfficientNet.from_name('efficientnet-b3')\n            self.model.load_state_dict(torch.load('../input/efficientnet-pytorch/efficientnet-b3-c8376fa2.pth'))\n        else:\n            self.model = EfficientNet.from_pretrained(None)            \n\n        in_features = self.model._fc.in_features\n        self.l0 = nn.Linear(in_features, config.CLASSES)\n\n    def forward(self, x):\n        bs, _, _, _ = x.shape\n        x = self.model.extract_features(x)\n        x = F.adaptive_avg_pool2d(x, 1).reshape(bs, -1)\n        l0 = self.l0(x)\n        return l0\n    \nclass EfficientNetB0(nn.Module):\n    def __init__(self, pretrained):\n        super(EfficientNetB0, self).__init__()\n        if pretrained == True:            \n            self.model = EfficientNet.from_name('efficientnet-b0')\n            self.model.load_state_dict(torch.load('../input/efficientnet-pytorch/efficientnet-b0-08094119.pth'))\n        else:\n            self.model = EfficientNet.from_pretrained(None)            \n\n        in_features = self.model._fc.in_features\n        self.l0 = nn.Linear(in_features, config.CLASSES)\n\n    def forward(self, x):\n        bs, _, _, _ = x.shape\n        x = self.model.extract_features(x)\n        x = F.adaptive_avg_pool2d(x, 1).reshape(bs, -1)\n        l0 = self.l0(x)\n        return l0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_tiles(img, mode=0):\n    result = []\n    h, w, c = img.shape\n    pad_h = (tile_size - h % tile_size) % tile_size + ((tile_size * mode) // 2)\n    pad_w = (tile_size - w % tile_size) % tile_size + ((tile_size * mode) // 2)\n\n    img2 = np.pad(img,[[pad_h // 2, pad_h - pad_h // 2], [pad_w // 2,pad_w - pad_w//2], [0,0]], constant_values=255)\n    img3 = img2.reshape(\n        img2.shape[0] // tile_size,\n        tile_size,\n        img2.shape[1] // tile_size,\n        tile_size,\n        3\n    )\n\n    img3 = img3.transpose(0,2,1,3,4).reshape(-1, tile_size, tile_size,3)\n    n_tiles_with_info = (img3.reshape(img3.shape[0],-1).sum(1) < tile_size ** 2 * 3 * 255).sum()\n    if len(img3) < n_tiles:\n        img3 = np.pad(img3,[[0,n_tiles-len(img3)],[0,0],[0,0],[0,0]], constant_values=255)\n    idxs = np.argsort(img3.reshape(img3.shape[0],-1).sum(-1))[:n_tiles]\n    img3 = img3[idxs]\n    for i in range(len(img3)):\n        result.append({'img':img3[i], 'idx':i})\n    return result, n_tiles_with_info >= n_tiles","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def write_image(path, img):\n    img = cv2.convertScaleAbs(img, alpha=(255.0))\n    cv2.imwrite(path, cv2.cvtColor(img, cv2.COLOR_RGB2BGR))  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tile_size = 256\nimage_size = 256\nn_tiles = 16\nbatch_size = 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class PANDADataset(Dataset):\n    def __init__(self,\n            df,\n            image_size,\n            n_tiles=n_tiles,\n            tile_mode=0,\n            rand=False,\n        ):\n\n        self.df = df.reset_index(drop=True)\n        self.image_size = image_size\n        self.n_tiles = n_tiles\n        self.tile_mode = tile_mode\n        self.rand = rand\n        # we are in validation part\n        self.aug = albumentations.Compose([\n            albumentations.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225], always_apply=True)\n        ])\n\n    def __len__(self):\n        return self.df.shape[0]\n    \n\n    def __getitem__(self, index):\n        row = self.df.iloc[index]\n        img_id = row.image_id\n        \n        tiff_file = os.path.join(data_dir, f'{img_id}.tiff')\n        image = skimage.io.MultiImage(tiff_file)[1]\n        tiles, OK = get_tiles(image, self.tile_mode)\n\n        if self.rand:\n            idxes = np.random.choice(list(range(self.n_tiles)), self.n_tiles, replace=False)\n        else:\n            idxes = list(range(self.n_tiles))\n\n        n_row_tiles = int(np.sqrt(self.n_tiles))\n        images = np.zeros((image_size * n_row_tiles, image_size * n_row_tiles, 3))\n        for h in range(n_row_tiles):\n            for w in range(n_row_tiles):\n                i = h * n_row_tiles + w\n    \n                if len(tiles) > idxes[i]:\n                    this_img = tiles[idxes[i]]['img']\n                else:\n                    this_img = np.ones((self.image_size, self.image_size, 3)).astype(np.uint8) * 255\n                this_img = 255 - this_img\n                h1 = h * image_size\n                w1 = w * image_size\n                images[h1:h1+image_size, w1:w1+image_size] = this_img\n\n                \n        images = images.astype(np.float32)\n        images /= 255\n        images = images.transpose(2, 0, 1)\n        \n        \n        img = images\n        img = np.transpose(img, (1, 2, 0)) # orig image has shape(3,1024, 1024), converting to (1024, 1024, 3)\n        img = 1 - img\n        img = cv2.resize(img, (768, 768))\n        write_image(f'{img_id}.png', img)\n        \n        # loading image\n        \n        img = skimage.io.MultiImage(f'{img_id}.png')[-1]\n#         img = cv2.resize(img[-1], (512, 512))\n\n        img = Image.fromarray(img).convert(\"RGB\")\n        img = self.aug(image=np.array(img))[\"image\"]\n        img = np.transpose(img, (2, 0, 1)).astype(np.float32)\n        \n\n        return { 'image': torch.tensor(img, dtype=torch.float) }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ENSEMBLES = [\n    {\n        'model_name': 'efficientnet-b0',\n        'model_weight': 'efficientnetB0_0.pth',\n        'ensemble_weight': 1 \n    }\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = config.device\nmodels = []\nfor ensemble in ENSEMBLES:\n    model = EfficientNetB3(pretrained=True)\n    model.load_state_dict(torch.load(ensemble['model_weight'], map_location=device))\n    model.to(device)\n    models.append(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_for_images_dir():\n    if config.DEBUG:\n        return os.path.exists('../input/prostate-cancer-grade-assessment/train_images')\n    else:\n        return os.path.exists('../input/prostate-cancer-grade-assessment/test_images')\n        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.eval()\npredictions = []\n\nif check_for_images_dir():\n  \n    test_dataset = PANDADataset(\n        df=test,\n        image_size=image_size,\n        n_tiles=n_tiles,\n        tile_mode=0\n    )\n\n\n    test_data_loader = torch.utils.data.DataLoader(\n        test_dataset,\n        batch_size=config.TEST_BATCH_SIZE,\n        shuffle=False,\n    )\n    \n    for model in models:\n        preds = []\n        for idx, d in tqdm(enumerate(test_data_loader), total=len(test_data_loader)):\n            inputs = d[\"image\"]\n            inputs = inputs.to(device)\n\n            with torch.no_grad():\n                outputs = model(inputs)\n            preds.append(outputs.to('cpu').numpy())\n                    \n        predictions.append(np.concatenate(preds))\n    predictions = np.mean(predictions, axis=0)\n    predictions = predictions.argmax(1)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save results","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if config.DEBUG:\n    def quadratic_weighted_kappa(y_hat, y):\n        return cohen_kappa_score(y_hat, y, weights='quadratic')\n\n    count = 0\n    for index, val in enumerate(test.isup_grade.values):\n        if predictions[index] == val:\n            count += 1\n\n    print(f\"Accuracy Train is {(count / test.shape[0])* 100}\")\n    print(f'Kappa Train is {quadratic_weighted_kappa(predictions, test.isup_grade.values)}')\nelse:\n    if len(predictions) > 0:\n        submission.isup_grade = predictions\n    submission.isup_grade = submission['isup_grade'].astype(int)\n    submission.to_csv('submission.csv',index=False)\n    print(submission.head())     ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## References:\n\n* https://www.kaggle.com/iafoss/panda-16x128x128-tiles","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# END NOTES\nI will keep on updating this kernel with my new findings and learning in order to help everyone who has just started in this competition.\n\n**<span style=\"color:Red\">Please upvote this kernel if you like it . It motivates me to produce more quality content :)**","execution_count":null}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}