{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Concatenate Tile Pooling\n### Inspired from Iafoss's work: https://www.kaggle.com/iafoss/panda-16x128x128-tiles\nThe biopsy images have a lot of empty space in them. We need to focus on the image portions that contain the actual biopsy for getting better results. Moreover, resizing all the images to a single resolution will also not work as the biopsy images provided are of different resolutions and resizing them to a single standard resolution will distort them and we won't be able to get important information out of the images.<br>\n\nWe are given MultiImages which contains each biopsy with three resolutions. For my models, I'll be using the lowest resolution images.\n\nIafoss has proposed a very good approach to handle this, called **Concatenate Tile Pooling**. He explains it as:<br><br>\n*'Instead of passing an entire image as an input, N tiles are selected from each image based on the number of tissue pixels and passed independently through the convolutional part. The outputs of the convolutional part is concatenated in a large single map for each image preceding pooling and FC head. Since any spatial information is eliminated by the pooling layer, the Concat Tile Pooling approach is nearly identical to passing an entire image through the convolutional part, excluding predictions for nearly empty regions, which do not contribute to the final prediction, and shuffle the remaining outputs into a square map of smaller size.'*<br>\n<br>\n\nSo, I have taken his idea and converted the Images into 49 tiles and then stacked them together into an image. We also handle the problem of distortion as all the generated images will be of 448x448x3 dimension and we won't have to rescale.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport os\nimport pandas as pd\nimport sys\n\nimport cv2\nimport matplotlib.pyplot as plt\nimport openslide\nfrom PIL import Image\nimport skimage.io\nfrom tqdm.notebook import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN = '../input/prostate-cancer-grade-assessment/train_images/'\nsz = 64 # Dimension of each tile.\nN = 49 # Number of tiles.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv = pd.read_csv('../input/prostate-cancer-grade-assessment/train.csv')\nimages = train_csv['image_id']","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def tile(img):\n    result = []\n    shape = img.shape\n    \n    '''Calculating the padding required in Horizontal and Vertical direction respectively'''\n    pad0,pad1 = (sz - shape[0]%sz)%sz, (sz - shape[1]%sz)%sz\n    \n    '''np.pad is used to pad the image. the second positional argument represents number of rows/columns to be padded, ie.\n        [[top_pad, bottom_pad], [left_pad, right_pad],[front_pad, back_pad]], with value 255'''\n    img = np.pad(img,[[pad0//2,pad0-pad0//2],[pad1//2,pad1-pad1//2],[0,0]],\n                constant_values=255)\n    \n    '''Reshaping the image into blocks of 128* 128\n    So, the img becomes 5 dimensional, and dimensions representthe following:\n    (num_vertical_blocks, height_each_block, num_horizontal_blocks, width_each_block, dimension)'''\n    img = img.reshape(img.shape[0]//sz,sz,img.shape[1]//sz,sz,3)\n    \n    '''Reshaping them into (num_horizontal_blocks x num_vertical_blocks) blocks of 128x128 in 3 dimensions'''\n    img = img.transpose(0,2,1,3,4).reshape(-1,sz,sz,3)\n    \n    '''If the number of blocks is less than the 16 or N, then pad remaining blocks with value 255 (White pixels).'''\n    if len(img) < N:\n        img = np.pad(img,[[0,N-len(img)],[0,0],[0,0],[0,0]],constant_values=255)\n        \n    '''Now get top N blocks from the image which have the lowest sum (which means they have fewer white pixels and hence more information).'''\n    idxs = np.argsort(img.reshape(img.shape[0],-1).sum(-1))[:N]\n    \n    '''Filter out the N blocks and then stack them into a single image of dimensions (512x512x3) with N tiles concatenated together.'''\n    img = img[idxs]\n    img = np.vstack((np.hstack((img[:7])),np.hstack((img[7:14])),np.hstack((img[14:21])),np.hstack((img[21:28])),np.hstack((img[28:35])),np.hstack((img[35:42])),np.hstack((img[42:49]))))\n    return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"'''Apply the tile() to all the images and convert them into Concatenated tile images. We'll use then use these images to train our models.'''\nfor x in tqdm(images):\n    '''Get the image of lowest resolution from the MultiImage.'''\n    img = skimage.io.MultiImage(os.path.join(TRAIN,f'{x}.tiff'))[-1]\n    img = tile(img)\n\n    img = Image.fromarray(img, 'RGB')\n    img.save(f'{x}.png')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Original Image","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img = skimage.io.MultiImage(os.path.join(TRAIN,f'003d4dd6bd61221ebc0bfb9350db333f.tiff'))[-1]\nprint(img.shape)\nfig = plt.figure(figsize = (10, 10))\nplt.imshow(img)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tiled Image","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img = tile(img)\nprint(img.shape)\nfig = plt.figure(figsize = (10, 10))\nplt.imshow(img)\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}