{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport skimage.io\nfrom tqdm.notebook import tqdm\nimport zipfile\nimport numpy as np\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN = '../input/prostate-cancer-grade-assessment/train_images/'\nMASKS = '../input/prostate-cancer-grade-assessment/train_label_masks/'\nOUT_TRAIN = 'train.zip'\nOUT_MASKS = 'masks.zip'\nsz = 224\nN = 24","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some notes for the below function. The first bullet point here refers to the the code below the first \"# note\" in the below function.\n* finds the values we should pad h & w by so that our dims are multiples of our sz\n* pads our images nd masks with chosen  constant vals\n* * reshapes our images into an array of 7 x 128 x 8 x 128 x 3 if img was 896x1024x3\n* swaps the order so our e.g dims would now be 7x8x128x128x3 and reshapes so that our example dim would be 56x128x128x3. In our e.g the 56 elements tiles filled by column first as per reshape.\n* same process for our masks\n* if our img or mask has less than N tiles we pad\n* three things occur here in this one line, we reshape our img so that in our example it would have dim of 56 x (3x128x128). We sum along the last dimension, so we have 56 values, we then order those values in ascending order and return the index of the N smallest values. In the subsequent lines we select the 16 tiles with most tissue.\n* The function returns a list of of 16 dictionaries, each dictionary contains an array describing an image tile, the corresponding mask array, and the index of the tile in the ordering of tiles based on most tissue value (descending order.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def tile(img, mask):\n    result = []\n    shape = img.shape\n    pad0,pad1 = (sz - shape[0]%sz)%sz, (sz - shape[1]%sz)%sz\n    img = np.pad(img,[[pad0//2,pad0-pad0//2],[pad1//2,pad1-pad1//2],[0,0]],\n                constant_values=255)\n    mask = np.pad(mask,[[pad0//2,pad0-pad0//2],[pad1//2,pad1-pad1//2],[0,0]],\n                constant_values=0)\n    img = img.reshape(img.shape[0]//sz,sz,img.shape[1]//sz,sz,3)\n    img = img.transpose(0,2,1,3,4).reshape(-1,sz,sz,3)\n    mask = mask.reshape(mask.shape[0]//sz,sz,mask.shape[1]//sz,sz,3)\n    mask = mask.transpose(0,2,1,3,4).reshape(-1,sz,sz,3)\n    if len(img) < N:\n        mask = np.pad(mask,[[0,N-len(img)],[0,0],[0,0],[0,0]],constant_values=0)\n        img = np.pad(img,[[0,N-len(img)],[0,0],[0,0],[0,0]],constant_values=255)\n    idxs = np.argsort(img.reshape(img.shape[0],-1).sum(-1))[:N]\n    img = img[idxs]\n    mask = mask[idxs]\n    for i in range(len(img)):\n        result.append({'img':img[i], 'mask':mask[i], 'idx':i})\n    return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nnames = [name[:-10] for name in os.listdir(MASKS)]\ntest_names = names[0:10]\ndef img_load(name):\n    img = skimage.io.MultiImage(os.path.join(TRAIN,name+'.tiff'))[-2]\n    return(img)\ndef mask_load(name):\n    mask = skimage.io.MultiImage(os.path.join(MASKS,name+'_mask.tiff'))[-2]\n    return(mask)\n\nimages = [img_load(name) for name in test_names]\nmasks = [mask_load(name) for name in test_names]\nimg = images[0]\nmask = masks[0]\ntiled_test = tile(img, mask)\n# tiled test is a list of 12 dictionaries, each dictionary has an img array, mask array and idx num_ones = (msk == 1).sum()\n# gonna need the train CSV to load masks in different ways according to data_providers\n# we can makeuse of all Radboudumc examples as they do have labels by gleason score, we call also make use of any karolinska example with only one gleason score e.g. 3+3 4+4 etc\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Radboudumc: Prostate glands are individually labelled. Valid values are:\n\n0: background (non tissue) or unknown\n1: stroma (connective tissue, non-epithelium tissue)\n2: healthy (benign) epithelium\n3: cancerous epithelium (Gleason 3)\n4: cancerous epithelium (Gleason 4)\n5: cancerous epithelium (Gleason 5)\n\nKarolinska: Regions are labelled. Valid values:\n0: background (non tissue) or unknown\n1: benign tissue (stroma and epithelium combined)\n2: cancerous tissue (stroma and epithelium combined)\n\nThe label masks of Radboudumc were semi-automatically generated by several deep learning algorithms, contain noise, and can be considered as weakly-supervised labels. The label masks of Karolinska were semi-autotomatically generated based on annotations by a pathologist.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Notes for the below cell in similar format to above.\n* we first instantiate our lists of tile information.\n* we access image id's by ignoring the last 10 characters of mask file names\n* set the distinct zip files we will write to\n* Use tqdm to create a progress bar\n* load our image and masks as array using skimage\n* apply our tile function above and obtain our list of dictionaries which each describe a tile and corresponding.\n* for each tile we calculate mean and std of px val px val squared and append valds to our lists of tile info. \n* we write to png and reverse rgb to bgr for reading with PIL\n* create same file name for tile by id and idx for both imgs and masks","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"x_tot,x2_tot = [],[]\nwith zipfile.ZipFile(OUT_TRAIN, 'w') as img_out,\\\n zipfile.ZipFile(OUT_MASKS, 'w') as mask_out:\n    for name in tqdm(names):\n        img = skimage.io.MultiImage(os.path.join(TRAIN,name+'.tiff'))[-2]\n        mask = skimage.io.MultiImage(os.path.join(MASKS,name+'_mask.tiff'))[-2]\n        tiles = tile(img,mask)\n        for t in tiles:\n            img,mask,idx = t['img'],t['mask'],t['idx']\n            x_tot.append((img/255.0).reshape(-1,3).mean(0))\n            x2_tot.append(((img/255.0)**2).reshape(-1,3).mean(0)) \n            #if read with PIL RGB turns into BGR\n            img[:,:,0] = ((img[:,:,0]/255) - 0.8094)/ 0.4055\n            img[:,:,1] = ((img[:,:,1]/255) - 0.6067)/ 0.5094\n            img[:,:,2] = ((img[:,:,2]/255) - 0.7383)/ 0.4158\n            img = cv2.imencode('.png',cv2.cvtColor(img, cv2.COLOR_RGB2BGR))[1]\n            img_out.writestr(f'{name}_{idx}.png', img)\n            mask = cv2.imencode('.png',mask[:,:,0])[1]\n            mask_out.writestr(f'{name}_{idx}.png', mask)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#image stats\nimg_avr =  np.array(x_tot).mean(0)\nimg_std =  np.sqrt(np.array(x2_tot).mean(0) - img_avr**2)\nprint('mean:',img_avr, ', std:', np.sqrt(img_std))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}