First of all, Thank you very much to organizers.\nSecond I would like to thanks my team. We had such a positive, encouraging working environment. Our team contribution generates most of the ideas which you will read below and are shared by members.
\n\nI used medium resolution, the only preprocessing I did was to remove the white background and store medium resolution on SSD drive:
\n\n```
\n\ndef crop_white(image, value: int = 255):\n assert image.shape[2] == 3\n assert image.dtype == np.uint8\n ys, = (image.min((1, 2)) < value).nonzero()\n xs, = (image.min(0).min(1) < value).nonzero()\n if len(xs) == 0 or len(ys) == 0:\n return image\n return image[ys.min():ys.max() + 1, xs.min():xs.max() + 1]
\n\n```
\n\nLike in APTOS competition, it was essential to clean images from pen marks, etc. I have used excellent work from this post: https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/151323 This also reduced the gap between CV and LB
\n\nAugmentation occurred at two levels. (Slide and Tile):
\n\nAfter the biopsy slide is open, we do random padding and applying one of the following transformations (similar to R Guo).
\n\n\ndef get_transforms_train():\n transforms=A.Compose(\n [\n A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.05, rotate_limit=10, border_mode=cv2.BORDER_CONSTANT, p=0.5,value=(255,255,255)),\n A.OneOf([\n A.Flip(p=0.5),\n A.RandomRotate90(p=0.5),\n ], p=0.3\n )\n ]\n )\n return transforms\n
For each tile I used standard fastai GPU augmentations: rotate=(-10, 10),flip vertically (p=0.5). For the padding I used reflection it gave a slight boost on CV
I decided to use a very simple model resnet34 but throughout competitions ended up doing few modifications
\n\nThe main idea is builds up on @iafoss. Aftter resnet enccoder we reshape features to look like a square in a following way: x = x.view(x.shape[0], x.shape[1], x.shape[2]//int(np.sqrt(N)), -1) . Here N represents Number of Tiles. After this we pass all the features to SqueezeExcite Block
After reshaping features, we added 1 SE block to enable the network to learn features for individual slides based on tiles.
\n\nexperiment done by @cateek \n```
\n\ndef make_divisible(v, divisor=8, min_value=None):\n min_value = min_value or divisor\n new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)\n # Make sure that round down does not go down by more than 10%.\n if new_v < 0.9 * v:\n new_v += divisor\n return new_v\ndef sigmoid(x, inplace: bool = False):\n return x.sigmoid_() if inplace else x.sigmoid()\nclass SqueezeExcite(nn.Module):\n def init(self, in_chs, se_ratio=0.25, reduced_base_chs=None,\n act_layer=nn.ReLU, gate_fn=sigmoid, divisor=1, **_):\n super(SqueezeExcite, self).init()\n self.gate_fn = gate_fn\n reduced_chs = make_divisible((reduced_base_chs or in_chs) * se_ratio, divisor)\n self.avg_pool = nn.AdaptiveAvgPool2d(1)\n self.conv_reduce = nn.Conv2d(in_chs, reduced_chs, 1, bias=True)\n self.act1 = act_layer(inplace=True)\n self.conv_expand = nn.Conv2d(reduced_chs, in_chs, 1, bias=True)\n def forward(self, x):\n x_se = self.avg_pool(x)\n x_se = self.conv_reduce(x_se)\n x_se = self.act1(x_se)\n x_se = self.conv_expand(x_se)\n x = x * self.gate_fn(x_se)\n return x\n```
\n\nOnce the feature passed thru SqueezeExcite Layer, I did Normal pooling. Our experiment showed that the batch normalization layer was messing with the last layer's features, so we removed it and saw a slight jump on local cv.
\n\n\nself.pool = nn.Sequential(AdaptiveConcatPool2d(),\n Flatten(),\n nn.Linear(2*nc,512),\n nn.ReLU(inplace=True),\n nn.Dropout(0.4),\n nn.Linear(512,7), \n
I used two heads. One head was for classification second was for regression. I noticed that training with two looses makes training much smoother (with sigmoid trick below) and yields higher local CV (0.88 -> 0.90). In the final prediction, I use output only for the regression head.
\n\nOne small modification that I did before calculating loss is that the regression head used sigmoid to scale outputs between (-1. 6.). This enables much smoother training without bumps and faster convergence.
\n\n```
\n\ndef sigmoid_range(x, low, high):\n return torch.sigmoid(x) * (high - low) + low \n```
\n\nI trained in two phases. In the First phase was trained with 49 tiles and later finetuned with 81 tiles. Both phases were using standard one cycle.
\n\nI trained 5 fold wich resulted on the CV of 0.911 and PB: 0.922.
\n\nOur Best Ensemble was simple average. Of 4 models.
\n\n\n@drhb resnet34 5 FOLD (CV -0.911) + \n@rguo97 5 FOLD (two stage attention model CV 0.92 ) + \n@xiejialun FOLD (EFNET) (CV 0.915-0.917) +\n@cateek Se 1 FOLD (CV -0.91) Final Standing\n
@xiejialun https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/169303\n@rguo97 https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/169108#940504
\n\nLB: 0.914 PB: 0.937
First of all, Thank you very much to organizers.\nSecond I would like to thanks my team. We had such a positive, encouraging working environment. Our team contribution generates most of the ideas which you will read below and are shared by members.
\n\nI used medium resolution, the only preprocessing I did was to remove the white background and store medium resolution on SSD drive:
\n\n```
\n\ndef crop_white(image, value: int = 255):\n assert image.shape[2] == 3\n assert image.dtype == np.uint8\n ys, = (image.min((1, 2)) < value).nonzero()\n xs, = (image.min(0).min(1) < value).nonzero()\n if len(xs) == 0 or len(ys) == 0:\n return image\n return image[ys.min():ys.max() + 1, xs.min():xs.max() + 1]
\n\n```
\n\nLike in APTOS competition, it was essential to clean images from pen marks, etc. I have used excellent work from this post: https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/151323 This also reduced the gap between CV and LB
\n\nAugmentation occurred at two levels. (Slide and Tile):
\n\nAfter the biopsy slide is open, we do random padding and applying one of the following transformations (similar to R Guo).
\n\n\ndef get_transforms_train():\n transforms=A.Compose(\n [\n A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.05, rotate_limit=10, border_mode=cv2.BORDER_CONSTANT, p=0.5,value=(255,255,255)),\n A.OneOf([\n A.Flip(p=0.5),\n A.RandomRotate90(p=0.5),\n ], p=0.3\n )\n ]\n )\n return transforms\n
For each tile I used standard fastai GPU augmentations: rotate=(-10, 10),flip vertically (p=0.5). For the padding I used reflection it gave a slight boost on CV
I decided to use a very simple model resnet34 but throughout competitions ended up doing few modifications
\n\nThe main idea is builds up on @iafoss. Aftter resnet enccoder we reshape features to look like a square in a following way: x = x.view(x.shape[0], x.shape[1], x.shape[2]//int(np.sqrt(N)), -1) . Here N represents Number of Tiles. After this we pass all the features to SqueezeExcite Block
After reshaping features, we added 1 SE block to enable the network to learn features for individual slides based on tiles.
\n\nexperiment done by @cateek \n```
\n\ndef make_divisible(v, divisor=8, min_value=None):\n min_value = min_value or divisor\n new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)\n # Make sure that round down does not go down by more than 10%.\n if new_v < 0.9 * v:\n new_v += divisor\n return new_v\ndef sigmoid(x, inplace: bool = False):\n return x.sigmoid_() if inplace else x.sigmoid()\nclass SqueezeExcite(nn.Module):\n def init(self, in_chs, se_ratio=0.25, reduced_base_chs=None,\n act_layer=nn.ReLU, gate_fn=sigmoid, divisor=1, **_):\n super(SqueezeExcite, self).init()\n self.gate_fn = gate_fn\n reduced_chs = make_divisible((reduced_base_chs or in_chs) * se_ratio, divisor)\n self.avg_pool = nn.AdaptiveAvgPool2d(1)\n self.conv_reduce = nn.Conv2d(in_chs, reduced_chs, 1, bias=True)\n self.act1 = act_layer(inplace=True)\n self.conv_expand = nn.Conv2d(reduced_chs, in_chs, 1, bias=True)\n def forward(self, x):\n x_se = self.avg_pool(x)\n x_se = self.conv_reduce(x_se)\n x_se = self.act1(x_se)\n x_se = self.conv_expand(x_se)\n x = x * self.gate_fn(x_se)\n return x\n```
\n\nOnce the feature passed thru SqueezeExcite Layer, I did Normal pooling. Our experiment showed that the batch normalization layer was messing with the last layer's features, so we removed it and saw a slight jump on local cv.
\n\n\nself.pool = nn.Sequential(AdaptiveConcatPool2d(),\n Flatten(),\n nn.Linear(2*nc,512),\n nn.ReLU(inplace=True),\n nn.Dropout(0.4),\n nn.Linear(512,7), \n
I used two heads. One head was for classification second was for regression. I noticed that training with two looses makes training much smoother (with sigmoid trick below) and yields higher local CV (0.88 -> 0.90). In the final prediction, I use output only for the regression head.
\n\nOne small modification that I did before calculating loss is that the regression head used sigmoid to scale outputs between (-1. 6.). This enables much smoother training without bumps and faster convergence.
\n\n```
\n\ndef sigmoid_range(x, low, high):\n return torch.sigmoid(x) * (high - low) + low \n```
\n\nI trained in two phases. In the First phase was trained with 49 tiles and later finetuned with 81 tiles. Both phases were using standard one cycle.
\n\nI trained 5 fold wich resulted on the CV of 0.911 and PB: 0.922.
\n\nOur Best Ensemble was simple average. Of 4 models.
\n\n\n@drhb resnet34 5 FOLD (CV -0.911) + \n@rguo97 5 FOLD (two stage attention model CV 0.92 ) + \n@xiejialun FOLD (EFNET) (CV 0.915-0.917) +\n@cateek Se 1 FOLD (CV -0.91) Final Standing\n
@xiejialun https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/169303\n@rguo97 https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/169108#940504
\n\nLB: 0.914 PB: 0.937
Thanks to all our teammates and congratulations to all winners. \nI provided two models in our final solution, and they work on median resolution and half-scale high resolution (2x median).
\n\nMedian Resolution Model
\n\nThe median level model takes tiles from @iafoss tiling method as input. Instead of caching all tiles, I give a random offset to the grid during training, this gives around 0.005 boost in CV.
\n\nI used a concatenation of attention pooling and max pooling in my model. I also added two auxiliary tasks, one for classification of isup_grade and another for classifying benign or malicious tiles. I used 48 192x192 tiles for median resolution.
\n\n5 fold median level models with seresnext50 backbone scores around 0.92 in private LB.
\n\nHigh Resolution Model
\n\nIn the second month, I was working on higher resolution. My initial trial was to enlarge tile size and tile number. But even with half-scale high resolution images, my poor machine(2080) can hardly hold for 1 batch size. So I was trying to reduce input dimensions.
\n\nMy median resolution model has an attention pooling, which automatically calculated weights for each tile during pooling. After visualizing it, I found it fantastic to remove useless tiles. Tiles with high attention are often malicious.\n.png?generation=1595468713784626&alt=media)
Therefore, I input all tiles in median resolution with foreground into the network, calculate their attentions and save corresponding tiles in higher resolution according to the attention. I saved top 64 tiles with 384x384 size. In order to deal with overfitting, I saved several group of tiles for each slide, each was padded with different offsets in tiling.
\n\nDuring training, I randomly sampled 32 tiles in these top 64 tiles according to their attention and do validation/inference with top 32 tiles.
\n\nThis model scores over 0.92 in CV and around 0.93 in private LB.
\n\nThings worked\n- Add random padding during tiling
\n\nEnlarge batchsize. While training high resolution models, I used gradient checkpoint and apex to reduce GPU memory usage. Finally with RTX Titan, I can train high resolution efficientnet-b0 with batch size 8.
Deal with noise. The label is noisy, I used mse loss for karolinska and huber loss for radboud. I also tried using 0.3* oof prediction + 0.7* groundtruth as new label and train with mse loss. Both strategy improves CV by 0.005-0.01.
congrates!!๐ \nHigh Resolution Model during inference, you input all tiles in median resolution then getting top 32 tiles. Top 32 tiles Input into high resolution model get final results. So the median resolution model plays a preprocessing role that attention top tiles, and high resolution model predicts isup.\nDid I get it right?
", "rawMarkdown": "congrates!!๐ \nHigh Resolution Model during inference, you input all tiles in median resolution then getting top 32 tiles. Top 32 tiles Input into high resolution model get final results. So the median resolution model plays a preprocessing role that attention top tiles, and high resolution model predicts isup.\nDid I get it right?", "votes": 1 }, { "id": 940798, "postDate": "2020-07-23T05:37:04.543Z", "content": "Yes.
", "rawMarkdown": "Yes.", "votes": 1 }, { "id": 941819, "postDate": "2020-07-23T12:33:08.353Z", "content": "I look forward to seeing the code, to see how that attention thing works ! The image above is quite appealing :)
", "rawMarkdown": "I look forward to seeing the code, to see how that attention thing works ! The image above is quite appealing :) ", "votes": 1 }, { "id": 942327, "postDate": "2020-07-23T17:43:59.273Z", "content": "```python\nclass AttentionPool(nn.Module):\n def init(self,in_ch,hidden=512,dropout=0.25):\n super().init()\n self.in_ch=in_ch
\n\n module=[nn.Linear(in_ch,hidden,bias=True),\n nn.Tanh()\n ]\n if dropout>0:\n module.append(nn.Dropout(dropout))\n module.append(nn.Linear(hidden,1,bias=True))\n self.attention=nn.Sequential(*module)\n\ndef forward(self,x):\n num_patch=x.size(1)\n x=x.view(-1,x.size(2))\n A=self.attention(x)\n A=A.view(-1,num_patch,1)\n wt=F.softmax(A,dim=1)\n return (x.view(-1,num_patch,self.in_ch)*wt).sum(dim=1),A\n```
", "rawMarkdown": "```python\nclass AttentionPool(nn.Module):\n def __init__(self,in_ch,hidden=512,dropout=0.25):\n super().__init__()\n self.in_ch=in_ch\n\n module=[nn.Linear(in_ch,hidden,bias=True),\n nn.Tanh()\n ]\n if dropout>0:\n module.append(nn.Dropout(dropout))\n module.append(nn.Linear(hidden,1,bias=True))\n self.attention=nn.Sequential(*module)\n\n def forward(self,x):\n num_patch=x.size(1)\n x=x.view(-1,x.size(2))\n A=self.attention(x)\n A=A.view(-1,num_patch,1)\n wt=F.softmax(A,dim=1)\n return (x.view(-1,num_patch,self.in_ch)*wt).sum(dim=1),A\n```\n\n", "votes": 3 }, { "id": 944902, "postDate": "2020-07-25T12:45:01.127Z", "content": "@rguo97 I see that this layer has two outputs, does this relate to what you said above about adding \n two auxiliary tasks, one for classifying the isup grade, and the other for classifying benign/malicious tiles?
\n\nIf so, could you tell which output is for which task (I am guessing the first output is for classifying tiles, but I am not sure), and how you went on about classifying tiles into benign/malicious categories? Did you simply resize the masks and use those as labels?
", "rawMarkdown": "@rguo97 I see that this layer has two outputs, does this relate to what you said above about adding \n two auxiliary tasks, one for classifying the isup grade, and the other for classifying benign/malicious tiles? \n\nIf so, could you tell which output is for which task (I am guessing the first output is for classifying tiles, but I am not sure), and how you went on about classifying tiles into benign/malicious categories? Did you simply resize the masks and use those as labels?" }, { "id": 945510, "postDate": "2020-07-25T22:33:03.267Z", "content": "The first output is used for following regression and classification, second one is just attention. I used the attention output to sample higher resolution tiles. The tile prediction head is before this attention pooling. I used mask to generate labels, and will not use tiles for back propagation if their mask values are all 0(unknown).\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3601023%2Fa67618109df67663c6afcf3a873054d2%2F.PNG?generation=1595716262962256&alt=media\")
Thank you, this definitely makes things much clearer. To fully see if I understand it, the general schedule looks something like this, correct?:
\n\nThe tile prediction head can then be seen as a raw classifier that should give a better distinction between isup grades of 0-2 (benign) and malignant (3-5), and the actual regression/classification heads are then the usual heads that predict the actual isup grades. Is that correct?
", "rawMarkdown": "Thank you, this definitely makes things much clearer. To fully see if I understand it, the general schedule looks something like this, correct?:\n\n1. Train the above schematic model on median tiles, if the mask are all unknown, exclude that specific tile from backprop.\n2. Use the trained model's attention layer to extract tiles with high attention values, and upscale these coordinates.\n\nThe tile prediction head can then be seen as a raw classifier that should give a better distinction between isup grades of 0-2 (benign) and malignant (3-5), and the actual regression/classification heads are then the usual heads that predict the actual isup grades. Is that correct?" }, { "id": 945596, "postDate": "2020-07-26T01:57:16.447Z", "content": "Yes. Except tiles are cropped from highest resolution in tiff file then downsampled by 2.
", "rawMarkdown": "Yes. Except tiles are cropped from highest resolution in tiff file then downsampled by 2.\n", "votes": 1 }, { "id": 946018, "postDate": "2020-07-26T09:32:33.117Z", "content": "I see, thank you very much!
", "rawMarkdown": "I see, thank you very much! " }, { "id": 965163, "postDate": "2020-08-10T12:29:41.453Z", "content": "Hi @rguo97!
\nwhat is shown in the second of three picture?
Masks, gray and green are benign (isup 1/2).
", "rawMarkdown": "Masks, gray and green are benign (isup 1/2)." } ] }, { "id": 941984, "postDate": "2020-07-23T14:26:25.893Z", "content": "Thanks to Kaggle team and organizers for the interesting competition! Also big thanks to my teammates! \nAfter some experiments with low resolution and attempts to specifically address label noise issue, I concentrated on designing a generally robust solution. I found training to be very sensitive to batch size and number of tiles. Therefore I decided to stick with resnext50 with groupnorm and weights stantardization, which allowed to train with batch size 1.
\n\nMy work is based on @iafoss 's kernel. I extracted tiles from WSI using skimage view_as_windows, removing completely white tiles. I applied very mild random_crop+rotate to whole slide, which allowed to get slightly different tiles during each iteration. I used 49 tiles of shape 224x224, taking ~1/3 of tiles from an array, sorted by tissue amount, and the rest of tiles were taken at random.
Model was trained for 50 epochs with reduce on plateau scheduler, and finetuned for another 20 epochs with cosine decay. Although the latter resulted in nothing more, than overfitting. \nI used regression formulation with a single output and smoothL1 loss.
\n\nTile level augmentations used(courtesy of albumentations library):\n\nOneOf([\n RGBShift(p=1),\n RandomGamma(p=1),\n], p=0.5),\nRandomBrightnessContrast(p=0.7),\nOneOf([\n RandomRotate90(p=1),\n Flip(p=1),\n Rotate(limit=10, border_mode=0, value=(255, 255, 255), p=1),\n ShiftScaleRotate(shift_limit=0.15, scale_limit=0.1, rotate_limit=10, border_mode=0, value=(255, 255, 255), p=1),\n], p=0.25),\nOneOf([\n Cutout(num_holes=50, max_h_size=10, max_w_size=10, fill_value=0, p=1),\n Cutout(num_holes=70, max_h_size=7, max_w_size=7, fill_value=0, p=1),\n Cutout(num_holes=100, max_h_size=5, max_w_size=5, fill_value=0, p=1),\n], p=0.2),\n\nOut of necessity to save submits, I really did only 3 submission of this model\nLocal / Public / Private:\n0.896 / 0.885 / 0.922\n0.903 / 0.892 / 0.920\n0.904 / 0.893 / 0.914\nOne with the highest local/public score was finetuned for another 20 epochs, as described earlier, and led to overfitting.
Things that didn't work:\n1. Various stain normalization methods\n2. Online erroneous labels mining\n3. Overlapping tiles \n4. smaller models with batchnorm
", "rawMarkdown": "Thanks to Kaggle team and organizers for the interesting competition! Also big thanks to my teammates! \nAfter some experiments with low resolution and attempts to specifically address label noise issue, I concentrated on designing a generally robust solution. I found training to be very sensitive to batch size and number of tiles. Therefore I decided to stick with resnext50 with groupnorm and [weights stantardization](https://arxiv.org/abs/1903.10520), which allowed to train with batch size 1. \n\nMy work is based on @iafoss 's kernel. I extracted tiles from WSI using `skimage view_as_windows`, removing completely white tiles. I applied very mild random_crop+rotate to whole slide, which allowed to get slightly different tiles during each iteration. I used 49 tiles of shape 224x224, taking ~1/3 of tiles from an array, sorted by tissue amount, and the rest of tiles were taken at random.\n \nModel was trained for 50 epochs with reduce on plateau scheduler, and finetuned for another 20 epochs with cosine decay. Although the latter resulted in nothing more, than overfitting. \nI used regression formulation with a single output and smoothL1 loss.\n\nTile level augmentations used(courtesy of albumentations library):\n```\nOneOf([\n\tRGBShift(p=1),\n\tRandomGamma(p=1),\n], p=0.5),\nRandomBrightnessContrast(p=0.7),\nOneOf([\n\tRandomRotate90(p=1),\n\tFlip(p=1),\n\tRotate(limit=10, border_mode=0, value=(255, 255, 255), p=1),\n\tShiftScaleRotate(shift_limit=0.15, scale_limit=0.1, rotate_limit=10, border_mode=0, value=(255, 255, 255), p=1),\n], p=0.25),\nOneOf([\n\tCutout(num_holes=50, max_h_size=10, max_w_size=10, fill_value=0, p=1),\n\tCutout(num_holes=70, max_h_size=7, max_w_size=7, fill_value=0, p=1),\n\tCutout(num_holes=100, max_h_size=5, max_w_size=5, fill_value=0, p=1),\n], p=0.2),\n```\nOut of necessity to save submits, I really did only 3 submission of this model\nLocal / Public / Private:\n0.896 / 0.885 / 0.922\n0.903 / 0.892 / 0.920\n0.904 / 0.893 / 0.914\nOne with the highest local/public score was finetuned for another 20 epochs, as described earlier, and led to overfitting. \n\nThings that didn't work:\n1. Various stain normalization methods\n2. Online erroneous labels mining\n3. Overlapping tiles \n4. smaller models with batchnorm\n ", "votes": 7 }, { "id": 944247, "postDate": "2020-07-25T00:48:16.613Z", "content": "@drhabib , Congratulations. It seems that you didn't remove noise from labels, so I wanted to ask what was the spread of your LB result for different seeds if u checked it. I have a hypotheses that noise removal reduces it making the results more stable, but most of my experiments ran with noise removal, so I cannot do a comparison. In my case the std is ~0.003 for 5 models (4-folds) trained with different seeds, while I saw some people at the forum reported really huge differences of 0.03. I just wanted to check how it looks for your models?
", "rawMarkdown": "@drhabib , Congratulations. It seems that you didn't remove noise from labels, so I wanted to ask what was the spread of your LB result for different seeds if u checked it. I have a hypotheses that noise removal reduces it making the results more stable, but most of my experiments ran with noise removal, so I cannot do a comparison. In my case the std is ~0.003 for 5 models (4-folds) trained with different seeds, while I saw some people at the forum reported really huge differences of 0.03. I just wanted to check how it looks for your models?", "votes": 6, "replies": [ { "id": 945534, "postDate": "2020-07-25T23:35:25.830Z", "content": "Thank you for the questions. If I understand your question correctly, we have not tried to do diffrent seeds me and @cateek had split which we created when we merged and we used this split to run all our experiments...
\n\nBy the way. I just want to say again thank you very much for your wonderful kernels and insight during the competition.
", "rawMarkdown": "Thank you for the questions. If I understand your question correctly, we have not tried to do diffrent seeds me and @cateek had split which we created when we merged and we used this split to run all our experiments... \n\nBy the way. I just want to say again thank you very much for your wonderful kernels and insight during the competition. ", "votes": 1 }, { "id": 945536, "postDate": "2020-07-25T23:38:39.077Z", "content": "You are very welcome
", "rawMarkdown": "You are very welcome ", "votes": 1 } ] }, { "id": 979773, "postDate": "2020-08-21T05:38:47.167Z", "content": "Great work @drhabib and an amazing write up! Great to see Fastai implementations on top of the LB!
", "rawMarkdown": "Great work @drhabib and an amazing write up! Great to see Fastai implementations on top of the LB! ", "votes": 1 }, { "id": 945741, "postDate": "2020-07-26T05:34:46.317Z", "content": "Cool!
", "rawMarkdown": "Cool!", "votes": 1 }, { "id": 945036, "postDate": "2020-07-25T14:25:40.343Z", "content": "Congratulation!!. thank you for your insight!. I learned a lot ~
", "rawMarkdown": "Congratulation!!. thank you for your insight!. I learned a lot ~ ", "votes": 1 }, { "id": 944962, "postDate": "2020-07-25T13:30:13.717Z", "content": "@drhabib \nCongratulation. and thanks for sharing. I didn't know about such a trick on sigmoid. Happy to learn. =)
", "rawMarkdown": "@drhabib \nCongratulation. and thanks for sharing. I didn't know about such a trick on sigmoid. Happy to learn. =)", "votes": 1 }, { "id": 941547, "postDate": "2020-07-23T09:39:59.217Z", "content": "Training\n\"I trained in two phases. In the First phase was trained with 49 tiles and later finetuned with 81 tiles. Both phases were using standard one cycle.\"\nI don't quite understand your statement, could you elaborate on it?
", "rawMarkdown": "Training\n\"I trained in two phases. In the First phase was trained with 49 tiles and later finetuned with 81 tiles. Both phases were using standard one cycle.\"\nI don't quite understand your statement, could you elaborate on it?", "votes": 1, "replies": [ { "id": 941806, "postDate": "2020-07-23T12:27:15.980Z", "content": "I train first using 49 Tiles. Save The Weights of the model. Change Number of Tiles to 81, Load the saved weights and continue training. Both 49 and 82 Tiles use one cycle learning rate schedule.
", "rawMarkdown": "I train first using 49 Tiles. Save The Weights of the model. Change Number of Tiles to 81, Load the saved weights and continue training. Both 49 and 82 Tiles use one cycle learning rate schedule. ", "votes": 2 }, { "id": 942795, "postDate": "2020-07-24T02:15:45.333Z", "content": "Have you ever tried using only one stage? Will the performance make a difference๏ผ
", "rawMarkdown": "Have you ever tried using only one stage? Will the performance make a difference๏ผ" } ] }, { "id": 941235, "postDate": "2020-07-23T06:24:12.820Z", "content": "congrats @drhabib and thanks for good explanation!
\n\nI didnโt participate in this competition.\nThanks to a good explanation, I learned a lot. ๐
", "rawMarkdown": "congrats @drhabib and thanks for good explanation!\n\nI didnโt participate in this competition.\nThanks to a good explanation, I learned a lot. ๐ ", "votes": 1 }, { "id": 940811, "postDate": "2020-07-23T05:49:30.240Z", "content": "Congratulations, very successful
", "rawMarkdown": "Congratulations, very successful\n", "votes": 1 }, { "id": 940711, "postDate": "2020-07-23T05:12:54.667Z", "content": "Congrats! Do you plan to share your code?
\n\nAlso, this may be the first time fastai has been used to win a competition! cc: @jhoward
", "rawMarkdown": "Congrats! Do you plan to share your code?\n\nAlso, this may be the first time fastai has been used to win a competition! cc: @jhoward ", "votes": 1, "replies": [ { "id": 940808, "postDate": "2020-07-23T05:46:28.330Z", "content": "Thank you very much! Yes I am planing to share code :)
", "rawMarkdown": "Thank you very much! Yes I am planing to share code :) ", "votes": 1 } ] }, { "id": 940610, "postDate": "2020-07-23T03:48:32.943Z", "content": "Great solution and thanks for the explanation. Can you say how many epochs you trained for ?
", "rawMarkdown": "Great solution and thanks for the explanation. Can you say how many epochs you trained for ?", "votes": 1, "replies": [ { "id": 940613, "postDate": "2020-07-23T03:50:50.297Z", "content": "In my case is it was 60=) For other it was between (10-40).
", "rawMarkdown": "In my case is it was 60=) For other it was between (10-40).", "votes": 2 }, { "id": 940801, "postDate": "2020-07-23T05:41:14.747Z", "content": "Thanks for the reply. How long approx did it take to run an epoch ?
", "rawMarkdown": "Thanks for the reply. How long approx did it take to run an epoch ? ", "votes": 1 }, { "id": 940806, "postDate": "2020-07-23T05:45:07.990Z", "content": "for me 11 min. For @rguo97 on high resolution 2nd level takes 30โ40 min . For rest of the team between 7-20 min.
", "rawMarkdown": "for me 11 min. For @rguo97 on high resolution 2nd level takes 30โ40 min . For rest of the team between 7-20 min. ", "votes": 1 } ] }, { "id": 940576, "postDate": "2020-07-23T03:12:19.547Z", "content": "Congratulations for the 2nd place!
\n\nYour team is high on both public and private LB, so I want to ask: do you have stable cv - LB relationship? How did you chose the final subs? Thanks.
", "rawMarkdown": "Congratulations for the 2nd place!\n\nYour team is high on both public and private LB, so I want to ask: do you have stable cv - LB relationship? How did you chose the final subs? Thanks.", "votes": 1, "replies": [ { "id": 940580, "postDate": "2020-07-23T03:18:46.483Z", "content": "Our best CV ensemble is a little bit higher than best LB ensemble. CV-LB relationship is still not very reliable but in my case, CV and private-LB has good relationship.
", "rawMarkdown": "Our best CV ensemble is a little bit higher than best LB ensemble. CV-LB relationship is still not very reliable but in my case, CV and private-LB has good relationship.", "votes": 2 }, { "id": 940597, "postDate": "2020-07-23T03:39:27.133Z", "content": "Thanks. So I assume you chose best cv sub and best public LB sub as final subs. And the former turned out to be 2nd place private?
", "rawMarkdown": "Thanks. So I assume you chose best cv sub and best public LB sub as final subs. And the former turned out to be 2nd place private?", "votes": 1 }, { "id": 940600, "postDate": "2020-07-23T03:40:36.577Z", "content": "Yes, but best CV scores 0.937 and best LB scores 0.935
", "rawMarkdown": "Yes, but best CV scores 0.937 and best LB scores 0.935", "votes": 2 }, { "id": 940625, "postDate": "2020-07-23T03:59:37.283Z", "content": "wow that's very stable. Both of your subs would be 2nd private. Looks like diverse ensembles can mitigate the cv-LB inconsistency to some extent ๐
", "rawMarkdown": "wow that's very stable. Both of your subs would be 2nd private. Looks like diverse ensembles can mitigate the cv-LB inconsistency to some extent ๐ ", "votes": 3 }, { "id": 941781, "postDate": "2020-07-23T12:12:12.137Z", "content": "> Yes, but best CV scores 0.937 and best LB scores 0.935
\n\nThat's an incredible feat !
", "rawMarkdown": "> Yes, but best CV scores 0.937 and best LB scores 0.935\n\nThat's an incredible feat ! " } ] }, { "id": 940532, "postDate": "2020-07-23T02:18:50.043Z", "content": "Congrats!!๐
", "rawMarkdown": "Congrats!!๐ ", "votes": 1 }, { "id": 940500, "postDate": "2020-07-23T01:45:24.337Z", "content": "Congrats!๐๐ I'm so curious about your idea.
", "rawMarkdown": "Congrats!๐๐ I'm so curious about your idea. ", "votes": 1 }, { "id": 940470, "postDate": "2020-07-23T01:09:13.353Z", "content": "Great game!\nLike APTOS, I guess ensambling was a key solution. Looking forward to specific solutions.
", "rawMarkdown": "Great game!\nLike APTOS, I guess ensambling was a key solution. Looking forward to specific solutions.", "votes": 1, "replies": [ { "id": 940657, "postDate": "2020-07-23T04:27:26.640Z", "content": "yes ensembling helped us a lot =) Congratulations on 1st place =)
", "rawMarkdown": "yes ensembling helped us a lot =) Congratulations on 1st place =)", "votes": 1 }, { "id": 944351, "postDate": "2020-07-25T03:35:52.450Z", "content": "Thank you! Your solution was a lot more stable both on LB/PB and I learned a lot from it!
", "rawMarkdown": "Thank you! Your solution was a lot more stable both on LB/PB and I learned a lot from it!", "votes": 1 } ] }, { "id": 940454, "postDate": "2020-07-23T00:38:59.567Z", "content": "Congrats and wait for your update
", "rawMarkdown": "Congrats and wait for your update\n\n", "votes": 1 }, { "id": 941901, "postDate": "2020-07-23T13:38:44.137Z", "content": "1) Yes 50, 50\n2) nn.MSELoss()
Very cool.
", "rawMarkdown": "Very cool." }, { "id": 941944, "postDate": "2020-07-23T14:03:28.400Z", "rawMarkdown": "", "isDeleted": true }, { "id": 977015, "postDate": "2020-08-19T08:27:55.960Z", "content": "Thanks for sharing and congrats!
", "rawMarkdown": "Thanks for sharing and congrats! ", "votes": 1 } ], "comments": [ { "id": 940504, "author_name": "R Guo", "author_url": "", "post_date": "2020-07-23T01:48:44.607000", "content": "Thanks to all our teammates and congratulations to all winners. \nI provided two models in our final solution, and they work on median resolution and half-scale high resolution (2x median).
\n\nMedian Resolution Model
\n\nThe median level model takes tiles from @iafoss tiling method as input. Instead of caching all tiles, I give a random offset to the grid during training, this gives around 0.005 boost in CV.
\n\nI used a concatenation of attention pooling and max pooling in my model. I also added two auxiliary tasks, one for classification of isup_grade and another for classifying benign or malicious tiles. I used 48 192x192 tiles for median resolution.
\n\n5 fold median level models with seresnext50 backbone scores around 0.92 in private LB.
\n\nHigh Resolution Model
\n\nIn the second month, I was working on higher resolution. My initial trial was to enlarge tile size and tile number. But even with half-scale high resolution images, my poor machine(2080) can hardly hold for 1 batch size. So I was trying to reduce input dimensions.
\n\nMy median resolution model has an attention pooling, which automatically calculated weights for each tile during pooling. After visualizing it, I found it fantastic to remove useless tiles. Tiles with high attention are often malicious.\n.png?generation=1595468713784626&alt=media)
Therefore, I input all tiles in median resolution with foreground into the network, calculate their attentions and save corresponding tiles in higher resolution according to the attention. I saved top 64 tiles with 384x384 size. In order to deal with overfitting, I saved several group of tiles for each slide, each was padded with different offsets in tiling.
\n\nDuring training, I randomly sampled 32 tiles in these top 64 tiles according to their attention and do validation/inference with top 32 tiles.
\n\nThis model scores over 0.92 in CV and around 0.93 in private LB.
\n\nThings worked\n- Add random padding during tiling
\n\nEnlarge batchsize. While training high resolution models, I used gradient checkpoint and apex to reduce GPU memory usage. Finally with RTX Titan, I can train high resolution efficientnet-b0 with batch size 8.
Deal with noise. The label is noisy, I used mse loss for karolinska and huber loss for radboud. I also tried using 0.3* oof prediction + 0.7* groundtruth as new label and train with mse loss. Both strategy improves CV by 0.005-0.01.
congrates!!๐ \nHigh Resolution Model during inference, you input all tiles in median resolution then getting top 32 tiles. Top 32 tiles Input into high resolution model get final results. So the median resolution model plays a preprocessing role that attention top tiles, and high resolution model predicts isup.\nDid I get it right?
", "votes": 1, "replies": [] }, { "id": 940798, "author_name": "R Guo", "author_url": "", "post_date": "2020-07-23T05:37:04.543000", "content": "Yes.
", "votes": 1, "replies": [] }, { "id": 941819, "author_name": "Benjamin Dubreu", "author_url": "", "post_date": "2020-07-23T12:33:08.353000", "content": "I look forward to seeing the code, to see how that attention thing works ! The image above is quite appealing :)
", "votes": 1, "replies": [] }, { "id": 942327, "author_name": "R Guo", "author_url": "", "post_date": "2020-07-23T17:43:59.273000", "content": "```python\nclass AttentionPool(nn.Module):\n def init(self,in_ch,hidden=512,dropout=0.25):\n super().init()\n self.in_ch=in_ch
\n\n module=[nn.Linear(in_ch,hidden,bias=True),\n nn.Tanh()\n ]\n if dropout>0:\n module.append(nn.Dropout(dropout))\n module.append(nn.Linear(hidden,1,bias=True))\n self.attention=nn.Sequential(*module)\n\ndef forward(self,x):\n num_patch=x.size(1)\n x=x.view(-1,x.size(2))\n A=self.attention(x)\n A=A.view(-1,num_patch,1)\n wt=F.softmax(A,dim=1)\n return (x.view(-1,num_patch,self.in_ch)*wt).sum(dim=1),A\n```
", "votes": 3, "replies": [] }, { "id": 944902, "author_name": "Stephan", "author_url": "", "post_date": "2020-07-25T12:45:01.127000", "content": "@rguo97 I see that this layer has two outputs, does this relate to what you said above about adding \n two auxiliary tasks, one for classifying the isup grade, and the other for classifying benign/malicious tiles?
\n\nIf so, could you tell which output is for which task (I am guessing the first output is for classifying tiles, but I am not sure), and how you went on about classifying tiles into benign/malicious categories? Did you simply resize the masks and use those as labels?
", "votes": 0, "replies": [] }, { "id": 945510, "author_name": "R Guo", "author_url": "", "post_date": "2020-07-25T22:33:03.267000", "content": "The first output is used for following regression and classification, second one is just attention. I used the attention output to sample higher resolution tiles. The tile prediction head is before this attention pooling. I used mask to generate labels, and will not use tiles for back propagation if their mask values are all 0(unknown).\n
Thank you, this definitely makes things much clearer. To fully see if I understand it, the general schedule looks something like this, correct?:
\n\nThe tile prediction head can then be seen as a raw classifier that should give a better distinction between isup grades of 0-2 (benign) and malignant (3-5), and the actual regression/classification heads are then the usual heads that predict the actual isup grades. Is that correct?
", "votes": 0, "replies": [] }, { "id": 945596, "author_name": "R Guo", "author_url": "", "post_date": "2020-07-26T01:57:16.447000", "content": "Yes. Except tiles are cropped from highest resolution in tiff file then downsampled by 2.
", "votes": 1, "replies": [] }, { "id": 946018, "author_name": "Stephan", "author_url": "", "post_date": "2020-07-26T09:32:33.117000", "content": "I see, thank you very much!
", "votes": 0, "replies": [] }, { "id": 965163, "author_name": "", "author_url": "", "post_date": "2020-08-10T12:29:41.453000", "content": "Hi @rguo97!
\nwhat is shown in the second of three picture?
Masks, gray and green are benign (isup 1/2).
", "votes": 0, "replies": [] } ] }, { "id": 941984, "author_name": "Eek The Cat", "author_url": "", "post_date": "2020-07-23T14:26:25.893000", "content": "Thanks to Kaggle team and organizers for the interesting competition! Also big thanks to my teammates! \nAfter some experiments with low resolution and attempts to specifically address label noise issue, I concentrated on designing a generally robust solution. I found training to be very sensitive to batch size and number of tiles. Therefore I decided to stick with resnext50 with groupnorm and weights stantardization, which allowed to train with batch size 1.
\n\nMy work is based on @iafoss 's kernel. I extracted tiles from WSI using skimage view_as_windows, removing completely white tiles. I applied very mild random_crop+rotate to whole slide, which allowed to get slightly different tiles during each iteration. I used 49 tiles of shape 224x224, taking ~1/3 of tiles from an array, sorted by tissue amount, and the rest of tiles were taken at random.
Model was trained for 50 epochs with reduce on plateau scheduler, and finetuned for another 20 epochs with cosine decay. Although the latter resulted in nothing more, than overfitting. \nI used regression formulation with a single output and smoothL1 loss.
\n\nTile level augmentations used(courtesy of albumentations library):\n\nOneOf([\n RGBShift(p=1),\n RandomGamma(p=1),\n], p=0.5),\nRandomBrightnessContrast(p=0.7),\nOneOf([\n RandomRotate90(p=1),\n Flip(p=1),\n Rotate(limit=10, border_mode=0, value=(255, 255, 255), p=1),\n ShiftScaleRotate(shift_limit=0.15, scale_limit=0.1, rotate_limit=10, border_mode=0, value=(255, 255, 255), p=1),\n], p=0.25),\nOneOf([\n Cutout(num_holes=50, max_h_size=10, max_w_size=10, fill_value=0, p=1),\n Cutout(num_holes=70, max_h_size=7, max_w_size=7, fill_value=0, p=1),\n Cutout(num_holes=100, max_h_size=5, max_w_size=5, fill_value=0, p=1),\n], p=0.2),\n\nOut of necessity to save submits, I really did only 3 submission of this model\nLocal / Public / Private:\n0.896 / 0.885 / 0.922\n0.903 / 0.892 / 0.920\n0.904 / 0.893 / 0.914\nOne with the highest local/public score was finetuned for another 20 epochs, as described earlier, and led to overfitting.
Things that didn't work:\n1. Various stain normalization methods\n2. Online erroneous labels mining\n3. Overlapping tiles \n4. smaller models with batchnorm
", "votes": 7, "replies": [] }, { "id": 944247, "author_name": "Iafoss", "author_url": "", "post_date": "2020-07-25T00:48:16.613000", "content": "@drhabib , Congratulations. It seems that you didn't remove noise from labels, so I wanted to ask what was the spread of your LB result for different seeds if u checked it. I have a hypotheses that noise removal reduces it making the results more stable, but most of my experiments ran with noise removal, so I cannot do a comparison. In my case the std is ~0.003 for 5 models (4-folds) trained with different seeds, while I saw some people at the forum reported really huge differences of 0.03. I just wanted to check how it looks for your models?
", "votes": 6, "replies": [ { "id": 945534, "author_name": "DrHB", "author_url": "", "post_date": "2020-07-25T23:35:25.830000", "content": "Thank you for the questions. If I understand your question correctly, we have not tried to do diffrent seeds me and @cateek had split which we created when we merged and we used this split to run all our experiments...
\n\nBy the way. I just want to say again thank you very much for your wonderful kernels and insight during the competition.
", "votes": 1, "replies": [] }, { "id": 945536, "author_name": "Iafoss", "author_url": "", "post_date": "2020-07-25T23:38:39.077000", "content": "You are very welcome
", "votes": 1, "replies": [] } ] }, { "id": 979773, "author_name": "Harish Vadlamani", "author_url": "", "post_date": "2020-08-21T05:38:47.167000", "content": "Great work @drhabib and an amazing write up! Great to see Fastai implementations on top of the LB!
", "votes": 1, "replies": [] }, { "id": 945741, "author_name": "Vincent Junitio Ungu", "author_url": "", "post_date": "2020-07-26T05:34:46.317000", "content": "Cool!
", "votes": 1, "replies": [] }, { "id": 945036, "author_name": "JJShadow", "author_url": "", "post_date": "2020-07-25T14:25:40.343000", "content": "Congratulation!!. thank you for your insight!. I learned a lot ~
", "votes": 1, "replies": [] }, { "id": 944962, "author_name": "Innat", "author_url": "", "post_date": "2020-07-25T13:30:13.717000", "content": "@drhabib \nCongratulation. and thanks for sharing. I didn't know about such a trick on sigmoid. Happy to learn. =)
", "votes": 1, "replies": [] }, { "id": 941547, "author_name": "", "author_url": "", "post_date": "2020-07-23T09:39:59.217000", "content": "Training\n\"I trained in two phases. In the First phase was trained with 49 tiles and later finetuned with 81 tiles. Both phases were using standard one cycle.\"\nI don't quite understand your statement, could you elaborate on it?
", "votes": 1, "replies": [ { "id": 941806, "author_name": "DrHB", "author_url": "", "post_date": "2020-07-23T12:27:15.980000", "content": "I train first using 49 Tiles. Save The Weights of the model. Change Number of Tiles to 81, Load the saved weights and continue training. Both 49 and 82 Tiles use one cycle learning rate schedule.
", "votes": 2, "replies": [] }, { "id": 942795, "author_name": "", "author_url": "", "post_date": "2020-07-24T02:15:45.333000", "content": "Have you ever tried using only one stage? Will the performance make a difference๏ผ
", "votes": 0, "replies": [] } ] }, { "id": 941235, "author_name": "Heroseo", "author_url": "", "post_date": "2020-07-23T06:24:12.820000", "content": "congrats @drhabib and thanks for good explanation!
\n\nI didnโt participate in this competition.\nThanks to a good explanation, I learned a lot. ๐
", "votes": 1, "replies": [] }, { "id": 940811, "author_name": "Hamza Tanรง", "author_url": "", "post_date": "2020-07-23T05:49:30.240000", "content": "Congratulations, very successful
", "votes": 1, "replies": [] }, { "id": 940711, "author_name": "ilovescience", "author_url": "", "post_date": "2020-07-23T05:12:54.667000", "content": "Congrats! Do you plan to share your code?
\n\nAlso, this may be the first time fastai has been used to win a competition! cc: @jhoward
", "votes": 1, "replies": [ { "id": 940808, "author_name": "DrHB", "author_url": "", "post_date": "2020-07-23T05:46:28.330000", "content": "Thank you very much! Yes I am planing to share code :)
", "votes": 1, "replies": [] } ] }, { "id": 940610, "author_name": "jwelliav", "author_url": "", "post_date": "2020-07-23T03:48:32.943000", "content": "Great solution and thanks for the explanation. Can you say how many epochs you trained for ?
", "votes": 1, "replies": [ { "id": 940613, "author_name": "DrHB", "author_url": "", "post_date": "2020-07-23T03:50:50.297000", "content": "In my case is it was 60=) For other it was between (10-40).
", "votes": 2, "replies": [] }, { "id": 940801, "author_name": "jwelliav", "author_url": "", "post_date": "2020-07-23T05:41:14.747000", "content": "Thanks for the reply. How long approx did it take to run an epoch ?
", "votes": 1, "replies": [] }, { "id": 940806, "author_name": "DrHB", "author_url": "", "post_date": "2020-07-23T05:45:07.990000", "content": "for me 11 min. For @rguo97 on high resolution 2nd level takes 30โ40 min . For rest of the team between 7-20 min.
", "votes": 1, "replies": [] } ] }, { "id": 940576, "author_name": "Bo", "author_url": "", "post_date": "2020-07-23T03:12:19.547000", "content": "Congratulations for the 2nd place!
\n\nYour team is high on both public and private LB, so I want to ask: do you have stable cv - LB relationship? How did you chose the final subs? Thanks.
", "votes": 1, "replies": [ { "id": 940580, "author_name": "R Guo", "author_url": "", "post_date": "2020-07-23T03:18:46.483000", "content": "Our best CV ensemble is a little bit higher than best LB ensemble. CV-LB relationship is still not very reliable but in my case, CV and private-LB has good relationship.
", "votes": 2, "replies": [] }, { "id": 940597, "author_name": "Bo", "author_url": "", "post_date": "2020-07-23T03:39:27.133000", "content": "Thanks. So I assume you chose best cv sub and best public LB sub as final subs. And the former turned out to be 2nd place private?
", "votes": 1, "replies": [] }, { "id": 940600, "author_name": "R Guo", "author_url": "", "post_date": "2020-07-23T03:40:36.577000", "content": "Yes, but best CV scores 0.937 and best LB scores 0.935
", "votes": 2, "replies": [] }, { "id": 940625, "author_name": "Bo", "author_url": "", "post_date": "2020-07-23T03:59:37.283000", "content": "wow that's very stable. Both of your subs would be 2nd private. Looks like diverse ensembles can mitigate the cv-LB inconsistency to some extent ๐
", "votes": 3, "replies": [] }, { "id": 941781, "author_name": "Benjamin Dubreu", "author_url": "", "post_date": "2020-07-23T12:12:12.137000", "content": "> Yes, but best CV scores 0.937 and best LB scores 0.935
\n\nThat's an incredible feat !
", "votes": 0, "replies": [] } ] }, { "id": 940532, "author_name": "wangyunpeng_bio", "author_url": "", "post_date": "2020-07-23T02:18:50.043000", "content": "Congrats!!๐
", "votes": 1, "replies": [] }, { "id": 940500, "author_name": "Aroddary", "author_url": "", "post_date": "2020-07-23T01:45:24.337000", "content": "Congrats!๐๐ I'm so curious about your idea.
", "votes": 1, "replies": [] }, { "id": 940470, "author_name": "arutema47", "author_url": "", "post_date": "2020-07-23T01:09:13.353000", "content": "Great game!\nLike APTOS, I guess ensambling was a key solution. Looking forward to specific solutions.
", "votes": 1, "replies": [ { "id": 940657, "author_name": "DrHB", "author_url": "", "post_date": "2020-07-23T04:27:26.640000", "content": "yes ensembling helped us a lot =) Congratulations on 1st place =)
", "votes": 1, "replies": [] }, { "id": 944351, "author_name": "arutema47", "author_url": "", "post_date": "2020-07-25T03:35:52.450000", "content": "Thank you! Your solution was a lot more stable both on LB/PB and I learned a lot from it!
", "votes": 1, "replies": [] } ] }, { "id": 940454, "author_name": "Youngno Yoon", "author_url": "", "post_date": "2020-07-23T00:38:59.567000", "content": "Congrats and wait for your update
", "votes": 1, "replies": [] }, { "id": 941901, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2020-07-23T13:38:44.137000", "content": "1) Yes 50, 50\n2) nn.MSELoss()
Very cool.
", "votes": 0, "replies": [] }, { "id": 941944, "author_name": "", "author_url": "", "post_date": "2020-07-23T14:03:28.400000", "content": "", "votes": 0, "replies": [] }, { "id": 977015, "author_name": "Abishek Sudarshan", "author_url": "", "post_date": "2020-08-19T08:27:55.960000", "content": "Thanks for sharing and congrats!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "940434": "First of all, Thank you very much to organizers.\nSecond I would like to thanks my team. We had such a positive, encouraging working environment. Our team contribution generates most of the ideas which you will read below and are shared by members. \n\n\n#Simple Resnet34 (DrHB)\n\n# Image Preprocessing \nI used medium resolution, the only preprocessing I did was to remove the white background and store medium resolution on SSD drive: \n\n```\n#function taken from R Guo\ndef crop_white(image, value: int = 255):\n assert image.shape[2] == 3\n assert image.dtype == np.uint8\n ys, = (image.min((1, 2)) < value).nonzero()\n xs, = (image.min(0).min(1) < value).nonzero()\n if len(xs) == 0 or len(ys) == 0:\n return image\n return image[ys.min():ys.max() + 1, xs.min():xs.max() + 1]\n\n```\n\n# Cleaning data\nLike in APTOS competition, it was essential to clean images from pen marks, etc. I have used excellent work from this post: https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/151323 This also reduced the gap between CV and LB\n\n\n# Image Augmenatiosn\nAugmentation occurred at two levels. (Slide and Tile): \n### 1) Full slide\nAfter the biopsy slide is open, we do random padding and applying one of the following transformations (similar to R Guo). \n\n```\ndef get_transforms_train():\n transforms=A.Compose(\n [\n A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.05, rotate_limit=10, border_mode=cv2.BORDER_CONSTANT, p=0.5,value=(255,255,255)),\n A.OneOf([\n A.Flip(p=0.5),\n A.RandomRotate90(p=0.5),\n ], p=0.3\n )\n ]\n )\n return transforms\n```\n\n### 2) Tile\nFor each tile I used standard fastai `GPU` augmentations: `rotate=(-10, 10`),` flip vertically (p=0.5)`. For the padding I used `reflection` it gave a slight boost on CV \n\n# Model \n\nI decided to use a very simple model resnet34 but throughout competitions ended up doing few modifications \n\n### 1) Making square features:\nThe main idea is builds up on @iafoss. Aftter resnet enccoder we reshape features to look like a square in a following way: `x = x.view(x.shape[0], x.shape[1], x.shape[2]//int(np.sqrt(N)), -1)` . Here `N` represents Number of Tiles. After this we pass all the features to `SqueezeExcite` Block\n\n#### 2) SqueezeExcite block\nAfter reshaping features, we added 1 SE block to enable the network to learn features for individual slides based on tiles. \n\nexperiment done by @cateek \n```\n#code adopted \n#https://github.com/rwightman/pytorch-image-models/tree/master/timm/models\ndef make_divisible(v, divisor=8, min_value=None):\n min_value = min_value or divisor\n new_v = max(min_value, int(v + divisor / 2) // divisor * divisor)\n # Make sure that round down does not go down by more than 10%.\n if new_v < 0.9 * v:\n new_v += divisor\n return new_v\ndef sigmoid(x, inplace: bool = False):\n return x.sigmoid_() if inplace else x.sigmoid()\nclass SqueezeExcite(nn.Module):\n def __init__(self, in_chs, se_ratio=0.25, reduced_base_chs=None,\n act_layer=nn.ReLU, gate_fn=sigmoid, divisor=1, **_):\n super(SqueezeExcite, self).__init__()\n self.gate_fn = gate_fn\n reduced_chs = make_divisible((reduced_base_chs or in_chs) * se_ratio, divisor)\n self.avg_pool = nn.AdaptiveAvgPool2d(1)\n self.conv_reduce = nn.Conv2d(in_chs, reduced_chs, 1, bias=True)\n self.act1 = act_layer(inplace=True)\n self.conv_expand = nn.Conv2d(reduced_chs, in_chs, 1, bias=True)\n def forward(self, x):\n x_se = self.avg_pool(x)\n x_se = self.conv_reduce(x_se)\n x_se = self.act1(x_se)\n x_se = self.conv_expand(x_se)\n x = x * self.gate_fn(x_se)\n return x\n```\n\n### 3) Pooling Layer \nOnce the feature passed thru SqueezeExcite Layer, I did Normal pooling. Our experiment showed that the batch normalization layer was messing with the last layer's features, so we removed it and saw a slight jump on local cv. \n\n\n\n```\nself.pool = nn.Sequential(AdaptiveConcatPool2d(),\n Flatten(),\n nn.Linear(2*nc,512),\n nn.ReLU(inplace=True),\n nn.Dropout(0.4),\n nn.Linear(512,7), \n```\n\n\n### 4) Final Head\nI used two heads. One head was for classification second was for regression. I noticed that training with two looses makes training much smoother (with sigmoid trick below) and yields higher local CV (0.88 -> 0.90). In the final prediction, I use output only for the regression head. \n\nOne small modification that I did before calculating loss is that the regression head used sigmoid to scale outputs between (-1. 6.). This enables much smoother training without bumps and faster convergence.\n\n\n```\n#idea taken from fastai\ndef sigmoid_range(x, low, high):\n return torch.sigmoid(x) * (high - low) + low \n```\n\n# Training \nI trained in two phases. In the First phase was trained with 49 tiles and later finetuned with 81 tiles. Both phases were using standard one cycle.\n\n# Final Model \nI trained 5 fold wich resulted on the CV of 0.911 and PB: 0.922.\n\nOur Best Ensemble was simple average. Of 4 models. \n\n```\n@drhb resnet34 5 FOLD (CV -0.911) + \n@rguo97 5 FOLD (two stage attention model CV 0.92 ) + \n@xiejialun FOLD (EFNET) (CV 0.915-0.917) +\n@cateek Se 1 FOLD (CV -0.91) Final Standing\n```\n\n@xiejialun https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/169303\n@rguo97 https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/169108#940504\n\n `LB: 0.914 PB: 0.937`\n\n\n\n", "940504": "Thanks to all our teammates and congratulations to all winners. \nI provided two models in our final solution, and they work on median resolution and half-scale high resolution (2x median).\n\n **Median Resolution Model**\n\nThe median level model takes tiles from @iafoss tiling method as input. Instead of caching all tiles, I give a random offset to the grid during training, this gives around 0.005 boost in CV. \n\nI used a concatenation of [attention pooling](http://proceedings.mlr.press/v80/ilse18a/ilse18a.pdf) and max pooling in my model. I also added two auxiliary tasks, one for classification of isup_grade and another for classifying benign or malicious tiles. I used 48 192x192 tiles for median resolution.\n\n5 fold median level models with seresnext50 backbone scores around 0.92 in private LB.\n\n**High Resolution Model**\n\nIn the second month, I was working on higher resolution. My initial trial was to enlarge tile size and tile number. But even with half-scale high resolution images, my poor machine(2080) can hardly hold for 1 batch size. So I was trying to reduce input dimensions. \n\nMy median resolution model has an attention pooling, which automatically calculated weights for each tile during pooling. After visualizing it, I found it fantastic to remove useless tiles. Tiles with high attention are often malicious.\n.png?generation=1595468713784626&alt=media)\n\nTherefore, I input all tiles in median resolution with foreground into the network, calculate their attentions and save corresponding tiles in higher resolution according to the attention. I saved top 64 tiles with 384x384 size. In order to deal with overfitting, I saved several group of tiles for each slide, each was padded with different offsets in tiling. \n\nDuring training, I randomly sampled 32 tiles in these top 64 tiles according to their attention and do validation/inference with top 32 tiles. \n\nThis model scores over 0.92 in CV and around 0.93 in private LB.\n\n**Things worked**\n- Add random padding during tiling\n\n- Enlarge batchsize. While training high resolution models, I used gradient checkpoint and apex to reduce GPU memory usage. Finally with RTX Titan, I can train high resolution efficientnet-b0 with batch size 8.\n\n- Deal with noise. The label is noisy, I used mse loss for karolinska and huber loss for radboud. I also tried using 0.3* oof prediction + 0.7* groundtruth as new label and train with mse loss. Both strategy improves CV by 0.005-0.01.", "941984": "Thanks to Kaggle team and organizers for the interesting competition! Also big thanks to my teammates! \nAfter some experiments with low resolution and attempts to specifically address label noise issue, I concentrated on designing a generally robust solution. I found training to be very sensitive to batch size and number of tiles. Therefore I decided to stick with resnext50 with groupnorm and [weights stantardization](https://arxiv.org/abs/1903.10520), which allowed to train with batch size 1. \n\nMy work is based on @iafoss 's kernel. I extracted tiles from WSI using `skimage view_as_windows`, removing completely white tiles. I applied very mild random_crop+rotate to whole slide, which allowed to get slightly different tiles during each iteration. I used 49 tiles of shape 224x224, taking ~1/3 of tiles from an array, sorted by tissue amount, and the rest of tiles were taken at random.\n \nModel was trained for 50 epochs with reduce on plateau scheduler, and finetuned for another 20 epochs with cosine decay. Although the latter resulted in nothing more, than overfitting. \nI used regression formulation with a single output and smoothL1 loss.\n\nTile level augmentations used(courtesy of albumentations library):\n```\nOneOf([\n\tRGBShift(p=1),\n\tRandomGamma(p=1),\n], p=0.5),\nRandomBrightnessContrast(p=0.7),\nOneOf([\n\tRandomRotate90(p=1),\n\tFlip(p=1),\n\tRotate(limit=10, border_mode=0, value=(255, 255, 255), p=1),\n\tShiftScaleRotate(shift_limit=0.15, scale_limit=0.1, rotate_limit=10, border_mode=0, value=(255, 255, 255), p=1),\n], p=0.25),\nOneOf([\n\tCutout(num_holes=50, max_h_size=10, max_w_size=10, fill_value=0, p=1),\n\tCutout(num_holes=70, max_h_size=7, max_w_size=7, fill_value=0, p=1),\n\tCutout(num_holes=100, max_h_size=5, max_w_size=5, fill_value=0, p=1),\n], p=0.2),\n```\nOut of necessity to save submits, I really did only 3 submission of this model\nLocal / Public / Private:\n0.896 / 0.885 / 0.922\n0.903 / 0.892 / 0.920\n0.904 / 0.893 / 0.914\nOne with the highest local/public score was finetuned for another 20 epochs, as described earlier, and led to overfitting. \n\nThings that didn't work:\n1. Various stain normalization methods\n2. Online erroneous labels mining\n3. Overlapping tiles \n4. smaller models with batchnorm\n ", "944247": "@drhabib , Congratulations. It seems that you didn't remove noise from labels, so I wanted to ask what was the spread of your LB result for different seeds if u checked it. I have a hypotheses that noise removal reduces it making the results more stable, but most of my experiments ran with noise removal, so I cannot do a comparison. In my case the std is ~0.003 for 5 models (4-folds) trained with different seeds, while I saw some people at the forum reported really huge differences of 0.03. I just wanted to check how it looks for your models?", "979773": "Great work @drhabib and an amazing write up! Great to see Fastai implementations on top of the LB! ", "945741": "Cool!", "945036": "Congratulation!!. thank you for your insight!. I learned a lot ~ ", "944962": "@drhabib \nCongratulation. and thanks for sharing. I didn't know about such a trick on sigmoid. Happy to learn. =)", "941547": "Training\n\"I trained in two phases. In the First phase was trained with 49 tiles and later finetuned with 81 tiles. Both phases were using standard one cycle.\"\nI don't quite understand your statement, could you elaborate on it?", "941235": "congrats @drhabib and thanks for good explanation!\n\nI didnโt participate in this competition.\nThanks to a good explanation, I learned a lot. ๐ ", "940811": "Congratulations, very successful\n", "940711": "Congrats! Do you plan to share your code?\n\nAlso, this may be the first time fastai has been used to win a competition! cc: @jhoward ", "940610": "Great solution and thanks for the explanation. Can you say how many epochs you trained for ?", "940576": "Congratulations for the 2nd place!\n\nYour team is high on both public and private LB, so I want to ask: do you have stable cv - LB relationship? How did you chose the final subs? Thanks.", "940532": "Congrats!!๐ ", "940500": "Congrats!๐๐ I'm so curious about your idea. ", "940470": "Great game!\nLike APTOS, I guess ensambling was a key solution. Looking forward to specific solutions.", "940454": "Congrats and wait for your update\n\n", "941901": "1. For final head, how do you weight the losses? equally 50/50?\n2. For final head, what loss did you use for regression? MSE? MAE? I like that sigmoid trick, have never seen before.", "943738": "Very cool.", "941944": "", "977015": "Thanks for sharing and congrats! " } }