{
  "id": 169141,
  "title": "9th Place Solution [ResNeSt might be the key]",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/169141",
  "author_name": "Sanchit Singh",
  "post_date": "2020-07-23T03:16:31.805000",
  "votes": 18,
  "comment_count": 0,
  "views": 0,
  "content": "<p>First of all Thank you very much to organizers.</p>\n\n<p>My solution is simple.\nI ended up with the following models in my final ensemble submission.</p>\n\n<p>Label Smoothing + Ordinal regression + efficientnet_b0 x 2\nLabel Smoothing + Ordinal regression + GeM + ResNeSt50 x 2</p>\n\n<p>I was not confident with efficientnet_b0 since there was difference of about .02-.03 between the CV (.90XX) and Public LB (.87XX). It seemed to me like it was overfitting the LB despite using Label Smoothing and less stable.\nWhereas, ResNeSt50(Split-Attention Network) gave me stable difference between Public LB (.85XX) and CV (.86XX) along with Label Smoothing.</p>\n\n<p>We all knew that qwk was not a stable metric for given amount of Public Test Images. We also knew that training data seemed to have ample amount of noise.\nSo, My intuition was to select one less scoring more stable model and one best Public LB scoring model for ensembling. <br>\nI manually weighted the models during ensembling. I observed that giving more weight to efficientnet_b0 (Best Public LB scoring Model) leads to decrease in Public LB, therefore I selected the ensemble having equal weights (simple average).</p>\n\n<p>After the releasing of Private LB the same Resnest50 gave a private LB score of .90XX.</p>\n\n<p>I was pretty sure to atleast land in bronze level zone, but shake up seemed to be very rough eventually landing me in gold zone. (completely unexpected)</p>\n\n<p>For training, I used both the techniques for tiling the images (List of tiles and Single large Image of Tiles). I used 36x256x256 (level - 1) tiles along with simple augmentations like hflip,flip,transpose,rotation..</p>\n\n<p><strong>What More Could Have been Done ??</strong></p>\n\n<p>Since I was using kaggle and Colab for training, I was constrained to 30 hr weekly and 12hr limit respectively. It is clear that I don't had enough gpu compute to do more experiments.</p>\n\n<p>But the following seemed to be much effective approach :- </p>\n\n<p>I also tried training seresnext50 + AdaptiveConcatPool + Classification (pretty sure that Ordinal Regression would have scored better) on Single Large Image of tiles dataset with noisy labels removed (Images Having Pen marks ) along with <strong>Progressive Resizing</strong> (ie) training in following sequence:-</p>\n\n<p>16x128x128 (Level-2) --&gt; 16x160x160 (Level-2) --&gt; 20x224x224 (Level-1) and so on....</p>\n\n<p>Make sure to load weights from previous stage while training the next stage.</p>\n\n<p>Due to the limited compute power I wasn't able to train seresnext50 on larger image size, my final input image size was 20x224x224 which gave me CV of .81XX and Public LB of .82XX (pretty stable).\nWhen the Private LB came, the resnext50 model trained in very same way mentioned above gave private LB of .89XX.\nSince I wasn't able to train this on larger images I wasn't able to use it in final ensemble.</p>\n\n<p>I am pretty sure it would have given a decent boost to my final score.</p>",
  "messages": [
    {
      "id": 940578,
      "postDate": "2020-07-23T03:16:31.807Z",
      "content": "<p>First of all Thank you very much to organizers.</p>\n\n<p>My solution is simple.\nI ended up with the following models in my final ensemble submission.</p>\n\n<p>Label Smoothing + Ordinal regression + efficientnet_b0 x 2\nLabel Smoothing + Ordinal regression + GeM + ResNeSt50 x 2</p>\n\n<p>I was not confident with efficientnet_b0 since there was difference of about .02-.03 between the CV (.90XX) and Public LB (.87XX). It seemed to me like it was overfitting the LB despite using Label Smoothing and less stable.\nWhereas, ResNeSt50(Split-Attention Network) gave me stable difference between Public LB (.85XX) and CV (.86XX) along with Label Smoothing.</p>\n\n<p>We all knew that qwk was not a stable metric for given amount of Public Test Images. We also knew that training data seemed to have ample amount of noise.\nSo, My intuition was to select one less scoring more stable model and one best Public LB scoring model for ensembling. <br>\nI manually weighted the models during ensembling. I observed that giving more weight to efficientnet_b0 (Best Public LB scoring Model) leads to decrease in Public LB, therefore I selected the ensemble having equal weights (simple average).</p>\n\n<p>After the releasing of Private LB the same Resnest50 gave a private LB score of .90XX.</p>\n\n<p>I was pretty sure to atleast land in bronze level zone, but shake up seemed to be very rough eventually landing me in gold zone. (completely unexpected)</p>\n\n<p>For training, I used both the techniques for tiling the images (List of tiles and Single large Image of Tiles). I used 36x256x256 (level - 1) tiles along with simple augmentations like hflip,flip,transpose,rotation..</p>\n\n<p><strong>What More Could Have been Done ??</strong></p>\n\n<p>Since I was using kaggle and Colab for training, I was constrained to 30 hr weekly and 12hr limit respectively. It is clear that I don't had enough gpu compute to do more experiments.</p>\n\n<p>But the following seemed to be much effective approach :- </p>\n\n<p>I also tried training seresnext50 + AdaptiveConcatPool + Classification (pretty sure that Ordinal Regression would have scored better) on Single Large Image of tiles dataset with noisy labels removed (Images Having Pen marks ) along with <strong>Progressive Resizing</strong> (ie) training in following sequence:-</p>\n\n<p>16x128x128 (Level-2) --&gt; 16x160x160 (Level-2) --&gt; 20x224x224 (Level-1) and so on....</p>\n\n<p>Make sure to load weights from previous stage while training the next stage.</p>\n\n<p>Due to the limited compute power I wasn't able to train seresnext50 on larger image size, my final input image size was 20x224x224 which gave me CV of .81XX and Public LB of .82XX (pretty stable).\nWhen the Private LB came, the resnext50 model trained in very same way mentioned above gave private LB of .89XX.\nSince I wasn't able to train this on larger images I wasn't able to use it in final ensemble.</p>\n\n<p>I am pretty sure it would have given a decent boost to my final score.</p>",
      "rawMarkdown": "First of all Thank you very much to organizers.\n\nMy solution is simple.\nI ended up with the following models in my final ensemble submission.\n\nLabel Smoothing + Ordinal regression + efficientnet_b0 x 2\nLabel Smoothing + Ordinal regression + GeM + ResNeSt50 x 2\n\nI was not confident with efficientnet_b0 since there was difference of about .02-.03 between the CV (.90XX) and Public LB (.87XX). It seemed to me like it was overfitting the LB despite using Label Smoothing and less stable.\nWhereas, ResNeSt50(Split-Attention Network) gave me stable difference between Public LB (.85XX) and CV (.86XX) along with Label Smoothing.\n\nWe all knew that qwk was not a stable metric for given amount of Public Test Images. We also knew that training data seemed to have ample amount of noise.\nSo, My intuition was to select one less scoring more stable model and one best Public LB scoring model for ensembling.  \nI manually weighted the models during ensembling. I observed that giving more weight to efficientnet_b0 (Best Public LB scoring Model) leads to decrease in Public LB, therefore I selected the ensemble having equal weights (simple average).\n\nAfter the releasing of Private LB the same Resnest50 gave a private LB score of .90XX.\n\nI was pretty sure to atleast land in bronze level zone, but shake up seemed to be very rough eventually landing me in gold zone. (completely unexpected)\n\nFor training, I used both the techniques for tiling the images (List of tiles and Single large Image of Tiles). I used 36x256x256 (level - 1) tiles along with simple augmentations like hflip,flip,transpose,rotation..\n\n**What More Could Have been Done ??**\n\nSince I was using kaggle and Colab for training, I was constrained to 30 hr weekly and 12hr limit respectively. It is clear that I don't had enough gpu compute to do more experiments.\n\nBut the following seemed to be much effective approach :- \n\nI also tried training seresnext50 + AdaptiveConcatPool + Classification (pretty sure that Ordinal Regression would have scored better) on Single Large Image of tiles dataset with noisy labels removed (Images Having Pen marks ) along with **Progressive Resizing** (ie) training in following sequence:-\n\n16x128x128 (Level-2) --&gt; 16x160x160 (Level-2) --&gt; 20x224x224 (Level-1) and so on....\n\nMake sure to load weights from previous stage while training the next stage.\n \nDue to the limited compute power I wasn't able to train seresnext50 on larger image size, my final input image size was 20x224x224 which gave me CV of .81XX and Public LB of .82XX (pretty stable).\nWhen the Private LB came, the resnext50 model trained in very same way mentioned above gave private LB of .89XX.\nSince I wasn't able to train this on larger images I wasn't able to use it in final ensemble.\n\nI am pretty sure it would have given a decent boost to my final score.",
      "votes": 18
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "940578": "First of all Thank you very much to organizers.\n\nMy solution is simple.\nI ended up with the following models in my final ensemble submission.\n\nLabel Smoothing + Ordinal regression + efficientnet_b0 x 2\nLabel Smoothing + Ordinal regression + GeM + ResNeSt50 x 2\n\nI was not confident with efficientnet_b0 since there was difference of about .02-.03 between the CV (.90XX) and Public LB (.87XX). It seemed to me like it was overfitting the LB despite using Label Smoothing and less stable.\nWhereas, ResNeSt50(Split-Attention Network) gave me stable difference between Public LB (.85XX) and CV (.86XX) along with Label Smoothing.\n\nWe all knew that qwk was not a stable metric for given amount of Public Test Images. We also knew that training data seemed to have ample amount of noise.\nSo, My intuition was to select one less scoring more stable model and one best Public LB scoring model for ensembling.  \nI manually weighted the models during ensembling. I observed that giving more weight to efficientnet_b0 (Best Public LB scoring Model) leads to decrease in Public LB, therefore I selected the ensemble having equal weights (simple average).\n\nAfter the releasing of Private LB the same Resnest50 gave a private LB score of .90XX.\n\nI was pretty sure to atleast land in bronze level zone, but shake up seemed to be very rough eventually landing me in gold zone. (completely unexpected)\n\nFor training, I used both the techniques for tiling the images (List of tiles and Single large Image of Tiles). I used 36x256x256 (level - 1) tiles along with simple augmentations like hflip,flip,transpose,rotation..\n\n**What More Could Have been Done ??**\n\nSince I was using kaggle and Colab for training, I was constrained to 30 hr weekly and 12hr limit respectively. It is clear that I don't had enough gpu compute to do more experiments.\n\nBut the following seemed to be much effective approach :- \n\nI also tried training seresnext50 + AdaptiveConcatPool + Classification (pretty sure that Ordinal Regression would have scored better) on Single Large Image of tiles dataset with noisy labels removed (Images Having Pen marks ) along with **Progressive Resizing** (ie) training in following sequence:-\n\n16x128x128 (Level-2) --&gt; 16x160x160 (Level-2) --&gt; 20x224x224 (Level-1) and so on....\n\nMake sure to load weights from previous stage while training the next stage.\n \nDue to the limited compute power I wasn't able to train seresnext50 on larger image size, my final input image size was 20x224x224 which gave me CV of .81XX and Public LB of .82XX (pretty stable).\nWhen the Private LB came, the resnext50 model trained in very same way mentioned above gave private LB of .89XX.\nSince I wasn't able to train this on larger images I wasn't able to use it in final ensemble.\n\nI am pretty sure it would have given a decent boost to my final score."
  }
}