{
  "id": 432666,
  "title": "102nd place solution",
  "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/432666",
  "author_name": "luddite^",
  "post_date": "2023-08-18T10:28:37.872000",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>First of all, I want to thank the organizers who hosted such a great competition and kagglers who generously shared their knowledge. <br>\nI was not intended to write my solution because my rank is not fascinating and I'm busy with the entrance exam for graduate college, but I couldn't resist the temptation of a duck T-shirt…</p>\n<h1>Model</h1>\n<p>I created two models: TwoStageModel and TwoStageModelMultiChannel.</p>\n<pre><code>config.seg_model = \n\n (nn.Module):\n     ():\n        ().__init__()\n        self.config = config\n        self.model = seg_models[config.seg_model](\n            encoder_name=config.encoder_name,\n            encoder_weights=,\n            in_channels=,\n            classes=,\n            activation=,\n        )\n        self.model2= seg_models[config.seg_model](\n            encoder_name=,\n            encoder_weights=,\n            in_channels=,\n            classes=,\n            activation=,\n        )\n        \n        self.conv2d = nn.Conv2d(, , config.kernel_size, padding=config.padding)\n\n     ():\n        out = self.model(x)\n        out2 = torch.cat([out, x], dim=)\n        out2 = self.conv2d(out2)\n        out2 = self.model2(out2)\n         out2, out\n\n (nn.Module):\n     ():\n        ().__init__()\n        self.config = config\n        self.model = seg_models[config.seg_model](\n            encoder_name=config.encoder_name,\n            encoder_weights=,\n            in_channels=,\n            classes=,\n            activation=,\n        )\n        self.model2= seg_models[config.seg_model](\n            encoder_name=,\n            encoder_weights=,\n            in_channels=,\n            classes=,\n            activation=,\n        )\n        self.conv2d1 = nn.Conv2d(, , config.kernel_size, padding=config.padding)\n        self.conv2d2 = nn.Conv2d(, , config.kernel_size, padding=config.padding)\n        self.conv2d3 = nn.Conv2d(+, , config.kernel_size, padding=config.padding)\n\n     ():\n\n        out = self.conv2d1(x)\n        out2 = self.conv2d2(out)\n        out3 = self.model(out2)\n        out2 = torch.cat([out3, out, x], dim=)\n        out2 = self.conv2d3(out2)\n        out2 = self.model2(out2)\n         out2, out3\n</code></pre>\n<h1>Encoder</h1>\n<p>I used 'timm-resnest26d', 'efficientnet-b6', 'vgg19_bn', and so on. Large models game me better performance.</p>\n<h1>Data</h1>\n<p>I used false-color images for ContrailsModel2Stage and all 9 bands for TwoStageModelMultiChannel.</p>\n<h1>Augmentation</h1>\n<p>HorizontalFlip(p=0.1),<br>\nVerticalFlip(p=0.1),<br>\nRandomRotate90(p=0.1),<br>\nShiftScaleRotate(shift_limit=0.0625/2, scale_limit=0.2/2, rotate_limit=15, p=0.1, border_mode=cv2.BORDER_REFLECT)</p>\n<h1>TTA</h1>\n<p>As mentioned in others' write-ups, data flip/rotation TTA severely hurt cv scores. I felt something wrong but thought this is because of <a href=\"https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/419994\" target=\"_blank\">data leakage</a> and ignored.<br>\nInstead of flip/rotation TTA, I applied a 1-pixel slide TTA and calculated the max value of original predictions and TTA predictions. 1-pixel right-down slide and left-up slide were effective and gave me about a 0.01 score boost, but 1-pixel right-up slide and left-down slide didn't work. I should have given more thought to why this happens…<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9268451%2Fe104d2aef5442825ea965e711a0b5856%2F2023-08-18%20191942.png?generation=1692354026668584&amp;alt=media\" alt=\"\"></p>",
  "messages": [
    {
      "id": 2396551,
      "postDate": "2023-08-18T10:28:37.873Z",
      "content": "<p>First of all, I want to thank the organizers who hosted such a great competition and kagglers who generously shared their knowledge. <br>\nI was not intended to write my solution because my rank is not fascinating and I'm busy with the entrance exam for graduate college, but I couldn't resist the temptation of a duck T-shirt…</p>\n<h1>Model</h1>\n<p>I created two models: TwoStageModel and TwoStageModelMultiChannel.</p>\n<pre><code>config.seg_model = \n\n (nn.Module):\n     ():\n        ().__init__()\n        self.config = config\n        self.model = seg_models[config.seg_model](\n            encoder_name=config.encoder_name,\n            encoder_weights=,\n            in_channels=,\n            classes=,\n            activation=,\n        )\n        self.model2= seg_models[config.seg_model](\n            encoder_name=,\n            encoder_weights=,\n            in_channels=,\n            classes=,\n            activation=,\n        )\n        \n        self.conv2d = nn.Conv2d(, , config.kernel_size, padding=config.padding)\n\n     ():\n        out = self.model(x)\n        out2 = torch.cat([out, x], dim=)\n        out2 = self.conv2d(out2)\n        out2 = self.model2(out2)\n         out2, out\n\n (nn.Module):\n     ():\n        ().__init__()\n        self.config = config\n        self.model = seg_models[config.seg_model](\n            encoder_name=config.encoder_name,\n            encoder_weights=,\n            in_channels=,\n            classes=,\n            activation=,\n        )\n        self.model2= seg_models[config.seg_model](\n            encoder_name=,\n            encoder_weights=,\n            in_channels=,\n            classes=,\n            activation=,\n        )\n        self.conv2d1 = nn.Conv2d(, , config.kernel_size, padding=config.padding)\n        self.conv2d2 = nn.Conv2d(, , config.kernel_size, padding=config.padding)\n        self.conv2d3 = nn.Conv2d(+, , config.kernel_size, padding=config.padding)\n\n     ():\n\n        out = self.conv2d1(x)\n        out2 = self.conv2d2(out)\n        out3 = self.model(out2)\n        out2 = torch.cat([out3, out, x], dim=)\n        out2 = self.conv2d3(out2)\n        out2 = self.model2(out2)\n         out2, out3\n</code></pre>\n<h1>Encoder</h1>\n<p>I used 'timm-resnest26d', 'efficientnet-b6', 'vgg19_bn', and so on. Large models game me better performance.</p>\n<h1>Data</h1>\n<p>I used false-color images for ContrailsModel2Stage and all 9 bands for TwoStageModelMultiChannel.</p>\n<h1>Augmentation</h1>\n<p>HorizontalFlip(p=0.1),<br>\nVerticalFlip(p=0.1),<br>\nRandomRotate90(p=0.1),<br>\nShiftScaleRotate(shift_limit=0.0625/2, scale_limit=0.2/2, rotate_limit=15, p=0.1, border_mode=cv2.BORDER_REFLECT)</p>\n<h1>TTA</h1>\n<p>As mentioned in others' write-ups, data flip/rotation TTA severely hurt cv scores. I felt something wrong but thought this is because of <a href=\"https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/419994\" target=\"_blank\">data leakage</a> and ignored.<br>\nInstead of flip/rotation TTA, I applied a 1-pixel slide TTA and calculated the max value of original predictions and TTA predictions. 1-pixel right-down slide and left-up slide were effective and gave me about a 0.01 score boost, but 1-pixel right-up slide and left-down slide didn't work. I should have given more thought to why this happens…<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9268451%2Fe104d2aef5442825ea965e711a0b5856%2F2023-08-18%20191942.png?generation=1692354026668584&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "First of all, I want to thank the organizers who hosted such a great competition and kagglers who generously shared their knowledge. \nI was not intended to write my solution because my rank is not fascinating and I'm busy with the entrance exam for graduate college, but I couldn't resist the temptation of a duck T-shirt...\n\n# Model\nI created two models: TwoStageModel and TwoStageModelMultiChannel.\n```python\nconfig.seg_model = 'Unet'\n\nclass ContrailsModel2Stage(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.config = config\n        self.model = seg_models[config.seg_model](\n            encoder_name=config.encoder_name,\n            encoder_weights=\"imagenet\",\n            in_channels=3,\n            classes=1,\n            activation=None,\n        )\n        self.model2= seg_models[config.seg_model](\n            encoder_name='timm-resnest14d',\n            encoder_weights=\"imagenet\",\n            in_channels=3,\n            classes=1,\n            activation=None,\n        )\n        # add conv2d, input channel=3, output channel=1, kernel_size=3, padding=1\n        self.conv2d = nn.Conv2d(4, 3, config.kernel_size, padding=config.padding)\n\n    def forward(self, x):\n        out = self.model(x)\n        out2 = torch.cat([out, x], dim=1)\n        out2 = self.conv2d(out2)\n        out2 = self.model2(out2)\n        return out2, out\n\nclass TwoStageModelMultiChannel(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.config = config\n        self.model = seg_models[config.seg_model](\n            encoder_name=config.encoder_name,\n            encoder_weights=\"imagenet\",\n            in_channels=3,\n            classes=1,\n            activation=None,\n        )\n        self.model2= seg_models[config.seg_model](\n            encoder_name='timm-resnest14d',\n            encoder_weights=\"imagenet\",\n            in_channels=3,\n            classes=1,\n            activation=None,\n        )\n        self.conv2d1 = nn.Conv2d(9, 6, config.kernel_size, padding=config.padding)\n        self.conv2d2 = nn.Conv2d(6, 3, config.kernel_size, padding=config.padding)\n        self.conv2d3 = nn.Conv2d(15+1, 3, config.kernel_size, padding=config.padding)\n\n    def forward(self, x):\n\n        out = self.conv2d1(x)\n        out2 = self.conv2d2(out)\n        out3 = self.model(out2)\n        out2 = torch.cat([out3, out, x], dim=1)\n        out2 = self.conv2d3(out2)\n        out2 = self.model2(out2)\n        return out2, out3\n```\n\n# Encoder\nI used 'timm-resnest26d', 'efficientnet-b6', 'vgg19_bn', and so on. Large models game me better performance.\n\n# Data\nI used false-color images for ContrailsModel2Stage and all 9 bands for TwoStageModelMultiChannel.\n\n# Augmentation\nHorizontalFlip(p=0.1),\nVerticalFlip(p=0.1),\nRandomRotate90(p=0.1),\nShiftScaleRotate(shift_limit=0.0625/2, scale_limit=0.2/2, rotate_limit=15, p=0.1, border_mode=cv2.BORDER_REFLECT)\n\n# TTA\nAs mentioned in others' write-ups, data flip/rotation TTA severely hurt cv scores. I felt something wrong but thought this is because of [data leakage](https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/419994) and ignored.\nInstead of flip/rotation TTA, I applied a 1-pixel slide TTA and calculated the max value of original predictions and TTA predictions. 1-pixel right-down slide and left-up slide were effective and gave me about a 0.01 score boost, but 1-pixel right-up slide and left-down slide didn't work. I should have given more thought to why this happens...\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9268451%2Fe104d2aef5442825ea965e711a0b5856%2F2023-08-18%20191942.png?generation=1692354026668584&alt=media)",
      "votes": 3
    },
    {
      "id": 2396604,
      "postDate": "2023-08-18T10:55:30.803Z",
      "content": "<p>Hahah, same! I also want SWAG ✨ to be stylish. </p>\n<p>Good luck with the exam!</p>",
      "rawMarkdown": "Hahah, same! I also want SWAG ✨ to be stylish. \n\nGood luck with the exam!",
      "votes": 1,
      "replies": [
        {
          "id": 2396607,
          "postDate": "2023-08-18T11:01:48.213Z",
          "content": "<p>Thank you!<br>\nMay the duck T-shirt with you.</p>",
          "rawMarkdown": "Thank you!\nMay the duck T-shirt with you.",
          "votes": 1
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2396604,
      "author_name": "Man of the year",
      "author_url": "",
      "post_date": "2023-08-18T10:55:30.803000",
      "content": "<p>Hahah, same! I also want SWAG ✨ to be stylish. </p>\n<p>Good luck with the exam!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2396607,
          "author_name": "luddite^",
          "author_url": "",
          "post_date": "2023-08-18T11:01:48.213000",
          "content": "<p>Thank you!<br>\nMay the duck T-shirt with you.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2396551": "First of all, I want to thank the organizers who hosted such a great competition and kagglers who generously shared their knowledge. \nI was not intended to write my solution because my rank is not fascinating and I'm busy with the entrance exam for graduate college, but I couldn't resist the temptation of a duck T-shirt...\n\n# Model\nI created two models: TwoStageModel and TwoStageModelMultiChannel.\n```python\nconfig.seg_model = 'Unet'\n\nclass ContrailsModel2Stage(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.config = config\n        self.model = seg_models[config.seg_model](\n            encoder_name=config.encoder_name,\n            encoder_weights=\"imagenet\",\n            in_channels=3,\n            classes=1,\n            activation=None,\n        )\n        self.model2= seg_models[config.seg_model](\n            encoder_name='timm-resnest14d',\n            encoder_weights=\"imagenet\",\n            in_channels=3,\n            classes=1,\n            activation=None,\n        )\n        # add conv2d, input channel=3, output channel=1, kernel_size=3, padding=1\n        self.conv2d = nn.Conv2d(4, 3, config.kernel_size, padding=config.padding)\n\n    def forward(self, x):\n        out = self.model(x)\n        out2 = torch.cat([out, x], dim=1)\n        out2 = self.conv2d(out2)\n        out2 = self.model2(out2)\n        return out2, out\n\nclass TwoStageModelMultiChannel(nn.Module):\n    def __init__(self, config):\n        super().__init__()\n        self.config = config\n        self.model = seg_models[config.seg_model](\n            encoder_name=config.encoder_name,\n            encoder_weights=\"imagenet\",\n            in_channels=3,\n            classes=1,\n            activation=None,\n        )\n        self.model2= seg_models[config.seg_model](\n            encoder_name='timm-resnest14d',\n            encoder_weights=\"imagenet\",\n            in_channels=3,\n            classes=1,\n            activation=None,\n        )\n        self.conv2d1 = nn.Conv2d(9, 6, config.kernel_size, padding=config.padding)\n        self.conv2d2 = nn.Conv2d(6, 3, config.kernel_size, padding=config.padding)\n        self.conv2d3 = nn.Conv2d(15+1, 3, config.kernel_size, padding=config.padding)\n\n    def forward(self, x):\n\n        out = self.conv2d1(x)\n        out2 = self.conv2d2(out)\n        out3 = self.model(out2)\n        out2 = torch.cat([out3, out, x], dim=1)\n        out2 = self.conv2d3(out2)\n        out2 = self.model2(out2)\n        return out2, out3\n```\n\n# Encoder\nI used 'timm-resnest26d', 'efficientnet-b6', 'vgg19_bn', and so on. Large models game me better performance.\n\n# Data\nI used false-color images for ContrailsModel2Stage and all 9 bands for TwoStageModelMultiChannel.\n\n# Augmentation\nHorizontalFlip(p=0.1),\nVerticalFlip(p=0.1),\nRandomRotate90(p=0.1),\nShiftScaleRotate(shift_limit=0.0625/2, scale_limit=0.2/2, rotate_limit=15, p=0.1, border_mode=cv2.BORDER_REFLECT)\n\n# TTA\nAs mentioned in others' write-ups, data flip/rotation TTA severely hurt cv scores. I felt something wrong but thought this is because of [data leakage](https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/419994) and ignored.\nInstead of flip/rotation TTA, I applied a 1-pixel slide TTA and calculated the max value of original predictions and TTA predictions. 1-pixel right-down slide and left-up slide were effective and gave me about a 0.01 score boost, but 1-pixel right-up slide and left-down slide didn't work. I should have given more thought to why this happens...\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9268451%2Fe104d2aef5442825ea965e711a0b5856%2F2023-08-18%20191942.png?generation=1692354026668584&alt=media)",
    "2396604": "Hahah, same! I also want SWAG ✨ to be stylish. \n\nGood luck with the exam!"
  }
}