\n
![\"models-table.jpg\"](\"https://i.postimg.cc/CLDGWYD1/models-table.jpg\")
\nCosine scheduler with warm up = 0.02 or 0.03 for all models
Thank you Kaggle and Google Research for hosting this competition.
\nHere's our solution in brief.
\nPreprocessing / Dataset
\n384 and 512 Image size - Bilinear resize images
Augmentations (Following two slightly different augmentations were part of our selected ensemble:)
def ():\n transform = albu.(\n [ \n albu.([\n albu.(p = ),\n albu.(p = ),\n ], p = ),\n albu.(limit = , p = ),\n albu.(p = ),\n albu.(\n mean = (, , ),\n std = (, , ),\n max_pixel_value = \n ),\n (transpose_mask = True),\n ],\n p = , is_check_shapes=False\n )\n return transform\nself.transform = {\n : A.Compose([\n A.ShiftScaleRotate(=0.20, =0, =0.1, =0.25, =cv2.BORDER_CONSTANT, =0),\n A.GridDistortion(=0.25),\n A.HorizontalFlip(=0.25),\n A.VerticalFlip(=0.25),\n A.Resize(G.img_size),\n A.RandomCrop(G.crop_size),\n A.Normalize(=CFG.pp_params[],=CFG.pp_params[]),\n A.pytorch.transforms.ToTensorV2(=)\n ], =1.0),\n }\nValidation (Two different CV schemes in ensemble)
\n.dup_ids = .df.dup_id.unique()\n ():\n dup_idx = .dup_ids[idx]\n rows = .df[.df.dup_id == dup_idx]\n …\nThings that worked
\nModels (UNet)
\n
\nCosine scheduler with warm up = 0.02 or 0.03 for all models
Thresholds
\nA threshold change could have landed us in the money zone. After the deadline submission, threshold of 0.45 was best with pvt. 0.713x and public 0.712x and we selected 0.5 threshold.
Things that didn’t work for us
\nPseudo labels (PL)
\nTest time augmentation
\nTraining with other bands, frames
\nWeighted loss
\nAdjusting decoder channels
\nVarious other backbones
\nPartial Code available here
\nhttps://github.com/furu-kaggle/ICRGW
A big thank you to my amazing team mates @ragnar123 @optimo and @kunihikofurugori 🙏
", "messages": [ { "id": 2385316, "postDate": "2023-08-11T09:09:02.767Z", "content": "Thank you Kaggle and Google Research for hosting this competition.
\nHere's our solution in brief.
\nPreprocessing / Dataset
\n384 and 512 Image size - Bilinear resize images
Augmentations (Following two slightly different augmentations were part of our selected ensemble:)
def ():\n transform = albu.(\n [ \n albu.([\n albu.(p = ),\n albu.(p = ),\n ], p = ),\n albu.(limit = , p = ),\n albu.(p = ),\n albu.(\n mean = (, , ),\n std = (, , ),\n max_pixel_value = \n ),\n (transpose_mask = True),\n ],\n p = , is_check_shapes=False\n )\n return transform\nself.transform = {\n : A.Compose([\n A.ShiftScaleRotate(=0.20, =0, =0.1, =0.25, =cv2.BORDER_CONSTANT, =0),\n A.GridDistortion(=0.25),\n A.HorizontalFlip(=0.25),\n A.VerticalFlip(=0.25),\n A.Resize(G.img_size),\n A.RandomCrop(G.crop_size),\n A.Normalize(=CFG.pp_params[],=CFG.pp_params[]),\n A.pytorch.transforms.ToTensorV2(=)\n ], =1.0),\n }\nValidation (Two different CV schemes in ensemble)
\n.dup_ids = .df.dup_id.unique()\n ():\n dup_idx = .dup_ids[idx]\n rows = .df[.df.dup_id == dup_idx]\n …\nThings that worked
\nModels (UNet)
\n
\nCosine scheduler with warm up = 0.02 or 0.03 for all models
Thresholds
\nA threshold change could have landed us in the money zone. After the deadline submission, threshold of 0.45 was best with pvt. 0.713x and public 0.712x and we selected 0.5 threshold.
Things that didn’t work for us
\nPseudo labels (PL)
\nTest time augmentation
\nTraining with other bands, frames
\nWeighted loss
\nAdjusting decoder channels
\nVarious other backbones
\nPartial Code available here
\nhttps://github.com/furu-kaggle/ICRGW
A big thank you to my amazing team mates @ragnar123 @optimo and @kunihikofurugori 🙏
", "rawMarkdown": "Thank you Kaggle and Google Research for hosting this competition. \n\nHere's our solution in brief. \n\n**Preprocessing / Dataset**\n- 384 and 512 Image size - Bilinear resize images \n\n- Augmentations (Following two slightly different augmentations were part of our selected ensemble:) \n```\ndef train_transform():\n transform = albu.Compose(\n [ \n albu.OneOf([\n albu.HorizontalFlip(p = 1.0),\n albu.VerticalFlip(p = 1.0),\n ], p = 0.50),\n albu.Rotate(limit = 180, p = 0.50),\n albu.Transpose(p = 0.50),\n albu.Normalize(\n mean = (0.485, 0.456, 0.406),\n std = (0.229, 0.224, 0.225),\n max_pixel_value = 1.0\n ),\n ToTensorV2(transpose_mask = True),\n ],\n p = 1, is_check_shapes=False\n )\n return transform\n```\n```\nself.transform = {\n \"train\": A.Compose([\n A.ShiftScaleRotate(scale_limit=0.20, rotate_limit=0, shift_limit=0.1, p=0.25, border_mode=cv2.BORDER_CONSTANT, value=0),\n A.GridDistortion(p=0.25),\n A.HorizontalFlip(p=0.25),\n A.VerticalFlip(p=0.25),\n A.Resize(*CFG.img_size),\n A.RandomCrop(*CFG.crop_size),\n A.Normalize(mean=CFG.pp_params[\"mean\"],std=CFG.pp_params[\"std\"]),\n A.pytorch.transforms.ToTensorV2(transpose_mask=True)\n ], p=1.0),\n }\n```\n\n**Validation** (Two different CV schemes in ensemble) \n\n- 4 fold CV with geographical binning i.e similar images were kept in same fold\n Also sampling with no duplicates (excerpt from dataset class)\n```\nself.dup_ids = self.df.dup_id.unique()\ndef __getitem__(self, idx):\n dup_idx = self.dup_ids[idx]\n rows = self.df[self.df.dup_id == dup_idx]\n …\n```\n\n- Random 5 fold split on training and given validation dataset. \n\n**Things that worked**\n\n - Soft labels - Instead of using human_pixel_masks, we used average of human_individual_masks from the training set and for validation we kept human_pixel_masks. \n - Loss function - We used SoftBCEWithLogitsLoss with soft labels. Also worked combination of dice loss and BCE loss \n - Usual Ash color images\n\n**Models (UNet)** \n[](https://postimg.cc/rKVRrBfX)\nCosine scheduler with warm up = 0.02 or 0.03 for all models\n\n**Thresholds**\nA threshold change could have landed us in the money zone. After the deadline submission, threshold of 0.45 was best with pvt. 0.713x and public 0.712x and we selected 0.5 threshold. \n\n**Things that didn’t work for us**\n\nPseudo labels (PL)\n\nTest time augmentation\n\nTraining with other bands, frames\n\nWeighted loss\n\nAdjusting decoder channels \n\nVarious other backbones \n\n\n**Partial Code available here** \n[](url)[https://github.com/furu-kaggle/ICRGW](https://github.com/furu-kaggle/ICRGW)\n\n---\nA big thank you to my amazing team mates @ragnar123 @optimo and @kunihikofurugori 🙏\n", "votes": 28 }, { "id": 2386326, "postDate": "2023-08-11T21:22:35.380Z", "content": "Thanks for the detailed configuration and scores for various models. Since I had trouble training large models, I'll retry if I can reach your scores. There must be professional art (probably in the augmentation), difficult to imitate, to train efficientnet-b7 for 100 epochs and reach potential gold.
\nWhat is your final prediction? If you are using weighted mean, you can tune the threshold together with the linear coefficients.
", "rawMarkdown": "Thanks for the detailed configuration and scores for various models. Since I had trouble training large models, I'll retry if I can reach your scores. There must be professional art (probably in the augmentation), difficult to imitate, to train efficientnet-b7 for 100 epochs and reach potential gold.\n\nWhat is your final prediction? If you are using weighted mean, you can tune the threshold together with the linear coefficients.\n\n", "votes": 1, "replies": [ { "id": 2386373, "postDate": "2023-08-11T22:24:03.457Z", "content": "Thank you for your comment and Congratulations on 1st prize !
\nWe trained B7 together as a team and that too in last 1-2 days. LR was important parameter here.
\nWe missed 0.5 pixel shift, I guess it would take fewer epochs with correct pixel shift.
\nAll we needed was to set threshold to 0.4 instead of 0.5 to be in the money/gold.
![\"contrails2.jpg\"](\"https://i.postimg.cc/RhCwq47m/contrails2.jpg\")
\"We missed 0.5 pixel shift,\"
\nso even without the mask alignment, it still work?
\nor becuase the use of soft labels corrected that error?
i hope to know the improvement if you add the \"0.5 pixel shift\"
\nThanks and congrats to your good work!
", "rawMarkdown": "\"We missed 0.5 pixel shift,\"\n\nso even without the mask alignment, it still work?\nor becuase the use of soft labels corrected that error?\n\ni hope to know the improvement if you add the \"0.5 pixel shift\"\n\nThanks and congrats to your good work!", "votes": 1 }, { "id": 2390139, "postDate": "2023-08-14T12:00:23.380Z", "content": "Hi @hengck23,
\nThanks for your comment, prompted me to run an experiment.
\nI decided to run one fold of timm-efficientnet-b7 for comparison with and without pixel correction (no other change in code except the pixel correction part)
Results with pixel correction
\ncv: 0.7087 private: 0.69119 public: 0.67749 (Best score @ epoch 92)
Previous timm-eff-b7 without pixel correction:
\ncv: 0.7067 private: 0.69131 public: 0.69833 (Best score @ epoch 89)
Just with this change, there isn't significant difference at least for this one, @optimo might have more to say with another experiment
", "rawMarkdown": "Hi @hengck23, \nThanks for your comment, prompted me to run an experiment.\nI decided to run one fold of timm-efficientnet-b7 for comparison with and without pixel correction (no other change in code except the pixel correction part)\n\n1. Results with pixel correction\ncv: 0.7087 private: 0.69119 public: 0.67749 (Best score @ epoch 92)\n \n2. Previous timm-eff-b7 without pixel correction: \ncv: 0.7067 private: 0.69131 public: 0.69833 (Best score @ epoch 89)\n\nJust with this change, there isn't significant difference at least for this one, @optimo might have more to say with another experiment", "votes": 2 }, { "id": 2390456, "postDate": "2023-08-14T15:11:56.263Z", "content": "I did a simple experiment with the key learnings from other write-ups:
\nimage = scipy.ndimage.shift(image, (0.5, 0.5, 0))1 single fold trained on full training data : validation score (on validation data) without TTA 0.690, with 4xTTA 0.696, public LB with 4xTTA 0.7033 public LB with 4xTTA 0.70886
\nI had tried all those techniques before but never with the shift, I think it really helps for the convergence and training of the model.
\nI think single model single fold gold medal was feasible once you had all the correct ingredients.
", "rawMarkdown": "I did a simple experiment with the key learnings from other write-ups:\n- shift the image at 256 resolution by 0.5 pixels : `image = scipy.ndimage.shift(image, (0.5, 0.5, 0))`\n- 40 epochs training (instead of 100) with rotations augmentations\n- remove last upscale block in decoder so that input 512 yields output 256\n- b7 backbone\n- mean of individual targets as target\n\n1 single fold trained on full training data : validation score (on validation data) without TTA 0.690, with 4xTTA 0.696, public LB with 4xTTA 0.7033 public LB with 4xTTA 0.70886\n\nI had tried all those techniques before but never with the shift, I think it really helps for the convergence and training of the model.\n\nI think single model single fold gold medal was feasible once you had all the correct ingredients.", "votes": 5 } ] }, { "id": 2386398, "postDate": "2023-08-11T23:01:08.440Z", "content": "Actually the final solution to train correctly the efficientnet b7 is remarkably simple. Kudos to @ragnar123 who ended up with this training code.
", "rawMarkdown": "Actually the final solution to train correctly the efficientnet b7 is remarkably simple. Kudos to @ragnar123 who ended up with this training code.", "votes": 3 } ] }, { "id": 2390259, "postDate": "2023-08-14T13:16:19.603Z", "content": "Good to hear, keep it up.
", "rawMarkdown": "Good to hear, keep it up." }, { "id": 2385345, "postDate": "2023-08-11T09:24:04.470Z", "content": "Great Writeup Thanks
", "rawMarkdown": "Great Writeup Thanks", "votes": 2 } ], "comments": [ { "id": 2386326, "author_name": "🐢 Jun Koda", "author_url": "", "post_date": "2023-08-11T21:22:35.380000", "content": "Thanks for the detailed configuration and scores for various models. Since I had trouble training large models, I'll retry if I can reach your scores. There must be professional art (probably in the augmentation), difficult to imitate, to train efficientnet-b7 for 100 epochs and reach potential gold.
\nWhat is your final prediction? If you are using weighted mean, you can tune the threshold together with the linear coefficients.
", "votes": 1, "replies": [ { "id": 2386373, "author_name": "RB", "author_url": "", "post_date": "2023-08-11T22:24:03.457000", "content": "Thank you for your comment and Congratulations on 1st prize !
\nWe trained B7 together as a team and that too in last 1-2 days. LR was important parameter here.
\nWe missed 0.5 pixel shift, I guess it would take fewer epochs with correct pixel shift.
\nAll we needed was to set threshold to 0.4 instead of 0.5 to be in the money/gold.
\"We missed 0.5 pixel shift,\"
\nso even without the mask alignment, it still work?
\nor becuase the use of soft labels corrected that error?
i hope to know the improvement if you add the \"0.5 pixel shift\"
\nThanks and congrats to your good work!
", "votes": 1, "replies": [] }, { "id": 2390139, "author_name": "RB", "author_url": "", "post_date": "2023-08-14T12:00:23.380000", "content": "Hi @hengck23,
\nThanks for your comment, prompted me to run an experiment.
\nI decided to run one fold of timm-efficientnet-b7 for comparison with and without pixel correction (no other change in code except the pixel correction part)
Results with pixel correction
\ncv: 0.7087 private: 0.69119 public: 0.67749 (Best score @ epoch 92)
Previous timm-eff-b7 without pixel correction:
\ncv: 0.7067 private: 0.69131 public: 0.69833 (Best score @ epoch 89)
Just with this change, there isn't significant difference at least for this one, @optimo might have more to say with another experiment
", "votes": 2, "replies": [] }, { "id": 2390456, "author_name": "Optimo", "author_url": "", "post_date": "2023-08-14T15:11:56.263000", "content": "I did a simple experiment with the key learnings from other write-ups:
\nimage = scipy.ndimage.shift(image, (0.5, 0.5, 0))1 single fold trained on full training data : validation score (on validation data) without TTA 0.690, with 4xTTA 0.696, public LB with 4xTTA 0.7033 public LB with 4xTTA 0.70886
\nI had tried all those techniques before but never with the shift, I think it really helps for the convergence and training of the model.
\nI think single model single fold gold medal was feasible once you had all the correct ingredients.
", "votes": 5, "replies": [] } ] }, { "id": 2386398, "author_name": "Optimo", "author_url": "", "post_date": "2023-08-11T23:01:08.440000", "content": "Actually the final solution to train correctly the efficientnet b7 is remarkably simple. Kudos to @ragnar123 who ended up with this training code.
", "votes": 3, "replies": [] } ] }, { "id": 2390259, "author_name": "shakaut hassain⚡", "author_url": "", "post_date": "2023-08-14T13:16:19.603000", "content": "Good to hear, keep it up.
", "votes": 0, "replies": [] }, { "id": 2385345, "author_name": "AK", "author_url": "", "post_date": "2023-08-11T09:24:04.470000", "content": "Great Writeup Thanks
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "2385316": "Thank you Kaggle and Google Research for hosting this competition. \n\nHere's our solution in brief. \n\n**Preprocessing / Dataset**\n- 384 and 512 Image size - Bilinear resize images \n\n- Augmentations (Following two slightly different augmentations were part of our selected ensemble:) \n```\ndef train_transform():\n transform = albu.Compose(\n [ \n albu.OneOf([\n albu.HorizontalFlip(p = 1.0),\n albu.VerticalFlip(p = 1.0),\n ], p = 0.50),\n albu.Rotate(limit = 180, p = 0.50),\n albu.Transpose(p = 0.50),\n albu.Normalize(\n mean = (0.485, 0.456, 0.406),\n std = (0.229, 0.224, 0.225),\n max_pixel_value = 1.0\n ),\n ToTensorV2(transpose_mask = True),\n ],\n p = 1, is_check_shapes=False\n )\n return transform\n```\n```\nself.transform = {\n \"train\": A.Compose([\n A.ShiftScaleRotate(scale_limit=0.20, rotate_limit=0, shift_limit=0.1, p=0.25, border_mode=cv2.BORDER_CONSTANT, value=0),\n A.GridDistortion(p=0.25),\n A.HorizontalFlip(p=0.25),\n A.VerticalFlip(p=0.25),\n A.Resize(*CFG.img_size),\n A.RandomCrop(*CFG.crop_size),\n A.Normalize(mean=CFG.pp_params[\"mean\"],std=CFG.pp_params[\"std\"]),\n A.pytorch.transforms.ToTensorV2(transpose_mask=True)\n ], p=1.0),\n }\n```\n\n**Validation** (Two different CV schemes in ensemble) \n\n- 4 fold CV with geographical binning i.e similar images were kept in same fold\n Also sampling with no duplicates (excerpt from dataset class)\n```\nself.dup_ids = self.df.dup_id.unique()\ndef __getitem__(self, idx):\n dup_idx = self.dup_ids[idx]\n rows = self.df[self.df.dup_id == dup_idx]\n …\n```\n\n- Random 5 fold split on training and given validation dataset. \n\n**Things that worked**\n\n - Soft labels - Instead of using human_pixel_masks, we used average of human_individual_masks from the training set and for validation we kept human_pixel_masks. \n - Loss function - We used SoftBCEWithLogitsLoss with soft labels. Also worked combination of dice loss and BCE loss \n - Usual Ash color images\n\n**Models (UNet)** \n[](https://postimg.cc/rKVRrBfX)\nCosine scheduler with warm up = 0.02 or 0.03 for all models\n\n**Thresholds**\nA threshold change could have landed us in the money zone. After the deadline submission, threshold of 0.45 was best with pvt. 0.713x and public 0.712x and we selected 0.5 threshold. \n\n**Things that didn’t work for us**\n\nPseudo labels (PL)\n\nTest time augmentation\n\nTraining with other bands, frames\n\nWeighted loss\n\nAdjusting decoder channels \n\nVarious other backbones \n\n\n**Partial Code available here** \n[](url)[https://github.com/furu-kaggle/ICRGW](https://github.com/furu-kaggle/ICRGW)\n\n---\nA big thank you to my amazing team mates @ragnar123 @optimo and @kunihikofurugori 🙏\n", "2386326": "Thanks for the detailed configuration and scores for various models. Since I had trouble training large models, I'll retry if I can reach your scores. There must be professional art (probably in the augmentation), difficult to imitate, to train efficientnet-b7 for 100 epochs and reach potential gold.\n\nWhat is your final prediction? If you are using weighted mean, you can tune the threshold together with the linear coefficients.\n\n", "2390259": "Good to hear, keep it up.", "2385345": "Great Writeup Thanks" } }