![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F99cce57c10157369fcc5c263ebfff4be%2Fcontrail.drawio.png?generation=1691723146332719&alt=media\")
Business context: https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/overview
\nData context: https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/data
My approach is simplified version of the 3D model found in the author's paper [1].
\nFirst, I hypothesized that features on earlier layers are not contributed much to the final prediction, since contrails are drifted from frames to frames. So I come up with the idea of only applying temporal modulation in the later layers of feature maps.
\nI tried some experiments with applying temporal modulation to earlier layers, and found only applying last layer gives the best performance. Note that the rest of the feature maps are just sliced by the current time frame (T=4).
\n
In order for faster loading and saving disk space, I quantized false-color image to uint8.
I did it based on the assumption that 'if humans label at uint8 resolution, then training an AI model at that resolution would be sufficient'.
As other participants are already discussed in publicly, any kind of geometric augmentations fails to improve models' performance (after competition, it turns ouf to be because of label misalignment though).
\nSo I just applied light augmentations of RandomResizedCrop and HorizontalFlip.
cfg.geometric_transform = A.Compose(\n [\n A.RandomResizedCrop(height=, width=, scale=(, ), p=),\n A.HorizontalFlip(p=),\n ]\n)\nMy CV strategy is very naive. I only trained by images on training frames and evaluated on validation images.
In the final submission, it might have been advisable to use all the images for training or to take the average across folds, but considering the remaining time until the competition's end, it was not realistic, so I gave up.
\nThe ensemble policy is:
\nhflip (x2)I tested several backbones including recently published ones, and find EfficientNet and PVTv2 as ensemble seeds have good training time & performance tradeoff.
\nThe final submission was composed of 24 models in total.
\n| backbone | \nused time frames | \ninput resolution | \nTTA | \nrandom seed | \n
|---|---|---|---|---|
| efficientnet_b3 | \n[1, 2, 3, 4] | \n768 | \nhflip (x2) | \nx4 | \n
| pvt_v2_b3 | \n[1, 2, 3, 4] | \n640 | \nhflip (x2) | \nx4 | \n
| pvt_v2_b5 | \n[1, 2, 3, 4] | \n512 | \nhflip (x2) | \nx4 | \n
The CV score on validation set is 0.692, and private LB score is 0.700.
\nIn my moderate machine environment (RTX 3090 Ti x1), Training on higher resolution tend to get CUDA memory overflow error, or result in too small batch sizes. So I used gradient check-pointing and FP16 training to overcome this issue.
\nUsing metadata, I found the geometric distribution of contrails are very different between training and validation sets.
\nAs the authors state in paper [1], I believe there are geographical observation points that is only appeared in the training data, and not included in the validation. I also implemented a fold split based on geographical distribution, but I was unable to effectively utilize this information.
\n\n\nTo further boost the number of positives in the dataset, we also included some GOES-16 ABI imagery at locations in the US where Google Street View images of the sky contained contrails.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fca513333ca52eca20a62c9ed0a5cc082%2Fgeometric_distributions.png?generation=1691723168388936&alt=media\")
Thank you for holding this competition. In addition to tackling an interesting challenge, I was able to review and recap all the methods I used in previous CV competitions.
\nIn late submission, I tested how misalignment of labels affect the score.
\nI also tested the effect of soft labels as some participants said effective.
To reduce training cost, I simply tested on the basic 2D U-Net architecture where only current frame is used, and no temporal feature modulator.
\nOther setups are like these:
| Parameter | \nValue | \n
|---|---|
| backend | \nefficientnet_b3 | \n
| crop_size | \n512x512 | \n
| loss | \nBCE(pos_weight=8) + Dice | \n
| augmentation | \nShiftScaleRotate(p=0.6), hflip | \n
| ensemble | \n4x seed ensemble | \n
| TTA | \nhflip | \n
Since label was misaligned by (0.5, 0.5) pixels, I upscale the input image by 2x, and shifted (1, 1) pixels. It is simpler than correcting labels on training time and inverse transform them on inference time.
\nCorrection was implemented by the below algorithm.
\nNote that additional shift coefficient (0.5, 0.5) is added to transformation matrix as a workaround of the open issue of OpenCV's warpAffine function[2].
M = np.array([[, , + ], [, , + ]], dtype=np.float32)\nimg = cv2.warpAffine(\n img, M, ( * W, * H), flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT\n)\nSoft labels are generated from human_individual_masks.npy.
\nSince the data tab on the competition page saids:
\n\nPixels were considered a contrail when >50% of the labelers annotated it as such. Individual annotations (
\nhuman_individual_masks.npy) as well as the aggregated ground truth annotations (human_pixel_masks.npy) are included in the training data.
I created the soft labels in the below processing:
\nhuman_pixel_mask = np.clip(individual_pixel_mask.mean(-) * , , )\nI trained three types of models and tested on validation data (CV) and leader board(Public and Private):
\nThe result shows both MC and SL has independent positive gains by 0.59% and 0.70% respectively on the private LBs.
\n| description | \nensemble seeds | \ncrop_size | \nCV | \nPublic | \nPrivate | \ngain | \n
|---|---|---|---|---|---|---|
| baseline | \n4x | \n512 | \n0.6672 | \n0.67818 | \n0.67669 | \n- | \n
| +MC | \n4x | \n512 | \n0.6708 | \n0.68131 | \n0.68259 | \n0.59% | \n
| +MC +SL | \n4x | \n512 | \n0.6766 | \n0.69623 | \n0.68956 | \n1.29% | \n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F85d08acac7f886c3270dd1da47992811%2FEffect%20of%20misalignment%20correction%20%20soft%20labels.png?generation=1691897900692345&alt=media\")
Business context: https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/overview
\nData context: https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/data
My approach is simplified version of the 3D model found in the author's paper [1].
\nFirst, I hypothesized that features on earlier layers are not contributed much to the final prediction, since contrails are drifted from frames to frames. So I come up with the idea of only applying temporal modulation in the later layers of feature maps.
\nI tried some experiments with applying temporal modulation to earlier layers, and found only applying last layer gives the best performance. Note that the rest of the feature maps are just sliced by the current time frame (T=4).
\n
In order for faster loading and saving disk space, I quantized false-color image to uint8.
I did it based on the assumption that 'if humans label at uint8 resolution, then training an AI model at that resolution would be sufficient'.
As other participants are already discussed in publicly, any kind of geometric augmentations fails to improve models' performance (after competition, it turns ouf to be because of label misalignment though).
\nSo I just applied light augmentations of RandomResizedCrop and HorizontalFlip.
cfg.geometric_transform = A.Compose(\n [\n A.RandomResizedCrop(height=, width=, scale=(, ), p=),\n A.HorizontalFlip(p=),\n ]\n)\nMy CV strategy is very naive. I only trained by images on training frames and evaluated on validation images.
In the final submission, it might have been advisable to use all the images for training or to take the average across folds, but considering the remaining time until the competition's end, it was not realistic, so I gave up.
\nThe ensemble policy is:
\nhflip (x2)I tested several backbones including recently published ones, and find EfficientNet and PVTv2 as ensemble seeds have good training time & performance tradeoff.
\nThe final submission was composed of 24 models in total.
\n| backbone | \nused time frames | \ninput resolution | \nTTA | \nrandom seed | \n
|---|---|---|---|---|
| efficientnet_b3 | \n[1, 2, 3, 4] | \n768 | \nhflip (x2) | \nx4 | \n
| pvt_v2_b3 | \n[1, 2, 3, 4] | \n640 | \nhflip (x2) | \nx4 | \n
| pvt_v2_b5 | \n[1, 2, 3, 4] | \n512 | \nhflip (x2) | \nx4 | \n
The CV score on validation set is 0.692, and private LB score is 0.700.
\nIn my moderate machine environment (RTX 3090 Ti x1), Training on higher resolution tend to get CUDA memory overflow error, or result in too small batch sizes. So I used gradient check-pointing and FP16 training to overcome this issue.
\nUsing metadata, I found the geometric distribution of contrails are very different between training and validation sets.
\nAs the authors state in paper [1], I believe there are geographical observation points that is only appeared in the training data, and not included in the validation. I also implemented a fold split based on geographical distribution, but I was unable to effectively utilize this information.
\n\n\nTo further boost the number of positives in the dataset, we also included some GOES-16 ABI imagery at locations in the US where Google Street View images of the sky contained contrails.
\n
Thank you for holding this competition. In addition to tackling an interesting challenge, I was able to review and recap all the methods I used in previous CV competitions.
\nIn late submission, I tested how misalignment of labels affect the score.
\nI also tested the effect of soft labels as some participants said effective.
To reduce training cost, I simply tested on the basic 2D U-Net architecture where only current frame is used, and no temporal feature modulator.
\nOther setups are like these:
| Parameter | \nValue | \n
|---|---|
| backend | \nefficientnet_b3 | \n
| crop_size | \n512x512 | \n
| loss | \nBCE(pos_weight=8) + Dice | \n
| augmentation | \nShiftScaleRotate(p=0.6), hflip | \n
| ensemble | \n4x seed ensemble | \n
| TTA | \nhflip | \n
Since label was misaligned by (0.5, 0.5) pixels, I upscale the input image by 2x, and shifted (1, 1) pixels. It is simpler than correcting labels on training time and inverse transform them on inference time.
\nCorrection was implemented by the below algorithm.
\nNote that additional shift coefficient (0.5, 0.5) is added to transformation matrix as a workaround of the open issue of OpenCV's warpAffine function[2].
M = np.array([[, , + ], [, , + ]], dtype=np.float32)\nimg = cv2.warpAffine(\n img, M, ( * W, * H), flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT\n)\nSoft labels are generated from human_individual_masks.npy.
\nSince the data tab on the competition page saids:
\n\nPixels were considered a contrail when >50% of the labelers annotated it as such. Individual annotations (
\nhuman_individual_masks.npy) as well as the aggregated ground truth annotations (human_pixel_masks.npy) are included in the training data.
I created the soft labels in the below processing:
\nhuman_pixel_mask = np.clip(individual_pixel_mask.mean(-) * , , )\nI trained three types of models and tested on validation data (CV) and leader board(Public and Private):
\nThe result shows both MC and SL has independent positive gains by 0.59% and 0.70% respectively on the private LBs.
\n| description | \nensemble seeds | \ncrop_size | \nCV | \nPublic | \nPrivate | \ngain | \n
|---|---|---|---|---|---|---|
| baseline | \n4x | \n512 | \n0.6672 | \n0.67818 | \n0.67669 | \n- | \n
| +MC | \n4x | \n512 | \n0.6708 | \n0.68131 | \n0.68259 | \n0.59% | \n
| +MC +SL | \n4x | \n512 | \n0.6766 | \n0.69623 | \n0.68956 | \n1.29% | \n
Interesting!
\nHave you considered frame t=4 as skip connection to the decoder?
", "rawMarkdown": "Interesting!\n\nHave you considered frame t=4 as skip connection to the decoder? ", "votes": 1, "replies": [ { "id": 2385527, "postDate": "2023-08-11T11:55:49.463Z", "content": "\n\n\nHave you considered frame t=4 as skip connection to the decoder?
\n
Well, it was once in my mind, but never tested. I believe it would stabilize training especially on deeper modulator (in my case, N=3). I will put this on stock of my arsenal. Thank you for pointing this out.
", "rawMarkdown": "@mayurimk\n\n> Have you considered frame t=4 as skip connection to the decoder?\n\nWell, it was once in my mind, but never tested. I believe it would stabilize training especially on deeper modulator (in my case, N=3). I will put this on stock of my arsenal. Thank you for pointing this out." } ] }, { "id": 2385324, "postDate": "2023-08-11T09:13:44.223Z", "content": "Many congratulations on the solo sliver and thank you for the great write up!
\nCould you please clarify this part:
\n\n", "rawMarkdown": "Many congratulations on the solo sliver and thank you for the great write up!\n\nCould you please clarify this part:\n>Temporal Feature Modulator\n\n>First, I hypothesized that features on earlier layers are not contributed much to the final prediction, since contrails are drifted from frames to frames. So I come up with the idea of only applying temporal modulation in the later layers of feature maps.\n>\nI tried some experiments with applying temporal modulation to earlier layers, and found only applying last layer gives the best performance. Note that the rest of the feature maps are just sliced by the current time frame (T=4).", "votes": 1, "replies": [ { "id": 2385347, "postDate": "2023-08-11T09:25:20.130Z", "content": "Temporal Feature Modulator
\nFirst, I hypothesized that features on earlier layers are not contributed much to the final prediction, since contrails are drifted from frames to frames. So I come up with the idea of only applying temporal modulation in the later layers of feature maps.
\nI tried some experiments with applying temporal modulation to earlier layers, and found only applying last layer gives the best performance. Note that the rest of the feature maps are just sliced by the current time frame (T=4).
\n
@tonymarkchris
\nDoes this helps?
():\n cfg = self.cfg\n\n crop_size = cfg.crop_size self.training cfg.infer_crop_size\n x = batch[]\n B, C, T, H, W = x.shape\n\n x = rearrange(x, )\n (H != crop_size) (W != crop_size):\n x = F.interpolate(x, size=(crop_size, crop_size), mode=, align_corners=)\n\n features = self.encoder(x)\n i ((features[:-])):\n x = rearrange(features[i], , t=T)\n x = x[:, :, self.current_frame_relative, :, :]\n features[i] = x\n\n x = features[-]\n x = rearrange(x, , t=T)\n x = self.temporal_encoder(x)\n x = x[:, :, self.current_frame_relative - cfg.temporal_offset, :, :]\n features[-] = x\n\n decoder_output = self.decoder(*features)\n logit = self.segmentation_head(decoder_output)\n\n _, _, h, w = logit.shape\n (h != H) (w != W):\n logit = F.interpolate(logit, size=(H, W), mode=, align_corners=)\n\n output = (logit=logit)\n\n output\nthanks for sharing, can u also share the encoder code snippet?
", "rawMarkdown": "thanks for sharing, can u also share the encoder code snippet?" }, { "id": 2402194, "postDate": "2023-08-22T03:49:30.203Z", "content": "Well, it's nothing special. It is just as I wrote on the block diagram (3 layers of Conv3D + BN + ReLU). I think you can implement it by yourself.
", "rawMarkdown": "Well, it's nothing special. It is just as I wrote on the block diagram (3 layers of Conv3D + BN + ReLU). I think you can implement it by yourself." } ] } ] }, { "id": 2398367, "postDate": "2023-08-19T16:17:41.670Z", "content": "Congratulations and thanks for sharing such a detailed writeup!
\nDo you know why HFlip worked for you even though you didn't correct label misalignment?
Thanks. Well, actually, I didn’t conducted detailed comparison (which I should have done). However, I believe augmentation in this competition (without misalignment correction) has both positive and negative effect.
\nIn the negative effect, v/h flips and rotation surely introduces label noise, but after enough steps of training, the model would learn “expected contrail label” which is close to mean of flipped/rotated contrails.
\nOn the positive effect, the augmentation would inject model with certain amount of generalization as in the general context of machine learning. Additionally, the larger model with time frame information might even have chance to learn flipped/rotated direction, where the model could learn shifted labels correctly.
\nWhat discussed above is just hypothesis, but considering some top solution says they conducted intense augmentation with larger models, I believe my hypothesis is somehow true.
", "rawMarkdown": "Thanks. Well, actually, I didn’t conducted detailed comparison (which I should have done). However, I believe augmentation in this competition (without misalignment correction) has both positive and negative effect.\n\nIn the negative effect, v/h flips and rotation surely introduces label noise, but after enough steps of training, the model would learn “expected contrail label” which is close to mean of flipped/rotated contrails.\n\nOn the positive effect, the augmentation would inject model with certain amount of generalization as in the general context of machine learning. Additionally, the larger model with time frame information might even have chance to learn flipped/rotated direction, where the model could learn shifted labels correctly.\n\nWhat discussed above is just hypothesis, but considering some top solution says they conducted intense augmentation with larger models, I believe my hypothesis is somehow true." } ] }, { "id": 2388440, "postDate": "2023-08-13T11:43:26.357Z", "content": "Hi @tatamikenn Thank you for sharing your solution and after deadline experiments
\nI'm wondering why multiply *2 here
\nhuman_pixel_mask = np.clip(individual_pixel_mask.mean(-1) * 2, 0, 1)
@RB
\nBecause I want to make the pixel value greater than 0.5 makes 1 (not 0.5). However, I believe it's not big difference to simply taking average, since setting proper threshold should gave us equivalent result.
Interesting!
\nHave you considered frame t=4 as skip connection to the decoder?
", "votes": 1, "replies": [ { "id": 2385527, "author_name": "Bilzard", "author_url": "", "post_date": "2023-08-11T11:55:49.463000", "content": "\n\n\nHave you considered frame t=4 as skip connection to the decoder?
\n
Well, it was once in my mind, but never tested. I believe it would stabilize training especially on deeper modulator (in my case, N=3). I will put this on stock of my arsenal. Thank you for pointing this out.
", "votes": 0, "replies": [] } ] }, { "id": 2385324, "author_name": "DJ_Xia", "author_url": "", "post_date": "2023-08-11T09:13:44.223000", "content": "Many congratulations on the solo sliver and thank you for the great write up!
\nCould you please clarify this part:
\n\n", "votes": 1, "replies": [ { "id": 2385347, "author_name": "Bilzard", "author_url": "", "post_date": "2023-08-11T09:25:20.130000", "content": "Temporal Feature Modulator
\nFirst, I hypothesized that features on earlier layers are not contributed much to the final prediction, since contrails are drifted from frames to frames. So I come up with the idea of only applying temporal modulation in the later layers of feature maps.
\nI tried some experiments with applying temporal modulation to earlier layers, and found only applying last layer gives the best performance. Note that the rest of the feature maps are just sliced by the current time frame (T=4).
\n
@tonymarkchris
\nDoes this helps?
():\n cfg = self.cfg\n\n crop_size = cfg.crop_size self.training cfg.infer_crop_size\n x = batch[]\n B, C, T, H, W = x.shape\n\n x = rearrange(x, )\n (H != crop_size) (W != crop_size):\n x = F.interpolate(x, size=(crop_size, crop_size), mode=, align_corners=)\n\n features = self.encoder(x)\n i ((features[:-])):\n x = rearrange(features[i], , t=T)\n x = x[:, :, self.current_frame_relative, :, :]\n features[i] = x\n\n x = features[-]\n x = rearrange(x, , t=T)\n x = self.temporal_encoder(x)\n x = x[:, :, self.current_frame_relative - cfg.temporal_offset, :, :]\n features[-] = x\n\n decoder_output = self.decoder(*features)\n logit = self.segmentation_head(decoder_output)\n\n _, _, h, w = logit.shape\n (h != H) (w != W):\n logit = F.interpolate(logit, size=(H, W), mode=, align_corners=)\n\n output = (logit=logit)\n\n output\nthanks for sharing, can u also share the encoder code snippet?
", "votes": 0, "replies": [] }, { "id": 2402194, "author_name": "Bilzard", "author_url": "", "post_date": "2023-08-22T03:49:30.203000", "content": "Well, it's nothing special. It is just as I wrote on the block diagram (3 layers of Conv3D + BN + ReLU). I think you can implement it by yourself.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2398367, "author_name": "delai50", "author_url": "", "post_date": "2023-08-19T16:17:41.670000", "content": "Congratulations and thanks for sharing such a detailed writeup!
\nDo you know why HFlip worked for you even though you didn't correct label misalignment?
Thanks. Well, actually, I didn’t conducted detailed comparison (which I should have done). However, I believe augmentation in this competition (without misalignment correction) has both positive and negative effect.
\nIn the negative effect, v/h flips and rotation surely introduces label noise, but after enough steps of training, the model would learn “expected contrail label” which is close to mean of flipped/rotated contrails.
\nOn the positive effect, the augmentation would inject model with certain amount of generalization as in the general context of machine learning. Additionally, the larger model with time frame information might even have chance to learn flipped/rotated direction, where the model could learn shifted labels correctly.
\nWhat discussed above is just hypothesis, but considering some top solution says they conducted intense augmentation with larger models, I believe my hypothesis is somehow true.
", "votes": 0, "replies": [] } ] }, { "id": 2388440, "author_name": "RB", "author_url": "", "post_date": "2023-08-13T11:43:26.357000", "content": "Hi @tatamikenn Thank you for sharing your solution and after deadline experiments
\nI'm wondering why multiply *2 here
\nhuman_pixel_mask = np.clip(individual_pixel_mask.mean(-1) * 2, 0, 1)
@RB
\nBecause I want to make the pixel value greater than 0.5 makes 1 (not 0.5). However, I believe it's not big difference to simply taking average, since setting proper threshold should gave us equivalent result.