Based on the knowledge shared on public discussion, I want to share the notable results which is obtained by experiments after the competition.
\nI appreciate participants who kindly shared their solutions.
First, my largest interest was
\nThese results are already added to my solution write-up[4], so you can refer to it first if you haven't read.
\nIn addition, I tested performance gain of these architectural choices:
\nIn this article, I discussed the experiment result of 3-5.
\nGiven the constraints of computing resources, all experiments were conducted using a single fold and a single random state. This means the results might not be entirely robust. For more definitive conclusions, multiple trials for each architecture would be necessary.
\nI evaluated various backbones using a simple 2D U-Net architecture that only utilizes the current frame. The models were trained on training images and evaluated on validation images. For all experiments, I employed label correction and soft-labels. The results are presented in Table 1. Among the options, CoaT-lite medium demonstrated the best balance of performance and training time. Consequently, I adopted this backbone as the default for subsequent experiments.
Table 1: CV score of various backbones
\n| backbone | \ncrop_size | \nepochs | \ngradient checkpointing | \nbatch_size | \nCV | \ntraining time/epoch [sec] | \n
|---|---|---|---|---|---|---|
| efficientnet_b3 | \n512 | \n20 | \nFALSE | \n28 | \n0.6629 | \n187 | \n
| maxvit_tiny_tf_512.in1k | \n512 | \n20 | \nFALSE | \n19 | \n0.6725 | \n396 | \n
| maxvit_small_tf_512.in1k | \n512 | \n20 | \nTRUE | \n21 | \n0.672 | \n709 | \n
| pvt_v2_b3 | \n512 | \n20 | \nFALSE | \n20 | \n0.6681 | \n341 | \n
| coat_lite_medium | \n512 | \n20 | \nFALSE | \n20 | \n0.6787 | \n413 | \n
| coat_small | \n512 | \n20 | \nFALSE | \n19 | \n0.6713 | \n513 | \n
PixelShuffle[1] offers an alternative method for up-scaling resolution while reducing feature channels. Following the approach mentioned by the 2nd place team in their solution[2], I implemented the PixelShuffle block as a replacement for the traditional up-scaling block, leading to what I refer to as the PS-U-Net architecture.
\nWhen comparing the U-Net with the PS-U-Net, there wasn't a significant performance gain (CV: 0.6787 to 0.6779). However, the PS-U-Net had the advantage of being more memory-efficient due to the reduced feature channels. Given this efficiency, I chose to use the PS-U-Net as the default architecture for subsequent experiments.
\nThanks to the correction of misalignment, we can now employ a more intensive augmentation strategy, as mentioned in the 1st place solution[3]. I adapted Koda's augmentation approach with slight modifications, as shown below:
\ncfg.geometric_transform = A.Compose(\n [\n A.RandomRotate90(p=),\n A.HorizontalFlip(p=),\n A.ShiftScaleRotate(\n rotate_limit=, scale_limit=, border_mode=cv2.BORDER_CONSTANT, p=\n ),\n ]\n)\nThe differences are:
\nborader_mode as constant in order to avoid unintentional artifactsThe experimental results highlighted the substantial impact of this intensive augmentation. It led to an increase in the CV score by 0.58% (from 0.6779 to 0.6837). This underscores that addressing misalignment is crucial for success in this competition.
\nNext, I transitioned from the PS-U-Net architecture to the PS-FPN, where I replaced the up-scaling block of the FPN with PixelShuffle blocks.
\nThis change resulted in a modest improvement of 0.24% in CV scores (from 0.6837 to 0.6861). However, it's uncertain whether this gain can be solely attributed to the architectural modification. To draw a more definitive conclusion, averaging across multiple seeds would be advisable. Nonetheless, based on these preliminary results, I opted to use the PS-FPN architecture as the standard for subsequent experiments.
\nLastly, I tested the impact of different architectural choice of T-Mixer.
\nEspecially, I wanted to compare which mixer is superior to this task: Convolutions and Transformer. So I compared below three architecture:
\nThe last pattern was just a spontaneous idea I had. However, since this backbone performs best in 2D settings, I believed it might be superior as temporal mixer. In this experiment, I evaluated on CV and LB. Additionally, I employed TTA to ensure robust evaluation.
\nNote that training Transformer and CoaT T-Mixer were unstable, sometimes resulting in NaN predictions. Stability techniques were applied to address this (see Appendix A). In contrast, Conv3d T-Mixer training was stable without any adjustments.
\nTable 2 displays the results. The first row represents a 2D U-Net without a T-Mixer, while the second uses a 2.5D PS-FPN with a Conv3D T-Mixer. Incorporating time frame information improved both CV and Private LB scores significantly. This indicates the value of temporal information even after alignment corrections.
\nTable 2's 3rd and 4th rows show results for Transformer and CoaT T-Mixers. While CoaT T-Mixer had the highest CV score, its private LB score was the lowest. 3D Conv T-Mixer and Transformer T-Mixer had comparable performances.
\nTable 3 presents ensemble model results. The best Private score came from an ensemble of three T-Mixer architectures, suggesting model diversity. Due to resource constraints, only one random seed was tested, but seed averaging or cross-fold averaging might yield better scores.
\nTable2: Comparison of Different T-Mixer Architectures
\n| id | \nused frames | \nT-Mixer | \nseed | \nTTA | \ncrop_size | \nCV | \nPublic | \nPrivate | \n
|---|---|---|---|---|---|---|---|---|
| 1 | \n[4] | \nNone | \nx1 | \nhflip, rot(90, 180, 270) | \n512 | \n0.691 | \n0.70244 | \n0.70462 | \n
| 2 | \n[1, 2, 3, 4] | \n3d Conv | \nx1 | \nhflip, rot(90, 180, 270) | \n512 | \n0.6978 | \n0.7147 | \n0.71681 | \n
| 3 | \n[1, 2, 3, 4] | \nTransformer | \nx1 | \nhflip, rot(90, 180, 270) | \n512 | \n0.6958 | \n0.70576 | \n0.71641 | \n
| 4 | \n[1, 2, 3, 4] | \nCoaT(3D pos enc) | \nx1 | \nhflip, rot(90, 180, 270) | \n512 | \n0.6988 | \n0.71553 | \n0.71229 | \n
Table3: Result of Ensemble
\n| ensemble | \nCV | \nPublic | \nPrivate | \n
|---|---|---|---|
| [2, 4] | \n0.7009 | \n0.718 | \n0.71827 | \n
| [2, 3, 4] | \n0.701 | \n0.71638 | \n0.72156 | \n
During my experiments with the Transformer and CoaT T-Mixer, I noticed numerous NaN values in the model's predictions. To stabilize the training, I implemented several techniques:
\nDespite trying the first two techniques, the issue persisted.
\nIt's worth noting that I consistently used AMP (Automatic Mixed Precision) throughout my experiments. Upon realizing that the issue didn't manifest in FP32 precision, I suspected that an excessively large epsilon might be causing instability, especially since similar issues have been reported with FP16 training[6].
\nIncorporating all these techniques resolved the problem, so I employed them in CoaT and Transformer T-Mixer experiments.
", "messages": [ { "id": 2397864, "postDate": "2023-08-19T09:28:06.487Z", "content": "Based on the knowledge shared on public discussion, I want to share the notable results which is obtained by experiments after the competition.
\nI appreciate participants who kindly shared their solutions.
First, my largest interest was
\nThese results are already added to my solution write-up[4], so you can refer to it first if you haven't read.
\nIn addition, I tested performance gain of these architectural choices:
\nIn this article, I discussed the experiment result of 3-5.
\nGiven the constraints of computing resources, all experiments were conducted using a single fold and a single random state. This means the results might not be entirely robust. For more definitive conclusions, multiple trials for each architecture would be necessary.
\nI evaluated various backbones using a simple 2D U-Net architecture that only utilizes the current frame. The models were trained on training images and evaluated on validation images. For all experiments, I employed label correction and soft-labels. The results are presented in Table 1. Among the options, CoaT-lite medium demonstrated the best balance of performance and training time. Consequently, I adopted this backbone as the default for subsequent experiments.
Table 1: CV score of various backbones
\n| backbone | \ncrop_size | \nepochs | \ngradient checkpointing | \nbatch_size | \nCV | \ntraining time/epoch [sec] | \n
|---|---|---|---|---|---|---|
| efficientnet_b3 | \n512 | \n20 | \nFALSE | \n28 | \n0.6629 | \n187 | \n
| maxvit_tiny_tf_512.in1k | \n512 | \n20 | \nFALSE | \n19 | \n0.6725 | \n396 | \n
| maxvit_small_tf_512.in1k | \n512 | \n20 | \nTRUE | \n21 | \n0.672 | \n709 | \n
| pvt_v2_b3 | \n512 | \n20 | \nFALSE | \n20 | \n0.6681 | \n341 | \n
| coat_lite_medium | \n512 | \n20 | \nFALSE | \n20 | \n0.6787 | \n413 | \n
| coat_small | \n512 | \n20 | \nFALSE | \n19 | \n0.6713 | \n513 | \n
PixelShuffle[1] offers an alternative method for up-scaling resolution while reducing feature channels. Following the approach mentioned by the 2nd place team in their solution[2], I implemented the PixelShuffle block as a replacement for the traditional up-scaling block, leading to what I refer to as the PS-U-Net architecture.
\nWhen comparing the U-Net with the PS-U-Net, there wasn't a significant performance gain (CV: 0.6787 to 0.6779). However, the PS-U-Net had the advantage of being more memory-efficient due to the reduced feature channels. Given this efficiency, I chose to use the PS-U-Net as the default architecture for subsequent experiments.
\nThanks to the correction of misalignment, we can now employ a more intensive augmentation strategy, as mentioned in the 1st place solution[3]. I adapted Koda's augmentation approach with slight modifications, as shown below:
\ncfg.geometric_transform = A.Compose(\n [\n A.RandomRotate90(p=),\n A.HorizontalFlip(p=),\n A.ShiftScaleRotate(\n rotate_limit=, scale_limit=, border_mode=cv2.BORDER_CONSTANT, p=\n ),\n ]\n)\nThe differences are:
\nborader_mode as constant in order to avoid unintentional artifactsThe experimental results highlighted the substantial impact of this intensive augmentation. It led to an increase in the CV score by 0.58% (from 0.6779 to 0.6837). This underscores that addressing misalignment is crucial for success in this competition.
\nNext, I transitioned from the PS-U-Net architecture to the PS-FPN, where I replaced the up-scaling block of the FPN with PixelShuffle blocks.
\nThis change resulted in a modest improvement of 0.24% in CV scores (from 0.6837 to 0.6861). However, it's uncertain whether this gain can be solely attributed to the architectural modification. To draw a more definitive conclusion, averaging across multiple seeds would be advisable. Nonetheless, based on these preliminary results, I opted to use the PS-FPN architecture as the standard for subsequent experiments.
\nLastly, I tested the impact of different architectural choice of T-Mixer.
\nEspecially, I wanted to compare which mixer is superior to this task: Convolutions and Transformer. So I compared below three architecture:
\nThe last pattern was just a spontaneous idea I had. However, since this backbone performs best in 2D settings, I believed it might be superior as temporal mixer. In this experiment, I evaluated on CV and LB. Additionally, I employed TTA to ensure robust evaluation.
\nNote that training Transformer and CoaT T-Mixer were unstable, sometimes resulting in NaN predictions. Stability techniques were applied to address this (see Appendix A). In contrast, Conv3d T-Mixer training was stable without any adjustments.
\nTable 2 displays the results. The first row represents a 2D U-Net without a T-Mixer, while the second uses a 2.5D PS-FPN with a Conv3D T-Mixer. Incorporating time frame information improved both CV and Private LB scores significantly. This indicates the value of temporal information even after alignment corrections.
\nTable 2's 3rd and 4th rows show results for Transformer and CoaT T-Mixers. While CoaT T-Mixer had the highest CV score, its private LB score was the lowest. 3D Conv T-Mixer and Transformer T-Mixer had comparable performances.
\nTable 3 presents ensemble model results. The best Private score came from an ensemble of three T-Mixer architectures, suggesting model diversity. Due to resource constraints, only one random seed was tested, but seed averaging or cross-fold averaging might yield better scores.
\nTable2: Comparison of Different T-Mixer Architectures
\n| id | \nused frames | \nT-Mixer | \nseed | \nTTA | \ncrop_size | \nCV | \nPublic | \nPrivate | \n
|---|---|---|---|---|---|---|---|---|
| 1 | \n[4] | \nNone | \nx1 | \nhflip, rot(90, 180, 270) | \n512 | \n0.691 | \n0.70244 | \n0.70462 | \n
| 2 | \n[1, 2, 3, 4] | \n3d Conv | \nx1 | \nhflip, rot(90, 180, 270) | \n512 | \n0.6978 | \n0.7147 | \n0.71681 | \n
| 3 | \n[1, 2, 3, 4] | \nTransformer | \nx1 | \nhflip, rot(90, 180, 270) | \n512 | \n0.6958 | \n0.70576 | \n0.71641 | \n
| 4 | \n[1, 2, 3, 4] | \nCoaT(3D pos enc) | \nx1 | \nhflip, rot(90, 180, 270) | \n512 | \n0.6988 | \n0.71553 | \n0.71229 | \n
Table3: Result of Ensemble
\n| ensemble | \nCV | \nPublic | \nPrivate | \n
|---|---|---|---|
| [2, 4] | \n0.7009 | \n0.718 | \n0.71827 | \n
| [2, 3, 4] | \n0.701 | \n0.71638 | \n0.72156 | \n
During my experiments with the Transformer and CoaT T-Mixer, I noticed numerous NaN values in the model's predictions. To stabilize the training, I implemented several techniques:
\nDespite trying the first two techniques, the issue persisted.
\nIt's worth noting that I consistently used AMP (Automatic Mixed Precision) throughout my experiments. Upon realizing that the issue didn't manifest in FP32 precision, I suspected that an excessively large epsilon might be causing instability, especially since similar issues have been reported with FP16 training[6].
\nIncorporating all these techniques resolved the problem, so I employed them in CoaT and Transformer T-Mixer experiments.
", "rawMarkdown": "Based on the knowledge shared on public discussion, I want to share the notable results which is obtained by experiments after the competition.\nI appreciate participants who kindly shared their solutions.\n\n## Overview\n\nFirst, my largest interest was\n\n1. How much gain can be obtained by correction of misalignment?\n2. How much gain can be obtained by soft labels?\n\nThese results are already added to my solution write-up[4], so you can refer to it first if you haven't read.\n\nIn addition, I tested performance gain of these architectural choices:\n\n3. backbones\n4. decoders, especially the impact of PixelShuffle\n5. T-Mixer (same as *Temporal Modulator* in my original solution)\n\nIn this article, I discussed the experiment result of 3-5.\n\n## Experiment Detail\n\n### Note\n\nGiven the constraints of computing resources, all experiments were conducted using a single fold and a single random state. This means the results might not be entirely robust. For more definitive conclusions, multiple trials for each architecture would be necessary.\n\n### Choice of backbones\n\nI evaluated various backbones using a simple 2D U-Net architecture that only utilizes the current frame. The models were trained on `training` images and evaluated on `validation` images. For all experiments, I employed label correction and soft-labels. The results are presented in Table 1. Among the options, CoaT-lite medium demonstrated the best balance of performance and training time. Consequently, I adopted this backbone as the default for subsequent experiments.\n\n**Table 1: CV score of various backbones**\n\n| backbone | crop_size | epochs | gradient checkpointing | batch_size | CV | training time/epoch [sec] |\n|-------------------------|-----------|--------|---------------|------------|--------|----------------|\n| efficientnet_b3 | 512 | 20 | FALSE | 28 | 0.6629 | 187 |\n| maxvit_tiny_tf_512.in1k | 512 | 20 | FALSE | 19 | 0.6725 | 396 |\n| maxvit_small_tf_512.in1k| 512 | 20 | TRUE | 21 | 0.672 | 709 |\n| pvt_v2_b3 | 512 | 20 | FALSE | 20 | 0.6681 | 341 |\n| coat_lite_medium | 512 | 20 | FALSE | 20 | 0.6787 | 413 |\n| coat_small | 512 | 20 | FALSE | 19 | 0.6713 | 513 |\n\n\n### Changing Up-scaling to PixelShuffle\n\nPixelShuffle[1] offers an alternative method for up-scaling resolution while reducing feature channels. Following the approach mentioned by the 2nd place team in their solution[2], I implemented the PixelShuffle block as a replacement for the traditional up-scaling block, leading to what I refer to as the PS-U-Net architecture.\n\nWhen comparing the U-Net with the PS-U-Net, there wasn't a significant performance gain (CV: 0.6787 to 0.6779). However, the PS-U-Net had the advantage of being more memory-efficient due to the reduced feature channels. Given this efficiency, I chose to use the PS-U-Net as the default architecture for subsequent experiments.\n\n### The Effects of Intensive Augmentation\n\nThanks to the correction of misalignment, we can now employ a more intensive augmentation strategy, as mentioned in the 1st place solution[3]. I adapted Koda's augmentation approach with slight modifications, as shown below:\n\n\n```python\ncfg.geometric_transform = A.Compose(\n [\n A.RandomRotate90(p=1),\n A.HorizontalFlip(p=0.5),\n A.ShiftScaleRotate(\n rotate_limit=45, scale_limit=0.2, border_mode=cv2.BORDER_CONSTANT, p=0.6\n ),\n ]\n)\n```\n\nThe differences are:\n1. slightly increase probability (0.5 -> 0.6)\n2. set `borader_mode` as constant in order to avoid unintentional artifacts\n\nThe experimental results highlighted the substantial impact of this intensive augmentation. It led to an increase in the CV score by 0.58% (from 0.6779 to 0.6837). This underscores that addressing misalignment is crucial for success in this competition.\n\n### Changing Decoder Architecture: PS-U-Net -> PS-FPN\n\nNext, I transitioned from the PS-U-Net architecture to the PS-FPN, where I replaced the up-scaling block of the FPN with PixelShuffle blocks.\n\nThis change resulted in a modest improvement of 0.24% in CV scores (from 0.6837 to 0.6861). However, it's uncertain whether this gain can be solely attributed to the architectural modification. To draw a more definitive conclusion, averaging across multiple seeds would be advisable. Nonetheless, based on these preliminary results, I opted to use the PS-FPN architecture as the standard for subsequent experiments.\n\n### Comparing T-Mixer Architecture\n\nLastly, I tested the impact of different architectural choice of T-Mixer.\n\nEspecially, I wanted to compare which mixer is superior to this task: Convolutions and Transformer. So I compared below three architecture:\n\n1. Conv3d\n2. Transformer\n3. CoaT with (3D positional encoding)\n\nThe last pattern was just a spontaneous idea I had. However, since this backbone performs best in 2D settings, I believed it might be superior as temporal mixer. In this experiment, I evaluated on CV and LB. Additionally, I employed TTA to ensure robust evaluation.\n\nNote that training Transformer and CoaT T-Mixer were unstable, sometimes resulting in NaN predictions. Stability techniques were applied to address this (see Appendix A). In contrast, Conv3d T-Mixer training was stable without any adjustments.\n\nTable 2 displays the results. The first row represents a 2D U-Net without a T-Mixer, while the second uses a 2.5D PS-FPN with a Conv3D T-Mixer. Incorporating time frame information improved both CV and Private LB scores significantly. This indicates the value of temporal information even after alignment corrections.\n\nTable 2's 3rd and 4th rows show results for Transformer and CoaT T-Mixers. While CoaT T-Mixer had the highest CV score, its private LB score was the lowest. 3D Conv T-Mixer and Transformer T-Mixer had comparable performances.\n\nTable 3 presents ensemble model results. The best Private score came from an ensemble of three T-Mixer architectures, suggesting model diversity. Due to resource constraints, only one random seed was tested, but seed averaging or cross-fold averaging might yield better scores.\n\n\n**Table2: Comparison of Different T-Mixer Architectures**\n\n| id | used frames | T-Mixer | seed | TTA | crop_size | CV | Public | Private |\n|----|----------------|-------------------|------|----------------------------|-----------|--------|---------|---------|\n| 1 | [4] | None | x1 | hflip, rot(90, 180, 270) | 512 | 0.691 | 0.70244 | 0.70462 |\n| 2 | [1, 2, 3, 4] | 3d Conv | x1 | hflip, rot(90, 180, 270) | 512 | 0.6978 | 0.7147 | 0.71681 |\n| 3 | [1, 2, 3, 4] | Transformer | x1 | hflip, rot(90, 180, 270) | 512 | 0.6958 | 0.70576 | 0.71641 |\n| 4 | [1, 2, 3, 4] | CoaT(3D pos enc) | x1 | hflip, rot(90, 180, 270) | 512 | 0.6988 | 0.71553 | 0.71229 |\n\n**Table3: Result of Ensemble**\n\n| ensemble | CV | Public | Private |\n|------------|--------|--------|---------|\n| [2, 4] | 0.7009 | 0.718 | 0.71827 |\n| [2, 3, 4] | 0.701 | 0.71638| 0.72156 |\n\n## References\n\n* [1] [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://arxiv.org/abs/1609.05158v2)\n* [2] https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/430491\n* [3] https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/430618\n* [4] https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/430794\n* [5] [Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://arxiv.org/abs/2004.01461)\n* [6] [Adam+Half Precision = NaNs?](https://discuss.pytorch.org/t/adam-half-precision-nans/1765/5)\n\n## Appendix\n\n### A. Tips to stabilize Transformer/CoaT T-Mixer\n\nDuring my experiments with the Transformer and CoaT T-Mixer, I noticed numerous NaN values in the model's predictions. To stabilize the training, I implemented several techniques:\n\n1. Reduced the learning rate.\n2. Applied Gradient Centralization[5].\n3. Used an epsilon value of 1e-4 wherever feasible.\n\nDespite trying the first two techniques, the issue persisted.\n\nIt's worth noting that I consistently used AMP (Automatic Mixed Precision) throughout my experiments. Upon realizing that the issue didn't manifest in FP32 precision, I suspected that an excessively large epsilon might be causing instability, especially since similar issues have been reported with FP16 training[6].\n\nIncorporating all these techniques resolved the problem, so I employed them in CoaT and Transformer T-Mixer experiments.\n", "votes": 27 }, { "id": 2400844, "postDate": "2023-08-21T09:03:26.763Z", "content": "Awesome stuff. Reproducing what worked for others is no easy task.
", "rawMarkdown": "Awesome stuff. Reproducing what worked for others is no easy task.", "votes": 1, "replies": [ { "id": 2402214, "postDate": "2023-08-22T04:10:06.943Z", "content": "Thanks. I have a lot of inspiration from your team's solution. Thank you for sharing this informative write-up.
", "rawMarkdown": "Thanks. I have a lot of inspiration from your team's solution. Thank you for sharing this informative write-up.", "votes": 1 } ] }, { "id": 2397968, "postDate": "2023-08-19T11:03:06.053Z", "content": "What a great idea and writeup! Thanks!
\nCan you share the code for the 3D Conv T-Mixer part?
Or share the entire code! what a great ablation study! 😀
\n\n\nUsed an epsilon value of 1e-4 wherever feasible
\n
Btw, what epsilon value are you refering to?
", "rawMarkdown": "Or share the entire code! what a great ablation study! 😀\n\n>Used an epsilon value of 1e-4 wherever feasible\n\nBtw, what epsilon value are you refering to?\n", "votes": 1, "replies": [ { "id": 2400019, "postDate": "2023-08-20T18:21:28.793Z", "content": "A higher value of epsilon in AdamW and the like helps with NaNs when doing mixed precision training.
", "rawMarkdown": "A higher value of epsilon in AdamW and the like helps with NaNs when doing mixed precision training.", "votes": 1 }, { "id": 2400024, "postDate": "2023-08-20T18:24:38.367Z", "content": "\n\nOr share the entire code! what a great ablation study! 😀
\n
Yep!
\nI'm asking because I've tried 3d time mixer (just a couple of 3d convs+2d reshape applied to the stride 32 encoder layer) and it was super expensive in terms of memory consumption and speed.
\n\nCan you share the code for the 3D Conv T-Mixer part?
\n
Well, I can share this part, but it's nothing special. its just as I wrote on my original solution's block diagram[4]. I think you can easily implement it by yourself.
\n\n\nOr share the entire code! what a great ablation study! 😀
\n
Sorry, but I am now working on other competition, and I don't have much time for that.
\nYou know, my code is full of external code, and I should follow licenses for each repository.
\n\nI'm asking because I've tried 3d time mixer (just a couple of 3d convs+2d reshape applied to the stride 32 encoder layer) and it was super expensive in terms of memory consumption and speed.
\n
If you have problem with memory consumption, my advice is to use gradient checkpointing[7]. For example, some backbone in timm library already be implemented with this functionality.
\n", "rawMarkdown": "> Can you share the code for the 3D Conv T-Mixer part?\n\nWell, I can share this part, but it's nothing special. its just as I wrote on my original solution's block diagram[4]. I think you can easily implement it by yourself.\n\n> Or share the entire code! what a great ablation study! 😀\n\nSorry, but I am now working on other competition, and I don't have much time for that.\nYou know, my code is full of external code, and I should follow licenses for each repository.\n\n> I'm asking because I've tried 3d time mixer (just a couple of 3d convs+2d reshape applied to the stride 32 encoder layer) and it was super expensive in terms of memory consumption and speed.\n\nIf you have problem with memory consumption, my advice is to use gradient checkpointing[7]. For example, some backbone in timm library already be implemented with this functionality.\n\n- [7] https://pytorch.org/docs/stable/checkpoint.html" } ] } ] } ], "comments": [ { "id": 2400844, "author_name": "Theo Viel", "author_url": "", "post_date": "2023-08-21T09:03:26.763000", "content": "Awesome stuff. Reproducing what worked for others is no easy task.
", "votes": 1, "replies": [ { "id": 2402214, "author_name": "Bilzard", "author_url": "", "post_date": "2023-08-22T04:10:06.943000", "content": "Thanks. I have a lot of inspiration from your team's solution. Thank you for sharing this informative write-up.
", "votes": 1, "replies": [] } ] }, { "id": 2397968, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-08-19T11:03:06.053000", "content": "What a great idea and writeup! Thanks!
\nCan you share the code for the 3D Conv T-Mixer part?
Or share the entire code! what a great ablation study! 😀
\n\n\nUsed an epsilon value of 1e-4 wherever feasible
\n
Btw, what epsilon value are you refering to?
", "votes": 1, "replies": [ { "id": 2400019, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-08-20T18:21:28.793000", "content": "A higher value of epsilon in AdamW and the like helps with NaNs when doing mixed precision training.
", "votes": 1, "replies": [] }, { "id": 2400024, "author_name": "DennisSakva", "author_url": "", "post_date": "2023-08-20T18:24:38.367000", "content": "\n\nOr share the entire code! what a great ablation study! 😀
\n
Yep!
\nI'm asking because I've tried 3d time mixer (just a couple of 3d convs+2d reshape applied to the stride 32 encoder layer) and it was super expensive in terms of memory consumption and speed.
\n\nCan you share the code for the 3D Conv T-Mixer part?
\n
Well, I can share this part, but it's nothing special. its just as I wrote on my original solution's block diagram[4]. I think you can easily implement it by yourself.
\n\n\nOr share the entire code! what a great ablation study! 😀
\n
Sorry, but I am now working on other competition, and I don't have much time for that.
\nYou know, my code is full of external code, and I should follow licenses for each repository.
\n\nI'm asking because I've tried 3d time mixer (just a couple of 3d convs+2d reshape applied to the stride 32 encoder layer) and it was super expensive in terms of memory consumption and speed.
\n
If you have problem with memory consumption, my advice is to use gradient checkpointing[7]. For example, some backbone in timm library already be implemented with this functionality.
\n", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2397864": "Based on the knowledge shared on public discussion, I want to share the notable results which is obtained by experiments after the competition.\nI appreciate participants who kindly shared their solutions.\n\n## Overview\n\nFirst, my largest interest was\n\n1. How much gain can be obtained by correction of misalignment?\n2. How much gain can be obtained by soft labels?\n\nThese results are already added to my solution write-up[4], so you can refer to it first if you haven't read.\n\nIn addition, I tested performance gain of these architectural choices:\n\n3. backbones\n4. decoders, especially the impact of PixelShuffle\n5. T-Mixer (same as *Temporal Modulator* in my original solution)\n\nIn this article, I discussed the experiment result of 3-5.\n\n## Experiment Detail\n\n### Note\n\nGiven the constraints of computing resources, all experiments were conducted using a single fold and a single random state. This means the results might not be entirely robust. For more definitive conclusions, multiple trials for each architecture would be necessary.\n\n### Choice of backbones\n\nI evaluated various backbones using a simple 2D U-Net architecture that only utilizes the current frame. The models were trained on `training` images and evaluated on `validation` images. For all experiments, I employed label correction and soft-labels. The results are presented in Table 1. Among the options, CoaT-lite medium demonstrated the best balance of performance and training time. Consequently, I adopted this backbone as the default for subsequent experiments.\n\n**Table 1: CV score of various backbones**\n\n| backbone | crop_size | epochs | gradient checkpointing | batch_size | CV | training time/epoch [sec] |\n|-------------------------|-----------|--------|---------------|------------|--------|----------------|\n| efficientnet_b3 | 512 | 20 | FALSE | 28 | 0.6629 | 187 |\n| maxvit_tiny_tf_512.in1k | 512 | 20 | FALSE | 19 | 0.6725 | 396 |\n| maxvit_small_tf_512.in1k| 512 | 20 | TRUE | 21 | 0.672 | 709 |\n| pvt_v2_b3 | 512 | 20 | FALSE | 20 | 0.6681 | 341 |\n| coat_lite_medium | 512 | 20 | FALSE | 20 | 0.6787 | 413 |\n| coat_small | 512 | 20 | FALSE | 19 | 0.6713 | 513 |\n\n\n### Changing Up-scaling to PixelShuffle\n\nPixelShuffle[1] offers an alternative method for up-scaling resolution while reducing feature channels. Following the approach mentioned by the 2nd place team in their solution[2], I implemented the PixelShuffle block as a replacement for the traditional up-scaling block, leading to what I refer to as the PS-U-Net architecture.\n\nWhen comparing the U-Net with the PS-U-Net, there wasn't a significant performance gain (CV: 0.6787 to 0.6779). However, the PS-U-Net had the advantage of being more memory-efficient due to the reduced feature channels. Given this efficiency, I chose to use the PS-U-Net as the default architecture for subsequent experiments.\n\n### The Effects of Intensive Augmentation\n\nThanks to the correction of misalignment, we can now employ a more intensive augmentation strategy, as mentioned in the 1st place solution[3]. I adapted Koda's augmentation approach with slight modifications, as shown below:\n\n\n```python\ncfg.geometric_transform = A.Compose(\n [\n A.RandomRotate90(p=1),\n A.HorizontalFlip(p=0.5),\n A.ShiftScaleRotate(\n rotate_limit=45, scale_limit=0.2, border_mode=cv2.BORDER_CONSTANT, p=0.6\n ),\n ]\n)\n```\n\nThe differences are:\n1. slightly increase probability (0.5 -> 0.6)\n2. set `borader_mode` as constant in order to avoid unintentional artifacts\n\nThe experimental results highlighted the substantial impact of this intensive augmentation. It led to an increase in the CV score by 0.58% (from 0.6779 to 0.6837). This underscores that addressing misalignment is crucial for success in this competition.\n\n### Changing Decoder Architecture: PS-U-Net -> PS-FPN\n\nNext, I transitioned from the PS-U-Net architecture to the PS-FPN, where I replaced the up-scaling block of the FPN with PixelShuffle blocks.\n\nThis change resulted in a modest improvement of 0.24% in CV scores (from 0.6837 to 0.6861). However, it's uncertain whether this gain can be solely attributed to the architectural modification. To draw a more definitive conclusion, averaging across multiple seeds would be advisable. Nonetheless, based on these preliminary results, I opted to use the PS-FPN architecture as the standard for subsequent experiments.\n\n### Comparing T-Mixer Architecture\n\nLastly, I tested the impact of different architectural choice of T-Mixer.\n\nEspecially, I wanted to compare which mixer is superior to this task: Convolutions and Transformer. So I compared below three architecture:\n\n1. Conv3d\n2. Transformer\n3. CoaT with (3D positional encoding)\n\nThe last pattern was just a spontaneous idea I had. However, since this backbone performs best in 2D settings, I believed it might be superior as temporal mixer. In this experiment, I evaluated on CV and LB. Additionally, I employed TTA to ensure robust evaluation.\n\nNote that training Transformer and CoaT T-Mixer were unstable, sometimes resulting in NaN predictions. Stability techniques were applied to address this (see Appendix A). In contrast, Conv3d T-Mixer training was stable without any adjustments.\n\nTable 2 displays the results. The first row represents a 2D U-Net without a T-Mixer, while the second uses a 2.5D PS-FPN with a Conv3D T-Mixer. Incorporating time frame information improved both CV and Private LB scores significantly. This indicates the value of temporal information even after alignment corrections.\n\nTable 2's 3rd and 4th rows show results for Transformer and CoaT T-Mixers. While CoaT T-Mixer had the highest CV score, its private LB score was the lowest. 3D Conv T-Mixer and Transformer T-Mixer had comparable performances.\n\nTable 3 presents ensemble model results. The best Private score came from an ensemble of three T-Mixer architectures, suggesting model diversity. Due to resource constraints, only one random seed was tested, but seed averaging or cross-fold averaging might yield better scores.\n\n\n**Table2: Comparison of Different T-Mixer Architectures**\n\n| id | used frames | T-Mixer | seed | TTA | crop_size | CV | Public | Private |\n|----|----------------|-------------------|------|----------------------------|-----------|--------|---------|---------|\n| 1 | [4] | None | x1 | hflip, rot(90, 180, 270) | 512 | 0.691 | 0.70244 | 0.70462 |\n| 2 | [1, 2, 3, 4] | 3d Conv | x1 | hflip, rot(90, 180, 270) | 512 | 0.6978 | 0.7147 | 0.71681 |\n| 3 | [1, 2, 3, 4] | Transformer | x1 | hflip, rot(90, 180, 270) | 512 | 0.6958 | 0.70576 | 0.71641 |\n| 4 | [1, 2, 3, 4] | CoaT(3D pos enc) | x1 | hflip, rot(90, 180, 270) | 512 | 0.6988 | 0.71553 | 0.71229 |\n\n**Table3: Result of Ensemble**\n\n| ensemble | CV | Public | Private |\n|------------|--------|--------|---------|\n| [2, 4] | 0.7009 | 0.718 | 0.71827 |\n| [2, 3, 4] | 0.701 | 0.71638| 0.72156 |\n\n## References\n\n* [1] [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://arxiv.org/abs/1609.05158v2)\n* [2] https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/430491\n* [3] https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/430618\n* [4] https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/430794\n* [5] [Gradient Centralization: A New Optimization Technique for Deep Neural Networks](https://arxiv.org/abs/2004.01461)\n* [6] [Adam+Half Precision = NaNs?](https://discuss.pytorch.org/t/adam-half-precision-nans/1765/5)\n\n## Appendix\n\n### A. Tips to stabilize Transformer/CoaT T-Mixer\n\nDuring my experiments with the Transformer and CoaT T-Mixer, I noticed numerous NaN values in the model's predictions. To stabilize the training, I implemented several techniques:\n\n1. Reduced the learning rate.\n2. Applied Gradient Centralization[5].\n3. Used an epsilon value of 1e-4 wherever feasible.\n\nDespite trying the first two techniques, the issue persisted.\n\nIt's worth noting that I consistently used AMP (Automatic Mixed Precision) throughout my experiments. Upon realizing that the issue didn't manifest in FP32 precision, I suspected that an excessively large epsilon might be causing instability, especially since similar issues have been reported with FP16 training[6].\n\nIncorporating all these techniques resolved the problem, so I employed them in CoaT and Transformer T-Mixer experiments.\n", "2400844": "Awesome stuff. Reproducing what worked for others is no easy task.", "2397968": "What a great idea and writeup! Thanks!\nCan you share the code for the 3D Conv T-Mixer part?" } }