![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F0d32daa11aecc54e4573a294b244ec79%2F21_architecture_tereka.png?generation=1722302269909845&alt=media\")
Thank you for holding this competition. We enjoyed a lot, as well as a lot of leaning experience. This was also really tough competition we ever participated. We literary training the model and making ensemble few hours before the competition closed, and finally we managed to win a gold medal.
\nThe following is the summary of 10th place solution by team Updraft. More detailed writeup might be shared by each members.
\nOur solution is ensemble of each 4 member's regression models.
\nThe models in the final submission contains 23 models with different architectures and learning conditions.
\nWe developed the following architectures.
We basically trained our models using 1-7y LR data, and evaluate with 1/6 subsample of 8y LR data.
\nSome model also use:
We used 1-MSE instead of R2 after std normalized target.
\nWe use this metrics because some member reported R2 loses correlation after host replaced the test dataset.
\n(However, later we found both are correlated when using sufficiently small eps for normalization.)
\nIn normalization, we used the following formula:
eps = ** (-)\ntarget = target / maximum(std(target), eps)\nBasically 1/7 ~ 1/6 subsample of LR data is used because of relatively poor machine resource (RTX4090 x 1).
\nWhat worked most was:
We extracted climate-invariant features which is suggested by Tom Beucler et. al. [1].
\nConceptually, these features are invariant under both cool and warm temperature.
\nSo these features are expected to cancel out the effect of yearly warm-up in ClimSim dataset (which I observed by EDA).
During the competition, we couldn't train models with this features on the full LR dataset, but it showed a significant gain of ~1% on a 1/7 to 1/6 subsample of LR dataset.
\nNote that the late submission results shows this features still have a significant gain on the full LR dataset (see Appendix A1).
\nWe used the following loss which we called confidence-aware MSE loss instead of MSE.
\nIntuitively it has the effect of loosening the contributions of the large error by hard samples.
\nThis idea was variant of the @ryomak 's previously shared solution[2].
\nNote that variance of target was regressed by model in addition to mean value. So our model's output variable is 2x larger than basic models.
loss = * (target - pred) ** / var + * torch.log(var)\nAs we shared on a public discussion[3], most of error came from extreme outliers. So we came to think that hard samples are good representatives of all dataset: if we doped small amount of hard samples from 1/1 full LR data, it would help to reduces regression error even if we used 1/n sub-sampled LR data.
\nWe extracted ~14% of hard samples which have the squared error greater than threshold using one of our best models, and doped to train other models. We found this has ~0.x% gain using this method.
\nWe also found adding tiny amount of neighbor time-frame sample from the same location also helps gaining accuracy.
Basically we used the following two architecture
\nThe second architecture was inspired by stacked hourglass architecture[5] which stacks U-Net and gradually improves the accuracy of output variable by regressing error of previous stack (like boosting). In addition, replacing ConvTranspose decoder in the usual U-Net to PixelShuffle gained ~0.x%.
\nWe often encounter gradient explosion when training deeper transformers (~8-12 layers).
\nWe mitigated this issue by utilizing trick we called tanh normalization.
\nIt is just inserting tanh activation before calculating softmax attention:
():\n x = rearrange(x, )\n\n qkv = self.qkv(x)\n qkv = rearrange(qkv, , h=self.num_heads)\n q, k, v = qkv.chunk(, dim=-)\n\n attn = (q @ k.transpose(-, -)) * self.scale\n attn = attn.tanh() * self.attn_clip_val \n attn = F.softmax(attn, dim=-)\n\n x = attn @ v\n x = rearrange(x, )\n\n x = self.proj(x)\n x = rearrange(x, )\n x\nIn the previous competition, some Kagglers reported removing Dropout layer has significant improvement on regression tasks[6]. We observed similar phenomena on this competition.
\nPublic/Private LB: 0.77784/0.77023\n\nModel\n Transformer + Convolution + LSTM\n Architecture Points\n Learnable Positional Embedding(nn.Embedding 60)\n several Transformer Layers\n Final Layer LSTM\n\nFeatures\n Original Dataset(standard scaler)\n\nDataset\n 70M(full low resolution dataset)\n\nLoss\n Huber Loss(> MSE)\n\nNot Worked\n Mixup\n EMA\n Deeper model, I tried some experiment, but it’s not stable.\n pretrained High resolution model, but low resolution and some high resolution data is worked in public LB.\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F194b6d4d267ae9c693f3fba5ab444cad%2F21_architecture_phalanx.png?generation=1722302291403618&alt=media\")
TBD
\nWe used tree type of ensemble method and final submission is weighted ensemble of these:
\nFor type2, basically we can't use the data which trained the seed model to fit parameter in the stacking.
\nSo we take the following scheme:
For the final submission, we ensemble the type-1 ~ type-3 model with manually fitted weight:
\nfinal_prediction = type1 * w1 + type2 * w2 + type3 * w3\nThe late submission result shows assigning 100% weight for type1 model performs the best on the private LB.
\n| Model | \nLB(Public) | \nLB(Private) | \nfinal submission | \n
|---|---|---|---|
| type1 * 0.5 + type2 * 0.3 + type3 * 0.2 | \n0.78803 | \n0.78503 | \n✔️ | \n
| type1 * 0.75 + type3 * 0.25 | \n0.78801 | \n0.78487 | \n✔️ | \n
| type1 * 0.6 * type2 * 0.4 | \n0.78777 | \n0.78535 | \n\n |
| type1 | \n0.78771 | \n0.78536 | \n\n |
We trained ensemble weight per (target, model). This is based on the study of EDA where we found each model has their expertize: some models performs good at regressing mixing ratio and some are good at regressing wind velocity etc.
\nWe tested the following variations and found ensemble without normalization and with bias performs the best.
\n| normalize weight? | \nwith bias? | \nLB(Public) | \nLB(Private) | \n
|---|---|---|---|
| ✔️ | \n- | \n0.78727 | \n0.78532 | \n
| - | \n- | \n0.78739 | \n0.78563 | \n
| - | \n✔️ | \n0.78740 | \n0.78569 | \n
During the competition, we only trained our models using climate-invariant features on a subsample of the LR data.However, after the competition ended, we successfully trained Phalanx's lightweight CNN model train model with climate-invariant features on the full sample of the LR dataset in feasible training time (~1 day) with a relatively \"low-throughput\" GPU (RTX 4090).
\nThe result of late submission shows it has significant gain of 0.05% on private LB. So climate-invariant feature has actually significant impact on this task.
\n| model | \nCV(1-MSE) | \nLB(Public) | \nLB(Private) | \n
|---|---|---|---|
| single w/o LR-full-clim-invariant | \n0.78231 | \n0.78025 | \n0.77721 | \n
| single w LR-full-clim-invariant | \n0.78434 | \n0.7814 | \n0.78017 | \n
| ensemble w/o LR-full-clim-invariant | \n0.79192 | \n0.78740 | \n0.78569 | \n
| ensemble w LR-full-clim-invariant | \n0.79218 | \n0.78771 | \n0.78618 | \n
Thank you for holding this competition. We enjoyed a lot, as well as a lot of leaning experience. This was also really tough competition we ever participated. We literary training the model and making ensemble few hours before the competition closed, and finally we managed to win a gold medal.
\nThe following is the summary of 10th place solution by team Updraft. More detailed writeup might be shared by each members.
\nOur solution is ensemble of each 4 member's regression models.
\nThe models in the final submission contains 23 models with different architectures and learning conditions.
\nWe developed the following architectures.
We basically trained our models using 1-7y LR data, and evaluate with 1/6 subsample of 8y LR data.
\nSome model also use:
We used 1-MSE instead of R2 after std normalized target.
\nWe use this metrics because some member reported R2 loses correlation after host replaced the test dataset.
\n(However, later we found both are correlated when using sufficiently small eps for normalization.)
\nIn normalization, we used the following formula:
eps = ** (-)\ntarget = target / maximum(std(target), eps)\nBasically 1/7 ~ 1/6 subsample of LR data is used because of relatively poor machine resource (RTX4090 x 1).
\nWhat worked most was:
We extracted climate-invariant features which is suggested by Tom Beucler et. al. [1].
\nConceptually, these features are invariant under both cool and warm temperature.
\nSo these features are expected to cancel out the effect of yearly warm-up in ClimSim dataset (which I observed by EDA).
During the competition, we couldn't train models with this features on the full LR dataset, but it showed a significant gain of ~1% on a 1/7 to 1/6 subsample of LR dataset.
\nNote that the late submission results shows this features still have a significant gain on the full LR dataset (see Appendix A1).
\nWe used the following loss which we called confidence-aware MSE loss instead of MSE.
\nIntuitively it has the effect of loosening the contributions of the large error by hard samples.
\nThis idea was variant of the @ryomak 's previously shared solution[2].
\nNote that variance of target was regressed by model in addition to mean value. So our model's output variable is 2x larger than basic models.
loss = * (target - pred) ** / var + * torch.log(var)\nAs we shared on a public discussion[3], most of error came from extreme outliers. So we came to think that hard samples are good representatives of all dataset: if we doped small amount of hard samples from 1/1 full LR data, it would help to reduces regression error even if we used 1/n sub-sampled LR data.
\nWe extracted ~14% of hard samples which have the squared error greater than threshold using one of our best models, and doped to train other models. We found this has ~0.x% gain using this method.
\nWe also found adding tiny amount of neighbor time-frame sample from the same location also helps gaining accuracy.
Basically we used the following two architecture
\nThe second architecture was inspired by stacked hourglass architecture[5] which stacks U-Net and gradually improves the accuracy of output variable by regressing error of previous stack (like boosting). In addition, replacing ConvTranspose decoder in the usual U-Net to PixelShuffle gained ~0.x%.
\nWe often encounter gradient explosion when training deeper transformers (~8-12 layers).
\nWe mitigated this issue by utilizing trick we called tanh normalization.
\nIt is just inserting tanh activation before calculating softmax attention:
():\n x = rearrange(x, )\n\n qkv = self.qkv(x)\n qkv = rearrange(qkv, , h=self.num_heads)\n q, k, v = qkv.chunk(, dim=-)\n\n attn = (q @ k.transpose(-, -)) * self.scale\n attn = attn.tanh() * self.attn_clip_val \n attn = F.softmax(attn, dim=-)\n\n x = attn @ v\n x = rearrange(x, )\n\n x = self.proj(x)\n x = rearrange(x, )\n x\nIn the previous competition, some Kagglers reported removing Dropout layer has significant improvement on regression tasks[6]. We observed similar phenomena on this competition.
\nPublic/Private LB: 0.77784/0.77023\n\nModel\n Transformer + Convolution + LSTM\n Architecture Points\n Learnable Positional Embedding(nn.Embedding 60)\n several Transformer Layers\n Final Layer LSTM\n\nFeatures\n Original Dataset(standard scaler)\n\nDataset\n 70M(full low resolution dataset)\n\nLoss\n Huber Loss(> MSE)\n\nNot Worked\n Mixup\n EMA\n Deeper model, I tried some experiment, but it’s not stable.\n pretrained High resolution model, but low resolution and some high resolution data is worked in public LB.\n
TBD
\nWe used tree type of ensemble method and final submission is weighted ensemble of these:
\nFor type2, basically we can't use the data which trained the seed model to fit parameter in the stacking.
\nSo we take the following scheme:
For the final submission, we ensemble the type-1 ~ type-3 model with manually fitted weight:
\nfinal_prediction = type1 * w1 + type2 * w2 + type3 * w3\nThe late submission result shows assigning 100% weight for type1 model performs the best on the private LB.
\n| Model | \nLB(Public) | \nLB(Private) | \nfinal submission | \n
|---|---|---|---|
| type1 * 0.5 + type2 * 0.3 + type3 * 0.2 | \n0.78803 | \n0.78503 | \n✔️ | \n
| type1 * 0.75 + type3 * 0.25 | \n0.78801 | \n0.78487 | \n✔️ | \n
| type1 * 0.6 * type2 * 0.4 | \n0.78777 | \n0.78535 | \n\n |
| type1 | \n0.78771 | \n0.78536 | \n\n |
We trained ensemble weight per (target, model). This is based on the study of EDA where we found each model has their expertize: some models performs good at regressing mixing ratio and some are good at regressing wind velocity etc.
\nWe tested the following variations and found ensemble without normalization and with bias performs the best.
\n| normalize weight? | \nwith bias? | \nLB(Public) | \nLB(Private) | \n
|---|---|---|---|
| ✔️ | \n- | \n0.78727 | \n0.78532 | \n
| - | \n- | \n0.78739 | \n0.78563 | \n
| - | \n✔️ | \n0.78740 | \n0.78569 | \n
During the competition, we only trained our models using climate-invariant features on a subsample of the LR data.However, after the competition ended, we successfully trained Phalanx's lightweight CNN model train model with climate-invariant features on the full sample of the LR dataset in feasible training time (~1 day) with a relatively \"low-throughput\" GPU (RTX 4090).
\nThe result of late submission shows it has significant gain of 0.05% on private LB. So climate-invariant feature has actually significant impact on this task.
\n| model | \nCV(1-MSE) | \nLB(Public) | \nLB(Private) | \n
|---|---|---|---|
| single w/o LR-full-clim-invariant | \n0.78231 | \n0.78025 | \n0.77721 | \n
| single w LR-full-clim-invariant | \n0.78434 | \n0.7814 | \n0.78017 | \n
| ensemble w/o LR-full-clim-invariant | \n0.79192 | \n0.78740 | \n0.78569 | \n
| ensemble w LR-full-clim-invariant | \n0.79218 | \n0.78771 | \n0.78618 | \n
Good job and congrats on on gold! Can you please elaborate on 'confidence aware' and 'variance prediction'? (I actually also used 'confidence loss' but I'm not sure if you mean the same thing)
", "rawMarkdown": "Good job and congrats on on gold! Can you please elaborate on 'confidence aware' and 'variance prediction'? (I actually also used 'confidence loss' but I'm not sure if you mean the same thing)", "votes": 1, "replies": [ { "id": 2940269, "postDate": "2024-07-29T23:07:39.163Z", "content": "Thanks. I introduced this loss so that the model predicts difficulty of targets. I think it’s close to Huber loss, which other team mates used.
\nGood job and congrats on on gold! Can you please elaborate on 'confidence aware' and 'variance prediction'? (I actually also used 'confidence loss' but I'm not sure if you mean the same thing)
", "votes": 1, "replies": [ { "id": 2940269, "author_name": "Bilzard", "author_url": "", "post_date": "2024-07-29T23:07:39.163000", "content": "Thanks. I introduced this loss so that the model predicts difficulty of targets. I think it’s close to Huber loss, which other team mates used.
\n