![\"jpg](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5724098%2F8c7335ec3f1f94f0f40e041832b28303%2Floss.png?generation=1722302507888184&alt=media\")
Thanks to the hosts @jerrylin96 and official kaggle team for organizing this interesting competition! Our team learned a lot from it!
\nI am very pleased to share our team's solution. The complete code is available on GitHub: https://github.com/ChunhanLi/2nd-kaggle-LEAP
We do not have extensive domain knowledge in atmospheric forecasting; we simply framed the competition task as a 1D seq2seq multi-target regression task. We trained with various model structures and meticulously tuned them, ultimately using a hill-climbing algorithm for model ensembling based on validation set scores.
\nThe four highest-priority key points in the solution are summarized as follows:
\nTraining data
\nTraining data is crucial for this competition. According to this GitHub repository, the Kaggle data is sampled from this low-resolution dataset. Utilizing the entire low-resolution dataset can boost performance by almost 0.01 in this competition.
Cross validation
\nOur models are trained using data from the first 7 years and the first half (January to June) of year 8, totaling approximately 75 million data points. We set the stride_sample parameter to 7 and used data from July to December of year 8 and January of year 9 as our hold-out validation dataset. This validation set has shown perfect correlation with the leaderboard scores.
Smooth L1 loss
\nThe Smooth L1 loss is a robust loss function used primarily in regression tasks, combining the advantages of L2 and L1 losses. It is less sensitive to outliers than L2 and avoids the non-differentiability at zero seen with L1. The function is defined as follows:
This approach helps stabilize the training process and often improves performance in noisy data situations.
\n![\"jpg](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5724098%2F566356fb54e82e5218a83c0746d8de42%2Ff1.png?generation=1722302556372586&alt=media\")
Auxiliary diff loss
\nWe use an auxiliary 'Diff Loss,' proposed by @zui0711 , to enhance learning. This loss calculates the difference between real and predicted values at adjacent levels of a target, using Smooth L1 loss to quantify the error. The approach is implemented as follows:
torch.no_grad():\n outputs = model(inputs)\n loss = criterion(outputs, labels)\n i ():\n output_diff = outputs[:, +*i+:+*(i+)] - outputs[:, +*i:+*(i+)-]\n label_diff = labels[:, +*i+:+*(i+)] - labels[:, +*i:+*(i+)-]\n loss += criterion(output_diff, label_diff) / \nCosine annealing scheduler
\nThe cosine annealing schedule adjusts the learning rate in a cosine-shaped curve to help the model escape local minima and improve convergence. We implemented this with learning rate decays at the 3rd and 9th epochs.
![\"jpg](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5724098%2Fe965962b0d05d483bf9d1bf690014863%2Ff2.png?generation=1722302675640626&alt=media\")
During multi-objective learning tasks, different learning objectives can interact in complex ways, promoting or inhibiting each other. Our experiments found that while different target groups initially promoted each other, they began to interfere towards the end of training, potentially due to complex semantic constraints.
\nInspired by the top solution from the 2021 VPP competition, we divided 368 features into seven groups, six of which are series of measurements of different metrics along the atmospheric column, and one group consists of eight unique single targets. After the training process with 364 full outputs was completed, we fine-tuned these groups again. This allowed each model with different architectures to achieve an improvement ranging from 0.0005 to 0.0015. Due to time and resource constraints, we only fine-tuned each group for one epoch. @max2020
\nAfter obtaining raw model predictions, we denormalize them using the standard deviation and mean. Following community discussions, we adjust certain variables of ptend_q0002 based on their linear relationships. Finally, we apply the weight values from the official sample submission file to process the predictions.
\nTable 1
\n| Model id | \npb | \nfig | \nExp | \n
|---|---|---|---|
| forcewithme_reslstm_cv0.789_lb0.783 | \n0.78275 | \nFigure 3(a) | \n18 | \n
| forcewithme_gf_reslstm_cv0.790_lb0.785 | \n0.7840 | \nFigure 3(a) | \n32 | \n
| forcewithme_gf_cnnlstm_cv0.789_lb0.787 | \n0.7826 | \nFigure 3(b) | \n39 | \n
| forcewithme_gf_lstmmamba_cv0.7885_lb0.7853 | \n0.7814 | \nFigure 3(c) | \n40 | \n
| forcewithme_gf_LstmMambaMixed_cv0.7886_lb0.7858 | \n0.7821 | \nFigure 3(d) | \n- | \n
| forcewithme_gf037_1LSTM1mamba-5_state16_cv0.7896_LBunknown | \n0.7830 | \nFigure 3(d) | \n37 | \n
| forcewithme_gf038_2LSTM1mamba-3_state64_cv0.7897_LB0.787 | \n0.7836 | \nFigure 3(d) | \n38 | \n
![\"jpg](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5724098%2F978e5b0a2e223d5024fb69cb1031e526%2Ff3.png?generation=1722301967906869&alt=media\")
The final ensemble strategy contains 6 models of ForcewithMe. These 6 models, the private score, the corresponding architectures, and the corresponding \"exp id\" in table 2 are in the table above. The model architectures are easy to understand and implement with the corresponding figures and the code. Here are some details and insights:
\nAll models show local scores and public leaderboard (LB) scores within their model ids, which correspond directly to filenames, model names, and submission files (without extensions). The table supplements these with private LB scores and details on the model architectures.
All models are primarily built with LSTM as main components. This is due to our finding that LSTMs significantly outperform other architectures such as CNNs, MHSA, NNs, and GRUs in this competition.
Inspired by ResNet and Transformer, all my models incorporate residual connection. We believe residuals make models converge faster and perform better.
Figure 3(a) depicts a simple ResLSTM structure, which surprisingly achieved the highest private LB score among all models. Moreover, It took over two weeks for my another submission to beat this residual LSTM model on both local score and on the public leaderboard. Additionally, ResLSTM is still better on the private LB. exp32 has the same architecture to exp 18. The former one resumes the weights of the latter one and applies group fine-tuning.
Figure 3b illustrates a model combining small kernel convolutions, large kernel convolutions, and LSTMs as encoders, with the encoded results concatenated and fed into an LSTM backbone. This design leverages the presumed relationships between adjacent atmospheric layers, aiming to model local information and positional relations. Although it performs less well offline and on the private LB compared to the pure resLSTM model, it scores higher on the public LB and provides gains during the ensemble.
The remaining four models combine LSTMs with MAMBA. While LSTMs typically have more layers, MAMBA occupies the same number of layers as the LSTMs in model forcewithme_gf037_1LSTM1mamba-5_state16_cv0.7896_LBunknown.
The models mixing LSTM and Mamba (3d) in every block achieve very good scores offline and on the public LB.
Despite individual MAMBA-based models not outperforming the pure LSTM structure, they play a significant role in the final ensemble.
There are two versions of MAMBA available: MAMBA and MAMBA-2 in the \"mamba-ssm\" package. We exclusively used MAMBA.
The default parameters for MAMBA are: d_model=512, d_state=16, d_conv=4, expand=2. Other than d_model, the parameter settings follow the official repository.
The models forcewithme_gf037_1LSTM1mamba-5_state16_cv0.7896_LBunknown have 1 LSTM and 1 MAMBA in each block, and it has 5 blocks, the mamba uses default params setting. While the models forcewithme_gf_LstmMambaMixed_cv0.7886_lb0.7858 and forcewithme_gf038_2LSTM1mamba-3_state64 have 2 LSTMs and 1 mamba in each block, and they have 3 blocks.
The models forcewithme_gf_LstmMambaMixed_cv0.7886_lb0.7858 and forcewithme_gf038_2LSTM1mamba-3_state64 differ only in that the latter has d_state set to 64. They share figure 3(d) as they are almost the same.
For installation and usage of MAMBA, please refer to the official repository: https://github.com/state-spaces/mamba?tab=readme-ov-file. It's licensed under Apache-2.0, allowing for free use, including commercial purposes, provided the license requirements are met.
In the Max2020 part, models 14, 15, 21, and 22 are all improvements based on the model by @forcewithme, integrating LSTM with skip connections. Model 22 is our team’s highest-performing single model, providing the best results in local scoring, Leader Board scoring, and private scoring. Its architecture is shown in Figure 4. Regarding the learning rate schedule, a cosine decay learning rate was used, with decays occurring at 3 and 9 epochs. The loss function employed was the smooth L1 loss with a beta of 0.5. The Table 2 contains the detailed performance of my five models.
\n![\"jpg](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5724098%2F82ce64bb111824f2eee6c53954286c7a%2Ff4.png?generation=1722302228698175&alt=media\")
The structure of Joseph's models mainly consists of the use of multi-layers Res-ConvLSTM blocks and a TimeDistributed fully-connected layer. The input size is [bs, 60, 25] and output size is [bs, 368], the last dimension of the input includes 16 sequence-features and 9 scalar-features. After passing the multi-layers Res-ConvLSTM blocks and fully-connected layer, it will result in a size of [bs, 60, 14], the last dimension includes 6 sequence-labels and 8 scalar-labels. For scalar-labels part which is [bs, 60, 8], we just average in the second axis to get the shape of [bs, 8]. For sequence-labels part which is [bs, 60, 6], we reshape it to the shape of [bs, 360]. Finally, we concatenate the two parts to get the output. Besides, a reverse and shift type of augmentation is implemented in one of the models. The idea is we randomly reverse or shift the sequence-features and sequence-labels respectively in the data loader.
The structure of Adam's models are like: Inputs -> Position encoding -> 1DCNN -> 3-4 layers LSTM -> 1 layer transformer -> Outputs. Most models used all the low-resolution dataset and the others only use sampling data. The model trained by sampling data can contribute to the ensemble a bit. For the loss, Adam used SmoothL1 loss and auxiliary diff loss as mentioned before. For the scheduler, Adam used ReduceLROnPlateau scheduler with a factor of 0.2 and patience of 2.
\nAdam's best single model has LB 0.78594 and PB 0.78141. The ensemble of Adam's models has LB 0.79050 and PB 0.78575.
\nZuiye's models are mainly based on two architectures. The first one consists of 2 LSTM layers followed by a MultiheadAttention layer. The other one consists of 3 parallel Convolutional layers with 3 different kernel sizes and next 2 LSTM layers followed by a MultiheadAttention layer just like the first architecture. Zuiye's best single model gets LB 0.78696 / PB 0.78205 and the ensemble of Zuiye's own models (with hill climb) gets LB 0.79050 / PB 0.78614.
\nWe use the hill climb method to search blend weights of each model. The steps of hill climb are the following: @hookman
\nnew\\_ensemble = w * best\\_ensemble + (1-w) * new\\_oofWeights of the best model are shown in the table below:
\nTable 2
| Exp id | \nweight | \ncv | \nlb | \npb | \n
|---|---|---|---|---|
| forcewithme_exp32 | \n0.166556 | \n0.790 | \n0.7865 | \n0.78398 | \n
| forcewithme_exp37 | \n0.158625 | \n0.7896 | \n0.78618 | \n0.78293 | \n
| forcewithme_exp38 | \n0.139194 | \n0.7897 | \n0.78719 | \n0.78362 | \n
| max_exp22 | \n0.120125 | \n0.7908 | \n0.78793 | \n0.78434 | \n
| Jo_exp912 | \n0.111971 | \n0.78935 | \n0.78528 | \n0.78150 | \n
| max_exp21 | \n0.104738 | \n0.7904 | \n0.78752 | \n0.78425 | \n
| forcewithme_exp39 | \n0.098977 | \n0.789 | \n0.78699 | \n0.78257 | \n
| max_exp14 | \n0.093088 | \n0.7905 | \n0.78641 | \n0.78214 | \n
| max_exp10 | \n0.092157 | \n0.7888 | \n0.78619 | \n0.78213 | \n
| forcewithme_exp40 | \n0.082941 | \n0.7885 | \n0.7853 | \n0.78261 | \n
| max_exp015 | \n0.052500 | \n0.7905 | \n0.78695 | \n0.78244 | \n
| adam_exp197 | \n0.048994 | \n0.7855 | \n0.78269 | \n0.777 | \n
| adam_exp200 | \n-0.047132 | \n0.7836 | \n0.78010 | \n0.77434 | \n
| adam_exp195 | \n-0.049875 | \n0.78569 | \n0.78334 | \n0.77753 | \n
| Jo_exp907 | \n-0.083779 | \n0.7855 | \n0.78289 | \n0.77873 | \n
| forcewithme_exp18 | \n-0.089079 | \n0.7890 | \n0.7863 | \n0.78272 | \n
| Ensemble | \n1.0 | \n0.7955 | \n0.79211 | \n0.78856 | \n
It's worth noting that we've provided the parameter weights files for all models on Google Drive. Using these for ensemble submissions results in slightly higher LB and PB scores, by a margin of 0.0002. The discrepancies stem from two main issues:
\nFirstly, the jo_exp907.pt model file was missing, and the model had to be rerun post-competition, which led to some differences from the original. Secondly, during the competition, an incorrect model file was used under forcewithme_exp18 (corresponding to the forcewithme_reslstm_cv0.789_lb0.783 folder). This has now been corrected. These two points have caused a very minor difference in our final results. Although we believe this difference does not impact the reproducibility of our overall approach, we mention it here to avoid any confusion.
Thanks to the hosts @jerrylin96 and official kaggle team for organizing this interesting competition! Our team learned a lot from it!
\nI am very pleased to share our team's solution. The complete code is available on GitHub: https://github.com/ChunhanLi/2nd-kaggle-LEAP
We do not have extensive domain knowledge in atmospheric forecasting; we simply framed the competition task as a 1D seq2seq multi-target regression task. We trained with various model structures and meticulously tuned them, ultimately using a hill-climbing algorithm for model ensembling based on validation set scores.
\nThe four highest-priority key points in the solution are summarized as follows:
\nTraining data
\nTraining data is crucial for this competition. According to this GitHub repository, the Kaggle data is sampled from this low-resolution dataset. Utilizing the entire low-resolution dataset can boost performance by almost 0.01 in this competition.
Cross validation
\nOur models are trained using data from the first 7 years and the first half (January to June) of year 8, totaling approximately 75 million data points. We set the stride_sample parameter to 7 and used data from July to December of year 8 and January of year 9 as our hold-out validation dataset. This validation set has shown perfect correlation with the leaderboard scores.
Smooth L1 loss
\nThe Smooth L1 loss is a robust loss function used primarily in regression tasks, combining the advantages of L2 and L1 losses. It is less sensitive to outliers than L2 and avoids the non-differentiability at zero seen with L1. The function is defined as follows:
This approach helps stabilize the training process and often improves performance in noisy data situations.
\nAuxiliary diff loss
\nWe use an auxiliary 'Diff Loss,' proposed by @zui0711 , to enhance learning. This loss calculates the difference between real and predicted values at adjacent levels of a target, using Smooth L1 loss to quantify the error. The approach is implemented as follows:
torch.no_grad():\n outputs = model(inputs)\n loss = criterion(outputs, labels)\n i ():\n output_diff = outputs[:, +*i+:+*(i+)] - outputs[:, +*i:+*(i+)-]\n label_diff = labels[:, +*i+:+*(i+)] - labels[:, +*i:+*(i+)-]\n loss += criterion(output_diff, label_diff) / \nCosine annealing scheduler
\nThe cosine annealing schedule adjusts the learning rate in a cosine-shaped curve to help the model escape local minima and improve convergence. We implemented this with learning rate decays at the 3rd and 9th epochs.
During multi-objective learning tasks, different learning objectives can interact in complex ways, promoting or inhibiting each other. Our experiments found that while different target groups initially promoted each other, they began to interfere towards the end of training, potentially due to complex semantic constraints.
\nInspired by the top solution from the 2021 VPP competition, we divided 368 features into seven groups, six of which are series of measurements of different metrics along the atmospheric column, and one group consists of eight unique single targets. After the training process with 364 full outputs was completed, we fine-tuned these groups again. This allowed each model with different architectures to achieve an improvement ranging from 0.0005 to 0.0015. Due to time and resource constraints, we only fine-tuned each group for one epoch. @max2020
\nAfter obtaining raw model predictions, we denormalize them using the standard deviation and mean. Following community discussions, we adjust certain variables of ptend_q0002 based on their linear relationships. Finally, we apply the weight values from the official sample submission file to process the predictions.
\nTable 1
\n| Model id | \npb | \nfig | \nExp | \n
|---|---|---|---|
| forcewithme_reslstm_cv0.789_lb0.783 | \n0.78275 | \nFigure 3(a) | \n18 | \n
| forcewithme_gf_reslstm_cv0.790_lb0.785 | \n0.7840 | \nFigure 3(a) | \n32 | \n
| forcewithme_gf_cnnlstm_cv0.789_lb0.787 | \n0.7826 | \nFigure 3(b) | \n39 | \n
| forcewithme_gf_lstmmamba_cv0.7885_lb0.7853 | \n0.7814 | \nFigure 3(c) | \n40 | \n
| forcewithme_gf_LstmMambaMixed_cv0.7886_lb0.7858 | \n0.7821 | \nFigure 3(d) | \n- | \n
| forcewithme_gf037_1LSTM1mamba-5_state16_cv0.7896_LBunknown | \n0.7830 | \nFigure 3(d) | \n37 | \n
| forcewithme_gf038_2LSTM1mamba-3_state64_cv0.7897_LB0.787 | \n0.7836 | \nFigure 3(d) | \n38 | \n
The final ensemble strategy contains 6 models of ForcewithMe. These 6 models, the private score, the corresponding architectures, and the corresponding \"exp id\" in table 2 are in the table above. The model architectures are easy to understand and implement with the corresponding figures and the code. Here are some details and insights:
\nAll models show local scores and public leaderboard (LB) scores within their model ids, which correspond directly to filenames, model names, and submission files (without extensions). The table supplements these with private LB scores and details on the model architectures.
All models are primarily built with LSTM as main components. This is due to our finding that LSTMs significantly outperform other architectures such as CNNs, MHSA, NNs, and GRUs in this competition.
Inspired by ResNet and Transformer, all my models incorporate residual connection. We believe residuals make models converge faster and perform better.
Figure 3(a) depicts a simple ResLSTM structure, which surprisingly achieved the highest private LB score among all models. Moreover, It took over two weeks for my another submission to beat this residual LSTM model on both local score and on the public leaderboard. Additionally, ResLSTM is still better on the private LB. exp32 has the same architecture to exp 18. The former one resumes the weights of the latter one and applies group fine-tuning.
Figure 3b illustrates a model combining small kernel convolutions, large kernel convolutions, and LSTMs as encoders, with the encoded results concatenated and fed into an LSTM backbone. This design leverages the presumed relationships between adjacent atmospheric layers, aiming to model local information and positional relations. Although it performs less well offline and on the private LB compared to the pure resLSTM model, it scores higher on the public LB and provides gains during the ensemble.
The remaining four models combine LSTMs with MAMBA. While LSTMs typically have more layers, MAMBA occupies the same number of layers as the LSTMs in model forcewithme_gf037_1LSTM1mamba-5_state16_cv0.7896_LBunknown.
The models mixing LSTM and Mamba (3d) in every block achieve very good scores offline and on the public LB.
Despite individual MAMBA-based models not outperforming the pure LSTM structure, they play a significant role in the final ensemble.
There are two versions of MAMBA available: MAMBA and MAMBA-2 in the \"mamba-ssm\" package. We exclusively used MAMBA.
The default parameters for MAMBA are: d_model=512, d_state=16, d_conv=4, expand=2. Other than d_model, the parameter settings follow the official repository.
The models forcewithme_gf037_1LSTM1mamba-5_state16_cv0.7896_LBunknown have 1 LSTM and 1 MAMBA in each block, and it has 5 blocks, the mamba uses default params setting. While the models forcewithme_gf_LstmMambaMixed_cv0.7886_lb0.7858 and forcewithme_gf038_2LSTM1mamba-3_state64 have 2 LSTMs and 1 mamba in each block, and they have 3 blocks.
The models forcewithme_gf_LstmMambaMixed_cv0.7886_lb0.7858 and forcewithme_gf038_2LSTM1mamba-3_state64 differ only in that the latter has d_state set to 64. They share figure 3(d) as they are almost the same.
For installation and usage of MAMBA, please refer to the official repository: https://github.com/state-spaces/mamba?tab=readme-ov-file. It's licensed under Apache-2.0, allowing for free use, including commercial purposes, provided the license requirements are met.
In the Max2020 part, models 14, 15, 21, and 22 are all improvements based on the model by @forcewithme, integrating LSTM with skip connections. Model 22 is our team’s highest-performing single model, providing the best results in local scoring, Leader Board scoring, and private scoring. Its architecture is shown in Figure 4. Regarding the learning rate schedule, a cosine decay learning rate was used, with decays occurring at 3 and 9 epochs. The loss function employed was the smooth L1 loss with a beta of 0.5. The Table 2 contains the detailed performance of my five models.
\nThe structure of Joseph's models mainly consists of the use of multi-layers Res-ConvLSTM blocks and a TimeDistributed fully-connected layer. The input size is [bs, 60, 25] and output size is [bs, 368], the last dimension of the input includes 16 sequence-features and 9 scalar-features. After passing the multi-layers Res-ConvLSTM blocks and fully-connected layer, it will result in a size of [bs, 60, 14], the last dimension includes 6 sequence-labels and 8 scalar-labels. For scalar-labels part which is [bs, 60, 8], we just average in the second axis to get the shape of [bs, 8]. For sequence-labels part which is [bs, 60, 6], we reshape it to the shape of [bs, 360]. Finally, we concatenate the two parts to get the output. Besides, a reverse and shift type of augmentation is implemented in one of the models. The idea is we randomly reverse or shift the sequence-features and sequence-labels respectively in the data loader.
The structure of Adam's models are like: Inputs -> Position encoding -> 1DCNN -> 3-4 layers LSTM -> 1 layer transformer -> Outputs. Most models used all the low-resolution dataset and the others only use sampling data. The model trained by sampling data can contribute to the ensemble a bit. For the loss, Adam used SmoothL1 loss and auxiliary diff loss as mentioned before. For the scheduler, Adam used ReduceLROnPlateau scheduler with a factor of 0.2 and patience of 2.
\nAdam's best single model has LB 0.78594 and PB 0.78141. The ensemble of Adam's models has LB 0.79050 and PB 0.78575.
\nZuiye's models are mainly based on two architectures. The first one consists of 2 LSTM layers followed by a MultiheadAttention layer. The other one consists of 3 parallel Convolutional layers with 3 different kernel sizes and next 2 LSTM layers followed by a MultiheadAttention layer just like the first architecture. Zuiye's best single model gets LB 0.78696 / PB 0.78205 and the ensemble of Zuiye's own models (with hill climb) gets LB 0.79050 / PB 0.78614.
\nWe use the hill climb method to search blend weights of each model. The steps of hill climb are the following: @hookman
\nnew\\_ensemble = w * best\\_ensemble + (1-w) * new\\_oofWeights of the best model are shown in the table below:
\nTable 2
| Exp id | \nweight | \ncv | \nlb | \npb | \n
|---|---|---|---|---|
| forcewithme_exp32 | \n0.166556 | \n0.790 | \n0.7865 | \n0.78398 | \n
| forcewithme_exp37 | \n0.158625 | \n0.7896 | \n0.78618 | \n0.78293 | \n
| forcewithme_exp38 | \n0.139194 | \n0.7897 | \n0.78719 | \n0.78362 | \n
| max_exp22 | \n0.120125 | \n0.7908 | \n0.78793 | \n0.78434 | \n
| Jo_exp912 | \n0.111971 | \n0.78935 | \n0.78528 | \n0.78150 | \n
| max_exp21 | \n0.104738 | \n0.7904 | \n0.78752 | \n0.78425 | \n
| forcewithme_exp39 | \n0.098977 | \n0.789 | \n0.78699 | \n0.78257 | \n
| max_exp14 | \n0.093088 | \n0.7905 | \n0.78641 | \n0.78214 | \n
| max_exp10 | \n0.092157 | \n0.7888 | \n0.78619 | \n0.78213 | \n
| forcewithme_exp40 | \n0.082941 | \n0.7885 | \n0.7853 | \n0.78261 | \n
| max_exp015 | \n0.052500 | \n0.7905 | \n0.78695 | \n0.78244 | \n
| adam_exp197 | \n0.048994 | \n0.7855 | \n0.78269 | \n0.777 | \n
| adam_exp200 | \n-0.047132 | \n0.7836 | \n0.78010 | \n0.77434 | \n
| adam_exp195 | \n-0.049875 | \n0.78569 | \n0.78334 | \n0.77753 | \n
| Jo_exp907 | \n-0.083779 | \n0.7855 | \n0.78289 | \n0.77873 | \n
| forcewithme_exp18 | \n-0.089079 | \n0.7890 | \n0.7863 | \n0.78272 | \n
| Ensemble | \n1.0 | \n0.7955 | \n0.79211 | \n0.78856 | \n
It's worth noting that we've provided the parameter weights files for all models on Google Drive. Using these for ensemble submissions results in slightly higher LB and PB scores, by a margin of 0.0002. The discrepancies stem from two main issues:
\nFirstly, the jo_exp907.pt model file was missing, and the model had to be rerun post-competition, which led to some differences from the original. Secondly, during the competition, an incorrect model file was used under forcewithme_exp18 (corresponding to the forcewithme_reslstm_cv0.789_lb0.783 folder). This has now been corrected. These two points have caused a very minor difference in our final results. Although we believe this difference does not impact the reproducibility of our overall approach, we mention it here to avoid any confusion.
How did you shuffle/load training data? I had trouble getting it into memory and if you don't adequately shuffle larger batches of data (if you load them separately) the learning becomes noisy.
", "rawMarkdown": "How did you shuffle/load training data? I had trouble getting it into memory and if you don't adequately shuffle larger batches of data (if you load them separately) the learning becomes noisy. ", "replies": [ { "id": 2975095, "postDate": "2024-08-31T12:33:01.800Z", "content": "We rented a high-memory server, and reading the entire dataset requires at least 300GB of memory space.
", "rawMarkdown": "We rented a high-memory server, and reading the entire dataset requires at least 300GB of memory space." } ] }, { "id": 2940737, "postDate": "2024-07-30T12:05:46.200Z", "content": "Congratulations on achieving second place in the LEAP - Atmospheric Physics using AI (ClimSim) competition! @max2020
\nYour innovative approach, framing the task as a 1D seq2seq multi-target regression, and the use of a hill-climbing algorithm for model ensembling are truly commendable. The strategic use of the entire low-resolution dataset, Smooth L1 loss, auxiliary diff loss, and group fine-tuning methods significantly enhanced performance. Your detailed and transparent sharing of the solution on GitHub is a valuable resource for the community. Thank you for your contribution and for recognizing the support of the hosts and the Kaggle team.
Thank you for the explanation.
", "rawMarkdown": "Thank you for the explanation. " } ], "comments": [ { "id": 2968112, "author_name": "zpanj", "author_url": "", "post_date": "2024-08-23T14:48:26.997000", "content": "How did you shuffle/load training data? I had trouble getting it into memory and if you don't adequately shuffle larger batches of data (if you load them separately) the learning becomes noisy.
", "votes": 0, "replies": [ { "id": 2975095, "author_name": "Max2020", "author_url": "", "post_date": "2024-08-31T12:33:01.800000", "content": "We rented a high-memory server, and reading the entire dataset requires at least 300GB of memory space.
", "votes": 0, "replies": [] } ] }, { "id": 2940737, "author_name": "", "author_url": "", "post_date": "2024-07-30T12:05:46.200000", "content": "Congratulations on achieving second place in the LEAP - Atmospheric Physics using AI (ClimSim) competition! @max2020
\nYour innovative approach, framing the task as a 1D seq2seq multi-target regression, and the use of a hill-climbing algorithm for model ensembling are truly commendable. The strategic use of the entire low-resolution dataset, Smooth L1 loss, auxiliary diff loss, and group fine-tuning methods significantly enhanced performance. Your detailed and transparent sharing of the solution on GitHub is a valuable resource for the community. Thank you for your contribution and for recognizing the support of the hosts and the Kaggle team.
Thank you for the explanation.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2940309": "Thanks to the hosts @jerrylin96 and official kaggle team for organizing this interesting competition! Our team learned a lot from it!\nI am very pleased to share our team's solution. The complete code is available on GitHub: [https://github.com/ChunhanLi/2nd-kaggle-LEAP](https://github.com/ChunhanLi/2nd-kaggle-LEAP)\n\n## Summary\nWe do not have extensive domain knowledge in atmospheric forecasting; we simply framed the competition task as a 1D seq2seq multi-target regression task. We trained with various model structures and meticulously tuned them, ultimately using a hill-climbing algorithm for model ensembling based on validation set scores.\n\nThe four highest-priority key points in the solution are summarized as follows:\n- **Data size is super important in this competition.** Using the entire [low-resolution dataset](https://huggingface.co/datasets/LEAP/ClimSim_low-res) can significantly enhance performance.\n- **The smoothl1 loss function, auxiliary diff loss and the cosine annealing learning rate schedule were adopted for model optimization.**\n- **Group fine-tuning methods were introduced in the training pipeline.**\n- **The team focused on developing diverse model structures and ensembling them.**\n\n\n## Solution\n\n### Data and Cross validation\n\n**Training data** \nTraining data is crucial for this competition. According to this [GitHub repository](https://github.com/leap-stc/ClimSim), the Kaggle data is sampled from this [low-resolution dataset](https://huggingface.co/datasets/LEAP/ClimSim_low-res). Utilizing the entire low-resolution dataset can boost performance by almost 0.01 in this competition.\n\n**Cross validation** \nOur models are trained using data from the first 7 years and the first half (January to June) of year 8, totaling approximately 75 million data points. We set the `stride_sample` parameter to 7 and used data from July to December of year 8 and January of year 9 as our hold-out validation dataset. This validation set has shown perfect correlation with the leaderboard scores.\n\n### Model optimization\n\n**Smooth L1 loss** \nThe Smooth L1 loss is a robust loss function used primarily in regression tasks, combining the advantages of L2 and L1 losses. It is less sensitive to outliers than L2 and avoids the non-differentiability at zero seen with L1. The function is defined as follows:\n\n\n