Thanks to Learning the Earth with Artificial Intelligence and Physics (LEAP) NSF Science and Technology Center and Kaggle for hosting this fun competition, which was very stable in CV, LB and PB.
\nConsidering that my English is not particularly good, this article is being written with the assistance of chatgpt for translation. I will check and make corrections.
\ninfer code to prove that no leak: LEAP_11th_PB0.78491
\nBusiness context: https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/overviewData context: https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/dataMy final submission consisted of 6 models (with 5 different architectures) integrated together(important). All models use residual structures. All models are trained on the full dataset(low-res) from 0001-02 to 0008-01, and validated on approximately 1.7 million rows of data sampled from 0008-02 to 0009-01. Using both standardization, log transformation and embedding for input process. The models have undergone meticulous training through multiple stages(important). The weights for the ensemble of models were determined using a hill-climbing algorithm base on data from 0008-02 to 0009-01.
\n| Model Architecture | \nNumber of Block | \nEmbedding Dimension | \nNumber of Parameters | \nCV | \nfile name | \n
|---|---|---|---|---|---|
| bi-LSTM + MLP | \n7 | \n8 | \n5070534 | \n0.71969 | \nv4_1 | \n
| bi-LSTM + MLP | \n17 | \n16 | \n49089054 | \n0.72703 | \nv4_1 | \n
| bi-LSTM + MLP + BN | \n7 | \n8 | \n5238534 | \n0.72493 | \nv4_8 | \n
| bi-GRU + MLP + BN | \n9 | \n8 | \n5286134 | \n0.72536 | \nv5_1 | \n
| bi-RNN + MLP + BN | \n17 | \n8 | \n4511734 | \n0.71692 | \nv8_1 | \n
| 1D-CNN + BN | \n11 | \n8 | \n16854334 | \n0.71459 | \nv6 | \n
| Ensemble | \n\n | \n | \n | 0.73472 (PB: 78491) | \n\n |
Based on my adversarial verification experiments, there is a significant difference(AUC=1) in data between years. According to the organizer's information, there should be one billion pieces of data in theory (but there are only about 80 million on HuggingFace). Therefore, it can be speculated that the test set is from data in the later years. Considering the above information, I used data from 0001-02 to 0008-01 for training, and data from 0008-02 to 0009-01 for validation. Perhaps this is why I was able to climb up the PB ranking and enter the gold medal zone.
\nThe original input had 556 columns. I removed 66 columns that only had one value, leaving 490 columns. For each column of data, I perform two operations: one is to directly standardize it (X - x_mean) / x_std, and the other is to apply log after taking the absolute value and then standardize it (CV +0.003). (log_X - log_x_mean) / log_x_std So the shape of the input for my model is (980,)
For each of the 980 inputs to the model, an embedding is assigned with an initialization range of [-0.05, 0.05]. This initialization range is crucial as other initialization methods may result in poorer performance.
\nBy padding and reshape, the input eventually becomes (bs, 60, 25*emb_dim).
\nFor bi-LSTM + MLP + BN, bi-GRU + MLP + BN, bi-RNN + MLP + BN, The model structures are all arranged in the following manner, with the only difference being the encoding layer.
pre_seq = seq \n i ((self.encoder_layers)):\n encoder_layer = self.encoder_layers[i]\n seq, _ = encoder_layer(seq) \n ffn = self.mlp[i]\n seq = ffn(seq) \n seq = seq.contiguous().view(bs, -) \n seq = self.mlp_bn[i](seq) \n seq = seq.contiguous().view(bs, , -) \n seq = F.gelu(seq)\n seq = seq + pre_seq \n pre_seq = seq\nFor 1D-CNN + BN, model structure as below
pre_x = x\n i ((self.cnns)):\n x = self.cnns[i](x) + pre_x\n pre_x = x\nThe complete code for the model can be found in the 'model' folder here. https://www.kaggle.com/datasets/uesugierii/leap-final?select=model
\nDuring the training process, it was found that training directly on the full dataset (low-res) resulted in lower scores. After performing EDA, it was discovered that the main cause of the unstable training was outliers.
\nTo address this, I employed two independent methods for mitigation:
\nnp.clip(x, -10, 10), np.clip(y, -10, 10)), first fitting on a smaller range (15 epochs, lr 1e-3), and then fitting on the original data (15 epochs, lr 1e-4). The rationale behind this is to first allow the model to learn and adapt to relatively normal weather conditions before generalizing to more extreme weather variations.To gain a slight improvement in performance towards the end, I would train the model for a few epochs on an extremely large batch size (10240) and a very low learning rate (1e-5). Some models were able to achieve further enhancements (CV +0.004).
\nptend trick: https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/499896#2791290
", "messages": [ { "id": 2940546, "postDate": "2024-07-30T07:24:53.777Z", "content": "Thanks to Learning the Earth with Artificial Intelligence and Physics (LEAP) NSF Science and Technology Center and Kaggle for hosting this fun competition, which was very stable in CV, LB and PB.
\nConsidering that my English is not particularly good, this article is being written with the assistance of chatgpt for translation. I will check and make corrections.
\ninfer code to prove that no leak: LEAP_11th_PB0.78491
\nBusiness context: https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/overviewData context: https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/dataMy final submission consisted of 6 models (with 5 different architectures) integrated together(important). All models use residual structures. All models are trained on the full dataset(low-res) from 0001-02 to 0008-01, and validated on approximately 1.7 million rows of data sampled from 0008-02 to 0009-01. Using both standardization, log transformation and embedding for input process. The models have undergone meticulous training through multiple stages(important). The weights for the ensemble of models were determined using a hill-climbing algorithm base on data from 0008-02 to 0009-01.
\n| Model Architecture | \nNumber of Block | \nEmbedding Dimension | \nNumber of Parameters | \nCV | \nfile name | \n
|---|---|---|---|---|---|
| bi-LSTM + MLP | \n7 | \n8 | \n5070534 | \n0.71969 | \nv4_1 | \n
| bi-LSTM + MLP | \n17 | \n16 | \n49089054 | \n0.72703 | \nv4_1 | \n
| bi-LSTM + MLP + BN | \n7 | \n8 | \n5238534 | \n0.72493 | \nv4_8 | \n
| bi-GRU + MLP + BN | \n9 | \n8 | \n5286134 | \n0.72536 | \nv5_1 | \n
| bi-RNN + MLP + BN | \n17 | \n8 | \n4511734 | \n0.71692 | \nv8_1 | \n
| 1D-CNN + BN | \n11 | \n8 | \n16854334 | \n0.71459 | \nv6 | \n
| Ensemble | \n\n | \n | \n | 0.73472 (PB: 78491) | \n\n |
Based on my adversarial verification experiments, there is a significant difference(AUC=1) in data between years. According to the organizer's information, there should be one billion pieces of data in theory (but there are only about 80 million on HuggingFace). Therefore, it can be speculated that the test set is from data in the later years. Considering the above information, I used data from 0001-02 to 0008-01 for training, and data from 0008-02 to 0009-01 for validation. Perhaps this is why I was able to climb up the PB ranking and enter the gold medal zone.
\nThe original input had 556 columns. I removed 66 columns that only had one value, leaving 490 columns. For each column of data, I perform two operations: one is to directly standardize it (X - x_mean) / x_std, and the other is to apply log after taking the absolute value and then standardize it (CV +0.003). (log_X - log_x_mean) / log_x_std So the shape of the input for my model is (980,)
For each of the 980 inputs to the model, an embedding is assigned with an initialization range of [-0.05, 0.05]. This initialization range is crucial as other initialization methods may result in poorer performance.
\nBy padding and reshape, the input eventually becomes (bs, 60, 25*emb_dim).
\nFor bi-LSTM + MLP + BN, bi-GRU + MLP + BN, bi-RNN + MLP + BN, The model structures are all arranged in the following manner, with the only difference being the encoding layer.
pre_seq = seq \n i ((self.encoder_layers)):\n encoder_layer = self.encoder_layers[i]\n seq, _ = encoder_layer(seq) \n ffn = self.mlp[i]\n seq = ffn(seq) \n seq = seq.contiguous().view(bs, -) \n seq = self.mlp_bn[i](seq) \n seq = seq.contiguous().view(bs, , -) \n seq = F.gelu(seq)\n seq = seq + pre_seq \n pre_seq = seq\nFor 1D-CNN + BN, model structure as below
pre_x = x\n i ((self.cnns)):\n x = self.cnns[i](x) + pre_x\n pre_x = x\nThe complete code for the model can be found in the 'model' folder here. https://www.kaggle.com/datasets/uesugierii/leap-final?select=model
\nDuring the training process, it was found that training directly on the full dataset (low-res) resulted in lower scores. After performing EDA, it was discovered that the main cause of the unstable training was outliers.
\nTo address this, I employed two independent methods for mitigation:
\nnp.clip(x, -10, 10), np.clip(y, -10, 10)), first fitting on a smaller range (15 epochs, lr 1e-3), and then fitting on the original data (15 epochs, lr 1e-4). The rationale behind this is to first allow the model to learn and adapt to relatively normal weather conditions before generalizing to more extreme weather variations.To gain a slight improvement in performance towards the end, I would train the model for a few epochs on an extremely large batch size (10240) and a very low learning rate (1e-5). Some models were able to achieve further enhancements (CV +0.004).
\nptend trick: https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/499896#2791290
", "rawMarkdown": "Thanks to Learning the Earth with Artificial Intelligence and Physics (LEAP) NSF Science and Technology Center and Kaggle for hosting this fun competition, which was very stable in CV, LB and PB.\n\nConsidering that my English is not particularly good, this article is being written with the assistance of chatgpt for translation. I will check and make corrections.\n\ninfer code to prove that no leak: [LEAP_11th_PB0.78491](https://www.kaggle.com/code/uesugierii/leap-11th-pb0-78491/notebook)\n\n## Context\n\n+ `Business context`: [https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/overview](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/overview)\n+ `Data context`: [https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/data](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/data)\n\n## Overview of the approach\n\nMy final submission consisted of 6 models (with 5 different architectures) integrated together(important). All models use residual structures. All models are trained on the full dataset(low-res) from 0001-02 to 0008-01, and validated on approximately 1.7 million rows of data sampled from 0008-02 to 0009-01. Using both standardization, log transformation and embedding for input process. The models have undergone meticulous training through multiple stages(important). The weights for the ensemble of models were determined using a hill-climbing algorithm base on data from 0008-02 to 0009-01.\n\n| Model Architecture | Number of Block | Embedding Dimension | Number of Parameters | CV | file name | \n|:------------------:|:---------------:|:-------------------:|:--------------------:|:-------:|:--------------------------------------------------------------------------:| \n| bi-LSTM + MLP | 7 | 8 | 5070534 | 0.71969 | [v4_1](https://www.kaggle.com/datasets/uesugierii/leap-final?select=model) | \n| bi-LSTM + MLP | 17 | 16 | 49089054 | 0.72703 | [v4_1](https://www.kaggle.com/datasets/uesugierii/leap-final?select=model) | \n| bi-LSTM + MLP + BN | 7 | 8 | 5238534 | 0.72493 | [v4_8](https://www.kaggle.com/datasets/uesugierii/leap-final?select=model) | \n| bi-GRU + MLP + BN | 9 | 8 | 5286134 | 0.72536 | [v5_1](https://www.kaggle.com/datasets/uesugierii/leap-final?select=model) | \n| bi-RNN + MLP + BN | 17 | 8 | 4511734 | 0.71692 | [v8_1](https://www.kaggle.com/datasets/uesugierii/leap-final?select=model) | \n| 1D-CNN + BN | 11 | 8 | 16854334 | 0.71459 | [v6](https://www.kaggle.com/datasets/uesugierii/leap-final?select=model) | \n| Ensemble | | | | 0.73472 (PB: 78491) | | \n\n## Solution Detail\n\n### data process detail\n\nBased on my adversarial verification experiments, there is a significant difference(AUC=1) in data between years. According to the organizer's information, there should be one billion pieces of data in theory (but there are only about 80 million on HuggingFace). Therefore, it can be speculated that the test set is from data in the later years. Considering the above information, I used data from 0001-02 to 0008-01 for training, and data from 0008-02 to 0009-01 for validation. Perhaps this is why I was able to climb up the PB ranking and enter the gold medal zone.\n\nThe original input had 556 columns. I removed 66 columns that only had one value, leaving 490 columns. For each column of data, I perform two operations: one is to directly standardize it `(X - x_mean) / x_std`, and the other is to apply log after taking the absolute value and then standardize it (CV +0.003). `(log_X - log_x_mean) / log_x_std` So the shape of the input for my model is (980,)\n\nFor each of the 980 inputs to the model, an embedding is assigned with an initialization range of [-0.05, 0.05]. This initialization range is **crucial** as other initialization methods may result in poorer performance.\n\nBy padding and reshape, the input eventually becomes (bs, 60, 25*emb_dim).\n\n### model detail (important)\n\nFor `bi-LSTM + MLP + BN`, `bi-GRU + MLP + BN`, `bi-RNN + MLP + BN`, The model structures are all arranged in the following manner, with the only difference being the encoding layer.\n\n```python\npre_seq = seq # (bs, 60, 25*emb_dim)\nfor i in range(len(self.encoder_layers)):\n encoder_layer = self.encoder_layers[i]\n seq, _ = encoder_layer(seq) # (bs, 60, 25*emb_dim*2)\n ffn = self.mlp[i]\n seq = ffn(seq) # (bs, 60, 25*emb_dim)\n seq = seq.contiguous().view(bs, -1) # (bs, 60*25*emb_dim)\n seq = self.mlp_bn[i](seq) # (bs, 60*25*emb_dim)\n seq = seq.contiguous().view(bs, 60, -1) # (bs, 60, 25*emb_dim)\n seq = F.gelu(seq)\n seq = seq + pre_seq # (bs, 60*25*emb_dim)\n pre_seq = seq\n```\n\nFor `1D-CNN + BN`, model structure as below\n\n```python\npre_x = x\nfor i in range(len(self.cnns)):\n x = self.cnns[i](x) + pre_x\n pre_x = x\n```\n\nThe complete code for the model can be found in the 'model' folder here. [https://www.kaggle.com/datasets/uesugierii/leap-final?select=model](https://www.kaggle.com/datasets/uesugierii/leap-final?select=model)\n\n### training detail (important)\n\nDuring the training process, it was found that training directly on the full dataset (low-res) resulted in lower scores. After performing EDA, it was discovered that the main cause of the unstable training was outliers.\n\nTo address this, I employed two independent methods for mitigation: \n\n1. First method involved clipping the value range of the input and target(`np.clip(x, -10, 10)`, `np.clip(y, -10, 10)`), first fitting on a smaller range (15 epochs, lr 1e-3), and then fitting on the original data (15 epochs, lr 1e-4). The rationale behind this is to first allow the model to learn and adapt to relatively normal weather conditions before generalizing to more extreme weather variations.\n2. Second method: first let the model train on one month's data(15 epochs, lr 1e-3), then on one year's data(15 epochs, lr 1e-3), and finally on all available data(15 epochs, lr 1e-4). The main idea behind this is to have the model initially learn the climate variations over short time spans, then gradually expand the time range to ensure a smoother learning process and reduce the difficulty of learning.\n\nTo gain a slight improvement in performance towards the end, I would train the model for a few epochs on an extremely large batch size (10240) and a very low learning rate (1e-5). Some models were able to achieve further enhancements (CV +0.004).\n\n### What didn't work for me\n\n+ Physics knowledge, neural network methods for solving partial differential equations\n + I'm not sure if it's my lack of understanding of these knowledge that's causing the lack of effect, or if there really don't work\n+ U-Net, U-Net++\n+ Transformers\n+ Squeezeformer\n+ Assist in training using target columns with weight 0\n+ More complex embedding schemes, such as bucketing + embedding, AutoDis, and so on\n\n## Sources\n\nptend trick: https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/499896#2791290\n", "votes": 11 }, { "id": 2940710, "postDate": "2024-07-30T11:28:45.077Z", "content": "Congrats on your 11th place finish, @uesugierii!
\nYour approach using multiple models and detailed training process is impressive. It's clear you put a lot of effort into optimizing your solution. Thanks for sharing your insights and methods.
Congrats on your 11th place finish, @uesugierii!
\nYour approach using multiple models and detailed training process is impressive. It's clear you put a lot of effort into optimizing your solution. Thanks for sharing your insights and methods.