Thanks Kaggle and host for organizing this competition, it was a good practice for training sequence models.
\nAlso thanks to my teammates @nischaydnk and @nikhilmishradev, I learned a lot from you!
We used simple transformer model (vanilla), but with couple of changes, -
\nWe first broadcast global features to match the shape of vertical features tensors, also we create some handcrafted features e.g. differences between adjacent timestamps for vertical features.
\nWe separate embedding projection between features from different so called spaces (we learn projection weights separately for vertical features, global features and hand-crafted features), after we just concat the embedding, apply abs. positional encoding and pass to transformer
\nSimple vanilla transformer w/ beit-v2 blocks, please refer to the 2nd place icecube solution, thanks @iafoss and @drhabib for publishing the code
\nInstead of taking the mean of transformer hidden states and applying MLP to get predicitons, we use fno1d layers to predict per-token targets, please refer to the neuraloperator lib, conv decoder should've worked as well
\nNum params of the model ~16M, n=8 transformer layers d=320 and m=4 fno1d layers.
\nOne of the main challenges of this competition was optimizing the code in way that ensures fast random reading from 80M samples dataset that takes 700Gb~ space.
\nWe used .hdf5 files for that, - we first create .hdf5 file for each month and during training, in each worker we open a separate instance of .h5py file for each month.
\nIt was improtant to have a fast reading drive (e.g. M2 NVME, simple SATA SSD didn’t work) in order to achieve good GPU utilization, with this setup whole training on 8x4090 took around 100Gb~ of RAM in DDP setting, which is a reasonable amount for such machine.
\nWe collected statistics from original kaggle data (mean/std) for both features and targets multipled by sample_sub weights and applied mean/std normalization whenever we trained our models (that be kaggle data or full low-res dataset)
\ntransformer-engine library is a very good thing if you want to train vanilla transformer (like here, when you don't need custom things such as learnable attention bias ref. recent Ribonanza competition)nvcr.io/nvidia/pytorch:24.05-py3 docker imageTarget was (x(t+1) - x(t))/1200,
x(t+1) directly, reconstruct original target laterTBA
", "messages": [ { "id": 2941753, "postDate": "2024-07-31T09:09:59.857Z", "content": "Thanks Kaggle and host for organizing this competition, it was a good practice for training sequence models.
\nAlso thanks to my teammates @nischaydnk and @nikhilmishradev, I learned a lot from you!
We used simple transformer model (vanilla), but with couple of changes, -
\nWe first broadcast global features to match the shape of vertical features tensors, also we create some handcrafted features e.g. differences between adjacent timestamps for vertical features.
\nWe separate embedding projection between features from different so called spaces (we learn projection weights separately for vertical features, global features and hand-crafted features), after we just concat the embedding, apply abs. positional encoding and pass to transformer
\nSimple vanilla transformer w/ beit-v2 blocks, please refer to the 2nd place icecube solution, thanks @iafoss and @drhabib for publishing the code
\nInstead of taking the mean of transformer hidden states and applying MLP to get predicitons, we use fno1d layers to predict per-token targets, please refer to the neuraloperator lib, conv decoder should've worked as well
\nNum params of the model ~16M, n=8 transformer layers d=320 and m=4 fno1d layers.
\nOne of the main challenges of this competition was optimizing the code in way that ensures fast random reading from 80M samples dataset that takes 700Gb~ space.
\nWe used .hdf5 files for that, - we first create .hdf5 file for each month and during training, in each worker we open a separate instance of .h5py file for each month.
\nIt was improtant to have a fast reading drive (e.g. M2 NVME, simple SATA SSD didn’t work) in order to achieve good GPU utilization, with this setup whole training on 8x4090 took around 100Gb~ of RAM in DDP setting, which is a reasonable amount for such machine.
\nWe collected statistics from original kaggle data (mean/std) for both features and targets multipled by sample_sub weights and applied mean/std normalization whenever we trained our models (that be kaggle data or full low-res dataset)
\ntransformer-engine library is a very good thing if you want to train vanilla transformer (like here, when you don't need custom things such as learnable attention bias ref. recent Ribonanza competition)nvcr.io/nvidia/pytorch:24.05-py3 docker imageTarget was (x(t+1) - x(t))/1200,
x(t+1) directly, reconstruct original target laterTBA
", "rawMarkdown": "Thanks Kaggle and host for organizing this competition, it was a good practice for training sequence models. \nAlso thanks to my teammates @nischaydnk and @nikhilmishradev, I learned a lot from you! \n\n### Model summary\n\nWe used simple transformer model (vanilla), but with couple of changes, - \n\n#### projection:\nWe first broadcast global features to match the shape of vertical features tensors, also we create some handcrafted features e.g. differences between adjacent timestamps for vertical features.\n\nWe separate embedding projection between features from different so called spaces (we learn projection weights separately for vertical features, global features and hand-crafted features), after we just concat the embedding, apply abs. positional encoding and pass to transformer\n\n#### encoder:\n\nSimple vanilla transformer w/ beit-v2 blocks, please refer to the [2nd place icecube solution](https://github.com/DrHB/icecube-2nd-place), thanks @iafoss and @drhabib for publishing the code\n\n#### decoder:\n\nInstead of taking the mean of transformer hidden states and applying MLP to get predicitons, we use fno1d layers to predict per-token targets, please refer to the [neuraloperator](https://github.com/neuraloperator/neuraloperator) lib, conv decoder should've worked as well\n\nNum params of the model ~16M, n=8 transformer layers d=320 and m=4 fno1d layers.\n\n### Training\n\n- use multiple losses applied to the single head (weighted), MSE and 0.1 * L1, we haven’t tried huberloss, should perform as well\n- AdamW with 4e-3, batch_size=4096, train for 5-7 epochs on the low-res dataset\n\n### Dataset\n\nOne of the main challenges of this competition was optimizing the code in way that ensures fast random reading from 80M samples dataset that takes 700Gb~ space. \n\nWe used .hdf5 files for that, - we first create .hdf5 file for each month and during training, in each worker we open a separate instance of .h5py file for each month. \n\nIt was improtant to have a fast reading drive (e.g. M2 NVME, simple SATA SSD didn’t work) in order to achieve good GPU utilization, with this setup whole training on 8x4090 took around 100Gb~ of RAM in DDP setting, which is a reasonable amount for such machine.\n\n### Normalization\nWe collected statistics from original kaggle data (mean/std) for both features and targets multipled by sample_sub weights and applied mean/std normalization whenever we trained our models (that be kaggle data or full low-res dataset)\n\n### Notes\n\n- It was important to use bf16/fp32 instead of fp16 training due to wide range of features even after normalization\n- `transformer-engine` library is a very good thing if you want to train vanilla transformer (like here, when you don't need custom things such as learnable attention bias ref. recent Ribonanza competition)\nIt takes way less memory than eager pytorch implementation allowing for larger batch sizes and generally we observed 30% speedup compared to vanilla transformer with flashattention implemented using eager torch, to run it we used latest `nvcr.io/nvidia/pytorch:24.05-py3` docker image\n\n### Tried, but didn't work\nTarget was `(x(t+1) - x(t))/1200`, \n\n- predict `x(t+1)` directly, reconstruct original target later\n- autoencoder pretrain (given t0, predict x(t0) when doing continious MLM)\n- train diffusion transformer, the loss was good, but the reconstruction quality suffered\n\n### code:\nTBA", "votes": 19 }, { "id": 2945014, "postDate": "2024-08-03T01:00:14.483Z", "content": "Hey @martynoveduard , thank you for publishing the solution and Congratulation to the Team for the Gold Medal !!
\nI had a question, as you have mentioned that it was important to use bf16/fp32 instead of fp16, even after normalizing the data first… How big of an impact did you observe on the score because of it?
", "rawMarkdown": "Hey @martynoveduard , thank you for publishing the solution and Congratulation to the Team for the Gold Medal !!\n\nI had a question, as you have mentioned that it was important to use bf16/fp32 instead of fp16, even after normalizing the data first... How big of an impact did you observe on the score because of it?", "replies": [ { "id": 2945082, "postDate": "2024-08-03T03:58:44.490Z", "content": "Thank you! For some old single model I observed quite a difference, -
\n| training precision | \npublic score | \nprivate score | \n
|---|---|---|
| bf16 | \n0.7796 | \n0.7766 | \n
| fp16 | \n0.7703 | \n0.7664 | \n
Hey @martynoveduard , thank you for publishing the solution and Congratulation to the Team for the Gold Medal !!
\nI had a question, as you have mentioned that it was important to use bf16/fp32 instead of fp16, even after normalizing the data first… How big of an impact did you observe on the score because of it?
", "votes": 0, "replies": [ { "id": 2945082, "author_name": "slime", "author_url": "", "post_date": "2024-08-03T03:58:44.490000", "content": "Thank you! For some old single model I observed quite a difference, -
\n| training precision | \npublic score | \nprivate score | \n
|---|---|---|
| bf16 | \n0.7796 | \n0.7766 | \n
| fp16 | \n0.7703 | \n0.7664 | \n