![\"JNet](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F17111680%2F15261fc27912dd8005aaaf6deeaf7101%2Fjnet.png?generation=1722330153587269&alt=media\")
\nCongratulations to all participants! I learned a lot in the past few months and want to thank Kaggle and the hosts for organizing this competition. It has been a truly rewarding experience, and I am excited to share my solution with you all.
\nOpen source code is available on github repo.
\nOur solution utilizes the PyTorch Lightning framework for training. All model architectures are based on the transformer encoder. Rotary positional encoding proved effective. Data preparation involved normalization, log transformation, min-max scaling, and clamping. Post-processing employed a column-wise ensemble method. While KAN linear embedding did not improve the score, it contributed as one of the models in the ensemble. We used 90% of the data for training and the remaining 10% for validation.
\nOur solution provides a convenient and neat approach for training machine learning models using the PTLit (PyTorch Lightning module) class. The training and validation loops are fully encapsulated within the class, simplifying the process. When multiple models are used, the class automatically averages their outputs to create an ensemble.
mdlit = PTLit(mask, learning_rate, step_size, gamma, [YOUR_MODEL()])\nmdlit = PTLit(mask, learning_rate, step_size, gamma, [YOUR_1ST_MODEL(), YOUR_2ND_MODEL()])\nbase_models = PTLit.load_from_checkpoint().models\nbase_models += PTLit.load_from_checkpoint().models\nmdlit = PTLit(mask, learning_rate, step_size, gamma, base_models)\nIn our experiment, switching from the original positional encoding to Rotary positional encoding (RoPE) resulted in an improvement of approximately 0.005 in the public score. Our best single model combined RoPE with a basic transformer encoder, achieving a score of 0.751 on the public leaderboard and 0.744 on the private leaderboard.
\nOur JNet model architecture processes the input sequence of shape (B, 60, 25) through a series of Conv1d layers, each followed by GELU activation functions, to extract hierarchical features. The outputs of these layers are concatenated into a combined feature representation, an embedding. This embedding is enriched with Rotary Positional Encoding (RoPE) to provide positional information and is subsequently processed by a transformer encoder. Finally, a linear layer generates the output sequence of shape (B, 60, 14).
\n\n \n
Our KANformer model architecture begins with an input sequence of shape (B, 60, 25) and applies a series of 60 KAN linear layers (25, 256) to each token of the sequence. These layers transform the input into an embedding of shape (B, 60, 256). This embedding is then processed by a transformer encoder to capture dependencies within the sequence. Finally, a linear layer generates the output sequence of shape (B, 60, 14). While KAN linear embedding did not improve the score, it contributed as one of the models in the ensemble.
\n\n ![\"KAN](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F17111680%2F3786e3b064eab9366cb451b3b4308cc8%2Fjnet.png?generation=1722329905316227&alt=media\")
\n
Without domain knowledge, we analyzed the distribution histograms of all feature columns one by one to determine the need for log transformation to address skewness. Afterward, we manually decided whether to apply normalization or min-max scaling to each feature column.
\nFor the label columns, direct normalization caused extreme values for the minimum and maximum. To mitigate this, we first clamped the label columns and then applied normalization. While clamping made it easier for the model to learn initially, it did not result in a better score after convergence. However, this approach can still be useful for curriculum learning. We included models trained on clamped data as part of the ensemble in the final submission.
\nWe used optuna to search for the best weights to ensemble models for each column. The code is as follows:
num_targets = label.size()\npreds_em = torch.zeros_like(preds[])\nalphas = torch.zeros(num_targets, num_models)\n\n ():\n weights = []\n remaining_sum = \n j (num_models - ):\n w_i = trial.suggest_float(, , remaining_sum, step=)\n w = w_i / \n weights.append(w)\n remaining_sum -= w_i\n col = (weight * preds[m][:, i] m, weight (weights))\n r2_score(col, labeln[:, i])\n\n i tqdm((num_targets)):\n mask[i]:\n study = optuna.create_study(direction=)\n study.optimize( trial: objective(trial, i), n_trials=)\n best_weights = []\n remaining_sum = \n j (num_models - ):\n w = study.best_params[]\n best_weights.append(w / )\n remaining_sum -= w\n best_weights.append(remaining_sum / ) \n alphas[i] = torch.tensor(best_weights)\n preds_em[:, i] = (weight * preds[m][:, i] m, weight (best_weights))\nWe experimented with changing the linear output layer to a convolutional output layer and a KAN linear output layer. However, these modifications increased the likelihood of overfitting. Due to time constraints, we were unable to implement adequate regularization techniques to mitigate this issue effectively.
\n[1] Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021, April 20). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv.org. https://arxiv.org/abs/2104.09864
\n[2] Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačić, M., Hou, T. Y., & Tegmark, M. (2024, April 30). KAN: Kolmogorov-Arnold Networks. arXiv.org. https://arxiv.org/abs/2404.19756
", "messages": [ { "id": 2940627, "postDate": "2024-07-30T09:04:09.013Z", "content": "Congratulations to all participants! I learned a lot in the past few months and want to thank Kaggle and the hosts for organizing this competition. It has been a truly rewarding experience, and I am excited to share my solution with you all.
\nOpen source code is available on github repo.
\nOur solution utilizes the PyTorch Lightning framework for training. All model architectures are based on the transformer encoder. Rotary positional encoding proved effective. Data preparation involved normalization, log transformation, min-max scaling, and clamping. Post-processing employed a column-wise ensemble method. While KAN linear embedding did not improve the score, it contributed as one of the models in the ensemble. We used 90% of the data for training and the remaining 10% for validation.
\nOur solution provides a convenient and neat approach for training machine learning models using the PTLit (PyTorch Lightning module) class. The training and validation loops are fully encapsulated within the class, simplifying the process. When multiple models are used, the class automatically averages their outputs to create an ensemble.
mdlit = PTLit(mask, learning_rate, step_size, gamma, [YOUR_MODEL()])\nmdlit = PTLit(mask, learning_rate, step_size, gamma, [YOUR_1ST_MODEL(), YOUR_2ND_MODEL()])\nbase_models = PTLit.load_from_checkpoint().models\nbase_models += PTLit.load_from_checkpoint().models\nmdlit = PTLit(mask, learning_rate, step_size, gamma, base_models)\nIn our experiment, switching from the original positional encoding to Rotary positional encoding (RoPE) resulted in an improvement of approximately 0.005 in the public score. Our best single model combined RoPE with a basic transformer encoder, achieving a score of 0.751 on the public leaderboard and 0.744 on the private leaderboard.
\nOur JNet model architecture processes the input sequence of shape (B, 60, 25) through a series of Conv1d layers, each followed by GELU activation functions, to extract hierarchical features. The outputs of these layers are concatenated into a combined feature representation, an embedding. This embedding is enriched with Rotary Positional Encoding (RoPE) to provide positional information and is subsequently processed by a transformer encoder. Finally, a linear layer generates the output sequence of shape (B, 60, 14).
\n\n \n
Our KANformer model architecture begins with an input sequence of shape (B, 60, 25) and applies a series of 60 KAN linear layers (25, 256) to each token of the sequence. These layers transform the input into an embedding of shape (B, 60, 256). This embedding is then processed by a transformer encoder to capture dependencies within the sequence. Finally, a linear layer generates the output sequence of shape (B, 60, 14). While KAN linear embedding did not improve the score, it contributed as one of the models in the ensemble.
\n\n \n
Without domain knowledge, we analyzed the distribution histograms of all feature columns one by one to determine the need for log transformation to address skewness. Afterward, we manually decided whether to apply normalization or min-max scaling to each feature column.
\nFor the label columns, direct normalization caused extreme values for the minimum and maximum. To mitigate this, we first clamped the label columns and then applied normalization. While clamping made it easier for the model to learn initially, it did not result in a better score after convergence. However, this approach can still be useful for curriculum learning. We included models trained on clamped data as part of the ensemble in the final submission.
\nWe used optuna to search for the best weights to ensemble models for each column. The code is as follows:
num_targets = label.size()\npreds_em = torch.zeros_like(preds[])\nalphas = torch.zeros(num_targets, num_models)\n\n ():\n weights = []\n remaining_sum = \n j (num_models - ):\n w_i = trial.suggest_float(, , remaining_sum, step=)\n w = w_i / \n weights.append(w)\n remaining_sum -= w_i\n col = (weight * preds[m][:, i] m, weight (weights))\n r2_score(col, labeln[:, i])\n\n i tqdm((num_targets)):\n mask[i]:\n study = optuna.create_study(direction=)\n study.optimize( trial: objective(trial, i), n_trials=)\n best_weights = []\n remaining_sum = \n j (num_models - ):\n w = study.best_params[]\n best_weights.append(w / )\n remaining_sum -= w\n best_weights.append(remaining_sum / ) \n alphas[i] = torch.tensor(best_weights)\n preds_em[:, i] = (weight * preds[m][:, i] m, weight (best_weights))\nWe experimented with changing the linear output layer to a convolutional output layer and a KAN linear output layer. However, these modifications increased the likelihood of overfitting. Due to time constraints, we were unable to implement adequate regularization techniques to mitigate this issue effectively.
\n[1] Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., & Liu, Y. (2021, April 20). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv.org. https://arxiv.org/abs/2104.09864
\n[2] Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačić, M., Hou, T. Y., & Tegmark, M. (2024, April 30). KAN: Kolmogorov-Arnold Networks. arXiv.org. https://arxiv.org/abs/2404.19756
", "rawMarkdown": "Congratulations to all participants! I learned a lot in the past few months and want to thank Kaggle and the hosts for organizing this competition. It has been a truly rewarding experience, and I am excited to share my solution with you all.\n\n## Context\n\n* Bussiness context:\n \n
\n \n
Nice work! Thanks for sharing code to use PyTorch lightning! Very helpful!
", "rawMarkdown": "Nice work! Thanks for sharing code to use PyTorch lightning! Very helpful!", "votes": 1 }, { "id": 2940895, "postDate": "2024-07-30T14:31:09.687Z", "content": "Thank you for sharing with us. I definitely try this code in this competition for learning.
", "rawMarkdown": "Thank you for sharing with us. I definitely try this code in this competition for learning.", "votes": 1 }, { "id": 2940758, "postDate": "2024-07-30T12:26:46.613Z", "content": "Impressive work, @lingjikong!
\nYour solution, with its use of PyTorch Lightning and column-wise ensemble, shows great attention to detail. The integration of Rotary Positional Encoding and your various model architectures, like JNet and KANformer, highlight a thoughtful approach to improving performance. I appreciate the detailed breakdown of your preprocessing and postprocessing techniques—it’s clear a lot of effort went into optimizing every step. Sharing your code and insights is incredibly valuable for the community.
\nKeep up the great work, and thanks for sharing your learnings.
Nice work! Thanks for sharing code to use PyTorch lightning! Very helpful!
", "votes": 1, "replies": [] }, { "id": 2940895, "author_name": "", "author_url": "", "post_date": "2024-07-30T14:31:09.687000", "content": "Thank you for sharing with us. I definitely try this code in this competition for learning.
", "votes": 1, "replies": [] }, { "id": 2940758, "author_name": "", "author_url": "", "post_date": "2024-07-30T12:26:46.613000", "content": "Impressive work, @lingjikong!
\nYour solution, with its use of PyTorch Lightning and column-wise ensemble, shows great attention to detail. The integration of Rotary Positional Encoding and your various model architectures, like JNet and KANformer, highlight a thoughtful approach to improving performance. I appreciate the detailed breakdown of your preprocessing and postprocessing techniques—it’s clear a lot of effort went into optimizing every step. Sharing your code and insights is incredibly valuable for the community.
\nKeep up the great work, and thanks for sharing your learnings.
\n \n
\n \n