Looking forward to publishing solutions. The very stable situation in the leaderboard gives reason to assume that the solutions will be interesting
", "messages": [ { "id": 2924119, "postDate": "2024-07-16T10:45:23.977Z", "content": "i hade a simple transformer encoder with input (N, 60, 25), only thing that might be a bit \"sophisticated\" was that i was also passing the input variable wise (N, 25, 60) with learned embeddings into a second transformer encoder and then do cross attention with the result of the first encoder. I did not have time to study if the extra encoder + cross attention gave any value though.
\n ():\n \n src2 = src1.permute(, , ) \n \n src1 = self.input_adapter1(src1) \n src2 = self.input_adapter2(src2) \n\n \n src1 = self.positional_encoding_1(src1)\n src2 = self.learned_encoding_2(src2)\n\n \n encoded1 = self.encoder1(src1)\n encoded2 = self.encoder2(src2)\n\n \n cross_attended1 = self.cross_attention_1(encoded1, encoded2, encoded2).flatten(start_dim=)\n cross_attended2 = self.cross_attention_2(encoded2, encoded1, encoded1).flatten(start_dim=)\n\n \n combined = torch.cat([cross_attended1, cross_attended2], dim=)\n\n output = self.output_linear(combined)\n output\nThanks! Have you considered any other permutations? how much has this feature improved?
", "rawMarkdown": "Thanks! Have you considered any other permutations? how much has this feature improved?", "replies": [ { "id": 2924200, "postDate": "2024-07-16T11:42:59.860Z", "content": "hey no, i am not sure how the extra thingy with the cross attention adds, i had no time to try without it. What i did try is to have conv1d layers between the transformer encoder layers but i did not manage to make it work better in a few attempts and since there were only a few
\ndays left i did not try further
Neat trick! How did you come up with the idea of using cross-attention? I used a single encoder model, but the score wasn't as good as yours.
", "rawMarkdown": "Neat trick! How did you come up with the idea of using cross-attention? I used a single encoder model, but the score wasn't as good as yours. ", "votes": 1, "replies": [ { "id": 2936544, "postDate": "2024-07-26T08:51:40.897Z", "content": "i just was fixated that i need to do something also variable-wise and then somehow concat the result, well cross_attention seemed a better idea. but maybe you had worse score for some other reason.
", "rawMarkdown": "i just was fixated that i need to do something also variable-wise and then somehow concat the result, well cross_attention seemed a better idea. but maybe you had worse score for some other reason.", "votes": 1 } ] } ] }, { "id": 2936456, "postDate": "2024-07-26T06:55:19.917Z", "content": "I was only 93rd in the competition, but I used a convolutional neural network with a big twist: it also ingested the predictions of a CatBoost decision tree model as additional input features. So the two techniques were effectively in series. (I also tried combining their predictions as various weighted sums, which didn't help, as the CNN was just better.)
\nApart from the actual machine learning, I found the data handling a challenge on the computer available to me, and one notable thing I did was write a class which gave me fast random access to the whole huge training file, using low-level file operations, so I could access it in arbitrary batches if I wanted to. I'd be interested to know what kind of hardware the more successful teams used for this challenge.
\nWith a quiet weekend at home with Covid, I indulged myself my writing up my adventures in a report. As I am nowhere near the top of the leaderboard, it's all pubic here: https://github.com/charlie-wartnaby/kaggle-leap-atmospheric-physics
", "rawMarkdown": "I was only 93rd in the competition, but I used a convolutional neural network with a big twist: it also ingested the predictions of a CatBoost decision tree model as additional input features. So the two techniques were effectively in series. (I also tried combining their predictions as various weighted sums, which didn't help, as the CNN was just better.)\n\nApart from the actual machine learning, I found the data handling a challenge on the computer available to me, and one notable thing I did was write a class which gave me fast random access to the whole huge training file, using low-level file operations, so I could access it in arbitrary batches if I wanted to. I'd be interested to know what kind of hardware the more successful teams used for this challenge.\n\nWith a quiet weekend at home with Covid, I indulged myself my writing up my adventures in a report. As I am nowhere near the top of the leaderboard, it's all pubic here: https://github.com/charlie-wartnaby/kaggle-leap-atmospheric-physics", "votes": 1, "replies": [ { "id": 2936546, "postDate": "2024-07-26T08:54:31.317Z", "content": "for me the trick was to use polars and to read batches of the big csv, train then load another df which was another chunk of the csv. It is easy in polars and the time took to process these dfs was only like 2-3 seconds while the training took much more time so efficiency because of that was not an issue. https://github.com/VCharatsidis/kaggle_LEAP/blob/master/seq2scalar/train_weightless_last_efficient.py
", "rawMarkdown": "for me the trick was to use polars and to read batches of the big csv, train then load another df which was another chunk of the csv. It is easy in polars and the time took to process these dfs was only like 2-3 seconds while the training took much more time so efficiency because of that was not an issue. https://github.com/VCharatsidis/kaggle_LEAP/blob/master/seq2scalar/train_weightless_last_efficient.py", "votes": 3 } ] }, { "id": 2924022, "postDate": "2024-07-16T09:00:24.310Z", "content": "Looking forward to publishing solutions. The very stable situation in the leaderboard gives reason to assume that the solutions will be interesting
", "rawMarkdown": "Looking forward to publishing solutions. The very stable situation in the leaderboard gives reason to assume that the solutions will be interesting", "votes": 2 } ], "comments": [ { "id": 2924119, "author_name": "Vasilis", "author_url": "", "post_date": "2024-07-16T10:45:23.977000", "content": "i hade a simple transformer encoder with input (N, 60, 25), only thing that might be a bit \"sophisticated\" was that i was also passing the input variable wise (N, 25, 60) with learned embeddings into a second transformer encoder and then do cross attention with the result of the first encoder. I did not have time to study if the extra encoder + cross attention gave any value though.
\n ():\n \n src2 = src1.permute(, , ) \n \n src1 = self.input_adapter1(src1) \n src2 = self.input_adapter2(src2) \n\n \n src1 = self.positional_encoding_1(src1)\n src2 = self.learned_encoding_2(src2)\n\n \n encoded1 = self.encoder1(src1)\n encoded2 = self.encoder2(src2)\n\n \n cross_attended1 = self.cross_attention_1(encoded1, encoded2, encoded2).flatten(start_dim=)\n cross_attended2 = self.cross_attention_2(encoded2, encoded1, encoded1).flatten(start_dim=)\n\n \n combined = torch.cat([cross_attended1, cross_attended2], dim=)\n\n output = self.output_linear(combined)\n output\nThanks! Have you considered any other permutations? how much has this feature improved?
", "votes": 0, "replies": [ { "id": 2924200, "author_name": "Vasilis", "author_url": "", "post_date": "2024-07-16T11:42:59.860000", "content": "hey no, i am not sure how the extra thingy with the cross attention adds, i had no time to try without it. What i did try is to have conv1d layers between the transformer encoder layers but i did not manage to make it work better in a few attempts and since there were only a few
\ndays left i did not try further
Neat trick! How did you come up with the idea of using cross-attention? I used a single encoder model, but the score wasn't as good as yours.
", "votes": 1, "replies": [ { "id": 2936544, "author_name": "Vasilis", "author_url": "", "post_date": "2024-07-26T08:51:40.897000", "content": "i just was fixated that i need to do something also variable-wise and then somehow concat the result, well cross_attention seemed a better idea. but maybe you had worse score for some other reason.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2936456, "author_name": "Charlie Wartnaby", "author_url": "", "post_date": "2024-07-26T06:55:19.917000", "content": "I was only 93rd in the competition, but I used a convolutional neural network with a big twist: it also ingested the predictions of a CatBoost decision tree model as additional input features. So the two techniques were effectively in series. (I also tried combining their predictions as various weighted sums, which didn't help, as the CNN was just better.)
\nApart from the actual machine learning, I found the data handling a challenge on the computer available to me, and one notable thing I did was write a class which gave me fast random access to the whole huge training file, using low-level file operations, so I could access it in arbitrary batches if I wanted to. I'd be interested to know what kind of hardware the more successful teams used for this challenge.
\nWith a quiet weekend at home with Covid, I indulged myself my writing up my adventures in a report. As I am nowhere near the top of the leaderboard, it's all pubic here: https://github.com/charlie-wartnaby/kaggle-leap-atmospheric-physics
", "votes": 1, "replies": [ { "id": 2936546, "author_name": "Vasilis", "author_url": "", "post_date": "2024-07-26T08:54:31.317000", "content": "for me the trick was to use polars and to read batches of the big csv, train then load another df which was another chunk of the csv. It is easy in polars and the time took to process these dfs was only like 2-3 seconds while the training took much more time so efficiency because of that was not an issue. https://github.com/VCharatsidis/kaggle_LEAP/blob/master/seq2scalar/train_weightless_last_efficient.py
", "votes": 3, "replies": [] } ] } ], "raw_markdown_by_id": { "2924119": "i hade a simple transformer encoder with input (N, 60, 25), only thing that might be a bit \"sophisticated\" was that i was also passing the input variable wise (N, 25, 60) with learned embeddings into a second transformer encoder and then do cross attention with the result of the first encoder. I did not have time to study if the extra encoder + cross attention gave any value though.\n\n```\n def forward(self, src1):\n # Reshape and reorder to N, 25, 60\n src2 = src1.permute(0, 2, 1) # Change to N, 25, 60\n # Prepare inputs\n src1 = self.input_adapter1(src1) # N, 60, 25 -> N, 60, d_model\n src2 = self.input_adapter2(src2) # N, 25, 60 -> N, 25, d_model\n\n # Add positional encoding\n src1 = self.positional_encoding_1(src1)\n src2 = self.learned_encoding_2(src2)\n\n # Encode with transformers\n encoded1 = self.encoder1(src1)\n encoded2 = self.encoder2(src2)\n\n # Apply cross-attention\n cross_attended1 = self.cross_attention_1(encoded1, encoded2, encoded2).flatten(start_dim=1)\n cross_attended2 = self.cross_attention_2(encoded2, encoded1, encoded1).flatten(start_dim=1)\n\n # Combine outputs from both paths\n combined = torch.cat([cross_attended1, cross_attended2], dim=1)\n\n output = self.output_linear(combined)\n return output\n```", "2936456": "I was only 93rd in the competition, but I used a convolutional neural network with a big twist: it also ingested the predictions of a CatBoost decision tree model as additional input features. So the two techniques were effectively in series. (I also tried combining their predictions as various weighted sums, which didn't help, as the CNN was just better.)\n\nApart from the actual machine learning, I found the data handling a challenge on the computer available to me, and one notable thing I did was write a class which gave me fast random access to the whole huge training file, using low-level file operations, so I could access it in arbitrary batches if I wanted to. I'd be interested to know what kind of hardware the more successful teams used for this challenge.\n\nWith a quiet weekend at home with Covid, I indulged myself my writing up my adventures in a report. As I am nowhere near the top of the leaderboard, it's all pubic here: https://github.com/charlie-wartnaby/kaggle-leap-atmospheric-physics", "2924022": "Looking forward to publishing solutions. The very stable situation in the leaderboard gives reason to assume that the solutions will be interesting" } }