![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7160377%2F0c6c734c99c85c76f40e3835532442c6%2Ftimeformer_input.png?generation=1759276778105413&alt=media\")
First of all, I want to express my gratitude to the organizers for hosting this competition. I had a fantastic time working on it.
\nAnd also I was a part of a great team with @solverworld @vitalykudelya @asimandia. Thank you guys, you are the best!
\nOur final solution is composed of three entirely different parts. In this post, I will describe the Deep Learning part.
\nIt utilizes two models, which I've named TimeFormer and PlanetFormer.
\nTimeFormer processes three adjacent signals (5625x3) as input and predicts three corresponding mean and sigma values.
\nIt also has a head that predicts signal zones over time. For zone labeling, I adapted an algorithm from last year's winning solution and tuned it for the new dataset.
\n
To further improve the zone interpretation, @asimandia spent several sleepless nights manually labeling the zones (t1, t2, t3, t4) for the entire dataset. This manual annotation was used during model training and provided a slight performance boost.
\nWe had planned to use these labels to enhance the signal zone detector for better inference quality, but unfortunately, we ran out of time.
\nDuring inference, the model is applied with an overlap, and its predictions for neighboring signals are averaged.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7160377%2F7274145ef0c9fc1d37af22cc2bac920d%2Ftimeformer.png?generation=1759276890354426&alt=media\")
On its own, TimeFormer can be used for prediction and achieves a public leaderboard score of approximately 0.46. However, it has limitations in capturing the global context.
\nTo analyze the entire signal, I run TimeFormer on each wave and extract the post-pooling embedding. This embedding provides a good temporal description of the signal. This process transforms a (5625x283) signal into 283 embeddings (32x283).
\nFun observation: Here is a PCA visualization of the TimeFormer embeddings. I find it a bit suspicious that in a competition about space, the embeddings looks like the Alien :)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7160377%2Fc31d2bf22c5573e2a2a176c573a93d88%2Ftimeformer_embeddings.png?generation=1759277189238192&alt=media\")
PlanetFormer takes the 283 embeddings from TimeFormer as input and outputs 283 mean and 283 std values.
\nThe target variable exhibits a highly consistent structure, which can be leveraged to reduce the number of model parameters and mitigate overfitting. Instead of predicting 283 distinct values, the mean head predicts only 21 coefficients for a 20th-degree polynomial used to model the target.
\nSimilarly, the sigma head (which actually predicts log_var for numerical stability) outputs just 7 values instead of 283. The underlying idea is that uncertainty is likely concentrated in specific, adjacent groups of waves.
I model this uncertainty using a set of basis functions: torch.ones(), sin(pi * x), cos(pi * x), sin(2pi * x), cos(2pi * x), sin(3pi * x), and cos(3pi * x). The output of the log_var head determines the weights for combining these functions.
Both the mean and sigma heads also utilize MoE for their predictions.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7160377%2F53d7e19e00418f3a9f5d2d82cb53fc84%2Fplanetformer.png?generation=1759277234626437&alt=media\")
PlanetFormer achieves a public LB score of approximately 0.48-0.49.
\nTo further enhance prediction accuracy, the outputs of TimeFormer and PlanetFormer are ensembled with weights of [0.15, 0.85].
\nAs indicated earlier, the models operate with binning=1. Before being fed into the models, the signal is processed with a low-pass filter. Transit zone was centered. Additionally, a \"time-axis flip\" augmentation was applied during the training of both models.
Both models were trained using a custom loss function that I designed to closely approximate the competition's evaluation metric. The FGS channel is assigned a weight of 57.8641. Here is the code for the loss:
\n (nn.Module):\n ():\n ().__init__()\n .naive_mean = naive_mean\n .naive_sigma = naive_sigma\n .fsg_sigma_true = fsg_sigma_true\n .airs_sigma_true = airs_sigma_true\n\n ():\n pred_sigma = torch.sqrt(torch.exp(log_var_pred)).clamp(=)\n\n neg_log_likelihood = * (\n torch.log( * np.pi * pred_sigma.()) + \n (target - pred_mean).() / pred_sigma.()\n )\n\n sigma_penalty = torch.where(\n pred_sigma < .airs_sigma_true,\n (.airs_sigma_true - pred_sigma).() * ,\n torch.zeros_like(pred_sigma)\n )\n\n \n fgs_mask = torch.where(weights > )\n sigma_penalty[fgs_mask] = torch.where(\n pred_sigma[fgs_mask] < .fsg_sigma_true,\n (.fsg_sigma_true - pred_sigma[fgs_mask]).() * ,\n sigma_penalty[fgs_mask]\n )\n\n total_loss = (neg_log_likelihood + sigma_penalty) * weights\n\n total_loss.mean()\nFirst of all, I want to express my gratitude to the organizers for hosting this competition. I had a fantastic time working on it.
\nAnd also I was a part of a great team with @solverworld @vitalykudelya @asimandia. Thank you guys, you are the best!
\nOur final solution is composed of three entirely different parts. In this post, I will describe the Deep Learning part.
\nIt utilizes two models, which I've named TimeFormer and PlanetFormer.
\nTimeFormer processes three adjacent signals (5625x3) as input and predicts three corresponding mean and sigma values.
\nIt also has a head that predicts signal zones over time. For zone labeling, I adapted an algorithm from last year's winning solution and tuned it for the new dataset.
\n
To further improve the zone interpretation, @asimandia spent several sleepless nights manually labeling the zones (t1, t2, t3, t4) for the entire dataset. This manual annotation was used during model training and provided a slight performance boost.
\nWe had planned to use these labels to enhance the signal zone detector for better inference quality, but unfortunately, we ran out of time.
\nDuring inference, the model is applied with an overlap, and its predictions for neighboring signals are averaged.
\n
On its own, TimeFormer can be used for prediction and achieves a public leaderboard score of approximately 0.46. However, it has limitations in capturing the global context.
\nTo analyze the entire signal, I run TimeFormer on each wave and extract the post-pooling embedding. This embedding provides a good temporal description of the signal. This process transforms a (5625x283) signal into 283 embeddings (32x283).
\nFun observation: Here is a PCA visualization of the TimeFormer embeddings. I find it a bit suspicious that in a competition about space, the embeddings looks like the Alien :)
\n
PlanetFormer takes the 283 embeddings from TimeFormer as input and outputs 283 mean and 283 std values.
\nThe target variable exhibits a highly consistent structure, which can be leveraged to reduce the number of model parameters and mitigate overfitting. Instead of predicting 283 distinct values, the mean head predicts only 21 coefficients for a 20th-degree polynomial used to model the target.
\nSimilarly, the sigma head (which actually predicts log_var for numerical stability) outputs just 7 values instead of 283. The underlying idea is that uncertainty is likely concentrated in specific, adjacent groups of waves.
I model this uncertainty using a set of basis functions: torch.ones(), sin(pi * x), cos(pi * x), sin(2pi * x), cos(2pi * x), sin(3pi * x), and cos(3pi * x). The output of the log_var head determines the weights for combining these functions.
Both the mean and sigma heads also utilize MoE for their predictions.
\n
PlanetFormer achieves a public LB score of approximately 0.48-0.49.
\nTo further enhance prediction accuracy, the outputs of TimeFormer and PlanetFormer are ensembled with weights of [0.15, 0.85].
\nAs indicated earlier, the models operate with binning=1. Before being fed into the models, the signal is processed with a low-pass filter. Transit zone was centered. Additionally, a \"time-axis flip\" augmentation was applied during the training of both models.
Both models were trained using a custom loss function that I designed to closely approximate the competition's evaluation metric. The FGS channel is assigned a weight of 57.8641. Here is the code for the loss:
\n (nn.Module):\n ():\n ().__init__()\n .naive_mean = naive_mean\n .naive_sigma = naive_sigma\n .fsg_sigma_true = fsg_sigma_true\n .airs_sigma_true = airs_sigma_true\n\n ():\n pred_sigma = torch.sqrt(torch.exp(log_var_pred)).clamp(=)\n\n neg_log_likelihood = * (\n torch.log( * np.pi * pred_sigma.()) + \n (target - pred_mean).() / pred_sigma.()\n )\n\n sigma_penalty = torch.where(\n pred_sigma < .airs_sigma_true,\n (.airs_sigma_true - pred_sigma).() * ,\n torch.zeros_like(pred_sigma)\n )\n\n \n fgs_mask = torch.where(weights > )\n sigma_penalty[fgs_mask] = torch.where(\n pred_sigma[fgs_mask] < .fsg_sigma_true,\n (.fsg_sigma_true - pred_sigma[fgs_mask]).() * ,\n sigma_penalty[fgs_mask]\n )\n\n total_loss = (neg_log_likelihood + sigma_penalty) * weights\n\n total_loss.mean()\nKudos for predicting poly coefficients!
\nThat was one of the first things I tried but I couldn't get it working
\nNice to see another attention based solution
Thanks! A huge congrats to you as well!
", "rawMarkdown": "Thanks! A huge congrats to you as well!" } ] } ], "comments": [ { "id": 3296516, "author_name": "sroger", "author_url": "", "post_date": "2025-10-01T04:42:18.423000", "content": "Kudos for predicting poly coefficients!
\nThat was one of the first things I tried but I couldn't get it working
\nNice to see another attention based solution
Thanks! A huge congrats to you as well!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "3296448": "First of all, I want to express my gratitude to the organizers for hosting this competition. I had a fantastic time working on it. \n\nAnd also I was a part of a great team with @solverworld @vitalykudelya @asimandia. Thank you guys, you are the best!\n\nOur final solution is composed of three entirely different parts. In this post, I will describe the Deep Learning part.\n\nIt utilizes two models, which I've named **TimeFormer** and **PlanetFormer**.\n\n-----\n\n## TimeFormer\n\nTimeFormer processes three adjacent signals (5625x3) as input and predicts three corresponding mean and sigma values.\n\nIt also has a head that predicts signal zones over time. For zone labeling, I adapted an algorithm from last year's winning solution and tuned it for the new dataset.\n\n\n\nTo further improve the zone interpretation, @asimandia spent several sleepless nights manually labeling the zones (t1, t2, t3, t4) for the entire dataset. This manual annotation was used during model training and provided a slight performance boost.\n\nWe had planned to use these labels to enhance the signal zone detector for better inference quality, but unfortunately, we ran out of time.\n\nDuring inference, the model is applied with an overlap, and its predictions for neighboring signals are averaged.\n\n### Architectural Tricks:\n\n * **Learnable embeddings** for each zone, complementing the standard positional embeddings.\n * A hybrid approach using both **convolutions with various kernel sizes and attention**.\n * **Zone-Aware Attention**: I injected prior knowledge about statistically important signal zones directly into the attention mechanism. This was achieved by adding a learned bias to the query and key vectors to guide the model's focus: $$Q' = Q + \\alpha \\cdot \\text{proj}(E_{\\text{zone}})$$\n * **Gated Residual Connections**\n * **Mixture of Experts (MoE) Heads**\n * A dedicated head that learns to **predict signal zones** after the transformer layer.\n\n\n\nOn its own, TimeFormer can be used for prediction and achieves a public leaderboard score of approximately **0.46**. However, it has limitations in capturing the global context.\n\nTo analyze the entire signal, I run TimeFormer on each wave and extract the post-pooling embedding. This embedding provides a good temporal description of the signal. This process transforms a (5625x283) signal into 283 embeddings (32x283).\n\n**Fun observation**: Here is a PCA visualization of the TimeFormer embeddings. I find it a bit suspicious that in a competition about space, the embeddings looks like the Alien :)\n\n\n\n-----\n\n## PlanetFormer\n\nPlanetFormer takes the 283 embeddings from TimeFormer as input and outputs 283 mean and 283 std values.\n\nThe target variable exhibits a highly consistent structure, which can be leveraged to reduce the number of model parameters and mitigate overfitting. Instead of predicting 283 distinct values, the **mean head** predicts only 21 coefficients for a 20th-degree polynomial used to model the target.\n\nSimilarly, the **sigma head** (which actually predicts `log_var` for numerical stability) outputs just 7 values instead of 283. The underlying idea is that uncertainty is likely concentrated in specific, adjacent groups of waves.\n\nI model this uncertainty using a set of basis functions: `torch.ones()`, `sin(pi * x)`, `cos(pi * x)`, `sin(2pi * x)`, `cos(2pi * x)`, `sin(3pi * x)`, and `cos(3pi * x)`. The output of the `log_var` head determines the weights for combining these functions.\n\nBoth the mean and sigma heads also utilize MoE for their predictions.\n\n\n\nPlanetFormer achieves a public LB score of approximately **0.48-0.49**.\n\nTo further enhance prediction accuracy, the outputs of TimeFormer and PlanetFormer are ensembled with weights of **[0.15, 0.85]**.\n\n-----\n\n## Signal Preprocessing and Augmentations\n\nAs indicated earlier, the models operate with `binning=1`. Before being fed into the models, the signal is processed with a **low-pass filter**. Transit zone was **centered**. Additionally, a **\"time-axis flip\" augmentation** was applied during the training of both models.\n\n-----\n\n## Loss Function\n\nBoth models were trained using a custom loss function that I designed to closely approximate the competition's evaluation metric. The FGS channel is assigned a weight of 57.8641. Here is the code for the loss:\n\n```python\nclass CompetitionLoss(nn.Module):\n def __init__(self, naive_mean=1.46890195e-02, naive_sigma=1.06613525e-02,\n fsg_sigma_true=1e-6, airs_sigma_true=1e-5):\n super().__init__()\n self.naive_mean = naive_mean\n self.naive_sigma = naive_sigma\n self.fsg_sigma_true = fsg_sigma_true\n self.airs_sigma_true = airs_sigma_true\n \n def forward(self, pred_mean, log_var_pred, target, weights):\n pred_sigma = torch.sqrt(torch.exp(log_var_pred)).clamp(min=1e-15)\n\n neg_log_likelihood = 0.5 * (\n torch.log(2 * np.pi * pred_sigma.pow(2)) + \n (target - pred_mean).pow(2) / pred_sigma.pow(2)\n )\n\n sigma_penalty = torch.where(\n pred_sigma < self.airs_sigma_true,\n (self.airs_sigma_true - pred_sigma).pow(2) * 100,\n torch.zeros_like(pred_sigma)\n )\n \n # Extra penalty for FGS1 channel\n fgs_mask = torch.where(weights > 1)\n sigma_penalty[fgs_mask] = torch.where(\n pred_sigma[fgs_mask] < self.fsg_sigma_true,\n (self.fsg_sigma_true - pred_sigma[fgs_mask]).pow(2) * 1000,\n sigma_penalty[fgs_mask]\n )\n \n total_loss = (neg_log_likelihood + sigma_penalty) * weights\n \n return total_loss.mean()\n```", "3296516": "Kudos for predicting poly coefficients!\nThat was one of the first things I tried but I couldn't get it working\nNice to see another attention based solution" } }