![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F11013868%2Fe4232f20af128341677ff6cd50855a13%2Fairs_foreground.png?generation=1759905097463948&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F11013868%2F8882fbdfb638672c375ef830a02c2424%2Ffgs1_foreground.png?generation=1759905119086938&alt=media\")
The code for my solution is available in this public github repository.
\nI have documented my solution in this pdf file. The description became quite long, so I am making a shorter writeup. If you want all the details you should read the pdf file.
\nMy Kaggle notebook is available here.
\nThe idea for my solution is to implement a model that mimics the physics of a planet moving in front of a star, while explicitly taking limb darkening into account. This is done to obtain high quality estimates of the transit depth, which then require minimal use of modeling later in the pipeline. In fact, the final depth estimates for the submission are simply constructed with linear regression on answers from the physical model. In particular, the depth is estimated independently of the standard deviation. The standard deviation is modeled with small neural networks. These downstream models were required to be simple due to the small number of planets in the training data.
\nBefore applying the physics-based model, a starting point for the search is provided by a simpler model that involves Kalman Smoothers. I also implemented preprocessing similar to what was suggested by the organizers. Finally, I made use of principal component analysis to refine the spectrum produced by the physical model.
\nAt a high level, my solution contains the following elements:
\nAll in all, my approach ended up being relatively complex, but I believe that the use of explicit modeling of physics may be worth the effort to obtain good results. The main downside is that such an approach is unlikely to generalize to other problems, which makes it inherently less interesting.
\nI generally follow the approach suggested by the organizers. I perform no binning, so the time series for FGS1 have length 67500, and the time series for AIRS have length 5625.
\nI treat hot and dead pixels as missing values. I replace such missing values with linear interpolation of neighboring pixels, four neighbors for FGS1 and two neighbors with the same wavelength for AIRS. Note that clusters of missing values remain missing. If one of the four most significant pixels are missing for AIRS, I remove that entire wavelength. The later model is set up to handle such missing values.
\nI use a rolling window of length 255 to discard outliers beyond five standard deviations from the centered rolling mean, again treating such values as missing.
\nInspired by last year's first place solution, I use the rim of the image to estimate foreground noise, which I then subtract from the center of the image. Except, I do not explicitly use the rim as in last year's first place solution. Instead I use the six rows with lowest intensity for AIRS, and the six columns and then six rows with lowest intensity for FGS1. The following figure illustrates the foreground noise processing step for planet 34983. Note that the foreground noise is not uniform, so I may actually throw away too much of the data.
\nI was pressed on time and did not put effort into verifying that this form of preprocessing was beneficial. It seemed reasonable, so I spent my time elsewhere.
\n
\n
My main model explicitly takes into account limb darkening, but it is non-trivial to optimize without ending up in an undesired local optimum. To improve the search, an initial solution is provided by a simpler model. This initial model was my baseline submission, and it is likely more complex than it needs to be when further refinement is performed later.
\nThe idea for the initial model is to apply a Kalman Smoother to a Local Linear Trend model, to get a very smooth underlying level and trend from which the beginning and end of the transit can easily be extracted. The Kalman Smoother is designed to handle missing values, and the computation is performed twice: Once with all the data, and once with the data from the transit removed (set to NaN). The depth of the transit is extracted from the difference of the levels resulting from these two computations.
\nThe beginning, middle, and end of the transit are identified with the first Kalman Smoother. I identify as breakpoints the indices of the minimum and maximum values of the trend component. The middle of the transit is defined to be the average over these breakpoints, including both AIRS and FGS1. For AIRS, I use the sum across the entire spectrum to decrease noise. The beginning of the transit is the latest point prior to the first breakpoint where either the trend reaches zero, or the one-step difference of the trend is above that of the breakpoint. The end of the transit is defined similarly, but for the earliest point after the second breakpoint. The data from AIRS and FGS1 is combined by choosing the latest of the two beginnings and the earliest of the two ends. Transit identification for planet 34983 is illustrated in the following figure.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F11013868%2Fb58825be75e47ddee8febdc4665d0e52%2Ftransit_identification.png?generation=1759905402945608&alt=media\"){kind=spacer}
My model that explicitly takes into account limb darkening simulates a planet moving in front of a star, and estimates the amount of light observed at every time step. Limb darkening means that the intensity of the star is highest at the center and is reduced toward the edge (the limb). The formula that I use for limb darkening is based on this note. The exact formula used is:
\nintensity(x, u, v) = 1 - u * ((1 - v) * (1 - \\sqrt{1 - x^2}) + v * x^2),
\nwhere x is the distance to the center of the star, assuming that the star has radius 1, and u and v are parameters that depend on the star. Unfortunately, I made a sign error in my code and changed the last plus into a minus, which meant that v behaved unexpectedly, so I gave up on directly optimizing v and instead treated it as a hyperparameter.
\nThe model for constructing the transit uses six parameters:
\nImportantly, the transit is differentiable with respect to these six parameters, which means that the parameters can be optimized to produce the transit with the best fit for some training data.
\nThe following figures show examples of the transits generated by my model for the same parameters, without and with limb darkening. Note that without limb darkening, the valley of the transit is very flat, because the observed light mainly depends on whether the planet is fully inside the boundary of the star or not. With limb darkening, the observed light continues being reduced as the planet gets closer to the center of the star, because it then covers an area with higher intensity.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F11013868%2F3de5b60398979d49ba04984c71458f04%2Fanimated_transit_0.gif?generation=1759906015242854&alt=media\"){kind=spacer}
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F11013868%2F1cda2d631ff0611792a9fd0609571255%2Fanimated_transit_1.gif?generation=1759906035105339&alt=media\"){kind=spacer}
I made a separate forum post about generating these animations.
\nThe parameters optimizing the fit of the simulated transits to the given signals are refined in ten stages. We are given a time series for FGS1 and 282 time series for AIRS, defining a spectrum, as well as the transit depth, transit start, and transit end from the initial model described in Section 3. Our first goal is to compute transits for three time series:
\nThe r^2 from the first and last transits are used directly in the linear regression for producing the final depth for the submission. The second transit is included because simply summing AIRS across the spectrum has a lower noise than the other two time series, and basing the computation of d_{min}, t_{min}, and t_{length} on the least noisy data produces the best result.
\nThe transits are optimized jointly, so that they share parameters d_{min}, t_{min}, and t_{length}. v is a frozen hyperparameter, and r and u are optimized independently for the three time series. I minimize a weighted sum of the mean squared errors from the three time series, with a higher weight on the second time series. The transit itself is a graph between 0 and 1, so in order match the signal, it is scaled by a polynomial f whose coefficients are computed with linear regression. These coefficients are computed independently for each of the three time series, and they are not treated as parameters, i.e., the transit is defined by its parameters, and the polynomial f is then chosen to minimize the mean squared error against the signal. Additionally, I apply L2 regularization to the coefficients of f for terms with degree at least 2.
\nThe optimization is performed in three stages with an increasing degree of freedom for the polynomial f. The reason that f is not just unconstrained is that it would then adjust to the signal dip, before the appropriate depth was reflected in the parameters. For the first stage we also only use data from inside the transit, since the degree of f is limited to 1. For the second stage, the degree of f is increased to 4 and all the data is used, but severe L2 regularization is applied. For the third stage, the L2 regularization is reduced to a moderate level.
\nNext, we perform similar computations for all 282 time series in the AIRS spectrum. The initial parameters are taken from the third time series above and broadcasted to all 282 time series. The parameters for d_{min}, t_{min}, and t_{length} remain frozen throughout the rest of the computation. The degree of f and the amount of L2 regularization will also remain the same. For the fourth stage, u also remains frozen. Thus, we compute r and the polynomial f independently for each time series.
\nNext, we want to incorporate the gain drift into the computation. This part is based on last year's first place solution. They observed that the data generated by the ExoSim2 simulator used a gain drift factor 1 + f(t) * g(\\lambda), where f is a polynomial of time, similar to what I used above, and g is a polynomial of the wavelength. They used polynomials of degree four, and so will I. At time t and wavelength \\lambda, the overall prediction is then of the form:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F11013868%2Fba4f20e390e843e674fbddd35a1787d5%2Fprediction_formula.png?generation=1759907357269826&alt=media\")
where I(\\lambda) is the intensity of the star at wavelength \\lambda, transit(t, r_\\lambda, u_\\lambda, v, d_{min}, t_{min}, t_{length}) is the transit factor at time t and wavelength \\lambda, and (1 + f(t) * g(\\lambda)) is the gain drift.
\nIn order to stabilize the computation, I alternate between optimizing the coefficients of f and g and optimizing the vectors r and u. When optimizing r and u, the intensity I(\\lambda) is derived independently for each \\lambda with linear regression, as it was done for f in the first three stages, i.e., I(\\lambda) is treated as a polynomial of degree zero. When optimizing f and g, the intensity and the transit are fixed from the previous stage, and the coefficients of g are again derived with linear regression. Thus, we never treat coefficients of g as true parameters. I perform three such alternations.
\nI regularize the vector u during these last stages by adding 200 * var(diff(u)) / mean(u) to the error, where var(diff(u)) is the variance of one-step differences of u, and mean(u) is the mean of u. This is done to ensure that u changes smoothly over the spectrum. r remains noisy, however, at this stage. I also use L2 regularization for f and g. For f regularization is applied to terms with degree at least 2, and for g regularization is applied to terms with degree at least 1.
\nThe depth, i.e., r^2, provided to the submission is obtained with simple linear regression on the r^2 parameters produced in the previous section. For planets with multiple instances, the average depth is used for the submission. Before applying linear regression, the AIRS spectrum is smoothed with a Kalman Smoother in the same way as described in Section 3. Recall from Section 2 that parts of the spectrum are missing, in case there were important missing pixels. This issue is fixed by the Kalman Smoother, since it works even with missing values.
\nThe linear regression for AIRS uses the features:
\nThe linear regression for FGS1 uses only a single feature:
\nThe training data contained outliers that reduced the quality of the solution. This was handled simply by omitting the 18 planets with worst mean squared error from the linear regression.
\nThe standard deviation component of the solution is computed with two independent Neural networks, one for AIRS and one for FGS1. I use 11 features for AIRS and 7 features for FGS1. The features are described in the full documentation. The most important feature was the limb darkening coefficient u.
\nThe neural networks are essentially multilayer perceptrons (MLPs), but with only a single hidden layer. Instead of using a single activation function, the hidden layer computes two vectors in parallel, applies a sigmoid activation function to one and relu to the other, and takes the product of the two vectors before applying dropout. I also use batch normalization for both vectors. The output is transformed with one plus an elu activation function and scaled by a parameter.
\nThe inputs for the MLPs are normalized with a StandardScaler, i.e., I subtract the mean and divide by the standard deviation. The inputs are clipped by the MLP itself to simplify hyperparameter optimization. Hyperparameter optimization was performed with manual experimentation. I initially planned to use Optuna, but I found satisfactory parameters before I got around to it. I used five fold cross validation with three repetitions. I selected features in the same way.
\nFor AIRS, the hidden dimension and dropout are 256 and 0.4. For FGS1, the hidden dimension and dropout are 32 and 0.1. In both cases I use an Adam optimizer with weight decay. For AIRS I trained for 5 epochs, and for FGS1 I trained for 150 epochs. Note that there was much more training data for AIRS due to working with a spectrum, which explains why it worked better with fewer epochs.
\nApplying principal component analysis (PCA) to the ARIEL target spectrum revealed that a small number of eigenvectors explained most of the variance. This is because certain molecules are more likely to make up the atmosphere of a planet. The idea of using PCA came from last year's second place solution. By target spectrum, I mean the matrix obtained by subtracting for each row of the target matrix the mean of that row. I only do this for the AIRS part of the data. I chose to use 7 components, which explained more than 99.5% of the variance.
\nFor each AIRS spectrum obtained from the limb darkening model from Section 4, I fit a linear regression model on the 7 most significant eigenvectors to the spectrum of interest, i.e., I find the best way to approximate the spectrum with spectra from the original target matrix. I then use this approximation as a replacement for the given spectrum, the idea being that the original target data is more accurate than the estimates produced by the limb darkening model.
\nIt is possible that the planet in question is not well represented by the training data. In that case relying on the PCA spectrum replacement would produce worse results. I therefore check whether the Euclidean distance of the original spectrum and the replacement spectrum is below the threshold 0.0015. The threshold is chosen such that about 90% of the training data itself would use PCA replacement. I use the PCA replacement spectrum if that is the case, otherwise I only use the original spectrum. To make this distinction I train two entirely separate models on all the data: One with and one without the PCA spectrum.
\nWhen using the PCA replacement spectrum, it does not technically replace the original spectrum. It appears as an additional feature in the linear regression model for depth estimation. The PCA spectrum generally dominates the original spectrum, however, being assigned about 20 times as high a weight.
\nBoth with and without PCA I train five models and average the result, so in total I train ten models with different seeds. This number could have been much larger while staying within the time limit, but I did not experiment with this aspect.
\nI processed data and trained models locally with an RTX 5090. In total, the running time of a Kaggle notebook with a P100 was about twice as long as with my RTX 5090. I expected a higher speedup with a significantly faster GPU, but the code has not been properly optimized (the electricity consumption was very low), and calculations are generally performed with 64 bit precision. The time spent on the individual steps with an RTX 5090 was:
\nThe limb darkening model from Section 4 produces good depth estimates by itself, and one can imagine further extensions of the model that include additional physics simulations. The way the transit is computed with a discrete integral is very memory inefficient, and it is the performance bottleneck for my submission. It is a small piece of code, and it should be possible to significantly speed up this computation, maybe even by writing a small custom kernel.
\nThe downside of the limb darkening model is that it is very specific to this problem, and achieving high quality results is quite tedious. General purpose models are more likely to be optimized and to be well understood by the research community, so using a very bespoke model is a definite disadvantage. Perhaps the real underlying research question here is to develop optimization techniques that perform well on a wide range of bespoke models. In order for that to be a successful area of research it would require that a large number of bespoke models were made easily available for research. That is clearly outside the scope of the Ariel Challenge, however.
\nIt is a definite downside to my approach that the depth estimate is computed entirely independently of the standard deviation. I would expect that improvements can be made on this front, possibly be adding a correction that is computed together with the standard deviation. I briefly tried this, but it caused overfitting, so I gave up on the idea. I would also expect that it should be possible to make a better model for standard deviation by itself. I generally did not put much time into this area.
\nThe Fermi Paradox asks \"Why are there no signs of intelligence elsewhere in the universe?\" The opportunities for life to appear are literally astronomical. I would argue, however, that astronomical numbers are quite small in this context, since they are only polynomial in nature. The probability of intelligent life appearing may very well be exponential in nature, for example if an initial critical step is for a sequence of molecules to randomly come together in a specific order. Even with slower growing functions than exponentials, the inverse of the probability of intelligent life may dwarf any number in the astronomical range. Thus, it may be plausible that the answer to the question \"Are we alone in an infinite universe?\" is no, while the answer to \"Are we alone in the observable universe?\" is yes with very high probability. This all assumes that life appears spontaneously. In a distant future it seems plausible that humanity will seed other planets with life. Perhaps life on Earth itself originates from such seeds planted by other civilizations or higher entities. Just something to keep in mind: If life was seeded by ancient civilizations, how should that affect the way that we search for it?
", "messages": [ { "id": 3299476, "postDate": "2025-10-08T07:35:14.200Z", "content": "The code for my solution is available in this public github repository.
\nI have documented my solution in this pdf file. The description became quite long, so I am making a shorter writeup. If you want all the details you should read the pdf file.
\nMy Kaggle notebook is available here.
\nThe idea for my solution is to implement a model that mimics the physics of a planet moving in front of a star, while explicitly taking limb darkening into account. This is done to obtain high quality estimates of the transit depth, which then require minimal use of modeling later in the pipeline. In fact, the final depth estimates for the submission are simply constructed with linear regression on answers from the physical model. In particular, the depth is estimated independently of the standard deviation. The standard deviation is modeled with small neural networks. These downstream models were required to be simple due to the small number of planets in the training data.
\nBefore applying the physics-based model, a starting point for the search is provided by a simpler model that involves Kalman Smoothers. I also implemented preprocessing similar to what was suggested by the organizers. Finally, I made use of principal component analysis to refine the spectrum produced by the physical model.
\nAt a high level, my solution contains the following elements:
\nAll in all, my approach ended up being relatively complex, but I believe that the use of explicit modeling of physics may be worth the effort to obtain good results. The main downside is that such an approach is unlikely to generalize to other problems, which makes it inherently less interesting.
\nI generally follow the approach suggested by the organizers. I perform no binning, so the time series for FGS1 have length 67500, and the time series for AIRS have length 5625.
\nI treat hot and dead pixels as missing values. I replace such missing values with linear interpolation of neighboring pixels, four neighbors for FGS1 and two neighbors with the same wavelength for AIRS. Note that clusters of missing values remain missing. If one of the four most significant pixels are missing for AIRS, I remove that entire wavelength. The later model is set up to handle such missing values.
\nI use a rolling window of length 255 to discard outliers beyond five standard deviations from the centered rolling mean, again treating such values as missing.
\nInspired by last year's first place solution, I use the rim of the image to estimate foreground noise, which I then subtract from the center of the image. Except, I do not explicitly use the rim as in last year's first place solution. Instead I use the six rows with lowest intensity for AIRS, and the six columns and then six rows with lowest intensity for FGS1. The following figure illustrates the foreground noise processing step for planet 34983. Note that the foreground noise is not uniform, so I may actually throw away too much of the data.
\nI was pressed on time and did not put effort into verifying that this form of preprocessing was beneficial. It seemed reasonable, so I spent my time elsewhere.
\n
\n
My main model explicitly takes into account limb darkening, but it is non-trivial to optimize without ending up in an undesired local optimum. To improve the search, an initial solution is provided by a simpler model. This initial model was my baseline submission, and it is likely more complex than it needs to be when further refinement is performed later.
\nThe idea for the initial model is to apply a Kalman Smoother to a Local Linear Trend model, to get a very smooth underlying level and trend from which the beginning and end of the transit can easily be extracted. The Kalman Smoother is designed to handle missing values, and the computation is performed twice: Once with all the data, and once with the data from the transit removed (set to NaN). The depth of the transit is extracted from the difference of the levels resulting from these two computations.
\nThe beginning, middle, and end of the transit are identified with the first Kalman Smoother. I identify as breakpoints the indices of the minimum and maximum values of the trend component. The middle of the transit is defined to be the average over these breakpoints, including both AIRS and FGS1. For AIRS, I use the sum across the entire spectrum to decrease noise. The beginning of the transit is the latest point prior to the first breakpoint where either the trend reaches zero, or the one-step difference of the trend is above that of the breakpoint. The end of the transit is defined similarly, but for the earliest point after the second breakpoint. The data from AIRS and FGS1 is combined by choosing the latest of the two beginnings and the earliest of the two ends. Transit identification for planet 34983 is illustrated in the following figure.
\n
My model that explicitly takes into account limb darkening simulates a planet moving in front of a star, and estimates the amount of light observed at every time step. Limb darkening means that the intensity of the star is highest at the center and is reduced toward the edge (the limb). The formula that I use for limb darkening is based on this note. The exact formula used is:
\nintensity(x, u, v) = 1 - u * ((1 - v) * (1 - \\sqrt{1 - x^2}) + v * x^2),
\nwhere x is the distance to the center of the star, assuming that the star has radius 1, and u and v are parameters that depend on the star. Unfortunately, I made a sign error in my code and changed the last plus into a minus, which meant that v behaved unexpectedly, so I gave up on directly optimizing v and instead treated it as a hyperparameter.
\nThe model for constructing the transit uses six parameters:
\nImportantly, the transit is differentiable with respect to these six parameters, which means that the parameters can be optimized to produce the transit with the best fit for some training data.
\nThe following figures show examples of the transits generated by my model for the same parameters, without and with limb darkening. Note that without limb darkening, the valley of the transit is very flat, because the observed light mainly depends on whether the planet is fully inside the boundary of the star or not. With limb darkening, the observed light continues being reduced as the planet gets closer to the center of the star, because it then covers an area with higher intensity.
\n
\n
I made a separate forum post about generating these animations.
\nThe parameters optimizing the fit of the simulated transits to the given signals are refined in ten stages. We are given a time series for FGS1 and 282 time series for AIRS, defining a spectrum, as well as the transit depth, transit start, and transit end from the initial model described in Section 3. Our first goal is to compute transits for three time series:
\nThe r^2 from the first and last transits are used directly in the linear regression for producing the final depth for the submission. The second transit is included because simply summing AIRS across the spectrum has a lower noise than the other two time series, and basing the computation of d_{min}, t_{min}, and t_{length} on the least noisy data produces the best result.
\nThe transits are optimized jointly, so that they share parameters d_{min}, t_{min}, and t_{length}. v is a frozen hyperparameter, and r and u are optimized independently for the three time series. I minimize a weighted sum of the mean squared errors from the three time series, with a higher weight on the second time series. The transit itself is a graph between 0 and 1, so in order match the signal, it is scaled by a polynomial f whose coefficients are computed with linear regression. These coefficients are computed independently for each of the three time series, and they are not treated as parameters, i.e., the transit is defined by its parameters, and the polynomial f is then chosen to minimize the mean squared error against the signal. Additionally, I apply L2 regularization to the coefficients of f for terms with degree at least 2.
\nThe optimization is performed in three stages with an increasing degree of freedom for the polynomial f. The reason that f is not just unconstrained is that it would then adjust to the signal dip, before the appropriate depth was reflected in the parameters. For the first stage we also only use data from inside the transit, since the degree of f is limited to 1. For the second stage, the degree of f is increased to 4 and all the data is used, but severe L2 regularization is applied. For the third stage, the L2 regularization is reduced to a moderate level.
\nNext, we perform similar computations for all 282 time series in the AIRS spectrum. The initial parameters are taken from the third time series above and broadcasted to all 282 time series. The parameters for d_{min}, t_{min}, and t_{length} remain frozen throughout the rest of the computation. The degree of f and the amount of L2 regularization will also remain the same. For the fourth stage, u also remains frozen. Thus, we compute r and the polynomial f independently for each time series.
\nNext, we want to incorporate the gain drift into the computation. This part is based on last year's first place solution. They observed that the data generated by the ExoSim2 simulator used a gain drift factor 1 + f(t) * g(\\lambda), where f is a polynomial of time, similar to what I used above, and g is a polynomial of the wavelength. They used polynomials of degree four, and so will I. At time t and wavelength \\lambda, the overall prediction is then of the form:
\n
where I(\\lambda) is the intensity of the star at wavelength \\lambda, transit(t, r_\\lambda, u_\\lambda, v, d_{min}, t_{min}, t_{length}) is the transit factor at time t and wavelength \\lambda, and (1 + f(t) * g(\\lambda)) is the gain drift.
\nIn order to stabilize the computation, I alternate between optimizing the coefficients of f and g and optimizing the vectors r and u. When optimizing r and u, the intensity I(\\lambda) is derived independently for each \\lambda with linear regression, as it was done for f in the first three stages, i.e., I(\\lambda) is treated as a polynomial of degree zero. When optimizing f and g, the intensity and the transit are fixed from the previous stage, and the coefficients of g are again derived with linear regression. Thus, we never treat coefficients of g as true parameters. I perform three such alternations.
\nI regularize the vector u during these last stages by adding 200 * var(diff(u)) / mean(u) to the error, where var(diff(u)) is the variance of one-step differences of u, and mean(u) is the mean of u. This is done to ensure that u changes smoothly over the spectrum. r remains noisy, however, at this stage. I also use L2 regularization for f and g. For f regularization is applied to terms with degree at least 2, and for g regularization is applied to terms with degree at least 1.
\nThe depth, i.e., r^2, provided to the submission is obtained with simple linear regression on the r^2 parameters produced in the previous section. For planets with multiple instances, the average depth is used for the submission. Before applying linear regression, the AIRS spectrum is smoothed with a Kalman Smoother in the same way as described in Section 3. Recall from Section 2 that parts of the spectrum are missing, in case there were important missing pixels. This issue is fixed by the Kalman Smoother, since it works even with missing values.
\nThe linear regression for AIRS uses the features:
\nThe linear regression for FGS1 uses only a single feature:
\nThe training data contained outliers that reduced the quality of the solution. This was handled simply by omitting the 18 planets with worst mean squared error from the linear regression.
\nThe standard deviation component of the solution is computed with two independent Neural networks, one for AIRS and one for FGS1. I use 11 features for AIRS and 7 features for FGS1. The features are described in the full documentation. The most important feature was the limb darkening coefficient u.
\nThe neural networks are essentially multilayer perceptrons (MLPs), but with only a single hidden layer. Instead of using a single activation function, the hidden layer computes two vectors in parallel, applies a sigmoid activation function to one and relu to the other, and takes the product of the two vectors before applying dropout. I also use batch normalization for both vectors. The output is transformed with one plus an elu activation function and scaled by a parameter.
\nThe inputs for the MLPs are normalized with a StandardScaler, i.e., I subtract the mean and divide by the standard deviation. The inputs are clipped by the MLP itself to simplify hyperparameter optimization. Hyperparameter optimization was performed with manual experimentation. I initially planned to use Optuna, but I found satisfactory parameters before I got around to it. I used five fold cross validation with three repetitions. I selected features in the same way.
\nFor AIRS, the hidden dimension and dropout are 256 and 0.4. For FGS1, the hidden dimension and dropout are 32 and 0.1. In both cases I use an Adam optimizer with weight decay. For AIRS I trained for 5 epochs, and for FGS1 I trained for 150 epochs. Note that there was much more training data for AIRS due to working with a spectrum, which explains why it worked better with fewer epochs.
\nApplying principal component analysis (PCA) to the ARIEL target spectrum revealed that a small number of eigenvectors explained most of the variance. This is because certain molecules are more likely to make up the atmosphere of a planet. The idea of using PCA came from last year's second place solution. By target spectrum, I mean the matrix obtained by subtracting for each row of the target matrix the mean of that row. I only do this for the AIRS part of the data. I chose to use 7 components, which explained more than 99.5% of the variance.
\nFor each AIRS spectrum obtained from the limb darkening model from Section 4, I fit a linear regression model on the 7 most significant eigenvectors to the spectrum of interest, i.e., I find the best way to approximate the spectrum with spectra from the original target matrix. I then use this approximation as a replacement for the given spectrum, the idea being that the original target data is more accurate than the estimates produced by the limb darkening model.
\nIt is possible that the planet in question is not well represented by the training data. In that case relying on the PCA spectrum replacement would produce worse results. I therefore check whether the Euclidean distance of the original spectrum and the replacement spectrum is below the threshold 0.0015. The threshold is chosen such that about 90% of the training data itself would use PCA replacement. I use the PCA replacement spectrum if that is the case, otherwise I only use the original spectrum. To make this distinction I train two entirely separate models on all the data: One with and one without the PCA spectrum.
\nWhen using the PCA replacement spectrum, it does not technically replace the original spectrum. It appears as an additional feature in the linear regression model for depth estimation. The PCA spectrum generally dominates the original spectrum, however, being assigned about 20 times as high a weight.
\nBoth with and without PCA I train five models and average the result, so in total I train ten models with different seeds. This number could have been much larger while staying within the time limit, but I did not experiment with this aspect.
\nI processed data and trained models locally with an RTX 5090. In total, the running time of a Kaggle notebook with a P100 was about twice as long as with my RTX 5090. I expected a higher speedup with a significantly faster GPU, but the code has not been properly optimized (the electricity consumption was very low), and calculations are generally performed with 64 bit precision. The time spent on the individual steps with an RTX 5090 was:
\nThe limb darkening model from Section 4 produces good depth estimates by itself, and one can imagine further extensions of the model that include additional physics simulations. The way the transit is computed with a discrete integral is very memory inefficient, and it is the performance bottleneck for my submission. It is a small piece of code, and it should be possible to significantly speed up this computation, maybe even by writing a small custom kernel.
\nThe downside of the limb darkening model is that it is very specific to this problem, and achieving high quality results is quite tedious. General purpose models are more likely to be optimized and to be well understood by the research community, so using a very bespoke model is a definite disadvantage. Perhaps the real underlying research question here is to develop optimization techniques that perform well on a wide range of bespoke models. In order for that to be a successful area of research it would require that a large number of bespoke models were made easily available for research. That is clearly outside the scope of the Ariel Challenge, however.
\nIt is a definite downside to my approach that the depth estimate is computed entirely independently of the standard deviation. I would expect that improvements can be made on this front, possibly be adding a correction that is computed together with the standard deviation. I briefly tried this, but it caused overfitting, so I gave up on the idea. I would also expect that it should be possible to make a better model for standard deviation by itself. I generally did not put much time into this area.
\nThe Fermi Paradox asks \"Why are there no signs of intelligence elsewhere in the universe?\" The opportunities for life to appear are literally astronomical. I would argue, however, that astronomical numbers are quite small in this context, since they are only polynomial in nature. The probability of intelligent life appearing may very well be exponential in nature, for example if an initial critical step is for a sequence of molecules to randomly come together in a specific order. Even with slower growing functions than exponentials, the inverse of the probability of intelligent life may dwarf any number in the astronomical range. Thus, it may be plausible that the answer to the question \"Are we alone in an infinite universe?\" is no, while the answer to \"Are we alone in the observable universe?\" is yes with very high probability. This all assumes that life appears spontaneously. In a distant future it seems plausible that humanity will seed other planets with life. Perhaps life on Earth itself originates from such seeds planted by other civilizations or higher entities. Just something to keep in mind: If life was seeded by ancient civilizations, how should that affect the way that we search for it?
", "rawMarkdown": "The code for my solution is available in [this public github repository](https://github.com/thomasdueholm/kaggle_ariel_2025).\n\nI have documented my solution in [this pdf file](https://github.com/thomasdueholm/kaggle_ariel_2025/blob/main/model_summary/model_summary.pdf). The description became quite long, so I am making a shorter writeup. If you want all the details you should read the pdf file.\n\nMy Kaggle notebook is available [here](https://www.kaggle.com/code/thomasdueholmhansen/ariel-2025-version-1).\n\n# 1. Model architecture\n\nThe idea for my solution is to implement a model that mimics the physics of a planet moving in front of a star, while explicitly taking limb darkening into account. This is done to obtain high quality estimates of the transit depth, which then require minimal use of modeling later in the pipeline. In fact, the final depth estimates for the submission are simply constructed with linear regression on answers from the physical model. In particular, the depth is estimated independently of the standard deviation. The standard deviation is modeled with small neural networks. These downstream models were required to be simple due to the small number of planets in the training data.\n\nBefore applying the physics-based model, a starting point for the search is provided by a simpler model that involves Kalman Smoothers. I also implemented preprocessing similar to what was suggested by the organizers. Finally, I made use of principal component analysis to refine the spectrum produced by the physical model.\n\nAt a high level, my solution contains the following elements:\n\n1. Preprocessing\n2. Initial depth and transit window estimation\n3. Depth estimation with explicit limb darkening\n4. A simple linear regression model for producing the depth for the submission\n5. A neural network for producing the standard deviation for the submission\n6. A distinction between whether or not to use principal component analysis\n\nAll in all, my approach ended up being relatively complex, but I believe that the use of explicit modeling of physics may be worth the effort to obtain good results. The main downside is that such an approach is unlikely to generalize to other problems, which makes it inherently less interesting.\n\n# 2. Preprocessing\n\nI generally follow [the approach suggested by the organizers](https://www.kaggle.com/code/gordonyip/calibrating-and-binning-ariel-data). I perform no binning, so the time series for FGS1 have length 67500, and the time series for AIRS have length 5625.\n\nI treat hot and dead pixels as missing values. I replace such missing values with linear interpolation of neighboring pixels, four neighbors for FGS1 and two neighbors with the same wavelength for AIRS. Note that clusters of missing values remain missing. If one of the four most significant pixels are missing for AIRS, I remove that entire wavelength. The later model is set up to handle such missing values.\n\nI use a rolling window of length 255 to discard outliers beyond five standard deviations from the centered rolling mean, again treating such values as missing.\n\nInspired by [last year's first place solution](https://www.kaggle.com/competitions/ariel-data-challenge-2024/writeups/c-number-daiwakun-1st-place-solution), I use the rim of the image to estimate foreground noise, which I then subtract from the center of the image. Except, I do not explicitly use the rim as in last year's first place solution. Instead I use the six rows with lowest intensity for AIRS, and the six columns and then six rows with lowest intensity for FGS1. The following figure illustrates the foreground noise processing step for planet 34983. Note that the foreground noise is not uniform, so I may actually throw away too much of the data.\n\nI was pressed on time and did not put effort into verifying that this form of preprocessing was beneficial. It seemed reasonable, so I spent my time elsewhere.\n\n\n\n\n# 3. Initial depth and transit window estimation\n\nMy main model explicitly takes into account limb darkening, but it is non-trivial to optimize without ending up in an undesired local optimum. To improve the search, an initial solution is provided by a simpler model. This initial model was my baseline submission, and it is likely more complex than it needs to be when further refinement is performed later.\n\nThe idea for the initial model is to apply a Kalman Smoother to a [Local Linear Trend model](https://www.statsmodels.org/dev/examples/notebooks/generated/statespace_local_linear_trend.html), to get a very smooth underlying level and trend from which the beginning and end of the transit can easily be extracted. The Kalman Smoother is designed to handle missing values, and the computation is performed twice: Once with all the data, and once with the data from the transit removed (set to NaN). The depth of the transit is extracted from the difference of the levels resulting from these two computations.\n\nThe beginning, middle, and end of the transit are identified with the first Kalman Smoother. I identify as breakpoints the indices of the minimum and maximum values of the trend component. The middle of the transit is defined to be the average over these breakpoints, including both AIRS and FGS1. For AIRS, I use the sum across the entire spectrum to decrease noise. The beginning of the transit is the latest point prior to the first breakpoint where either the trend reaches zero, or the one-step difference of the trend is above that of the breakpoint. The end of the transit is defined similarly, but for the earliest point after the second breakpoint. The data from AIRS and FGS1 is combined by choosing the latest of the two beginnings and the earliest of the two ends. Transit identification for planet 34983 is illustrated in the following figure.\n\n\n\n# 4. Depth estimation with explicit limb darkening\n\n## 4.1. Transit simulation\n\nMy model that explicitly takes into account limb darkening simulates a planet moving in front of a star, and estimates the amount of light observed at every time step. Limb darkening means that the intensity of the star is highest at the center and is reduced toward the edge (the limb). The formula that I use for limb darkening is based on [this note](https://www.astro.uvic.ca/~tatum/stellatm/atm6.pdf). The exact formula used is:\n\nintensity(x, u, v) = 1 - u * ((1 - v) * (1 - \\sqrt{1 - x^2}) + v * x^2),\n\nwhere x is the distance to the center of the star, assuming that the star has radius 1, and u and v are parameters that depend on the star. Unfortunately, I made a sign error in my code and changed the last plus into a minus, which meant that v behaved unexpectedly, so I gave up on directly optimizing v and instead treated it as a hyperparameter.\n\nThe model for constructing the transit uses six parameters:\n\n- r: The radius of the planet, scaled such that the radius of the star is 1.\n- u: The limb darkening parameter u described above.\n- v: The limb darkening parameter v described above.\n- d_{min}: The closest distance from the center of the planet to the center of the star during the transit.\n- t_{min}: The time at which the planet is closest to the center of the star. The timeframe goes from 0 to 1.\n- t_{length}: The transit starts when the planet first overlaps the star, and ends when the planet no longer overlaps the star. The transit length t_{length} is the time between these two points. Note that the transit length can be greater than 1 if the timeframe does not cover the entire transit.\n\nImportantly, the transit is differentiable with respect to these six parameters, which means that the parameters can be optimized to produce the transit with the best fit for some training data.\n\nThe following figures show examples of the transits generated by my model for the same parameters, without and with limb darkening. Note that without limb darkening, the valley of the transit is very flat, because the observed light mainly depends on whether the planet is fully inside the boundary of the star or not. With limb darkening, the observed light continues being reduced as the planet gets closer to the center of the star, because it then covers an area with higher intensity.\n\n\n\n\nI made [a separate forum post](https://www.kaggle.com/competitions/ariel-data-challenge-2025/discussion/609406) about generating these animations.\n\n## 4.2. Parameter optimization\n\nThe parameters optimizing the fit of the simulated transits to the given signals are refined in ten stages. We are given a time series for FGS1 and 282 time series for AIRS, defining a spectrum, as well as the transit depth, transit start, and transit end from the initial model described in Section 3. Our first goal is to compute transits for three time series:\n\n1. FGS1.\n2. The sum of AIRS over the spectrum.\n3. The weighted sum of AIRS over the spectrum, with weights being the inverse of the mean of the signals, i.e., all 282 time series should have equal importance regardless of magnitude.\n\nThe r^2 from the first and last transits are used directly in the linear regression for producing the final depth for the submission. The second transit is included because simply summing AIRS across the spectrum has a lower noise than the other two time series, and basing the computation of d_{min}, t_{min}, and t_{length} on the least noisy data produces the best result.\n\nThe transits are optimized jointly, so that they share parameters d_{min}, t_{min}, and t_{length}. v is a frozen hyperparameter, and r and u are optimized independently for the three time series. I minimize a weighted sum of the mean squared errors from the three time series, with a higher weight on the second time series. The transit itself is a graph between 0 and 1, so in order match the signal, it is scaled by a polynomial f whose coefficients are computed with linear regression. These coefficients are computed independently for each of the three time series, and they are not treated as parameters, i.e., the transit is defined by its parameters, and the polynomial f is then chosen to minimize the mean squared error against the signal. Additionally, I apply L2 regularization to the coefficients of f for terms with degree at least 2.\n\nThe optimization is performed in three stages with an increasing degree of freedom for the polynomial f. The reason that f is not just unconstrained is that it would then adjust to the signal dip, before the appropriate depth was reflected in the parameters. For the first stage we also only use data from inside the transit, since the degree of f is limited to 1. For the second stage, the degree of f is increased to 4 and all the data is used, but severe L2 regularization is applied. For the third stage, the L2 regularization is reduced to a moderate level.\n\nNext, we perform similar computations for all 282 time series in the AIRS spectrum. The initial parameters are taken from the third time series above and broadcasted to all 282 time series. The parameters for d_{min}, t_{min}, and t_{length} remain frozen throughout the rest of the computation. The degree of f and the amount of L2 regularization will also remain the same. For the fourth stage, u also remains frozen. Thus, we compute r and the polynomial f independently for each time series.\n\nNext, we want to incorporate the *gain drift* into the computation. This part is based on [last year's first place solution](https://www.kaggle.com/competitions/ariel-data-challenge-2024/writeups/c-number-daiwakun-1st-place-solution). They observed that the data generated by the ExoSim2 simulator used a gain drift factor 1 + f(t) * g(\\lambda), where f is a polynomial of time, similar to what I used above, and g is a polynomial of the wavelength. They used polynomials of degree four, and so will I. At time t and wavelength \\lambda, the overall prediction is then of the form:\n\n\n\nwhere I(\\lambda) is the intensity of the star at wavelength \\lambda, transit(t, r_\\lambda, u_\\lambda, v, d_{min}, t_{min}, t_{length}) is the transit factor at time t and wavelength \\lambda, and (1 + f(t) * g(\\lambda)) is the gain drift.\n\nIn order to stabilize the computation, I alternate between optimizing the coefficients of f and g and optimizing the vectors r and u. When optimizing r and u, the intensity I(\\lambda) is derived independently for each \\lambda with linear regression, as it was done for f in the first three stages, i.e., I(\\lambda) is treated as a polynomial of degree zero. When optimizing f and g, the intensity and the transit are fixed from the previous stage, and the coefficients of g are again derived with linear regression. Thus, we never treat coefficients of g as true parameters. I perform three such alternations.\n\nI regularize the vector u during these last stages by adding 200 * var(diff(u)) / mean(u) to the error, where var(diff(u)) is the variance of one-step differences of u, and mean(u) is the mean of u. This is done to ensure that u changes smoothly over the spectrum. r remains noisy, however, at this stage. I also use L2 regularization for f and g. For f regularization is applied to terms with degree at least 2, and for g regularization is applied to terms with degree at least 1.\n\n## 4.3. Spectrum smoothing and final depth estimation\n\nThe depth, i.e., r^2, provided to the submission is obtained with simple linear regression on the r^2 parameters produced in the previous section. For planets with multiple instances, the average depth is used for the submission. Before applying linear regression, the AIRS spectrum is smoothed with a Kalman Smoother in the same way as described in Section 3. Recall from Section 2 that parts of the spectrum are missing, in case there were important missing pixels. This issue is fixed by the Kalman Smoother, since it works even with missing values.\n\nThe linear regression for AIRS uses the features:\n1. r^2 from the third transit, weighted sum of AIRS, after stage 3 in Section 4.2, broadcasted to the entire spectrum.\n2. The mean of the smoothed r^2 for the spectrum, broadcasted to the entire spectrum.\n3. The difference between the smoothed r^2 for the spectrum and its mean.\n4. Possibly a feature derived with principal component analysis (PCA) as described later in Section 6.\n\nThe linear regression for FGS1 uses only a single feature:\n1. r^2 from the first transit, FGS1, after stage 3 in Section 4.2.\n\nThe training data contained outliers that reduced the quality of the solution. This was handled simply by omitting the 18 planets with worst mean squared error from the linear regression.\n\n# 5. Computation of uncertainty\n\nThe standard deviation component of the solution is computed with two independent Neural networks, one for AIRS and one for FGS1. I use 11 features for AIRS and 7 features for FGS1. The features are described in [the full documentation](https://github.com/thomasdueholm/kaggle_ariel_2025/blob/main/model_summary/model_summary.pdf). The most important feature was the limb darkening coefficient u.\n\nThe neural networks are essentially multilayer perceptrons (MLPs), but with only a single hidden layer. Instead of using a single activation function, the hidden layer computes two vectors in parallel, applies a sigmoid activation function to one and relu to the other, and takes the product of the two vectors before applying dropout. I also use batch normalization for both vectors. The output is transformed with one plus an elu activation function and scaled by a parameter.\n\nThe inputs for the MLPs are normalized with a StandardScaler, i.e., I subtract the mean and divide by the standard deviation. The inputs are clipped by the MLP itself to simplify hyperparameter optimization. Hyperparameter optimization was performed with manual experimentation. I initially planned to use Optuna, but I found satisfactory parameters before I got around to it. I used five fold cross validation with three repetitions. I selected features in the same way.\n\nFor AIRS, the hidden dimension and dropout are 256 and 0.4. For FGS1, the hidden dimension and dropout are 32 and 0.1. In both cases I use an Adam optimizer with weight decay. For AIRS I trained for 5 epochs, and for FGS1 I trained for 150 epochs. Note that there was much more training data for AIRS due to working with a spectrum, which explains why it worked better with fewer epochs.\n\n# 6. Principal component analysis (PCA)\n\nApplying principal component analysis (PCA) to the ARIEL target spectrum revealed that a small number of eigenvectors explained most of the variance. This is because certain molecules are more likely to make up the atmosphere of a planet. The idea of using PCA came from [last year's second place solution](https://www.kaggle.com/competitions/ariel-data-challenge-2024/writeups/jeroen-cottaar-2nd-place-solution-pure-bayesian-in). By target spectrum, I mean the matrix obtained by subtracting for each row of the target matrix the mean of that row. I only do this for the AIRS part of the data. I chose to use 7 components, which explained more than 99.5% of the variance.\n\nFor each AIRS spectrum obtained from the limb darkening model from Section 4, I fit a linear regression model on the 7 most significant eigenvectors to the spectrum of interest, i.e., I find the best way to approximate the spectrum with spectra from the original target matrix. I then use this approximation as a replacement for the given spectrum, the idea being that the original target data is more accurate than the estimates produced by the limb darkening model.\n\nIt is possible that the planet in question is not well represented by the training data. In that case relying on the PCA spectrum replacement would produce worse results. I therefore check whether the Euclidean distance of the original spectrum and the replacement spectrum is below the threshold 0.0015. The threshold is chosen such that about 90% of the training data itself would use PCA replacement. I use the PCA replacement spectrum if that is the case, otherwise I only use the original spectrum. To make this distinction I train two entirely separate models on all the data: One with and one without the PCA spectrum.\n\nWhen using the PCA replacement spectrum, it does not technically replace the original spectrum. It appears as an additional feature in the linear regression model for depth estimation. The PCA spectrum generally dominates the original spectrum, however, being assigned about 20 times as high a weight.\n\n# 7. Ensembling\n\nBoth with and without PCA I train five models and average the result, so in total I train ten models with different seeds. This number could have been much larger while staying within the time limit, but I did not experiment with this aspect.\n\n# 8. Computation time\n\nI processed data and trained models locally with an RTX 5090. In total, the running time of a Kaggle notebook with a P100 was about twice as long as with my RTX 5090. I expected a higher speedup with a significantly faster GPU, but the code has not been properly optimized (the electricity consumption was very low), and calculations are generally performed with 64 bit precision. The time spent on the individual steps with an RTX 5090 was:\n\n- Preprocessing (Section 2): 17 minutes.\n- Initial depth estimation (Section 3): 1 minute.\n- Depth estimation with limb darkening (Section 4): 3 hours and 35 minutes.\n- Model training: 11 minutes.\n\n# 9. Conclusion\n\nThe limb darkening model from Section 4 produces good depth estimates by itself, and one can imagine further extensions of the model that include additional physics simulations. The way the transit is computed with a discrete integral is very memory inefficient, and it is the performance bottleneck for my submission. It is a small piece of code, and it should be possible to significantly speed up this computation, maybe even by writing a small custom kernel.\n\nThe downside of the limb darkening model is that it is very specific to this problem, and achieving high quality results is quite tedious. General purpose models are more likely to be optimized and to be well understood by the research community, so using a very bespoke model is a definite disadvantage. Perhaps the real underlying research question here is to develop optimization techniques that perform well on a wide range of bespoke models. In order for that to be a successful area of research it would require that a large number of bespoke models were made easily available for research. That is clearly outside the scope of the Ariel Challenge, however.\n\nIt is a definite downside to my approach that the depth estimate is computed entirely independently of the standard deviation. I would expect that improvements can be made on this front, possibly be adding a correction that is computed together with the standard deviation. I briefly tried this, but it caused overfitting, so I gave up on the idea. I would also expect that it should be possible to make a better model for standard deviation by itself. I generally did not put much time into this area.\n\n# 10. Are we alone in the universe?\n\nThe Fermi Paradox asks \"Why are there no signs of intelligence elsewhere in the universe?\" The opportunities for life to appear are literally astronomical. I would argue, however, that astronomical numbers are quite small in this context, since they are only polynomial in nature. The probability of intelligent life appearing may very well be exponential in nature, for example if an initial critical step is for a sequence of molecules to randomly come together in a specific order. Even with slower growing functions than exponentials, the inverse of the probability of intelligent life may dwarf any number in the astronomical range. Thus, it may be plausible that the answer to the question \"Are we alone in an infinite universe?\" is no, while the answer to \"Are we alone in the observable universe?\" is yes with very high probability. This all assumes that life appears spontaneously. In a distant future it seems plausible that humanity will seed other planets with life. Perhaps life on Earth itself originates from such seeds planted by other civilizations or higher entities. Just something to keep in mind: If life was seeded by ancient civilizations, how should that affect the way that we search for it?", "votes": 8 } ], "comments": [], "raw_markdown_by_id": { "3299476": "The code for my solution is available in [this public github repository](https://github.com/thomasdueholm/kaggle_ariel_2025).\n\nI have documented my solution in [this pdf file](https://github.com/thomasdueholm/kaggle_ariel_2025/blob/main/model_summary/model_summary.pdf). The description became quite long, so I am making a shorter writeup. If you want all the details you should read the pdf file.\n\nMy Kaggle notebook is available [here](https://www.kaggle.com/code/thomasdueholmhansen/ariel-2025-version-1).\n\n# 1. Model architecture\n\nThe idea for my solution is to implement a model that mimics the physics of a planet moving in front of a star, while explicitly taking limb darkening into account. This is done to obtain high quality estimates of the transit depth, which then require minimal use of modeling later in the pipeline. In fact, the final depth estimates for the submission are simply constructed with linear regression on answers from the physical model. In particular, the depth is estimated independently of the standard deviation. The standard deviation is modeled with small neural networks. These downstream models were required to be simple due to the small number of planets in the training data.\n\nBefore applying the physics-based model, a starting point for the search is provided by a simpler model that involves Kalman Smoothers. I also implemented preprocessing similar to what was suggested by the organizers. Finally, I made use of principal component analysis to refine the spectrum produced by the physical model.\n\nAt a high level, my solution contains the following elements:\n\n1. Preprocessing\n2. Initial depth and transit window estimation\n3. Depth estimation with explicit limb darkening\n4. A simple linear regression model for producing the depth for the submission\n5. A neural network for producing the standard deviation for the submission\n6. A distinction between whether or not to use principal component analysis\n\nAll in all, my approach ended up being relatively complex, but I believe that the use of explicit modeling of physics may be worth the effort to obtain good results. The main downside is that such an approach is unlikely to generalize to other problems, which makes it inherently less interesting.\n\n# 2. Preprocessing\n\nI generally follow [the approach suggested by the organizers](https://www.kaggle.com/code/gordonyip/calibrating-and-binning-ariel-data). I perform no binning, so the time series for FGS1 have length 67500, and the time series for AIRS have length 5625.\n\nI treat hot and dead pixels as missing values. I replace such missing values with linear interpolation of neighboring pixels, four neighbors for FGS1 and two neighbors with the same wavelength for AIRS. Note that clusters of missing values remain missing. If one of the four most significant pixels are missing for AIRS, I remove that entire wavelength. The later model is set up to handle such missing values.\n\nI use a rolling window of length 255 to discard outliers beyond five standard deviations from the centered rolling mean, again treating such values as missing.\n\nInspired by [last year's first place solution](https://www.kaggle.com/competitions/ariel-data-challenge-2024/writeups/c-number-daiwakun-1st-place-solution), I use the rim of the image to estimate foreground noise, which I then subtract from the center of the image. Except, I do not explicitly use the rim as in last year's first place solution. Instead I use the six rows with lowest intensity for AIRS, and the six columns and then six rows with lowest intensity for FGS1. The following figure illustrates the foreground noise processing step for planet 34983. Note that the foreground noise is not uniform, so I may actually throw away too much of the data.\n\nI was pressed on time and did not put effort into verifying that this form of preprocessing was beneficial. It seemed reasonable, so I spent my time elsewhere.\n\n\n\n\n# 3. Initial depth and transit window estimation\n\nMy main model explicitly takes into account limb darkening, but it is non-trivial to optimize without ending up in an undesired local optimum. To improve the search, an initial solution is provided by a simpler model. This initial model was my baseline submission, and it is likely more complex than it needs to be when further refinement is performed later.\n\nThe idea for the initial model is to apply a Kalman Smoother to a [Local Linear Trend model](https://www.statsmodels.org/dev/examples/notebooks/generated/statespace_local_linear_trend.html), to get a very smooth underlying level and trend from which the beginning and end of the transit can easily be extracted. The Kalman Smoother is designed to handle missing values, and the computation is performed twice: Once with all the data, and once with the data from the transit removed (set to NaN). The depth of the transit is extracted from the difference of the levels resulting from these two computations.\n\nThe beginning, middle, and end of the transit are identified with the first Kalman Smoother. I identify as breakpoints the indices of the minimum and maximum values of the trend component. The middle of the transit is defined to be the average over these breakpoints, including both AIRS and FGS1. For AIRS, I use the sum across the entire spectrum to decrease noise. The beginning of the transit is the latest point prior to the first breakpoint where either the trend reaches zero, or the one-step difference of the trend is above that of the breakpoint. The end of the transit is defined similarly, but for the earliest point after the second breakpoint. The data from AIRS and FGS1 is combined by choosing the latest of the two beginnings and the earliest of the two ends. Transit identification for planet 34983 is illustrated in the following figure.\n\n\n\n# 4. Depth estimation with explicit limb darkening\n\n## 4.1. Transit simulation\n\nMy model that explicitly takes into account limb darkening simulates a planet moving in front of a star, and estimates the amount of light observed at every time step. Limb darkening means that the intensity of the star is highest at the center and is reduced toward the edge (the limb). The formula that I use for limb darkening is based on [this note](https://www.astro.uvic.ca/~tatum/stellatm/atm6.pdf). The exact formula used is:\n\nintensity(x, u, v) = 1 - u * ((1 - v) * (1 - \\sqrt{1 - x^2}) + v * x^2),\n\nwhere x is the distance to the center of the star, assuming that the star has radius 1, and u and v are parameters that depend on the star. Unfortunately, I made a sign error in my code and changed the last plus into a minus, which meant that v behaved unexpectedly, so I gave up on directly optimizing v and instead treated it as a hyperparameter.\n\nThe model for constructing the transit uses six parameters:\n\n- r: The radius of the planet, scaled such that the radius of the star is 1.\n- u: The limb darkening parameter u described above.\n- v: The limb darkening parameter v described above.\n- d_{min}: The closest distance from the center of the planet to the center of the star during the transit.\n- t_{min}: The time at which the planet is closest to the center of the star. The timeframe goes from 0 to 1.\n- t_{length}: The transit starts when the planet first overlaps the star, and ends when the planet no longer overlaps the star. The transit length t_{length} is the time between these two points. Note that the transit length can be greater than 1 if the timeframe does not cover the entire transit.\n\nImportantly, the transit is differentiable with respect to these six parameters, which means that the parameters can be optimized to produce the transit with the best fit for some training data.\n\nThe following figures show examples of the transits generated by my model for the same parameters, without and with limb darkening. Note that without limb darkening, the valley of the transit is very flat, because the observed light mainly depends on whether the planet is fully inside the boundary of the star or not. With limb darkening, the observed light continues being reduced as the planet gets closer to the center of the star, because it then covers an area with higher intensity.\n\n\n\n\nI made [a separate forum post](https://www.kaggle.com/competitions/ariel-data-challenge-2025/discussion/609406) about generating these animations.\n\n## 4.2. Parameter optimization\n\nThe parameters optimizing the fit of the simulated transits to the given signals are refined in ten stages. We are given a time series for FGS1 and 282 time series for AIRS, defining a spectrum, as well as the transit depth, transit start, and transit end from the initial model described in Section 3. Our first goal is to compute transits for three time series:\n\n1. FGS1.\n2. The sum of AIRS over the spectrum.\n3. The weighted sum of AIRS over the spectrum, with weights being the inverse of the mean of the signals, i.e., all 282 time series should have equal importance regardless of magnitude.\n\nThe r^2 from the first and last transits are used directly in the linear regression for producing the final depth for the submission. The second transit is included because simply summing AIRS across the spectrum has a lower noise than the other two time series, and basing the computation of d_{min}, t_{min}, and t_{length} on the least noisy data produces the best result.\n\nThe transits are optimized jointly, so that they share parameters d_{min}, t_{min}, and t_{length}. v is a frozen hyperparameter, and r and u are optimized independently for the three time series. I minimize a weighted sum of the mean squared errors from the three time series, with a higher weight on the second time series. The transit itself is a graph between 0 and 1, so in order match the signal, it is scaled by a polynomial f whose coefficients are computed with linear regression. These coefficients are computed independently for each of the three time series, and they are not treated as parameters, i.e., the transit is defined by its parameters, and the polynomial f is then chosen to minimize the mean squared error against the signal. Additionally, I apply L2 regularization to the coefficients of f for terms with degree at least 2.\n\nThe optimization is performed in three stages with an increasing degree of freedom for the polynomial f. The reason that f is not just unconstrained is that it would then adjust to the signal dip, before the appropriate depth was reflected in the parameters. For the first stage we also only use data from inside the transit, since the degree of f is limited to 1. For the second stage, the degree of f is increased to 4 and all the data is used, but severe L2 regularization is applied. For the third stage, the L2 regularization is reduced to a moderate level.\n\nNext, we perform similar computations for all 282 time series in the AIRS spectrum. The initial parameters are taken from the third time series above and broadcasted to all 282 time series. The parameters for d_{min}, t_{min}, and t_{length} remain frozen throughout the rest of the computation. The degree of f and the amount of L2 regularization will also remain the same. For the fourth stage, u also remains frozen. Thus, we compute r and the polynomial f independently for each time series.\n\nNext, we want to incorporate the *gain drift* into the computation. This part is based on [last year's first place solution](https://www.kaggle.com/competitions/ariel-data-challenge-2024/writeups/c-number-daiwakun-1st-place-solution). They observed that the data generated by the ExoSim2 simulator used a gain drift factor 1 + f(t) * g(\\lambda), where f is a polynomial of time, similar to what I used above, and g is a polynomial of the wavelength. They used polynomials of degree four, and so will I. At time t and wavelength \\lambda, the overall prediction is then of the form:\n\n\n\nwhere I(\\lambda) is the intensity of the star at wavelength \\lambda, transit(t, r_\\lambda, u_\\lambda, v, d_{min}, t_{min}, t_{length}) is the transit factor at time t and wavelength \\lambda, and (1 + f(t) * g(\\lambda)) is the gain drift.\n\nIn order to stabilize the computation, I alternate between optimizing the coefficients of f and g and optimizing the vectors r and u. When optimizing r and u, the intensity I(\\lambda) is derived independently for each \\lambda with linear regression, as it was done for f in the first three stages, i.e., I(\\lambda) is treated as a polynomial of degree zero. When optimizing f and g, the intensity and the transit are fixed from the previous stage, and the coefficients of g are again derived with linear regression. Thus, we never treat coefficients of g as true parameters. I perform three such alternations.\n\nI regularize the vector u during these last stages by adding 200 * var(diff(u)) / mean(u) to the error, where var(diff(u)) is the variance of one-step differences of u, and mean(u) is the mean of u. This is done to ensure that u changes smoothly over the spectrum. r remains noisy, however, at this stage. I also use L2 regularization for f and g. For f regularization is applied to terms with degree at least 2, and for g regularization is applied to terms with degree at least 1.\n\n## 4.3. Spectrum smoothing and final depth estimation\n\nThe depth, i.e., r^2, provided to the submission is obtained with simple linear regression on the r^2 parameters produced in the previous section. For planets with multiple instances, the average depth is used for the submission. Before applying linear regression, the AIRS spectrum is smoothed with a Kalman Smoother in the same way as described in Section 3. Recall from Section 2 that parts of the spectrum are missing, in case there were important missing pixels. This issue is fixed by the Kalman Smoother, since it works even with missing values.\n\nThe linear regression for AIRS uses the features:\n1. r^2 from the third transit, weighted sum of AIRS, after stage 3 in Section 4.2, broadcasted to the entire spectrum.\n2. The mean of the smoothed r^2 for the spectrum, broadcasted to the entire spectrum.\n3. The difference between the smoothed r^2 for the spectrum and its mean.\n4. Possibly a feature derived with principal component analysis (PCA) as described later in Section 6.\n\nThe linear regression for FGS1 uses only a single feature:\n1. r^2 from the first transit, FGS1, after stage 3 in Section 4.2.\n\nThe training data contained outliers that reduced the quality of the solution. This was handled simply by omitting the 18 planets with worst mean squared error from the linear regression.\n\n# 5. Computation of uncertainty\n\nThe standard deviation component of the solution is computed with two independent Neural networks, one for AIRS and one for FGS1. I use 11 features for AIRS and 7 features for FGS1. The features are described in [the full documentation](https://github.com/thomasdueholm/kaggle_ariel_2025/blob/main/model_summary/model_summary.pdf). The most important feature was the limb darkening coefficient u.\n\nThe neural networks are essentially multilayer perceptrons (MLPs), but with only a single hidden layer. Instead of using a single activation function, the hidden layer computes two vectors in parallel, applies a sigmoid activation function to one and relu to the other, and takes the product of the two vectors before applying dropout. I also use batch normalization for both vectors. The output is transformed with one plus an elu activation function and scaled by a parameter.\n\nThe inputs for the MLPs are normalized with a StandardScaler, i.e., I subtract the mean and divide by the standard deviation. The inputs are clipped by the MLP itself to simplify hyperparameter optimization. Hyperparameter optimization was performed with manual experimentation. I initially planned to use Optuna, but I found satisfactory parameters before I got around to it. I used five fold cross validation with three repetitions. I selected features in the same way.\n\nFor AIRS, the hidden dimension and dropout are 256 and 0.4. For FGS1, the hidden dimension and dropout are 32 and 0.1. In both cases I use an Adam optimizer with weight decay. For AIRS I trained for 5 epochs, and for FGS1 I trained for 150 epochs. Note that there was much more training data for AIRS due to working with a spectrum, which explains why it worked better with fewer epochs.\n\n# 6. Principal component analysis (PCA)\n\nApplying principal component analysis (PCA) to the ARIEL target spectrum revealed that a small number of eigenvectors explained most of the variance. This is because certain molecules are more likely to make up the atmosphere of a planet. The idea of using PCA came from [last year's second place solution](https://www.kaggle.com/competitions/ariel-data-challenge-2024/writeups/jeroen-cottaar-2nd-place-solution-pure-bayesian-in). By target spectrum, I mean the matrix obtained by subtracting for each row of the target matrix the mean of that row. I only do this for the AIRS part of the data. I chose to use 7 components, which explained more than 99.5% of the variance.\n\nFor each AIRS spectrum obtained from the limb darkening model from Section 4, I fit a linear regression model on the 7 most significant eigenvectors to the spectrum of interest, i.e., I find the best way to approximate the spectrum with spectra from the original target matrix. I then use this approximation as a replacement for the given spectrum, the idea being that the original target data is more accurate than the estimates produced by the limb darkening model.\n\nIt is possible that the planet in question is not well represented by the training data. In that case relying on the PCA spectrum replacement would produce worse results. I therefore check whether the Euclidean distance of the original spectrum and the replacement spectrum is below the threshold 0.0015. The threshold is chosen such that about 90% of the training data itself would use PCA replacement. I use the PCA replacement spectrum if that is the case, otherwise I only use the original spectrum. To make this distinction I train two entirely separate models on all the data: One with and one without the PCA spectrum.\n\nWhen using the PCA replacement spectrum, it does not technically replace the original spectrum. It appears as an additional feature in the linear regression model for depth estimation. The PCA spectrum generally dominates the original spectrum, however, being assigned about 20 times as high a weight.\n\n# 7. Ensembling\n\nBoth with and without PCA I train five models and average the result, so in total I train ten models with different seeds. This number could have been much larger while staying within the time limit, but I did not experiment with this aspect.\n\n# 8. Computation time\n\nI processed data and trained models locally with an RTX 5090. In total, the running time of a Kaggle notebook with a P100 was about twice as long as with my RTX 5090. I expected a higher speedup with a significantly faster GPU, but the code has not been properly optimized (the electricity consumption was very low), and calculations are generally performed with 64 bit precision. The time spent on the individual steps with an RTX 5090 was:\n\n- Preprocessing (Section 2): 17 minutes.\n- Initial depth estimation (Section 3): 1 minute.\n- Depth estimation with limb darkening (Section 4): 3 hours and 35 minutes.\n- Model training: 11 minutes.\n\n# 9. Conclusion\n\nThe limb darkening model from Section 4 produces good depth estimates by itself, and one can imagine further extensions of the model that include additional physics simulations. The way the transit is computed with a discrete integral is very memory inefficient, and it is the performance bottleneck for my submission. It is a small piece of code, and it should be possible to significantly speed up this computation, maybe even by writing a small custom kernel.\n\nThe downside of the limb darkening model is that it is very specific to this problem, and achieving high quality results is quite tedious. General purpose models are more likely to be optimized and to be well understood by the research community, so using a very bespoke model is a definite disadvantage. Perhaps the real underlying research question here is to develop optimization techniques that perform well on a wide range of bespoke models. In order for that to be a successful area of research it would require that a large number of bespoke models were made easily available for research. That is clearly outside the scope of the Ariel Challenge, however.\n\nIt is a definite downside to my approach that the depth estimate is computed entirely independently of the standard deviation. I would expect that improvements can be made on this front, possibly be adding a correction that is computed together with the standard deviation. I briefly tried this, but it caused overfitting, so I gave up on the idea. I would also expect that it should be possible to make a better model for standard deviation by itself. I generally did not put much time into this area.\n\n# 10. Are we alone in the universe?\n\nThe Fermi Paradox asks \"Why are there no signs of intelligence elsewhere in the universe?\" The opportunities for life to appear are literally astronomical. I would argue, however, that astronomical numbers are quite small in this context, since they are only polynomial in nature. The probability of intelligent life appearing may very well be exponential in nature, for example if an initial critical step is for a sequence of molecules to randomly come together in a specific order. Even with slower growing functions than exponentials, the inverse of the probability of intelligent life may dwarf any number in the astronomical range. Thus, it may be plausible that the answer to the question \"Are we alone in an infinite universe?\" is no, while the answer to \"Are we alone in the observable universe?\" is yes with very high probability. This all assumes that life appears spontaneously. In a distant future it seems plausible that humanity will seed other planets with life. Perhaps life on Earth itself originates from such seeds planted by other civilizations or higher entities. Just something to keep in mind: If life was seeded by ancient civilizations, how should that affect the way that we search for it?" } }