![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14984949%2Fdb7b70449a7c43d02d157f90105f8f09%2Fdownload.png?generation=1756280700460046&alt=media\")
I have a model that does quite well on most planets, but there are some occasional planets that it just can't model properly. I'm pretty convinced by now there is something very weird going on here - either a bug in the data generation, or some physical effect I've never heard of.
\nI can't really describe the problem in full without revealing my model, but there's one planet that allows me to show the effect I mean in a simple way. This is #1349926825. It has two transits, and this is the difference between the AIRS measurements for the two planets (X axis is wavelength, Y axis is time):
\n
The effect I'm trying to highlight is more apparent after a low pass filter:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14984949%2Fa2e525eb363c8c797f137131e7375ddc%2Fdownload%20(1).png?generation=1756280741462536&alt=media\")
It appears there is a clear difference in the transit depth between the two transits. Nothing in the expected physics explains this properly. What I can come up with:
\nAnyone have any ideas? Also tagging @gordonyip
\nNOTE: some light preprocessing was applied to the figures above, which cannot explain what we see:
\nI have a model that does quite well on most planets, but there are some occasional planets that it just can't model properly. I'm pretty convinced by now there is something very weird going on here - either a bug in the data generation, or some physical effect I've never heard of.
\nI can't really describe the problem in full without revealing my model, but there's one planet that allows me to show the effect I mean in a simple way. This is #1349926825. It has two transits, and this is the difference between the AIRS measurements for the two planets (X axis is wavelength, Y axis is time):
\n
The effect I'm trying to highlight is more apparent after a low pass filter:
\n
It appears there is a clear difference in the transit depth between the two transits. Nothing in the expected physics explains this properly. What I can come up with:
\nAnyone have any ideas? Also tagging @gordonyip
\nNOTE: some light preprocessing was applied to the figures above, which cannot explain what we see:
\nDepending on how advanced the simulation is, the discrepancy might be caused by the planet passing in front of some anomaly on the surface of the star in one case but not the other. For example something like this coronal hole:
\nhttps://www.reddit.com/r/spaceporn/comments/1n87k64/a_giant_southernhemisphere_coronal_hole_is_now/
", "rawMarkdown": "Depending on how advanced the simulation is, the discrepancy might be caused by the planet passing in front of some anomaly on the surface of the star in one case but not the other. For example something like this coronal hole:\n\nhttps://www.reddit.com/r/spaceporn/comments/1n87k64/a_giant_southernhemisphere_coronal_hole_is_now/", "votes": 1, "replies": [ { "id": 3281693, "postDate": "2025-09-04T17:48:41.110Z", "content": "I did consider that - but it doesn't seem consistent with a constant error over the entire transit (but rather with a varying transit depth profile during the transit).
", "rawMarkdown": "I did consider that - but it doesn't seem consistent with a constant error over the entire transit (but rather with a varying transit depth profile during the transit)." }, { "id": 3281713, "postDate": "2025-09-04T18:27:59.233Z", "content": "It could also be that this data mimics the effect that the second observation is another planet which is misidentified :)
", "rawMarkdown": "It could also be that this data mimics the effect that the second observation is another planet which is misidentified :)" } ] }, { "id": 3278697, "postDate": "2025-08-30T11:35:55.487Z", "content": "If anyone wants to audit my results or continue the analysis, here's a notebook that reproduces the key results (not quite the ones above, but refined one from the discussion with Oleh below):
\nhttps://www.kaggle.com/code/jeroencottaar/demonstrate-apparent-label-issue
", "rawMarkdown": "If anyone wants to audit my results or continue the analysis, here's a notebook that reproduces the key results (not quite the ones above, but refined one from the discussion with Oleh below):\n\nhttps://www.kaggle.com/code/jeroencottaar/demonstrate-apparent-label-issue", "votes": 1 }, { "id": 3277122, "postDate": "2025-08-27T11:35:00.877Z", "content": "I still have not added additional observations of exoplanets to my code. I checked my solution for this planet and the first observation is well described.
\nI guess that you need to be a bit more careful when comparing two observations with a simple difference. I am not sure what is your scaling per \"column\" and if it truly removes the background effect. Is this scale factor dependent in time and lambda? To me it looks like the effect is due to background residuals as there are differences in the shoulders (low/high t) as well.
", "rawMarkdown": "I still have not added additional observations of exoplanets to my code. I checked my solution for this planet and the first observation is well described.\n\nI guess that you need to be a bit more careful when comparing two observations with a simple difference. I am not sure what is your scaling per \"column\" and if it truly removes the background effect. Is this scale factor dependent in time and lambda? To me it looks like the effect is due to background residuals as there are differences in the shoulders (low/high t) as well.", "replies": [ { "id": 3277128, "postDate": "2025-08-27T11:47:01.017Z", "content": "Thanks for thinking along. Here's the same plots without the column scaling applied - you can see the high-frequent difference is really confined to the transit itself.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14984949%2F2a4f88d886ece06d15de045a0904d46e%2Fdownload%20(3).png?generation=1756295213994864&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14984949%2Fd5ba48fa999436bd073c492a3aef1fc0%2Fdownload%20(4).png?generation=1756295218234986&alt=media\")
I agree, now it is more conclusive that there is something wrong in the spectrum. Can you also check the ratio instead of difference as in this case the signal should be canceled if it is the same in the modeling: B1(t, lambda)xS(lambda) over B2(t, lambda)xS(lambda) ?
", "rawMarkdown": "I agree, now it is more conclusive that there is something wrong in the spectrum. Can you also check the ratio instead of difference as in this case the signal should be canceled if it is the same in the modeling: B1(t, lambda)xS(lambda) over B2(t, lambda)xS(lambda) ?" }, { "id": 3277135, "postDate": "2025-08-27T12:01:13.323Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14984949%2F5bc7dea15d4e3d7510f516f70099b594%2FScreenshot%202025-08-27%20140029.png?generation=1756296057720968&alt=media\")
This indeed shows it even clearer (at least after low pass filtering).
", "rawMarkdown": "\n\nThis indeed shows it even clearer (at least after low pass filtering).", "votes": 1 }, { "id": 3277141, "postDate": "2025-08-27T12:10:54.323Z", "content": "Now it is evident from data (no modeling dependency) that the ground spectrum used to generate data for two observations is different. I hope that this can be solved because combination of observations should decrease the sigmas by sqrt(2) so quite important
", "rawMarkdown": "Now it is evident from data (no modeling dependency) that the ground spectrum used to generate data for two observations is different. I hope that this can be solved because combination of observations should decrease the sigmas by sqrt(2) so quite important" }, { "id": 3277172, "postDate": "2025-08-27T13:05:37.007Z", "content": "Note that what I want to highlight is that the measurements are inconstent with the labels, i.e. the ground truth spectrum. I just pick this one with 2 transits because it's easy to prove - if the 2 transits are not consistent, at least one of them is not consistent with the ground truth. But I see several single-transit planets with the same issue: I can't find a way to explain the signal from the ground truth.
", "rawMarkdown": "Note that what I want to highlight is that the measurements are inconstent with the labels, i.e. the ground truth spectrum. I just pick this one with 2 transits because it's easy to prove - if the 2 transits are not consistent, at least one of them is not consistent with the ground truth. But I see several single-transit planets with the same issue: I can't find a way to explain the signal from the ground truth." }, { "id": 3294713, "postDate": "2025-09-26T15:12:14.643Z", "content": "Just as additional cross-check, here is what I get from simple curve fits for both observations of this planet:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6327791%2Fd682230c45178088cea16a442448bc69%2Fobs12.png?generation=1758899494206963&alt=media\")
Looks to me compatible…
\nProbably the differences in the raw data ratio within the transit zone that we see is due to the LD effect, which is not canceled out
", "rawMarkdown": "Just as additional cross-check, here is what I get from simple curve fits for both observations of this planet:\n\n\nLooks to me compatible...\n\nProbably the differences in the raw data ratio within the transit zone that we see is due to the LD effect, which is not canceled out" } ] } ] }, { "id": 3277116, "postDate": "2025-08-27T11:13:28.137Z", "content": "Hi @jeroencottaar ,
\nHad a quick look at #1349926825, looks normal on initial look but i might be missing something.
\nHere is my quick plot of the (normalised) white light curve of the two observations (ignore y label) - ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18942071%2Fb2d043a205596e7141237a93200ecb47%2Fwlc_alignment.png?generation=1756292774784909&alt=media\")
They look reasonably aligned to me - i have also checked the alignment between each spectroscopic lightcurves (org obs and the repeat obs)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18942071%2Fb306aa27ce1ec1c4167d2ba747e64f90%2Fcorreletion_between_obs.png?generation=1756292824142987&alt=media\")
despite the fact that they had different t0s, They look positively correlated to each other which is reassuring ( I have omitted some cosmic ray which can get very large)
", "rawMarkdown": "Hi @jeroencottaar , \n\nHad a quick look at #1349926825, looks normal on initial look but i might be missing something. \n\nHere is my quick plot of the (normalised) white light curve of the two observations (ignore y label) - \n\nThey look reasonably aligned to me - i have also checked the alignment between each spectroscopic lightcurves (org obs and the repeat obs)\n\n\n\ndespite the fact that they had different t0s, They look positively correlated to each other which is reassuring ( I have omitted some cosmic ray which can get very large)\n\n", "replies": [ { "id": 3277119, "postDate": "2025-08-27T11:23:55.920Z", "content": "Indeed the overall profiles make sense. But my point is that the difference between the AIRS measurements, shown above, can only be explained by the transit depths being different. Not a big difference (don't have the number at hand), but there must be a difference nonetheless. And in modeling, neither of the individual transit depths corresponds to the given labels (I don't show these results above).
\nOf course the explanation could just be some physics that I'm missing, and then of course you can't reveal it. But this only happens on a small subset of planets, while the vast majority model just fine.
", "rawMarkdown": "Indeed the overall profiles make sense. But my point is that the difference between the AIRS measurements, shown above, can only be explained by the transit depths being different. Not a big difference (don't have the number at hand), but there must be a difference nonetheless. And in modeling, neither of the individual transit depths corresponds to the given labels (I don't show these results above).\n\nOf course the explanation could just be some physics that I'm missing, and then of course you can't reveal it. But this only happens on a small subset of planets, while the vast majority model just fine. ", "votes": 1, "replies": [ { "id": 3277536, "postDate": "2025-08-28T06:44:44.970Z", "content": "@gordonyip Throughout the data, I'm seeing patterns that defy all explanation based on my knowledge of the physics, and that are entirely unlike anything seen in last year's data. I simply cannot properly model the data using the given labels. However, the data perfect follows the expected behavior with different labels.
\nIs there any possibility that there is some noise on the provided training labels (and also on the test labels)? Similar to how there is noise on the transit parameters such as orbital period. It's the only explanation I can come up with…
", "rawMarkdown": "@gordonyip Throughout the data, I'm seeing patterns that defy all explanation based on my knowledge of the physics, and that are entirely unlike anything seen in last year's data. I simply cannot properly model the data using the given labels. However, the data perfect follows the expected behavior with different labels.\n\nIs there any possibility that there is some noise on the provided training labels (and also on the test labels)? Similar to how there is noise on the transit parameters such as orbital period. It's the only explanation I can come up with...", "votes": 1 }, { "id": 3277601, "postDate": "2025-08-28T10:59:14.360Z", "content": "Hi @jeroencottaar, We took a bit of time to dive into the data product before we reply again.
\n
\nWe can confirm that both observations in #1349926825 were generated using the same GT spectrum, and that the planetary properties are the same. To ensure the GT spectrum corresponds to the observations, we extracted the apparent transit depth from the white light curve and compared it to the mean of the GT spectrum (see figure below). It shows a positive correlation. While we understand the discrepancy is between AIRS measurements, given that the spectrum's dynamic range is quite large, it would be very unlikely to have an incorrect input spectrum with a very similar \"level\" (mean of a spectrum).
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18942071%2Fefc4d9c7cb18d26619e083a0efd1392e%2Fcorrelation.png?generation=1756378402803552&alt=media\")
\n
\nWe have also done some quick spectral extraction on our end, and the two observations returned very similar spectra.
\nLooking at the figures from the discussion, it appears (of course we could be wrong) that the discrepancy is about ~1% or represents a small variation in the transit region, which could be due to residual noise not being properly accounted for.
While the changes to the lightcurve itself look minimal this time around, the additional effects compared to ADC24 could actually produce biased results if noise is not properly accounted for. How to properly account for it remains a hot topic of discussion, even within the science consortium.
\nI hope this helps clarify things? Please let us know if you have more questions. We will try to answer as best we can without revealing the mechanism. However, we can advise that noise is difficult to estimate.
", "rawMarkdown": "Hi @jeroencottaar, We took a bit of time to dive into the data product before we reply again. \n \nWe can confirm that both observations in #1349926825 were generated using the same GT spectrum, and that the planetary properties are the same. To ensure the GT spectrum corresponds to the observations, we extracted the apparent transit depth from the white light curve and compared it to the mean of the GT spectrum (see figure below). It shows a positive correlation. While we understand the discrepancy is between AIRS measurements, given that the spectrum's dynamic range is quite large, it would be very unlikely to have an incorrect input spectrum with a very similar \"level\" (mean of a spectrum).\n\n \n \nWe have also done some quick spectral extraction on our end, and the two observations returned very similar spectra.\nLooking at the figures from the discussion, it appears (of course we could be wrong) that the discrepancy is about ~1% or represents a small variation in the transit region, which could be due to residual noise not being properly accounted for.\n\nWhile the changes to the lightcurve itself look minimal this time around, the additional effects compared to ADC24 could actually produce biased results if noise is not properly accounted for. How to properly account for it remains a hot topic of discussion, even within the science consortium.\n\nI hope this helps clarify things? Please let us know if you have more questions. We will try to answer as best we can without revealing the mechanism. However, we can advise that noise is difficult to estimate.", "votes": 2 }, { "id": 3277618, "postDate": "2025-08-28T11:52:46.970Z", "content": "@gordonyip, thanks for following up.
\nI'm indeed talking about small differences, on the order of 1%. And the transit depth extracted from either transit looks approximately OK. But it's actually quite far off - it leads to an error of >1000 PPM, far more than would be expected based on error sources that I know of.
\nI'll highlight again the last figure out of the discussion with @olehkivernyk above, which shows the ratio between the data for the two transits. Note that this is not the result of any transit modeling, it's just the data as imported with minimal preprocessing (such as inpainting). There are high-frequent (in wavelength) differences of ~1% that occur only inside the transit zone, while everything outside behaves the same over the transits.
\nSo if this is caused by noise terms that are unaccounted for, they somehow manifest only during the transit, or have some odd interaction with the transit. I just can't imagine this, and also can't find anything in the literature that would cause something like this (admittedly I haven't searched that hard yet…)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F14984949%2F501abef13b3052ce6f728b05418f3c26%2FScreenshot%202025-08-27%20140029.png?generation=1756381986818529&alt=media\")
This is an interesting thing to notice! Can I ask to what extent the ups and downs here correlate with the ground-truth spectrum for this planet? E.g., does that vertical big blue bar at around 120 or so on the x-axis of your plot correspond to a bump in the ground-truth spectrum?
", "rawMarkdown": "This is an interesting thing to notice! Can I ask to what extent the ups and downs here correlate with the ground-truth spectrum for this planet? E.g., does that vertical big blue bar at around 120 or so on the x-axis of your plot correspond to a bump in the ground-truth spectrum? " }, { "id": 3277997, "postDate": "2025-08-29T03:56:01.623Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F84327%2F767b4922b9ffedad69bd9c98ed8d0b72%2Ftemp.png?generation=1756439553713730&alt=media\")
Answering my own question: here is the ground-truth spectrum for planet 1349926825; if there is any similarity to the pattern of vertical bands in Jeroen's plot, it is not strong enough for my eye to see it clearly
", "rawMarkdown": "\n\nAnswering my own question: here is the ground-truth spectrum for planet 1349926825; if there is any similarity to the pattern of vertical bands in Jeroen's plot, it is not strong enough for my eye to see it clearly" }, { "id": 3281460, "postDate": "2025-09-04T10:38:11.977Z", "content": "Seems similar? The blue bar looks to be the dip at 120 with two yellow peaks (~0.088) to the left and the smaller peak to the right ~160 (~0.078).
\nThough I'm not sure why the ratio of two transits would correlate with the spectrum in this manner…
", "rawMarkdown": "Seems similar? The blue bar looks to be the dip at 120 with two yellow peaks (~0.088) to the left and the smaller peak to the right ~160 (~0.078).\n\nThough I'm not sure why the ratio of two transits would correlate with the spectrum in this manner..." }, { "id": 3294605, "postDate": "2025-09-26T12:00:33.400Z", "content": "\n\nHi @jeroencottaar, We took a bit of time to dive into the data product before we reply again.
\n
\n
\n We can confirm that both observations in #1349926825 were generated using the same GT spectrum, and that the planetary properties are the same. To ensure the GT spectrum corresponds to the observations, we extracted the apparent transit depth from the white light curve and compared it to the mean of the GT spectrum (see figure below). It shows a positive correlation. While we understand the discrepancy is between AIRS measurements, given that the spectrum's dynamic range is quite large, it would be very unlikely to have an incorrect input spectrum with a very similar \"level\" (mean of a spectrum).\n
\n
\n We have also done some quick spectral extraction on our end, and the two observations returned very similar spectra.
\n Looking at the figures from the discussion, it appears (of course we could be wrong) that the discrepancy is about ~1% or represents a small variation in the transit region, which could be due to residual noise not being properly accounted for.While the changes to the lightcurve itself look minimal this time around, the additional effects compared to ADC24 could actually produce biased results if noise is not properly accounted for. How to properly account for it remains a hot topic of discussion, even within the science consortium.
\nI hope this helps clarify things? Please let us know if you have more questions. We will try to answer as best we can without revealing the mechanism. However, we can advise that noise is difficult to estimate.
\n
@gordonyip Now that the competition is over, are you willing to reveal more about what's going on here? I spent a lot of time on this and got no closer to figuring it out.
", "rawMarkdown": "> Hi @jeroencottaar, We took a bit of time to dive into the data product before we reply again. \n> \n> We can confirm that both observations in #1349926825 were generated using the same GT spectrum, and that the planetary properties are the same. To ensure the GT spectrum corresponds to the observations, we extracted the apparent transit depth from the white light curve and compared it to the mean of the GT spectrum (see figure below). It shows a positive correlation. While we understand the discrepancy is between AIRS measurements, given that the spectrum's dynamic range is quite large, it would be very unlikely to have an incorrect input spectrum with a very similar \"level\" (mean of a spectrum).\n> \n> \n> \n> We have also done some quick spectral extraction on our end, and the two observations returned very similar spectra.\n> Looking at the figures from the discussion, it appears (of course we could be wrong) that the discrepancy is about ~1% or represents a small variation in the transit region, which could be due to residual noise not being properly accounted for.\n> \n> While the changes to the lightcurve itself look minimal this time around, the additional effects compared to ADC24 could actually produce biased results if noise is not properly accounted for. How to properly account for it remains a hot topic of discussion, even within the science consortium.\n> \n> I hope this helps clarify things? Please let us know if you have more questions. We will try to answer as best we can without revealing the mechanism. However, we can advise that noise is difficult to estimate.\n\n@gordonyip Now that the competition is over, are you willing to reveal more about what's going on here? I spent a lot of time on this and got no closer to figuring it out." }, { "id": 3296581, "postDate": "2025-10-01T08:05:26.550Z", "content": "Very happy to - We can confirm as we said before, the simulation is done using the same GT spectrum with both observations. We think (but this is based on looking at the diagram so we could be wrong), this could be caused by not fitting gain drift and the transit model together, There is also a change in the host star's brightness between the observation (to simulate the fact that stars do change brightness), but we think the effect caused by this should not be substantial if you treat each observation independently.
", "rawMarkdown": "Very happy to - We can confirm as we said before, the simulation is done using the same GT spectrum with both observations. We think (but this is based on looking at the diagram so we could be wrong), this could be caused by not fitting gain drift and the transit model together, There is also a change in the host star's brightness between the observation (to simulate the fact that stars do change brightness), but we think the effect caused by this should not be substantial if you treat each observation independently. " }, { "id": 3296582, "postDate": "2025-10-01T08:05:53.330Z", "content": "leave your question below - I have the experts with me during the conference and can ask them :)
", "rawMarkdown": "leave your question below - I have the experts with me during the conference and can ask them :)" }, { "id": 3296590, "postDate": "2025-10-01T08:15:57.670Z", "content": "As I mentioned here, I managed to reproduce the true spectrum from both observations within 1-2 sigma. They are treated as independent measurements. Therefore, I attribute the effect in raw data ratio to the LD - there is no constrain that the LD between observations and vs lambda should be exactly the same.
", "rawMarkdown": "As I mentioned [here](https://www.kaggle.com/competitions/ariel-data-challenge-2025/discussion/602425#3294713), I managed to reproduce the true spectrum from both observations within 1-2 sigma. They are treated as independent measurements. Therefore, I attribute the effect in raw data ratio to the LD - there is no constrain that the LD between observations and vs lambda should be exactly the same." }, { "id": 3296592, "postDate": "2025-10-01T08:23:36.657Z", "content": "Ah! yes, sorry for missing it , yes i can confirm on our end that the parameters used to produce LD effect is not the same as the input stellar parameters nor the provided (they have been perturbed around the input value.
", "rawMarkdown": "Ah! yes, sorry for missing it , yes i can confirm on our end that the parameters used to produce LD effect is not the same as the input stellar parameters nor the provided (they have been perturbed around the input value. " }, { "id": 3296597, "postDate": "2025-10-01T08:39:46.700Z", "content": "I don't think this (or gain drift) explains the effect though. Note that there is no modeling involved to get this plot - it's just the ratio of the photon counts between the transit (with some preprocessing, most notably low pass filtering - see also the code).
\nWhat we're seeing is:
\nThis excludes anything related to gain drift as a root cause; this would not be limited to the transit specifically. Note also that it's unrelated to how gain drift is modeled, since no modeling takes place here at all. (As a side note, my full model does model gain drift and the transit simultaneously.)
\nLimb darkening isn't so easy to 100% exclude. Still, it shouldn't be constant in time over the transit (but this is a bit trickier to draw conclusion on, because the low pass filtering could be masking this). But it would also mean high-frequent limb darkening variation over wavelength, which I have otherwise seen no evidence of.
", "rawMarkdown": "I don't think this (or gain drift) explains the effect though. Note that there is no modeling involved to get this plot - it's just the ratio of the photon counts between the transit (with some preprocessing, most notably low pass filtering - see also the [code](https://www.kaggle.com/code/jeroencottaar/demonstrate-apparent-label-issue)).\n\nWhat we're seeing is:\n\n- The photon counts are consistent outside the transit. Note that it's low-pass-filtered, so this is still reasonably despite the change in star brightness, which is presumably low-frequent in wavelength.\n- The photon counts are inconsistent inside the transit. The error is pretty much constant over time, but high-frequent in wavelength.\n\nThis excludes anything related to gain drift as a root cause; this would not be limited to the transit specifically. Note also that it's unrelated to how gain drift is modeled, since no modeling takes place here at all. (As a side note, my full model does model gain drift and the transit simultaneously.)\n\nLimb darkening isn't so easy to 100% exclude. Still, it shouldn't be constant in time over the transit (but this is a bit trickier to draw conclusion on, because the low pass filtering could be masking this). But it would also mean high-frequent limb darkening variation over wavelength, which I have otherwise seen no evidence of." } ] }, { "id": 3296812, "postDate": "2025-10-01T16:45:55.837Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 3281690, "author_name": "Thomas Dueholm Hansen", "author_url": "", "post_date": "2025-09-04T17:39:19.830000", "content": "Depending on how advanced the simulation is, the discrepancy might be caused by the planet passing in front of some anomaly on the surface of the star in one case but not the other. For example something like this coronal hole:
\nhttps://www.reddit.com/r/spaceporn/comments/1n87k64/a_giant_southernhemisphere_coronal_hole_is_now/
", "votes": 1, "replies": [ { "id": 3281693, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-09-04T17:48:41.110000", "content": "I did consider that - but it doesn't seem consistent with a constant error over the entire transit (but rather with a varying transit depth profile during the transit).
", "votes": 0, "replies": [] }, { "id": 3281713, "author_name": "Oleh Kivernyk", "author_url": "", "post_date": "2025-09-04T18:27:59.233000", "content": "It could also be that this data mimics the effect that the second observation is another planet which is misidentified :)
", "votes": 0, "replies": [] } ] }, { "id": 3278697, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-08-30T11:35:55.487000", "content": "If anyone wants to audit my results or continue the analysis, here's a notebook that reproduces the key results (not quite the ones above, but refined one from the discussion with Oleh below):
\nhttps://www.kaggle.com/code/jeroencottaar/demonstrate-apparent-label-issue
", "votes": 1, "replies": [] }, { "id": 3277122, "author_name": "Oleh Kivernyk", "author_url": "", "post_date": "2025-08-27T11:35:00.877000", "content": "I still have not added additional observations of exoplanets to my code. I checked my solution for this planet and the first observation is well described.
\nI guess that you need to be a bit more careful when comparing two observations with a simple difference. I am not sure what is your scaling per \"column\" and if it truly removes the background effect. Is this scale factor dependent in time and lambda? To me it looks like the effect is due to background residuals as there are differences in the shoulders (low/high t) as well.
", "votes": 0, "replies": [ { "id": 3277128, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-08-27T11:47:01.017000", "content": "Thanks for thinking along. Here's the same plots without the column scaling applied - you can see the high-frequent difference is really confined to the transit itself.
\n
I agree, now it is more conclusive that there is something wrong in the spectrum. Can you also check the ratio instead of difference as in this case the signal should be canceled if it is the same in the modeling: B1(t, lambda)xS(lambda) over B2(t, lambda)xS(lambda) ?
", "votes": 0, "replies": [] }, { "id": 3277135, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-08-27T12:01:13.323000", "content": "
This indeed shows it even clearer (at least after low pass filtering).
", "votes": 1, "replies": [] }, { "id": 3277141, "author_name": "Oleh Kivernyk", "author_url": "", "post_date": "2025-08-27T12:10:54.323000", "content": "Now it is evident from data (no modeling dependency) that the ground spectrum used to generate data for two observations is different. I hope that this can be solved because combination of observations should decrease the sigmas by sqrt(2) so quite important
", "votes": 0, "replies": [] }, { "id": 3277172, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-08-27T13:05:37.007000", "content": "Note that what I want to highlight is that the measurements are inconstent with the labels, i.e. the ground truth spectrum. I just pick this one with 2 transits because it's easy to prove - if the 2 transits are not consistent, at least one of them is not consistent with the ground truth. But I see several single-transit planets with the same issue: I can't find a way to explain the signal from the ground truth.
", "votes": 0, "replies": [] }, { "id": 3294713, "author_name": "Oleh Kivernyk", "author_url": "", "post_date": "2025-09-26T15:12:14.643000", "content": "Just as additional cross-check, here is what I get from simple curve fits for both observations of this planet:
\n
Looks to me compatible…
\nProbably the differences in the raw data ratio within the transit zone that we see is due to the LD effect, which is not canceled out
", "votes": 0, "replies": [] } ] } ] }, { "id": 3277116, "author_name": "Gordon Yip", "author_url": "", "post_date": "2025-08-27T11:13:28.137000", "content": "Hi @jeroencottaar ,
\nHad a quick look at #1349926825, looks normal on initial look but i might be missing something.
\nHere is my quick plot of the (normalised) white light curve of the two observations (ignore y label) -
They look reasonably aligned to me - i have also checked the alignment between each spectroscopic lightcurves (org obs and the repeat obs)
\n
despite the fact that they had different t0s, They look positively correlated to each other which is reassuring ( I have omitted some cosmic ray which can get very large)
", "votes": 0, "replies": [ { "id": 3277119, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-08-27T11:23:55.920000", "content": "Indeed the overall profiles make sense. But my point is that the difference between the AIRS measurements, shown above, can only be explained by the transit depths being different. Not a big difference (don't have the number at hand), but there must be a difference nonetheless. And in modeling, neither of the individual transit depths corresponds to the given labels (I don't show these results above).
\nOf course the explanation could just be some physics that I'm missing, and then of course you can't reveal it. But this only happens on a small subset of planets, while the vast majority model just fine.
", "votes": 1, "replies": [ { "id": 3277536, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-08-28T06:44:44.970000", "content": "@gordonyip Throughout the data, I'm seeing patterns that defy all explanation based on my knowledge of the physics, and that are entirely unlike anything seen in last year's data. I simply cannot properly model the data using the given labels. However, the data perfect follows the expected behavior with different labels.
\nIs there any possibility that there is some noise on the provided training labels (and also on the test labels)? Similar to how there is noise on the transit parameters such as orbital period. It's the only explanation I can come up with…
", "votes": 1, "replies": [] }, { "id": 3277601, "author_name": "Gordon Yip", "author_url": "", "post_date": "2025-08-28T10:59:14.360000", "content": "Hi @jeroencottaar, We took a bit of time to dive into the data product before we reply again.
\n
\nWe can confirm that both observations in #1349926825 were generated using the same GT spectrum, and that the planetary properties are the same. To ensure the GT spectrum corresponds to the observations, we extracted the apparent transit depth from the white light curve and compared it to the mean of the GT spectrum (see figure below). It shows a positive correlation. While we understand the discrepancy is between AIRS measurements, given that the spectrum's dynamic range is quite large, it would be very unlikely to have an incorrect input spectrum with a very similar \"level\" (mean of a spectrum).
\n
\nWe have also done some quick spectral extraction on our end, and the two observations returned very similar spectra.
\nLooking at the figures from the discussion, it appears (of course we could be wrong) that the discrepancy is about ~1% or represents a small variation in the transit region, which could be due to residual noise not being properly accounted for.
While the changes to the lightcurve itself look minimal this time around, the additional effects compared to ADC24 could actually produce biased results if noise is not properly accounted for. How to properly account for it remains a hot topic of discussion, even within the science consortium.
\nI hope this helps clarify things? Please let us know if you have more questions. We will try to answer as best we can without revealing the mechanism. However, we can advise that noise is difficult to estimate.
", "votes": 2, "replies": [] }, { "id": 3277618, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-08-28T11:52:46.970000", "content": "@gordonyip, thanks for following up.
\nI'm indeed talking about small differences, on the order of 1%. And the transit depth extracted from either transit looks approximately OK. But it's actually quite far off - it leads to an error of >1000 PPM, far more than would be expected based on error sources that I know of.
\nI'll highlight again the last figure out of the discussion with @olehkivernyk above, which shows the ratio between the data for the two transits. Note that this is not the result of any transit modeling, it's just the data as imported with minimal preprocessing (such as inpainting). There are high-frequent (in wavelength) differences of ~1% that occur only inside the transit zone, while everything outside behaves the same over the transits.
\nSo if this is caused by noise terms that are unaccounted for, they somehow manifest only during the transit, or have some odd interaction with the transit. I just can't imagine this, and also can't find anything in the literature that would cause something like this (admittedly I haven't searched that hard yet…)
\n
This is an interesting thing to notice! Can I ask to what extent the ups and downs here correlate with the ground-truth spectrum for this planet? E.g., does that vertical big blue bar at around 120 or so on the x-axis of your plot correspond to a bump in the ground-truth spectrum?
", "votes": 0, "replies": [] }, { "id": 3277997, "author_name": "particlebbq", "author_url": "", "post_date": "2025-08-29T03:56:01.623000", "content": "
Answering my own question: here is the ground-truth spectrum for planet 1349926825; if there is any similarity to the pattern of vertical bands in Jeroen's plot, it is not strong enough for my eye to see it clearly
", "votes": 0, "replies": [] }, { "id": 3281460, "author_name": "sroger", "author_url": "", "post_date": "2025-09-04T10:38:11.977000", "content": "Seems similar? The blue bar looks to be the dip at 120 with two yellow peaks (~0.088) to the left and the smaller peak to the right ~160 (~0.078).
\nThough I'm not sure why the ratio of two transits would correlate with the spectrum in this manner…
", "votes": 0, "replies": [] }, { "id": 3294605, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-09-26T12:00:33.400000", "content": "\n\nHi @jeroencottaar, We took a bit of time to dive into the data product before we reply again.
\n
\n
\n We can confirm that both observations in #1349926825 were generated using the same GT spectrum, and that the planetary properties are the same. To ensure the GT spectrum corresponds to the observations, we extracted the apparent transit depth from the white light curve and compared it to the mean of the GT spectrum (see figure below). It shows a positive correlation. While we understand the discrepancy is between AIRS measurements, given that the spectrum's dynamic range is quite large, it would be very unlikely to have an incorrect input spectrum with a very similar \"level\" (mean of a spectrum).\n
\n
\n We have also done some quick spectral extraction on our end, and the two observations returned very similar spectra.
\n Looking at the figures from the discussion, it appears (of course we could be wrong) that the discrepancy is about ~1% or represents a small variation in the transit region, which could be due to residual noise not being properly accounted for.While the changes to the lightcurve itself look minimal this time around, the additional effects compared to ADC24 could actually produce biased results if noise is not properly accounted for. How to properly account for it remains a hot topic of discussion, even within the science consortium.
\nI hope this helps clarify things? Please let us know if you have more questions. We will try to answer as best we can without revealing the mechanism. However, we can advise that noise is difficult to estimate.
\n
@gordonyip Now that the competition is over, are you willing to reveal more about what's going on here? I spent a lot of time on this and got no closer to figuring it out.
", "votes": 0, "replies": [] }, { "id": 3296581, "author_name": "Gordon Yip", "author_url": "", "post_date": "2025-10-01T08:05:26.550000", "content": "Very happy to - We can confirm as we said before, the simulation is done using the same GT spectrum with both observations. We think (but this is based on looking at the diagram so we could be wrong), this could be caused by not fitting gain drift and the transit model together, There is also a change in the host star's brightness between the observation (to simulate the fact that stars do change brightness), but we think the effect caused by this should not be substantial if you treat each observation independently.
", "votes": 0, "replies": [] }, { "id": 3296582, "author_name": "Gordon Yip", "author_url": "", "post_date": "2025-10-01T08:05:53.330000", "content": "leave your question below - I have the experts with me during the conference and can ask them :)
", "votes": 0, "replies": [] }, { "id": 3296590, "author_name": "Oleh Kivernyk", "author_url": "", "post_date": "2025-10-01T08:15:57.670000", "content": "As I mentioned here, I managed to reproduce the true spectrum from both observations within 1-2 sigma. They are treated as independent measurements. Therefore, I attribute the effect in raw data ratio to the LD - there is no constrain that the LD between observations and vs lambda should be exactly the same.
", "votes": 0, "replies": [] }, { "id": 3296592, "author_name": "Gordon Yip", "author_url": "", "post_date": "2025-10-01T08:23:36.657000", "content": "Ah! yes, sorry for missing it , yes i can confirm on our end that the parameters used to produce LD effect is not the same as the input stellar parameters nor the provided (they have been perturbed around the input value.
", "votes": 0, "replies": [] }, { "id": 3296597, "author_name": "Jeroen Cottaar", "author_url": "", "post_date": "2025-10-01T08:39:46.700000", "content": "I don't think this (or gain drift) explains the effect though. Note that there is no modeling involved to get this plot - it's just the ratio of the photon counts between the transit (with some preprocessing, most notably low pass filtering - see also the code).
\nWhat we're seeing is:
\nThis excludes anything related to gain drift as a root cause; this would not be limited to the transit specifically. Note also that it's unrelated to how gain drift is modeled, since no modeling takes place here at all. (As a side note, my full model does model gain drift and the transit simultaneously.)
\nLimb darkening isn't so easy to 100% exclude. Still, it shouldn't be constant in time over the transit (but this is a bit trickier to draw conclusion on, because the low pass filtering could be masking this). But it would also mean high-frequent limb darkening variation over wavelength, which I have otherwise seen no evidence of.
", "votes": 0, "replies": [] } ] }, { "id": 3296812, "author_name": "", "author_url": "", "post_date": "2025-10-01T16:45:55.837000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "3277040": "I have a model that does quite well on most planets, but there are some occasional planets that it just can't model properly. I'm pretty convinced by now there is something very weird going on here - either a bug in the data generation, or some physical effect I've never heard of.\n\nI can't really describe the problem in full without revealing my model, but there's one planet that allows me to show the effect I mean in a simple way. This is #1349926825. It has two transits, and this is the difference between the AIRS measurements for the two planets (X axis is wavelength, Y axis is time):\n\n\n\nThe effect I'm trying to highlight is more apparent after a low pass filter:\n\n.png?generation=1756280741462536&alt=media)\n\nIt appears there is a clear difference in the transit depth between the two transits. Nothing in the expected physics explains this properly. What I can come up with:\n- The planet's properties are actually different between the two transits, and we are expected to average them => that would be odd without us being explicitly informed, and as mentioned I see this problem on more planets, also with 1 transit.\n- There is a significant foreground signal that differs between the transits => there is nothing near large enough, which can be seen from analysis of the raw AIRS signals.\n\nAnyone have any ideas? Also tagging @gordonyip \n\n\nNOTE: some light preprocessing was applied to the figures above, which cannot explain what we see:\n- Standard preprocessing as provided by host (with minor modifications)\n- Shift one signal by 50 pixels in time to overlap the transits\n- Apply a scaling per column to correct for different stellar spectrum (since this is a fixed scaling per column, it does not cause the observed differences within columns)", "3281690": "Depending on how advanced the simulation is, the discrepancy might be caused by the planet passing in front of some anomaly on the surface of the star in one case but not the other. For example something like this coronal hole:\n\nhttps://www.reddit.com/r/spaceporn/comments/1n87k64/a_giant_southernhemisphere_coronal_hole_is_now/", "3278697": "If anyone wants to audit my results or continue the analysis, here's a notebook that reproduces the key results (not quite the ones above, but refined one from the discussion with Oleh below):\n\nhttps://www.kaggle.com/code/jeroencottaar/demonstrate-apparent-label-issue", "3277122": "I still have not added additional observations of exoplanets to my code. I checked my solution for this planet and the first observation is well described.\n\nI guess that you need to be a bit more careful when comparing two observations with a simple difference. I am not sure what is your scaling per \"column\" and if it truly removes the background effect. Is this scale factor dependent in time and lambda? To me it looks like the effect is due to background residuals as there are differences in the shoulders (low/high t) as well.", "3277116": "Hi @jeroencottaar , \n\nHad a quick look at #1349926825, looks normal on initial look but i might be missing something. \n\nHere is my quick plot of the (normalised) white light curve of the two observations (ignore y label) - \n\nThey look reasonably aligned to me - i have also checked the alignment between each spectroscopic lightcurves (org obs and the repeat obs)\n\n\n\ndespite the fact that they had different t0s, They look positively correlated to each other which is reassuring ( I have omitted some cosmic ray which can get very large)\n\n" } }