{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":101849,"databundleVersionId":12846694,"sourceType":"competition"}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on June 27, 2025. By Prata, Marília (mpwolke)","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-06-27T02:56:51.787611Z","iopub.execute_input":"2025-06-27T02:56:51.787981Z","iopub.status.idle":"2025-06-27T02:57:15.017235Z","shell.execute_reply.started":"2025-06-27T02:56:51.787955Z","shell.execute_reply":"2025-06-27T02:57:15.016096Z"},"_kg_hide-input":true,"_kg_hide-output":true,"collapsed":true,"jupyter":{"outputs_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Competition Citation\n\n@misc{ariel-data-challenge-2025,\n    author = {Kai Hou Yip, Lorenzo V. Mugnai, Rebecca L. Coates, Andrea Bocchieri, Orphée Faucoz, Arun Nambiyath Govindan, Giuseppe Morello, Andreas Papageorgiou, Angèle Syty, Tara Tahseen, Sohier Dane, Maggie Demkin, Jean-Philippe Beaulieu, Sudeshna Boro Saikia, Giovanni Bruno, Quentin Changeat, Camilla Danielski, Pascale Danto, Jack Davey, Pierre Drossart, Paul Eccleston, Billy Edwards, Clare Jenner, Ryan King, Theresa Lueftinger, Michiel Min, Nikolaos Nikolaou, Leonardo Pagliaro, Enzo Pascale, Emilie Panek, Alice Radcliffe, Luís F. Simões, Patricio Cubillos Vallejos, Tiziano Zingales, Giovanna Tinetti, Ingo P. Waldmann. NeurIPS - Ariel Data Challenge 2025. https://kaggle.com/competitions/ariel-data-challenge-2025, Unpublished. Kaggle.   },\n    \n    title = {NeurIPS - Ariel Data Challenge 2025},\n    \n    year = {2025},\n    \n    howpublished = {\\url{https://kaggle.com/competitions/ariel-data-challenge-2025}},\n    note = {Kaggle}\n}","metadata":{}},{"cell_type":"markdown","source":"## Description\n\n\"When an exoplanet passes in front of its host star, a small amount of starlight filters through the planet's atmosphere. This technique, known as transit spectroscopy, enables researchers to analyze atmospheric composition. However, these signals are incredibly faint, often hidden by complex, time-dependent noise from both the instruments and the stars themselves.\"\n\n\"In this competition, you'll work with simulated data from ESA's Ariel mission, which aims to characterize 1,000 exoplanets after its 2029 launch. Your goal is to recover the true exoplanet spectrum from these noisy observations.\" \n\nhttps://www.kaggle.com/competitions/ariel-data-challenge-2025","metadata":{}},{"cell_type":"markdown","source":"## [train/test]_star_info.csv\n\nplanet_id: Unique identifier for the star-planet system.\n\nRs: Stellar radius in solar radii (R☉).\n\nMs: Stellar mass in solar masses (M☉).\n\nTs: Stellar effective temperature in Kelvin.\n\nMp: Planetary mass in Earth masses (M⊕).\n\ne: Orbital eccentricity (dimensionless).\n\nP: Orbital period in days.\n\nsma: Semi-major axis in stellar radii (Rs), showing the orbital distance relative to the stellar radii.\n\ni: Orbital inclination in degrees.","metadata":{}},{"cell_type":"code","source":"star = pd.read_csv('/kaggle/input/ariel-data-challenge-2025/train_star_info.csv')\nstar.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:16:57.30923Z","iopub.execute_input":"2025-06-27T03:16:57.309553Z","iopub.status.idle":"2025-06-27T03:16:57.332602Z","shell.execute_reply.started":"2025-06-27T03:16:57.30953Z","shell.execute_reply":"2025-06-27T03:16:57.331866Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## train.csv Ground truth spectra.","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/ariel-data-challenge-2025/train.csv')\ntrain.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:19:30.187054Z","iopub.execute_input":"2025-06-27T03:19:30.187349Z","iopub.status.idle":"2025-06-27T03:19:30.36122Z","shell.execute_reply.started":"2025-06-27T03:19:30.187309Z","shell.execute_reply":"2025-06-27T03:19:30.360356Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Calibration Files\n\n\"Calibration files record the electronic characteristics of the sensor and serve as \"supporting frames\" used in image post-processing to image signal to noise ratio. This website contains a brief introduction about the concept of calibration frames. This notebook may also be useful. The calibration files are largely duplicated.\"\n\n**[train/test]/[planet_id]/[AIRS-CH0/FGS1]_calibration/dark.parquet:** Dark frames are exposures taken with the shutter closed, capturing the thermal noise and bias level of the sensor. These are used to subtract the dark current from science images.\n\n**[train/test]/[planet_id]/[AIRS-CH0/FGS1]_calibration/dead.parquet:** Identifies dead or hot pixels on the sensor. Dead pixels do not respond to light, while hot pixels consistently produce high signal levels regardless of incoming light.\n\n**[train/test]/[planet_id]/[AIRS-CH0/FGS1]_calibration/flat.parquet:** Flat field frames are created by imaging a uniformly illuminated surface. They are used to correct for variations in pixel-to-pixel sensitivity and optical system irregularities.\n\n**[train/test]/[planet_id]/[AIRS-CH0/FGS1]_calibration/linear_corr.parquet:** Information about the linearity correction of the sensor. The response of the pixels in the detector becomes less linear as they fill with electrons, approaching the point of saturation, where the pixel can no longer collect additional electrons and its response to light becomes flat. For an accurate estimate of the signal, the instrument's response as a function of the received charge is calibrated, and the correction is calculated using a polynomial of degree n. This polynomial allows for the conversion of the number of electrons collected/measured by the pixel into the number of electrons that the detector would have generated with a linear response.\n\n**[train/test]/[planet_id]/[AIRS-CH0/FGS1]_calibration/read.parquet:** Read noise frames capture the electronic noise introduced during the readout process of the sensor. This noise is present even when no light falls on the detector.\n\nhttps://www.kaggle.com/competitions/ariel-data-challenge-2025/data","metadata":{}},{"cell_type":"markdown","source":"## axis_info.parquet \n\nAxis information for both instruments (**AIRS-CH0 and FGS1**).","metadata":{}},{"cell_type":"code","source":"axis = pd.read_parquet(\"/kaggle/input/ariel-data-challenge-2025/axis_info.parquet\")\naxis.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:25:58.661013Z","iopub.execute_input":"2025-06-27T03:25:58.661768Z","iopub.status.idle":"2025-06-27T03:25:58.684816Z","shell.execute_reply.started":"2025-06-27T03:25:58.66174Z","shell.execute_reply":"2025-06-27T03:25:58.683999Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Calibration frames  (parquet files)\n\n**Dark, Dead, Flat, Linear_corr and Read** parquet frames\n\nCalibration frames are essential for eye-catching images, although it can be time consuming to acquire them you will notice a difference in your images when you do take the time.\n\nhttps://practicalastrophotography.com/a-brief-guide-to-calibration-frames/","metadata":{}},{"cell_type":"markdown","source":"### Dead Calibration frame  - Atmospheric Infrared Sounder (AIRS)\n\nOn AIRS-CH0_calibration - **Atmospheric Infrared Sounder (AIRS)**\n\nhttps://eospso.gsfc.nasa.gov/sites/default/files/atbd/AIRS_L1B_ATBD_Part_1.pdf","metadata":{}},{"cell_type":"code","source":"dead = pd.read_parquet(\"/kaggle/input/ariel-data-challenge-2025/train/1010375142/AIRS-CH0_calibration_0/dead.parquet\")\ndead.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:31:05.426179Z","iopub.execute_input":"2025-06-27T03:31:05.426734Z","iopub.status.idle":"2025-06-27T03:31:05.478172Z","shell.execute_reply.started":"2025-06-27T03:31:05.426709Z","shell.execute_reply":"2025-06-27T03:31:05.477478Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Dark calibration  (On FGS1_calibration)\n\nFGS1_calibration likely refers to the calibration process for the **Fine Guidance Sensor 1 (FGS1)** on the Hubble Space Telescope (HST)","metadata":{}},{"cell_type":"code","source":"dar = pd.read_parquet(\"/kaggle/input/ariel-data-challenge-2025/train/1024292144/FGS1_calibration_0/dark.parquet\")\ndar.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:33:42.998688Z","iopub.execute_input":"2025-06-27T03:33:42.999011Z","iopub.status.idle":"2025-06-27T03:33:43.030383Z","shell.execute_reply.started":"2025-06-27T03:33:42.99899Z","shell.execute_reply":"2025-06-27T03:33:43.029713Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Matshow\n\nClueless about these parquet - Thak You Cyberia for matshow. Thanks to Cyberia I learned matshow.","metadata":{}},{"cell_type":"code","source":"#Cyberia https://www.kaggle.com/code/cyberia/eda-adc2024/notebook\n\ndark = dar.values.reshape(32, 32)  # Reshape to 2D array 32 columns\nplt.matshow(dark)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T04:03:26.980797Z","iopub.execute_input":"2025-06-27T04:03:26.981107Z","iopub.status.idle":"2025-06-27T04:03:27.340892Z","shell.execute_reply.started":"2025-06-27T04:03:26.981086Z","shell.execute_reply":"2025-06-27T04:03:27.340143Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Flat calibration frame\n\nOn AIRS-CH0_calibration - **Atmospheric Infrared Sounder (AIRS)**","metadata":{}},{"cell_type":"code","source":"flat = pd.read_parquet(\"/kaggle/input/ariel-data-challenge-2025/train/1029552010/AIRS-CH0_calibration_0/flat.parquet\")\nflat.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:52:25.949624Z","iopub.execute_input":"2025-06-27T03:52:25.949983Z","iopub.status.idle":"2025-06-27T03:52:26.000716Z","shell.execute_reply.started":"2025-06-27T03:52:25.949959Z","shell.execute_reply":"2025-06-27T03:52:25.999832Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Flat didn't change much since last time. Or maybe, nothing changed.","metadata":{}},{"cell_type":"code","source":"fla = flat.values  \nplt.matshow(fla)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T04:22:18.056179Z","iopub.execute_input":"2025-06-27T04:22:18.056795Z","iopub.status.idle":"2025-06-27T04:22:18.276555Z","shell.execute_reply.started":"2025-06-27T04:22:18.05677Z","shell.execute_reply":"2025-06-27T04:22:18.275736Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Linear_corr calibration frame","metadata":{}},{"cell_type":"code","source":"linCorr = pd.read_parquet(\"/kaggle/input/ariel-data-challenge-2025/train/1031303815/FGS1_calibration_0/linear_corr.parquet\")\nlinCorr.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:56:53.803448Z","iopub.execute_input":"2025-06-27T03:56:53.804188Z","iopub.status.idle":"2025-06-27T03:56:53.840256Z","shell.execute_reply.started":"2025-06-27T03:56:53.804159Z","shell.execute_reply":"2025-06-27T03:56:53.839605Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Linear_corr (Matshow)","metadata":{}},{"cell_type":"code","source":"linear = linCorr.values  \nplt.matshow(linear)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T04:05:45.013156Z","iopub.execute_input":"2025-06-27T04:05:45.013928Z","iopub.status.idle":"2025-06-27T04:05:45.327509Z","shell.execute_reply.started":"2025-06-27T04:05:45.013904Z","shell.execute_reply":"2025-06-27T04:05:45.326766Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Read Calibration frame","metadata":{}},{"cell_type":"code","source":"read = pd.read_parquet(\"/kaggle/input/ariel-data-challenge-2025/train/104891231/AIRS-CH0_calibration_0/read.parquet\")\nread.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:58:51.935548Z","iopub.execute_input":"2025-06-27T03:58:51.936193Z","iopub.status.idle":"2025-06-27T03:58:51.983841Z","shell.execute_reply.started":"2025-06-27T03:58:51.936168Z","shell.execute_reply":"2025-06-27T03:58:51.982895Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Read parquet file sample\n\nQuite the same of my last Kaggle Notebook on ADC 2024","metadata":{}},{"cell_type":"code","source":"rea = read.values  \nplt.matshow(rea)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T04:07:37.325218Z","iopub.execute_input":"2025-06-27T04:07:37.326227Z","iopub.status.idle":"2025-06-27T04:07:37.576417Z","shell.execute_reply.started":"2025-06-27T04:07:37.326187Z","shell.execute_reply":"2025-06-27T04:07:37.575475Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"air = pd.read_parquet(\"/kaggle/input/ariel-data-challenge-2025/train/1029552010/FGS1_signal_0.parquet\")\nair.tail(3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T04:16:04.057246Z","iopub.execute_input":"2025-06-27T04:16:04.057594Z","iopub.status.idle":"2025-06-27T04:16:06.252062Z","shell.execute_reply.started":"2025-06-27T04:16:04.057542Z","shell.execute_reply":"2025-06-27T04:16:06.25129Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## FGS1_signal_0","metadata":{}},{"cell_type":"code","source":"AIRS = air.values  \nplt.matshow(AIRS)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T04:16:16.346374Z","iopub.execute_input":"2025-06-27T04:16:16.347099Z","iopub.status.idle":"2025-06-27T04:16:19.392893Z","shell.execute_reply.started":"2025-06-27T04:16:16.347071Z","shell.execute_reply":"2025-06-27T04:16:19.392081Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## submission file","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv('/kaggle/input/ariel-data-challenge-2025/sample_submission.csv')\nsubmission.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T02:57:45.342312Z","iopub.execute_input":"2025-06-27T02:57:45.34318Z","iopub.status.idle":"2025-06-27T02:57:45.402416Z","shell.execute_reply.started":"2025-06-27T02:57:45.34315Z","shell.execute_reply":"2025-06-27T02:57:45.401689Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Ariel Mission\n\n\"Ariel, the European Space Agency’s next-generation mission to observe the chemical makeup of distant extrasolar planets, underwent tests designed to simulate the intense acoustic environment of rocket launch at the National Satellite Test Facility (NSTF).\"  \n\n\"The NSTF’s Dynamics Suite is home to 48 large speakers and amplifiers which can be configured to simulate the incredibly loud acoustic environment typical of a rocket launch. This testing, known as direct field acoustic noise (DFAN) testing, is vital to ensure the structural integrity of space components.\"\n\n\"The NSTF team has successfully completed its first DFAN tests conducted on components for the Ariel space telescope, marking a significant milestone for the facility and the UK space sector.\"\n\nhttps://arielmission.space/","metadata":{}},{"cell_type":"markdown","source":"## wavelengths.csv \n\nThe wavelength grid for each ground truth spectrum in the dataset.","metadata":{}},{"cell_type":"code","source":"wave = pd.read_csv('/kaggle/input/ariel-data-challenge-2025/wavelengths.csv', delimiter=',', encoding='utf-8')\n#pd.set_option('display.max_columns', None)\nwave.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:12:16.41321Z","iopub.execute_input":"2025-06-27T03:12:16.413506Z","iopub.status.idle":"2025-06-27T03:12:16.447682Z","shell.execute_reply.started":"2025-06-27T03:12:16.413483Z","shell.execute_reply":"2025-06-27T03:12:16.446872Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## adc_info.csv \n\nAnalog-to-digital (ADC) conversion parameters (gain and offset) for restoring the original dynamic range of the data. Unlike last year's challenge, all planets use the same adc_info.","metadata":{}},{"cell_type":"code","source":"adc = pd.read_csv('/kaggle/input/ariel-data-challenge-2025/adc_info.csv', delimiter=',', encoding='utf-8')\n#pd.set_option('display.max_columns', None)\nadc.tail()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T03:14:02.237159Z","iopub.execute_input":"2025-06-27T03:14:02.237442Z","iopub.status.idle":"2025-06-27T03:14:02.258008Z","shell.execute_reply.started":"2025-06-27T03:14:02.23742Z","shell.execute_reply":"2025-06-27T03:14:02.257157Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## We have much more data this year. Much more : (\n\n\"Relative to last year's challenge this iteration:\"\n\n\"Contains more train and test data. Uses unique star models for each planet. Has repeated observations of some planets. Incorporates upgrades to the physics model.\"","metadata":{}},{"cell_type":"code","source":"#By Ambros https://www.kaggle.com/code/ambrosm/adc24-intro-training/notebook\n\nplanet_id = 1124057774\nf_signal = pd.read_parquet(f'/kaggle/input/ariel-data-challenge-2025/train/{planet_id}/FGS1_signal_0.parquet')\nf_signal","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T04:34:39.770772Z","iopub.execute_input":"2025-06-27T04:34:39.77148Z","iopub.status.idle":"2025-06-27T04:34:42.676792Z","shell.execute_reply.started":"2025-06-27T04:34:39.771452Z","shell.execute_reply":"2025-06-27T04:34:42.675917Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"#By Ambros https://www.kaggle.com/code/ambrosm/adc24-intro-training/notebook\n\n_, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))\nsns.heatmap(f_signal.iloc[0].values.reshape(32, 32), ax=ax1, vmin=0, vmax=52000)\nax1.set_aspect('equal')\nsns.heatmap(f_signal.iloc[1].values.reshape(32, 32), ax=ax2, vmin=0, vmax=52000)\nax2.set_aspect('equal')\nplt.suptitle('A pair of FGS1 images')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-27T04:35:53.483713Z","iopub.execute_input":"2025-06-27T04:35:53.484221Z","iopub.status.idle":"2025-06-27T04:35:54.113897Z","shell.execute_reply.started":"2025-06-27T04:35:53.484186Z","shell.execute_reply":"2025-06-27T04:35:54.112817Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Reduction of the signal intensity\n\n\"To see the time series, we first have to compute the difference between the even and the odd frames to get the net signal (67500 time steps). Then take the mean over all 1024 pixels. The net signal is very noisy, and we smoothen it by computing a moving average. The plot of the smoothened signal clearly shows that the **signal intensity is reduced** (i.e., the image gets darker) while the planet passes in front of the star (between time steps 23500 and 44000).\n\nThe left diagram shows a planet with a **strong reduction of the signal intensity**, the right diagram shows a planet with a **weak reduction**:\"\n\n**By Ambros** https://www.kaggle.com/code/ambrosm/adc24-intro-training/notebook\n\nI could only understand \"reduction of signal intensity\". About the computation, I'm still trying to digest. Better ask Ambros the author. I had only intended to plot the \"pair of FGS1 images\".\n\nThat's all. **Draft Session 1h:52m**","metadata":{}},{"cell_type":"markdown","source":"## Acknowledgements:\n\nCyberia https://www.kaggle.com/code/cyberia/eda-adc2024/notebook\n\nAmbros https://www.kaggle.com/code/ambrosm/adc24-intro-training/notebook","metadata":{}}]}