The first and second placed teams on the leaderboard appear to be experts in atmospheric science and machine learning.
\nFrom my experience with experiments, I’ve learned that domain knowledge (and EDA) can be crucial. @phalanx also emphasized this point in this topic, and I completely agree.
\nTo achieve higher scores, let’s share materials and knowledge relevant to this competition.
\nIf you have any useful information, please share it in the comments.
\nI’ll start by sharing some videos, code, and papers. Some of these have helped me understand domain knowledge and improve my model, while others I haven’t fully explored yet.
\nPapers using Neural Networks:
\nThe first and second placed teams on the leaderboard appear to be experts in atmospheric science and machine learning.
\nFrom my experience with experiments, I’ve learned that domain knowledge (and EDA) can be crucial. @phalanx also emphasized this point in this topic, and I completely agree.
\nTo achieve higher scores, let’s share materials and knowledge relevant to this competition.
\nIf you have any useful information, please share it in the comments.
\nI’ll start by sharing some videos, code, and papers. Some of these have helped me understand domain knowledge and improve my model, while others I haven’t fully explored yet.
\nPapers using Neural Networks:
\nI'm doubtful, TBH, how much domain knowledge is helpful in this comp.
\nNo, to be exact, I'm waiting to see even one thing, which is 'domain knowledge,' which is helpful in this competition.
\n1st and 2nd places might be possible to achieve by training for enough epochs and good finetuning on all data (i.e., all 70M samples that exist in HF).
Thanks. That's an interesting point. However, considering that many Kagglers are significantly outscored by the first and second places, I think there might be reasons beyond just the amount of data.
\nAt the very least, there might not be any domain knowledge causing a major score boost, but I have gained ideas that are useful for improving my model.
This book is good for me (sorry that this is a Japanese book, but similar books should be published in many countries.)
\n地球規模の気象学 大気の大循環から理解する新しい気象学
I’ve had trouble finding how domain knowledge can be used in this case. I come from a fluid dynamics background, with limited experience in ML. There is not enough information to enforce conservation, unless you use the full Climsim dataset. The only thing useful is enforcing positive values of the output scalars (as publicly stated in the climsim paper). This is my first Kaggle competition and I thought some domain knowledge would help me but it is not easy, but I’ve learned a lot. I am currently stuck at 0.56 score.
", "rawMarkdown": "I’ve had trouble finding how domain knowledge can be used in this case. I come from a fluid dynamics background, with limited experience in ML. There is not enough information to enforce conservation, unless you use the full Climsim dataset. The only thing useful is enforcing positive values of the output scalars (as publicly stated in the climsim paper). This is my first Kaggle competition and I thought some domain knowledge would help me but it is not easy, but I’ve learned a lot. I am currently stuck at 0.56 score.", "votes": 1, "replies": [ { "id": 2856146, "postDate": "2024-06-05T07:12:56.883Z", "content": "I hadn't seen the amount of work shared here… there is a lot to digest! but thanks for sharing!
", "rawMarkdown": "I hadn't seen the amount of work shared here... there is a lot to digest! but thanks for sharing!" } ] }, { "id": 2844971, "postDate": "2024-05-30T10:13:37.423Z", "content": "[Paper] Identifying Relations Between Deep Convection and the Large-\nScale Atmosphere Using Explainable Artificial Intelligence
\n[Video] Introduction to the Simple Cloud-Resolving E3SM Atmosphere Model
I haven't looked too deeply into it as I'm still working on my model arch, but I'm wondering if a major objective of this comp is to model deep or shallow convection types (or both). According to the video, deep convection seems to be the most problematic/error prone part of CRMs.
\nThanks for sharing!
", "rawMarkdown": "[Paper] [Identifying Relations Between Deep Convection and the Large-\nScale Atmosphere Using Explainable Artificial Intelligence](https://singh.sci.monash.edu/papers/Retsch_2022_relations.pdf)\n[Video] [Introduction to the Simple Cloud-Resolving E3SM Atmosphere Model](https://www.youtube.com/watch?v=jcMqYdWyfcg)\n\nI haven't looked too deeply into it as I'm still working on my model arch, but I'm wondering if a major objective of this comp is to model deep or shallow convection types (or both). According to the video, deep convection seems to be the most problematic/error prone part of CRMs.\n\nThanks for sharing!", "votes": 1 }, { "id": 2846827, "postDate": "2024-05-31T09:27:22.653Z", "content": "Did you get improvement from domain knowledge?
", "rawMarkdown": "Did you get improvement from domain knowledge?", "replies": [ { "id": 2848437, "postDate": "2024-06-01T03:14:20.650Z", "content": "Yes, I’ve gained some insights from domain knowledge, which have been helpful for improvements. While each individual improvement might not have a huge impact, parts of my model architecture are based on this domain knowledge.
\nIt would have taken more time to achieve the same score if I had modeled without considering domain knowledge.
", "rawMarkdown": "Yes, I’ve gained some insights from domain knowledge, which have been helpful for improvements. While each individual improvement might not have a huge impact, parts of my model architecture are based on this domain knowledge.\n\nIt would have taken more time to achieve the same score if I had modeled without considering domain knowledge.", "votes": 4 } ] }, { "id": 2846144, "postDate": "2024-05-31T00:17:50.143Z", "content": "I don't think there is sufficient data provided to apply any conservation laws, geospatial information or time dependent information. All of these would be useful in a predictive sense, but are computationally expensive.
\nThat being said, this is my first experience with ML. I do think that training multiple neural networks is an interesting idea (specifically that posed in the Yuval paper) considering that the objective variables likely have some dependency on one another. Thanks for posting this! I think its very good for generating new ideas.
", "rawMarkdown": "I don't think there is sufficient data provided to apply any conservation laws, geospatial information or time dependent information. All of these would be useful in a predictive sense, but are computationally expensive. \n\nThat being said, this is my first experience with ML. I do think that training multiple neural networks is an interesting idea (specifically that posed in the Yuval paper) considering that the objective variables likely have some dependency on one another. Thanks for posting this! I think its very good for generating new ideas. " }, { "id": 2849297, "postDate": "2024-06-01T14:07:03.217Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2846123, "postDate": "2024-05-30T23:10:30.237Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2845151, "author_name": "greySnow", "author_url": "", "post_date": "2024-05-30T12:20:57.173000", "content": "I'm doubtful, TBH, how much domain knowledge is helpful in this comp.
\nNo, to be exact, I'm waiting to see even one thing, which is 'domain knowledge,' which is helpful in this competition.
\n1st and 2nd places might be possible to achieve by training for enough epochs and good finetuning on all data (i.e., all 70M samples that exist in HF).
Thanks. That's an interesting point. However, considering that many Kagglers are significantly outscored by the first and second places, I think there might be reasons beyond just the amount of data.
\nAt the very least, there might not be any domain knowledge causing a major score boost, but I have gained ideas that are useful for improving my model.
This book is good for me (sorry that this is a Japanese book, but similar books should be published in many countries.)
\n地球規模の気象学 大気の大循環から理解する新しい気象学
I’ve had trouble finding how domain knowledge can be used in this case. I come from a fluid dynamics background, with limited experience in ML. There is not enough information to enforce conservation, unless you use the full Climsim dataset. The only thing useful is enforcing positive values of the output scalars (as publicly stated in the climsim paper). This is my first Kaggle competition and I thought some domain knowledge would help me but it is not easy, but I’ve learned a lot. I am currently stuck at 0.56 score.
", "votes": 1, "replies": [ { "id": 2856146, "author_name": "Juan D C F", "author_url": "", "post_date": "2024-06-05T07:12:56.883000", "content": "I hadn't seen the amount of work shared here… there is a lot to digest! but thanks for sharing!
", "votes": 0, "replies": [] } ] }, { "id": 2844971, "author_name": "sroger", "author_url": "", "post_date": "2024-05-30T10:13:37.423000", "content": "[Paper] Identifying Relations Between Deep Convection and the Large-\nScale Atmosphere Using Explainable Artificial Intelligence
\n[Video] Introduction to the Simple Cloud-Resolving E3SM Atmosphere Model
I haven't looked too deeply into it as I'm still working on my model arch, but I'm wondering if a major objective of this comp is to model deep or shallow convection types (or both). According to the video, deep convection seems to be the most problematic/error prone part of CRMs.
\nThanks for sharing!
", "votes": 1, "replies": [] }, { "id": 2846827, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2024-05-31T09:27:22.653000", "content": "Did you get improvement from domain knowledge?
", "votes": 0, "replies": [ { "id": 2848437, "author_name": "kami", "author_url": "", "post_date": "2024-06-01T03:14:20.650000", "content": "Yes, I’ve gained some insights from domain knowledge, which have been helpful for improvements. While each individual improvement might not have a huge impact, parts of my model architecture are based on this domain knowledge.
\nIt would have taken more time to achieve the same score if I had modeled without considering domain knowledge.
", "votes": 4, "replies": [] } ] }, { "id": 2846144, "author_name": "Anthony Meza", "author_url": "", "post_date": "2024-05-31T00:17:50.143000", "content": "I don't think there is sufficient data provided to apply any conservation laws, geospatial information or time dependent information. All of these would be useful in a predictive sense, but are computationally expensive.
\nThat being said, this is my first experience with ML. I do think that training multiple neural networks is an interesting idea (specifically that posed in the Yuval paper) considering that the objective variables likely have some dependency on one another. Thanks for posting this! I think its very good for generating new ideas.
", "votes": 0, "replies": [] }, { "id": 2849297, "author_name": "", "author_url": "", "post_date": "2024-06-01T14:07:03.217000", "content": "", "votes": 0, "replies": [] }, { "id": 2846123, "author_name": "", "author_url": "", "post_date": "2024-05-30T23:10:30.237000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2844924": "## Domain Knowledge Might Be Important\n\nThe first and second placed teams on the leaderboard appear to be experts in atmospheric science and machine learning.\n\nFrom my experience with experiments, I’ve learned that domain knowledge (and EDA) can be crucial. @phalanx also emphasized this point in [this topic](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506984), and I completely agree.\n\nTo achieve higher scores, let’s share materials and knowledge relevant to this competition.\n\n**If you have any useful information, please share it in the comments.**\n\n## Materials\n\nI’ll start by sharing some videos, code, and papers. Some of these have helped me understand domain knowledge and improve my model, while others I haven’t fully explored yet.\n\n- [paper] [ClimSim: A large multi-scale dataset for hybrid physics-ML climate emulation](https://arxiv.org/abs/2306.08754)\n - This paper is about the competition.\n- [code] https://github.com/leap-stc/ClimSim/tree/main\n - This is code related to the competition.\n- [video] https://www.youtube.com/@LEAP_STC\n - This is the YouTube channel of the competition host, LEAP.\n- [code] https://github.com/E3SM-Project/E3SM\n - This is my first time reading Fortran code!\n\nPapers using Neural Networks:\n\n- [A Moist Physics Parameterization Based on Deep Learning](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2020MS002076)\n- [Enforcing Analytic Constraints in Neural Networks Emulating Physical Systems](https://arxiv.org/abs/1909.00912)\n- [Causally-Informed Deep Learning to Improve Climate Models and Projections](https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2023JD039202)\n- [Stable Climate Simulations Using a Realistic General Circulation Model with Neural Network Parameterizations for Atmospheric Moist Physics and Radiation Processes](https://gmd.copernicus.org/articles/15/3923/2022/)\n- [Assessing the Potential of Deep Learning for Emulating Cloud Superparameterization in Climate Models With Real-Geography Boundary Conditions](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2020MS002385)\n- [Use of Neural Networks for Stable, Accurate, and Physically Consistent Parameterization of Subgrid Atmospheric Processes With Good Performance at Reduced Precision](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2020GL091363)\n- [Climate-Invariant Machine Learning](https://www.science.org/doi/10.1126/sciadv.adj7250)\n- [Could Machine Learning Break the Convection Parameterization Deadlock?](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2018GL078202)\n- [Correcting Coarse-Grid Weather and Climate Models by Machine Learning from Global Storm-Resolving Simulations](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2021MS002794)\n- [Machine-Learned Climate Model Corrections from a Global Storm-Resolving Model: Performance Across the Annual Cycle](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2022MS003400)\n\n", "2845151": "I'm doubtful, TBH, how much domain knowledge is helpful in this comp.\nNo, to be exact, I'm waiting to see even *one* thing, which is 'domain knowledge,' which is helpful in this competition.\n1st and 2nd places might be possible to achieve by training for enough epochs and good finetuning on all data (i.e., all 70M samples that exist in HF). ", "2886305": "This book is good for me (sorry that this is a Japanese book, but similar books should be published in many countries.)\n[地球規模の気象学 大気の大循環から理解する新しい気象学](https://www.amazon.co.jp/%E5%9C%B0%E7%90%83%E8%A6%8F%E6%A8%A1%E3%81%AE%E6%B0%97%E8%B1%A1%E5%AD%A6-%E5%A4%A7%E6%B0%97%E3%81%AE%E5%A4%A7%E5%BE%AA%E7%92%B0%E3%81%8B%E3%82%89%E7%90%86%E8%A7%A3%E3%81%99%E3%82%8B%E6%96%B0%E3%81%97%E3%81%84%E6%B0%97%E8%B1%A1%E5%AD%A6-%E3%83%96%E3%83%AB%E3%83%BC%E3%83%90%E3%83%83%E3%82%AF%E3%82%B9-%E4%BF%9D%E5%9D%82%E7%9B%B4%E7%B4%80-ebook/dp/B0CMQ7W37W/ref=sr_1_27?crid=3AJECTG52XQS2&dib=eyJ2IjoiMSJ9.--Zrqsq_dSCdbO6-iYXfw11LgEf2pacUwatJ-s1Xh_uEwRFcqNjeU2FY6Gw8-IJsFOoCv1AHiKnePxMkpmS6OIfz-5YTfutkl8_xJUETl3N-hTy9AOd2xZAi0W1x5Y6qprY7Jo8VyRY5VFf6RcNp4EvHOsteHPyLSoTJMMYGhc-v0-pNcAOGRTnAmCPCZRbA-u5CR38vhUU2FPKNjERffpUKtMldRjJoux3kzE6M3N_tEZpr-YHJkhCK8eUeTlK3LGXKe5FPQbPS2d17Mwn8V_8z4zg7rr3R-i02uZk9UTQ.YLhCjzq0ziJOBxdlZCZYJpBuRpCetzdNdhmfpJamA7g&dib_tag=se&keywords=%E6%B0%97%E8%B1%A1%E5%AD%A6&qid=1719155143&sprefix=%E6%B0%97%E8%B1%A1%E5%AD%A6%2Caps%2C166&sr=8-27)", "2852256": "I’ve had trouble finding how domain knowledge can be used in this case. I come from a fluid dynamics background, with limited experience in ML. There is not enough information to enforce conservation, unless you use the full Climsim dataset. The only thing useful is enforcing positive values of the output scalars (as publicly stated in the climsim paper). This is my first Kaggle competition and I thought some domain knowledge would help me but it is not easy, but I’ve learned a lot. I am currently stuck at 0.56 score.", "2844971": "[Paper] [Identifying Relations Between Deep Convection and the Large-\nScale Atmosphere Using Explainable Artificial Intelligence](https://singh.sci.monash.edu/papers/Retsch_2022_relations.pdf)\n[Video] [Introduction to the Simple Cloud-Resolving E3SM Atmosphere Model](https://www.youtube.com/watch?v=jcMqYdWyfcg)\n\nI haven't looked too deeply into it as I'm still working on my model arch, but I'm wondering if a major objective of this comp is to model deep or shallow convection types (or both). According to the video, deep convection seems to be the most problematic/error prone part of CRMs.\n\nThanks for sharing!", "2846827": "Did you get improvement from domain knowledge?", "2846144": "I don't think there is sufficient data provided to apply any conservation laws, geospatial information or time dependent information. All of these would be useful in a predictive sense, but are computationally expensive. \n\nThat being said, this is my first experience with ML. I do think that training multiple neural networks is an interesting idea (specifically that posed in the Yuval paper) considering that the objective variables likely have some dependency on one another. Thanks for posting this! I think its very good for generating new ideas. ", "2849297": "", "2846123": "" } }