![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5487737%2Fd89f52ecf20de68b91f838ebe7d18563%2Fshrek-shrek-rizz.gif?generation=1718777749835349&alt=media\")
Hi Kagglers,
\nWe have some good news and bad news. The good news is that we are extending the competition by 2 weeks! The new deadline is July 15, 2024 11:59PM UTC. We are incredibly impressed with your work so far and the engaging discussion that has been going on between competing teams. It’s truly awe-inspiring.
\nNow for the bad news: we are releasing a new test set and sample submission (weighting). As Kaggle user @tatamikenn pointed out, it is possible to reverse-engineer location information from the original test set. Many users (very fairly) entered this competition with the assumption that it would be a column to column regression problem, not a multi-column to column regression problem. Because we don’t want to give an unfair advantage to users who discovered this exploit early, we are making the tough decision to release a fresh test set + weighting + solution. For those that have invested a great deal of time in the multi-column approach, you must have realized that the sample IDs were pre-scrambled and of no use. Knowing this, it is fairly obvious that the intention of the competition designers was to not grant you access to this information. As Kaggle user @ryches puts it, using this information indeed goes against the spirit of the competition.
\nThis unfortunately means all previous submissions will be invalidated, and you will need to submit new predictions using the new test set and weighting. We sincerely apologize for the changes and we are very grateful to @tatamikenn for pointing this out. Thank you so much for your continued engagement and happy Kaggling!
\nEDIT 1: Thanks to @ymatioun for originally pointing the location leak out.
\nEDIT 2: As @churkinnikita points out, ptend_q0002 12-14 are no longer zeroed out.
For those who don't want to download everything, you can use -f option
\nkaggle competitions download -c leap-atmospheric-physics-ai-climsim -f sample_submission.csv\nkaggle competitions download -c leap-atmospheric-physics-ai-climsim -f test.csv\nThank you very much!
", "rawMarkdown": "Thank you very much!", "votes": 1 } ] }, { "id": 2878453, "postDate": "2024-06-19T00:30:57.173Z", "content": "Hi Kagglers,
\nWe have some good news and bad news. The good news is that we are extending the competition by 2 weeks! The new deadline is July 15, 2024 11:59PM UTC. We are incredibly impressed with your work so far and the engaging discussion that has been going on between competing teams. It’s truly awe-inspiring.
\nNow for the bad news: we are releasing a new test set and sample submission (weighting). As Kaggle user @tatamikenn pointed out, it is possible to reverse-engineer location information from the original test set. Many users (very fairly) entered this competition with the assumption that it would be a column to column regression problem, not a multi-column to column regression problem. Because we don’t want to give an unfair advantage to users who discovered this exploit early, we are making the tough decision to release a fresh test set + weighting + solution. For those that have invested a great deal of time in the multi-column approach, you must have realized that the sample IDs were pre-scrambled and of no use. Knowing this, it is fairly obvious that the intention of the competition designers was to not grant you access to this information. As Kaggle user @ryches puts it, using this information indeed goes against the spirit of the competition.
\nThis unfortunately means all previous submissions will be invalidated, and you will need to submit new predictions using the new test set and weighting. We sincerely apologize for the changes and we are very grateful to @tatamikenn for pointing this out. Thank you so much for your continued engagement and happy Kaggling!
\nEDIT 1: Thanks to @ymatioun for originally pointing the location leak out.
\nEDIT 2: As @churkinnikita points out, ptend_q0002 12-14 are no longer zeroed out.
When I check the new sample_submission.csv, the weight of 12-14 in ptend_q0002 is 1 instead of 0.
\nIs this correct?
I want to mention that the original person to reveal that there was a leak in test was @ymatioun so kudos to him too.
", "rawMarkdown": "I want to mention that the original person to reveal that [there was a leak](https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/511274#2868420) in test was @ymatioun so kudos to him too.", "votes": 20 }, { "id": 2878746, "postDate": "2024-06-19T06:16:07.517Z", "content": "\n\nWe have some good news and bad news. The good news is that we are extending the competition by 2 weeks!
\n
@jerrylin96 The weights of ptend_q0002 12–14 are now ones instead of zeros. Is this intended? This introduces a discrepancy between pre-update scoring and current scoring. Now we have three more \"extreme\" targets to predict.
", "rawMarkdown": "@jerrylin96 The weights of ptend_q0002 12–14 are now ones instead of zeros. Is this intended? This introduces a discrepancy between pre-update scoring and current scoring. Now we have three more \"extreme\" targets to predict.", "votes": 8, "replies": [ { "id": 2878959, "postDate": "2024-06-19T09:22:12.443Z", "content": "They are overwritten from -state_q0002 i / 1200 so it doesn't change anything.
", "rawMarkdown": "They are overwritten from -state_q0002 i / 1200 so it doesn't change anything. ", "votes": 3 }, { "id": 2879388, "postDate": "2024-06-19T14:28:41.747Z", "content": "This is correct. Apologies for not mentioning this earlier. I will update the post accordingly.
", "rawMarkdown": "This is correct. Apologies for not mentioning this earlier. I will update the post accordingly.", "votes": 3, "replies": [ { "id": 2879400, "postDate": "2024-06-19T14:46:33.460Z", "content": "@jerrylin96 In the context of Gunes' comment above, can you confirm whether you now want us to predict ptend_q0002_12-14? I'm a little confused, as I'm not sure what this update has to do with preventing the multi-column approach. Can you explain why this change has been made?
", "rawMarkdown": "@jerrylin96 In the context of Gunes' comment above, can you confirm whether you now want us to predict ptend_q0002_12-14? I'm a little confused, as I'm not sure what this update has to do with preventing the multi-column approach. Can you explain why this change has been made?", "votes": 2 }, { "id": 2879564, "postDate": "2024-06-19T17:02:55.933Z", "content": "You're correct, that part of the update does not have anything to do with preventing the multi-column approach. This was an unintentional update spurred by addressing the other issue quickly.
\nWhile it is unlikely we will reverse course on this, we can revisit this issue if the Kaggle community feels sufficiently strongly about zero'ing out those values again.
", "rawMarkdown": "You're correct, that part of the update does not have anything to do with preventing the multi-column approach. This was an unintentional update spurred by addressing the other issue quickly.\n\nWhile it is unlikely we will reverse course on this, we can revisit this issue if the Kaggle community feels sufficiently strongly about zero'ing out those values again." } ] } ] }, { "id": 2878523, "postDate": "2024-06-19T03:15:23.287Z", "content": "@jerrylin96 Thanks for the update!
\nJust to be clear.
\nWhat if someone predicts location, reconstructs map and aggregates location specific Information?(like lat, lon and more sophisticated)
\nIt's still possible even after this shuffling as top team says they can predict location with very high accuracy.
I think models are already capturing that information from the current features. Explicitly feeding the location via an embedding might be giving the edge.
", "rawMarkdown": "I think models are already capturing that information from the current features. Explicitly feeding the location via an embedding might be giving the edge.", "votes": 2 }, { "id": 2878779, "postDate": "2024-06-19T06:42:31.990Z", "content": "Even if you can predict with 100% accuracy latlon you will not be able to use it for multiple-columns-to-one model since the test set is also shuffled in time. And good luck predicting also time stamp with sufficient accuracy.
", "rawMarkdown": "Even if you can predict with 100% accuracy latlon you will not be able to use it for multiple-columns-to-one model since the test set is also shuffled in time. And good luck predicting also time stamp with sufficient accuracy.", "votes": 2 }, { "id": 2879029, "postDate": "2024-06-19T10:27:08.933Z", "content": "I think explicitly aggregating location information will be more advantageous, rather than model feeding it, but will it be allowed ? I think its a very important question to be addressed
", "rawMarkdown": "I think explicitly aggregating location information will be more advantageous, rather than model feeding it, but will it be allowed ? I think its a very important question to be addressed", "votes": 3, "replies": [ { "id": 2890460, "postDate": "2024-06-26T06:04:53.707Z", "content": "I have the same question, is it allowed to restore the lat/lon information of the grid? @jerrylin96
", "rawMarkdown": "I have the same question, is it allowed to restore the lat/lon information of the grid? @jerrylin96 " } ] } ] }, { "id": 2878487, "postDate": "2024-06-19T01:26:37.927Z", "content": "@jerrylin96 Can you confirm that train.csv has not changed? Because I remember that the size of this file was about 180G before, no more than 200G, but now it is 359.31G.
", "rawMarkdown": "@jerrylin96 Can you confirm that train.csv has not changed? Because I remember that the size of this file was about 180G before, no more than 200G, but now it is 359.31G.", "votes": 4, "replies": [ { "id": 2878511, "postDate": "2024-06-19T02:34:17.967Z", "content": "i just downloaded the train data, and it's about 169G
", "rawMarkdown": "i just downloaded the train data, and it's about 169G", "votes": 1, "replies": [ { "id": 2878520, "postDate": "2024-06-19T03:08:36.407Z", "content": "Thanks, maybe it's just a display issue.
", "rawMarkdown": "Thanks, maybe it's just a display issue." } ] } ] }, { "id": 2886907, "postDate": "2024-06-23T22:17:34.320Z", "content": "@jerrylin96 Can I ask the score difference of the benchmark model between new Public test and new Private test?
\nAre they close?
c.f.) In your previous comment, it had 0.005 score difference between old public test and old private test.
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2834423
sample_submission.csv weight file is 0 or 1, The difference between the new weights and the previous ones is quite large, with a CV of over 0.994. This needs to be confirmed - should we indeed adopt the new weights?
", "rawMarkdown": "sample_submission.csv weight file is 0 or 1, The difference between the new weights and the previous ones is quite large, with a CV of over 0.994. This needs to be confirmed - should we indeed adopt the new weights?", "votes": 2, "replies": [ { "id": 2878490, "postDate": "2024-06-19T01:35:14.710Z", "content": "Yes, use the new weights.
", "rawMarkdown": "Yes, use the new weights." }, { "id": 2879001, "postDate": "2024-06-19T10:13:01.857Z", "content": "You're probably using the wrong calculation formula.
\nThe discussion below may be helpful.
https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/495255
", "rawMarkdown": "You're probably using the wrong calculation formula.\nThe discussion below may be helpful.\n\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/495255", "votes": 3 }, { "id": 2880062, "postDate": "2024-06-20T02:45:55.670Z", "content": "I've also encountered this problem. How did you solve it? I changed the calculation method, but it still seems to be 0.994. Maybe there's something wrong with my calculation?
", "rawMarkdown": "I've also encountered this problem. How did you solve it? I changed the calculation method, but it still seems to be 0.994. Maybe there's something wrong with my calculation?", "votes": 2, "replies": [ { "id": 2880927, "postDate": "2024-06-20T13:12:04.917Z", "content": "Use sklearn not pytorch implements
", "rawMarkdown": "Use sklearn not pytorch implements " }, { "id": 2880949, "postDate": "2024-06-20T13:23:43.607Z", "content": "I also use pytorch, what is wrong with pytorch? 😅
", "rawMarkdown": "I also use pytorch, what is wrong with pytorch? 😅" }, { "id": 2881016, "postDate": "2024-06-20T13:45:35.403Z", "content": "Thank you, I'll double-check
", "rawMarkdown": "Thank you, I'll double-check" }, { "id": 2881784, "postDate": "2024-06-21T01:02:23.243Z", "content": "I have checked. When torchmetrics calculates very small numbers, r2_score is 1.
", "rawMarkdown": "I have checked. When torchmetrics calculates very small numbers, r2_score is 1.", "votes": 1 } ] } ] }, { "id": 2884450, "postDate": "2024-06-22T14:08:25.930Z", "content": "Project System is fast but lagging
", "rawMarkdown": "Project System is fast but lagging\n" }, { "id": 2880096, "postDate": "2024-06-20T03:56:47.560Z", "content": "\nI'm new in this competition, and I have one question about r2 calculation.
\nIn the description page, the following statement can be seen:
\nThis weighting can also be calculated without downloading this file. The top 12 levels (0-11 inclusive) for 'ptend_q0001', 'ptend_q0002', 'ptend_q0003', 'ptend_u', and 'ptend_v' can be ignored.
Is the r2 calculated with skipping these columns in submission stage?
\nIn other words, columns with weight=0 are skipped in r2 calculation?
yes, pretty much…
", "rawMarkdown": "yes, pretty much..." }, { "id": 2880432, "postDate": "2024-06-20T07:32:37.540Z", "content": "No. It is 1 for these cols.
", "rawMarkdown": "No. It is 1 for these cols.", "votes": 3, "replies": [ { "id": 2881786, "postDate": "2024-06-21T01:08:29.960Z", "content": "I don't understand what you mean, I checked these columns and they are 0.
", "rawMarkdown": "I don't understand what you mean, I checked these columns and they are 0." }, { "id": 2881844, "postDate": "2024-06-21T02:52:32.243Z", "content": "R2 value should be 1 for those columns with weight 0
", "rawMarkdown": "R2 value should be 1 for those columns with weight 0", "votes": 1 }, { "id": 2882139, "postDate": "2024-06-21T07:19:46.847Z", "content": "Thanks everyone.
\nI have another question. For convenience, I split the case 1 and 2 depending on target columns.
1). About the top 12 levels (0-11 inclusive) for 'ptend_q0001', 'ptend_q0002', 'ptend_q0003', 'ptend_u', and 'ptend_v'
\nThe reason why I asked the above question is that if y_true is approximately equal to y_pred, then ss_tot will become tiny in r2 calculation, leading to large negative values in each R2_i where i denotes each class.
\nIf the host set R2_i=1 for these columns, then there is no afraid in these targets.
2). About the other target columns having tiny values
\nBut the situation is not resolved for other targets such as ptend_q0002_18, 19 etc which has tiny values.
\nSince R2 calculation is performed on class-wise and then averaged over all classes, very tiny values of ss_tot would be again appeared in some R2_i calculation. Though the problem is alleviated in our local calculation by using some tricks such as adding epsilon to ss_tot or standardization, these approach is not applicable to submission stage because of the difference of r2 calculation.
from these points, my another question is:
\nHow should we avoid this negative explosion in some targets?
p.s.
\nThere is a thread somewhere about different scoring methods of R2, i was always using the one that you first sum of ss_res then sum of ss_tot and then divide res/tot and whatever score i had in my validation sets i had also in my submissions (with the old weights, with the new weights i am currently still training but i am confident its going to be the same).
", "rawMarkdown": "There is a thread somewhere about different scoring methods of R2, i was always using the one that you first sum of ss_res then sum of ss_tot and then divide res/tot and whatever score i had in my validation sets i had also in my submissions (with the old weights, with the new weights i am currently still training but i am confident its going to be the same)." }, { "id": 2882807, "postDate": "2024-06-21T15:13:26.243Z", "content": "i think you were right. When i omit this code in inference i get very negative values, with this code i am at 0.71 ```
\nREPLACE_TO = ['ptend_q0002_0', 'ptend_q0002_1', 'ptend_q0002_2', 'ptend_q0002_3', 'ptend_q0002_4', 'ptend_q0002_5', 'ptend_q0002_6', 'ptend_q0002_7', 'ptend_q0002_8', 'ptend_q0002_9', 'ptend_q0002_10', 'ptend_q0002_11', 'ptend_q0002_12', 'ptend_q0002_13', 'ptend_q0002_14', 'ptend_q0002_15', 'ptend_q0002_16', 'ptend_q0002_17', 'ptend_q0002_18', 'ptend_q0002_19', 'ptend_q0002_20', 'ptend_q0002_21', 'ptend_q0002_22', 'ptend_q0002_23', 'ptend_q0002_24', 'ptend_q0002_25', 'ptend_q0002_26']
\nREPLACE_FROM = ['state_q0002_0', 'state_q0002_1', 'state_q0002_2', 'state_q0002_3', 'state_q0002_4', 'state_q0002_5', 'state_q0002_6', 'state_q0002_7', 'state_q0002_8', 'state_q0002_9', 'state_q0002_10', 'state_q0002_11', 'state_q0002_12', 'state_q0002_13', 'state_q0002_14', 'state_q0002_15', 'state_q0002_16', 'state_q0002_17', 'state_q0002_18', 'state_q0002_19', 'state_q0002_20', 'state_q0002_21', 'state_q0002_22', 'state_q0002_23', 'state_q0002_24', 'state_q0002_25', 'state_q0002_26']
df_test = pd.read_csv(test_file)
\nfor idx in range(0, 27):
\n df_p_test[f\"ptend_q0002_{idx}\"] = -df_test[f\"state_q0002_{idx}\"].to_numpy() / 1200
\n```
For those who don't want to download everything, you can use -f option
\nkaggle competitions download -c leap-atmospheric-physics-ai-climsim -f sample_submission.csv\nkaggle competitions download -c leap-atmospheric-physics-ai-climsim -f test.csv\nThank you very much!
", "votes": 1, "replies": [] } ] }, { "id": 2878528, "author_name": "kuto", "author_url": "", "post_date": "2024-06-19T03:23:51.943000", "content": "When I check the new sample_submission.csv, the weight of 12-14 in ptend_q0002 is 1 instead of 0.
\nIs this correct?
I want to mention that the original person to reveal that there was a leak in test was @ymatioun so kudos to him too.
", "votes": 20, "replies": [] }, { "id": 2878746, "author_name": "slime", "author_url": "", "post_date": "2024-06-19T06:16:07.517000", "content": "\n\nWe have some good news and bad news. The good news is that we are extending the competition by 2 weeks!
\n
@jerrylin96 The weights of ptend_q0002 12–14 are now ones instead of zeros. Is this intended? This introduces a discrepancy between pre-update scoring and current scoring. Now we have three more \"extreme\" targets to predict.
", "votes": 8, "replies": [ { "id": 2878959, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-06-19T09:22:12.443000", "content": "They are overwritten from -state_q0002 i / 1200 so it doesn't change anything.
", "votes": 3, "replies": [] }, { "id": 2879388, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-06-19T14:28:41.747000", "content": "This is correct. Apologies for not mentioning this earlier. I will update the post accordingly.
", "votes": 3, "replies": [ { "id": 2879400, "author_name": "Sarah Jeffreson", "author_url": "", "post_date": "2024-06-19T14:46:33.460000", "content": "@jerrylin96 In the context of Gunes' comment above, can you confirm whether you now want us to predict ptend_q0002_12-14? I'm a little confused, as I'm not sure what this update has to do with preventing the multi-column approach. Can you explain why this change has been made?
", "votes": 2, "replies": [] }, { "id": 2879564, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-06-19T17:02:55.933000", "content": "You're correct, that part of the update does not have anything to do with preventing the multi-column approach. This was an unintentional update spurred by addressing the other issue quickly.
\nWhile it is unlikely we will reverse course on this, we can revisit this issue if the Kaggle community feels sufficiently strongly about zero'ing out those values again.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2878523, "author_name": "Camaro", "author_url": "", "post_date": "2024-06-19T03:15:23.287000", "content": "@jerrylin96 Thanks for the update!
\nJust to be clear.
\nWhat if someone predicts location, reconstructs map and aggregates location specific Information?(like lat, lon and more sophisticated)
\nIt's still possible even after this shuffling as top team says they can predict location with very high accuracy.
I think models are already capturing that information from the current features. Explicitly feeding the location via an embedding might be giving the edge.
", "votes": 2, "replies": [] }, { "id": 2878779, "author_name": "greySnow", "author_url": "", "post_date": "2024-06-19T06:42:31.990000", "content": "Even if you can predict with 100% accuracy latlon you will not be able to use it for multiple-columns-to-one model since the test set is also shuffled in time. And good luck predicting also time stamp with sufficient accuracy.
", "votes": 2, "replies": [] }, { "id": 2879029, "author_name": "NikhilMishra", "author_url": "", "post_date": "2024-06-19T10:27:08.933000", "content": "I think explicitly aggregating location information will be more advantageous, rather than model feeding it, but will it be allowed ? I think its a very important question to be addressed
", "votes": 3, "replies": [ { "id": 2890460, "author_name": "Xintong Zhao", "author_url": "", "post_date": "2024-06-26T06:04:53.707000", "content": "I have the same question, is it allowed to restore the lat/lon information of the grid? @jerrylin96
", "votes": 0, "replies": [] } ] } ] }, { "id": 2878487, "author_name": "heng", "author_url": "", "post_date": "2024-06-19T01:26:37.927000", "content": "@jerrylin96 Can you confirm that train.csv has not changed? Because I remember that the size of this file was about 180G before, no more than 200G, but now it is 359.31G.
", "votes": 4, "replies": [ { "id": 2878511, "author_name": "yelim421", "author_url": "", "post_date": "2024-06-19T02:34:17.967000", "content": "i just downloaded the train data, and it's about 169G
", "votes": 1, "replies": [ { "id": 2878520, "author_name": "heng", "author_url": "", "post_date": "2024-06-19T03:08:36.407000", "content": "Thanks, maybe it's just a display issue.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2886907, "author_name": "Bilzard", "author_url": "", "post_date": "2024-06-23T22:17:34.320000", "content": "@jerrylin96 Can I ask the score difference of the benchmark model between new Public test and new Private test?
\nAre they close?
c.f.) In your previous comment, it had 0.005 score difference between old public test and old private test.
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/506490#2834423
sample_submission.csv weight file is 0 or 1, The difference between the new weights and the previous ones is quite large, with a CV of over 0.994. This needs to be confirmed - should we indeed adopt the new weights?
", "votes": 2, "replies": [ { "id": 2878490, "author_name": "Jerry Lin", "author_url": "", "post_date": "2024-06-19T01:35:14.710000", "content": "Yes, use the new weights.
", "votes": 0, "replies": [] }, { "id": 2879001, "author_name": "ohkawa3", "author_url": "", "post_date": "2024-06-19T10:13:01.857000", "content": "You're probably using the wrong calculation formula.
\nThe discussion below may be helpful.
https://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/495255
", "votes": 3, "replies": [] }, { "id": 2880062, "author_name": "Chengwei Yan", "author_url": "", "post_date": "2024-06-20T02:45:55.670000", "content": "I've also encountered this problem. How did you solve it? I changed the calculation method, but it still seems to be 0.994. Maybe there's something wrong with my calculation?
", "votes": 2, "replies": [ { "id": 2880927, "author_name": "lhwcv", "author_url": "", "post_date": "2024-06-20T13:12:04.917000", "content": "Use sklearn not pytorch implements
", "votes": 0, "replies": [] }, { "id": 2880949, "author_name": "Vasilis", "author_url": "", "post_date": "2024-06-20T13:23:43.607000", "content": "I also use pytorch, what is wrong with pytorch? 😅
", "votes": 0, "replies": [] }, { "id": 2881016, "author_name": "Chengwei Yan", "author_url": "", "post_date": "2024-06-20T13:45:35.403000", "content": "Thank you, I'll double-check
", "votes": 0, "replies": [] }, { "id": 2881784, "author_name": "ynhuhu", "author_url": "", "post_date": "2024-06-21T01:02:23.243000", "content": "I have checked. When torchmetrics calculates very small numbers, r2_score is 1.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2884450, "author_name": "Devesh Kumar Gola", "author_url": "", "post_date": "2024-06-22T14:08:25.930000", "content": "Project System is fast but lagging
", "votes": 0, "replies": [] }, { "id": 2880096, "author_name": "MBOOK", "author_url": "", "post_date": "2024-06-20T03:56:47.560000", "content": "\nI'm new in this competition, and I have one question about r2 calculation.
\nIn the description page, the following statement can be seen:
\nThis weighting can also be calculated without downloading this file. The top 12 levels (0-11 inclusive) for 'ptend_q0001', 'ptend_q0002', 'ptend_q0003', 'ptend_u', and 'ptend_v' can be ignored.
Is the r2 calculated with skipping these columns in submission stage?
\nIn other words, columns with weight=0 are skipped in r2 calculation?
yes, pretty much…
", "votes": 0, "replies": [] }, { "id": 2880432, "author_name": "greySnow", "author_url": "", "post_date": "2024-06-20T07:32:37.540000", "content": "No. It is 1 for these cols.
", "votes": 3, "replies": [ { "id": 2881786, "author_name": "ynhuhu", "author_url": "", "post_date": "2024-06-21T01:08:29.960000", "content": "I don't understand what you mean, I checked these columns and they are 0.
", "votes": 0, "replies": [] }, { "id": 2881844, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2024-06-21T02:52:32.243000", "content": "R2 value should be 1 for those columns with weight 0
", "votes": 1, "replies": [] }, { "id": 2882139, "author_name": "MBOOK", "author_url": "", "post_date": "2024-06-21T07:19:46.847000", "content": "Thanks everyone.
\nI have another question. For convenience, I split the case 1 and 2 depending on target columns.
1). About the top 12 levels (0-11 inclusive) for 'ptend_q0001', 'ptend_q0002', 'ptend_q0003', 'ptend_u', and 'ptend_v'
\nThe reason why I asked the above question is that if y_true is approximately equal to y_pred, then ss_tot will become tiny in r2 calculation, leading to large negative values in each R2_i where i denotes each class.
\nIf the host set R2_i=1 for these columns, then there is no afraid in these targets.
2). About the other target columns having tiny values
\nBut the situation is not resolved for other targets such as ptend_q0002_18, 19 etc which has tiny values.
\nSince R2 calculation is performed on class-wise and then averaged over all classes, very tiny values of ss_tot would be again appeared in some R2_i calculation. Though the problem is alleviated in our local calculation by using some tricks such as adding epsilon to ss_tot or standardization, these approach is not applicable to submission stage because of the difference of r2 calculation.
from these points, my another question is:
\nHow should we avoid this negative explosion in some targets?
p.s.
\nThere is a thread somewhere about different scoring methods of R2, i was always using the one that you first sum of ss_res then sum of ss_tot and then divide res/tot and whatever score i had in my validation sets i had also in my submissions (with the old weights, with the new weights i am currently still training but i am confident its going to be the same).
", "votes": 0, "replies": [] }, { "id": 2882807, "author_name": "Vasilis", "author_url": "", "post_date": "2024-06-21T15:13:26.243000", "content": "i think you were right. When i omit this code in inference i get very negative values, with this code i am at 0.71 ```
\nREPLACE_TO = ['ptend_q0002_0', 'ptend_q0002_1', 'ptend_q0002_2', 'ptend_q0002_3', 'ptend_q0002_4', 'ptend_q0002_5', 'ptend_q0002_6', 'ptend_q0002_7', 'ptend_q0002_8', 'ptend_q0002_9', 'ptend_q0002_10', 'ptend_q0002_11', 'ptend_q0002_12', 'ptend_q0002_13', 'ptend_q0002_14', 'ptend_q0002_15', 'ptend_q0002_16', 'ptend_q0002_17', 'ptend_q0002_18', 'ptend_q0002_19', 'ptend_q0002_20', 'ptend_q0002_21', 'ptend_q0002_22', 'ptend_q0002_23', 'ptend_q0002_24', 'ptend_q0002_25', 'ptend_q0002_26']
\nREPLACE_FROM = ['state_q0002_0', 'state_q0002_1', 'state_q0002_2', 'state_q0002_3', 'state_q0002_4', 'state_q0002_5', 'state_q0002_6', 'state_q0002_7', 'state_q0002_8', 'state_q0002_9', 'state_q0002_10', 'state_q0002_11', 'state_q0002_12', 'state_q0002_13', 'state_q0002_14', 'state_q0002_15', 'state_q0002_16', 'state_q0002_17', 'state_q0002_18', 'state_q0002_19', 'state_q0002_20', 'state_q0002_21', 'state_q0002_22', 'state_q0002_23', 'state_q0002_24', 'state_q0002_25', 'state_q0002_26']
df_test = pd.read_csv(test_file)
\nfor idx in range(0, 27):
\n df_p_test[f\"ptend_q0002_{idx}\"] = -df_test[f\"state_q0002_{idx}\"].to_numpy() / 1200
\n```