![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5487737%2Fee1d48213af364532dc2ffdb67357c94%2Fleap_corr.png?generation=1714210918935331&alt=media\")
Hi Everyone. I hope you're doing well in this competition. I'm a little bit curious to see how things goes for you concerning CV and LB gaps. Feel free to share here.
\nI start first by sharing two experiments:
\nSo basically, I noticed that there is a big correlation between CV and LB. In my experiments for CV < 0.55 the gap is around 0.01x and when CV>0.60 the gap is around 0.03x.
\nHappy Kaggling !!!
", "messages": [ { "id": 2778613, "postDate": "2024-04-27T08:53:31.540Z", "content": "Hi Everyone. I hope you're doing well in this competition. I'm a little bit curious to see how things goes for you concerning CV and LB gaps. Feel free to share here.
\nI start first by sharing two experiments:
\nSo basically, I noticed that there is a big correlation between CV and LB. In my experiments for CV < 0.55 the gap is around 0.01x and when CV>0.60 the gap is around 0.03x.
\nHappy Kaggling !!!
", "rawMarkdown": "Hi Everyone. I hope you're doing well in this competition. I'm a little bit curious to see how things goes for you concerning CV and LB gaps. Feel free to share here.\n\nI start first by sharing two experiments:\n* A simple Xgboost : CV=0.43x / LB = 0.44x\n* A simple FCN : CV=0.51x / LB=0.52x\n\nSo basically, I noticed that there is a **big correlation between CV and LB**. In my experiments for CV < 0.55 the gap is around 0.01x and when CV>0.60 the gap is around 0.03x.\n\n\n**Happy Kaggling !!!**", "votes": 22 }, { "id": 2778710, "postDate": "2024-04-27T09:42:40.823Z", "content": "For constant or unpredictable targets I assign R2=0, for the rest I compute per-class R2 score and then take average of all scores,
\nthe validation is done on last 2* million samples.
| CV | \nLB | \n
|---|---|
| 0.38753 | \n0.56053 | \n
| 0.41259 | \n0.58585 | \n
| 0.43521 | \n0.60716 | \n
| 0.45459 | \n0.62662 | \n
I expect this competition have little to none shakeup.
", "rawMarkdown": "For constant or unpredictable targets I assign R2=0, for the rest I compute per-class R2 score and then take average of all scores,\nthe validation is done on last 2* million samples.\n\n| CV | LB |\n| --- | --- |\n| 0.38753 | 0.56053 |\n| 0.41259 | 0.58585 |\n| 0.43521 | 0.60716 | \n| 0.45459| 0.62662 |\n\n\n\nI expect this competition have little to none shakeup.", "votes": 7, "replies": [ { "id": 2778721, "postDate": "2024-04-27T09:46:34.183Z", "content": "Yes you're right. Actual R2 is inflated because for the constant var, predicting the same constant yiels an R2= 1 just inflating the computations.
", "rawMarkdown": "Yes you're right. Actual R2 is inflated because for the constant var, predicting the same constant yiels an R2= 1 just inflating the computations.", "votes": 5 }, { "id": 2778756, "postDate": "2024-04-27T10:27:15.437Z", "content": "validation on the last 10M samples or 1M samples? Thanks for sharing this insightful result.
", "rawMarkdown": "validation on the last 10M samples or 1M samples? Thanks for sharing this insightful result.\n", "votes": 3, "replies": [ { "id": 2778759, "postDate": "2024-04-27T10:28:16.270Z", "content": "Validation on 500 000 rows
", "rawMarkdown": "Validation on 500 000 rows", "votes": 4 } ] }, { "id": 2778931, "postDate": "2024-04-27T13:04:05.413Z", "content": "@martynoveduard, please what is meant by unpredictable targets, the constant targets is perhaps those with 0 as sample_submission(scale) ?
\nThanks
I didn't dig deep yet, but I found some targets that are neither constant or predictable (i.e. for these columns it's impossible to obtain R2 score > 0), I called them unpredictable
", "rawMarkdown": "I didn't dig deep yet, but I found some targets that are neither constant or predictable (i.e. for these columns it's impossible to obtain R2 score > 0), I called them unpredictable " } ] } ] }, { "id": 2779144, "postDate": "2024-04-27T14:43:53.600Z", "content": "In my experiments CV-LB are well correlated.
\n| CV | \nLB | \n
|---|---|
| 0.6274 | \n0.6241 | \n
| 0.6197 | \n0.6164 | \n
| 0.6075 | \n0.6081 | \n
Use the last 650,000 records as validation.
\nCV: 0.765
\nLB: 0.756
How is target with value ==0 calculated? R2 equals 1 for them? It seems to be impossible to predict ptend_q0002_15~ptend_q0002_26, and they make the cv score very bad …
", "rawMarkdown": "How is target with value ==0 calculated? R2 equals 1 for them? It seems to be impossible to predict ptend_q0002_15~ptend_q0002_26, and they make the cv score very bad ...", "votes": 1, "replies": [ { "id": 2828383, "postDate": "2024-05-22T04:52:53.913Z", "content": "You can have perfect scores for those:
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/502484
Thank you for the info, from the metric calculation, it seems that we get 1 for 0s. But the correlation between local CV and LB seems to be perfect, and my CV skips the columns with all 0.
", "rawMarkdown": "Thank you for the info, from the metric calculation, it seems that we get 1 for 0s. But the correlation between local CV and LB seems to be perfect, and my CV skips the columns with all 0. " } ] } ] }, { "id": 2778889, "postDate": "2024-04-27T12:29:35.387Z", "content": "Excuse the newbie question: I know LB stands for Leadboard [score], and the competition brief explains how that is calculated, but what exactly is CV in this context? Thanks!
", "rawMarkdown": "Excuse the newbie question: I know LB stands for Leadboard [score], and the competition brief explains how that is calculated, but what exactly is CV in this context? Thanks!", "votes": 1, "replies": [ { "id": 2779492, "postDate": "2024-04-27T17:11:37.537Z", "content": "The CV is the score on your validation dagta. It is supposed to be proper cross validation score, but sometimes it is the score on one validation fold. I hope it mates it clear
", "rawMarkdown": "The CV is the score on your validation dagta. It is supposed to be *proper cross validation score*, but sometimes it is the score on one validation fold. I hope it mates it clear", "votes": 7, "replies": [ { "id": 2779538, "postDate": "2024-04-27T17:29:27.430Z", "content": "Ah thanks, Googling it seemed to be related to cross-validation, but as that was a process not a metric I thought it must be something else! (Any idea what metric \"CV\" is then, anyone, out of curiosity? -- I don't need to know, was just plugging my lack of knowledge)
", "rawMarkdown": "Ah thanks, Googling it seemed to be related to cross-validation, but as that was a process not a metric I thought it must be something else! (Any idea what metric \"CV\" is then, anyone, out of curiosity? -- I don't need to know, was just plugging my lack of knowledge)", "votes": 1 }, { "id": 2781041, "postDate": "2024-04-28T14:21:04.960Z", "content": "\"CV score\" means \"score parsed on the validation data\".
", "rawMarkdown": "\"CV score\" means \"score parsed on the validation data\"." }, { "id": 2781524, "postDate": "2024-04-28T20:27:10.070Z", "content": "Thanks -- but what does 'score' mean mathematically?
", "rawMarkdown": "Thanks -- but what does 'score' mean mathematically?" }, { "id": 2781636, "postDate": "2024-04-28T22:27:35.533Z", "content": "Same 'score' (metric) as LB…
", "rawMarkdown": "Same 'score' (metric) as LB..." } ] }, { "id": 2779501, "postDate": "2024-04-27T17:15:27.510Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2830030, "postDate": "2024-05-23T00:57:44.230Z", "content": "current my best model:
\nCV: 0.7657
\nLB: 0.76360
Good to know
", "rawMarkdown": "Good to know" }, { "id": 2788978, "postDate": "2024-05-02T13:52:03.647Z", "content": "I've got quiet the deflated LB score compared to my CV score. CV around ~.6 dropping to ~.5 regularly.
\n*for reference, i'll define:
\nIn summary, what I'm doing is…
\nShould I be replacing y_pred_n with zero, or calculating y_train_s means and replacing y_pred_s at that point?
", "rawMarkdown": "I've got quiet the deflated LB score compared to my CV score. CV around ~.6 dropping to ~.5 regularly. \n\n*for reference, i'll define:\n - original scale \"_o\"\n - normalized variables \"_n\" \n - scaled by submission weights \"_s\"\n\nIn summary, what I'm doing is...\n1. y_o -> y_n\n2. model(y_n) -> y_pred_n\n3. y_pred_n -> y_pred_o\n4. r2_score for (y_o, y_pred_o)\n5. replace y_pred_o with mean of y_o where r2 is negative\n * This is where I perform cross val \n6. y_pred_o -> y_pred_s\n\nShould I be replacing y_pred_n with zero, or calculating y_train_s means and replacing y_pred_s at that point?" } ], "comments": [ { "id": 2778710, "author_name": "slime", "author_url": "", "post_date": "2024-04-27T09:42:40.823000", "content": "For constant or unpredictable targets I assign R2=0, for the rest I compute per-class R2 score and then take average of all scores,
\nthe validation is done on last 2* million samples.
| CV | \nLB | \n
|---|---|
| 0.38753 | \n0.56053 | \n
| 0.41259 | \n0.58585 | \n
| 0.43521 | \n0.60716 | \n
| 0.45459 | \n0.62662 | \n
I expect this competition have little to none shakeup.
", "votes": 7, "replies": [ { "id": 2778721, "author_name": "Ulrich G.", "author_url": "", "post_date": "2024-04-27T09:46:34.183000", "content": "Yes you're right. Actual R2 is inflated because for the constant var, predicting the same constant yiels an R2= 1 just inflating the computations.
", "votes": 5, "replies": [] }, { "id": 2778756, "author_name": "HungryLearner", "author_url": "", "post_date": "2024-04-27T10:27:15.437000", "content": "validation on the last 10M samples or 1M samples? Thanks for sharing this insightful result.
", "votes": 3, "replies": [ { "id": 2778759, "author_name": "Ulrich G.", "author_url": "", "post_date": "2024-04-27T10:28:16.270000", "content": "Validation on 500 000 rows
", "votes": 4, "replies": [] } ] }, { "id": 2778931, "author_name": "HungryLearner", "author_url": "", "post_date": "2024-04-27T13:04:05.413000", "content": "@martynoveduard, please what is meant by unpredictable targets, the constant targets is perhaps those with 0 as sample_submission(scale) ?
\nThanks
I didn't dig deep yet, but I found some targets that are neither constant or predictable (i.e. for these columns it's impossible to obtain R2 score > 0), I called them unpredictable
", "votes": 0, "replies": [] } ] } ] }, { "id": 2779144, "author_name": "Giba", "author_url": "", "post_date": "2024-04-27T14:43:53.600000", "content": "In my experiments CV-LB are well correlated.
\n| CV | \nLB | \n
|---|---|
| 0.6274 | \n0.6241 | \n
| 0.6197 | \n0.6164 | \n
| 0.6075 | \n0.6081 | \n
Use the last 650,000 records as validation.
\nCV: 0.765
\nLB: 0.756
How is target with value ==0 calculated? R2 equals 1 for them? It seems to be impossible to predict ptend_q0002_15~ptend_q0002_26, and they make the cv score very bad …
", "votes": 1, "replies": [ { "id": 2828383, "author_name": "DennisSakva", "author_url": "", "post_date": "2024-05-22T04:52:53.913000", "content": "You can have perfect scores for those:
\nhttps://www.kaggle.com/competitions/leap-atmospheric-physics-ai-climsim/discussion/502484
Thank you for the info, from the metric calculation, it seems that we get 1 for 0s. But the correlation between local CV and LB seems to be perfect, and my CV skips the columns with all 0.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2778889, "author_name": "Charlie Wartnaby", "author_url": "", "post_date": "2024-04-27T12:29:35.387000", "content": "Excuse the newbie question: I know LB stands for Leadboard [score], and the competition brief explains how that is calculated, but what exactly is CV in this context? Thanks!
", "votes": 1, "replies": [ { "id": 2779492, "author_name": "Ulrich G.", "author_url": "", "post_date": "2024-04-27T17:11:37.537000", "content": "The CV is the score on your validation dagta. It is supposed to be proper cross validation score, but sometimes it is the score on one validation fold. I hope it mates it clear
", "votes": 7, "replies": [ { "id": 2779538, "author_name": "Charlie Wartnaby", "author_url": "", "post_date": "2024-04-27T17:29:27.430000", "content": "Ah thanks, Googling it seemed to be related to cross-validation, but as that was a process not a metric I thought it must be something else! (Any idea what metric \"CV\" is then, anyone, out of curiosity? -- I don't need to know, was just plugging my lack of knowledge)
", "votes": 1, "replies": [] }, { "id": 2781041, "author_name": "Catadanna", "author_url": "", "post_date": "2024-04-28T14:21:04.960000", "content": "\"CV score\" means \"score parsed on the validation data\".
", "votes": 0, "replies": [] }, { "id": 2781524, "author_name": "Charlie Wartnaby", "author_url": "", "post_date": "2024-04-28T20:27:10.070000", "content": "Thanks -- but what does 'score' mean mathematically?
", "votes": 0, "replies": [] }, { "id": 2781636, "author_name": "greySnow", "author_url": "", "post_date": "2024-04-28T22:27:35.533000", "content": "Same 'score' (metric) as LB…
", "votes": 0, "replies": [] } ] }, { "id": 2779501, "author_name": "", "author_url": "", "post_date": "2024-04-27T17:15:27.510000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2830030, "author_name": "Bilzard", "author_url": "", "post_date": "2024-05-23T00:57:44.230000", "content": "current my best model:
\nCV: 0.7657
\nLB: 0.76360
Good to know
", "votes": 0, "replies": [] }, { "id": 2788978, "author_name": "Jekasm19", "author_url": "", "post_date": "2024-05-02T13:52:03.647000", "content": "I've got quiet the deflated LB score compared to my CV score. CV around ~.6 dropping to ~.5 regularly.
\n*for reference, i'll define:
\nIn summary, what I'm doing is…
\nShould I be replacing y_pred_n with zero, or calculating y_train_s means and replacing y_pred_s at that point?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2778613": "Hi Everyone. I hope you're doing well in this competition. I'm a little bit curious to see how things goes for you concerning CV and LB gaps. Feel free to share here.\n\nI start first by sharing two experiments:\n* A simple Xgboost : CV=0.43x / LB = 0.44x\n* A simple FCN : CV=0.51x / LB=0.52x\n\nSo basically, I noticed that there is a **big correlation between CV and LB**. In my experiments for CV < 0.55 the gap is around 0.01x and when CV>0.60 the gap is around 0.03x.\n\n\n**Happy Kaggling !!!**", "2778710": "For constant or unpredictable targets I assign R2=0, for the rest I compute per-class R2 score and then take average of all scores,\nthe validation is done on last 2* million samples.\n\n| CV | LB |\n| --- | --- |\n| 0.38753 | 0.56053 |\n| 0.41259 | 0.58585 |\n| 0.43521 | 0.60716 | \n| 0.45459| 0.62662 |\n\n\n\nI expect this competition have little to none shakeup.", "2779144": "In my experiments CV-LB are well correlated. \n| CV | LB |\n| --- | --- |\n| 0.6274 | 0.6241 |\n| 0.6197 | 0.6164 |\n| 0.6075 | 0.6081 |", "2826676": "Use the last 650,000 records as validation.\nCV: 0.765\nLB: 0.756", "2828263": "How is target with value ==0 calculated? R2 equals 1 for them? It seems to be impossible to predict ptend_q0002_15~ptend_q0002_26, and they make the cv score very bad ...", "2778889": "Excuse the newbie question: I know LB stands for Leadboard [score], and the competition brief explains how that is calculated, but what exactly is CV in this context? Thanks!", "2830030": "current my best model:\n\nCV: 0.7657\nLB: 0.76360", "2789401": "Good to know", "2788978": "I've got quiet the deflated LB score compared to my CV score. CV around ~.6 dropping to ~.5 regularly. \n\n*for reference, i'll define:\n - original scale \"_o\"\n - normalized variables \"_n\" \n - scaled by submission weights \"_s\"\n\nIn summary, what I'm doing is...\n1. y_o -> y_n\n2. model(y_n) -> y_pred_n\n3. y_pred_n -> y_pred_o\n4. r2_score for (y_o, y_pred_o)\n5. replace y_pred_o with mean of y_o where r2 is negative\n * This is where I perform cross val \n6. y_pred_o -> y_pred_s\n\nShould I be replacing y_pred_n with zero, or calculating y_train_s means and replacing y_pred_s at that point?" } }