{ "id": 447825, "title": "CV vs LB Thread", "url": "/competitions/UBC-OCEAN/discussion/447825", "author_name": "Reacher", "post_date": "2023-10-17T11:46:31.099000", "votes": 11, "comment_count": 16, "views": 0, "content": "

Split: 5-StratifiedKFold

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelCVLB
1 fold0.6160.27
1 fold0.6240.28
1 fold0.740.28(better)
5 folds0.799(mean)0.32
\n

Although the CV/LB gap is substantial, there is some correlation.
\nWhat does your CV/LB look like?

", "messages": [ { "id": 2485741, "postDate": "2023-10-17T12:34:54.687Z", "content": "

A few things probably contirubiting:

\n
    \n
  1. The LB score seems to be off by a few percent due to it calculating based on a non-existent 7th class.
    2. \n
  2. A little under 10% of the data has the wrong mask, by my count 46 images in the train set are significantly off and shouldn't be used. I'd assume about a bit under that amount to be off in the test set because there's fewer proportion of WSI's in the test set and all the errors seem to be in the WSI data.
    3. \n
  3. If 'other' class is not present in your CV you'll artificially have a higher score than the LB where 'other' is present (up to 14% penalty on current LB).
    4. \n
  4. The main challenge of the competition is to have out of distribution data in the test set so without using out of distribution data in train there will be a chronic cv vs lb gap
    5. \n
", "rawMarkdown": "A few things probably contirubiting:\n1. The [LB score seems to be off](https://www.kaggle.com/competitions/UBC-OCEAN/discussion/446760) by a few percent due to it calculating based on a non-existent 7th class.\n2. A little under 10% of the data has the [wrong mask](https://www.kaggle.com/competitions/UBC-OCEAN/discussion/447063), by my count 46 images in the train set are significantly off and shouldn't be used. I'd assume about a bit under that amount to be off in the test set because there's fewer proportion of WSI's in the test set and all the errors seem to be in the WSI data.\n3. If 'other' class is not present in your CV you'll artificially have a higher score than the LB where 'other' is present (up to 14% penalty on current LB).\n4. The main challenge of the competition is to have out of distribution data in the test set so without using out of distribution data in train there will be a chronic cv vs lb gap", "votes": 13, "replies": [ { "id": 2485746, "postDate": "2023-10-17T12:39:17.893Z", "content": "

Great points

", "rawMarkdown": "Great points", "votes": 2 }, { "id": 2485781, "postDate": "2023-10-17T13:03:11.783Z", "content": "

Thank you for your comment.
\nYou've covered almost all issues and the necessary steps for success in this competition.
\nMuch appreciated.

", "rawMarkdown": "Thank you for your comment.\nYou've covered almost all issues and the necessary steps for success in this competition. \nMuch appreciated. " } ] }, { "id": 2485700, "postDate": "2023-10-17T11:46:31.100Z", "content": "

Split: 5-StratifiedKFold

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelCVLB
1 fold0.6160.27
1 fold0.6240.28
1 fold0.740.28(better)
5 folds0.799(mean)0.32
\n

Although the CV/LB gap is substantial, there is some correlation.
\nWhat does your CV/LB look like?

", "rawMarkdown": "Split: 5-StratifiedKFold \n| Model | CV | LB |\n| --- | --- | --- |\n| 1 fold|0.616 | 0.27 |\n| 1 fold |0.624 | 0.28 |\n| 1 fold |0.74 | 0.28(better) |\n| 5 folds |0.799(mean) | 0.32 |\n\nAlthough the CV/LB gap is substantial, there is some correlation.\nWhat does your CV/LB look like?", "votes": 11 }, { "id": 2485775, "postDate": "2023-10-17T12:57:20.443Z", "content": "

EfficientNetV2B0
\n1 / 10 fold
\nCV: 0.82 (5 classes only)
\nLB: 0.37 (after rescore: 0.43)

", "rawMarkdown": "EfficientNetV2B0\n1 / 10 fold\nCV: 0.82 (5 classes only)\nLB: 0.37 (after rescore: 0.43)", "votes": 3, "replies": [ { "id": 2488463, "postDate": "2023-10-19T09:14:59.003Z", "content": "

That's really cool! If convinient, I wonder what is your model input size you use to resize high resolution images?

", "rawMarkdown": "That's really cool! If convinient, I wonder what is your model input size you use to resize high resolution images?", "replies": [ { "id": 2488515, "postDate": "2023-10-19T09:48:42.767Z", "content": "

should be 1024 I guess🤔

", "rawMarkdown": "should be 1024 I guess🤔" }, { "id": 2488802, "postDate": "2023-10-19T14:02:38.720Z", "content": "

恭喜你，答错了😄
\nIt's 512.

", "rawMarkdown": "恭喜你，答错了😄 \nIt's 512.", "votes": 1 }, { "id": 2500846, "postDate": "2023-10-27T03:55:04.987Z", "content": "

using thumbnail images ?

", "rawMarkdown": "using thumbnail images ?" } ] }, { "id": 2501454, "postDate": "2023-10-27T13:00:56.030Z", "content": "

@wuliaokaola Hello, may I ask what assessment index your cv refers to

", "rawMarkdown": "@wuliaokaola Hello, may I ask what assessment index your cv refers to", "replies": [ { "id": 2502070, "postDate": "2023-10-27T23:20:33.307Z", "content": "

It's balanced accuracy, the same as LB.

", "rawMarkdown": "It's [balanced accuracy](http://www.kaggle.com/competitions/UBC-OCEAN/overview/evaluation), the same as LB. " }, { "id": 2502268, "postDate": "2023-10-28T04:49:52.143Z", "content": "

Hi, did you train with thumbnails images and a few WSI images

", "rawMarkdown": "Hi, did you train with thumbnails images and a few WSI images" }, { "id": 2502350, "postDate": "2023-10-28T06:17:15.150Z", "content": "

I'm trying everything which can improve the score.

", "rawMarkdown": "I'm trying everything which can improve the score." }, { "id": 2502367, "postDate": "2023-10-28T06:35:42.847Z", "content": "

May I ask if you have adopted the training method of cutting patches from WSI, or is your current score still based on training with WSI thumbnails + TMA?

", "rawMarkdown": "May I ask if you have adopted the training method of cutting patches from WSI, or is your current score still based on training with WSI thumbnails + TMA?", "votes": 1 } ] } ] }, { "id": 2485704, "postDate": "2023-10-17T11:53:11.553Z", "content": "

It's not correlating for me well. Also it changes upon the data I use (whether I use tiles as additional data or not)
\nMy best submission yet (on the lb) is from 1 fold, cv = 0.51 and lb = 0.33
\nBut weirdly, with change in model/data. The same fold with cv = 0.59 has lb = 0.26 .
\nI think it'll take some time for our cv vs lb to stabilize

", "rawMarkdown": "It's not correlating for me well. Also it changes upon the data I use (whether I use tiles as additional data or not)\nMy best submission yet (on the lb) is from 1 fold, cv = 0.51 and lb = 0.33\nBut weirdly, with change in model/data. The same fold with cv = 0.59 has lb = 0.26 .\nI think it'll take some time for our cv vs lb to stabilize ", "votes": 4 }, { "id": 2485712, "postDate": "2023-10-17T11:58:27.967Z", "content": "

Do you predict or overwrite any Other class?

", "rawMarkdown": "Do you predict or overwrite any Other class?", "votes": 1, "replies": [ { "id": 2485723, "postDate": "2023-10-17T12:07:25.417Z", "content": "

Nope, I'm just training on classes present in the training dataset

", "rawMarkdown": "Nope, I'm just training on classes present in the training dataset" } ] } ], "comments": [ { "id": 2485741, "author_name": "David Austin", "author_url": "", "post_date": "2023-10-17T12:34:54.687000", "content": "

A few things probably contirubiting:

\n
    \n
  1. The LB score seems to be off by a few percent due to it calculating based on a non-existent 7th class.
    2. \n
  2. A little under 10% of the data has the wrong mask, by my count 46 images in the train set are significantly off and shouldn't be used. I'd assume about a bit under that amount to be off in the test set because there's fewer proportion of WSI's in the test set and all the errors seem to be in the WSI data.
    3. \n
  3. If 'other' class is not present in your CV you'll artificially have a higher score than the LB where 'other' is present (up to 14% penalty on current LB).
    4. \n
  4. The main challenge of the competition is to have out of distribution data in the test set so without using out of distribution data in train there will be a chronic cv vs lb gap
    5. \n
", "votes": 13, "replies": [ { "id": 2485746, "author_name": "pjmathematician", "author_url": "", "post_date": "2023-10-17T12:39:17.893000", "content": "

Great points

", "votes": 2, "replies": [] }, { "id": 2485781, "author_name": "Reacher", "author_url": "", "post_date": "2023-10-17T13:03:11.783000", "content": "

Thank you for your comment.
\nYou've covered almost all issues and the necessary steps for success in this competition.
\nMuch appreciated.

", "votes": 0, "replies": [] } ] }, { "id": 2485775, "author_name": "Johnny Lee", "author_url": "", "post_date": "2023-10-17T12:57:20.443000", "content": "

EfficientNetV2B0
\n1 / 10 fold
\nCV: 0.82 (5 classes only)
\nLB: 0.37 (after rescore: 0.43)

", "votes": 3, "replies": [ { "id": 2488463, "author_name": "Time Master", "author_url": "", "post_date": "2023-10-19T09:14:59.003000", "content": "

That's really cool! If convinient, I wonder what is your model input size you use to resize high resolution images?

", "votes": 0, "replies": [ { "id": 2488515, "author_name": "Seeing Times", "author_url": "", "post_date": "2023-10-19T09:48:42.767000", "content": "

should be 1024 I guess🤔

", "votes": 0, "replies": [] }, { "id": 2488802, "author_name": "Johnny Lee", "author_url": "", "post_date": "2023-10-19T14:02:38.720000", "content": "

恭喜你，答错了😄
\nIt's 512.

", "votes": 1, "replies": [] }, { "id": 2500846, "author_name": "Sonu Jha", "author_url": "", "post_date": "2023-10-27T03:55:04.987000", "content": "

using thumbnail images ?

", "votes": 0, "replies": [] } ] }, { "id": 2501454, "author_name": "allen wang", "author_url": "", "post_date": "2023-10-27T13:00:56.030000", "content": "

@wuliaokaola Hello, may I ask what assessment index your cv refers to

", "votes": 0, "replies": [ { "id": 2502070, "author_name": "Johnny Lee", "author_url": "", "post_date": "2023-10-27T23:20:33.307000", "content": "

It's balanced accuracy, the same as LB.

", "votes": 0, "replies": [] }, { "id": 2502268, "author_name": "allen wang", "author_url": "", "post_date": "2023-10-28T04:49:52.143000", "content": "

Hi, did you train with thumbnails images and a few WSI images

", "votes": 0, "replies": [] }, { "id": 2502350, "author_name": "Johnny Lee", "author_url": "", "post_date": "2023-10-28T06:17:15.150000", "content": "

I'm trying everything which can improve the score.

", "votes": 0, "replies": [] }, { "id": 2502367, "author_name": "Huang Jin Feng", "author_url": "", "post_date": "2023-10-28T06:35:42.847000", "content": "

May I ask if you have adopted the training method of cutting patches from WSI, or is your current score still based on training with WSI thumbnails + TMA?

", "votes": 1, "replies": [] } ] } ] }, { "id": 2485704, "author_name": "pjmathematician", "author_url": "", "post_date": "2023-10-17T11:53:11.553000", "content": "

It's not correlating for me well. Also it changes upon the data I use (whether I use tiles as additional data or not)
\nMy best submission yet (on the lb) is from 1 fold, cv = 0.51 and lb = 0.33
\nBut weirdly, with change in model/data. The same fold with cv = 0.59 has lb = 0.26 .
\nI think it'll take some time for our cv vs lb to stabilize

", "votes": 4, "replies": [] }, { "id": 2485712, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2023-10-17T11:58:27.967000", "content": "

Do you predict or overwrite any Other class?

", "votes": 1, "replies": [ { "id": 2485723, "author_name": "Reacher", "author_url": "", "post_date": "2023-10-17T12:07:25.417000", "content": "

Nope, I'm just training on classes present in the training dataset

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2485741": "A few things probably contirubiting:\n1. The [LB score seems to be off](https://www.kaggle.com/competitions/UBC-OCEAN/discussion/446760) by a few percent due to it calculating based on a non-existent 7th class.\n2. A little under 10% of the data has the [wrong mask](https://www.kaggle.com/competitions/UBC-OCEAN/discussion/447063), by my count 46 images in the train set are significantly off and shouldn't be used. I'd assume about a bit under that amount to be off in the test set because there's fewer proportion of WSI's in the test set and all the errors seem to be in the WSI data.\n3. If 'other' class is not present in your CV you'll artificially have a higher score than the LB where 'other' is present (up to 14% penalty on current LB).\n4. The main challenge of the competition is to have out of distribution data in the test set so without using out of distribution data in train there will be a chronic cv vs lb gap", "2485700": "Split: 5-StratifiedKFold \n| Model | CV | LB |\n| --- | --- | --- |\n| 1 fold|0.616 | 0.27 |\n| 1 fold |0.624 | 0.28 |\n| 1 fold |0.74 | 0.28(better) |\n| 5 folds |0.799(mean) | 0.32 |\n\nAlthough the CV/LB gap is substantial, there is some correlation.\nWhat does your CV/LB look like?", "2485775": "EfficientNetV2B0\n1 / 10 fold\nCV: 0.82 (5 classes only)\nLB: 0.37 (after rescore: 0.43)", "2485704": "It's not correlating for me well. Also it changes upon the data I use (whether I use tiles as additional data or not)\nMy best submission yet (on the lb) is from 1 fold, cv = 0.51 and lb = 0.33\nBut weirdly, with change in model/data. The same fold with cv = 0.59 has lb = 0.26 .\nI think it'll take some time for our cv vs lb to stabilize ", "2485712": "Do you predict or overwrite any Other class?" } }