2 Folds ensemble score comes less than a single fold score. \nAny right strategy to do ensemble ?
", "messages": [ { "id": 913433, "postDate": "2020-07-03T08:11:02.813Z", "content": "Blending has worked for me (with a couple of subtleties), but I'm blending different models rather than different folds of the same model.
\n\nOne point that has become really obvious is that it is incredibly easy to overfit this LB! Only 1000 samples, so that stands to reason... But from what I've seen I often get a model/set of models that outperform when it's clear that it's only LB overfitting, and I wouldn't expect the same to hold true on a larger test set.
\n\nInitially I thought that this wasn't a competition that would have a big shake up, but the more I'm playing around the more I think that a shake up is likely. It will be important not to get seduced by high public LB scores!
", "rawMarkdown": "Blending has worked for me (with a couple of subtleties), but I'm blending different models rather than different folds of the same model. \n\nOne point that has become really obvious is that it is incredibly easy to overfit this LB! Only 1000 samples, so that stands to reason... But from what I've seen I often get a model/set of models that outperform when it's clear that it's only LB overfitting, and I wouldn't expect the same to hold true on a larger test set.\n\nInitially I thought that this wasn't a competition that would have a big shake up, but the more I'm playing around the more I think that a shake up is likely. It will be important not to get seduced by high public LB scores!", "votes": 3, "replies": [ { "id": 914034, "postDate": "2020-07-03T15:29:48.467Z", "content": "How do you know when a model is lb overfitting?
", "rawMarkdown": "How do you know when a model is lb overfitting?", "votes": 1 }, { "id": 914065, "postDate": "2020-07-03T15:53:05.970Z", "content": "Typically by sanity checking with CV / other insights. For example, if I see a big leap in LB but CV doesn't show the same jump. Or if I've changed how I'm dealing with data and it shows a jump on the LB for one backbone but not another relative to their previous LB baselines... Stuff like that.
", "rawMarkdown": "Typically by sanity checking with CV / other insights. For example, if I see a big leap in LB but CV doesn't show the same jump. Or if I've changed how I'm dealing with data and it shows a jump on the LB for one backbone but not another relative to their previous LB baselines... Stuff like that.", "votes": 1 }, { "id": 914179, "postDate": "2020-07-03T16:59:54.353Z", "content": "in my case i see a resemblance btw my CV and LB.. both are above .90,although there is inconsistency in the number of times m able to produce same results.
", "rawMarkdown": "in my case i see a resemblance btw my CV and LB.. both are above .90,although there is inconsistency in the number of times m able to produce same results. " } ] }, { "id": 912814, "postDate": "2020-07-02T18:41:00.627Z", "content": "2 Folds ensemble score comes less than a single fold score. \nAny right strategy to do ensemble ?
", "rawMarkdown": "2 Folds ensemble score comes less than a single fold score. \nAny right strategy to do ensemble ?", "votes": 4 }, { "id": 912914, "postDate": "2020-07-02T20:07:10.553Z", "content": "Here they discuss and show code about it. The post it's abour ResNet architectures and due to the Code Requirements of this competition, i think that it's adecuate.
", "rawMarkdown": "[Here](https://discuss.pytorch.org/t/custom-ensemble-approach/52024) they discuss and show code about it. The post it's abour ResNet architectures and due to the Code Requirements of this competition, i think that it's adecuate.", "votes": 2 }, { "id": 913097, "postDate": "2020-07-03T00:53:06.903Z", "content": "Blending has also shown lower scores for me so far. Has someone tried stacking instead ?
", "rawMarkdown": "Blending has also shown lower scores for me so far. Has someone tried stacking instead ?" } ], "comments": [ { "id": 913433, "author_name": "fergusoci", "author_url": "", "post_date": "2020-07-03T08:11:02.813000", "content": "Blending has worked for me (with a couple of subtleties), but I'm blending different models rather than different folds of the same model.
\n\nOne point that has become really obvious is that it is incredibly easy to overfit this LB! Only 1000 samples, so that stands to reason... But from what I've seen I often get a model/set of models that outperform when it's clear that it's only LB overfitting, and I wouldn't expect the same to hold true on a larger test set.
\n\nInitially I thought that this wasn't a competition that would have a big shake up, but the more I'm playing around the more I think that a shake up is likely. It will be important not to get seduced by high public LB scores!
", "votes": 3, "replies": [ { "id": 914034, "author_name": "Shujun", "author_url": "", "post_date": "2020-07-03T15:29:48.467000", "content": "How do you know when a model is lb overfitting?
", "votes": 1, "replies": [] }, { "id": 914065, "author_name": "fergusoci", "author_url": "", "post_date": "2020-07-03T15:53:05.970000", "content": "Typically by sanity checking with CV / other insights. For example, if I see a big leap in LB but CV doesn't show the same jump. Or if I've changed how I'm dealing with data and it shows a jump on the LB for one backbone but not another relative to their previous LB baselines... Stuff like that.
", "votes": 1, "replies": [] }, { "id": 914179, "author_name": "Jaideep", "author_url": "", "post_date": "2020-07-03T16:59:54.353000", "content": "in my case i see a resemblance btw my CV and LB.. both are above .90,although there is inconsistency in the number of times m able to produce same results.
", "votes": 0, "replies": [] } ] }, { "id": 912914, "author_name": "Hiram Coria 🧬", "author_url": "", "post_date": "2020-07-02T20:07:10.553000", "content": "Here they discuss and show code about it. The post it's abour ResNet architectures and due to the Code Requirements of this competition, i think that it's adecuate.
", "votes": 2, "replies": [] }, { "id": 913097, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-07-03T00:53:06.903000", "content": "Blending has also shown lower scores for me so far. Has someone tried stacking instead ?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "913433": "Blending has worked for me (with a couple of subtleties), but I'm blending different models rather than different folds of the same model. \n\nOne point that has become really obvious is that it is incredibly easy to overfit this LB! Only 1000 samples, so that stands to reason... But from what I've seen I often get a model/set of models that outperform when it's clear that it's only LB overfitting, and I wouldn't expect the same to hold true on a larger test set.\n\nInitially I thought that this wasn't a competition that would have a big shake up, but the more I'm playing around the more I think that a shake up is likely. It will be important not to get seduced by high public LB scores!", "912814": "2 Folds ensemble score comes less than a single fold score. \nAny right strategy to do ensemble ?", "912914": "[Here](https://discuss.pytorch.org/t/custom-ensemble-approach/52024) they discuss and show code about it. The post it's abour ResNet architectures and due to the Code Requirements of this competition, i think that it's adecuate.", "913097": "Blending has also shown lower scores for me so far. Has someone tried stacking instead ?" } }