{ "id": 453158, "title": "In this challenge, do ensemble can increase score?", "url": "/competitions/UBC-OCEAN/discussion/453158", "author_name": "Quan Vu", "post_date": "2023-11-05T07:23:29.080000", "votes": 3, "comment_count": 16, "views": 0, "content": "

I split data to 5 fold, submission is ensemble by 5 fold model. Another submission, I ensemble 5 fold model 1 and 5 fold model 2. But score not improve, this is very weird. Does anyone have the same problem as me?

", "messages": [ { "id": 2513103, "postDate": "2023-11-05T07:23:29.080Z", "content": "

I split data to 5 fold, submission is ensemble by 5 fold model. Another submission, I ensemble 5 fold model 1 and 5 fold model 2. But score not improve, this is very weird. Does anyone have the same problem as me?

", "rawMarkdown": "I split data to 5 fold, submission is ensemble by 5 fold model. Another submission, I ensemble 5 fold model 1 and 5 fold model 2. But score not improve, this is very weird. Does anyone have the same problem as me?", "votes": 3 }, { "id": 2513294, "postDate": "2023-11-05T11:13:10.783Z", "content": "

Same problem, I ensembled the model that got 0.4 and the model that got 0.38, and got 0.38. We need to reconsider how we use softmax in ensemble.

", "rawMarkdown": "Same problem, I ensembled the model that got 0.4 and the model that got 0.38, and got 0.38. We need to reconsider how we use softmax in ensemble.", "votes": 1, "replies": [ { "id": 2513437, "postDate": "2023-11-05T12:51:41.787Z", "content": "

I ensemble 5 fold 0.36; 0.38; 0.36; 0.34; 0.4 -> ensemble LB score = 0.4. Ensemble 5 fold LB = 0.43 with 5 fold LB = 0.45 -> ensemble LB score = 0.45 πŸ˜…

", "rawMarkdown": "I ensemble 5 fold 0.36; 0.38; 0.36; 0.34; 0.4 -> ensemble LB score = 0.4. Ensemble 5 fold LB = 0.43 with 5 fold LB = 0.45 -> ensemble LB score = 0.45 πŸ˜…", "replies": [ { "id": 2513455, "postDate": "2023-11-05T13:11:05.763Z", "content": "

The scores 0.36; 0.38; 0.36; 0.34; 0.4, do you compute them on all images or just on TMA?

", "rawMarkdown": "The scores 0.36; 0.38; 0.36; 0.34; 0.4, do you compute them on all images or just on TMA?" }, { "id": 2513533, "postDate": "2023-11-05T14:16:07.807Z", "content": "

All image on public leader board.

", "rawMarkdown": "All image on public leader board." }, { "id": 2556843, "postDate": "2023-12-11T03:28:17.843Z", "content": "

Has this problem been solved

", "rawMarkdown": "Has this problem been solved" }, { "id": 2557857, "postDate": "2023-12-11T17:45:08.670Z", "content": "

No, this problem not solved. My ensemble model score lower or equal best single model.

", "rawMarkdown": "No, this problem not solved. My ensemble model score lower or equal best single model." } ] } ] }, { "id": 2513225, "postDate": "2023-11-05T09:50:29Z", "content": "

yes same problem ensemble decreases the score

", "rawMarkdown": "yes same problem ensemble decreases the score", "votes": 1 }, { "id": 2513334, "postDate": "2023-11-05T11:40:10.103Z", "content": "

Same here. Also, sometimes I have a new model that has better local validation/CV score, but then the LB score is worst… how are you evaluating whether a model will be better or no? Any ideas?..

", "rawMarkdown": "Same here. Also, sometimes I have a new model that has better local validation/CV score, but then the LB score is worst... how are you evaluating whether a model will be better or no? Any ideas?..", "votes": 2, "replies": [ { "id": 2513338, "postDate": "2023-11-05T11:42:26.373Z", "content": "

Does 'better local validation/CV score' means only of TMA scores?

", "rawMarkdown": "Does 'better local validation/CV score' means only of TMA scores?", "replies": [ { "id": 2513355, "postDate": "2023-11-05T11:50:57.543Z", "content": "

No, of all the images. Are you checking the score just on TMA to evaluate whether a model is better than another?

", "rawMarkdown": "No, of all the images. Are you checking the score just on TMA to evaluate whether a model is better than another?" }, { "id": 2513360, "postDate": "2023-11-05T11:56:36.373Z", "content": "

Yes, but there are only 25 TMA images, the scores hard to trust.
\nSo, I observe whether the accuracy of the WSI image of validation also increases.

", "rawMarkdown": "Yes, but there are only 25 TMA images, the scores hard to trust. \nSo, I observe whether the accuracy of the WSI image of validation also increases." }, { "id": 2513390, "postDate": "2023-11-05T12:22:57.680Z", "content": "

You mean if \"the accuracy of the WSI image of validation also decreases\"?. So it is not yet clear to me: do you check the validation score on the whole TMA+WSI images or just the TMA score?

", "rawMarkdown": "You mean if \"the accuracy of the WSI image of validation also decreases\"?. So it is not yet clear to me: do you check the validation score on the whole TMA+WSI images or just the TMA score?" }, { "id": 2513449, "postDate": "2023-11-05T13:07:06.710Z", "content": "

No, I check two validation datasets. (20% WSI and 25 TMA images)

", "rawMarkdown": "No, I check two validation datasets. (20% WSI and 25 TMA images)", "votes": 1 }, { "id": 2513466, "postDate": "2023-11-05T13:18:46.530Z", "content": "

But then If you use all TMA images for validation, you don't have any of these in the training set? I assume then for submission you retrain with all of them not to loose that valuable information, right?

", "rawMarkdown": "But then If you use all TMA images for validation, you don't have any of these in the training set? I assume then for submission you retrain with all of them not to loose that valuable information, right?" }, { "id": 2513475, "postDate": "2023-11-05T13:23:43.013Z", "content": "

Yes, I am now checking whether my strategy works well for TMA and will eventually use it for Training.

", "rawMarkdown": "Yes, I am now checking whether my strategy works well for TMA and will eventually use it for Training." } ] }, { "id": 2513432, "postDate": "2023-11-05T12:48:07.730Z", "content": "

πŸ˜‚ Some time, my CV increase but LB decrease, some ensemble model have better CV but LB = 0.3 πŸ˜…. The most confusing thing is ensemble cannot improve LB score. I think my code is bug, but i cannot found it πŸ˜…

", "rawMarkdown": "πŸ˜‚ Some time, my CV increase but LB decrease, some ensemble model have better CV but LB = 0.3 πŸ˜…. The most confusing thing is ensemble cannot improve LB score. I think my code is bug, but i cannot found it πŸ˜…", "votes": 1 } ] } ], "comments": [ { "id": 2513294, "author_name": "devchopin", "author_url": "", "post_date": "2023-11-05T11:13:10.783000", "content": "

Same problem, I ensembled the model that got 0.4 and the model that got 0.38, and got 0.38. We need to reconsider how we use softmax in ensemble.

", "votes": 1, "replies": [ { "id": 2513437, "author_name": "Quan Vu", "author_url": "", "post_date": "2023-11-05T12:51:41.787000", "content": "

I ensemble 5 fold 0.36; 0.38; 0.36; 0.34; 0.4 -> ensemble LB score = 0.4. Ensemble 5 fold LB = 0.43 with 5 fold LB = 0.45 -> ensemble LB score = 0.45 πŸ˜…

", "votes": 0, "replies": [ { "id": 2513455, "author_name": "Andreu Arderiu", "author_url": "", "post_date": "2023-11-05T13:11:05.763000", "content": "

The scores 0.36; 0.38; 0.36; 0.34; 0.4, do you compute them on all images or just on TMA?

", "votes": 0, "replies": [] }, { "id": 2513533, "author_name": "Quan Vu", "author_url": "", "post_date": "2023-11-05T14:16:07.807000", "content": "

All image on public leader board.

", "votes": 0, "replies": [] }, { "id": 2556843, "author_name": "allen wang", "author_url": "", "post_date": "2023-12-11T03:28:17.843000", "content": "

Has this problem been solved

", "votes": 0, "replies": [] }, { "id": 2557857, "author_name": "Quan Vu", "author_url": "", "post_date": "2023-12-11T17:45:08.670000", "content": "

No, this problem not solved. My ensemble model score lower or equal best single model.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2513225, "author_name": "Arunodhayan", "author_url": "", "post_date": "2023-11-05T09:50:29", "content": "

yes same problem ensemble decreases the score

", "votes": 1, "replies": [] }, { "id": 2513334, "author_name": "Andreu Arderiu", "author_url": "", "post_date": "2023-11-05T11:40:10.103000", "content": "

Same here. Also, sometimes I have a new model that has better local validation/CV score, but then the LB score is worst… how are you evaluating whether a model will be better or no? Any ideas?..

", "votes": 2, "replies": [ { "id": 2513338, "author_name": "devchopin", "author_url": "", "post_date": "2023-11-05T11:42:26.373000", "content": "

Does 'better local validation/CV score' means only of TMA scores?

", "votes": 0, "replies": [ { "id": 2513355, "author_name": "Andreu Arderiu", "author_url": "", "post_date": "2023-11-05T11:50:57.543000", "content": "

No, of all the images. Are you checking the score just on TMA to evaluate whether a model is better than another?

", "votes": 0, "replies": [] }, { "id": 2513360, "author_name": "devchopin", "author_url": "", "post_date": "2023-11-05T11:56:36.373000", "content": "

Yes, but there are only 25 TMA images, the scores hard to trust.
\nSo, I observe whether the accuracy of the WSI image of validation also increases.

", "votes": 0, "replies": [] }, { "id": 2513390, "author_name": "Andreu Arderiu", "author_url": "", "post_date": "2023-11-05T12:22:57.680000", "content": "

You mean if \"the accuracy of the WSI image of validation also decreases\"?. So it is not yet clear to me: do you check the validation score on the whole TMA+WSI images or just the TMA score?

", "votes": 0, "replies": [] }, { "id": 2513449, "author_name": "devchopin", "author_url": "", "post_date": "2023-11-05T13:07:06.710000", "content": "

No, I check two validation datasets. (20% WSI and 25 TMA images)

", "votes": 1, "replies": [] }, { "id": 2513466, "author_name": "Andreu Arderiu", "author_url": "", "post_date": "2023-11-05T13:18:46.530000", "content": "

But then If you use all TMA images for validation, you don't have any of these in the training set? I assume then for submission you retrain with all of them not to loose that valuable information, right?

", "votes": 0, "replies": [] }, { "id": 2513475, "author_name": "devchopin", "author_url": "", "post_date": "2023-11-05T13:23:43.013000", "content": "

Yes, I am now checking whether my strategy works well for TMA and will eventually use it for Training.

", "votes": 0, "replies": [] } ] }, { "id": 2513432, "author_name": "Quan Vu", "author_url": "", "post_date": "2023-11-05T12:48:07.730000", "content": "

πŸ˜‚ Some time, my CV increase but LB decrease, some ensemble model have better CV but LB = 0.3 πŸ˜…. The most confusing thing is ensemble cannot improve LB score. I think my code is bug, but i cannot found it πŸ˜…

", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "2513103": "I split data to 5 fold, submission is ensemble by 5 fold model. Another submission, I ensemble 5 fold model 1 and 5 fold model 2. But score not improve, this is very weird. Does anyone have the same problem as me?", "2513294": "Same problem, I ensembled the model that got 0.4 and the model that got 0.38, and got 0.38. We need to reconsider how we use softmax in ensemble.", "2513225": "yes same problem ensemble decreases the score", "2513334": "Same here. Also, sometimes I have a new model that has better local validation/CV score, but then the LB score is worst... how are you evaluating whether a model will be better or no? Any ideas?.." } }