![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F530dc8567f2044bbac20d0f81b7a3088%2Fens-fold_ensemble.drawio%20(1).png?generation=1691263031233671&alt=media\")
I am considering two ways to ensemble models for this competition.
\nIn the figures below, m1 = model 1, m2 = model 2, and m3 = model 3. Each model is considered to have a different structure and achieves similar accuracy on a holdout set.
\nAggregate predictions in each fold before the final ensemble.
\n
Aggregate predictions for each model type before the final ensemble.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F1397a2a808d984e0f490195631de9555%2Fens-model%20ensemble.drawio%20(1).png?generation=1691263014510718&alt=media\")
Due to the computational requirements of training each model, I have been unable to test the performance of each ensemble type using CV. I suspect that the difference is negligible, but I am interested to see what people think. Has anyone tried these methods in previous competitions?
\nNote: Each ensemble uses the median for aggregation
", "messages": [ { "id": 2375729, "postDate": "2023-08-05T21:07:38.493Z", "content": "Hello, I am not sure both aren't the exact same thing…
\nFrom my understanding, each fold-model would have 1/9th of weight in both cases.
I checked all possible predictions for 9 models, and 85.9% (440/512) of the time the final result is the same. Consider this counter example.
\nHere I am considering 0:3 in the 1st fold, 3:6 in the 2nd fold, and 6:9 in the final fold. Every third model has the same backbone [0, 3, 6], [1, 4, 7], [2, 5, 8].
\npreds = [1, 1, 1, 1, 0, 0, 0, 0, 1]
\nModel Ensemble.
\nm1 = [1,1,0] = 1
\nm2 = [1,0,0] = 0
\nm3 = [1,0,1] = 1
\n= 1
Fold Ensemble.
\nf1 = [1,1,1] = 1
\nf2 = [1,0,0] = 0
\nf3 = [0,0,1] = 0
\n= 0
I think that there is an error in the \"m1 = [1,1,0] = 0\" if I got it right, is it possible? Also the thing here is that you are applying the threshold at each sub prediction and most of the times it is better to ensemble the raw probabilities with mean or weighted average and just apply the threshold at the final step to have the final mask with 0s and 1s.
", "rawMarkdown": "I think that there is an error in the \"m1 = [1,1,0] = 0\" if I got it right, is it possible? Also the thing here is that you are applying the threshold at each sub prediction and most of the times it is better to ensemble the raw probabilities with mean or weighted average and just apply the threshold at the final step to have the final mask with 0s and 1s.", "votes": 1 }, { "id": 2377032, "postDate": "2023-08-06T21:46:50.340Z", "content": "The way I do things:
\npreds = [1, 1, 1, 1, 0, 0, 0, 0, 1]
\nModel Ensemble.\nm1 = [,,] = \nm2 = [,,] = \nm3 = [,,] = \n= / = , -> \nFold Ensemble.\nf1 = [,,] = \nf2 = [,,] = \nf3 = [,,] = \n= / = , -> \nOk I understand where you are coming from now. This is actually quite a smart solution!
\nI thought that you were ensembling prior to converting predictions to a 0 or 1.
", "rawMarkdown": "Ok I understand where you are coming from now. This is actually quite a smart solution! \n\nI thought that you were ensembling prior to converting predictions to a 0 or 1.", "votes": 1 } ] }, { "id": 2375783, "postDate": "2023-08-05T22:19:49.223Z", "content": "I also understand them both as the same thing, at the end you have 9 different checkpoints from [3 architectures x 3 train-sets] or viceversa. If the ensemble is done by a simple mean of probabilities at each level, both options should have same results. One could even simply leave the 9 predictions stacked and do the mean of all them together.
", "rawMarkdown": "I also understand them both as the same thing, at the end you have 9 different checkpoints from [3 architectures x 3 train-sets] or viceversa. If the ensemble is done by a simple mean of probabilities at each level, both options should have same results. One could even simply leave the 9 predictions stacked and do the mean of all them together.", "votes": 1, "replies": [ { "id": 2375808, "postDate": "2023-08-05T23:17:00.720Z", "content": "I should have been clear that I was assuming median aggregation.. In this case the predictions will differ 72/512 times (with 9 models)?
\nYes, I agree that for mean aggregation the weight of each model will be 1/9th of the total, and the two options will identical to a 9 prediction stack.
\nHere is the code to reproduce the results for median aggregation.
\n itertools\n numpy as np\n\n generate_binary_arrays(length):\n length <= :\n ValueError()\n =\n \n\n = \n = generate_binary_arrays(length)\n = \n = \n\n arr in all_arrays:\n \n = np.median([np.median(arr[:]), np.median(arr[:]), np.median(arr[:])])\n \n = np.median([np.median([arr[], arr[], arr[]]),np.median([arr[], arr[], arr[]]), np.median([arr[], arr[], arr[]])])\n (arr, p1, p2)\n +=\n p1 == p2:\n +=\n(sc, c, sc/c)\nI am considering two ways to ensemble models for this competition.
\nIn the figures below, m1 = model 1, m2 = model 2, and m3 = model 3. Each model is considered to have a different structure and achieves similar accuracy on a holdout set.
\nAggregate predictions in each fold before the final ensemble.
\n
Aggregate predictions for each model type before the final ensemble.
\n
Due to the computational requirements of training each model, I have been unable to test the performance of each ensemble type using CV. I suspect that the difference is negligible, but I am interested to see what people think. Has anyone tried these methods in previous competitions?
\nNote: Each ensemble uses the median for aggregation
", "rawMarkdown": "I am considering two ways to ensemble models for this competition.\n\nIn the figures below, m1 = model 1, m2 = model 2, and m3 = model 3. Each model is considered to have a different structure and achieves similar accuracy on a holdout set.\n\n### Option 1. Fold Ensemble\n\nAggregate predictions in each fold before the final ensemble.\n\n.png?generation=1691263031233671&alt=media)\n\n### Option 2. Model Ensemble\n\nAggregate predictions for each model type before the final ensemble.\n\n.png?generation=1691263014510718&alt=media)\n\nDue to the computational requirements of training each model, I have been unable to test the performance of each ensemble type using CV. I suspect that the difference is negligible, but I am interested to see what people think. Has anyone tried these methods in previous competitions?\n\nNote: **Each ensemble uses the median for aggregation**", "votes": 1 }, { "id": 2375755, "postDate": "2023-08-05T21:30:26.290Z", "rawMarkdown": "", "votes": -6, "isDeleted": true } ], "comments": [ { "id": 2375729, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-08-05T21:07:38.493000", "content": "Hello, I am not sure both aren't the exact same thing…
\nFrom my understanding, each fold-model would have 1/9th of weight in both cases.
I checked all possible predictions for 9 models, and 85.9% (440/512) of the time the final result is the same. Consider this counter example.
\nHere I am considering 0:3 in the 1st fold, 3:6 in the 2nd fold, and 6:9 in the final fold. Every third model has the same backbone [0, 3, 6], [1, 4, 7], [2, 5, 8].
\npreds = [1, 1, 1, 1, 0, 0, 0, 0, 1]
\nModel Ensemble.
\nm1 = [1,1,0] = 1
\nm2 = [1,0,0] = 0
\nm3 = [1,0,1] = 1
\n= 1
Fold Ensemble.
\nf1 = [1,1,1] = 1
\nf2 = [1,0,0] = 0
\nf3 = [0,0,1] = 0
\n= 0
I think that there is an error in the \"m1 = [1,1,0] = 0\" if I got it right, is it possible? Also the thing here is that you are applying the threshold at each sub prediction and most of the times it is better to ensemble the raw probabilities with mean or weighted average and just apply the threshold at the final step to have the final mask with 0s and 1s.
", "votes": 1, "replies": [] }, { "id": 2377032, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-08-06T21:46:50.340000", "content": "The way I do things:
\npreds = [1, 1, 1, 1, 0, 0, 0, 0, 1]
\nModel Ensemble.\nm1 = [,,] = \nm2 = [,,] = \nm3 = [,,] = \n= / = , -> \nFold Ensemble.\nf1 = [,,] = \nf2 = [,,] = \nf3 = [,,] = \n= / = , -> \nOk I understand where you are coming from now. This is actually quite a smart solution!
\nI thought that you were ensembling prior to converting predictions to a 0 or 1.
", "votes": 1, "replies": [] } ] }, { "id": 2375783, "author_name": "Enric Domingo", "author_url": "", "post_date": "2023-08-05T22:19:49.223000", "content": "I also understand them both as the same thing, at the end you have 9 different checkpoints from [3 architectures x 3 train-sets] or viceversa. If the ensemble is done by a simple mean of probabilities at each level, both options should have same results. One could even simply leave the 9 predictions stacked and do the mean of all them together.
", "votes": 1, "replies": [ { "id": 2375808, "author_name": "Bartley", "author_url": "", "post_date": "2023-08-05T23:17:00.720000", "content": "I should have been clear that I was assuming median aggregation.. In this case the predictions will differ 72/512 times (with 9 models)?
\nYes, I agree that for mean aggregation the weight of each model will be 1/9th of the total, and the two options will identical to a 9 prediction stack.
\nHere is the code to reproduce the results for median aggregation.
\n itertools\n numpy as np\n\n generate_binary_arrays(length):\n length <= :\n ValueError()\n =\n \n\n = \n = generate_binary_arrays(length)\n = \n = \n\n arr in all_arrays:\n \n = np.median([np.median(arr[:]), np.median(arr[:]), np.median(arr[:])])\n \n = np.median([np.median([arr[], arr[], arr[]]),np.median([arr[], arr[], arr[]]), np.median([arr[], arr[], arr[]])])\n (arr, p1, p2)\n +=\n p1 == p2:\n +=\n(sc, c, sc/c)\n