Does anyone else notice this? My best scoring checkpoints are very rarely have the best loss. I guess we can explain that by the nature of an unstable metric and use the best scoring checkpoints.
", "messages": [ { "id": 2572332, "postDate": "2023-12-24T06:12:37.603Z", "content": "Does anyone else notice this? My best scoring checkpoints are very rarely have the best loss. I guess we can explain that by the nature of an unstable metric and use the best scoring checkpoints.
", "rawMarkdown": "Does anyone else notice this? My best scoring checkpoints are very rarely have the best loss. I guess we can explain that by the nature of an unstable metric and use the best scoring checkpoints.", "votes": 11 }, { "id": 2580897, "postDate": "2023-12-31T10:57:47.837Z", "content": "I solved this by increasing data points in validation sets. Epochs with best validation loss also have the best balanced accuracy now.
", "rawMarkdown": "I solved this by increasing data points in validation sets. Epochs with best validation loss also have the best balanced accuracy now. ", "votes": 5 }, { "id": 2573205, "postDate": "2023-12-24T19:30:58.177Z", "content": "It is hard to believe the organisers spent valuable cancer research money for organising competition with improper scoring rule as a metric. Metrics such as 'balanced accuracy' are not proper scoring rules and don't lead models to well calibrated predictions! The results of putting such models into real life setting will result in incorrect probabilities of classifying cancer!
", "rawMarkdown": "It is hard to believe the organisers spent valuable cancer research money for organising competition with improper scoring rule as a metric. Metrics such as 'balanced accuracy' are not proper scoring rules and don't lead models to well calibrated predictions! The results of putting such models into real life setting will result in incorrect probabilities of classifying cancer!", "votes": 4, "replies": [ { "id": 2575138, "postDate": "2023-12-26T15:55:35.123Z", "content": "Metrics like macro F1 score would be unstable like this one. What do you suggest? Log loss?
", "rawMarkdown": "Metrics like macro F1 score would be unstable like this one. What do you suggest? Log loss?", "votes": 1, "replies": [ { "id": 2575211, "postDate": "2023-12-26T17:08:17.673Z", "content": "Dear @gunesevitan
\nThe log-loss would be an obvious choice as this is what most classifiers minimize and constitutes a strictly proper scoring rule, as does the multi-class Brier score. Indeed the problem of correctly assessing class predictions goes back well over 70 years and was effectively addressed by Brier in the paper Verification of Forecasts Expressed in Terms of Probability in 1950 and is well worth reading.
\nAll the best,
\ncarl
LOL; yet another classification competition that does not use a proper scoring rule….
\nAll the best,
\ncarl
Hi @gunesevitan , I think if you set a proper weight of the loss, they will have better correlation.
", "rawMarkdown": "Hi @gunesevitan , I think if you set a proper weight of the loss, they will have better correlation.", "votes": 1, "replies": [ { "id": 2572484, "postDate": "2023-12-24T09:41:32.117Z", "content": "I tried weighted sampling but it didn't work. I'll try weighted loss too but based on my understanding of the metric, isn't each sample should have equal weights?
", "rawMarkdown": "I tried weighted sampling but it didn't work. I'll try weighted loss too but based on my understanding of the metric, isn't each sample should have equal weights?", "replies": [ { "id": 2572664, "postDate": "2023-12-24T12:50:03.787Z", "content": "I think weight loss correlates more with bacc. For example, given there are 200 HGSC and 40 LGSC, when you use mean loss, if you make 1 more correct predictions for LGSC and 3 more wrong predictions for HGSC, the mean loss is worse while bacc is better.
I solved this by increasing data points in validation sets. Epochs with best validation loss also have the best balanced accuracy now.
", "votes": 5, "replies": [] }, { "id": 2573205, "author_name": "predict_addict", "author_url": "", "post_date": "2023-12-24T19:30:58.177000", "content": "It is hard to believe the organisers spent valuable cancer research money for organising competition with improper scoring rule as a metric. Metrics such as 'balanced accuracy' are not proper scoring rules and don't lead models to well calibrated predictions! The results of putting such models into real life setting will result in incorrect probabilities of classifying cancer!
", "votes": 4, "replies": [ { "id": 2575138, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2023-12-26T15:55:35.123000", "content": "Metrics like macro F1 score would be unstable like this one. What do you suggest? Log loss?
", "votes": 1, "replies": [ { "id": 2575211, "author_name": "Carl McBride Ellis", "author_url": "", "post_date": "2023-12-26T17:08:17.673000", "content": "Dear @gunesevitan
\nThe log-loss would be an obvious choice as this is what most classifiers minimize and constitutes a strictly proper scoring rule, as does the multi-class Brier score. Indeed the problem of correctly assessing class predictions goes back well over 70 years and was effectively addressed by Brier in the paper Verification of Forecasts Expressed in Terms of Probability in 1950 and is well worth reading.
\nAll the best,
\ncarl
LOL; yet another classification competition that does not use a proper scoring rule….
\nAll the best,
\ncarl
Hi @gunesevitan , I think if you set a proper weight of the loss, they will have better correlation.
", "votes": 1, "replies": [ { "id": 2572484, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2023-12-24T09:41:32.117000", "content": "I tried weighted sampling but it didn't work. I'll try weighted loss too but based on my understanding of the metric, isn't each sample should have equal weights?
", "votes": 0, "replies": [ { "id": 2572664, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-12-24T12:50:03.787000", "content": "I think weight loss correlates more with bacc. For example, given there are 200 HGSC and 40 LGSC, when you use mean loss, if you make 1 more correct predictions for LGSC and 3 more wrong predictions for HGSC, the mean loss is worse while bacc is better.