I believe there's an error in the LB metric.
\nAccording to the data description there are 6 labels in the test set: CC, EC, HGSC, LGSC, MC, Other, so if we make a submission just predicting a single one of these classes for the whole test set, our score should be 0.167 (1/6 * (1+0+0+0+0+0)). However, the score given for a single class prediction is 0.14 which is what we would expect for 7 classes, but not 6. I think what may be happening is there's a 7th class in the GT data that's throwing things off. @sohier can you look into this?
I believe there's an error in the LB metric.
\nAccording to the data description there are 6 labels in the test set: CC, EC, HGSC, LGSC, MC, Other, so if we make a submission just predicting a single one of these classes for the whole test set, our score should be 0.167 (1/6 * (1+0+0+0+0+0)). However, the score given for a single class prediction is 0.14 which is what we would expect for 7 classes, but not 6. I think what may be happening is there's a 7th class in the GT data that's throwing things off. @sohier can you look into this?
Thank you for flagging this. There were a few labels where inconsistent capitalization caused them to be treated as a distinct class. I'll fix this now.
", "rawMarkdown": "Thank you for flagging this. There were a few labels where inconsistent capitalization caused them to be treated as a distinct class. I'll fix this now.", "votes": 7 }, { "id": 2482644, "postDate": "2023-10-15T07:47:21.800Z", "content": "Nice catch. It's probably a typo on one of the labels.
", "rawMarkdown": "Nice catch. It's probably a typo on one of the labels.", "votes": 5 }, { "id": 2483015, "postDate": "2023-10-15T12:28:39.423Z", "content": "Maybe this can also explain why the gap between CV and LB is so big.
\n@ashleychow @homesmac @sohier Can you check it?
Maybe the distribution of classes in the test set is not uniform, and the proportion of that single class in the test set is 14%, that's why you're getting a score of 0.14
", "rawMarkdown": "Maybe the distribution of classes in the test set is not uniform, and the proportion of that single class in the test set is 14%, that's why you're getting a score of 0.14", "replies": [ { "id": 2483249, "postDate": "2023-10-15T15:32:58.323Z", "content": "It would be true for regular accuracy. For balanced accuracy, predicting only one class yields 1 / n_class score.
", "rawMarkdown": "It would be true for regular accuracy. For balanced accuracy, predicting only one class yields 1 / n_class score.", "votes": 8 } ] } ], "comments": [ { "id": 2487662, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-10-18T17:47:37.187000", "content": "Thank you for flagging this. There were a few labels where inconsistent capitalization caused them to be treated as a distinct class. I'll fix this now.
", "votes": 7, "replies": [] }, { "id": 2482644, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2023-10-15T07:47:21.800000", "content": "Nice catch. It's probably a typo on one of the labels.
", "votes": 5, "replies": [] }, { "id": 2483015, "author_name": "Johnny Lee", "author_url": "", "post_date": "2023-10-15T12:28:39.423000", "content": "Maybe this can also explain why the gap between CV and LB is so big.
\n@ashleychow @homesmac @sohier Can you check it?
Maybe the distribution of classes in the test set is not uniform, and the proportion of that single class in the test set is 14%, that's why you're getting a score of 0.14
", "votes": 0, "replies": [ { "id": 2483249, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2023-10-15T15:32:58.323000", "content": "It would be true for regular accuracy. For balanced accuracy, predicting only one class yields 1 / n_class score.
", "votes": 8, "replies": [] } ] } ], "raw_markdown_by_id": { "2479849": "I believe there's an error in the LB metric. \nAccording to the data description there are 6 labels in the test set: `CC, EC, HGSC, LGSC, MC, Other`, so if we make a submission just predicting a single one of these classes for the whole test set, our score should be 0.167 (1/6 * (1+0+0+0+0+0)). However, the score given for a single class prediction is 0.14 which is what we would expect for 7 classes, but not 6. I think what may be happening is there's a 7th class in the GT data that's throwing things off. @sohier can you look into this?", "2487662": "Thank you for flagging this. There were a few labels where inconsistent capitalization caused them to be treated as a distinct class. I'll fix this now.", "2482644": "Nice catch. It's probably a typo on one of the labels.", "2483015": "Maybe this can also explain why the gap between CV and LB is so big. \n@ashleychow @homesmac @sohier Can you check it?", "2483246": "Maybe the distribution of classes in the test set is not uniform, and the proportion of that single class in the test set is 14%, that's why you're getting a score of 0.14" } }