I believe there's a bug in Kaggle's implementation of the pF score here
\nAccording to equation (6) in the paper, cFP should only be computed when the GT is False. So this line is unnecessary and should be removed: cfp += 1 - prediction
As a sanity check, pF1 score is a generalization of F1 score, and F1 score is a special case of pF1 score when all the predictions are binary. But this is not the case in current implementation. I've illustrated it here
\nI already commented under the metric notebook, but not sure if anyone have seen it.
\n@sohier @maggiemd Could you take a look? Thanks.
", "messages": [ { "id": 2052040, "postDate": "2022-12-01T20:53:05.310Z", "content": "I believe there's a bug in Kaggle's implementation of the pF score here
\nAccording to equation (6) in the paper, cFP should only be computed when the GT is False. So this line is unnecessary and should be removed: cfp += 1 - prediction
As a sanity check, pF1 score is a generalization of F1 score, and F1 score is a special case of pF1 score when all the predictions are binary. But this is not the case in current implementation. I've illustrated it here
\nI already commented under the metric notebook, but not sure if anyone have seen it.
\n@sohier @maggiemd Could you take a look? Thanks.
", "rawMarkdown": "I believe there's a bug in Kaggle's implementation of the pF score [here](https://www.kaggle.com/code/sohier/probabilistic-f-score/notebook)\n\nAccording to equation (6) in the [paper](https://aclanthology.org/2020.eval4nlp-1.9.pdf), cFP should only be computed when the GT is False. So this line is unnecessary and should be removed: `cfp += 1 - prediction`\n\nAs a sanity check, pF1 score is a generalization of F1 score, and F1 score is a special case of pF1 score when all the predictions are binary. But this is not the case in current implementation. I've illustrated it [here](https://www.kaggle.com/code/boliu0/probabilistic-f-score-bug/notebook)\n\n\nI already commented under the metric notebook, but not sure if anyone have seen it. \n\n@sohier @maggiemd Could you take a look? Thanks. ", "votes": 15 }, { "id": 2052299, "postDate": "2022-12-02T04:33:23.173Z", "content": "The issue should be resolved now: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369921
", "rawMarkdown": "The issue should be resolved now: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369921", "replies": [ { "id": 2052301, "postDate": "2022-12-02T04:35:42.867Z", "content": "Thanks for fixing it so quickly
", "rawMarkdown": "Thanks for fixing it so quickly" } ] } ], "comments": [ { "id": 2052299, "author_name": "Sohier Dane", "author_url": "", "post_date": "2022-12-02T04:33:23.173000", "content": "The issue should be resolved now: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369921
", "votes": 0, "replies": [ { "id": 2052301, "author_name": "Bo", "author_url": "", "post_date": "2022-12-02T04:35:42.867000", "content": "Thanks for fixing it so quickly
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2052040": "I believe there's a bug in Kaggle's implementation of the pF score [here](https://www.kaggle.com/code/sohier/probabilistic-f-score/notebook)\n\nAccording to equation (6) in the [paper](https://aclanthology.org/2020.eval4nlp-1.9.pdf), cFP should only be computed when the GT is False. So this line is unnecessary and should be removed: `cfp += 1 - prediction`\n\nAs a sanity check, pF1 score is a generalization of F1 score, and F1 score is a special case of pF1 score when all the predictions are binary. But this is not the case in current implementation. I've illustrated it [here](https://www.kaggle.com/code/boliu0/probabilistic-f-score-bug/notebook)\n\n\nI already commented under the metric notebook, but not sure if anyone have seen it. \n\n@sohier @maggiemd Could you take a look? Thanks. ", "2052299": "The issue should be resolved now: https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369921" } }