Hello guys, I was wondering which techniques beside:
\ncan be used in Computer Vision to handle class imbalance.
\nAny other technique I am missing?
", "messages": [ { "id": 2058043, "postDate": "2022-12-07T15:17:52.593Z", "content": "Hello guys, I was wondering which techniques beside:
\ncan be used in Computer Vision to handle class imbalance.
\nAny other technique I am missing?
", "rawMarkdown": "Hello guys, I was wondering which techniques beside:\n- Undersampling majority class.\n- Oversampling minority class.\n- Focal Loss.\n- Weighted Loss.\n- Augmentations\n\ncan be used in Computer Vision to handle class imbalance.\n\nAny other technique I am missing?", "votes": 13 }, { "id": 2059490, "postDate": "2022-12-08T22:45:16.887Z", "content": "You were nice to quote a topic I had on a related topic, thanks for that. In that topic I was advocating that fixing imbalance was not really useful. It was mostly true because the metric at the time was roc-auc, hence it did not depend on prediction calibration.
\nHere the metric depends on calibration (or depends on the threshold you use for binary predictions). Fixing imbalance may therefore be more relevant here than it was when I wrote that topic.
", "rawMarkdown": "You were nice to quote a [topic ](https://www.kaggle.com/competitions/siim-isic-melanoma-classification/discussion/172892)I had on a related topic, thanks for that. In that topic I was advocating that fixing imbalance was not really useful. It was mostly true because the metric at the time was roc-auc, hence it did not depend on prediction calibration.\n\nHere the metric depends on calibration (or depends on the threshold you use for binary predictions). Fixing imbalance may therefore be more relevant here than it was when I wrote that topic.", "votes": 6, "replies": [ { "id": 2059537, "postDate": "2022-12-09T01:58:50.097Z", "content": "Thanks for clarifying on the matter.
", "rawMarkdown": "Thanks for clarifying on the matter." } ] }, { "id": 2059531, "postDate": "2022-12-09T01:33:53.350Z", "content": "Q: \"How to Handle Class Imbalance in Computer Vision?\"
\nA: Do experiments to verify
suggest start training with at least some m pos samples in a batch size of n.
\nin computer visions, the problems are multiple.e.g. if a positive sample is difficult to detect because of \"rare content\", then having balanced \"distribution\" may not help.
\nthere are a couple of issues in this competition:
\n[1] small lesion (maybe only 10x10 pixel in the whole of 1024x1024)
\n[2] \"uncommon lesion\" (may be common to human but uncommon in distribution), you cannot find \"similar neighbors\"
i am not sure what is the problem contribution for now:
\npoor performance is due to
\ne.g. 20% due to class imbalance
\n 10% due to small object
\n 10% due to rare objects, etc ….
another question you should ask is, how to show that the problem is class imbalance?
\ne.g. results improves when you use oversampling minority class
\ne.g. does failure of focal loss indicates that it is not a class imbalance problem?
I agree that one should always try and experiment if an idea boosts the performance but if we know in advance whether that boost is significant or not can save us a lot of time.
\nI tried Focal Loss, it completely changed the model's prediction distribution from an \"exponential\" one to a more \"right-skewed\". It did not improve my performance though.
\nIt appears so far that increasing resolution is very beneficial. But with Kaggle's GPU it's barely impossible to train with 1024.
", "rawMarkdown": "I agree that one should always try and experiment if an idea boosts the performance but if we know in advance whether that boost is significant or not can save us a lot of time.\n\nI tried Focal Loss, it completely changed the model's prediction distribution from an \"exponential\" one to a more \"right-skewed\". It did not improve my performance though.\n\nIt appears so far that increasing resolution is very beneficial. But with Kaggle's GPU it's barely impossible to train with 1024." } ] }, { "id": 2058762, "postDate": "2022-12-08T07:31:14.260Z", "content": "It is also pretty cool to see the inference notebooks shared publicly either explicitly using multi task learning with auxiliary loss or having everything in place to use one 🙂
\nVery suspect 😄 (but points to what people might be using)
", "rawMarkdown": "It is also pretty cool to see the inference notebooks shared publicly either explicitly using multi task learning with auxiliary loss or having everything in place to use one 🙂\n\nVery suspect 😄 (but points to what people might be using)", "votes": 1, "replies": [ { "id": 2059375, "postDate": "2022-12-08T19:23:06.490Z", "content": "Thanks @radek1. I recently read this excellent topic by @cpmpml with an extensive discussion on the matter.
\nTLDR:
\nImportant: I believe that that competition's metric was AUC-ROC, whereas here we have pF1.
\nI would definitely like to hear more grandmasters opinions on this matter.
", "rawMarkdown": "Thanks @radek1. I recently read this excellent [topic](https://www.kaggle.com/competitions/siim-isic-melanoma-classification/discussion/172892) by @cpmpml with an extensive discussion on the matter.\n\nTLDR: \n- some people suggests doing nothing\n- most people agree that oversampling leads to faster model convergence\n- some say that what matters is the **amount** of minority samples not the **ratio** between positive/negative.\n\n**Important:** I believe that that competition's metric was AUC-ROC, whereas here we have pF1. \n\nI would definitely like to hear more grandmasters opinions on this matter.", "votes": 2 } ] }, { "id": 2058760, "postDate": "2022-12-08T07:29:56.047Z", "content": "You were nice to quote a topic I had on a related topic, thanks for that. In that topic I was advocating that fixing imbalance was not really useful. It was mostly true because the metric at the time was roc-auc, hence it did not depend on prediction calibration.
\nHere the metric depends on calibration (or depends on the threshold you use for binary predictions). Fixing imbalance may therefore be more relevant here than it was when I wrote that topic.
", "votes": 6, "replies": [ { "id": 2059537, "author_name": "moth", "author_url": "", "post_date": "2022-12-09T01:58:50.097000", "content": "Thanks for clarifying on the matter.
", "votes": 0, "replies": [] } ] }, { "id": 2059531, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-09T01:33:53.350000", "content": "Q: \"How to Handle Class Imbalance in Computer Vision?\"
\nA: Do experiments to verify
suggest start training with at least some m pos samples in a batch size of n.
\nin computer visions, the problems are multiple.e.g. if a positive sample is difficult to detect because of \"rare content\", then having balanced \"distribution\" may not help.
\nthere are a couple of issues in this competition:
\n[1] small lesion (maybe only 10x10 pixel in the whole of 1024x1024)
\n[2] \"uncommon lesion\" (may be common to human but uncommon in distribution), you cannot find \"similar neighbors\"
i am not sure what is the problem contribution for now:
\npoor performance is due to
\ne.g. 20% due to class imbalance
\n 10% due to small object
\n 10% due to rare objects, etc ….
another question you should ask is, how to show that the problem is class imbalance?
\ne.g. results improves when you use oversampling minority class
\ne.g. does failure of focal loss indicates that it is not a class imbalance problem?
I agree that one should always try and experiment if an idea boosts the performance but if we know in advance whether that boost is significant or not can save us a lot of time.
\nI tried Focal Loss, it completely changed the model's prediction distribution from an \"exponential\" one to a more \"right-skewed\". It did not improve my performance though.
\nIt appears so far that increasing resolution is very beneficial. But with Kaggle's GPU it's barely impossible to train with 1024.
", "votes": 0, "replies": [] } ] }, { "id": 2058762, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-08T07:31:14.260000", "content": "It is also pretty cool to see the inference notebooks shared publicly either explicitly using multi task learning with auxiliary loss or having everything in place to use one 🙂
\nVery suspect 😄 (but points to what people might be using)
", "votes": 1, "replies": [ { "id": 2059375, "author_name": "moth", "author_url": "", "post_date": "2022-12-08T19:23:06.490000", "content": "Thanks @radek1. I recently read this excellent topic by @cpmpml with an extensive discussion on the matter.
\nTLDR:
\nImportant: I believe that that competition's metric was AUC-ROC, whereas here we have pF1.
\nI would definitely like to hear more grandmasters opinions on this matter.
", "votes": 2, "replies": [] } ] }, { "id": 2058760, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-12-08T07:29:56.047000", "content": "