{ "id": 459363, "title": "Balanced Accuracy Pytorch Implementation", "url": "/competitions/UBC-OCEAN/discussion/459363", "author_name": "SSS", "post_date": "2023-12-04T21:32:29.447000", "votes": 18, "comment_count": 5, "views": 0, "content": "

Hi,

\n

It's been a while since the competition started. I've not found the competition metric PyTorch implementation elsewhere, so I have rewritten the sklearn one.
\nLet me know if you find any bugs and good luck in the competition.

\n
 torch\n torchmetrics.classification  MulticlassConfusionMatrix\n warnings\n\n\n () -> torch.:\n    \n\n    mcm = MulticlassConfusionMatrix(num_classes=)\n    C = mcm(y_pred, y_true)\n    per_class = torch.diag(C) / C.(axis=)\n     torch.(torch.isnan(C)):\n        warnings.warn()\n        per_class = per_class[~torch.isnan(per_class)]\n    score = per_class.mean()\n\n     adjusted:\n        n_classes = (per_class)\n        chance =  / n_classes\n        score -= chance\n        score /=  - chance\n     score\n
", "messages": [ { "id": 2548943, "postDate": "2023-12-04T21:32:29.447Z", "content": "

Hi,

\n

It's been a while since the competition started. I've not found the competition metric PyTorch implementation elsewhere, so I have rewritten the sklearn one.
\nLet me know if you find any bugs and good luck in the competition.

\n
 torch\n torchmetrics.classification  MulticlassConfusionMatrix\n warnings\n\n\n () -> torch.:\n    \n\n    mcm = MulticlassConfusionMatrix(num_classes=)\n    C = mcm(y_pred, y_true)\n    per_class = torch.diag(C) / C.(axis=)\n     torch.(torch.isnan(C)):\n        warnings.warn()\n        per_class = per_class[~torch.isnan(per_class)]\n    score = per_class.mean()\n\n     adjusted:\n        n_classes = (per_class)\n        chance =  / n_classes\n        score -= chance\n        score /=  - chance\n     score\n
", "rawMarkdown": "Hi,\n\nIt's been a while since the competition started. I've not found the competition metric PyTorch implementation elsewhere, so I have rewritten the sklearn one.\nLet me know if you find any bugs and good luck in the competition.\n\n```python\nimport torch\nfrom torchmetrics.classification import MulticlassConfusionMatrix\nimport warnings\n\n\ndef torch_balanced_accuracy(\n y_true: torch.tensor,\n y_pred: torch.tensor,\n adjusted: bool = False\n) -> torch.float:\n \"\"\"Compute the balanced accuracy.\n The balanced accuracy in binary and multiclass classification problems to\n deal with imbalanced datasets. It is defined as the average of recall\n obtained on each class.\n\n Args:\n y_true : torch tensor of shape (n_samples,). Ground truth (correct) target values.\n y_pred : torch tensor of shape (n_samples,). Estimated targets as returned by a classifier.\n adjusted: default=False, When true, the result is adjusted for chance, so that random\n performance would score 0, while keeping perfect performance at a score of 1.\n\n Return\n balanced_accuracy: Balanced accuracy score.\n \"\"\"\n\n mcm = MulticlassConfusionMatrix(num_classes=5)\n C = mcm(y_pred, y_true)\n per_class = torch.diag(C) / C.sum(axis=1)\n if torch.any(torch.isnan(C)):\n warnings.warn(\"y_pred contains classes not in y_true\")\n per_class = per_class[~torch.isnan(per_class)]\n score = per_class.mean()\n\n if adjusted:\n n_classes = len(per_class)\n chance = 1 / n_classes\n score -= chance\n score /= 1 - chance\n return score\n```", "votes": 18 }, { "id": 2557295, "postDate": "2023-12-11T11:38:13.310Z", "content": "

You can use the Accuracy from Pytorch-Metric.
\nBy setting the average to \"macro\" it is equivalent to the balanced accuracy from sklearn:

\n
 sklearn.metrics import balanced_accuracy_score\n torchmetrics import Accuracy\n\n = np.array([, , , , , , , , , , , , , , , , , ])\n = np.array([, , , , , , , , , , , , , , , , , ])\n(balanced_accuracy_score(y_true, y_pred))\n\n = Accuracy(task=, num_classes=, average=)\n(accMetric(torch.Tensor(y_pred), torch.Tensor(y_true)))\n\n = Accuracy(task=, num_classes=, average=)\n(accMetric(torch.Tensor(y_pred), torch.Tensor(y_true)))\n\n\n\n\n\n\n
", "rawMarkdown": "You can use the [Accuracy from Pytorch-Metric](https://lightning.ai/docs/torchmetrics/stable/classification/accuracy.html).\nBy setting the average to \"macro\" it is equivalent to the balanced accuracy from sklearn:\n\n```\nfrom sklearn.metrics import balanced_accuracy_score\nfrom torchmetrics import Accuracy\n\ny_true = np.array([0, 1, 2, 0, 2, 0, 0, 0, 1, 2, 2, 1, 0, 2, 1, 1, 1, 0])\ny_pred = np.array([0, 1, 2, 0, 0, 1, 0, 0, 1, 2, 2, 1, 0, 1, 1, 0, 1, 0])\nprint(balanced_accuracy_score(y_true, y_pred))\n\naccMetric = Accuracy(task=\"multiclass\", num_classes=3, average=\"micro\")\nprint(accMetric(torch.Tensor(y_pred), torch.Tensor(y_true)))\n\naccMetric = Accuracy(task=\"multiclass\", num_classes=3, average=\"macro\")\nprint(accMetric(torch.Tensor(y_pred), torch.Tensor(y_true)))\n\n\n#Output:\n#0.7634920634920634\n#tensor(0.7778)\n#tensor(0.7635)\n```", "votes": 2, "replies": [ { "id": 2557359, "postDate": "2023-12-11T12:43:52.257Z", "content": "

@manuelkrmer,

\n

That's a good one. Though, the torchmetrics implementation looks different.

\n

All 3 coverge perfectly:

\n

\"\"

", "rawMarkdown": "@manuelkrmer,\n\nThat's a good one. Though, the torchmetrics implementation looks different.\n\nAll 3 coverge perfectly:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F4142d02fe56e15589972141d1903bf87%2FScreenshot%20from%202023-12-11%2008-53-22.png?generation=1702302821225687&alt=media)", "votes": 1, "replies": [ { "id": 2557453, "postDate": "2023-12-11T13:52:16.913Z", "content": "

You are right. The balance accuracy is actually the average of recall values for each class. While the \"macro accuracy\" will calculate the accuracy for every class with a one-vs-rest approach. Should be something like this

\n

$$Recall_c = \\frac{\\text{# correctly classified instances of class c}}{\\text{# all instances of class c}}$$

\n

$$Accuracy_c = \\frac{\\text{# correctly classified instances of class c} + \\text{# correctly classified instances of all other classes}}{\\text{# all instances}}$$

\n

In the end, I guess both metrices will give you a good estimate of the model performance (without the influence of class imbalance). But the former is used for scoring so I will change my implementation. Thanks for pointing it out!

", "rawMarkdown": "You are right. The balance accuracy is actually the average of recall values for each class. While the \"macro accuracy\" will calculate the accuracy for every class with a one-vs-rest approach. Should be something like this\n\n$$Recall_c = \\frac{\\text{# correctly classified instances of class c}}{\\text{# all instances of class c}}$$\n\n$$Accuracy_c = \\frac{\\text{# correctly classified instances of class c} + \\text{# correctly classified instances of all other classes}}{\\text{# all instances}}$$\n\nIn the end, I guess both metrices will give you a good estimate of the model performance (without the influence of class imbalance). But the former is used for scoring so I will change my implementation. Thanks for pointing it out!", "votes": 2 } ] } ] }, { "id": 2555968, "postDate": "2023-12-10T10:52:05.060Z", "content": "

thanks for sharing,

\n

I'm using following from sklearn

\n
 sklearn.metrics  balanced_accuracy_score\nbalanced_accuracy_score(labels, predictions)\n
\n

Interested to know what are differences.

", "rawMarkdown": "thanks for sharing, \n\nI'm using following from sklearn\n\n```python\nfrom sklearn.metrics import balanced_accuracy_score\nbalanced_accuracy_score(labels, predictions)\n```\n\nInterested to know what are differences.", "votes": 2, "replies": [ { "id": 2557486, "postDate": "2023-12-11T14:09:28.973Z", "content": "

@aliabbasi,

\n

Well, the difference is in the implementation.
\nThough, what you really mean is: \"why should you use sklearn, rather than Torch one?\"
\nThe answer is: sklearn uses numpy.ndarray or list as an input, so you are to convert torch.tensor during the training.

\n

You can either work with torch.tensor directly or do .numpy() operation (which is rather cheap). Though, you need to use .detach() every time you do this.

\n

Now you can research why sometimes you want to avoid things like .item(), .numpy(), .cpu() during the training.

", "rawMarkdown": "@aliabbasi, \n\nWell, the difference is in the implementation. \nThough, what you really mean is: \"why should you use sklearn, rather than Torch one?\"\nThe answer is: sklearn uses numpy.ndarray or list as an input, so you are to convert torch.tensor during the training. \n\nYou can either work with torch.tensor directly or do `.numpy()` operation (which is rather cheap). Though, you need to use `.detach()` every time you do this.\n\nNow you can research why sometimes you want to avoid things like `.item()`, `.numpy()`, `.cpu()` during the training. ", "votes": 3 } ] } ], "comments": [ { "id": 2557295, "author_name": "Manuel K", "author_url": "", "post_date": "2023-12-11T11:38:13.310000", "content": "

You can use the Accuracy from Pytorch-Metric.
\nBy setting the average to \"macro\" it is equivalent to the balanced accuracy from sklearn:

\n
 sklearn.metrics import balanced_accuracy_score\n torchmetrics import Accuracy\n\n = np.array([, , , , , , , , , , , , , , , , , ])\n = np.array([, , , , , , , , , , , , , , , , , ])\n(balanced_accuracy_score(y_true, y_pred))\n\n = Accuracy(task=, num_classes=, average=)\n(accMetric(torch.Tensor(y_pred), torch.Tensor(y_true)))\n\n = Accuracy(task=, num_classes=, average=)\n(accMetric(torch.Tensor(y_pred), torch.Tensor(y_true)))\n\n\n\n\n\n\n
", "votes": 2, "replies": [ { "id": 2557359, "author_name": "SSS", "author_url": "", "post_date": "2023-12-11T12:43:52.257000", "content": "

@manuelkrmer,

\n

That's a good one. Though, the torchmetrics implementation looks different.

\n

All 3 coverge perfectly:

\n

\"\"

", "votes": 1, "replies": [ { "id": 2557453, "author_name": "Manuel K", "author_url": "", "post_date": "2023-12-11T13:52:16.913000", "content": "

You are right. The balance accuracy is actually the average of recall values for each class. While the \"macro accuracy\" will calculate the accuracy for every class with a one-vs-rest approach. Should be something like this

\n

$$Recall_c = \\frac{\\text{# correctly classified instances of class c}}{\\text{# all instances of class c}}$$

\n

$$Accuracy_c = \\frac{\\text{# correctly classified instances of class c} + \\text{# correctly classified instances of all other classes}}{\\text{# all instances}}$$

\n

In the end, I guess both metrices will give you a good estimate of the model performance (without the influence of class imbalance). But the former is used for scoring so I will change my implementation. Thanks for pointing it out!

", "votes": 2, "replies": [] } ] } ] }, { "id": 2555968, "author_name": "Ali", "author_url": "", "post_date": "2023-12-10T10:52:05.060000", "content": "

thanks for sharing,

\n

I'm using following from sklearn

\n
 sklearn.metrics  balanced_accuracy_score\nbalanced_accuracy_score(labels, predictions)\n
\n

Interested to know what are differences.

", "votes": 2, "replies": [ { "id": 2557486, "author_name": "SSS", "author_url": "", "post_date": "2023-12-11T14:09:28.973000", "content": "

@aliabbasi,

\n

Well, the difference is in the implementation.
\nThough, what you really mean is: \"why should you use sklearn, rather than Torch one?\"
\nThe answer is: sklearn uses numpy.ndarray or list as an input, so you are to convert torch.tensor during the training.

\n

You can either work with torch.tensor directly or do .numpy() operation (which is rather cheap). Though, you need to use .detach() every time you do this.

\n

Now you can research why sometimes you want to avoid things like .item(), .numpy(), .cpu() during the training.

", "votes": 3, "replies": [] } ] } ], "raw_markdown_by_id": { "2548943": "Hi,\n\nIt's been a while since the competition started. I've not found the competition metric PyTorch implementation elsewhere, so I have rewritten the sklearn one.\nLet me know if you find any bugs and good luck in the competition.\n\n```python\nimport torch\nfrom torchmetrics.classification import MulticlassConfusionMatrix\nimport warnings\n\n\ndef torch_balanced_accuracy(\n y_true: torch.tensor,\n y_pred: torch.tensor,\n adjusted: bool = False\n) -> torch.float:\n \"\"\"Compute the balanced accuracy.\n The balanced accuracy in binary and multiclass classification problems to\n deal with imbalanced datasets. It is defined as the average of recall\n obtained on each class.\n\n Args:\n y_true : torch tensor of shape (n_samples,). Ground truth (correct) target values.\n y_pred : torch tensor of shape (n_samples,). Estimated targets as returned by a classifier.\n adjusted: default=False, When true, the result is adjusted for chance, so that random\n performance would score 0, while keeping perfect performance at a score of 1.\n\n Return\n balanced_accuracy: Balanced accuracy score.\n \"\"\"\n\n mcm = MulticlassConfusionMatrix(num_classes=5)\n C = mcm(y_pred, y_true)\n per_class = torch.diag(C) / C.sum(axis=1)\n if torch.any(torch.isnan(C)):\n warnings.warn(\"y_pred contains classes not in y_true\")\n per_class = per_class[~torch.isnan(per_class)]\n score = per_class.mean()\n\n if adjusted:\n n_classes = len(per_class)\n chance = 1 / n_classes\n score -= chance\n score /= 1 - chance\n return score\n```", "2557295": "You can use the [Accuracy from Pytorch-Metric](https://lightning.ai/docs/torchmetrics/stable/classification/accuracy.html).\nBy setting the average to \"macro\" it is equivalent to the balanced accuracy from sklearn:\n\n```\nfrom sklearn.metrics import balanced_accuracy_score\nfrom torchmetrics import Accuracy\n\ny_true = np.array([0, 1, 2, 0, 2, 0, 0, 0, 1, 2, 2, 1, 0, 2, 1, 1, 1, 0])\ny_pred = np.array([0, 1, 2, 0, 0, 1, 0, 0, 1, 2, 2, 1, 0, 1, 1, 0, 1, 0])\nprint(balanced_accuracy_score(y_true, y_pred))\n\naccMetric = Accuracy(task=\"multiclass\", num_classes=3, average=\"micro\")\nprint(accMetric(torch.Tensor(y_pred), torch.Tensor(y_true)))\n\naccMetric = Accuracy(task=\"multiclass\", num_classes=3, average=\"macro\")\nprint(accMetric(torch.Tensor(y_pred), torch.Tensor(y_true)))\n\n\n#Output:\n#0.7634920634920634\n#tensor(0.7778)\n#tensor(0.7635)\n```", "2555968": "thanks for sharing, \n\nI'm using following from sklearn\n\n```python\nfrom sklearn.metrics import balanced_accuracy_score\nbalanced_accuracy_score(labels, predictions)\n```\n\nInterested to know what are differences." } }