{"cells":[{"metadata":{},"cell_type":"markdown","source":"# QWK metric in PyTorch\nSince the metric is not implemented natively in PyTorch, I decided to implement it by myself. I followed the very good explanation by @reigHns in his [notebook](https://www.kaggle.com/reighns/understanding-the-quadratic-weighted-kappa) and realized it using only pure `torch` functions.\n\nThe function takes as input a `onehot` encoding of the target, while it can take a list of probabilities (apply `softmax` to the network output) or, in case of `binned` labels, the sum of the sigmoid logits from the network (apply `sigmoid` and `sum` to the network output).\n\n## Drawbacks of QWK metric (and loss)\nSince the coefficient is calculated with the outer product of the `outputs` and `targets` histograms, if there is no guess in any class (in any item of batch), a division by `0` occurs, leading the metric (and loss) to `nan`. It is clear that, the more the batch size, the less is the probability of such an event. Higher batch sizes (10, maybe) should be considered; in addition, the last batch of `Dataloader` should be dropped to avoid this problem (flag `drop_last` in the PyTorch `Dataloader` API). In the end, this metric is quite unstable for little batch sizes, so it should be used carefully.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"def quadratic_kappa_coefficient(output, target):\n    n_classes = target.shape[-1]\n    weights = torch.arange(0, n_classes, dtype=torch.float32, device=output.device) / (n_classes - 1)\n    weights = (weights - torch.unsqueeze(weights, -1)) ** 2\n\n    C = (output.t() @ target).t()  # confusion matrix\n\n    hist_true = torch.sum(target, dim=0).unsqueeze(-1)\n    hist_pred = torch.sum(output, dim=0).unsqueeze(-1)\n\n    E = hist_true @ hist_pred.t()  # Outer product of histograms\n    E = E / C.sum() # Normalize to the sum of C.\n\n    num = weights * C\n    den = weights * E\n\n    QWK = 1 - torch.sum(num) / torch.sum(den)\n    return QWK","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Use example from notebook\nReferring to the aforementioned notebook, each tensor is less by 1 because in that case there were 5 classes and it started from 1. To use the `torch` functions we must start from 0.\n\nIn this case, there are 5 classes and 10 items.  (i.e., batch size = 10).","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"target = torch.tensor([2,2,2,3,4,5,5,5,5,5]) - 1\noutput = torch.tensor([2,2,2,3,2,1,1,1,1,3]) - 1\n\noutput.shape, target.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Transform both arrays to `onehot` encoding.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\n\ntarget_onehot = F.one_hot(target, 5)\noutput_onehot = F.one_hot(output, 5)\n\noutput_onehot.shape, target_onehot.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Show same result from the aforementioned notebook. I have already checked the values and dimensions of the `C`, `E` and `weights` matrices, but feel free to check by yourself.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"quadratic_kappa_coefficient(output_onehot.type(torch.float32), target_onehot.type(torch.float32))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Implementation of loss function\nThis method can be easily integrated inside a `torch.nn.Module` to build, for example, the correlated loss.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def quadratic_kappa_loss(output, target, scale=2.0):\n    QWK = quadratic_kappa_coefficient(output, target)\n    loss = -torch.log(torch.sigmoid(scale * QWK))\n    return loss\n\nclass QWKLoss(torch.nn.Module):\n    def __init__(self, scale=2.0):\n        super().__init__()\n        self.scale = scale\n\n    def forward(self, output, target):\n        # Keep trace of output dtype for half precision training\n        target = F.one_hot(target.squeeze(), num_classes=6).to(target.device).type(output.dtype)\n        output = torch.softmax(output, dim=1)\n        return quadratic_kappa_loss(output, target, self.scale)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Implementation of metric function\nMoreover, it can be useful to define the metric function, even with binned labels for network training.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class QWKMetric(torch.nn.Module):\n    def __init__(self, binned=False):\n        super().__init__()\n        self.binned = binned\n\n    def forward(self, output, target):\n        # Keep trace of dtype for half precision training\n        dtype = output.dtype\n        target = F.one_hot(target.squeeze(), num_classes=6).to(target.device).type(dtype)\n        if self.binned:\n            output = torch.sigmoid(output).sum(1).round().long()\n            output = F.one_hot(output.squeeze(), num_classes=6).to(output.device).type(dtype)\n        else:\n            output = torch.softmax(output, dim=1)\n        return quadratic_kappa_coefficient(output, target)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = torch.randint(0, 6, (10, 1)).squeeze()\nprint(\"target: \", target)  # target class coming directly from the isup grades\n\noutput = torch.rand(10, 6)  # Logits from network, trained with not binned target\nprint(\"output: \", output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Binned metric and not binned loss examples","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"nb_loss = QWKLoss()\nb_metric = QWKMetric(binned=True)\nnbl = nb_loss(output, target)\nbl = b_metric(output, target)\nprint(\"not binned loss: \", nbl.item())\nprint(\"binned metric: \", bl.item())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}