{ "id": 155528, "title": "NaN QK values", "url": "/competitions/prostate-cancer-grade-assessment/discussion/155528", "author_name": "Jaideep", "post_date": "2020-06-02T03:52:41.802000", "votes": 1, "comment_count": 9, "views": 0, "content": "

I get NaN qk values during training on val set. \nCan any one help here why should that be so. \nI dont get same when i call qk function offline that is get the prediction on val set and then compute qk by passing over actuals and predictions.\n\ndef qk(y_pred, y):\n #print(y_pred.size(),y.size())\n return torch.tensor(cohen_kappa_score(torch.round(y_pred.squeeze(-1)), y, weights='quadratic'), device='cuda:0')\n

", "messages": [ { "id": 870943, "postDate": "2020-06-02T03:52:41.803Z", "content": "

I get NaN qk values during training on val set. \nCan any one help here why should that be so. \nI dont get same when i call qk function offline that is get the prediction on val set and then compute qk by passing over actuals and predictions.\n\ndef qk(y_pred, y):\n #print(y_pred.size(),y.size())\n return torch.tensor(cohen_kappa_score(torch.round(y_pred.squeeze(-1)), y, weights='quadratic'), device='cuda:0')\n

", "rawMarkdown": "I get NaN qk values during training on val set. \nCan any one help here why should that be so. \nI dont get same when i call qk function offline that is get the prediction on val set and then compute qk by passing over actuals and predictions.\n```\ndef qk(y_pred, y):\n #print(y_pred.size(),y.size())\n return torch.tensor(cohen_kappa_score(torch.round(y_pred.squeeze(-1)), y, weights='quadratic'), device='cuda:0')\n```", "votes": 1 }, { "id": 872149, "postDate": "2020-06-03T00:20:50.033Z", "content": "

QWK score becomes NaN if any of the predictions consists NaN values.

", "rawMarkdown": "QWK score becomes NaN if any of the predictions consists NaN values." }, { "id": 872056, "postDate": "2020-06-02T21:24:34.127Z", "content": "

\ndef quadratic_kappa(y_hat, y):\n return torch.tensor(cohen_kappa_score(torch.round(y_hat).cpu(), y.cpu(), weights='quadratic'),device='cuda:0')\n

\n\n

The issue might be that your are doing it on GPU i had this problem just transfer your tensors to CPU and put back to GPU. Hopefully this will help...

", "rawMarkdown": "```\ndef quadratic_kappa(y_hat, y):\n return torch.tensor(cohen_kappa_score(torch.round(y_hat).cpu(), y.cpu(), weights='quadratic'),device='cuda:0')\n```\n\nThe issue might be that your are doing it on `GPU` i had this problem just transfer your tensors to `CPU` and put back to `GPU`. Hopefully this will help...", "replies": [ { "id": 872147, "postDate": "2020-06-03T00:20:08.693Z", "content": "

Not possible. I have answered the reason already

", "rawMarkdown": "Not possible. I have answered the reason already", "votes": 1 } ] }, { "id": 871534, "postDate": "2020-06-02T12:44:33.517Z", "content": "

I also have the same question. Can the metric kappa be used normally?

", "rawMarkdown": "I also have the same question. Can the metric kappa be used normally?" }, { "id": 870946, "postDate": "2020-06-02T03:54:34.980Z", "content": "

| Train Val loss Qk\n 0.568884 0.523514 0.844231 10:05\n9 0.611826 0.539616 0.837371 10:02\n10 0.637843 0.527465 nan 10:04\n11 0.672980 0.531039 nan 10:03\n12 0.520446 0.520810 0.840317 10:02\n13 0.603619 0.519214 nan \n14 0.585406 0.518505 nan 10:05\n15 0.513156 0.530183 nan 10:05\n16 0.556478 0.514124 nan 10:07\n17 0.600684 0.524000 nan 10:12\n | |\n| --- | --- |\n| | |

", "rawMarkdown": "| ``` Train Val loss Qk\n 0.568884\t0.523514\t0.844231\t \t 10:05\n9\t0.611826\t0.539616\t0.837371 10:02\n10\t0.637843\t0.527465\tnan \t 10:04\n11\t0.672980\t0.531039\tnan\t \t 10:03\n12\t0.520446\t0.520810\t0.840317\t \t10:02\n13\t0.603619\t0.519214\tnan\t\n14\t0.585406\t0.518505\tnan\t 10:05\n15\t0.513156\t0.530183\tnan\t \t 10:05\n16\t0.556478\t0.514124\tnan\t 10:07\n17\t0.600684\t0.524000\tnan\t 10:12\n``` | |\n| --- | --- |\n| | |", "replies": [ { "id": 871154, "postDate": "2020-06-02T07:10:58.697Z", "content": "

I think what is happening is that the metric function you have, quadratic kappa from sklearn, will return nan if the predictions perfectly match with the labels. In one batch, if your model predicts that batch perfectly, you will get a nan value. Since the metric being displayed is cumulative, every batch is being added to that score. Since you get nan for one batch, I think it ruins the entire epoch since adding anything to nan will still return nan. Correct me if I am wrong on that.

", "rawMarkdown": "I think what is happening is that the metric function you have, quadratic kappa from sklearn, will return nan if the predictions perfectly match with the labels. In one batch, if your model predicts that batch perfectly, you will get a nan value. Since the metric being displayed is cumulative, every batch is being added to that score. Since you get nan for one batch, I think it ruins the entire epoch since adding anything to nan will still return nan. Correct me if I am wrong on that.", "votes": -1 }, { "id": 871332, "postDate": "2020-06-02T09:17:56.420Z", "content": "

hmm then its strange issue,i read similar response else where also. Why should metric prevent you from predicting 100 pct correct. \nAny way to overcome issue ? suppose if i get NaN can we replace that batch qk with 1 ?

", "rawMarkdown": "hmm then its strange issue,i read similar response else where also. Why should metric prevent you from predicting 100 pct correct. \nAny way to overcome issue ? suppose if i get NaN can we replace that batch qk with 1 ?" }, { "id": 871454, "postDate": "2020-06-02T11:27:39.370Z", "content": "

i rectified the issue . it was because the last batch had just 2 items and were in full agreement with labels 0,0 . This peculiar bug happens in case of 0 only it could be causing weights to be zero. This is some sckit should resolve the issue. it should return 1. in case of such complete agreement.

", "rawMarkdown": "i rectified the issue . it was because the last batch had just 2 items and were in full agreement with labels 0,0 . This peculiar bug happens in case of 0 only it could be causing weights to be zero. This is some sckit should resolve the issue. it should return 1. in case of such complete agreement. \n" } ] }, { "id": 872201, "postDate": "2020-06-03T01:50:45.247Z", "content": "

I got this problem too but solved. When there is only 1 label, the QWK would be NaN. For example, cohen_kappa_score([1], [1], weights='quadratic') . This would happen in the end of dataloader.\nYou can use DataLoader(drop_last=True) or justnp.concatenate all preds and true labels, to calculate QWK in the end.

", "rawMarkdown": "I got this problem too but solved. When there is only 1 label, the QWK would be NaN. For example, `cohen_kappa_score([1], [1], weights='quadratic')` . This would happen in the end of dataloader.\nYou can use `DataLoader(drop_last=True)` or just`np.concatenate` all preds and true labels, to calculate QWK in the end. ", "isDeleted": true } ], "comments": [ { "id": 872149, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2020-06-03T00:20:50.033000", "content": "

QWK score becomes NaN if any of the predictions consists NaN values.

", "votes": 0, "replies": [] }, { "id": 872056, "author_name": "DrHB", "author_url": "", "post_date": "2020-06-02T21:24:34.127000", "content": "

\ndef quadratic_kappa(y_hat, y):\n return torch.tensor(cohen_kappa_score(torch.round(y_hat).cpu(), y.cpu(), weights='quadratic'),device='cuda:0')\n

\n\n

The issue might be that your are doing it on GPU i had this problem just transfer your tensors to CPU and put back to GPU. Hopefully this will help...

", "votes": 0, "replies": [ { "id": 872147, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2020-06-03T00:20:08.693000", "content": "

Not possible. I have answered the reason already

", "votes": 1, "replies": [] } ] }, { "id": 871534, "author_name": "zhangeng", "author_url": "", "post_date": "2020-06-02T12:44:33.517000", "content": "

I also have the same question. Can the metric kappa be used normally?

", "votes": 0, "replies": [] }, { "id": 870946, "author_name": "Jaideep", "author_url": "", "post_date": "2020-06-02T03:54:34.980000", "content": "

| Train Val loss Qk\n 0.568884 0.523514 0.844231 10:05\n9 0.611826 0.539616 0.837371 10:02\n10 0.637843 0.527465 nan 10:04\n11 0.672980 0.531039 nan 10:03\n12 0.520446 0.520810 0.840317 10:02\n13 0.603619 0.519214 nan \n14 0.585406 0.518505 nan 10:05\n15 0.513156 0.530183 nan 10:05\n16 0.556478 0.514124 nan 10:07\n17 0.600684 0.524000 nan 10:12\n | |\n| --- | --- |\n| | |

", "votes": 0, "replies": [ { "id": 871154, "author_name": "Richard Xiao", "author_url": "", "post_date": "2020-06-02T07:10:58.697000", "content": "

I think what is happening is that the metric function you have, quadratic kappa from sklearn, will return nan if the predictions perfectly match with the labels. In one batch, if your model predicts that batch perfectly, you will get a nan value. Since the metric being displayed is cumulative, every batch is being added to that score. Since you get nan for one batch, I think it ruins the entire epoch since adding anything to nan will still return nan. Correct me if I am wrong on that.

", "votes": -1, "replies": [] }, { "id": 871332, "author_name": "Jaideep", "author_url": "", "post_date": "2020-06-02T09:17:56.420000", "content": "

hmm then its strange issue,i read similar response else where also. Why should metric prevent you from predicting 100 pct correct. \nAny way to overcome issue ? suppose if i get NaN can we replace that batch qk with 1 ?

", "votes": 0, "replies": [] }, { "id": 871454, "author_name": "Jaideep", "author_url": "", "post_date": "2020-06-02T11:27:39.370000", "content": "

i rectified the issue . it was because the last batch had just 2 items and were in full agreement with labels 0,0 . This peculiar bug happens in case of 0 only it could be causing weights to be zero. This is some sckit should resolve the issue. it should return 1. in case of such complete agreement.

", "votes": 0, "replies": [] } ] }, { "id": 872201, "author_name": "", "author_url": "", "post_date": "2020-06-03T01:50:45.247000", "content": "

I got this problem too but solved. When there is only 1 label, the QWK would be NaN. For example, cohen_kappa_score([1], [1], weights='quadratic') . This would happen in the end of dataloader.\nYou can use DataLoader(drop_last=True) or justnp.concatenate all preds and true labels, to calculate QWK in the end.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "870943": "I get NaN qk values during training on val set. \nCan any one help here why should that be so. \nI dont get same when i call qk function offline that is get the prediction on val set and then compute qk by passing over actuals and predictions.\n```\ndef qk(y_pred, y):\n #print(y_pred.size(),y.size())\n return torch.tensor(cohen_kappa_score(torch.round(y_pred.squeeze(-1)), y, weights='quadratic'), device='cuda:0')\n```", "872149": "QWK score becomes NaN if any of the predictions consists NaN values.", "872056": "```\ndef quadratic_kappa(y_hat, y):\n return torch.tensor(cohen_kappa_score(torch.round(y_hat).cpu(), y.cpu(), weights='quadratic'),device='cuda:0')\n```\n\nThe issue might be that your are doing it on `GPU` i had this problem just transfer your tensors to `CPU` and put back to `GPU`. Hopefully this will help...", "871534": "I also have the same question. Can the metric kappa be used normally?", "870946": "| ``` Train Val loss Qk\n 0.568884\t0.523514\t0.844231\t \t 10:05\n9\t0.611826\t0.539616\t0.837371 10:02\n10\t0.637843\t0.527465\tnan \t 10:04\n11\t0.672980\t0.531039\tnan\t \t 10:03\n12\t0.520446\t0.520810\t0.840317\t \t10:02\n13\t0.603619\t0.519214\tnan\t\n14\t0.585406\t0.518505\tnan\t 10:05\n15\t0.513156\t0.530183\tnan\t \t 10:05\n16\t0.556478\t0.514124\tnan\t 10:07\n17\t0.600684\t0.524000\tnan\t 10:12\n``` | |\n| --- | --- |\n| | |", "872201": "I got this problem too but solved. When there is only 1 label, the QWK would be NaN. For example, `cohen_kappa_score([1], [1], weights='quadratic')` . This would happen in the end of dataloader.\nYou can use `DataLoader(drop_last=True)` or just`np.concatenate` all preds and true labels, to calculate QWK in the end. " } }