Hey guys,\nI'm using Keras and trying to implement Cohen's Kappa Score(sklearn) as the \"metrics\".\nTo do so, I have to convert the tensor to a numpy array before making a computation.
\n\nAfter going over several forums, I think there is no good way of doing that. \n.numpy() does not work and several issues have been raised on github, without coming to a conclusion.\nIf anyone knows how to do this, please leave a comment!
", "messages": [ { "id": 822115, "postDate": "2020-04-26T17:15:25.173Z", "content": "Hello @ibtesama, I would write a callback that does predictions and runs the sklearn.metric after each epoch. Something like:
\n\n```\nclass PredictionCallback(keras.callbacks.Callback):
\n\ndef __init__(self, test_data, batch_size=32)\n self.test_data = test_data\n self.scores = [] \ndef on_epoch_end(self, batch, logs={}):\n test_preds = self.model.predict(self.test_data, batch_size=self.batch_size)\n score = sklearn.metric(self.test_data.labels, test_preds)\n print(score)\n self.scores.append(scores)\ncallback = PredictionCallback(test_data)\nmodel.fit(..., callbacks=[callback])
\n\nscores = callback.scores\n```
\n\nif you wanna obtain all predictions for all epochs, append them to a list (self.test_preds = []) similar to self.scores in above example.
", "rawMarkdown": "Hello @ibtesama, I would write a callback that does predictions and runs the sklearn.metric after each epoch. Something like:\n\n```\nclass PredictionCallback(keras.callbacks.Callback):\n \n def __init__(self, test_data, batch_size=32)\n self.test_data = test_data\n self.scores = [] \n def on_epoch_end(self, batch, logs={}):\n test_preds = self.model.predict(self.test_data, batch_size=self.batch_size)\n score = sklearn.metric(self.test_data.labels, test_preds)\n print(score)\n self.scores.append(scores)\n\ncallback = PredictionCallback(test_data)\nmodel.fit(..., callbacks=[callback])\n\n# obtain all scores\nscores = callback.scores\n```\n\nif you wanna obtain all predictions for all epochs, append them to a list (self.test_preds = []) similar to self.scores in above example.\n", "votes": 3, "replies": [ { "id": 828119, "postDate": "2020-04-30T19:17:42.740Z", "content": "Side note: If you happen to have memory issues with the implementation above or with model.predict in general (I had some memory leakage issues before), try tf-nightly (version 2.2.0-dev) and see if it solves the problem (it did for me). Unfortunately, memory issues have been a recurring problem [for me at least] when using Keras/tf.keras specifically. Interestingly, in the Kaggle kernel, I only have the leakage problem when using use_multiprocessing, not otherwise.
Hey, thanks for the help!
", "rawMarkdown": "Hey, thanks for the help!", "votes": 1 } ] }, { "id": 821752, "postDate": "2020-04-26T11:52:40.467Z", "content": "Hey guys,\nI'm using Keras and trying to implement Cohen's Kappa Score(sklearn) as the \"metrics\".\nTo do so, I have to convert the tensor to a numpy array before making a computation.
\n\nAfter going over several forums, I think there is no good way of doing that. \n.numpy() does not work and several issues have been raised on github, without coming to a conclusion.\nIf anyone knows how to do this, please leave a comment!
", "rawMarkdown": "Hey guys,\nI'm using Keras and trying to implement Cohen's Kappa Score(sklearn) as the \"metrics\".\nTo do so, I have to convert the tensor to a numpy array before making a computation.\n\nAfter going over several forums, I think there is no good way of doing that. \n.numpy() does not work and several issues have been raised on github, without coming to a conclusion.\nIf anyone knows how to do this, please leave a comment!", "votes": -3 }, { "id": 828794, "postDate": "2020-05-01T09:34:07.643Z", "content": "I have the same problem
", "rawMarkdown": "I have the same problem", "replies": [ { "id": 828823, "postDate": "2020-05-01T10:07:36.353Z", "content": "@zhangeng , How did you train your model then?
", "rawMarkdown": "@zhangeng , How did you train your model then?" }, { "id": 829029, "postDate": "2020-05-01T12:56:18.493Z", "content": "import tensorflow_addons as tfa
\n\nmetrics=[CohenKappa(num_classes=n_classes,weightage='quadratic')
", "rawMarkdown": "import tensorflow_addons as tfa\n\nmetrics=[CohenKappa(num_classes=n_classes,weightage='quadratic')", "votes": 2 }, { "id": 829071, "postDate": "2020-05-01T13:30:09.893Z", "content": "Thanks a lot.
", "rawMarkdown": "Thanks a lot.", "votes": 1 }, { "id": 829229, "postDate": "2020-05-01T15:59:06.043Z", "content": "@zhangeng but how do you make a submission with it? Internet should be disabled and tensorflow-addons is not installed by default.
", "rawMarkdown": "@zhangeng but how do you make a submission with it? Internet should be disabled and tensorflow-addons is not installed by default." }, { "id": 829582, "postDate": "2020-05-01T23:37:27.073Z", "content": "I trained on my local computer
", "rawMarkdown": "I trained on my local computer" }, { "id": 833659, "postDate": "2020-05-05T01:36:02.477Z", "content": "@ibtesama
\ndef metric(y_true, y_pred):\n return tf.compat.v1.py_func(numpy_metric ,(y_true, y_pred), tf.double)\ndef numpy_metric(y_true, y_pred):\n #numpy computing method\n return numpy_metric
@ibtesama This is the final answer to this post
", "rawMarkdown": "@ibtesama [This is the final answer to this post](https://www.tensorflow.org/api_docs/python/tf/compat/v1/py_func?hl=en)" }, { "id": 834035, "postDate": "2020-05-05T08:40:44.300Z", "content": "I'll try this. Thanks a lot @zhangeng
", "rawMarkdown": "I'll try this. Thanks a lot @zhangeng " } ] }, { "id": 824312, "postDate": "2020-04-28T09:46:44.557Z", "content": "Are you using just Keras or Tensorflow-Keras.?
\n\n.numpy() works when eager execution is enabled. When calling model.fit(), TensorFlow executes the training in graph mode and so .numpy() on a Tensor object will not work. Try decorating the method directly with tf.function().
Otherwise there is a very simple workaround for this, the Cohen Kappa Metric is already available in TensorFlow-Addons API. Follow this link
", "rawMarkdown": "Are you using just Keras or Tensorflow-Keras.?\n\n`.numpy() works when eager execution is enabled. When calling model.fit(), TensorFlow executes the training in graph mode and so .numpy() on a Tensor object will not work. Try decorating the method directly with tf.function().`\n\nOtherwise there is a very simple workaround for this, the Cohen Kappa Metric is already available in TensorFlow-Addons API. Follow this [link](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/146460)", "replies": [ { "id": 824323, "postDate": "2020-04-28T09:55:33.720Z", "content": "Eager execution is enabled by default in TF 2.0.(so that is done). \nEven the TensorFlow-Addons API you mentioned uses numpy array to make computations.
", "rawMarkdown": "Eager execution is enabled by default in TF 2.0.(so that is done). \nEven the TensorFlow-Addons API you mentioned uses numpy array to make computations." }, { "id": 824430, "postDate": "2020-04-28T11:03:32.537Z", "content": "I am very well aware that TensorFlow 2.0 is eager execution by default.
\n\nBut you see at times it doesnt work and this seems to be the same issue that I was having before where tf/keras was stuck in graph mode while they should be in eager mode. I tried decorating the function with tf.function() but it didn't work out.
\n\nThis error will throw only if the program execution is in graph mode.
", "rawMarkdown": "I am very well aware that TensorFlow 2.0 is eager execution by default. \n\nBut you see at times it doesnt work and this seems to be the same issue that I was having before where tf/keras was stuck in graph mode while they should be in eager mode. I tried decorating the function with tf.function() but it didn't work out. \n\nThis error will throw only if the program execution is in graph mode. " }, { "id": 824438, "postDate": "2020-04-28T11:11:23.413Z", "content": "There is a solution to this but its not a fix which is instead of calling .numpy() on tensor object try using eval() from tf.keras.backend. It will look something like,
\n\n\"eval()\" --> K.eval(my_tensor)
that's one way to convert a tensor into a numpy array in tf.keras (without a TensorFlow session).
", "rawMarkdown": "There is a solution to this but its not a fix which is instead of calling .numpy() on tensor object try using eval() from tf.keras.backend. It will look something like,\n\n` \"eval()\" --> K.eval(my_tensor)`\n\nthat's one way to convert a tensor into a numpy array in tf.keras (without a TensorFlow session).", "votes": 1 }, { "id": 824637, "postDate": "2020-04-28T13:58:06.330Z", "rawMarkdown": "", "isDeleted": true }, { "id": 824735, "postDate": "2020-04-28T15:06:14.747Z", "content": "@rhtsingh @sourcecode369 (Seems like the same profile). @kaggleteam
\n\nI have tried all of those things but the error remains.
", "rawMarkdown": "@rhtsingh @sourcecode369 (Seems like the same profile). @kaggleteam \n\nI have tried all of those things but the error remains." }, { "id": 824768, "postDate": "2020-04-28T15:39:11.837Z", "content": "Please don't declare such things without having a proper evidence.
\n\nI have been on Kaggle longer than you and if I had cheated I had been in ranks. I dont cheat neither am I running for medals. We work in the same place and this guy has joined Kaggle just a month ago.
", "rawMarkdown": "Please don't declare such things without having a proper evidence. \n\nI have been on Kaggle longer than you and if I had cheated I had been in ranks. I dont cheat neither am I running for medals. We work in the same place and this guy has joined Kaggle just a month ago. ", "votes": 1 }, { "id": 828542, "postDate": "2020-05-01T06:17:24.457Z", "content": "One person 's user name Rohit Singh and url sourcecode369\nAnother person's name Sourcecode369 and url Rohit Singh
\n\nI can understand how one can get confused by this coincidence .
", "rawMarkdown": "One person 's user name Rohit Singh and url sourcecode369\nAnother person's name Sourcecode369 and url Rohit Singh \n\nI can understand how one can get confused by this coincidence . ", "votes": -1 } ] }, { "id": 822041, "postDate": "2020-04-26T15:53:31.387Z", "content": "If you want to compute the value of a tensor , you just run the tensor in a session, there is no need to create a variable out of it. You can just run out in a session.
\n\nFor code simply follow this.
", "rawMarkdown": "\nIf you want to compute the value of a tensor , you just run the tensor in a session, there is no need to create a variable out of it. You can just run out in a session.\n\nFor code simply follow [this](https://kite.com/python/answers/how-to-convert-a-tensorflow-tensor-to-a-numpy-array-in-python).\n\n" }, { "id": 829793, "postDate": "2020-05-02T05:24:58.053Z", "rawMarkdown": "", "votes": -3, "isDeleted": true } ], "comments": [ { "id": 822115, "author_name": "Alex", "author_url": "", "post_date": "2020-04-26T17:15:25.173000", "content": "Hello @ibtesama, I would write a callback that does predictions and runs the sklearn.metric after each epoch. Something like:
\n\n```\nclass PredictionCallback(keras.callbacks.Callback):
\n\ndef __init__(self, test_data, batch_size=32)\n self.test_data = test_data\n self.scores = [] \ndef on_epoch_end(self, batch, logs={}):\n test_preds = self.model.predict(self.test_data, batch_size=self.batch_size)\n score = sklearn.metric(self.test_data.labels, test_preds)\n print(score)\n self.scores.append(scores)\ncallback = PredictionCallback(test_data)\nmodel.fit(..., callbacks=[callback])
\n\nscores = callback.scores\n```
\n\nif you wanna obtain all predictions for all epochs, append them to a list (self.test_preds = []) similar to self.scores in above example.
", "votes": 3, "replies": [ { "id": 828119, "author_name": "Alex", "author_url": "", "post_date": "2020-04-30T19:17:42.740000", "content": "Side note: If you happen to have memory issues with the implementation above or with model.predict in general (I had some memory leakage issues before), try tf-nightly (version 2.2.0-dev) and see if it solves the problem (it did for me). Unfortunately, memory issues have been a recurring problem [for me at least] when using Keras/tf.keras specifically. Interestingly, in the Kaggle kernel, I only have the leakage problem when using use_multiprocessing, not otherwise.
Hey, thanks for the help!
", "votes": 1, "replies": [] } ] }, { "id": 828794, "author_name": "zhangeng", "author_url": "", "post_date": "2020-05-01T09:34:07.643000", "content": "I have the same problem
", "votes": 0, "replies": [ { "id": 828823, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2020-05-01T10:07:36.353000", "content": "@zhangeng , How did you train your model then?
", "votes": 0, "replies": [] }, { "id": 829029, "author_name": "zhangeng", "author_url": "", "post_date": "2020-05-01T12:56:18.493000", "content": "import tensorflow_addons as tfa
\n\nmetrics=[CohenKappa(num_classes=n_classes,weightage='quadratic')
", "votes": 2, "replies": [] }, { "id": 829071, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2020-05-01T13:30:09.893000", "content": "Thanks a lot.
", "votes": 1, "replies": [] }, { "id": 829229, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2020-05-01T15:59:06.043000", "content": "@zhangeng but how do you make a submission with it? Internet should be disabled and tensorflow-addons is not installed by default.
", "votes": 0, "replies": [] }, { "id": 829582, "author_name": "zhangeng", "author_url": "", "post_date": "2020-05-01T23:37:27.073000", "content": "I trained on my local computer
", "votes": 0, "replies": [] }, { "id": 833659, "author_name": "zhangeng", "author_url": "", "post_date": "2020-05-05T01:36:02.477000", "content": "@ibtesama
\ndef metric(y_true, y_pred):\n return tf.compat.v1.py_func(numpy_metric ,(y_true, y_pred), tf.double)\ndef numpy_metric(y_true, y_pred):\n #numpy computing method\n return numpy_metric
@ibtesama This is the final answer to this post
", "votes": 0, "replies": [] }, { "id": 834035, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2020-05-05T08:40:44.300000", "content": "I'll try this. Thanks a lot @zhangeng
", "votes": 0, "replies": [] } ] }, { "id": 824312, "author_name": "torch", "author_url": "", "post_date": "2020-04-28T09:46:44.557000", "content": "Are you using just Keras or Tensorflow-Keras.?
\n\n.numpy() works when eager execution is enabled. When calling model.fit(), TensorFlow executes the training in graph mode and so .numpy() on a Tensor object will not work. Try decorating the method directly with tf.function().
Otherwise there is a very simple workaround for this, the Cohen Kappa Metric is already available in TensorFlow-Addons API. Follow this link
", "votes": 0, "replies": [ { "id": 824323, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2020-04-28T09:55:33.720000", "content": "Eager execution is enabled by default in TF 2.0.(so that is done). \nEven the TensorFlow-Addons API you mentioned uses numpy array to make computations.
", "votes": 0, "replies": [] }, { "id": 824430, "author_name": "torch", "author_url": "", "post_date": "2020-04-28T11:03:32.537000", "content": "I am very well aware that TensorFlow 2.0 is eager execution by default.
\n\nBut you see at times it doesnt work and this seems to be the same issue that I was having before where tf/keras was stuck in graph mode while they should be in eager mode. I tried decorating the function with tf.function() but it didn't work out.
\n\nThis error will throw only if the program execution is in graph mode.
", "votes": 0, "replies": [] }, { "id": 824438, "author_name": "torch", "author_url": "", "post_date": "2020-04-28T11:11:23.413000", "content": "There is a solution to this but its not a fix which is instead of calling .numpy() on tensor object try using eval() from tf.keras.backend. It will look something like,
\n\n\"eval()\" --> K.eval(my_tensor)
that's one way to convert a tensor into a numpy array in tf.keras (without a TensorFlow session).
", "votes": 1, "replies": [] }, { "id": 824637, "author_name": "", "author_url": "", "post_date": "2020-04-28T13:58:06.330000", "content": "", "votes": 0, "replies": [] }, { "id": 824735, "author_name": "Ibtesam Ahmed", "author_url": "", "post_date": "2020-04-28T15:06:14.747000", "content": "@rhtsingh @sourcecode369 (Seems like the same profile). @kaggleteam
\n\nI have tried all of those things but the error remains.
", "votes": 0, "replies": [] }, { "id": 824768, "author_name": "torch", "author_url": "", "post_date": "2020-04-28T15:39:11.837000", "content": "Please don't declare such things without having a proper evidence.
\n\nI have been on Kaggle longer than you and if I had cheated I had been in ranks. I dont cheat neither am I running for medals. We work in the same place and this guy has joined Kaggle just a month ago.
", "votes": 1, "replies": [] }, { "id": 828542, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-05-01T06:17:24.457000", "content": "One person 's user name Rohit Singh and url sourcecode369\nAnother person's name Sourcecode369 and url Rohit Singh
\n\nI can understand how one can get confused by this coincidence .
", "votes": -1, "replies": [] } ] }, { "id": 822041, "author_name": "Dracarys", "author_url": "", "post_date": "2020-04-26T15:53:31.387000", "content": "If you want to compute the value of a tensor , you just run the tensor in a session, there is no need to create a variable out of it. You can just run out in a session.
\n\nFor code simply follow this.
", "votes": 0, "replies": [] }, { "id": 829793, "author_name": "", "author_url": "", "post_date": "2020-05-02T05:24:58.053000", "content": "", "votes": -3, "replies": [] } ], "raw_markdown_by_id": { "822115": "Hello @ibtesama, I would write a callback that does predictions and runs the sklearn.metric after each epoch. Something like:\n\n```\nclass PredictionCallback(keras.callbacks.Callback):\n \n def __init__(self, test_data, batch_size=32)\n self.test_data = test_data\n self.scores = [] \n def on_epoch_end(self, batch, logs={}):\n test_preds = self.model.predict(self.test_data, batch_size=self.batch_size)\n score = sklearn.metric(self.test_data.labels, test_preds)\n print(score)\n self.scores.append(scores)\n\ncallback = PredictionCallback(test_data)\nmodel.fit(..., callbacks=[callback])\n\n# obtain all scores\nscores = callback.scores\n```\n\nif you wanna obtain all predictions for all epochs, append them to a list (self.test_preds = []) similar to self.scores in above example.\n", "821752": "Hey guys,\nI'm using Keras and trying to implement Cohen's Kappa Score(sklearn) as the \"metrics\".\nTo do so, I have to convert the tensor to a numpy array before making a computation.\n\nAfter going over several forums, I think there is no good way of doing that. \n.numpy() does not work and several issues have been raised on github, without coming to a conclusion.\nIf anyone knows how to do this, please leave a comment!", "828794": "I have the same problem", "824312": "Are you using just Keras or Tensorflow-Keras.?\n\n`.numpy() works when eager execution is enabled. When calling model.fit(), TensorFlow executes the training in graph mode and so .numpy() on a Tensor object will not work. Try decorating the method directly with tf.function().`\n\nOtherwise there is a very simple workaround for this, the Cohen Kappa Metric is already available in TensorFlow-Addons API. Follow this [link](https://www.kaggle.com/c/prostate-cancer-grade-assessment/discussion/146460)", "822041": "\nIf you want to compute the value of a tensor , you just run the tensor in a session, there is no need to create a variable out of it. You can just run out in a session.\n\nFor code simply follow [this](https://kite.com/python/answers/how-to-convert-a-tensorflow-tensor-to-a-numpy-array-in-python).\n\n", "829793": "" } }