{
  "id": 148572,
  "title": "Combine between tf.data and ImageDataGenerator !",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/148572",
  "author_name": "Yaheaal",
  "post_date": "2020-05-04T20:35:56.134000",
  "votes": 0,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I am trying to combine between tf.data and ImageDataGenerator by using this code :</p>\n\n<p>```\ndef prepare_training():\n    gen = ImageDataGenerator(rescale=1./255)\n    train_gen = gen.flow_from_dataframe(train,\n                                     x_col='image_id',\n                                     y_col=columns,\n                                     seed=SEED,\n                                     batch_size=BATCH_SIZE,\n                                     target_size=(SIZE,SIZE),\n                                     shuffle=False,\n                                     class_mode=\"raw\")</p>\n\n<pre><code>return train_gen\n</code></pre>\n\n<p>def prepare_train():\n    data = (\n        tf.data.Dataset\n        .from_generator(prepare_training, (tf.float32,tf.float32), output_shapes=([BATCH_SIZE,SIZE, SIZE,3], [BATCH_SIZE,nb_classes]))\n    return data\n```</p>\n\n<p>so when I don't use the TPU(TPU off) ,the code below works fine:\n<code>\nfor i in prepare_train():\n    print(i)\n    break\n</code>\nand It return the first batch , but when I turn on the TPU it shows this error:\n&gt; NotFoundError: No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}}.  Registered:  no registered [[PyFunc]]</p>\n\n<p>The <code>train</code>I've used in <code>gen.flow_from_dataframe</code> :<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2698359%2F5dd0e2b7083015af4adb20a3aae1a0f3%2FCapture.PNG?generation=1588625877538520&amp;alt=media\" alt=\"\">\nImage__id is the file_path for the data after saving it as png + each image ID in the training set.</p>\n\n<p>The ImageDataGenerator is working fine , I've test it before , the problem is with combining it with tf.data and TPU .Any ideas why am I getting this error, or can I do it in other ways ?</p>",
  "messages": [
    {
      "id": 834031,
      "postDate": "2020-05-05T08:39:18.720Z",
      "content": "<p>PyFunc is unsupported on the tpu_worker, as there is no python interpreter on the TPU Machines. \nYou can try using the StreamingFilesDataset which allows you to run a dataset with a pyfunc, and stream the resulting tensors over to the Cloud TPU. Hope that helps...</p>",
      "rawMarkdown": "PyFunc is unsupported on the tpu_worker, as there is no python interpreter on the TPU Machines. \nYou can try using the StreamingFilesDataset which allows you to run a dataset with a pyfunc, and stream the resulting tensors over to the Cloud TPU. Hope that helps...",
      "replies": [
        {
          "id": 834440,
          "postDate": "2020-05-05T14:40:42.977Z",
          "content": "<p>Do you have any code or video that explains how to do that ?\nThanks</p>",
          "rawMarkdown": "Do you have any code or video that explains how to do that ?\nThanks"
        }
      ]
    },
    {
      "id": 833467,
      "postDate": "2020-05-04T20:35:56.133Z",
      "content": "<p>I am trying to combine between tf.data and ImageDataGenerator by using this code :</p>\n\n<p>```\ndef prepare_training():\n    gen = ImageDataGenerator(rescale=1./255)\n    train_gen = gen.flow_from_dataframe(train,\n                                     x_col='image_id',\n                                     y_col=columns,\n                                     seed=SEED,\n                                     batch_size=BATCH_SIZE,\n                                     target_size=(SIZE,SIZE),\n                                     shuffle=False,\n                                     class_mode=\"raw\")</p>\n\n<pre><code>return train_gen\n</code></pre>\n\n<p>def prepare_train():\n    data = (\n        tf.data.Dataset\n        .from_generator(prepare_training, (tf.float32,tf.float32), output_shapes=([BATCH_SIZE,SIZE, SIZE,3], [BATCH_SIZE,nb_classes]))\n    return data\n```</p>\n\n<p>so when I don't use the TPU(TPU off) ,the code below works fine:\n<code>\nfor i in prepare_train():\n    print(i)\n    break\n</code>\nand It return the first batch , but when I turn on the TPU it shows this error:\n&gt; NotFoundError: No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}}.  Registered:  no registered [[PyFunc]]</p>\n\n<p>The <code>train</code>I've used in <code>gen.flow_from_dataframe</code> :<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2698359%2F5dd0e2b7083015af4adb20a3aae1a0f3%2FCapture.PNG?generation=1588625877538520&amp;alt=media\" alt=\"\">\nImage__id is the file_path for the data after saving it as png + each image ID in the training set.</p>\n\n<p>The ImageDataGenerator is working fine , I've test it before , the problem is with combining it with tf.data and TPU .Any ideas why am I getting this error, or can I do it in other ways ?</p>",
      "rawMarkdown": "I am trying to combine between tf.data and ImageDataGenerator by using this code :\n\n```\ndef prepare_training():\n    gen = ImageDataGenerator(rescale=1./255)\n    train_gen = gen.flow_from_dataframe(train,\n                                     x_col='image_id',\n                                     y_col=columns,\n                                     seed=SEED,\n                                     batch_size=BATCH_SIZE,\n                                     target_size=(SIZE,SIZE),\n                                     shuffle=False,\n                                     class_mode=\"raw\")\n\n    \n    return train_gen\n\ndef prepare_train():\n    data = (\n        tf.data.Dataset\n        .from_generator(prepare_training, (tf.float32,tf.float32), output_shapes=([BATCH_SIZE,SIZE, SIZE,3], [BATCH_SIZE,nb_classes]))\n    return data\n```\n\nso when I don't use the TPU(TPU off) ,the code below works fine:\n```\nfor i in prepare_train():\n    print(i)\n    break\n```\nand It return the first batch , but when I turn on the TPU it shows this error:\n&gt; NotFoundError: No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}}.  Registered:  no registered [[PyFunc]]\n\nThe `train `I've used in `gen.flow_from_dataframe` :![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2698359%2F5dd0e2b7083015af4adb20a3aae1a0f3%2FCapture.PNG?generation=1588625877538520&amp;alt=media)\nImage__id is the file_path for the data after saving it as png + each image ID in the training set.\n\nThe ImageDataGenerator is working fine , I've test it before , the problem is with combining it with tf.data and TPU .Any ideas why am I getting this error, or can I do it in other ways ?"
    }
  ],
  "comments": [
    {
      "id": 834031,
      "author_name": "Yugansh Goyal",
      "author_url": "",
      "post_date": "2020-05-05T08:39:18.720000",
      "content": "<p>PyFunc is unsupported on the tpu_worker, as there is no python interpreter on the TPU Machines. \nYou can try using the StreamingFilesDataset which allows you to run a dataset with a pyfunc, and stream the resulting tensors over to the Cloud TPU. Hope that helps...</p>",
      "votes": 0,
      "replies": [
        {
          "id": 834440,
          "author_name": "Yaheaal",
          "author_url": "",
          "post_date": "2020-05-05T14:40:42.977000",
          "content": "<p>Do you have any code or video that explains how to do that ?\nThanks</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "834031": "PyFunc is unsupported on the tpu_worker, as there is no python interpreter on the TPU Machines. \nYou can try using the StreamingFilesDataset which allows you to run a dataset with a pyfunc, and stream the resulting tensors over to the Cloud TPU. Hope that helps...",
    "833467": "I am trying to combine between tf.data and ImageDataGenerator by using this code :\n\n```\ndef prepare_training():\n    gen = ImageDataGenerator(rescale=1./255)\n    train_gen = gen.flow_from_dataframe(train,\n                                     x_col='image_id',\n                                     y_col=columns,\n                                     seed=SEED,\n                                     batch_size=BATCH_SIZE,\n                                     target_size=(SIZE,SIZE),\n                                     shuffle=False,\n                                     class_mode=\"raw\")\n\n    \n    return train_gen\n\ndef prepare_train():\n    data = (\n        tf.data.Dataset\n        .from_generator(prepare_training, (tf.float32,tf.float32), output_shapes=([BATCH_SIZE,SIZE, SIZE,3], [BATCH_SIZE,nb_classes]))\n    return data\n```\n\nso when I don't use the TPU(TPU off) ,the code below works fine:\n```\nfor i in prepare_train():\n    print(i)\n    break\n```\nand It return the first batch , but when I turn on the TPU it shows this error:\n&gt; NotFoundError: No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}}.  Registered:  no registered [[PyFunc]]\n\nThe `train `I've used in `gen.flow_from_dataframe` :![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2698359%2F5dd0e2b7083015af4adb20a3aae1a0f3%2FCapture.PNG?generation=1588625877538520&amp;alt=media)\nImage__id is the file_path for the data after saving it as png + each image ID in the training set.\n\nThe ImageDataGenerator is working fine , I've test it before , the problem is with combining it with tf.data and TPU .Any ideas why am I getting this error, or can I do it in other ways ?"
  }
}