{
  "id": 383608,
  "title": "TPU with ImageDataGeneator",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/383608",
  "author_name": "Igor Litvin",
  "post_date": "2023-02-04T09:43:49.407000",
  "votes": 0,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Good day.</p>\n<p>I try to use Kaggle TPU with ImageDataGeneator. It doesnt work. If you know how to sort it out please let me know.</p>\n<p>I spend almost 2 weeks with no result.</p>\n<p>What best to use for TPU instead  ImageDataGeneator? </p>\n<p>Thanks a lot for your help</p>\n<p>Igor</p>",
  "messages": [
    {
      "id": 2137437,
      "postDate": "2023-02-10T02:06:57.470Z",
      "content": "<p>Hi, do you use tf.keras.utils.Sequence as \"ImageDataGenerator\"? If yes, that won't work with TPU (TPU-VM can read from local) and it is not recommanded for distributed training. You need to use tf.data.Dataset for input pipeline. For TPU, you will need to generate tfrecords and read tfrecords from gcs. </p>",
      "rawMarkdown": "Hi, do you use tf.keras.utils.Sequence as \"ImageDataGenerator\"? If yes, that won't work with TPU (TPU-VM can read from local) and it is not recommanded for distributed training. You need to use tf.data.Dataset for input pipeline. For TPU, you will need to generate tfrecords and read tfrecords from gcs. "
    },
    {
      "id": 2129087,
      "postDate": "2023-02-04T10:03:43.383Z",
      "content": "<p>tracking issue: <a href=\"https://github.com/tensorflow/tensorflow/issues/34346\" target=\"_blank\">https://github.com/tensorflow/tensorflow/issues/34346</a><br>\n(maybe try: tpu-vm v3)</p>",
      "rawMarkdown": "tracking issue: https://github.com/tensorflow/tensorflow/issues/34346\n(maybe try: tpu-vm v3)"
    },
    {
      "id": 2129072,
      "postDate": "2023-02-04T09:43:49.407Z",
      "content": "<p>Good day.</p>\n<p>I try to use Kaggle TPU with ImageDataGeneator. It doesnt work. If you know how to sort it out please let me know.</p>\n<p>I spend almost 2 weeks with no result.</p>\n<p>What best to use for TPU instead  ImageDataGeneator? </p>\n<p>Thanks a lot for your help</p>\n<p>Igor</p>",
      "rawMarkdown": "Good day.\n\nI try to use Kaggle TPU with ImageDataGeneator. It doesnt work. If you know how to sort it out please let me know.\n\nI spend almost 2 weeks with no result.\n\nWhat best to use for TPU instead  ImageDataGeneator? \n\nThanks a lot for your help\n\nIgor"
    },
    {
      "id": 2129582,
      "postDate": "2023-02-04T18:03:47.573Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2137437,
      "author_name": "dingyan",
      "author_url": "",
      "post_date": "2023-02-10T02:06:57.470000",
      "content": "<p>Hi, do you use tf.keras.utils.Sequence as \"ImageDataGenerator\"? If yes, that won't work with TPU (TPU-VM can read from local) and it is not recommanded for distributed training. You need to use tf.data.Dataset for input pipeline. For TPU, you will need to generate tfrecords and read tfrecords from gcs. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2129087,
      "author_name": "Innat",
      "author_url": "",
      "post_date": "2023-02-04T10:03:43.383000",
      "content": "<p>tracking issue: <a href=\"https://github.com/tensorflow/tensorflow/issues/34346\" target=\"_blank\">https://github.com/tensorflow/tensorflow/issues/34346</a><br>\n(maybe try: tpu-vm v3)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2129582,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-02-04T18:03:47.573000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2137437": "Hi, do you use tf.keras.utils.Sequence as \"ImageDataGenerator\"? If yes, that won't work with TPU (TPU-VM can read from local) and it is not recommanded for distributed training. You need to use tf.data.Dataset for input pipeline. For TPU, you will need to generate tfrecords and read tfrecords from gcs. ",
    "2129087": "tracking issue: https://github.com/tensorflow/tensorflow/issues/34346\n(maybe try: tpu-vm v3)",
    "2129072": "Good day.\n\nI try to use Kaggle TPU with ImageDataGeneator. It doesnt work. If you know how to sort it out please let me know.\n\nI spend almost 2 weeks with no result.\n\nWhat best to use for TPU instead  ImageDataGeneator? \n\nThanks a lot for your help\n\nIgor",
    "2129582": ""
  }
}