{
  "id": 161693,
  "title": "Transform images to arrays for keras ",
  "url": "/competitions/prostate-cancer-grade-assessment/discussion/161693",
  "author_name": "Andreas Horlbeck",
  "post_date": "2020-06-25T19:18:03.376000",
  "votes": 0,
  "comment_count": 3,
  "views": 0,
  "content": "<p>hey,\ni have added the image data ( about 10.600 images as png-files with shape 512,512,3 - thanks to another guy on the platform here) to my project.\nThen I loaded these png files to array with the help of keras in order to use these arrays\nfor the following cnn-model. But when I convert the images to a whole array it takes years that\nall images have been converted to arrays. So what do you do?\nDo you convert the images to array once and the next time you just load the arrays again?\nHow and where can you store the arrays?\nI am really lost and need help.\nThanks a lot in advance!\nCheers!!!</p>",
  "messages": [
    {
      "id": 901944,
      "postDate": "2020-06-25T19:57:46.353Z",
      "content": "<p>ok stop, this works much much faster:</p>\n\n<h1>PREPROCESS DATA - READ IMAGE AS ARRAYS</h1>\n\n<p>img_size_x,img_size_y = (512, 512)</p>\n\n<p>images_array = np.empty(shape=(10616,img_size_x,img_size_y,3),dtype='float') </p>\n\n<p>i=0</p>\n\n<p>for img in tqdm(train.image_id):\n    image = tf.keras.preprocessing.image.load_img(os.path.join(path_images, f'{img}.png'))\n    input_arr = tf.keras.preprocessing.image.img_to_array(image)/255.\n    input_arr = np.array([input_arr]) \n    images_array[i] = input_arr\n    i=i+1</p>\n\n<p>no need to store it</p>",
      "rawMarkdown": "ok stop, this works much much faster:\n\n# PREPROCESS DATA - READ IMAGE AS ARRAYS\nimg_size_x,img_size_y = (512, 512)\n\nimages_array = np.empty(shape=(10616,img_size_x,img_size_y,3),dtype='float') \n\ni=0\n\nfor img in tqdm(train.image_id):\n    image = tf.keras.preprocessing.image.load_img(os.path.join(path_images, f'{img}.png'))\n    input_arr = tf.keras.preprocessing.image.img_to_array(image)/255.\n    input_arr = np.array([input_arr]) \n    images_array[i] = input_arr\n    i=i+1\n\nno need to store it"
    },
    {
      "id": 901924,
      "postDate": "2020-06-25T19:43:45.620Z",
      "content": "<p>ok now I have a workaorund:</p>\n\n<h1>PREPROCESS DATA - READ IMAGE AS ARRAYS</h1>\n\n<p>img_size_x,img_size_y = (512, 512)</p>\n\n<p>images_array = np.empty(shape=(1,img_size_x,img_size_y,3),dtype='float') </p>\n\n<p>i=0</p>\n\n<p>for img in tqdm(train.image_id):\n    image = tf.keras.preprocessing.image.load_img(os.path.join(path_images, f'{img}.png'))\n    input_arr = tf.keras.preprocessing.image.img_to_array(image)/255.\n    input_arr = np.array([input_arr]) \n    if i==0:\n        images_array = input_arr <br>\n    else:\n        images_array = np.concatenate((images_array, input_arr))\n    i=i+1</p>\n\n<pre><code>np.save('train_images_as_array.npy', images_array)\n</code></pre>\n\n<p>print('Shape of Full Data set:,', images_array.shape)</p>\n\n<p>np.save('train_images_as_array.npy', images_array)\narr=np.load('train_images_as_array.npy')\nprint(images_array)\nprint(arr)</p>",
      "rawMarkdown": "ok now I have a workaorund:\n# PREPROCESS DATA - READ IMAGE AS ARRAYS\nimg_size_x,img_size_y = (512, 512)\n\nimages_array = np.empty(shape=(1,img_size_x,img_size_y,3),dtype='float') \n\ni=0\n\nfor img in tqdm(train.image_id):\n    image = tf.keras.preprocessing.image.load_img(os.path.join(path_images, f'{img}.png'))\n    input_arr = tf.keras.preprocessing.image.img_to_array(image)/255.\n    input_arr = np.array([input_arr]) \n    if i==0:\n        images_array = input_arr    \n    else:\n        images_array = np.concatenate((images_array, input_arr))\n    i=i+1\n\n    np.save('train_images_as_array.npy', images_array)\n\nprint('Shape of Full Data set:,', images_array.shape)\n\nnp.save('train_images_as_array.npy', images_array)\narr=np.load('train_images_as_array.npy')\nprint(images_array)\nprint(arr)",
      "replies": [
        {
          "id": 903062,
          "postDate": "2020-06-26T14:50:47.133Z",
          "content": "<p>Can I ask what the purpose is of converting all the images to arrays, and then saving them into a single large npy file? Wouldn't that take up a major chunk of memory (read: impossibly large)?</p>",
          "rawMarkdown": "Can I ask what the purpose is of converting all the images to arrays, and then saving them into a single large npy file? Wouldn't that take up a major chunk of memory (read: impossibly large)?"
        }
      ]
    },
    {
      "id": 901891,
      "postDate": "2020-06-25T19:18:03.377Z",
      "content": "<p>hey,\ni have added the image data ( about 10.600 images as png-files with shape 512,512,3 - thanks to another guy on the platform here) to my project.\nThen I loaded these png files to array with the help of keras in order to use these arrays\nfor the following cnn-model. But when I convert the images to a whole array it takes years that\nall images have been converted to arrays. So what do you do?\nDo you convert the images to array once and the next time you just load the arrays again?\nHow and where can you store the arrays?\nI am really lost and need help.\nThanks a lot in advance!\nCheers!!!</p>",
      "rawMarkdown": "hey,\ni have added the image data ( about 10.600 images as png-files with shape 512,512,3 - thanks to another guy on the platform here) to my project.\nThen I loaded these png files to array with the help of keras in order to use these arrays\nfor the following cnn-model. But when I convert the images to a whole array it takes years that\nall images have been converted to arrays. So what do you do?\nDo you convert the images to array once and the next time you just load the arrays again?\nHow and where can you store the arrays?\nI am really lost and need help.\nThanks a lot in advance!\nCheers!!!"
    }
  ],
  "comments": [
    {
      "id": 901944,
      "author_name": "Andreas Horlbeck",
      "author_url": "",
      "post_date": "2020-06-25T19:57:46.353000",
      "content": "<p>ok stop, this works much much faster:</p>\n\n<h1>PREPROCESS DATA - READ IMAGE AS ARRAYS</h1>\n\n<p>img_size_x,img_size_y = (512, 512)</p>\n\n<p>images_array = np.empty(shape=(10616,img_size_x,img_size_y,3),dtype='float') </p>\n\n<p>i=0</p>\n\n<p>for img in tqdm(train.image_id):\n    image = tf.keras.preprocessing.image.load_img(os.path.join(path_images, f'{img}.png'))\n    input_arr = tf.keras.preprocessing.image.img_to_array(image)/255.\n    input_arr = np.array([input_arr]) \n    images_array[i] = input_arr\n    i=i+1</p>\n\n<p>no need to store it</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 901924,
      "author_name": "Andreas Horlbeck",
      "author_url": "",
      "post_date": "2020-06-25T19:43:45.620000",
      "content": "<p>ok now I have a workaorund:</p>\n\n<h1>PREPROCESS DATA - READ IMAGE AS ARRAYS</h1>\n\n<p>img_size_x,img_size_y = (512, 512)</p>\n\n<p>images_array = np.empty(shape=(1,img_size_x,img_size_y,3),dtype='float') </p>\n\n<p>i=0</p>\n\n<p>for img in tqdm(train.image_id):\n    image = tf.keras.preprocessing.image.load_img(os.path.join(path_images, f'{img}.png'))\n    input_arr = tf.keras.preprocessing.image.img_to_array(image)/255.\n    input_arr = np.array([input_arr]) \n    if i==0:\n        images_array = input_arr <br>\n    else:\n        images_array = np.concatenate((images_array, input_arr))\n    i=i+1</p>\n\n<pre><code>np.save('train_images_as_array.npy', images_array)\n</code></pre>\n\n<p>print('Shape of Full Data set:,', images_array.shape)</p>\n\n<p>np.save('train_images_as_array.npy', images_array)\narr=np.load('train_images_as_array.npy')\nprint(images_array)\nprint(arr)</p>",
      "votes": 0,
      "replies": [
        {
          "id": 903062,
          "author_name": "Stephan",
          "author_url": "",
          "post_date": "2020-06-26T14:50:47.133000",
          "content": "<p>Can I ask what the purpose is of converting all the images to arrays, and then saving them into a single large npy file? Wouldn't that take up a major chunk of memory (read: impossibly large)?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "901944": "ok stop, this works much much faster:\n\n# PREPROCESS DATA - READ IMAGE AS ARRAYS\nimg_size_x,img_size_y = (512, 512)\n\nimages_array = np.empty(shape=(10616,img_size_x,img_size_y,3),dtype='float') \n\ni=0\n\nfor img in tqdm(train.image_id):\n    image = tf.keras.preprocessing.image.load_img(os.path.join(path_images, f'{img}.png'))\n    input_arr = tf.keras.preprocessing.image.img_to_array(image)/255.\n    input_arr = np.array([input_arr]) \n    images_array[i] = input_arr\n    i=i+1\n\nno need to store it",
    "901924": "ok now I have a workaorund:\n# PREPROCESS DATA - READ IMAGE AS ARRAYS\nimg_size_x,img_size_y = (512, 512)\n\nimages_array = np.empty(shape=(1,img_size_x,img_size_y,3),dtype='float') \n\ni=0\n\nfor img in tqdm(train.image_id):\n    image = tf.keras.preprocessing.image.load_img(os.path.join(path_images, f'{img}.png'))\n    input_arr = tf.keras.preprocessing.image.img_to_array(image)/255.\n    input_arr = np.array([input_arr]) \n    if i==0:\n        images_array = input_arr    \n    else:\n        images_array = np.concatenate((images_array, input_arr))\n    i=i+1\n\n    np.save('train_images_as_array.npy', images_array)\n\nprint('Shape of Full Data set:,', images_array.shape)\n\nnp.save('train_images_as_array.npy', images_array)\narr=np.load('train_images_as_array.npy')\nprint(images_array)\nprint(arr)",
    "901891": "hey,\ni have added the image data ( about 10.600 images as png-files with shape 512,512,3 - thanks to another guy on the platform here) to my project.\nThen I loaded these png files to array with the help of keras in order to use these arrays\nfor the following cnn-model. But when I convert the images to a whole array it takes years that\nall images have been converted to arrays. So what do you do?\nDo you convert the images to array once and the next time you just load the arrays again?\nHow and where can you store the arrays?\nI am really lost and need help.\nThanks a lot in advance!\nCheers!!!"
  }
}