{
  "id": 343571,
  "title": "Data too big",
  "url": "/competitions/mayo-clinic-strip-ai/discussion/343571",
  "author_name": "Elviz",
  "post_date": "2022-08-11T19:59:17.025000",
  "votes": 0,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hello!</p>\n<p>When I try to collect all images into one array, It takes too much memory and session restart. Can anyone suggest an idea on how to read all images. Thank you in advance!</p>",
  "messages": [
    {
      "id": 1900344,
      "postDate": "2022-08-15T23:51:25.197Z",
      "content": "<p>These may be useful:<br>\n<a href=\"https://www.kaggle.com/code/realneuralnetwork/coatnet-strip-ai-inference\" target=\"_blank\">https://www.kaggle.com/code/realneuralnetwork/coatnet-strip-ai-inference</a><br>\n<a href=\"https://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715\" target=\"_blank\">https://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715</a><br>\nGood luck!</p>",
      "rawMarkdown": "These may be useful:\nhttps://www.kaggle.com/code/realneuralnetwork/coatnet-strip-ai-inference\nhttps://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715\nGood luck!",
      "votes": 1,
      "replies": [
        {
          "id": 1903328,
          "postDate": "2022-08-17T10:15:22.710Z",
          "content": "<p>Thank you!</p>",
          "rawMarkdown": "Thank you!\n"
        }
      ]
    },
    {
      "id": 1895261,
      "postDate": "2022-08-12T03:40:02.263Z",
      "content": "<p>If you want to do that to train a model, look into Data Generators. These will load each image or batch of images and feed to the model as you need them, instead of loading them all at once. This way you won't blow up memory.</p>",
      "rawMarkdown": "If you want to do that to train a model, look into Data Generators. These will load each image or batch of images and feed to the model as you need them, instead of loading them all at once. This way you won't blow up memory.",
      "votes": 2,
      "replies": [
        {
          "id": 1895508,
          "postDate": "2022-08-12T07:28:53.167Z",
          "content": "<p>Thank you! But I think to use data Generators there must be spesific folder structure:<br>\nTrain<br>\n    Class1<br>\n      Images<br>\n   Class 2 <br>\n    Images<br>\nTest<br>\n   Class1 <br>\n      Images<br>\n   Class 2<br>\n       Images</p>\n<p>But in this case there's only one folder with all images and csv file with labels. Can data generators still be used? If yes, how?</p>",
          "rawMarkdown": "Thank you! But I think to use data Generators there must be spesific folder structure:\nTrain\n    Class1\n      Images\n   Class 2 \n    Images\nTest\n   Class1 \n      Images\n   Class 2\n       Images\n\nBut in this case there's only one folder with all images and csv file with labels. Can data generators still be used? If yes, how?\n\n\n",
          "votes": 1
        },
        {
          "id": 1896146,
          "postDate": "2022-08-12T16:10:10.853Z",
          "content": "<p>All you have to do is use Shutil or OS library to create those folders and move the respective images to each folder, and you are good to go. But in that case you will need to use the JPEG dataset or you will not have enough HD space, or maybe do it on your computer and upload it.</p>",
          "rawMarkdown": "All you have to do is use Shutil or OS library to create those folders and move the respective images to each folder, and you are good to go. But in that case you will need to use the JPEG dataset or you will not have enough HD space, or maybe do it on your computer and upload it.",
          "votes": 1
        },
        {
          "id": 1896153,
          "postDate": "2022-08-12T16:11:16.660Z",
          "content": "<p>If you search for \"tiff to jpeg\" you will find a notebook of this competition that converted the tiff files to jpeg, keeping the original sizes and all</p>",
          "rawMarkdown": "If you search for \"tiff to jpeg\" you will find a notebook of this competition that converted the tiff files to jpeg, keeping the original sizes and all"
        },
        {
          "id": 1903330,
          "postDate": "2022-08-17T10:15:36.077Z",
          "content": "<p>Thank you very much!</p>",
          "rawMarkdown": "Thank you very much!\n"
        }
      ]
    },
    {
      "id": 1894938,
      "postDate": "2022-08-11T19:59:17.027Z",
      "content": "<p>Hello!</p>\n<p>When I try to collect all images into one array, It takes too much memory and session restart. Can anyone suggest an idea on how to read all images. Thank you in advance!</p>",
      "rawMarkdown": "Hello!\n\nWhen I try to collect all images into one array, It takes too much memory and session restart. Can anyone suggest an idea on how to read all images. Thank you in advance!"
    }
  ],
  "comments": [
    {
      "id": 1900344,
      "author_name": "Rodrigo M Carrillo Larco",
      "author_url": "",
      "post_date": "2022-08-15T23:51:25.197000",
      "content": "<p>These may be useful:<br>\n<a href=\"https://www.kaggle.com/code/realneuralnetwork/coatnet-strip-ai-inference\" target=\"_blank\">https://www.kaggle.com/code/realneuralnetwork/coatnet-strip-ai-inference</a><br>\n<a href=\"https://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715\" target=\"_blank\">https://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715</a><br>\nGood luck!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1903328,
          "author_name": "Elviz",
          "author_url": "",
          "post_date": "2022-08-17T10:15:22.710000",
          "content": "<p>Thank you!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1895261,
      "author_name": "Iuryck Santos",
      "author_url": "",
      "post_date": "2022-08-12T03:40:02.263000",
      "content": "<p>If you want to do that to train a model, look into Data Generators. These will load each image or batch of images and feed to the model as you need them, instead of loading them all at once. This way you won't blow up memory.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1895508,
          "author_name": "Elviz",
          "author_url": "",
          "post_date": "2022-08-12T07:28:53.167000",
          "content": "<p>Thank you! But I think to use data Generators there must be spesific folder structure:<br>\nTrain<br>\n    Class1<br>\n      Images<br>\n   Class 2 <br>\n    Images<br>\nTest<br>\n   Class1 <br>\n      Images<br>\n   Class 2<br>\n       Images</p>\n<p>But in this case there's only one folder with all images and csv file with labels. Can data generators still be used? If yes, how?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1896146,
          "author_name": "Iuryck Santos",
          "author_url": "",
          "post_date": "2022-08-12T16:10:10.853000",
          "content": "<p>All you have to do is use Shutil or OS library to create those folders and move the respective images to each folder, and you are good to go. But in that case you will need to use the JPEG dataset or you will not have enough HD space, or maybe do it on your computer and upload it.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1896153,
          "author_name": "Iuryck Santos",
          "author_url": "",
          "post_date": "2022-08-12T16:11:16.660000",
          "content": "<p>If you search for \"tiff to jpeg\" you will find a notebook of this competition that converted the tiff files to jpeg, keeping the original sizes and all</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1903330,
          "author_name": "Elviz",
          "author_url": "",
          "post_date": "2022-08-17T10:15:36.077000",
          "content": "<p>Thank you very much!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1900344": "These may be useful:\nhttps://www.kaggle.com/code/realneuralnetwork/coatnet-strip-ai-inference\nhttps://www.kaggle.com/code/rodrigocarrillo/import-model-trained-locally-20220715\nGood luck!",
    "1895261": "If you want to do that to train a model, look into Data Generators. These will load each image or batch of images and feed to the model as you need them, instead of loading them all at once. This way you won't blow up memory.",
    "1894938": "Hello!\n\nWhen I try to collect all images into one array, It takes too much memory and session restart. Can anyone suggest an idea on how to read all images. Thank you in advance!"
  }
}