{
  "id": 446041,
  "title": "Any idea how to avoid out of memory error in submitting when using image.open()?",
  "url": "/competitions/UBC-OCEAN/discussion/446041",
  "author_name": "Alexandra",
  "post_date": "2023-10-10T04:18:49.894000",
  "votes": 4,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Hey,</p>\n<p>when submitting I sequentially open the images, process them and then make the submission predictions but after an hour or so I run out of memory.</p>\n<p>Any tips?</p>\n<p>So far I also use gc.collect, image.close() etc. but it's not enough. I also used a non-gpu notebook version, but the ram still is full after some time and I get an out of memory error.</p>\n<p>Best,<br>\nAlex</p>",
  "messages": [
    {
      "id": 2475640,
      "postDate": "2023-10-10T04:18:49.893Z",
      "content": "<p>Hey,</p>\n<p>when submitting I sequentially open the images, process them and then make the submission predictions but after an hour or so I run out of memory.</p>\n<p>Any tips?</p>\n<p>So far I also use gc.collect, image.close() etc. but it's not enough. I also used a non-gpu notebook version, but the ram still is full after some time and I get an out of memory error.</p>\n<p>Best,<br>\nAlex</p>",
      "rawMarkdown": "Hey,\n\nwhen submitting I sequentially open the images, process them and then make the submission predictions but after an hour or so I run out of memory.\n\nAny tips?\n\nSo far I also use gc.collect, image.close() etc. but it's not enough. I also used a non-gpu notebook version, but the ram still is full after some time and I get an out of memory error.\n\nBest,\nAlex",
      "votes": 4
    },
    {
      "id": 2491921,
      "postDate": "2023-10-22T06:29:57.757Z",
      "content": "<p>Check my libvips notebook <a href=\"https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started\" target=\"_blank\">https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started</a></p>",
      "rawMarkdown": "Check my libvips notebook https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started",
      "votes": 1
    },
    {
      "id": 2491563,
      "postDate": "2023-10-21T18:01:16.583Z",
      "content": "<p>I had the same issue ,when submitting notebook - Notebook Out of Memory Error is given. Please can anyone knows how to handle this problem. </p>",
      "rawMarkdown": "I had the same issue ,when submitting notebook - Notebook Out of Memory Error is given. Please can anyone knows how to handle this problem. ",
      "votes": 2,
      "replies": [
        {
          "id": 2491688,
          "postDate": "2023-10-21T22:32:29.193Z",
          "content": "<p>try pyvips or other libraries that are more memory efficient, I think that works relatively well with extracting smaller patches e.g. 256x256 pixels. and don't forget gc.collect etc. </p>",
          "rawMarkdown": "try pyvips or other libraries that are more memory efficient, I think that works relatively well with extracting smaller patches e.g. 256x256 pixels. and don't forget gc.collect etc. "
        }
      ]
    },
    {
      "id": 2490944,
      "postDate": "2023-10-21T08:04:31.763Z",
      "content": "<p>I had the same issue, but then I decided to use batch_size=1, num_workers=1. It takes around 7 hours to submit. Did you manage to solve it?</p>",
      "rawMarkdown": "I had the same issue, but then I decided to use batch_size=1, num_workers=1. It takes around 7 hours to submit. Did you manage to solve it?",
      "replies": [
        {
          "id": 2491177,
          "postDate": "2023-10-21T11:47:13.487Z",
          "content": "<p>thanks for the reply. actually switched to another library opening the images (rasterio first, currently experimenting with pyvips) and the ram increase in the gpu notebooks also definitely helped. would probably still have out of ram error with normal image.open after a while.</p>",
          "rawMarkdown": "thanks for the reply. actually switched to another library opening the images (rasterio first, currently experimenting with pyvips) and the ram increase in the gpu notebooks also definitely helped. would probably still have out of ram error with normal image.open after a while.",
          "votes": 1,
          "replies": [
            {
              "id": 2491202,
              "postDate": "2023-10-21T12:14:00.977Z",
              "content": "<p>What's the running time with rasterio? Am trying out pyvips at the moment</p>",
              "rawMarkdown": "What's the running time with rasterio? Am trying out pyvips at the moment"
            },
            {
              "id": 2491218,
              "postDate": "2023-10-21T12:31:25.270Z",
              "content": "<p>Rasterio is absolutely wonderful.</p>",
              "rawMarkdown": "Rasterio is absolutely wonderful."
            },
            {
              "id": 2491247,
              "postDate": "2023-10-21T13:05:00.273Z",
              "content": "<p>rasterio not bad timewise but pyvips seems still a good margin faster to me, although I still need some more runs to see if pyvips works well for me.</p>",
              "rawMarkdown": "rasterio not bad timewise but pyvips seems still a good margin faster to me, although I still need some more runs to see if pyvips works well for me."
            },
            {
              "id": 2504863,
              "postDate": "2023-10-30T07:41:18.127Z",
              "content": "<p>I have also encountered the same issue when trying to use the full “.png” image for testing, and it throws a \"ram out of memory\" error. Have you resolved this issue, and if so, how did you address it? I also tried to use batch_size=1 and num_workers=1, but it also didn't work. :(</p>",
              "rawMarkdown": "I have also encountered the same issue when trying to use the full “.png” image for testing, and it throws a \"ram out of memory\" error. Have you resolved this issue, and if so, how did you address it? I also tried to use batch_size=1 and num_workers=1, but it also didn't work. :("
            }
          ]
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2491921,
      "author_name": "Gunes Evitan",
      "author_url": "",
      "post_date": "2023-10-22T06:29:57.757000",
      "content": "<p>Check my libvips notebook <a href=\"https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started\" target=\"_blank\">https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started</a></p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2491563,
      "author_name": "sunil thite",
      "author_url": "",
      "post_date": "2023-10-21T18:01:16.583000",
      "content": "<p>I had the same issue ,when submitting notebook - Notebook Out of Memory Error is given. Please can anyone knows how to handle this problem. </p>",
      "votes": 2,
      "replies": [
        {
          "id": 2491688,
          "author_name": "Alexandra",
          "author_url": "",
          "post_date": "2023-10-21T22:32:29.193000",
          "content": "<p>try pyvips or other libraries that are more memory efficient, I think that works relatively well with extracting smaller patches e.g. 256x256 pixels. and don't forget gc.collect etc. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2490944,
      "author_name": "samu2505",
      "author_url": "",
      "post_date": "2023-10-21T08:04:31.763000",
      "content": "<p>I had the same issue, but then I decided to use batch_size=1, num_workers=1. It takes around 7 hours to submit. Did you manage to solve it?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2491177,
          "author_name": "Alexandra",
          "author_url": "",
          "post_date": "2023-10-21T11:47:13.487000",
          "content": "<p>thanks for the reply. actually switched to another library opening the images (rasterio first, currently experimenting with pyvips) and the ram increase in the gpu notebooks also definitely helped. would probably still have out of ram error with normal image.open after a while.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2491202,
              "author_name": "samu2505",
              "author_url": "",
              "post_date": "2023-10-21T12:14:00.977000",
              "content": "<p>What's the running time with rasterio? Am trying out pyvips at the moment</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2491218,
              "author_name": "samu2505",
              "author_url": "",
              "post_date": "2023-10-21T12:31:25.270000",
              "content": "<p>Rasterio is absolutely wonderful.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2491247,
              "author_name": "Alexandra",
              "author_url": "",
              "post_date": "2023-10-21T13:05:00.273000",
              "content": "<p>rasterio not bad timewise but pyvips seems still a good margin faster to me, although I still need some more runs to see if pyvips works well for me.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2504863,
              "author_name": "yang_zhou",
              "author_url": "",
              "post_date": "2023-10-30T07:41:18.127000",
              "content": "<p>I have also encountered the same issue when trying to use the full “.png” image for testing, and it throws a \"ram out of memory\" error. Have you resolved this issue, and if so, how did you address it? I also tried to use batch_size=1 and num_workers=1, but it also didn't work. :(</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2475640": "Hey,\n\nwhen submitting I sequentially open the images, process them and then make the submission predictions but after an hour or so I run out of memory.\n\nAny tips?\n\nSo far I also use gc.collect, image.close() etc. but it's not enough. I also used a non-gpu notebook version, but the ram still is full after some time and I get an out of memory error.\n\nBest,\nAlex",
    "2491921": "Check my libvips notebook https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started",
    "2491563": "I had the same issue ,when submitting notebook - Notebook Out of Memory Error is given. Please can anyone knows how to handle this problem. ",
    "2490944": "I had the same issue, but then I decided to use batch_size=1, num_workers=1. It takes around 7 hours to submit. Did you manage to solve it?"
  }
}