{
  "id": 457460,
  "title": "Why GPU memory gets filled like this?",
  "url": "/competitions/UBC-OCEAN/discussion/457460",
  "author_name": "Gabriel Chehade",
  "post_date": "2023-11-24T23:14:09.002000",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi,<br>\nI've ran the pinned notebook <strong>[KerasCV] train and infer on thumbnails</strong> and I've noticed that when loading the model with the command <br>\n<code>resnet_backbone = keras_cv.models.ResNetV2Backbone.from_preset(\n    \"resnet152_v2\",\n)</code><br>\nthe GPU memory usage reaches about 15 GB and I don't understand why. I've ran the same command on a Colab notebook and the memory consumption increased by only 0.5 GB. This is a real problem because it is causing out-of-memory issues when I run the notebook.<br>\nCan somebody help?</p>",
  "messages": [
    {
      "id": 2538246,
      "postDate": "2023-11-26T00:08:44.600Z",
      "content": "<p>It's likey due to a difference in environment variables.<br>\nTry running this before loading the model</p>\n<pre><code> \n.environ[]=\n</code></pre>",
      "rawMarkdown": "It's likey due to a difference in environment variables.\nTry running this before loading the model\n```\nimport os\nos.environ['TF_FORCE_GPU_ALLOW_GROWTH']='true'\n```",
      "votes": 2,
      "replies": [
        {
          "id": 2555804,
          "postDate": "2023-12-10T08:05:12.413Z",
          "content": "<p>Thanks for the tip, but I feel like it's not working with <strong>KerasCV</strong> as the GPU memory gets filled when I train the model using the <code>model.fit</code> method. However my oom issues were probably not linked to this but maybe to the batch size I was using</p>",
          "rawMarkdown": "Thanks for the tip, but I feel like it's not working with **KerasCV** as the GPU memory gets filled when I train the model using the `model.fit` method. However my oom issues were probably not linked to this but maybe to the batch size I was using"
        }
      ]
    },
    {
      "id": 2537227,
      "postDate": "2023-11-24T23:14:09.003Z",
      "content": "<p>Hi,<br>\nI've ran the pinned notebook <strong>[KerasCV] train and infer on thumbnails</strong> and I've noticed that when loading the model with the command <br>\n<code>resnet_backbone = keras_cv.models.ResNetV2Backbone.from_preset(\n    \"resnet152_v2\",\n)</code><br>\nthe GPU memory usage reaches about 15 GB and I don't understand why. I've ran the same command on a Colab notebook and the memory consumption increased by only 0.5 GB. This is a real problem because it is causing out-of-memory issues when I run the notebook.<br>\nCan somebody help?</p>",
      "rawMarkdown": "Hi,\nI've ran the pinned notebook **[KerasCV] train and infer on thumbnails** and I've noticed that when loading the model with the command \n`resnet_backbone = keras_cv.models.ResNetV2Backbone.from_preset(\n    \"resnet152_v2\",\n)`\nthe GPU memory usage reaches about 15 GB and I don't understand why. I've ran the same command on a Colab notebook and the memory consumption increased by only 0.5 GB. This is a real problem because it is causing out-of-memory issues when I run the notebook.\nCan somebody help?",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 2538246,
      "author_name": "KSMCG90",
      "author_url": "",
      "post_date": "2023-11-26T00:08:44.600000",
      "content": "<p>It's likey due to a difference in environment variables.<br>\nTry running this before loading the model</p>\n<pre><code> \n.environ[]=\n</code></pre>",
      "votes": 2,
      "replies": [
        {
          "id": 2555804,
          "author_name": "Gabriel Chehade",
          "author_url": "",
          "post_date": "2023-12-10T08:05:12.413000",
          "content": "<p>Thanks for the tip, but I feel like it's not working with <strong>KerasCV</strong> as the GPU memory gets filled when I train the model using the <code>model.fit</code> method. However my oom issues were probably not linked to this but maybe to the batch size I was using</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2538246": "It's likey due to a difference in environment variables.\nTry running this before loading the model\n```\nimport os\nos.environ['TF_FORCE_GPU_ALLOW_GROWTH']='true'\n```",
    "2537227": "Hi,\nI've ran the pinned notebook **[KerasCV] train and infer on thumbnails** and I've noticed that when loading the model with the command \n`resnet_backbone = keras_cv.models.ResNetV2Backbone.from_preset(\n    \"resnet152_v2\",\n)`\nthe GPU memory usage reaches about 15 GB and I don't understand why. I've ran the same command on a Colab notebook and the memory consumption increased by only 0.5 GB. This is a real problem because it is causing out-of-memory issues when I run the notebook.\nCan somebody help?"
  }
}