{
  "id": 111398,
  "title": "Too high RAM usage",
  "url": "/competitions/rsna-intracranial-hemorrhage-detection/discussion/111398",
  "author_name": "Basil Pupkins",
  "post_date": "2019-10-05T10:57:16.641000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hello, Kagglers!\nForming up a training dataset I ran into a RAM problem: RAM is being consumed too fast and I cannot find the reason for it. I tried solving it in the following steps:\n-  Deleting unused variables\n-  Using garbage collector, i.e.: gc.collect()\n-  Using small datatypes where it's possible: uint8 and float16.</p>\n\n<p>Moreover, I checked the sizes of variables in the following way:\n<code>\nfrom sys import getsizeof\nvars = dir() #get all the variable names\ntotal_sum = 0\nfor v in vars: #for every variable name\n    bts = getsizeof(eval(v)) #get its size in bytes\n    mbts = bts/1000000  #get its size in megabytes\n    if mbts&gt;1: #I display only the\n                      #sizes of variables greater than 1 MB\n        print(v + \" \"+str(mbts)) \n    total_sum += bts #counting total sum of bytes\nprint(\"total_sum: \" + str(total_sum/1000000)) #total sum in MB\n</code>\nI got the following output:</p>\n\n<blockquote>\n  <p>X 14.680192</p>\n  \n  <p>X_any 3340.861584</p>\n  \n  <p>aligned_file_nums 5.394128</p>\n  \n  <p>id_index_dict 20.971616</p>\n  \n  <p>index_id_dict 20.971616</p>\n  \n  <p>labels 4.04566</p>\n  \n  <p>train_files 6.068432</p>\n  \n  <p>total_sum: 3413.844825</p>\n</blockquote>\n\n<p>i.e. ~3.4 GB.\nBut the RAM consuming meter shows, that I use &gt;12 GB.\nMaybe, someone encountered the same problem and knows the reason? I would be very grateful for your saggestions how to lower down RAM consumption or explanation, why it happens, maybe, I simply do not know or understand something.</p>",
  "messages": [
    {
      "id": 641940,
      "postDate": "2019-10-05T10:57:16.643Z",
      "content": "<p>Hello, Kagglers!\nForming up a training dataset I ran into a RAM problem: RAM is being consumed too fast and I cannot find the reason for it. I tried solving it in the following steps:\n-  Deleting unused variables\n-  Using garbage collector, i.e.: gc.collect()\n-  Using small datatypes where it's possible: uint8 and float16.</p>\n\n<p>Moreover, I checked the sizes of variables in the following way:\n<code>\nfrom sys import getsizeof\nvars = dir() #get all the variable names\ntotal_sum = 0\nfor v in vars: #for every variable name\n    bts = getsizeof(eval(v)) #get its size in bytes\n    mbts = bts/1000000  #get its size in megabytes\n    if mbts&gt;1: #I display only the\n                      #sizes of variables greater than 1 MB\n        print(v + \" \"+str(mbts)) \n    total_sum += bts #counting total sum of bytes\nprint(\"total_sum: \" + str(total_sum/1000000)) #total sum in MB\n</code>\nI got the following output:</p>\n\n<blockquote>\n  <p>X 14.680192</p>\n  \n  <p>X_any 3340.861584</p>\n  \n  <p>aligned_file_nums 5.394128</p>\n  \n  <p>id_index_dict 20.971616</p>\n  \n  <p>index_id_dict 20.971616</p>\n  \n  <p>labels 4.04566</p>\n  \n  <p>train_files 6.068432</p>\n  \n  <p>total_sum: 3413.844825</p>\n</blockquote>\n\n<p>i.e. ~3.4 GB.\nBut the RAM consuming meter shows, that I use &gt;12 GB.\nMaybe, someone encountered the same problem and knows the reason? I would be very grateful for your saggestions how to lower down RAM consumption or explanation, why it happens, maybe, I simply do not know or understand something.</p>",
      "rawMarkdown": "Hello, Kagglers!\nForming up a training dataset I ran into a RAM problem: RAM is being consumed too fast and I cannot find the reason for it. I tried solving it in the following steps:\n-  Deleting unused variables\n-  Using garbage collector, i.e.: gc.collect()\n-  Using small datatypes where it's possible: uint8 and float16.\n\nMoreover, I checked the sizes of variables in the following way:\n```\nfrom sys import getsizeof\nvars = dir() #get all the variable names\ntotal_sum = 0\nfor v in vars: #for every variable name\n    bts = getsizeof(eval(v)) #get its size in bytes\n    mbts = bts/1000000  #get its size in megabytes\n    if mbts&gt;1: #I display only the\n                      #sizes of variables greater than 1 MB\n        print(v + \" \"+str(mbts)) \n    total_sum += bts #counting total sum of bytes\nprint(\"total_sum: \" + str(total_sum/1000000)) #total sum in MB\n```\nI got the following output:\n&gt; X 14.680192\n\n&gt; X_any 3340.861584\n\n&gt;aligned\\_file_nums 5.394128\n\n&gt;id\\_index_dict 20.971616\n\n&gt;index\\_id_dict 20.971616\n\n&gt;labels 4.04566\n\n&gt;train_files 6.068432\n\n&gt;total_sum: 3413.844825\n\n\ni.e. ~3.4 GB.\nBut the RAM consuming meter shows, that I use &gt;12 GB.\nMaybe, someone encountered the same problem and knows the reason? I would be very grateful for your saggestions how to lower down RAM consumption or explanation, why it happens, maybe, I simply do not know or understand something.",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "641940": "Hello, Kagglers!\nForming up a training dataset I ran into a RAM problem: RAM is being consumed too fast and I cannot find the reason for it. I tried solving it in the following steps:\n-  Deleting unused variables\n-  Using garbage collector, i.e.: gc.collect()\n-  Using small datatypes where it's possible: uint8 and float16.\n\nMoreover, I checked the sizes of variables in the following way:\n```\nfrom sys import getsizeof\nvars = dir() #get all the variable names\ntotal_sum = 0\nfor v in vars: #for every variable name\n    bts = getsizeof(eval(v)) #get its size in bytes\n    mbts = bts/1000000  #get its size in megabytes\n    if mbts&gt;1: #I display only the\n                      #sizes of variables greater than 1 MB\n        print(v + \" \"+str(mbts)) \n    total_sum += bts #counting total sum of bytes\nprint(\"total_sum: \" + str(total_sum/1000000)) #total sum in MB\n```\nI got the following output:\n&gt; X 14.680192\n\n&gt; X_any 3340.861584\n\n&gt;aligned\\_file_nums 5.394128\n\n&gt;id\\_index_dict 20.971616\n\n&gt;index\\_id_dict 20.971616\n\n&gt;labels 4.04566\n\n&gt;train_files 6.068432\n\n&gt;total_sum: 3413.844825\n\n\ni.e. ~3.4 GB.\nBut the RAM consuming meter shows, that I use &gt;12 GB.\nMaybe, someone encountered the same problem and knows the reason? I would be very grateful for your saggestions how to lower down RAM consumption or explanation, why it happens, maybe, I simply do not know or understand something."
  }
}