{ "id": 192786, "title": "Memory leak when opening many files - minimal complete example", "url": "/competitions/rsna-str-pulmonary-embolism-detection/discussion/192786", "author_name": "Alexander Soare", "post_date": "2020-10-23T09:39:33.815000", "votes": 0, "comment_count": 6, "views": 0, "content": "

At the risk of sounding like a ranting lunatic I'm opening yet another thread about this. I've also raised it as an issue here.

\n

The issue is that having a simple loop like this:

\n
import pandas as pd\nfrom tqdm.auto import tqdm\nfrom turbojpeg import TurboJPEG\nimport cv2\n\nJPEG = TurboJPEG()\n\ntrain_df = pd.read_csv('../input/pe-train-csv/train.csv')\n\nfor path in tqdm(train_df.image_path):\n#     cv2.imread(path)\n    with open(path, 'rb') as f:\n        img = JPEG.decode(f.read())\n
\n

Ends up killing Kaggle's memory quota before running through the whole loop. This is with this public 512x512 dataset.

\n

Here is a kernel demonstrating what I'm talking about.

\n

Any help would be appreciated!

", "messages": [ { "id": 1058083, "postDate": "2020-10-23T10:11:28.193Z", "content": "

You should close a file after reading it with f.close() line. Not sure if it will fix the issue.

", "rawMarkdown": "You should close a file after reading it with `f.close()` line. Not sure if it will fix the issue.", "replies": [ { "id": 1058085, "postDate": "2020-10-23T10:15:35.030Z", "content": "

The with means that happens automatically after exiting that block. Anyway, to be sure I tried using f.close() without the with and nothing changed.

", "rawMarkdown": "The `with` means that happens automatically after exiting that block. Anyway, to be sure I tried using `f.close()` without the `with` and nothing changed." }, { "id": 1058099, "postDate": "2020-10-23T10:43:57.700Z", "content": "

I see. Well, it was just an idea.

", "rawMarkdown": "I see. Well, it was just an idea." }, { "id": 1058100, "postDate": "2020-10-23T10:44:36.470Z", "content": "

You can try to use garbage collector gc.collect()

", "rawMarkdown": "You can try to use garbage collector `gc.collect()`" }, { "id": 1058113, "postDate": "2020-10-23T10:59:57.430Z", "content": "
\n

I see. Well, it was just an idea.

\n
\n

Yes, don't take any offense!

\n

And I've also tried gc.collect() after del img. No help. I'm kind of wondering how anyone else did this competition. Is it just me struggling to even train a model?

", "rawMarkdown": "> I see. Well, it was just an idea.\n\nYes, don't take any offense!\n\nAnd I've also tried `gc.collect()` after `del img`. No help. I'm kind of wondering how anyone else did this competition. Is it just me struggling to even train a model?" }, { "id": 1058124, "postDate": "2020-10-23T11:11:23.303Z", "content": "

Indeed, it's quite hard to train model which will beat new baseline with only Kaggle resources. I'm also having hard time with that. I try to use TF records instead of images.

", "rawMarkdown": "Indeed, it's quite hard to train model which will beat new baseline with only Kaggle resources. I'm also having hard time with that. I try to use TF records instead of images." } ] }, { "id": 1058052, "postDate": "2020-10-23T09:39:33.817Z", "content": "

At the risk of sounding like a ranting lunatic I'm opening yet another thread about this. I've also raised it as an issue here.

\n

The issue is that having a simple loop like this:

\n
import pandas as pd\nfrom tqdm.auto import tqdm\nfrom turbojpeg import TurboJPEG\nimport cv2\n\nJPEG = TurboJPEG()\n\ntrain_df = pd.read_csv('../input/pe-train-csv/train.csv')\n\nfor path in tqdm(train_df.image_path):\n#     cv2.imread(path)\n    with open(path, 'rb') as f:\n        img = JPEG.decode(f.read())\n
\n

Ends up killing Kaggle's memory quota before running through the whole loop. This is with this public 512x512 dataset.

\n

Here is a kernel demonstrating what I'm talking about.

\n

Any help would be appreciated!

", "rawMarkdown": "At the risk of sounding like a ranting lunatic I'm opening yet another thread about [this](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/192623). I've also raised it as an issue [here](https://www.kaggle.com/product-feedback/192703).\n\nThe issue is that having a simple loop like this:\n\n```\nimport pandas as pd\nfrom tqdm.auto import tqdm\nfrom turbojpeg import TurboJPEG\nimport cv2\n\nJPEG = TurboJPEG()\n\ntrain_df = pd.read_csv('../input/pe-train-csv/train.csv')\n\nfor path in tqdm(train_df.image_path):\n# cv2.imread(path)\n with open(path, 'rb') as f:\n img = JPEG.decode(f.read())\n```\n\nEnds up killing Kaggle's memory quota before running through the whole loop. This is with this [public 512x512 dataset](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/185742).\n\nHere is a [kernel](https://www.kaggle.com/alexandersoare/memory-leak-example?scriptVersionId=45325483) demonstrating what I'm talking about.\n\nAny help would be appreciated!" } ], "comments": [ { "id": 1058083, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2020-10-23T10:11:28.193000", "content": "

You should close a file after reading it with f.close() line. Not sure if it will fix the issue.

", "votes": 0, "replies": [ { "id": 1058085, "author_name": "Alexander Soare", "author_url": "", "post_date": "2020-10-23T10:15:35.030000", "content": "

The with means that happens automatically after exiting that block. Anyway, to be sure I tried using f.close() without the with and nothing changed.

", "votes": 0, "replies": [] }, { "id": 1058099, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2020-10-23T10:43:57.700000", "content": "

I see. Well, it was just an idea.

", "votes": 0, "replies": [] }, { "id": 1058100, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2020-10-23T10:44:36.470000", "content": "

You can try to use garbage collector gc.collect()

", "votes": 0, "replies": [] }, { "id": 1058113, "author_name": "Alexander Soare", "author_url": "", "post_date": "2020-10-23T10:59:57.430000", "content": "
\n

I see. Well, it was just an idea.

\n
\n

Yes, don't take any offense!

\n

And I've also tried gc.collect() after del img. No help. I'm kind of wondering how anyone else did this competition. Is it just me struggling to even train a model?

", "votes": 0, "replies": [] }, { "id": 1058124, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2020-10-23T11:11:23.303000", "content": "

Indeed, it's quite hard to train model which will beat new baseline with only Kaggle resources. I'm also having hard time with that. I try to use TF records instead of images.

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1058083": "You should close a file after reading it with `f.close()` line. Not sure if it will fix the issue.", "1058052": "At the risk of sounding like a ranting lunatic I'm opening yet another thread about [this](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/192623). I've also raised it as an issue [here](https://www.kaggle.com/product-feedback/192703).\n\nThe issue is that having a simple loop like this:\n\n```\nimport pandas as pd\nfrom tqdm.auto import tqdm\nfrom turbojpeg import TurboJPEG\nimport cv2\n\nJPEG = TurboJPEG()\n\ntrain_df = pd.read_csv('../input/pe-train-csv/train.csv')\n\nfor path in tqdm(train_df.image_path):\n# cv2.imread(path)\n with open(path, 'rb') as f:\n img = JPEG.decode(f.read())\n```\n\nEnds up killing Kaggle's memory quota before running through the whole loop. This is with this [public 512x512 dataset](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/185742).\n\nHere is a [kernel](https://www.kaggle.com/alexandersoare/memory-leak-example?scriptVersionId=45325483) demonstrating what I'm talking about.\n\nAny help would be appreciated!" } }