I am getting this strange error even though my run with provided test images works fine and I have set a limit of image size so it won't kill the small RAM… is there something dramatically different with the used machine for submission? Also, the submission details state we have 9 hours runtime, and this kernel dies after 2 hours or so…
\n\n\nSubmission kernel: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet
\n
\n Conversion kernel: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-eda-load-crop-images
to show that I can convert all images up to 4e9 pixels without problem…
\nUPDATE:
\na good finding from: https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/336819
\n\n\nIf you don't use gpu, you have 16 Gb of RAM, but if you use gpu, it's reduced to 13 Gb on the kaggle notebook.
\n
Also, I was able to load and covert images up to 1.5e9 pixels with GPU
", "messages": [ { "id": 1851612, "postDate": "2022-07-11T11:51:32.903Z", "content": "I am getting this strange error even though my run with provided test images works fine and I have set a limit of image size so it won't kill the small RAM… is there something dramatically different with the used machine for submission? Also, the submission details state we have 9 hours runtime, and this kernel dies after 2 hours or so…
\n\n\nSubmission kernel: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet
\n
\n Conversion kernel: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-eda-load-crop-images
to show that I can convert all images up to 4e9 pixels without problem…
\nUPDATE:
\na good finding from: https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/336819
\n\n\nIf you don't use gpu, you have 16 Gb of RAM, but if you use gpu, it's reduced to 13 Gb on the kaggle notebook.
\n
Also, I was able to load and covert images up to 1.5e9 pixels with GPU
", "rawMarkdown": "I am getting this strange error even though my run with provided test images works fine and I have set a limit of image size so it won't kill the small RAM... is there something dramatically different with the used machine for submission? Also, the submission details state we have 9 hours runtime, and this kernel dies after 2 hours or so...\n\n> Submission kernel: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet\n> Conversion kernel: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-eda-load-crop-images\n\nto show that I can convert all images up to 4e9 pixels without problem...\n\n**UPDATE:**\n\na good finding from: https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/336819\n\n> If you don't use gpu, you have 16 Gb of RAM, but if you use gpu, it's reduced to 13 Gb on the kaggle notebook.\n\nAlso, I was able to load and covert images up to 1.5e9 pixels with GPU", "votes": 9 }, { "id": 1907793, "postDate": "2022-08-21T04:52:48.877Z", "content": "Hello! This is probably a re-working of Topic Author's solution, but I would like to make a few comments.
\nI too encountered the \"Notebook Exceeded Allowed Compute\" in my submission pipeline.
\nIt seems that the OOM was occurring when loading images using the tifffile.
Since the RAM limit for a GPU instance is 13 GB, I used the train data to investigate the maximum file size (width x height) that could not be loaded due to this limitation.
\nhttps://www.kaggle.com/code/bobfromjapan/mayo-filesize-inspection
The results showed that the largest image in train data, 6baf51_0.tif, has a size of 4896084492, which would exceed the 13GB System RAM of the GPU instance.
\nThe second largest image, b894f4_0.tif, is 4131662535 and can be loaded.
I tried to skip processing when the size exceeds 4131662535 as a workaround to avoid the \"Notebook Exceeded Allowed Compute\". Below is an example.
too_big_for_process = []\nfor path in test_image_paths:\n\n print(path)\n slide = OpenSlide(path)\n\n if slide.dimensions[0]*slide.dimensions[1] < 4131662535:\n image_id = os.path.splitext(os.path.basename(path))[0]\n image = tiff.imread(path)\n print(f\"{image.shape}\")\n cv2.imwrite(os.path.join(output_dir, f\"{image_id}.jpg\"), image[::scale,::scale,::-1])\n del image\n gc.collect()\n else:\n print(\"Skip process for avoiding OOM possibility.\")\n too_big_for_process.append(path)\nAfter the inference, if there were images that could not be processed, the value of prediction was given as a constant. This avoids the \"Notebook Exceeded Allowed Compute\" and the submission succeeds!
\nif len(too_big_for_process)>0:\n for i in too_big_for_process:\n preds.append((i, 0.82, 0.28))\nI ran into a time error, made it 20% faster to barely finish it in time with 1 fold (HUGE OVERFIT BTW, leaderboard is nothing as training on my set), although I am unsure, what will happen if my run 5 folds instead
", "rawMarkdown": "I ran into a time error, made it 20% faster to barely finish it in time with 1 fold (HUGE OVERFIT BTW, leaderboard is nothing as training on my set), although I am unsure, what will happen if my run 5 folds instead ", "replies": [ { "id": 1853807, "postDate": "2022-07-13T06:51:09.570Z", "content": "So you say it is timeout issue?
", "rawMarkdown": "So you say it is timeout issue? " }, { "id": 1853996, "postDate": "2022-07-13T11:20:06.410Z", "content": "I had to make tiles, which if I load images carefully avoiding memory errors, takes too much time…
", "rawMarkdown": "I had to make tiles, which if I load images carefully avoiding memory errors, takes too much time..." }, { "id": 1854026, "postDate": "2022-07-13T11:45:35.370Z", "content": "what does too much time mean? mine is killed after bout 2h of runtime…
\neven they state:
\n", "rawMarkdown": "what does too much time mean? mine is killed after bout 2h of runtime...\neven they state:\n\n> CPU Notebook <= 9 hours run-time\nGPU Notebook <= 9 hours run-time\nInternet access disabled" }, { "id": 1854098, "postDate": "2022-07-13T12:35:34.157Z", "content": "CPU Notebook <= 9 hours run-time
\n
\n GPU Notebook <= 9 hours run-time
\n Internet access disabled
It means, mine took more than 9 hours to executue, so if there occurs no error, instead of showing me the score it will show 'Notebook Timeout Error'
", "rawMarkdown": "It means, mine took more than 9 hours to executue, so if there occurs no error, instead of showing me the score it will show 'Notebook Timeout Error' " }, { "id": 1854295, "postDate": "2022-07-13T15:28:50.637Z", "content": "I see, the \"Notebook Timeout Error\" is quite clear… :)
", "rawMarkdown": "I see, the \"Notebook Timeout Error\" is quite clear... :)" } ] }, { "id": 1853553, "postDate": "2022-07-13T00:25:39.647Z", "content": "I think that the test is huge.. I keep getting errors every day when I try to run a submission..
\nMaybe try converting them in batches and you should see an improvement.
Hi, may you elaborate more on what you mean converting in batches?
", "rawMarkdown": "Hi, may you elaborate more on what you mean converting in batches? " } ] }, { "id": 1852084, "postDate": "2022-07-11T18:41:59.003Z", "content": "The real test set is MUCH larger than the provided test set. You need to debug with a representative dataset.
\nI recommend you do the following to debug:
\nYes, that is what I tried and I do not have any problem with the train dataset as I skipped all too large image even before loading…
", "rawMarkdown": "Yes, that is what I tried and I do not have any problem with the train dataset as I skipped all too large image even before loading... " }, { "id": 1852157, "postDate": "2022-07-11T20:46:57.130Z", "content": "So if you run your pipeline on a random sample of 280 training images (Save and Run All), you don't encounter any errors?
", "rawMarkdown": "So if you run your pipeline on a random sample of 280 training images (Save and Run All), you don't encounter any errors?" }, { "id": 1852175, "postDate": "2022-07-11T21:08:24.773Z", "content": "correct, I can convert all 751 out of 754 images from the training dataset on the limited instance and the three remaining are safely skipped… 👀
", "rawMarkdown": "correct, I can convert all 751 out of 754 images from the training dataset on the limited instance and the three remaining are safely skipped... 👀" }, { "id": 1852372, "postDate": "2022-07-12T03:08:03.150Z", "content": "Hmm, I'm not sure. Is this the notebook that crashes?
\nhttps://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet
", "rawMarkdown": "Hmm, I'm not sure. Is this the notebook that crashes?\n\nhttps://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet" }, { "id": 1852570, "postDate": "2022-07-12T06:56:09.523Z", "content": "Yes, this one
", "rawMarkdown": "Yes, this one " } ] }, { "id": 1851861, "postDate": "2022-07-11T15:30:42.160Z", "content": "Image.load itself will fail if the tiff is too big… I had my successful submission by using pyvips
", "rawMarkdown": "Image.load itself will fail if the tiff is too big... I had my successful submission by using pyvips", "replies": [ { "id": 1851879, "postDate": "2022-07-11T15:45:06.597Z", "content": "I see but that is not part of the environment so you have to download the OS package and then save it as a dataset…
\nbut I do not use Image.load, just Image.open, so do you think that these images can't be even opened? :(
Sry, I meant, Image.open opens the whole image, Image.load does not exist, typo, yes I think these images can not even be opened
", "rawMarkdown": "Sry, I meant, Image.open opens the whole image, Image.load does not exist, typo, yes I think these images can not even be opened" }, { "id": 1851979, "postDate": "2022-07-11T17:23:34.740Z", "content": "what I see, Image.open doe snot take almost any memory, it does when you call Image.read
I saw your submission notebook, I suggest you use pylibs anyways, it works for me…, although I have currently submitted my submission notebook, it will verify if pylibs works with very big sizes if this submission does not give an error (I first used it, imediately converting images to HigherThan1024x1024, it did not give me a submission error)
", "rawMarkdown": "I saw your submission notebook, I suggest you use pylibs anyways, it works for me..., although I have currently submitted my submission notebook, it will verify if pylibs works with very big sizes if this submission does not give an error (I first used it, imediately converting images to HigherThan1024x1024, it did not give me a submission error)" }, { "id": 1852140, "postDate": "2022-07-11T20:01:13.247Z", "content": "Do you have/ mind sharing your notebook with this loading? =)
", "rawMarkdown": "Do you have/ mind sharing your notebook with this loading? =) " }, { "id": 1856272, "postDate": "2022-07-15T08:26:50.497Z", "content": "just verified that Image.open does not load any data and I can apply it to all images in the test set :)
\n", "rawMarkdown": "just verified that `Image.open` does not load any data and I can apply it to all images in the test set :)\n> https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-fake-stats-prediction" }, { "id": 1894131, "postDate": "2022-08-11T09:37:49.917Z", "content": "https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-fake-stats-prediction
\n
when using pyvips, have you found memory leak?
\nonly run this line for every WSI, the memory will increase to 13g quikly
\nimg = pyvips.Image.thumbnail(path, width).numpy()
Have you tried to set this env. variable? os.environ['VIPS_DISC_THRESHOLD'] = '7gb'
no…
\ni think memory will also increase to 7gb, even with os.environ['VIPS_DISC_THRESHOLD'] = '7gb' , it's a memory leak bug of pyvips libary.
\neven though memory increases to 13g, it will not throw OOM error, the code still can run to the final submission.
How many threads are you running? Because eventually the 7gb would be used baby each not together…
", "rawMarkdown": "How many threads are you running? Because eventually the 7gb would be used baby each not together..." } ] } ], "comments": [ { "id": 1907793, "author_name": "bobfromjapan", "author_url": "", "post_date": "2022-08-21T04:52:48.877000", "content": "Hello! This is probably a re-working of Topic Author's solution, but I would like to make a few comments.
\nI too encountered the \"Notebook Exceeded Allowed Compute\" in my submission pipeline.
\nIt seems that the OOM was occurring when loading images using the tifffile.
Since the RAM limit for a GPU instance is 13 GB, I used the train data to investigate the maximum file size (width x height) that could not be loaded due to this limitation.
\nhttps://www.kaggle.com/code/bobfromjapan/mayo-filesize-inspection
The results showed that the largest image in train data, 6baf51_0.tif, has a size of 4896084492, which would exceed the 13GB System RAM of the GPU instance.
\nThe second largest image, b894f4_0.tif, is 4131662535 and can be loaded.
I tried to skip processing when the size exceeds 4131662535 as a workaround to avoid the \"Notebook Exceeded Allowed Compute\". Below is an example.
too_big_for_process = []\nfor path in test_image_paths:\n\n print(path)\n slide = OpenSlide(path)\n\n if slide.dimensions[0]*slide.dimensions[1] < 4131662535:\n image_id = os.path.splitext(os.path.basename(path))[0]\n image = tiff.imread(path)\n print(f\"{image.shape}\")\n cv2.imwrite(os.path.join(output_dir, f\"{image_id}.jpg\"), image[::scale,::scale,::-1])\n del image\n gc.collect()\n else:\n print(\"Skip process for avoiding OOM possibility.\")\n too_big_for_process.append(path)\nAfter the inference, if there were images that could not be processed, the value of prediction was given as a constant. This avoids the \"Notebook Exceeded Allowed Compute\" and the submission succeeds!
\nif len(too_big_for_process)>0:\n for i in too_big_for_process:\n preds.append((i, 0.82, 0.28))\nI ran into a time error, made it 20% faster to barely finish it in time with 1 fold (HUGE OVERFIT BTW, leaderboard is nothing as training on my set), although I am unsure, what will happen if my run 5 folds instead
", "votes": 0, "replies": [ { "id": 1853807, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-13T06:51:09.570000", "content": "So you say it is timeout issue?
", "votes": 0, "replies": [] }, { "id": 1853996, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-07-13T11:20:06.410000", "content": "I had to make tiles, which if I load images carefully avoiding memory errors, takes too much time…
", "votes": 0, "replies": [] }, { "id": 1854026, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-13T11:45:35.370000", "content": "what does too much time mean? mine is killed after bout 2h of runtime…
\neven they state:
\n", "votes": 0, "replies": [] }, { "id": 1854098, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-07-13T12:35:34.157000", "content": "CPU Notebook <= 9 hours run-time
\n
\n GPU Notebook <= 9 hours run-time
\n Internet access disabled
It means, mine took more than 9 hours to executue, so if there occurs no error, instead of showing me the score it will show 'Notebook Timeout Error'
", "votes": 0, "replies": [] }, { "id": 1854295, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-13T15:28:50.637000", "content": "I see, the \"Notebook Timeout Error\" is quite clear… :)
", "votes": 0, "replies": [] } ] }, { "id": 1853553, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-13T00:25:39.647000", "content": "I think that the test is huge.. I keep getting errors every day when I try to run a submission..
\nMaybe try converting them in batches and you should see an improvement.
Hi, may you elaborate more on what you mean converting in batches?
", "votes": 0, "replies": [] } ] }, { "id": 1852084, "author_name": "Darien Schettler", "author_url": "", "post_date": "2022-07-11T18:41:59.003000", "content": "The real test set is MUCH larger than the provided test set. You need to debug with a representative dataset.
\nI recommend you do the following to debug:
\nYes, that is what I tried and I do not have any problem with the train dataset as I skipped all too large image even before loading…
", "votes": 0, "replies": [] }, { "id": 1852157, "author_name": "Darien Schettler", "author_url": "", "post_date": "2022-07-11T20:46:57.130000", "content": "So if you run your pipeline on a random sample of 280 training images (Save and Run All), you don't encounter any errors?
", "votes": 0, "replies": [] }, { "id": 1852175, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-11T21:08:24.773000", "content": "correct, I can convert all 751 out of 754 images from the training dataset on the limited instance and the three remaining are safely skipped… 👀
", "votes": 0, "replies": [] }, { "id": 1852372, "author_name": "Darien Schettler", "author_url": "", "post_date": "2022-07-12T03:08:03.150000", "content": "Hmm, I'm not sure. Is this the notebook that crashes?
\nhttps://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet
", "votes": 0, "replies": [] }, { "id": 1852570, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-12T06:56:09.523000", "content": "Yes, this one
", "votes": 0, "replies": [] } ] }, { "id": 1851861, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-07-11T15:30:42.160000", "content": "Image.load itself will fail if the tiff is too big… I had my successful submission by using pyvips
", "votes": 0, "replies": [ { "id": 1851879, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-11T15:45:06.597000", "content": "I see but that is not part of the environment so you have to download the OS package and then save it as a dataset…
\nbut I do not use Image.load, just Image.open, so do you think that these images can't be even opened? :(
Sry, I meant, Image.open opens the whole image, Image.load does not exist, typo, yes I think these images can not even be opened
", "votes": 0, "replies": [] }, { "id": 1851979, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-11T17:23:34.740000", "content": "what I see, Image.open doe snot take almost any memory, it does when you call Image.read
I saw your submission notebook, I suggest you use pylibs anyways, it works for me…, although I have currently submitted my submission notebook, it will verify if pylibs works with very big sizes if this submission does not give an error (I first used it, imediately converting images to HigherThan1024x1024, it did not give me a submission error)
", "votes": 0, "replies": [] }, { "id": 1852140, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-11T20:01:13.247000", "content": "Do you have/ mind sharing your notebook with this loading? =)
", "votes": 0, "replies": [] }, { "id": 1856272, "author_name": "Jirka", "author_url": "", "post_date": "2022-07-15T08:26:50.497000", "content": "just verified that Image.open does not load any data and I can apply it to all images in the test set :)
\n", "votes": 0, "replies": [] }, { "id": 1894131, "author_name": "ddt0106", "author_url": "", "post_date": "2022-08-11T09:37:49.917000", "content": "https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-fake-stats-prediction
\n
when using pyvips, have you found memory leak?
\nonly run this line for every WSI, the memory will increase to 13g quikly
\nimg = pyvips.Image.thumbnail(path, width).numpy()
Have you tried to set this env. variable? os.environ['VIPS_DISC_THRESHOLD'] = '7gb'
no…
\ni think memory will also increase to 7gb, even with os.environ['VIPS_DISC_THRESHOLD'] = '7gb' , it's a memory leak bug of pyvips libary.
\neven though memory increases to 13g, it will not throw OOM error, the code still can run to the final submission.
How many threads are you running? Because eventually the 7gb would be used baby each not together…
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1851612": "I am getting this strange error even though my run with provided test images works fine and I have set a limit of image size so it won't kill the small RAM... is there something dramatically different with the used machine for submission? Also, the submission details state we have 9 hours runtime, and this kernel dies after 2 hours or so...\n\n> Submission kernel: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-baseline-flash-effnet\n> Conversion kernel: https://www.kaggle.com/code/jirkaborovec/bloodclots-classif-eda-load-crop-images\n\nto show that I can convert all images up to 4e9 pixels without problem...\n\n**UPDATE:**\n\na good finding from: https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/336819\n\n> If you don't use gpu, you have 16 Gb of RAM, but if you use gpu, it's reduced to 13 Gb on the kaggle notebook.\n\nAlso, I was able to load and covert images up to 1.5e9 pixels with GPU", "1907793": "Hello! This is probably a re-working of Topic Author's solution, but I would like to make a few comments.\n\nI too encountered the \"Notebook Exceeded Allowed Compute\" in my submission pipeline.\nIt seems that the OOM was occurring when loading images using the `tifffile`.\n\nSince the RAM limit for a GPU instance is 13 GB, I used the train data to investigate the maximum file size (width x height) that could not be loaded due to this limitation.\nhttps://www.kaggle.com/code/bobfromjapan/mayo-filesize-inspection\n\nThe results showed that the largest image in train data, `6baf51_0.tif`, has a size of `4896084492`, which would exceed the 13GB System RAM of the GPU instance.\nThe second largest image, `b894f4_0.tif`, is `4131662535` and can be loaded.\n\nI tried to skip processing when the size exceeds `4131662535` as a workaround to avoid the \"Notebook Exceeded Allowed Compute\". Below is an example.\n\n```\ntoo_big_for_process = []\nfor path in test_image_paths:\n\n print(path)\n slide = OpenSlide(path)\n\n if slide.dimensions[0]*slide.dimensions[1] < 4131662535:\n image_id = os.path.splitext(os.path.basename(path))[0]\n image = tiff.imread(path)\n print(f\"{image.shape}\")\n cv2.imwrite(os.path.join(output_dir, f\"{image_id}.jpg\"), image[::scale,::scale,::-1])\n del image\n gc.collect()\n else:\n print(\"Skip process for avoiding OOM possibility.\")\n too_big_for_process.append(path)\n\n```\nAfter the inference, if there were images that could not be processed, the value of prediction was given as a constant. This avoids the \"Notebook Exceeded Allowed Compute\" and the submission succeeds!\n\n```\nif len(too_big_for_process)>0:\n for i in too_big_for_process:\n preds.append((i, 0.82, 0.28))\n```", "1853611": "I ran into a time error, made it 20% faster to barely finish it in time with 1 fold (HUGE OVERFIT BTW, leaderboard is nothing as training on my set), although I am unsure, what will happen if my run 5 folds instead ", "1853553": "I think that the test is huge.. I keep getting errors every day when I try to run a submission..\nMaybe try converting them in batches and you should see an improvement.\n\n", "1852084": "The real test set is MUCH larger than the provided test set. You need to debug with a representative dataset.\n\nI recommend you do the following to debug:\n\n- instead of using the provided test data use a portion of the training data equivalent to the size of the hidden test set\n- Use something like debug==len(ss_df)==4 and then use that to initialize a more representative dataset.\n- if I had to guess I’d say PIL is holding some RAM and causing an eventual OOM. \n", "1851861": "Image.load itself will fail if the tiff is too big... I had my successful submission by using pyvips" } }