Notebook Exceeded Allowed Compute: This indicates you have violated a code requirement constraint during the rerun. This includes limitations in the execution environment, for example requesting more RAM or disk than available, or competition constraints, such as input data source type or size limits.
\nI've already make the reading and preprocessing of images to be done in batches of 4 images each time, after each I do gcCollect to free RAM and shutil.rmtree(output_path) to free disk space.
\nCompetition says I'm allowed to use pre-trained model, so I trained my model and saved it in a public dataset and reads it in instead of training from scratch so I don't need to process all files from \"/train\" folder anymore.
\nMy notebook only reads in the model weights saved from a public dataset and the competition files as inputs.
\nIt straight away read the \"/test\" folder and start processing them in batches, predict them and store result in a dataframe.
I feel the toughest part of this competition is in figuring this out..which demotivates me a lot since it's not about solving the problem statement but about the constraint of the environments…I simply cannot get any submission working after more than 12 attempts. What could possibly be wrong? Is openCV completely unusable here?
\nI read some earlier discussion ppl just switch to different image processing library is that the only way?
\nAnyone manage to post a submission still using openCV? I'd like some guidance please
@raymondsky most of us have gone through this issue in the beginning. Here and here are some previous threads of discussions on this, might be helpful to go through these and then clarify what worked and what did not. Maybe then some of us can be in a better position to help.
\nAs far as your explanation reveals, image reading is the likely issue - in my limited experience, there are large enough images in the hidden test dataset that do not allow reading image file and then even reasonable RAM intensive processing/computation, especially with the GPU where the RAM is only 13 Gb. If you are trying to read 4 images at a time before inference, definitely start from there - need to read one at a time, aggregate the results, perform inference etc.
\nHope this helps.
", "rawMarkdown": "@raymondsky most of us have gone through this issue in the beginning. [Here](https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/340944) and [here](https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/336484) are some previous threads of discussions on this, might be helpful to go through these and then clarify what worked and what did not. Maybe then some of us can be in a better position to help.\n\nAs far as your explanation reveals, image reading is the likely issue - in my limited experience, there are large enough images in the hidden test dataset that do not allow reading image file and then even reasonable RAM intensive processing/computation, especially with the GPU where the RAM is only 13 Gb. If you are trying to read 4 images at a time before inference, definitely start from there - need to read one at a time, aggregate the results, perform inference etc. \n\nHope this helps.", "votes": 1, "replies": [ { "id": 1918487, "postDate": "2022-08-29T15:57:40.193Z", "content": "Thanks, seems one of the solution is to simply skip files that's too large to read into the available RAM….
\nI'll give that a go hopefully finally resolve this nightmare problem I've been facing
", "rawMarkdown": "Thanks, seems one of the solution is to simply skip files that's too large to read into the available RAM....\n\nI'll give that a go hopefully finally resolve this nightmare problem I've been facing", "votes": 1 }, { "id": 1918502, "postDate": "2022-08-29T16:11:04.333Z", "content": "All the best :)
", "rawMarkdown": "All the best :)", "votes": 1 }, { "id": 1919963, "postDate": "2022-08-30T19:17:18.307Z", "content": "I've applied the same threshold and try exception from this notebook:
\nhttps://www.kaggle.com/code/realneuralnetwork/cnn-strip-ai-inference/notebook?scriptVersionId=100707854
still no luck….still hitting same error
\nNotebook Exceeded Allowed Compute
\nzzzzzzzzzzzzzzz
@raymondsky what file size limit have you used? Depending on your code's RAM usage, and with GPU, anything above 1.2 Gb might still need to be treated in a different way.
\nAlso, you mentioned you are inferring on 4 images at a time. Could you try one image at a time, I'm sure that should work with a conservative image size limit. Beyond that there's no reason unless your code is eating RAM for lunch while inference :)
", "rawMarkdown": "@raymondsky what file size limit have you used? Depending on your code's RAM usage, and with GPU, anything above 1.2 Gb might still need to be treated in a different way.\n\nAlso, you mentioned you are inferring on 4 images at a time. Could you try one image at a time, I'm sure that should work with a conservative image size limit. Beyond that there's no reason unless your code is eating RAM for lunch while inference :)" }, { "id": 1920235, "postDate": "2022-08-31T02:47:05.580Z", "content": "I've already done that mate, 1 patient image at a time, T-T
\nI read, resize and save a copy and then immediately delete and gc.collect()
\nHowever as part of the solution, I did do cv findContours and create new subimages….before i save the copies,
\nI also made sure I delete the saved images before next loop after getting prediction results.
\nI lowered my threshold form 1500000000 to 1000000000 and gone passed the error now, but I get Submission Scoring Error this time…..
If we skipped those larger than 1000000000 size how do we predict for them?
\nI saw in the sample notebook we set np.zeros((512,512,3), np.uint8)
\nhow can the ML predict this dummy image accurately?
With GPU and inference processing, I've also found it only possible to read upto 1.2 Gb image, but without any model running, I was able to check up to 2 Gb image reading without errors. If you are downscaling images before inference, one method could be to first read only large images, downscale and write the smaller sized images, then continue such that all images are below a certain size. I have tried this with training data set, which does not have many images above 1Gb in size.
\nThe code example relevant for avoiding OOM error is just the try: except: image read block. Resizing entire image to 512x512 is a drastic image reduction, which is what the author chose to do, and still got some good results on the public LB. Not sure what the secret there is :)
", "rawMarkdown": "With GPU and inference processing, I've also found it only possible to read upto 1.2 Gb image, but without any model running, I was able to check up to 2 Gb image reading without errors. If you are downscaling images before inference, one method could be to first read only large images, downscale and write the smaller sized images, then continue such that all images are below a certain size. I have tried this with training data set, which does not have many images above 1Gb in size.\n\nThe code example relevant for avoiding OOM error is just the try: except: image read block. Resizing entire image to 512x512 is a drastic image reduction, which is what the author chose to do, and still got some good results on the public LB. Not sure what the secret there is :)" } ] }, { "id": 1920242, "postDate": "2022-08-31T02:56:31.643Z", "content": "I just sort of fixed this same problem, where even though I was only reading one image at a time, I still ran out of memory during submission. As a very basic fix, I started reading in using openslide and specifying a small size and region in read_region. It's not the best data processing I'm sure, but I managed to get a submission to actually submit without error
", "rawMarkdown": "I just sort of fixed this same problem, where even though I was only reading one image at a time, I still ran out of memory during submission. As a very basic fix, I started reading in using openslide and specifying a small size and region in read_region. It's not the best data processing I'm sure, but I managed to get a submission to actually submit without error", "votes": 2, "replies": [ { "id": 1921704, "postDate": "2022-09-01T02:15:11.773Z", "content": "Thanks this is a life saver!!!
\nI've now gone passed Notebook Exceeded Allowed Compute.
I'm now stuck with Submission Scoring Error….oh gosh….good luck to me
", "rawMarkdown": "Thanks this is a life saver!!!\nI've now gone passed Notebook Exceeded Allowed Compute.\n\nI'm now stuck with Submission Scoring Error....oh gosh....good luck to me" } ] }, { "id": 1918557, "postDate": "2022-08-29T16:53:33.647Z", "content": "adding these logic and print out to see if any image needed to be skipped:
\n print(f'\\nProcessing Image {i}')\n image_path = f'{official_data_path}/train/{image_filename}'\n print('required RAM: ' + str(os.path.getsize(image_path)))\n print('available RAM: ' + str(psutil.virtual_memory().available))\n skip = False\n if psutil.virtual_memory().available<os.path.getsize(image_path):\n skip = True\n if not skip:\n #do stuffs here\nfingers crossed
", "rawMarkdown": "adding these logic and print out to see if any image needed to be skipped:\n\n print(f'\\nProcessing Image {i}')\n image_path = f'{official_data_path}/train/{image_filename}'\n print('required RAM: ' + str(os.path.getsize(image_path)))\n print('available RAM: ' + str(psutil.virtual_memory().available))\n skip = False\n if psutil.virtual_memory().availableI've already make the reading and preprocessing of images to be done in batches of 4 images each time, after each I do gcCollect to free RAM and shutil.rmtree(output_path) to free disk space.
\nCompetition says I'm allowed to use pre-trained model, so I trained my model and saved it in a public dataset and reads it in instead of training from scratch so I don't need to process all files from \"/train\" folder anymore.
\nMy notebook only reads in the model weights saved from a public dataset and the competition files as inputs.
\nIt straight away read the \"/test\" folder and start processing them in batches, predict them and store result in a dataframe.
I feel the toughest part of this competition is in figuring this out..which demotivates me a lot since it's not about solving the problem statement but about the constraint of the environments…I simply cannot get any submission working after more than 12 attempts. What could possibly be wrong? Is openCV completely unusable here?
\nI read some earlier discussion ppl just switch to different image processing library is that the only way?
\nAnyone manage to post a submission still using openCV? I'd like some guidance please
Honestly, I can't even read in 1 image in a reasonable amount of time. Pretty much have to process the images in patches with weak supervision. This article helped me get some direction on this matter :)
", "rawMarkdown": "Honestly, I can't even read in 1 image in a reasonable amount of time. Pretty much have to process the images in patches with weak supervision. [This article](https://towardsdatascience.com/from-patches-to-slides-how-to-train-deep-learning-models-on-gigapixel-images-with-weak-supervision-d2cd2081cfd7) helped me get some direction on this matter :)", "votes": 1, "isDeleted": true, "replies": [ { "id": 1920272, "postDate": "2022-08-31T03:44:12.550Z", "content": "@akhilkota thanks for sharing the article, there are some good ideas to reduce work load for the MIL approach.
", "rawMarkdown": "@akhilkota thanks for sharing the article, there are some good ideas to reduce work load for the MIL approach." } ] } ], "comments": [ { "id": 1917530, "author_name": "tdiceman", "author_url": "", "post_date": "2022-08-28T20:06:11.690000", "content": "@raymondsky most of us have gone through this issue in the beginning. Here and here are some previous threads of discussions on this, might be helpful to go through these and then clarify what worked and what did not. Maybe then some of us can be in a better position to help.
\nAs far as your explanation reveals, image reading is the likely issue - in my limited experience, there are large enough images in the hidden test dataset that do not allow reading image file and then even reasonable RAM intensive processing/computation, especially with the GPU where the RAM is only 13 Gb. If you are trying to read 4 images at a time before inference, definitely start from there - need to read one at a time, aggregate the results, perform inference etc.
\nHope this helps.
", "votes": 1, "replies": [ { "id": 1918487, "author_name": "Raymond sky", "author_url": "", "post_date": "2022-08-29T15:57:40.193000", "content": "Thanks, seems one of the solution is to simply skip files that's too large to read into the available RAM….
\nI'll give that a go hopefully finally resolve this nightmare problem I've been facing
", "votes": 1, "replies": [] }, { "id": 1918502, "author_name": "tdiceman", "author_url": "", "post_date": "2022-08-29T16:11:04.333000", "content": "All the best :)
", "votes": 1, "replies": [] }, { "id": 1919963, "author_name": "Raymond sky", "author_url": "", "post_date": "2022-08-30T19:17:18.307000", "content": "I've applied the same threshold and try exception from this notebook:
\nhttps://www.kaggle.com/code/realneuralnetwork/cnn-strip-ai-inference/notebook?scriptVersionId=100707854
still no luck….still hitting same error
\nNotebook Exceeded Allowed Compute
\nzzzzzzzzzzzzzzz
@raymondsky what file size limit have you used? Depending on your code's RAM usage, and with GPU, anything above 1.2 Gb might still need to be treated in a different way.
\nAlso, you mentioned you are inferring on 4 images at a time. Could you try one image at a time, I'm sure that should work with a conservative image size limit. Beyond that there's no reason unless your code is eating RAM for lunch while inference :)
", "votes": 0, "replies": [] }, { "id": 1920235, "author_name": "Raymond sky", "author_url": "", "post_date": "2022-08-31T02:47:05.580000", "content": "I've already done that mate, 1 patient image at a time, T-T
\nI read, resize and save a copy and then immediately delete and gc.collect()
\nHowever as part of the solution, I did do cv findContours and create new subimages….before i save the copies,
\nI also made sure I delete the saved images before next loop after getting prediction results.
\nI lowered my threshold form 1500000000 to 1000000000 and gone passed the error now, but I get Submission Scoring Error this time…..
If we skipped those larger than 1000000000 size how do we predict for them?
\nI saw in the sample notebook we set np.zeros((512,512,3), np.uint8)
\nhow can the ML predict this dummy image accurately?
With GPU and inference processing, I've also found it only possible to read upto 1.2 Gb image, but without any model running, I was able to check up to 2 Gb image reading without errors. If you are downscaling images before inference, one method could be to first read only large images, downscale and write the smaller sized images, then continue such that all images are below a certain size. I have tried this with training data set, which does not have many images above 1Gb in size.
\nThe code example relevant for avoiding OOM error is just the try: except: image read block. Resizing entire image to 512x512 is a drastic image reduction, which is what the author chose to do, and still got some good results on the public LB. Not sure what the secret there is :)
", "votes": 0, "replies": [] } ] }, { "id": 1920242, "author_name": "Cassie Brimmer", "author_url": "", "post_date": "2022-08-31T02:56:31.643000", "content": "I just sort of fixed this same problem, where even though I was only reading one image at a time, I still ran out of memory during submission. As a very basic fix, I started reading in using openslide and specifying a small size and region in read_region. It's not the best data processing I'm sure, but I managed to get a submission to actually submit without error
", "votes": 2, "replies": [ { "id": 1921704, "author_name": "Raymond sky", "author_url": "", "post_date": "2022-09-01T02:15:11.773000", "content": "Thanks this is a life saver!!!
\nI've now gone passed Notebook Exceeded Allowed Compute.
I'm now stuck with Submission Scoring Error….oh gosh….good luck to me
", "votes": 0, "replies": [] } ] }, { "id": 1918557, "author_name": "Raymond sky", "author_url": "", "post_date": "2022-08-29T16:53:33.647000", "content": "adding these logic and print out to see if any image needed to be skipped:
\n print(f'\\nProcessing Image {i}')\n image_path = f'{official_data_path}/train/{image_filename}'\n print('required RAM: ' + str(os.path.getsize(image_path)))\n print('available RAM: ' + str(psutil.virtual_memory().available))\n skip = False\n if psutil.virtual_memory().available<os.path.getsize(image_path):\n skip = True\n if not skip:\n #do stuffs here\nfingers crossed
", "votes": 2, "replies": [] }, { "id": 1920186, "author_name": "", "author_url": "", "post_date": "2022-08-31T01:30:55.483000", "content": "Honestly, I can't even read in 1 image in a reasonable amount of time. Pretty much have to process the images in patches with weak supervision. This article helped me get some direction on this matter :)
", "votes": 1, "replies": [ { "id": 1920272, "author_name": "tdiceman", "author_url": "", "post_date": "2022-08-31T03:44:12.550000", "content": "@akhilkota thanks for sharing the article, there are some good ideas to reduce work load for the MIL approach.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1917530": "@raymondsky most of us have gone through this issue in the beginning. [Here](https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/340944) and [here](https://www.kaggle.com/competitions/mayo-clinic-strip-ai/discussion/336484) are some previous threads of discussions on this, might be helpful to go through these and then clarify what worked and what did not. Maybe then some of us can be in a better position to help.\n\nAs far as your explanation reveals, image reading is the likely issue - in my limited experience, there are large enough images in the hidden test dataset that do not allow reading image file and then even reasonable RAM intensive processing/computation, especially with the GPU where the RAM is only 13 Gb. If you are trying to read 4 images at a time before inference, definitely start from there - need to read one at a time, aggregate the results, perform inference etc. \n\nHope this helps.", "1920242": "I just sort of fixed this same problem, where even though I was only reading one image at a time, I still ran out of memory during submission. As a very basic fix, I started reading in using openslide and specifying a small size and region in read_region. It's not the best data processing I'm sure, but I managed to get a submission to actually submit without error", "1918557": "adding these logic and print out to see if any image needed to be skipped:\n\n print(f'\\nProcessing Image {i}')\n image_path = f'{official_data_path}/train/{image_filename}'\n print('required RAM: ' + str(os.path.getsize(image_path)))\n print('available RAM: ' + str(psutil.virtual_memory().available))\n skip = False\n if psutil.virtual_memory().available