Hello everyone! I am receiving a \"Notebook Out of Memory\" error during submission and I am failing to see what's the root cause after making a couple of changes. Saving and Committing the notebook before submission is returning no errors. Here is where my suspicions were but no improvements were achieved upon making changes (to reduce consumed RAM and also consumed output directory disk space):
\nWhat could be the cause? Is it a RAM or output directory disk space issue? The notebook is committing successfully without any memory issues before submission. I am also not seeing this type of error on the 'Code Competitions - Errors & Debugging Tips' page. Thanks to all that are going to help.
", "messages": [ { "id": 2426165, "postDate": "2023-09-06T12:54:48.123Z", "content": "Hello everyone! I am receiving a \"Notebook Out of Memory\" error during submission and I am failing to see what's the root cause after making a couple of changes. Saving and Committing the notebook before submission is returning no errors. Here is where my suspicions were but no improvements were achieved upon making changes (to reduce consumed RAM and also consumed output directory disk space):
\nWhat could be the cause? Is it a RAM or output directory disk space issue? The notebook is committing successfully without any memory issues before submission. I am also not seeing this type of error on the 'Code Competitions - Errors & Debugging Tips' page. Thanks to all that are going to help.
", "rawMarkdown": "Hello everyone! I am receiving a \"Notebook Out of Memory\" error during submission and I am failing to see what's the root cause after making a couple of changes. Saving and Committing the notebook before submission is returning no errors. Here is where my suspicions were but no improvements were achieved upon making changes (to reduce consumed RAM and also consumed output directory disk space):\n\n1. There is a resize function which loops through and resizes the images and converts them to a list of numpy arrays for which I have reduced the number of the pre-processed images to reduce consumed RAM but still received a \"Notebook Out of Memory\" error.\n2. My suspicion was also on the output folder directory consumed disk space ('/kaggle/working/') which I suspected that it might be a disk space issue as I am saving separate models in that directory totaling up to 1.2GB (for which the limit is 19.6GB). I then changed the path to all my outputs that I am saving from '/kaggle/working/' output directory to just '/tmp/' directory, but no improvements were achieved.\n\nWhat could be the cause? Is it a RAM or output directory disk space issue? The notebook is committing successfully without any memory issues before submission. I am also not seeing this type of error on the 'Code Competitions - Errors & Debugging Tips' page. Thanks to all that are going to help.", "votes": 1 }, { "id": 2429632, "postDate": "2023-09-08T17:32:17.253Z", "content": "You could test doing the inference on the train dataset
", "rawMarkdown": "You could test doing the inference on the train dataset", "replies": [ { "id": 2429767, "postDate": "2023-09-08T19:37:52.093Z", "content": "(or a subset)
", "rawMarkdown": "(or a subset)", "replies": [ { "id": 2430596, "postDate": "2023-09-09T11:56:38.703Z", "content": "A straightfoward approach. Thanks for pointing this out.
", "rawMarkdown": "A straightfoward approach. Thanks for pointing this out." } ] } ] }, { "id": 2426892, "postDate": "2023-09-06T22:34:15.497Z", "content": "The test set that we get to play with is too small to generate RAM issues. Try running a test set that contains more patients and scans in interactive mode. I shared a set and there is at least one other small set that has been shared.
\nhttps://www.kaggle.com/competitions/rsna-2023-abdominal-trauma-detection/discussion/429050
\nhttps://www.kaggle.com/datasets/pcjimmmy/rsna-7-patients-test
", "rawMarkdown": "The test set that we get to play with is too small to generate RAM issues. Try running a test set that contains more patients and scans in interactive mode. I shared a set and there is at least one other small set that has been shared.\n\nhttps://www.kaggle.com/competitions/rsna-2023-abdominal-trauma-detection/discussion/429050\n\nhttps://www.kaggle.com/datasets/pcjimmmy/rsna-7-patients-test\n\n", "replies": [ { "id": 2427504, "postDate": "2023-09-07T09:26:43.300Z", "content": "Thanks with the new test dataset. Let me work with that one and try and see where I am losing it in my notebook.
", "rawMarkdown": "Thanks with the new test dataset. Let me work with that one and try and see where I am losing it in my notebook. " }, { "id": 2427575, "postDate": "2023-09-07T10:16:56.253Z", "content": "It would have been nice if the above test dataset was saved in dcm rather than png format to fully simulate what will be going on on the hidden test set i.e., from the conversion process of the dcm files to png, through the whole resizing and prediction process to see where exactly the RAM is being consumed as I am having no Memory issues with the above ready made png test dataset when I run it.
", "rawMarkdown": "It would have been nice if the above test dataset was saved in dcm rather than png format to fully simulate what will be going on on the hidden test set i.e., from the conversion process of the dcm files to png, through the whole resizing and prediction process to see where exactly the RAM is being consumed as I am having no Memory issues with the above ready made png test dataset when I run it.", "votes": 1, "replies": [ { "id": 2427782, "postDate": "2023-09-07T13:33:21.863Z", "content": "Good idea - attempted to add them and got a 'too many files' error - not seen that before. Will keep trying.
", "rawMarkdown": "Good idea - attempted to add them and got a 'too many files' error - not seen that before. Will keep trying." }, { "id": 2427872, "postDate": "2023-09-07T14:21:48.937Z", "content": "Does not seem to want to let me load more than 1000 files to dataset. Not sure I was aware of that limit. Will have time later to build a new seperate dataset, looks like probably with just 2 patients. Of course, not real hard to build a dataset for your self :)
\nCheon created a 6 patient dataset with dicom's that you can use.
", "rawMarkdown": "Does not seem to want to let me load more than 1000 files to dataset. Not sure I was aware of that limit. Will have time later to build a new seperate dataset, looks like probably with just 2 patients. Of course, not real hard to build a dataset for your self :)\n\n[Cheon](https://www.kaggle.com/datasets/cheonresearch/rsna-submission-test-data) created a 6 patient dataset with dicom's that you can use." }, { "id": 2430601, "postDate": "2023-09-09T12:00:38.120Z", "content": "Great, thanks. Directly doing inference on a subset of the train dataset could also be another straightforward way as pointed out by @gabrielrudloff above.
", "rawMarkdown": "Great, thanks. Directly doing inference on a subset of the train dataset could also be another straightforward way as pointed out by @gabrielrudloff above." }, { "id": 2448086, "postDate": "2023-09-20T12:02:45.023Z", "content": "Hi were you able to fix it or know the real reason for this to happen ?? if yes please let me know
", "rawMarkdown": "Hi were you able to fix it or know the real reason for this to happen ?? if yes please let me know " }, { "id": 2448140, "postDate": "2023-09-20T12:32:28.183Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2448144, "postDate": "2023-09-20T12:33:32.013Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2448149, "postDate": "2023-09-20T12:34:59.473Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2448150, "postDate": "2023-09-20T12:35:34.920Z", "content": "Not yet. When I tried using the provided competition notebook I then received a \"Notebook Timeout\" error instead. I am losing on all fronts, both my own notebook I built from scratch which is giving me a \"Notebook Out of Memory\" error and the provided competition notebook which is giving me a \"Notebook Timeout\" error. Too bad, I am fighting a two headed snake! Have tried to use the garbage collector (gc) module and del method to remove in memory variables to release more RAM but it didn't work. After receiving the timeout error with the competition notebook, I am now concerned that even if I solve the \"Notebook Out of Memory\" issue which is a RAM issue, I can eventually bump into the \"Notebook Timeout\" error again after solving the RAM issue. It seems the test dataset is really really big.
\nI will give an update on the solution if I manage to tackle the problem. @prateeksanghionward kindly update if you also manage to succeed from your side.
", "rawMarkdown": "Not yet. When I tried using the provided competition notebook I then received a \"Notebook Timeout\" error instead. I am losing on all fronts, both my own notebook I built from scratch which is giving me a \"Notebook Out of Memory\" error and the provided competition notebook which is giving me a \"Notebook Timeout\" error. Too bad, I am fighting a two headed snake! Have tried to use the garbage collector (gc) module and del method to remove in memory variables to release more RAM but it didn't work. After receiving the timeout error with the competition notebook, I am now concerned that even if I solve the \"Notebook Out of Memory\" issue which is a RAM issue, I can eventually bump into the \"Notebook Timeout\" error again after solving the RAM issue. It seems the test dataset is really really big. \n\nI will give an update on the solution if I manage to tackle the problem. @prateeksanghionward kindly update if you also manage to succeed from your side." }, { "id": 2456495, "postDate": "2023-09-26T08:32:40.387Z", "content": "ok cool . not yet able to tackle it
", "rawMarkdown": "ok cool . not yet able to tackle it " } ] } ] } ], "comments": [ { "id": 2429632, "author_name": "gabriel rudloff", "author_url": "", "post_date": "2023-09-08T17:32:17.253000", "content": "You could test doing the inference on the train dataset
", "votes": 0, "replies": [ { "id": 2429767, "author_name": "gabriel rudloff", "author_url": "", "post_date": "2023-09-08T19:37:52.093000", "content": "(or a subset)
", "votes": 0, "replies": [ { "id": 2430596, "author_name": "Wisdom James Murombo", "author_url": "", "post_date": "2023-09-09T11:56:38.703000", "content": "A straightfoward approach. Thanks for pointing this out.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2426892, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2023-09-06T22:34:15.497000", "content": "The test set that we get to play with is too small to generate RAM issues. Try running a test set that contains more patients and scans in interactive mode. I shared a set and there is at least one other small set that has been shared.
\nhttps://www.kaggle.com/competitions/rsna-2023-abdominal-trauma-detection/discussion/429050
\nhttps://www.kaggle.com/datasets/pcjimmmy/rsna-7-patients-test
", "votes": 0, "replies": [ { "id": 2427504, "author_name": "Wisdom James Murombo", "author_url": "", "post_date": "2023-09-07T09:26:43.300000", "content": "Thanks with the new test dataset. Let me work with that one and try and see where I am losing it in my notebook.
", "votes": 0, "replies": [] }, { "id": 2427575, "author_name": "Wisdom James Murombo", "author_url": "", "post_date": "2023-09-07T10:16:56.253000", "content": "It would have been nice if the above test dataset was saved in dcm rather than png format to fully simulate what will be going on on the hidden test set i.e., from the conversion process of the dcm files to png, through the whole resizing and prediction process to see where exactly the RAM is being consumed as I am having no Memory issues with the above ready made png test dataset when I run it.
", "votes": 1, "replies": [ { "id": 2427782, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2023-09-07T13:33:21.863000", "content": "Good idea - attempted to add them and got a 'too many files' error - not seen that before. Will keep trying.
", "votes": 0, "replies": [] }, { "id": 2427872, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2023-09-07T14:21:48.937000", "content": "Does not seem to want to let me load more than 1000 files to dataset. Not sure I was aware of that limit. Will have time later to build a new seperate dataset, looks like probably with just 2 patients. Of course, not real hard to build a dataset for your self :)
\nCheon created a 6 patient dataset with dicom's that you can use.
", "votes": 0, "replies": [] }, { "id": 2430601, "author_name": "Wisdom James Murombo", "author_url": "", "post_date": "2023-09-09T12:00:38.120000", "content": "Great, thanks. Directly doing inference on a subset of the train dataset could also be another straightforward way as pointed out by @gabrielrudloff above.
", "votes": 0, "replies": [] }, { "id": 2448086, "author_name": "Prateek Sanghi Onward", "author_url": "", "post_date": "2023-09-20T12:02:45.023000", "content": "Hi were you able to fix it or know the real reason for this to happen ?? if yes please let me know
", "votes": 0, "replies": [] }, { "id": 2448140, "author_name": "", "author_url": "", "post_date": "2023-09-20T12:32:28.183000", "content": "", "votes": 0, "replies": [] }, { "id": 2448144, "author_name": "", "author_url": "", "post_date": "2023-09-20T12:33:32.013000", "content": "", "votes": 0, "replies": [] }, { "id": 2448149, "author_name": "", "author_url": "", "post_date": "2023-09-20T12:34:59.473000", "content": "", "votes": 0, "replies": [] }, { "id": 2448150, "author_name": "Wisdom James Murombo", "author_url": "", "post_date": "2023-09-20T12:35:34.920000", "content": "Not yet. When I tried using the provided competition notebook I then received a \"Notebook Timeout\" error instead. I am losing on all fronts, both my own notebook I built from scratch which is giving me a \"Notebook Out of Memory\" error and the provided competition notebook which is giving me a \"Notebook Timeout\" error. Too bad, I am fighting a two headed snake! Have tried to use the garbage collector (gc) module and del method to remove in memory variables to release more RAM but it didn't work. After receiving the timeout error with the competition notebook, I am now concerned that even if I solve the \"Notebook Out of Memory\" issue which is a RAM issue, I can eventually bump into the \"Notebook Timeout\" error again after solving the RAM issue. It seems the test dataset is really really big.
\nI will give an update on the solution if I manage to tackle the problem. @prateeksanghionward kindly update if you also manage to succeed from your side.
", "votes": 0, "replies": [] }, { "id": 2456495, "author_name": "Prateek Sanghi Onward", "author_url": "", "post_date": "2023-09-26T08:32:40.387000", "content": "ok cool . not yet able to tackle it
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2426165": "Hello everyone! I am receiving a \"Notebook Out of Memory\" error during submission and I am failing to see what's the root cause after making a couple of changes. Saving and Committing the notebook before submission is returning no errors. Here is where my suspicions were but no improvements were achieved upon making changes (to reduce consumed RAM and also consumed output directory disk space):\n\n1. There is a resize function which loops through and resizes the images and converts them to a list of numpy arrays for which I have reduced the number of the pre-processed images to reduce consumed RAM but still received a \"Notebook Out of Memory\" error.\n2. My suspicion was also on the output folder directory consumed disk space ('/kaggle/working/') which I suspected that it might be a disk space issue as I am saving separate models in that directory totaling up to 1.2GB (for which the limit is 19.6GB). I then changed the path to all my outputs that I am saving from '/kaggle/working/' output directory to just '/tmp/' directory, but no improvements were achieved.\n\nWhat could be the cause? Is it a RAM or output directory disk space issue? The notebook is committing successfully without any memory issues before submission. I am also not seeing this type of error on the 'Code Competitions - Errors & Debugging Tips' page. Thanks to all that are going to help.", "2429632": "You could test doing the inference on the train dataset", "2426892": "The test set that we get to play with is too small to generate RAM issues. Try running a test set that contains more patients and scans in interactive mode. I shared a set and there is at least one other small set that has been shared.\n\nhttps://www.kaggle.com/competitions/rsna-2023-abdominal-trauma-detection/discussion/429050\n\nhttps://www.kaggle.com/datasets/pcjimmmy/rsna-7-patients-test\n\n" } }