![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1211645%2F5cde25d86a118a18aaf1c0fd531e87b6%2FScreenshot%202023-12-21%20at%2012.58.38.png?generation=1703131309840632&alt=media\")
Hi @sohier , @jonathanmcwilliams , @ashleychow
\nIn my submission notebook, I saved some files in /tmp/output/, which always have more than 70G storage.
I didn't change any code relative to the file saving part in my submission notebook. But all of the 5 submission notebook fails with Notebook Out of Disk, which never happened before. And I just submitted the same notebook successfully 24 hours ago.
Is the storage of kaggle submission notebook change in the last 24 hours? And anyone met the same problems in the last 24 hours?
\nThe discussion is going in another topics. The problem is solved. Thanks to the kaggle stuff @herbison. Please feel free to follow.
", "messages": [ { "id": 2560819, "postDate": "2023-12-14T03:17:36.230Z", "content": "Hi @sohier , @jonathanmcwilliams , @ashleychow
\nIn my submission notebook, I saved some files in /tmp/output/, which always have more than 70G storage.
I didn't change any code relative to the file saving part in my submission notebook. But all of the 5 submission notebook fails with Notebook Out of Disk, which never happened before. And I just submitted the same notebook successfully 24 hours ago.
Is the storage of kaggle submission notebook change in the last 24 hours? And anyone met the same problems in the last 24 hours?
\nThe discussion is going in another topics. The problem is solved. Thanks to the kaggle stuff @herbison. Please feel free to follow.
", "rawMarkdown": "Hi @sohier , @jonathanmcwilliams , @ashleychow \nIn my submission notebook, I saved some files in `/tmp/output/`, which always have more than 70G storage. \n\nI didn't change any code relative to the file saving part in my submission notebook. But all of the 5 submission notebook fails with `Notebook Out of Disk`, which never happened before. And I just submitted the same notebook successfully 24 hours ago.\n\nIs the storage of kaggle submission notebook change in the last 24 hours? And anyone met the same problems in the last 24 hours?\n\n*************************************\n\nThe discussion is going in [another topics](https://www.kaggle.com/discussions/product-feedback/461401). The problem ~~hasn't been solved yet~~ is solved. Thanks to the kaggle stuff @herbison. Please feel free to follow. ", "votes": 11 }, { "id": 2561765, "postDate": "2023-12-14T19:32:42.597Z", "content": "@forcewithme can you share a notebook with me that was previously working? You can remove your inference code; I should just need the part of your pipeline that opens the images in order to continue debugging.
", "rawMarkdown": "@forcewithme can you share a notebook with me that was previously working? You can remove your inference code; I should just need the part of your pipeline that opens the images in order to continue debugging.", "votes": 3, "replies": [ { "id": 2561898, "postDate": "2023-12-15T00:05:46.117Z", "content": "Hi @sohier , thank you! The relative part of my code is basically the same as this public notebook. Is there a similar observation in this notebook?
", "rawMarkdown": "Hi @sohier , thank you! The relative part of my code is basically the same as [this public notebook](https://www.kaggle.com/code/jirkaborovec/cancer-subtype-lightning-torch-inference-tiles). Is there a similar observation in this notebook?" } ] }, { "id": 2561176, "postDate": "2023-12-14T10:26:13.957Z", "content": "Same here. I've reported it to the PRODUCT FEEDBACK. You may leave your comments too.
", "rawMarkdown": "Same here. I've reported it to the [PRODUCT FEEDBACK](https://www.kaggle.com/discussions/product-feedback/461401). You may leave your comments too.", "votes": 4 }, { "id": 2563826, "postDate": "2023-12-16T15:48:36.283Z", "content": "I am also encountering the same error (Notebook Out of Disk).
\nI thought I had submitted a submit which should not have changed the amount of writes to disk, but the above error message was displayed.
\nIs there any update here?
", "rawMarkdown": "I am also encountering the same error (Notebook Out of Disk).\n\nI thought I had submitted a submit which should not have changed the amount of writes to disk, but the above error message was displayed.\n\nIs there any update here?", "votes": 1, "replies": [ { "id": 2564154, "postDate": "2023-12-16T20:49:21.733Z", "content": "If you're using pyvips try this workaround https://www.kaggle.com/discussions/product-feedback/461401#2562671
", "rawMarkdown": "If you're using pyvips try this workaround https://www.kaggle.com/discussions/product-feedback/461401#2562671", "votes": 1, "replies": [ { "id": 2564790, "postDate": "2023-12-17T12:27:46.463Z", "content": "Hello Dustin, I’m encountering the same problem: a notebook that ran before do not run anymore (out of disk) and the workaround your propose results in an exception. Did the env of the image the notebooks run have changed ?
", "rawMarkdown": "Hello Dustin, I’m encountering the same problem: a notebook that ran before do not run anymore (out of disk) and the workaround your propose results in an exception. Did the env of the image the notebooks run have changed ?" }, { "id": 2564941, "postDate": "2023-12-17T14:15:20.357Z", "content": "There was an image release recently, but if you're using the same notebook and you didn't set it to \"use latest\" (you're pinned to your original image still), then there have been no changes to the env.
\nThe code I suggested should work fine if the issue is pyvips using /tmp (and if the files are being cleaned up, you're not already using 20GBs of /kaggle/working space).
\nIf you're using a lot of space outside /kaggle/working, then I suggest you try to move your work into /kaggle/working for now.
", "rawMarkdown": "There was an image release recently, but if you're using the same notebook and you didn't set it to \"use latest\" (you're pinned to your original image still), then there have been no changes to the env.\n\nThe code I suggested should work fine if the issue is pyvips using /tmp (and if the files are being cleaned up, you're not already using 20GBs of /kaggle/working space).\n\nIf you're using a lot of space outside /kaggle/working, then I suggest you try to move your work into /kaggle/working for now.", "votes": 1 }, { "id": 2564978, "postDate": "2023-12-17T14:47:27.933Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2564980, "postDate": "2023-12-17T14:47:45.687Z", "content": "Hi Dustin, Does any recent release could potentially cause \"Notebook Threw Exception\" ? My recent submissions all failed with this info. And my notebook was only silghtly modifed compare to previously successfully submitted version. I'm really struggled to figure out why… And also I tested my code on train data, it works out fine.
", "rawMarkdown": "Hi Dustin, Does any recent release could potentially cause \"Notebook Threw Exception\" ? My recent submissions all failed with this info. And my notebook was only silghtly modifed compare to previously successfully submitted version. I'm really struggled to figure out why… And also I tested my code on train data, it works out fine.", "votes": 1 }, { "id": 2565004, "postDate": "2023-12-17T14:57:09.907Z", "content": "Thank you for the answer ! Indeed I did not change the env used (pinned to the first version). What could then explain the fails from notebooks that ran before ? (I really just re-launched a previous version)
", "rawMarkdown": "Thank you for the answer ! Indeed I did not change the env used (pinned to the first version). What could then explain the fails from notebooks that ran before ? (I really just re-launched a previous version) " }, { "id": 2568181, "postDate": "2023-12-20T10:13:57.863Z", "content": "Hey ! I don’t know why but my notebooks kept failing even with this solution. In my case I found that removing the pyvips cache helped ! So the workaround becomes:
\nimport os\nif not os.path.exists('/kaggle/working/tmp'):\n os.mkdir('/kaggle/working/tmp')\nos.environ['TMPDIR'] = '/kaggle/working/tmp'\n!export TMPDIR='/kaggle/working/tmp'\n!export VIPS_CONCURRENCY=1\npyvips.cache_set_max(0)
Hi @herbison @sohier , @jonathanmcwilliams , @ashleychow , Please help! It seems the problem remains. The new submissions still failed, reporting notebook out of disk. It has suspended my plan for this competition for 2 days. I think it also bothers many other competitors.
@forcewithme I'm looking into this.
\nThere should be no change in the size of the disk on the VM.
\nYou mention there is always more than 70GB of space, but thats actually around the maximum possible before it runs out, and I do see that your submissions appear to be using high /tmp disk space, are you able to free some space as you go to not exceed 60-70GBs at a time in /tmp?
", "rawMarkdown": "@forcewithme I'm looking into this. \n\nThere should be no change in the size of the disk on the VM. \n\nYou mention there is always more than 70GB of space, but thats actually around the maximum possible before it runs out, and I do see that your submissions appear to be using high /tmp disk space, are you able to free some space as you go to not exceed 60-70GBs at a time in /tmp?", "votes": 2, "replies": [ { "id": 2561489, "postDate": "2023-12-14T14:28:25.250Z", "content": "Hi @herbison , thanks for your reply! The disk space I need should be much lower than 70G. The same code has never had an error before. In contrast to the previous one, the code related to disk usage has not been modified at all.
", "rawMarkdown": "Hi @herbison , thanks for your reply! The disk space I need should be much lower than 70G. The same code has never had an error before. In contrast to the previous one, the code related to disk usage has not been modified at all. " }, { "id": 2561492, "postDate": "2023-12-14T14:32:36.967Z", "content": "In the new submission failed today. I saved the data in /kaggle/working instead of /tmp. I know /kaggle/working only has 15G. But my files should take much lower space than 15G. I have tested it on the training set of this competition and it goes well. The raw data training set have more than 700G, and the raw data in the test set has only 500G. So since the code work well on training set, it's not likely to crash on the test set.
The most recent failed out of disk submission I see used /tmp still, looking at the disk usage logs, it starts out using very little, but ramps up to > 70GBs and beyond and then fails.
", "rawMarkdown": "The most recent failed out of disk submission I see used /tmp still, looking at the disk usage logs, it starts out using very little, but ramps up to > 70GBs and beyond and then fails.", "votes": 1 }, { "id": 2561509, "postDate": "2023-12-14T14:49:00.567Z", "content": "\n\nThe most recent failed out of disk submission I see used /tmp still,
\n
It shouldn't happen. When I search /tmp in my code, I only find one line of deprecated code os.makedirs('/tmp/output', exist_ok=True). Obviously, this line of code shouldn't take any storage, and it's at about 3 minutes of the runtime. The fails of my submission happens randomly, sometimes 9 hours, sometimes 11 hours.
Both your successful run and your unsuccessful runs exceed 70GBs of /tmp space, it looks like you were just \"on the line\" and lucky with the successful ones.
\nThe jump in usage consistently happens quite late (like 8hrs into the run), so something is loading a bunch of files into /tmp around that time (it might not be intentional, ex. maybe you're working with a model/framework that dumps a bunch of files to /tmp?)
\nIt might be good to review this with an Save Version session (or interactive if you can keep it alive that long), and print out the /tmp disk usage periodically to review when it jumps.
\nSo to clarify, the amount of disk available has not changed, something in your code (or that your code depends on) is writing a lot of data in /tmp.
", "rawMarkdown": "Both your successful run and your unsuccessful runs exceed 70GBs of /tmp space, it looks like you were just \"on the line\" and lucky with the successful ones.\n\nThe jump in usage consistently happens quite late (like 8hrs into the run), so something is loading a bunch of files into /tmp around that time (it might not be intentional, ex. maybe you're working with a model/framework that dumps a bunch of files to /tmp?)\n\nIt might be good to review this with an Save Version session (or interactive if you can keep it alive that long), and print out the /tmp disk usage periodically to review when it jumps.\n\nSo to clarify, the amount of disk available has not changed, something in your code (or that your code depends on) is writing a lot of data in /tmp.", "votes": 1 }, { "id": 2561876, "postDate": "2023-12-14T22:53:22.787Z", "content": "The most likely culprit in most people's code is pyvips. This package uses /tmp to \"open\" the images we use for this competition (or create a virtual image). According to the main developer of the package, the image in /tmp should be automatically garbage collected upon closing. However, it seems that this expected behavior is no longer happening since two days ago, despite explicitly adding some del and gc.collect() within my code.
Can you confirm which wheel you're using to install pyvips?
Hi @patrob , I also explicitly adding some del and gc.collect() within my code. Not work.
", "rawMarkdown": "Hi @patrob , I also explicitly adding some del and gc.collect() within my code. Not work." }, { "id": 2561903, "postDate": "2023-12-15T00:09:28.570Z", "content": "@sohier I used the wheels from Jirka's notebook: https://www.kaggle.com/datasets/jirkaborovec/pyvips-python-and-deb-package-gpu
", "rawMarkdown": "@sohier I used the wheels from Jirka's notebook: [https://www.kaggle.com/datasets/jirkaborovec/pyvips-python-and-deb-package-gpu](url)", "votes": 1 }, { "id": 2561904, "postDate": "2023-12-15T00:10:54.653Z", "content": "Hi @sohier , I use the wheels produced in this public notebook. Someone may use the wheels in this dataset. Both haven’t been changed for 2 months.
", "rawMarkdown": "Hi @sohier , I use the wheels produced in [this public notebook](https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started). Someone may use the wheels in this [dataset](https://www.kaggle.com/datasets/jirkaborovec/pyvips-python-and-deb-package-gpu). Both haven’t been changed for 2 months. " }, { "id": 2562735, "postDate": "2023-12-15T16:19:17.973Z", "content": "@forcewithme Try the possible workaround I put in here:
\nhttps://www.kaggle.com/discussions/product-feedback/461401#2562671
I've been running the test notebook for an hour and it prevented unbounded /tmp growth.
\nI'm still looking into why /tmp grows when its written to like this, and what we can do about it, but this should at least let you continue working.
", "rawMarkdown": "@forcewithme Try the possible workaround I put in here:\nhttps://www.kaggle.com/discussions/product-feedback/461401#2562671\n\nI've been running the test notebook for an hour and it prevented unbounded /tmp growth.\n\nI'm still looking into why /tmp grows when its written to like this, and what we can do about it, but this should at least let you continue working.", "votes": 2 } ] } ] }, { "id": 2561266, "postDate": "2023-12-14T11:25:24.960Z", "content": "@herbison I've seen you deal with similar issues on other competitions. That probably also falls within your patch of work. Could you please have a look at this?
", "rawMarkdown": "@herbison I've seen you deal with similar issues on other competitions. That probably also falls within your patch of work. Could you please have a look at this?", "votes": 1 }, { "id": 2561204, "postDate": "2023-12-14T10:57:09.513Z", "content": "Same here, all my submissions failed with \"Notebook Out of Disk\", despite they being just parameter changes of already completed submissions (these changes shouldn't influence disk usage at all).
", "rawMarkdown": "Same here, all my submissions failed with \"Notebook Out of Disk\", despite they being just parameter changes of already completed submissions (these changes shouldn't influence disk usage at all).", "votes": 1 }, { "id": 2561160, "postDate": "2023-12-14T10:10:49.173Z", "content": "Same here, \"Notebook Out of Disk\" with notebooks that ran before yesterday. Might be a global problem
", "rawMarkdown": "Same here, \"Notebook Out of Disk\" with notebooks that ran before yesterday. Might be a global problem", "votes": 1 }, { "id": 2561128, "postDate": "2023-12-14T09:47:15.927Z", "content": "Same here. I've had three submissions with an out-of-disk message, which I never had before. Further, I have a submission that is still currently running after 10 hours; I normally can do inference on the TRAIN data in about 10 hours. Considering that the test data is composed of a majority of TMAs and is only 550GB, I would have expected my submission to be completed by now. Something is not right.
", "rawMarkdown": "Same here. I've had three submissions with an out-of-disk message, which I never had before. Further, I have a submission that is still currently running after 10 hours; I normally can do inference on the TRAIN data in about 10 hours. Considering that the test data is composed of a majority of TMAs and is only 550GB, I would have expected my submission to be completed by now. Something is not right.", "votes": 1 }, { "id": 2569079, "postDate": "2023-12-21T04:02:10.493Z", "content": "I have another problem.
\n
write too much means not enough space on disk?
", "rawMarkdown": "I have another problem.\n\n\n\n\nwrite too much means not enough space on disk?\n\n\n" }, { "id": 2561758, "postDate": "2023-12-14T19:18:26.450Z", "content": "Is there a workaround till this issue is solved? For example, have anyone tried to submit without saving anything beside the submission.csv?
", "rawMarkdown": "Is there a workaround till this issue is solved? For example, have anyone tried to submit without saving anything beside the submission.csv? " } ], "comments": [ { "id": 2561765, "author_name": "Sohier Dane", "author_url": "", "post_date": "2023-12-14T19:32:42.597000", "content": "@forcewithme can you share a notebook with me that was previously working? You can remove your inference code; I should just need the part of your pipeline that opens the images in order to continue debugging.
", "votes": 3, "replies": [ { "id": 2561898, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-12-15T00:05:46.117000", "content": "Hi @sohier , thank you! The relative part of my code is basically the same as this public notebook. Is there a similar observation in this notebook?
", "votes": 0, "replies": [] } ] }, { "id": 2561176, "author_name": "Johnny Lee", "author_url": "", "post_date": "2023-12-14T10:26:13.957000", "content": "Same here. I've reported it to the PRODUCT FEEDBACK. You may leave your comments too.
", "votes": 4, "replies": [] }, { "id": 2563826, "author_name": "fam_taro", "author_url": "", "post_date": "2023-12-16T15:48:36.283000", "content": "I am also encountering the same error (Notebook Out of Disk).
\nI thought I had submitted a submit which should not have changed the amount of writes to disk, but the above error message was displayed.
\nIs there any update here?
", "votes": 1, "replies": [ { "id": 2564154, "author_name": "Dustin", "author_url": "", "post_date": "2023-12-16T20:49:21.733000", "content": "If you're using pyvips try this workaround https://www.kaggle.com/discussions/product-feedback/461401#2562671
", "votes": 1, "replies": [ { "id": 2564790, "author_name": "prototype", "author_url": "", "post_date": "2023-12-17T12:27:46.463000", "content": "Hello Dustin, I’m encountering the same problem: a notebook that ran before do not run anymore (out of disk) and the workaround your propose results in an exception. Did the env of the image the notebooks run have changed ?
", "votes": 0, "replies": [] }, { "id": 2564941, "author_name": "Dustin", "author_url": "", "post_date": "2023-12-17T14:15:20.357000", "content": "There was an image release recently, but if you're using the same notebook and you didn't set it to \"use latest\" (you're pinned to your original image still), then there have been no changes to the env.
\nThe code I suggested should work fine if the issue is pyvips using /tmp (and if the files are being cleaned up, you're not already using 20GBs of /kaggle/working space).
\nIf you're using a lot of space outside /kaggle/working, then I suggest you try to move your work into /kaggle/working for now.
", "votes": 1, "replies": [] }, { "id": 2564978, "author_name": "", "author_url": "", "post_date": "2023-12-17T14:47:27.933000", "content": "", "votes": 0, "replies": [] }, { "id": 2564980, "author_name": "Yannan Chen", "author_url": "", "post_date": "2023-12-17T14:47:45.687000", "content": "Hi Dustin, Does any recent release could potentially cause \"Notebook Threw Exception\" ? My recent submissions all failed with this info. And my notebook was only silghtly modifed compare to previously successfully submitted version. I'm really struggled to figure out why… And also I tested my code on train data, it works out fine.
", "votes": 1, "replies": [] }, { "id": 2565004, "author_name": "prototype", "author_url": "", "post_date": "2023-12-17T14:57:09.907000", "content": "Thank you for the answer ! Indeed I did not change the env used (pinned to the first version). What could then explain the fails from notebooks that ran before ? (I really just re-launched a previous version)
", "votes": 0, "replies": [] }, { "id": 2568181, "author_name": "prototype", "author_url": "", "post_date": "2023-12-20T10:13:57.863000", "content": "Hey ! I don’t know why but my notebooks kept failing even with this solution. In my case I found that removing the pyvips cache helped ! So the workaround becomes:
\nimport os\nif not os.path.exists('/kaggle/working/tmp'):\n os.mkdir('/kaggle/working/tmp')\nos.environ['TMPDIR'] = '/kaggle/working/tmp'\n!export TMPDIR='/kaggle/working/tmp'\n!export VIPS_CONCURRENCY=1\npyvips.cache_set_max(0)
Hi @herbison @sohier , @jonathanmcwilliams , @ashleychow , Please help! It seems the problem remains. The new submissions still failed, reporting notebook out of disk. It has suspended my plan for this competition for 2 days. I think it also bothers many other competitors.
@forcewithme I'm looking into this.
\nThere should be no change in the size of the disk on the VM.
\nYou mention there is always more than 70GB of space, but thats actually around the maximum possible before it runs out, and I do see that your submissions appear to be using high /tmp disk space, are you able to free some space as you go to not exceed 60-70GBs at a time in /tmp?
", "votes": 2, "replies": [ { "id": 2561489, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-12-14T14:28:25.250000", "content": "Hi @herbison , thanks for your reply! The disk space I need should be much lower than 70G. The same code has never had an error before. In contrast to the previous one, the code related to disk usage has not been modified at all.
", "votes": 0, "replies": [] }, { "id": 2561492, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-12-14T14:32:36.967000", "content": "In the new submission failed today. I saved the data in /kaggle/working instead of /tmp. I know /kaggle/working only has 15G. But my files should take much lower space than 15G. I have tested it on the training set of this competition and it goes well. The raw data training set have more than 700G, and the raw data in the test set has only 500G. So since the code work well on training set, it's not likely to crash on the test set.
The most recent failed out of disk submission I see used /tmp still, looking at the disk usage logs, it starts out using very little, but ramps up to > 70GBs and beyond and then fails.
", "votes": 1, "replies": [] }, { "id": 2561509, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-12-14T14:49:00.567000", "content": "\n\nThe most recent failed out of disk submission I see used /tmp still,
\n
It shouldn't happen. When I search /tmp in my code, I only find one line of deprecated code os.makedirs('/tmp/output', exist_ok=True). Obviously, this line of code shouldn't take any storage, and it's at about 3 minutes of the runtime. The fails of my submission happens randomly, sometimes 9 hours, sometimes 11 hours.
Both your successful run and your unsuccessful runs exceed 70GBs of /tmp space, it looks like you were just \"on the line\" and lucky with the successful ones.
\nThe jump in usage consistently happens quite late (like 8hrs into the run), so something is loading a bunch of files into /tmp around that time (it might not be intentional, ex. maybe you're working with a model/framework that dumps a bunch of files to /tmp?)
\nIt might be good to review this with an Save Version session (or interactive if you can keep it alive that long), and print out the /tmp disk usage periodically to review when it jumps.
\nSo to clarify, the amount of disk available has not changed, something in your code (or that your code depends on) is writing a lot of data in /tmp.
", "votes": 1, "replies": [] }, { "id": 2561876, "author_name": "Patrick Robitaille", "author_url": "", "post_date": "2023-12-14T22:53:22.787000", "content": "The most likely culprit in most people's code is pyvips. This package uses /tmp to \"open\" the images we use for this competition (or create a virtual image). According to the main developer of the package, the image in /tmp should be automatically garbage collected upon closing. However, it seems that this expected behavior is no longer happening since two days ago, despite explicitly adding some del and gc.collect() within my code.
Can you confirm which wheel you're using to install pyvips?
Hi @patrob , I also explicitly adding some del and gc.collect() within my code. Not work.
", "votes": 0, "replies": [] }, { "id": 2561903, "author_name": "Patrick Robitaille", "author_url": "", "post_date": "2023-12-15T00:09:28.570000", "content": "@sohier I used the wheels from Jirka's notebook: https://www.kaggle.com/datasets/jirkaborovec/pyvips-python-and-deb-package-gpu
", "votes": 1, "replies": [] }, { "id": 2561904, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-12-15T00:10:54.653000", "content": "Hi @sohier , I use the wheels produced in this public notebook. Someone may use the wheels in this dataset. Both haven’t been changed for 2 months.
", "votes": 0, "replies": [] }, { "id": 2562735, "author_name": "Dustin", "author_url": "", "post_date": "2023-12-15T16:19:17.973000", "content": "@forcewithme Try the possible workaround I put in here:
\nhttps://www.kaggle.com/discussions/product-feedback/461401#2562671
I've been running the test notebook for an hour and it prevented unbounded /tmp growth.
\nI'm still looking into why /tmp grows when its written to like this, and what we can do about it, but this should at least let you continue working.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2561266, "author_name": "Patrick Robitaille", "author_url": "", "post_date": "2023-12-14T11:25:24.960000", "content": "@herbison I've seen you deal with similar issues on other competitions. That probably also falls within your patch of work. Could you please have a look at this?
", "votes": 1, "replies": [] }, { "id": 2561204, "author_name": "Adam Narai", "author_url": "", "post_date": "2023-12-14T10:57:09.513000", "content": "Same here, all my submissions failed with \"Notebook Out of Disk\", despite they being just parameter changes of already completed submissions (these changes shouldn't influence disk usage at all).
", "votes": 1, "replies": [] }, { "id": 2561160, "author_name": "prototype", "author_url": "", "post_date": "2023-12-14T10:10:49.173000", "content": "Same here, \"Notebook Out of Disk\" with notebooks that ran before yesterday. Might be a global problem
", "votes": 1, "replies": [] }, { "id": 2561128, "author_name": "Patrick Robitaille", "author_url": "", "post_date": "2023-12-14T09:47:15.927000", "content": "Same here. I've had three submissions with an out-of-disk message, which I never had before. Further, I have a submission that is still currently running after 10 hours; I normally can do inference on the TRAIN data in about 10 hours. Considering that the test data is composed of a majority of TMAs and is only 550GB, I would have expected my submission to be completed by now. Something is not right.
", "votes": 1, "replies": [] }, { "id": 2569079, "author_name": "shigengtian", "author_url": "", "post_date": "2023-12-21T04:02:10.493000", "content": "I have another problem.
\n
write too much means not enough space on disk?
", "votes": 0, "replies": [] }, { "id": 2561758, "author_name": "Adam Narai", "author_url": "", "post_date": "2023-12-14T19:18:26.450000", "content": "Is there a workaround till this issue is solved? For example, have anyone tried to submit without saving anything beside the submission.csv?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2560819": "Hi @sohier , @jonathanmcwilliams , @ashleychow \nIn my submission notebook, I saved some files in `/tmp/output/`, which always have more than 70G storage. \n\nI didn't change any code relative to the file saving part in my submission notebook. But all of the 5 submission notebook fails with `Notebook Out of Disk`, which never happened before. And I just submitted the same notebook successfully 24 hours ago.\n\nIs the storage of kaggle submission notebook change in the last 24 hours? And anyone met the same problems in the last 24 hours?\n\n*************************************\n\nThe discussion is going in [another topics](https://www.kaggle.com/discussions/product-feedback/461401). The problem ~~hasn't been solved yet~~ is solved. Thanks to the kaggle stuff @herbison. Please feel free to follow. ", "2561765": "@forcewithme can you share a notebook with me that was previously working? You can remove your inference code; I should just need the part of your pipeline that opens the images in order to continue debugging.", "2561176": "Same here. I've reported it to the [PRODUCT FEEDBACK](https://www.kaggle.com/discussions/product-feedback/461401). You may leave your comments too.", "2563826": "I am also encountering the same error (Notebook Out of Disk).\n\nI thought I had submitted a submit which should not have changed the amount of writes to disk, but the above error message was displayed.\n\nIs there any update here?", "2561429": "Hi @herbison @sohier , @jonathanmcwilliams , @ashleychow , Please help! It seems the problem remains. The new submissions still failed, reporting `notebook out of disk`. It has suspended my plan for this competition for 2 days. I think it also bothers many other competitors. ", "2561266": "@herbison I've seen you deal with similar issues on other competitions. That probably also falls within your patch of work. Could you please have a look at this?", "2561204": "Same here, all my submissions failed with \"Notebook Out of Disk\", despite they being just parameter changes of already completed submissions (these changes shouldn't influence disk usage at all).", "2561160": "Same here, \"Notebook Out of Disk\" with notebooks that ran before yesterday. Might be a global problem", "2561128": "Same here. I've had three submissions with an out-of-disk message, which I never had before. Further, I have a submission that is still currently running after 10 hours; I normally can do inference on the TRAIN data in about 10 hours. Considering that the test data is composed of a majority of TMAs and is only 550GB, I would have expected my submission to be completed by now. Something is not right.", "2569079": "I have another problem.\n\n\n\n\nwrite too much means not enough space on disk?\n\n\n", "2561758": "Is there a workaround till this issue is solved? For example, have anyone tried to submit without saving anything beside the submission.csv? " } }