I broke the images into 250,000 tiles and stored them in '/kaggle/working/tiles/', where they take up about 2.5GB. I want to have access to the tiles when I continue editing the kernel tomorrow, and so I tried to commit the kernel (from what I understand, this is how you preserve outputs). After the code had rerun, my kernel timed out, but I could see that a new version had been created. It said that it had an error, but I checked the notebook and there were no errors shown, and all of the tiles had been created. The 'Execution Info\" had a long error message that started with \"Failure Message: non-zero exit code: 123, stderr\". Does this mean that the tiles were not stored? If they were stored, how do I access them now? I already tried creating a new kernel, then going to \"add inputs\", and then searching for 'kernel input files', but my kernel doesn't even show up in the search.
", "messages": [ { "id": 909788, "postDate": "2020-06-30T20:28:56.750Z", "content": "You should try writing into a zip file (or a hdfs file). If you directly output thousands of images I'm pretty sure kaggle won't like that.
", "rawMarkdown": "You should try writing into a zip file (or a hdfs file). If you directly output thousands of images I'm pretty sure kaggle won't like that.", "votes": 1, "replies": [ { "id": 909984, "postDate": "2020-07-01T00:44:26.603Z", "content": "Do you think the number of files matter? The disc space is only 2.5GB, which is below the 5GB limit. Or is that limit only for active kernels? I'm not sure what amount of output can be saved.
", "rawMarkdown": "Do you think the number of files matter? The disc space is only 2.5GB, which is below the 5GB limit. Or is that limit only for active kernels? I'm not sure what amount of output can be saved." }, { "id": 909988, "postDate": "2020-07-01T00:48:38.853Z", "content": "I think I tried it once without zipping and it gave me an error. Could be that some process is listing the files in the output section of a kernel and too many make it crash.
\n\nNormally, if you use an archive as a dataset later on I'm pretty sure they unzip it anyway.
", "rawMarkdown": "I think I tried it once without zipping and it gave me an error. Could be that some process is listing the files in the output section of a kernel and too many make it crash.\n\nNormally, if you use an archive as a dataset later on I'm pretty sure they unzip it anyway." } ] }, { "id": 909776, "postDate": "2020-06-30T20:17:44.767Z", "content": "Sorry, I think it's not possible to access data of a kernel version that resulted in an error. I had a similar issue, too. Something was wrong with one or more images that I tried to split into tiles... or with my code :)
\n\nThis is probably far from beeing best practice but it helped me to move on:\n1. do a test commit and only write 250 files\n2. if that was successful wrap your tile export code in a try block
", "rawMarkdown": "Sorry, I think it's not possible to access data of a kernel version that resulted in an error. I had a similar issue, too. Something was wrong with one or more images that I tried to split into tiles... or with my code :)\n\nThis is probably far from beeing best practice but it helped me to move on:\n1. do a test commit and only write 250 files\n2. if that was successful wrap your tile export code in a try block", "votes": 1, "replies": [ { "id": 909982, "postDate": "2020-07-01T00:42:17.797Z", "content": "Those are good ideas. Thanks Jack.
", "rawMarkdown": "Those are good ideas. Thanks Jack." } ] }, { "id": 910471, "postDate": "2020-07-01T06:57:54.220Z", "content": "I have similar issues. I was running iafoss kernel which uses data from here --> https://www.kaggle.com/iafoss/panda-16x128x128-tiles-data
\n\nA few days ago, it was running OK & I even committed it. But yesterday, I run again & it crashed, as kernel can't see the files in train.zip and mask.zip (previously it was OK reading from zipped files, I don't need to unzip files in my code)
", "rawMarkdown": "I have similar issues. I was running iafoss kernel which uses data from here --> https://www.kaggle.com/iafoss/panda-16x128x128-tiles-data\n\nA few days ago, it was running OK & I even committed it. But yesterday, I run again & it crashed, as kernel can't see the files in train.zip and mask.zip (previously it was OK reading from zipped files, I don't need to unzip files in my code)" }, { "id": 909624, "postDate": "2020-06-30T17:43:02.437Z", "content": "I broke the images into 250,000 tiles and stored them in '/kaggle/working/tiles/', where they take up about 2.5GB. I want to have access to the tiles when I continue editing the kernel tomorrow, and so I tried to commit the kernel (from what I understand, this is how you preserve outputs). After the code had rerun, my kernel timed out, but I could see that a new version had been created. It said that it had an error, but I checked the notebook and there were no errors shown, and all of the tiles had been created. The 'Execution Info\" had a long error message that started with \"Failure Message: non-zero exit code: 123, stderr\". Does this mean that the tiles were not stored? If they were stored, how do I access them now? I already tried creating a new kernel, then going to \"add inputs\", and then searching for 'kernel input files', but my kernel doesn't even show up in the search.
", "rawMarkdown": "I broke the images into 250,000 tiles and stored them in '/kaggle/working/tiles/', where they take up about 2.5GB. I want to have access to the tiles when I continue editing the kernel tomorrow, and so I tried to commit the kernel (from what I understand, this is how you preserve outputs). After the code had rerun, my kernel timed out, but I could see that a new version had been created. It said that it had an error, but I checked the notebook and there were no errors shown, and all of the tiles had been created. The 'Execution Info\" had a long error message that started with \"Failure Message: non-zero exit code: 123, stderr\". Does this mean that the tiles were not stored? If they were stored, how do I access them now? I already tried creating a new kernel, then going to \"add inputs\", and then searching for 'kernel input files', but my kernel doesn't even show up in the search." } ], "comments": [ { "id": 909788, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-06-30T20:28:56.750000", "content": "You should try writing into a zip file (or a hdfs file). If you directly output thousands of images I'm pretty sure kaggle won't like that.
", "votes": 1, "replies": [ { "id": 909984, "author_name": "shk", "author_url": "", "post_date": "2020-07-01T00:44:26.603000", "content": "Do you think the number of files matter? The disc space is only 2.5GB, which is below the 5GB limit. Or is that limit only for active kernels? I'm not sure what amount of output can be saved.
", "votes": 0, "replies": [] }, { "id": 909988, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-07-01T00:48:38.853000", "content": "I think I tried it once without zipping and it gave me an error. Could be that some process is listing the files in the output section of a kernel and too many make it crash.
\n\nNormally, if you use an archive as a dataset later on I'm pretty sure they unzip it anyway.
", "votes": 0, "replies": [] } ] }, { "id": 909776, "author_name": "Jack Byte", "author_url": "", "post_date": "2020-06-30T20:17:44.767000", "content": "Sorry, I think it's not possible to access data of a kernel version that resulted in an error. I had a similar issue, too. Something was wrong with one or more images that I tried to split into tiles... or with my code :)
\n\nThis is probably far from beeing best practice but it helped me to move on:\n1. do a test commit and only write 250 files\n2. if that was successful wrap your tile export code in a try block
", "votes": 1, "replies": [ { "id": 909982, "author_name": "shk", "author_url": "", "post_date": "2020-07-01T00:42:17.797000", "content": "Those are good ideas. Thanks Jack.
", "votes": 0, "replies": [] } ] }, { "id": 910471, "author_name": "Sidney Ng", "author_url": "", "post_date": "2020-07-01T06:57:54.220000", "content": "I have similar issues. I was running iafoss kernel which uses data from here --> https://www.kaggle.com/iafoss/panda-16x128x128-tiles-data
\n\nA few days ago, it was running OK & I even committed it. But yesterday, I run again & it crashed, as kernel can't see the files in train.zip and mask.zip (previously it was OK reading from zipped files, I don't need to unzip files in my code)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "909788": "You should try writing into a zip file (or a hdfs file). If you directly output thousands of images I'm pretty sure kaggle won't like that.", "909776": "Sorry, I think it's not possible to access data of a kernel version that resulted in an error. I had a similar issue, too. Something was wrong with one or more images that I tried to split into tiles... or with my code :)\n\nThis is probably far from beeing best practice but it helped me to move on:\n1. do a test commit and only write 250 files\n2. if that was successful wrap your tile export code in a try block", "910471": "I have similar issues. I was running iafoss kernel which uses data from here --> https://www.kaggle.com/iafoss/panda-16x128x128-tiles-data\n\nA few days ago, it was running OK & I even committed it. But yesterday, I run again & it crashed, as kernel can't see the files in train.zip and mask.zip (previously it was OK reading from zipped files, I don't need to unzip files in my code)", "909624": "I broke the images into 250,000 tiles and stored them in '/kaggle/working/tiles/', where they take up about 2.5GB. I want to have access to the tiles when I continue editing the kernel tomorrow, and so I tried to commit the kernel (from what I understand, this is how you preserve outputs). After the code had rerun, my kernel timed out, but I could see that a new version had been created. It said that it had an error, but I checked the notebook and there were no errors shown, and all of the tiles had been created. The 'Execution Info\" had a long error message that started with \"Failure Message: non-zero exit code: 123, stderr\". Does this mean that the tiles were not stored? If they were stored, how do I access them now? I already tried creating a new kernel, then going to \"add inputs\", and then searching for 'kernel input files', but my kernel doesn't even show up in the search." } }