Anyone still having trouble downloading the data through kaggle api? I think i am having trouble with the dataset.in colab i tried to download this competition dataset through it's api.i was downloading that 224x224 png image dataset of tom,i can see in output tab that the dataset has downloaded but Whenever i try to expand the dataset to see images,i see nothing and sometimes it keeps on crashing. I felt extremely disappointed, then downloaded the full dataset in my local computer to upload manually in google drive.but when i try to upload the dataset after 4/5 minutes my browser crashes and computer stops working, anyone having similar problem? Stage-2 is coming bit worried about this gigantic dataset because in stage-2 i assume stage-1's train and test dataset will be merged for stage-2 training set means we get much bigger dataset.am i wrong?
", "messages": [ { "id": 664867, "postDate": "2019-11-04T10:20:15.947Z", "content": "@mobassir , I had the same experience on Colab with this competition's data. This made me have a 2nd thought about submitting for stage-2 since the training data will be even larger. The Kaggle 30G per week for this type of competition is more than a joke really. What were they think???
", "rawMarkdown": "@mobassir , I had the same experience on Colab with this competition's data. This made me have a 2nd thought about submitting for stage-2 since the training data will be even larger. The Kaggle 30G per week for this type of competition is more than a joke really. What were they think???", "votes": 1, "replies": [ { "id": 664875, "postDate": "2019-11-04T10:30:04.490Z", "content": "some people publish way of getting 0.066 for personal gpu users and on the other hand kaggle sets gpu quota..!!\ni am also very frustrated mate :(\nreally very helpless :(
", "rawMarkdown": "some people publish way of getting 0.066 for personal gpu users and on the other hand kaggle sets gpu quota..!!\ni am also very frustrated mate :(\nreally very helpless :( " } ] }, { "id": 664353, "postDate": "2019-11-03T14:41:35.800Z", "content": "I think kaggle's GPU is also not reliable,\nthe connection is easy to fail in China.
", "rawMarkdown": "I think kaggle's GPU is also not reliable,\nthe connection is easy to fail in China.", "votes": 1, "replies": [ { "id": 664800, "postDate": "2019-11-04T08:38:41.690Z", "content": "why it is failing in china?
", "rawMarkdown": "why it is failing in china?" }, { "id": 664895, "postDate": "2019-11-04T11:12:32.193Z", "content": "when you train, and when the train stage is not completed,then the kaggle notebook will lose connection.\nIt always happen.
", "rawMarkdown": "when you train, and when the train stage is not completed,then the kaggle notebook will lose connection.\nIt always happen." } ] }, { "id": 664115, "postDate": "2019-11-03T07:21:03.793Z", "content": "The stage 2 dataset is 156GB training set( stage 2 training set = stage 1 training set + stage 1 test set ) + 27GB test set (stage 2 test set).
", "rawMarkdown": "The stage 2 dataset is 156GB training set( stage 2 training set = stage 1 training set + stage 1 test set ) + 27GB test set (stage 2 test set).", "votes": 1, "replies": [ { "id": 664194, "postDate": "2019-11-03T10:03:01.230Z", "content": "more trouble up in front :(
", "rawMarkdown": "more trouble up in front :(" } ] }, { "id": 664288, "postDate": "2019-11-03T13:02:53.683Z", "content": "@mobassir - Google Colab is highly unreliable. I have faced so many instances where the code was executing for 6 hours and suddenly I would be getting an error message, no GPU. Unfortunately, even Google Cloud is also not that great. Even if internet connectivity goes for one second, entire code goes for a toss. Almost all GPU's online except for Kaggle Kernel GPU's seems to be highly unreliable.
", "rawMarkdown": "@mobassir - Google Colab is highly unreliable. I have faced so many instances where the code was executing for 6 hours and suddenly I would be getting an error message, no GPU. Unfortunately, even Google Cloud is also not that great. Even if internet connectivity goes for one second, entire code goes for a toss. Almost all GPU's online except for Kaggle Kernel GPU's seems to be highly unreliable. ", "votes": 2, "replies": [ { "id": 664307, "postDate": "2019-11-03T13:27:55.350Z", "content": "@manojprabhaakr \nyou are right mate :(
", "rawMarkdown": "@manojprabhaakr \nyou are right mate :( ", "votes": 1 }, { "id": 665044, "postDate": "2019-11-04T15:14:08.257Z", "content": "in googlce cloud one must start session as tmux .. if internet goes off u can always reconnect to same notebook once internet is back
", "rawMarkdown": "in googlce cloud one must start session as tmux .. if internet goes off u can always reconnect to same notebook once internet is back" } ] }, { "id": 664086, "postDate": "2019-11-03T06:08:25.043Z", "content": "Anyone still having trouble downloading the data through kaggle api? I think i am having trouble with the dataset.in colab i tried to download this competition dataset through it's api.i was downloading that 224x224 png image dataset of tom,i can see in output tab that the dataset has downloaded but Whenever i try to expand the dataset to see images,i see nothing and sometimes it keeps on crashing. I felt extremely disappointed, then downloaded the full dataset in my local computer to upload manually in google drive.but when i try to upload the dataset after 4/5 minutes my browser crashes and computer stops working, anyone having similar problem? Stage-2 is coming bit worried about this gigantic dataset because in stage-2 i assume stage-1's train and test dataset will be merged for stage-2 training set means we get much bigger dataset.am i wrong?
", "rawMarkdown": "Anyone still having trouble downloading the data through kaggle api? I think i am having trouble with the dataset.in colab i tried to download this competition dataset through it's api.i was downloading that 224x224 png image dataset of tom,i can see in output tab that the dataset has downloaded but Whenever i try to expand the dataset to see images,i see nothing and sometimes it keeps on crashing. I felt extremely disappointed, then downloaded the full dataset in my local computer to upload manually in google drive.but when i try to upload the dataset after 4/5 minutes my browser crashes and computer stops working, anyone having similar problem? Stage-2 is coming bit worried about this gigantic dataset because in stage-2 i assume stage-1's train and test dataset will be merged for stage-2 training set means we get much bigger dataset.am i wrong? \n", "votes": 2 }, { "id": 664193, "postDate": "2019-11-03T09:58:40.537Z", "content": "@Mobassir In my experience, colab crashes often when the dataset size is more than a few gigabytes (more generally, when the dataset is more than 6 or 7 gigabytes). And you are trying to download the 224x224 png images which I think amounts to 16 gigabytes in total (train + test). Most probably, the reason for crashing is the dataset size.
\n\nTry sticking to Kaggle kernels for this competition. I know that the 30 hour / week GPU limit is a problem but we need to make the best use of it. This is one of those few vision competitions where we need extensive GPU power. Even I am facing problems with my GTX 1060 (decent GPU, but not good enough for this competition).
\n\nAnd best of luck for 2nd Stage.
", "rawMarkdown": "@Mobassir In my experience, colab crashes often when the dataset size is more than a few gigabytes (more generally, when the dataset is more than 6 or 7 gigabytes). And you are trying to download the 224x224 png images which I think amounts to 16 gigabytes in total (train + test). Most probably, the reason for crashing is the dataset size. \n\nTry sticking to Kaggle kernels for this competition. I know that the 30 hour / week GPU limit is a problem but we need to make the best use of it. This is one of those few vision competitions where we need extensive GPU power. Even I am facing problems with my GTX 1060 (decent GPU, but not good enough for this competition). \n\n**And best of luck for 2nd Stage.**", "replies": [ { "id": 664197, "postDate": "2019-11-03T10:12:07.487Z", "content": "lots of 0.66 ,courtesy : own gpu \non the other hand colab is not responding and kaggle sets gpu quota,this competition is gone for people like me :(
", "rawMarkdown": "lots of 0.66 ,courtesy : own gpu \non the other hand colab is not responding and kaggle sets gpu quota,this competition is gone for people like me :(" } ] }, { "id": 664351, "postDate": "2019-11-03T14:39:04.120Z", "content": "Hi all ,I have an idea....\nfor online GPU,if connection failed...\nwe can upload the temporary model to google drive.\nand then download it and recover from it.
", "rawMarkdown": "Hi all ,I have an idea....\nfor online GPU,if connection failed...\nwe can upload the temporary model to google drive.\nand then download it and recover from it.", "replies": [ { "id": 664799, "postDate": "2019-11-04T08:38:16.100Z", "content": "you can't upload more than 15gb file in google drive
", "rawMarkdown": "you can't upload more than 15gb file in google drive" }, { "id": 664808, "postDate": "2019-11-04T08:49:18.450Z", "content": "ask student to help you ,if you have an student google drive account,you will have great storage.
", "rawMarkdown": "ask student to help you ,if you have an student google drive account,you will have great storage." }, { "id": 664856, "postDate": "2019-11-04T10:09:53.393Z", "content": "yeah,i am student and i have my varsity email and using that email and google drive i can get huge amount of storage but not everyone can get the same
", "rawMarkdown": "yeah,i am student and i have my varsity email and using that email and google drive i can get huge amount of storage but not everyone can get the same" } ] }, { "id": 664350, "postDate": "2019-11-03T14:37:46.033Z", "content": "will team have more quota?
", "rawMarkdown": "will team have more quota?", "replies": [ { "id": 664798, "postDate": "2019-11-04T08:37:43.560Z", "content": "that means?
", "rawMarkdown": "that means?" }, { "id": 664801, "postDate": "2019-11-04T08:39:41.093Z", "content": "Does team-style will have more quota of GPU?\nI am not sure,I just guess.
", "rawMarkdown": "Does team-style will have more quota of GPU?\nI am not sure,I just guess." }, { "id": 664803, "postDate": "2019-11-04T08:43:35.560Z", "content": "each and every team mate gets 30 hours gpu quota per week
", "rawMarkdown": "each and every team mate gets 30 hours gpu quota per week" } ] } ], "comments": [ { "id": 664867, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-11-04T10:20:15.947000", "content": "@mobassir , I had the same experience on Colab with this competition's data. This made me have a 2nd thought about submitting for stage-2 since the training data will be even larger. The Kaggle 30G per week for this type of competition is more than a joke really. What were they think???
", "votes": 1, "replies": [ { "id": 664875, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-04T10:30:04.490000", "content": "some people publish way of getting 0.066 for personal gpu users and on the other hand kaggle sets gpu quota..!!\ni am also very frustrated mate :(\nreally very helpless :(
", "votes": 0, "replies": [] } ] }, { "id": 664353, "author_name": "Tian Bingyang", "author_url": "", "post_date": "2019-11-03T14:41:35.800000", "content": "I think kaggle's GPU is also not reliable,\nthe connection is easy to fail in China.
", "votes": 1, "replies": [ { "id": 664800, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-04T08:38:41.690000", "content": "why it is failing in china?
", "votes": 0, "replies": [] }, { "id": 664895, "author_name": "Tian Bingyang", "author_url": "", "post_date": "2019-11-04T11:12:32.193000", "content": "when you train, and when the train stage is not completed,then the kaggle notebook will lose connection.\nIt always happen.
", "votes": 0, "replies": [] } ] }, { "id": 664115, "author_name": "Zehao Zhao", "author_url": "", "post_date": "2019-11-03T07:21:03.793000", "content": "The stage 2 dataset is 156GB training set( stage 2 training set = stage 1 training set + stage 1 test set ) + 27GB test set (stage 2 test set).
", "votes": 1, "replies": [ { "id": 664194, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-03T10:03:01.230000", "content": "more trouble up in front :(
", "votes": 0, "replies": [] } ] }, { "id": 664288, "author_name": "Manoj Prabhakar", "author_url": "", "post_date": "2019-11-03T13:02:53.683000", "content": "@mobassir - Google Colab is highly unreliable. I have faced so many instances where the code was executing for 6 hours and suddenly I would be getting an error message, no GPU. Unfortunately, even Google Cloud is also not that great. Even if internet connectivity goes for one second, entire code goes for a toss. Almost all GPU's online except for Kaggle Kernel GPU's seems to be highly unreliable.
", "votes": 2, "replies": [ { "id": 664307, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-03T13:27:55.350000", "content": "@manojprabhaakr \nyou are right mate :(
", "votes": 1, "replies": [] }, { "id": 665044, "author_name": "Jaideep", "author_url": "", "post_date": "2019-11-04T15:14:08.257000", "content": "in googlce cloud one must start session as tmux .. if internet goes off u can always reconnect to same notebook once internet is back
", "votes": 0, "replies": [] } ] }, { "id": 664193, "author_name": "Sovit Ranjan Rath", "author_url": "", "post_date": "2019-11-03T09:58:40.537000", "content": "@Mobassir In my experience, colab crashes often when the dataset size is more than a few gigabytes (more generally, when the dataset is more than 6 or 7 gigabytes). And you are trying to download the 224x224 png images which I think amounts to 16 gigabytes in total (train + test). Most probably, the reason for crashing is the dataset size.
\n\nTry sticking to Kaggle kernels for this competition. I know that the 30 hour / week GPU limit is a problem but we need to make the best use of it. This is one of those few vision competitions where we need extensive GPU power. Even I am facing problems with my GTX 1060 (decent GPU, but not good enough for this competition).
\n\nAnd best of luck for 2nd Stage.
", "votes": 0, "replies": [ { "id": 664197, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-03T10:12:07.487000", "content": "lots of 0.66 ,courtesy : own gpu \non the other hand colab is not responding and kaggle sets gpu quota,this competition is gone for people like me :(
", "votes": 0, "replies": [] } ] }, { "id": 664351, "author_name": "Tian Bingyang", "author_url": "", "post_date": "2019-11-03T14:39:04.120000", "content": "Hi all ,I have an idea....\nfor online GPU,if connection failed...\nwe can upload the temporary model to google drive.\nand then download it and recover from it.
", "votes": 0, "replies": [ { "id": 664799, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-04T08:38:16.100000", "content": "you can't upload more than 15gb file in google drive
", "votes": 0, "replies": [] }, { "id": 664808, "author_name": "Tian Bingyang", "author_url": "", "post_date": "2019-11-04T08:49:18.450000", "content": "ask student to help you ,if you have an student google drive account,you will have great storage.
", "votes": 0, "replies": [] }, { "id": 664856, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-04T10:09:53.393000", "content": "yeah,i am student and i have my varsity email and using that email and google drive i can get huge amount of storage but not everyone can get the same
", "votes": 0, "replies": [] } ] }, { "id": 664350, "author_name": "Tian Bingyang", "author_url": "", "post_date": "2019-11-03T14:37:46.033000", "content": "will team have more quota?
", "votes": 0, "replies": [ { "id": 664798, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-04T08:37:43.560000", "content": "that means?
", "votes": 0, "replies": [] }, { "id": 664801, "author_name": "Tian Bingyang", "author_url": "", "post_date": "2019-11-04T08:39:41.093000", "content": "Does team-style will have more quota of GPU?\nI am not sure,I just guess.
", "votes": 0, "replies": [] }, { "id": 664803, "author_name": "Mobassir", "author_url": "", "post_date": "2019-11-04T08:43:35.560000", "content": "each and every team mate gets 30 hours gpu quota per week
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "664867": "@mobassir , I had the same experience on Colab with this competition's data. This made me have a 2nd thought about submitting for stage-2 since the training data will be even larger. The Kaggle 30G per week for this type of competition is more than a joke really. What were they think???", "664353": "I think kaggle's GPU is also not reliable,\nthe connection is easy to fail in China.", "664115": "The stage 2 dataset is 156GB training set( stage 2 training set = stage 1 training set + stage 1 test set ) + 27GB test set (stage 2 test set).", "664288": "@mobassir - Google Colab is highly unreliable. I have faced so many instances where the code was executing for 6 hours and suddenly I would be getting an error message, no GPU. Unfortunately, even Google Cloud is also not that great. Even if internet connectivity goes for one second, entire code goes for a toss. Almost all GPU's online except for Kaggle Kernel GPU's seems to be highly unreliable. ", "664086": "Anyone still having trouble downloading the data through kaggle api? I think i am having trouble with the dataset.in colab i tried to download this competition dataset through it's api.i was downloading that 224x224 png image dataset of tom,i can see in output tab that the dataset has downloaded but Whenever i try to expand the dataset to see images,i see nothing and sometimes it keeps on crashing. I felt extremely disappointed, then downloaded the full dataset in my local computer to upload manually in google drive.but when i try to upload the dataset after 4/5 minutes my browser crashes and computer stops working, anyone having similar problem? Stage-2 is coming bit worried about this gigantic dataset because in stage-2 i assume stage-1's train and test dataset will be merged for stage-2 training set means we get much bigger dataset.am i wrong? \n", "664193": "@Mobassir In my experience, colab crashes often when the dataset size is more than a few gigabytes (more generally, when the dataset is more than 6 or 7 gigabytes). And you are trying to download the 224x224 png images which I think amounts to 16 gigabytes in total (train + test). Most probably, the reason for crashing is the dataset size. \n\nTry sticking to Kaggle kernels for this competition. I know that the 30 hour / week GPU limit is a problem but we need to make the best use of it. This is one of those few vision competitions where we need extensive GPU power. Even I am facing problems with my GTX 1060 (decent GPU, but not good enough for this competition). \n\n**And best of luck for 2nd Stage.**", "664351": "Hi all ,I have an idea....\nfor online GPU,if connection failed...\nwe can upload the temporary model to google drive.\nand then download it and recover from it.", "664350": "will team have more quota?" } }