![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F474194%2F58ccf1040fd52f03c4c85b346f2015b7%2Fexport_cookies.png?generation=1587732481592548&alt=media\")
After taking part in the DFDC competition, I got used to working with larger data sizes 😄 \nI learned much from this post of DFDC: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121194
\n\nThese are the steps that I take to download the files faster using aria2, thanks to @abhishek \n1. You download cookies when you are logged in on the Kaggle website. You can use a web extension for this. For example on Chrome I have an extension called cookies.txt. \nThe cookies can then be exported by the extension and saved as a text file cookies.txt:\n
On your machine you install aria2:\nsudo apt-get install aria2
Put the cookies.txt in the directory you want your data to be installed and then use following command to download the data:\naria2c -c -x 16 -s 16 --load-cookies cookies.txt -p https://www.kaggle.com/c/18647/download-all
Took me less than 20 minutes on a cloud instance
", "messages": [ { "id": 820420, "postDate": "2020-04-25T12:33:10.677Z", "content": "After taking part in the DFDC competition, I got used to working with larger data sizes 😄 \nI learned much from this post of DFDC: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121194
\n\nThese are the steps that I take to download the files faster using aria2, thanks to @abhishek \n1. You download cookies when you are logged in on the Kaggle website. You can use a web extension for this. For example on Chrome I have an extension called cookies.txt. \nThe cookies can then be exported by the extension and saved as a text file cookies.txt:\n
On your machine you install aria2:\nsudo apt-get install aria2
Put the cookies.txt in the directory you want your data to be installed and then use following command to download the data:\naria2c -c -x 16 -s 16 --load-cookies cookies.txt -p https://www.kaggle.com/c/18647/download-all
Took me less than 20 minutes on a cloud instance
", "rawMarkdown": "After taking part in the DFDC competition, I got used to working with larger data sizes 😄 \nI learned much from this post of DFDC: https://www.kaggle.com/c/deepfake-detection-challenge/discussion/121194\n\nThese are the steps that I take to download the files faster using aria2, thanks to @abhishek \n1. You download cookies when you are logged in on the Kaggle website. You can use a web extension for this. For example on Chrome I have an extension called [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg?hl=nl). \nThe cookies can then be exported by the extension and saved as a text file *cookies.txt*:\n\n\n2. On your machine you install aria2:\n`sudo apt-get install aria2`\n \n3. Put the cookies.txt in the directory you want your data to be installed and then use following command to download the data:\n`aria2c -c -x 16 -s 16 --load-cookies cookies.txt -p https://www.kaggle.com/c/18647/download-all`\n\nTook me less than 20 minutes on a cloud instance", "votes": 8 }, { "id": 903969, "postDate": "2020-06-27T08:52:41.557Z", "content": "Thanks a lot for your advice, my problem has been solved by using aria2. Thank you again.
", "rawMarkdown": "Thanks a lot for your advice, my problem has been solved by using aria2. Thank you again.", "votes": 1, "replies": [ { "id": 904110, "postDate": "2020-06-27T11:21:37.660Z", "content": "With pleasure :)
", "rawMarkdown": "With pleasure :)" } ] }, { "id": 826247, "postDate": "2020-04-29T14:30:00.877Z", "content": "Thanks @rafiko1 for sharing this.\nEarlier I was not aware of this before.
", "rawMarkdown": "Thanks @rafiko1 for sharing this.\nEarlier I was not aware of this before.", "votes": 1 }, { "id": 821435, "postDate": "2020-04-26T07:09:17.257Z", "content": "First, Congratulations on deep fake challenges. And thanks for this idea to download large data set efficiently. However, have you tried to download via colab? I was hoping to get all files in a single zip file. But it downloads all files separately.
", "rawMarkdown": "First, Congratulations on deep fake challenges. And thanks for this idea to download large data set efficiently. However, have you tried to download via colab? I was hoping to get all files in a single zip file. But it downloads all files separately. ", "votes": 1, "replies": [ { "id": 821547, "postDate": "2020-04-26T08:44:07.920Z", "content": "Thank you! No, I didn't try to download it on colab. Have you tried aria2 there? For me it downloads only a single zip file.
", "rawMarkdown": "Thank you! No, I didn't try to download it on colab. Have you tried aria2 there? For me it downloads only a single zip file.", "votes": 1 }, { "id": 821553, "postDate": "2020-04-26T08:46:58.097Z", "content": "Not yet. But I will. I will report you if I face any problem with aria2 there. :)
Thanks a lot for your advice, my problem has been solved by using aria2. Thank you again.
", "votes": 1, "replies": [ { "id": 904110, "author_name": "Rafi Hai", "author_url": "", "post_date": "2020-06-27T11:21:37.660000", "content": "With pleasure :)
", "votes": 0, "replies": [] } ] }, { "id": 826247, "author_name": "Sanket Parate", "author_url": "", "post_date": "2020-04-29T14:30:00.877000", "content": "Thanks @rafiko1 for sharing this.\nEarlier I was not aware of this before.
", "votes": 1, "replies": [] }, { "id": 821435, "author_name": "Innat", "author_url": "", "post_date": "2020-04-26T07:09:17.257000", "content": "First, Congratulations on deep fake challenges. And thanks for this idea to download large data set efficiently. However, have you tried to download via colab? I was hoping to get all files in a single zip file. But it downloads all files separately.
", "votes": 1, "replies": [ { "id": 821547, "author_name": "Rafi Hai", "author_url": "", "post_date": "2020-04-26T08:44:07.920000", "content": "Thank you! No, I didn't try to download it on colab. Have you tried aria2 there? For me it downloads only a single zip file.
", "votes": 1, "replies": [] }, { "id": 821553, "author_name": "Innat", "author_url": "", "post_date": "2020-04-26T08:46:58.097000", "content": "Not yet. But I will. I will report you if I face any problem with aria2 there. :)