Hey,
\nThis challenge seems very interesting but do you recommend participating in this challenge having only GPU quota from Kaggle? I mean there is tons of data and I doubt whether 12 hours available for a session is enough.
Any thoughts?
\nThanks in advance,
\nJan
Hey,
\nThis challenge seems very interesting but do you recommend participating in this challenge having only GPU quota from Kaggle? I mean there is tons of data and I doubt whether 12 hours available for a session is enough.
Any thoughts?
\nThanks in advance,
\nJan
If I was going to use notebooks only, I wouldn't join this competition.
", "rawMarkdown": "If I was going to use notebooks only, I wouldn't join this competition.", "votes": 1 }, { "id": 2367594, "postDate": "2023-07-31T16:52:58.047Z", "content": "The challenge is pretty interesting - but running it only on kaggle - not sure I would do it.
\nBreaking training time into a couple of notebooks (if needed) is pretty easy to do. However, the weekly quota will be the problem. On my local machines I expect to do a wide range of experiments I will pretty much have at least 4 machines running 24/7 and will easily exceed the quota of hours. My Python skills are pretty low level so I can only make small changes per experiment :)
\nRunning the predictions will burn additional quota time. I do all training local and only use kaggle for the predictions code - it's not easy to stay within the quota IF you have a lot of different models to score and tend to use up the full 5 submissions per day.
\nA potential solution is to find someone who has lots of resources and team up. This works best if your pretty sharp in Python and the person with resources is like myself. But do team up with only someone where the profile shows they have been very active on kaggle in recent months.
", "rawMarkdown": "The challenge is pretty interesting - but running it only on kaggle - not sure I would do it.\n\nBreaking training time into a couple of notebooks (if needed) is pretty easy to do. However, the weekly quota will be the problem. On my local machines I expect to do a wide range of experiments I will pretty much have at least 4 machines running 24/7 and will easily exceed the quota of hours. My Python skills are pretty low level so I can only make small changes per experiment :)\n\nRunning the predictions will burn additional quota time. I do all training local and only use kaggle for the predictions code - it's not easy to stay within the quota IF you have a lot of different models to score and tend to use up the full 5 submissions per day.\n\nA potential solution is to find someone who has lots of resources and team up. This works best if your pretty sharp in Python and the person with resources is like myself. But do team up with only someone where the profile shows they have been very active on kaggle in recent months. ", "votes": 1 }, { "id": 2367072, "postDate": "2023-07-31T10:40:26.473Z", "content": "i have a solution like the following don't know if it is accepted or not? You will train the model in \"notebook 1\", it can fight with GPU runtime more than 12 hours, you get the model. Download and then re-upload kaggle as a dataset. In the new notebook \"notebook 2\", you only do the prediction and submission task using the model you uploaded. I guess if your notebook only does the prediction task without the time consuming training it will take very little less than 12 hours
", "rawMarkdown": "i have a solution like the following don't know if it is accepted or not? You will train the model in \"notebook 1\", it can fight with GPU runtime more than 12 hours, you get the model. Download and then re-upload kaggle as a dataset. In the new notebook \"notebook 2\", you only do the prediction and submission task using the model you uploaded. I guess if your notebook only does the prediction task without the time consuming training it will take very little less than 12 hours", "votes": 1, "replies": [ { "id": 2367096, "postDate": "2023-07-31T10:54:10.883Z", "content": "Nah, the notebook can't run more than 12 hours. Kaggle will interrupt it. And for submission, you have 9 hours of runtime
", "rawMarkdown": "Nah, the notebook can't run more than 12 hours. Kaggle will interrupt it. And for submission, you have 9 hours of runtime", "replies": [ { "id": 2367215, "postDate": "2023-07-31T12:39:28.497Z", "content": "oh. can we save weights and load it in other notebook and continue training? (in other accounts). And the notebook that we submit just make prediction with trained model. I think the training process take many time and the predicting process take less time
", "rawMarkdown": "oh. can we save weights and load it in other notebook and continue training? (in other accounts). And the notebook that we submit just make prediction with trained model. I think the training process take many time and the predicting process take less time", "votes": 1 }, { "id": 2387240, "postDate": "2023-08-12T14:18:06.517Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2367451, "postDate": "2023-07-31T14:55:52.367Z", "content": "@nguyenhoainam27 your idea to train elsewhere and predict on the submission kernel is the preferred way to attempt code competitions on Kaggle.
\nUsing paperspace/ vast.ai/ colab pro+ or AWS cluster for extra GPUs is also feasible in my opinion.
quite expensive for a single home user :/
\nBtw, do you know how much is a single \"computing unit\" in colab? Something like how long i can run 24GB RTX3090 for one \"computing unit\"?
\nvast.ai seems the most reasonable
", "rawMarkdown": "quite expensive for a single home user :/\n\nBtw, do you know how much is a single \"computing unit\" in colab? Something like how long i can run 24GB RTX3090 for one \"computing unit\"?\n\nvast.ai seems the most reasonable" } ] } ] }, { "id": 2395327, "postDate": "2023-08-17T13:06:41.860Z", "content": "I used to use Colab TPU before I got my desktop.
", "rawMarkdown": "I used to use Colab TPU before I got my desktop." }, { "id": 2372449, "postDate": "2023-08-03T17:16:24.123Z", "content": "As others have suggested, you can save your state and break training over multiple sessions. One other suggestion is use the 20 hours you get for TPU training also. there is a starter notebook by @awsaf49: https://www.kaggle.com/code/awsaf49/rsna-atd-cnn-tpu-train
", "rawMarkdown": "As others have suggested, you can save your state and break training over multiple sessions. One other suggestion is use the 20 hours you get for TPU training also. there is a starter notebook by @awsaf49: https://www.kaggle.com/code/awsaf49/rsna-atd-cnn-tpu-train", "replies": [ { "id": 2388247, "postDate": "2023-08-13T09:00:15.790Z", "content": "I’m kinda surprised this is allowed. I know it’s technically possible to save weights and reload them for inference, but I thought the purpose of the 9-hour rule was that training and inference are bounded to force us into simpler models, rather than just a pay-to-win scenario. Otherwise, can’t someone with access to a lot of compute just win the competition by uploading a dataset of their weights and running inference on Kaggle?
", "rawMarkdown": "I’m kinda surprised this is allowed. I know it’s technically possible to save weights and reload them for inference, but I thought the purpose of the 9-hour rule was that training and inference are bounded to force us into simpler models, rather than just a pay-to-win scenario. Otherwise, can’t someone with access to a lot of compute just win the competition by uploading a dataset of their weights and running inference on Kaggle?", "votes": 1, "replies": [ { "id": 2388937, "postDate": "2023-08-13T17:30:50.780Z", "content": "9-hour rule is only for inference, where one can use weights trained outside the notebook. Sadly Kaggle is pay-to-win a little :/ One with more significant computational supplies can benefit from extensive grid search or running a wide range of experiments, which is inaccessible for 30h-quota players.
", "rawMarkdown": " 9-hour rule is only for inference, where one can use weights trained outside the notebook. Sadly Kaggle is pay-to-win a little :/ One with more significant computational supplies can benefit from extensive grid search or running a wide range of experiments, which is inaccessible for 30h-quota players.", "votes": 1 }, { "id": 2448178, "postDate": "2023-09-20T13:01:48.813Z", "content": "How should we load trained state dict in inference notebook because I am unable to sync my working/ directory among sessions or notebooks.
\nAnd the competition only allows public dataset so if i upload my state dict as dataset and make it public everyone will be able to see it.
If I was going to use notebooks only, I wouldn't join this competition.
", "votes": 1, "replies": [] }, { "id": 2367594, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2023-07-31T16:52:58.047000", "content": "The challenge is pretty interesting - but running it only on kaggle - not sure I would do it.
\nBreaking training time into a couple of notebooks (if needed) is pretty easy to do. However, the weekly quota will be the problem. On my local machines I expect to do a wide range of experiments I will pretty much have at least 4 machines running 24/7 and will easily exceed the quota of hours. My Python skills are pretty low level so I can only make small changes per experiment :)
\nRunning the predictions will burn additional quota time. I do all training local and only use kaggle for the predictions code - it's not easy to stay within the quota IF you have a lot of different models to score and tend to use up the full 5 submissions per day.
\nA potential solution is to find someone who has lots of resources and team up. This works best if your pretty sharp in Python and the person with resources is like myself. But do team up with only someone where the profile shows they have been very active on kaggle in recent months.
", "votes": 1, "replies": [] }, { "id": 2367072, "author_name": "Vietnamese-NHNAM", "author_url": "", "post_date": "2023-07-31T10:40:26.473000", "content": "i have a solution like the following don't know if it is accepted or not? You will train the model in \"notebook 1\", it can fight with GPU runtime more than 12 hours, you get the model. Download and then re-upload kaggle as a dataset. In the new notebook \"notebook 2\", you only do the prediction and submission task using the model you uploaded. I guess if your notebook only does the prediction task without the time consuming training it will take very little less than 12 hours
", "votes": 1, "replies": [ { "id": 2367096, "author_name": "JanGlinko2", "author_url": "", "post_date": "2023-07-31T10:54:10.883000", "content": "Nah, the notebook can't run more than 12 hours. Kaggle will interrupt it. And for submission, you have 9 hours of runtime
", "votes": 0, "replies": [ { "id": 2367215, "author_name": "Vietnamese-NHNAM", "author_url": "", "post_date": "2023-07-31T12:39:28.497000", "content": "oh. can we save weights and load it in other notebook and continue training? (in other accounts). And the notebook that we submit just make prediction with trained model. I think the training process take many time and the predicting process take less time
", "votes": 1, "replies": [] }, { "id": 2387240, "author_name": "", "author_url": "", "post_date": "2023-08-12T14:18:06.517000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2367451, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2023-07-31T14:55:52.367000", "content": "@nguyenhoainam27 your idea to train elsewhere and predict on the submission kernel is the preferred way to attempt code competitions on Kaggle.
\nUsing paperspace/ vast.ai/ colab pro+ or AWS cluster for extra GPUs is also feasible in my opinion.
quite expensive for a single home user :/
\nBtw, do you know how much is a single \"computing unit\" in colab? Something like how long i can run 24GB RTX3090 for one \"computing unit\"?
\nvast.ai seems the most reasonable
", "votes": 0, "replies": [] } ] } ] }, { "id": 2395327, "author_name": "junseonglee11", "author_url": "", "post_date": "2023-08-17T13:06:41.860000", "content": "I used to use Colab TPU before I got my desktop.
", "votes": 0, "replies": [] }, { "id": 2372449, "author_name": "coderRKJ", "author_url": "", "post_date": "2023-08-03T17:16:24.123000", "content": "As others have suggested, you can save your state and break training over multiple sessions. One other suggestion is use the 20 hours you get for TPU training also. there is a starter notebook by @awsaf49: https://www.kaggle.com/code/awsaf49/rsna-atd-cnn-tpu-train
", "votes": 0, "replies": [ { "id": 2388247, "author_name": "zacstewart", "author_url": "", "post_date": "2023-08-13T09:00:15.790000", "content": "I’m kinda surprised this is allowed. I know it’s technically possible to save weights and reload them for inference, but I thought the purpose of the 9-hour rule was that training and inference are bounded to force us into simpler models, rather than just a pay-to-win scenario. Otherwise, can’t someone with access to a lot of compute just win the competition by uploading a dataset of their weights and running inference on Kaggle?
", "votes": 1, "replies": [ { "id": 2388937, "author_name": "JanGlinko2", "author_url": "", "post_date": "2023-08-13T17:30:50.780000", "content": "9-hour rule is only for inference, where one can use weights trained outside the notebook. Sadly Kaggle is pay-to-win a little :/ One with more significant computational supplies can benefit from extensive grid search or running a wide range of experiments, which is inaccessible for 30h-quota players.
", "votes": 1, "replies": [] }, { "id": 2448178, "author_name": "Type4Human", "author_url": "", "post_date": "2023-09-20T13:01:48.813000", "content": "How should we load trained state dict in inference notebook because I am unable to sync my working/ directory among sessions or notebooks.
\nAnd the competition only allows public dataset so if i upload my state dict as dataset and make it public everyone will be able to see it.