Is anyone else noticing that they are unable to attach the V100 GPU to their instances? I can attach other GPU's but the V100's across all the servers seem to be in use - both preempted and non-preempted instances.
\n\nIf you are able to attach a V100 GPU to your instance, please share the region, location and whether you're using preempted or not. Thanks.\n
\nEdit: I just tried the us-west1-a on 10/8 and it works - V100 preemptible. Thanks a lot guys!
Got a V100 on us-west1-a.
I created a new instance with V100 yesterday. I tried several zones, but V100 wasn't available until I tried us-west1-a.
Is anyone else noticing that they are unable to attach the V100 GPU to their instances? I can attach other GPU's but the V100's across all the servers seem to be in use - both preempted and non-preempted instances.
\n\nIf you are able to attach a V100 GPU to your instance, please share the region, location and whether you're using preempted or not. Thanks.\n
\nEdit: I just tried the us-west1-a on 10/8 and it works - V100 preemptible. Thanks a lot guys!
Try asia-east1-c
", "rawMarkdown": "Try asia-east1-c" }, { "id": 644050, "postDate": "2019-10-08T08:48:34.273Z", "content": "Yesterday I got a 1 * V100 instance on us-west1-b.
Hi guys, I’ve never used V100 before, just want to know : what is a training speed boost you get with V100 over P100 for basic baseline models?
", "rawMarkdown": "Hi guys, I’ve never used V100 before, just want to know : what is a training speed boost you get with V100 over P100 for basic baseline models?", "replies": [ { "id": 643614, "postDate": "2019-10-07T17:26:20.803Z", "content": "I wish I could say based on experience. However, based on the fp32(single precision) TFLOPs, The V100 has 14TFLOPS, while P100 has 9.3TFLOPS. In most cases you'll be comparing fp32, but the V100 does have tensor cores that can run larger batch sizes in mixed precision which would likely speed up training and inference. I'll update you once I get get my hands on the a VM with a V100.
", "rawMarkdown": "I wish I could say based on experience. However, based on the fp32(single precision) TFLOPs, The V100 has 14TFLOPS, while P100 has 9.3TFLOPS. In most cases you'll be comparing fp32, but the V100 does have tensor cores that can run larger batch sizes in mixed precision which would likely speed up training and inference. I'll update you once I get get my hands on the a VM with a V100.", "votes": 1 }, { "id": 644929, "postDate": "2019-10-09T14:33:55.303Z", "content": "@ratthachat From my testing in mixed precision - the V100 is 2.4-2.8x faster, which is well worth it considering that the cost/hr of a V100 2.35x the cost/hr for the T4. And just running the same tests on the P100, I get same speed as the T4. From what I can tell, the V100 utilizes the same CPU more than the T4 so my guess is that it can iterate over the same batchsize faster so the CPU is idling less.
", "rawMarkdown": "@ratthachat From my testing in mixed precision - the V100 is 2.4-2.8x faster, which is well worth it considering that the cost/hr of a V100 2.35x the cost/hr for the T4. And just running the same tests on the P100, I get same speed as the T4. From what I can tell, the V100 utilizes the same CPU more than the T4 so my guess is that it can iterate over the same batchsize faster so the CPU is idling less.", "votes": 1 }, { "id": 644953, "postDate": "2019-10-09T15:02:54.250Z", "content": "@teeyee314 Are you running preemptible instance or regular instance? At first I tried preemptible instance, but noticed that the performance while training the model isn't consistent. Specially the first epoch takes almost 2x longer time than other epochs. Now I am running a regular V100 instance and the performance is consistent.
", "rawMarkdown": "@teeyee314 Are you running preemptible instance or regular instance? At first I tried preemptible instance, but noticed that the performance while training the model isn't consistent. Specially the first epoch takes almost 2x longer time than other epochs. Now I am running a regular V100 instance and the performance is consistent.", "votes": 1 }, { "id": 644967, "postDate": "2019-10-09T15:26:06.370Z", "content": "Strange. I haven't used regular instances, only colab, kaggle kernel and gcp preemptibles. Very consistent pytorch and apex training times. Are you loading from pngs/jpgs or from dicoms?
", "rawMarkdown": "Strange. I haven't used regular instances, only colab, kaggle kernel and gcp preemptibles. Very consistent pytorch and apex training times. Are you loading from pngs/jpgs or from dicoms?", "votes": 1 }, { "id": 644985, "postDate": "2019-10-09T15:43:31.697Z", "content": "I am loading png images (the datasets pre-processed by you). I am also using pytorch (fastai). I trained several efficientnet models. Previously I trained on colab and kaggle kernel, where training times were consistent.
\n\nOn GCP preemptibles, training times were consistent at first. But later I noticed that the first epoch always took more than 2x times. Then I switched to a regular instance.
\n\nCan you please share how long does it take to run an epoch on V100? I am currently training an Efficientnet B5 model with 224 png images and it takes around 1h per epoch (batch size 64 & fp16).
", "rawMarkdown": "I am loading png images (the datasets pre-processed by you). I am also using pytorch (fastai). I trained several efficientnet models. Previously I trained on colab and kaggle kernel, where training times were consistent.\n\nOn GCP preemptibles, training times were consistent at first. But later I noticed that the first epoch always took more than 2x times. Then I switched to a regular instance. \n\nCan you please share how long does it take to run an epoch on V100? I am currently training an Efficientnet B5 model with 224 png images and it takes around 1h per epoch (batch size 64 & fp16).", "votes": 1 } ] } ], "comments": [ { "id": 643620, "author_name": "Alexander Abstreiter", "author_url": "", "post_date": "2019-10-07T17:36:03.877000", "content": "Got a V100 on us-west1-a.
I created a new instance with V100 yesterday. I tried several zones, but V100 wasn't available until I tried us-west1-a.
Try asia-east1-c
", "votes": 0, "replies": [] }, { "id": 644050, "author_name": "kambarakun", "author_url": "", "post_date": "2019-10-08T08:48:34.273000", "content": "Yesterday I got a 1 * V100 instance on us-west1-b.
Hi guys, I’ve never used V100 before, just want to know : what is a training speed boost you get with V100 over P100 for basic baseline models?
", "votes": 0, "replies": [ { "id": 643614, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-07T17:26:20.803000", "content": "I wish I could say based on experience. However, based on the fp32(single precision) TFLOPs, The V100 has 14TFLOPS, while P100 has 9.3TFLOPS. In most cases you'll be comparing fp32, but the V100 does have tensor cores that can run larger batch sizes in mixed precision which would likely speed up training and inference. I'll update you once I get get my hands on the a VM with a V100.
", "votes": 1, "replies": [] }, { "id": 644929, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-09T14:33:55.303000", "content": "@ratthachat From my testing in mixed precision - the V100 is 2.4-2.8x faster, which is well worth it considering that the cost/hr of a V100 2.35x the cost/hr for the T4. And just running the same tests on the P100, I get same speed as the T4. From what I can tell, the V100 utilizes the same CPU more than the T4 so my guess is that it can iterate over the same batchsize faster so the CPU is idling less.
", "votes": 1, "replies": [] }, { "id": 644953, "author_name": "Atikur Rahman", "author_url": "", "post_date": "2019-10-09T15:02:54.250000", "content": "@teeyee314 Are you running preemptible instance or regular instance? At first I tried preemptible instance, but noticed that the performance while training the model isn't consistent. Specially the first epoch takes almost 2x longer time than other epochs. Now I am running a regular V100 instance and the performance is consistent.
", "votes": 1, "replies": [] }, { "id": 644967, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-09T15:26:06.370000", "content": "Strange. I haven't used regular instances, only colab, kaggle kernel and gcp preemptibles. Very consistent pytorch and apex training times. Are you loading from pngs/jpgs or from dicoms?
", "votes": 1, "replies": [] }, { "id": 644985, "author_name": "Atikur Rahman", "author_url": "", "post_date": "2019-10-09T15:43:31.697000", "content": "I am loading png images (the datasets pre-processed by you). I am also using pytorch (fastai). I trained several efficientnet models. Previously I trained on colab and kaggle kernel, where training times were consistent.
\n\nOn GCP preemptibles, training times were consistent at first. But later I noticed that the first epoch always took more than 2x times. Then I switched to a regular instance.
\n\nCan you please share how long does it take to run an epoch on V100? I am currently training an Efficientnet B5 model with 224 png images and it takes around 1h per epoch (batch size 64 & fp16).
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "643620": "Got a V100 on `us-west1-a`.", "643034": "I created a new instance with V100 yesterday. I tried several zones, but V100 wasn't available until I tried `us-west1-a`.\n\n- n1-highmem-8\n- us-west1-a\n- preempted", "643019": "Is anyone else noticing that they are unable to attach the V100 GPU to their instances? I can attach other GPU's but the V100's across all the servers seem to be in use - both preempted and non-preempted instances. \n\nIf you are able to attach a V100 GPU to your instance, please share the region, location and whether you're using preempted or not. Thanks.\n