To quote @taindow from his kernel: \"Using mixed precision along with efficientnet-b0 and a little bit of pre-processing, a single pass of the entire 670k image dataset should take approx. 45m (at 224x224 resolution).\"
I'm using Keras (with tensorflow backend) and it take's about 2-2.5 hours per epoch, using efficientnetb0 and 224x224 input (full train set). It's also ran on a Kaggle kernel. I would like to know if anyone experience the same? And do you have any idea of how to solve this for keras/tensorflow? Or is just that keras/tensorflow hasn't optimized for EfficientNets yet? I actually also find most of the keras_applications models relatively slow (InceptionResNetV2 takes ~6 hours per epoch etc.).
\nBriefly what I've tried:
\n* 'channels_first' and 'channels_last' (So far I only made it work for EfficietNets)\n* Different implementations (all in keras/tensorflow)\n* Adding this to my script (from IBM's website):\nconfig = tf.ConfigProto()\nconfig.graph_options.rewrite_options.layout_optimizer=rewriter_config_pb2.RewriterConfig.OFF\nsess = tf.Session(config=config)\nkeras.backend.tensorflow_backend.set_session(sess) # set new TensorFlow session as the default\n* Locally (RTX 2070) or Kaggle kernel
Doing 'channels_first' or adding that code to the script improved the training time by ~25% for EfficientNets, which is nice, but It's still relatively slow. It would be nice if you guys could share your training run-times! Thank you in advance.
", "messages": [ { "id": 636801, "postDate": "2019-09-30T08:05:14.570Z", "content": "To quote @taindow from his kernel: \"Using mixed precision along with efficientnet-b0 and a little bit of pre-processing, a single pass of the entire 670k image dataset should take approx. 45m (at 224x224 resolution).\"
I'm using Keras (with tensorflow backend) and it take's about 2-2.5 hours per epoch, using efficientnetb0 and 224x224 input (full train set). It's also ran on a Kaggle kernel. I would like to know if anyone experience the same? And do you have any idea of how to solve this for keras/tensorflow? Or is just that keras/tensorflow hasn't optimized for EfficientNets yet? I actually also find most of the keras_applications models relatively slow (InceptionResNetV2 takes ~6 hours per epoch etc.).
\nBriefly what I've tried:
\n* 'channels_first' and 'channels_last' (So far I only made it work for EfficietNets)\n* Different implementations (all in keras/tensorflow)\n* Adding this to my script (from IBM's website):\nconfig = tf.ConfigProto()\nconfig.graph_options.rewrite_options.layout_optimizer=rewriter_config_pb2.RewriterConfig.OFF\nsess = tf.Session(config=config)\nkeras.backend.tensorflow_backend.set_session(sess) # set new TensorFlow session as the default\n* Locally (RTX 2070) or Kaggle kernel
Doing 'channels_first' or adding that code to the script improved the training time by ~25% for EfficientNets, which is nice, but It's still relatively slow. It would be nice if you guys could share your training run-times! Thank you in advance.
", "rawMarkdown": "To quote @taindow from his kernel: \"Using mixed precision along with efficientnet-b0 and a little bit of pre-processing, a single pass of the entire 670k image dataset should take approx. 45m (at 224x224 resolution).\"pytorch can give up to 50% training speed boost over keras\nand you can also you apex, which further 2x your pytorch training speed
", "rawMarkdown": "pytorch can give up to 50% training speed boost over keras\nand you can also you apex, which further 2x your pytorch training speed", "votes": 3, "replies": [ { "id": 636884, "postDate": "2019-09-30T10:02:25.847Z", "content": "That is great, Thank you for the info! What a huge difference.. I will have to learn pytorch :-)
", "rawMarkdown": "That is great, Thank you for the info! What a huge difference.. I will have to learn pytorch :-) " }, { "id": 636946, "postDate": "2019-09-30T12:11:13.213Z", "content": "@moewie94 sorry I never use apex before. Will using apex affect the training accuracy compared to normal pytorch only? thanks so much
", "rawMarkdown": "@moewie94 sorry I never use apex before. Will using apex affect the training accuracy compared to normal pytorch only? thanks so much" }, { "id": 637051, "postDate": "2019-09-30T15:32:55.193Z", "content": "I just implemented a Pytorch + apex version of my public kernel (not public/commited). I quickly tested ResNet50, ResNext50, InceptionV3 and EfficientNets. Training time of ResNet50 is pretty similar to before, but the other 3 are much faster now.
It's still not 45min/epoch for EfficientNetB0, which probably means that my DataGenerator/DataLoader is the bottleneck (I'm using the dcms and not resized png files)
", "rawMarkdown": "I just implemented a Pytorch + apex version of my public kernel (not public/commited). I quickly tested ResNet50, ResNext50, InceptionV3 and EfficientNets. Training time of ResNet50 is pretty similar to before, but the other 3 are much faster now.You can use pytorch profiler to find out exactly what is bottleneck...
\n\n\n\nor you can use this nice package\nhttps://github.com/rkern/line_profiler
", "rawMarkdown": "You can use pytorch profiler to find out exactly what is bottleneck... \n\nhttps://pytorch.org/docs/stable/autograd.html?highlight=autograd%20profiler#torch.autograd.profiler.profile\n\nor you can use this nice package\nhttps://github.com/rkern/line_profiler", "votes": 2 }, { "id": 637588, "postDate": "2019-10-01T06:26:53.057Z", "content": "@fiyeroleung yeah it will affect the performance a little bit but the tradeoff is worth it
", "rawMarkdown": "@fiyeroleung yeah it will affect the performance a little bit but the tradeoff is worth it" }, { "id": 638673, "postDate": "2019-10-02T08:59:05.620Z", "content": "thanks so much. Sadly I only own a 1080ti which wont be benefited from FP16 calculation.🤕
", "rawMarkdown": "thanks so much. Sadly I only own a 1080ti which wont be benefited from FP16 calculation.🤕 " } ] }, { "id": 636826, "postDate": "2019-09-30T08:51:00.050Z", "content": "size=224x224 batch_size=16 efficientnet b4 7hrs/epoch. Only uses ~35% gpu. No cpu/io bottleneck. I’ll try your suggestion.
", "rawMarkdown": "size=224x224 batch_size=16 efficientnet b4 7hrs/epoch. Only uses ~35% gpu. No cpu/io bottleneck. I’ll try your suggestion.", "votes": 1, "replies": [ { "id": 636835, "postDate": "2019-09-30T08:59:00.190Z", "content": "for that size you can use batch_size=64 and it will work fast(note : don't use full data)
", "rawMarkdown": "for that size you can use batch_size=64 and it will work fast(note : don't use full data)", "votes": 1 }, { "id": 636849, "postDate": "2019-09-30T09:08:57.687Z", "content": "With full dataset that’s the best batch size I can afford. I’ll try use 10k positive and 10k negative subset later. Thanks for your suggestion!
", "rawMarkdown": "With full dataset that’s the best batch size I can afford. I’ll try use 10k positive and 10k negative subset later. Thanks for your suggestion!", "votes": 1 }, { "id": 636990, "postDate": "2019-09-30T13:32:47.887Z", "content": "You dont have to go for b4 or complex model. The best strategy is use smaller model and do bunch of experiments (e.g tfms, training schedule and etc). After you are satisfied you can scale up.. My current LB standing is model trained on b0 with 224 image size...
This is really solid advice. Start small and experiment. More parameters and bigger images can come later..
", "rawMarkdown": "This is really solid advice. Start small and experiment. More parameters and bigger images can come later.. ", "votes": 2 }, { "id": 637032, "postDate": "2019-09-30T15:05:35.173Z", "content": "Thanks for the pro tips. 👍 It’s the first time I try EfficientNet so maybe I got a bit too excited that I forget the importance of fast iteration.
", "rawMarkdown": "Thanks for the pro tips. 👍 It’s the first time I try EfficientNet so maybe I got a bit too excited that I forget the importance of fast iteration.", "votes": 1 }, { "id": 637033, "postDate": "2019-09-30T15:10:29.517Z", "content": "Its all good =) I have to learn this stuff hard way as well =)
", "rawMarkdown": "Its all good =) I have to learn this stuff hard way as well =) ", "votes": 1 } ] }, { "id": 636807, "postDate": "2019-09-30T08:16:15.753Z", "content": "Absolutely @akensert ,no doubt about it,i have wasted so many gpu hours during working on this kernel from my quota dealing with kernel errors,it doesn't finish for even 3 epochs for 256x256 using only 340000 images within due time(9 hours in kaggle),i think keras team need to boost their model performance,keras is having lot of c/c++ down there,i don't know why it is still not as fast as expected :(
\n\nHow can i know in advance that my model will finish training within 9 hours? can you tell me? i don't want to waste gpu hours from my kaggle quota anymore,thanks in advance,specially thanks a lot for this post
", "rawMarkdown": "Absolutely @akensert ,no doubt about it,i have wasted so many gpu hours during working on[ this kernel](https://www.kaggle.com/mobassir/keras-efficientnetb4-for-intracranial-hemorrhage) from my quota dealing with kernel errors,it doesn't finish for even 3 epochs for 256x256 using only 340000 images within due time(9 hours in kaggle),i think keras team need to boost their model performance,keras is having lot of c/c++ down there,i don't know why it is still not as fast as expected :(\n\nHow can i know in advance that my model will finish training within 9 hours? can you tell me? i don't want to waste gpu hours from my kaggle quota anymore,thanks in advance,specially thanks a lot for this post", "votes": 1, "replies": [ { "id": 636815, "postDate": "2019-09-30T08:30:12.210Z", "content": "Yeah, I also expect it to perform relatively good. I've been sitting with Keras for quite some time now (and I really like it), but right now I'm considering going to pytorch. Or perhaps try a different backend.
To answer your question: I would run the model training in interactive mode just for a brief moment (with verbose=1), to approximate the total run time. And be a bit pessimistic about the run-time :-)
@akensert thank you,one more question if you don't mind?\nhow slow or fast gtx 1080 8gb and gtx 1080 16gb in our local machine compared to kaggle kernels gpu?\ni want to make pc for deep learning that will work as good as kaggle kernels or just slightly better than that.\nso, is gtx 1080 8gb with 16gb ddr4 and i7 is enough?\nor even gtx 1080 16gb with 16gb ddr4 and i7 is not better than kaggle kernels?\ni want to make pc for deep learning within range 1000-1200 dollar,if you could recommend me any laptop or desktop within this range then it will be highly appreciated,,,kaggle gpu quota hours limit sucks :(
", "rawMarkdown": "@akensert thank you,one more question if you don't mind?\nhow slow or fast gtx 1080 8gb and gtx 1080 16gb in our local machine compared to kaggle kernels gpu?\ni want to make pc for deep learning that will work as good as kaggle kernels or just slightly better than that.\nso, is gtx 1080 8gb with 16gb ddr4 and i7 is enough?\nor even gtx 1080 16gb with 16gb ddr4 and i7 is not better than kaggle kernels?\ni want to make pc for deep learning within range 1000-1200 dollar,if you could recommend me any laptop or desktop within this range then it will be highly appreciated,,,kaggle gpu quota hours limit sucks :(" }, { "id": 636858, "postDate": "2019-09-30T09:23:09.003Z", "content": "@mobassir I would do a google search on this for sure! If you ask for my opinion, I would think that if you went for a gtx 1080 you would get similar or slightly faster model training than the Kaggle kernels'. For big datasets, which can't be read directly to memory, a fast CPU with multiple cores (perhaps hexacore) could be good. The GPU memory, 8 GB (which I have on my RTX 2070) I sometimes find a bit limited (but it's OK).
", "rawMarkdown": "@mobassir I would do a google search on this for sure! If you ask for my opinion, I would think that if you went for a gtx 1080 you would get similar or slightly faster model training than the Kaggle kernels'. For big datasets, which can't be read directly to memory, a fast CPU with multiple cores (perhaps hexacore) could be good. The GPU memory, 8 GB (which I have on my RTX 2070) I sometimes find a bit limited (but it's OK).", "votes": 1 } ] }, { "id": 637876, "postDate": "2019-10-01T10:35:12.100Z", "content": "I tried to use tf.train.experimental.enable_mixed_precision_graph_rewrite with slightly larger batch size. Now EfficientNetB0 takes <2hr per epoch instead of ~3.5hr. It should have been even faster because I read DCM files and resize the image on the fly and there's CPU bottleneck with 2 GPUs running simultaneously.
The interesting topic! I'm struggling now with slowness. So I subscribe here. :)
", "rawMarkdown": "The interesting topic! I'm struggling now with slowness. So I subscribe here. :)" } ], "comments": [ { "id": 636875, "author_name": "DatNT", "author_url": "", "post_date": "2019-09-30T09:46:28.853000", "content": "pytorch can give up to 50% training speed boost over keras\nand you can also you apex, which further 2x your pytorch training speed
", "votes": 3, "replies": [ { "id": 636884, "author_name": "Alex", "author_url": "", "post_date": "2019-09-30T10:02:25.847000", "content": "That is great, Thank you for the info! What a huge difference.. I will have to learn pytorch :-)
", "votes": 0, "replies": [] }, { "id": 636946, "author_name": "Salaryman", "author_url": "", "post_date": "2019-09-30T12:11:13.213000", "content": "@moewie94 sorry I never use apex before. Will using apex affect the training accuracy compared to normal pytorch only? thanks so much
", "votes": 0, "replies": [] }, { "id": 637051, "author_name": "Alex", "author_url": "", "post_date": "2019-09-30T15:32:55.193000", "content": "I just implemented a Pytorch + apex version of my public kernel (not public/commited). I quickly tested ResNet50, ResNext50, InceptionV3 and EfficientNets. Training time of ResNet50 is pretty similar to before, but the other 3 are much faster now.
It's still not 45min/epoch for EfficientNetB0, which probably means that my DataGenerator/DataLoader is the bottleneck (I'm using the dcms and not resized png files)
", "votes": 1, "replies": [] }, { "id": 637053, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-30T15:43:10.343000", "content": "You can use pytorch profiler to find out exactly what is bottleneck...
\n\n\n\nor you can use this nice package\nhttps://github.com/rkern/line_profiler
", "votes": 2, "replies": [] }, { "id": 637588, "author_name": "DatNT", "author_url": "", "post_date": "2019-10-01T06:26:53.057000", "content": "@fiyeroleung yeah it will affect the performance a little bit but the tradeoff is worth it
", "votes": 0, "replies": [] }, { "id": 638673, "author_name": "Salaryman", "author_url": "", "post_date": "2019-10-02T08:59:05.620000", "content": "thanks so much. Sadly I only own a 1080ti which wont be benefited from FP16 calculation.🤕
", "votes": 0, "replies": [] } ] }, { "id": 636826, "author_name": "Yifeng (Ethan) Zou", "author_url": "", "post_date": "2019-09-30T08:51:00.050000", "content": "size=224x224 batch_size=16 efficientnet b4 7hrs/epoch. Only uses ~35% gpu. No cpu/io bottleneck. I’ll try your suggestion.
", "votes": 1, "replies": [ { "id": 636835, "author_name": "Mobassir", "author_url": "", "post_date": "2019-09-30T08:59:00.190000", "content": "for that size you can use batch_size=64 and it will work fast(note : don't use full data)
", "votes": 1, "replies": [] }, { "id": 636849, "author_name": "Yifeng (Ethan) Zou", "author_url": "", "post_date": "2019-09-30T09:08:57.687000", "content": "With full dataset that’s the best batch size I can afford. I’ll try use 10k positive and 10k negative subset later. Thanks for your suggestion!
", "votes": 1, "replies": [] }, { "id": 636990, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-30T13:32:47.887000", "content": "You dont have to go for b4 or complex model. The best strategy is use smaller model and do bunch of experiments (e.g tfms, training schedule and etc). After you are satisfied you can scale up.. My current LB standing is model trained on b0 with 224 image size...
This is really solid advice. Start small and experiment. More parameters and bigger images can come later..
", "votes": 2, "replies": [] }, { "id": 637032, "author_name": "Yifeng (Ethan) Zou", "author_url": "", "post_date": "2019-09-30T15:05:35.173000", "content": "Thanks for the pro tips. 👍 It’s the first time I try EfficientNet so maybe I got a bit too excited that I forget the importance of fast iteration.
", "votes": 1, "replies": [] }, { "id": 637033, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-30T15:10:29.517000", "content": "Its all good =) I have to learn this stuff hard way as well =)
", "votes": 1, "replies": [] } ] }, { "id": 636807, "author_name": "Mobassir", "author_url": "", "post_date": "2019-09-30T08:16:15.753000", "content": "Absolutely @akensert ,no doubt about it,i have wasted so many gpu hours during working on this kernel from my quota dealing with kernel errors,it doesn't finish for even 3 epochs for 256x256 using only 340000 images within due time(9 hours in kaggle),i think keras team need to boost their model performance,keras is having lot of c/c++ down there,i don't know why it is still not as fast as expected :(
\n\nHow can i know in advance that my model will finish training within 9 hours? can you tell me? i don't want to waste gpu hours from my kaggle quota anymore,thanks in advance,specially thanks a lot for this post
", "votes": 1, "replies": [ { "id": 636815, "author_name": "Alex", "author_url": "", "post_date": "2019-09-30T08:30:12.210000", "content": "Yeah, I also expect it to perform relatively good. I've been sitting with Keras for quite some time now (and I really like it), but right now I'm considering going to pytorch. Or perhaps try a different backend.
To answer your question: I would run the model training in interactive mode just for a brief moment (with verbose=1), to approximate the total run time. And be a bit pessimistic about the run-time :-)
@akensert thank you,one more question if you don't mind?\nhow slow or fast gtx 1080 8gb and gtx 1080 16gb in our local machine compared to kaggle kernels gpu?\ni want to make pc for deep learning that will work as good as kaggle kernels or just slightly better than that.\nso, is gtx 1080 8gb with 16gb ddr4 and i7 is enough?\nor even gtx 1080 16gb with 16gb ddr4 and i7 is not better than kaggle kernels?\ni want to make pc for deep learning within range 1000-1200 dollar,if you could recommend me any laptop or desktop within this range then it will be highly appreciated,,,kaggle gpu quota hours limit sucks :(
", "votes": 0, "replies": [] }, { "id": 636858, "author_name": "Alex", "author_url": "", "post_date": "2019-09-30T09:23:09.003000", "content": "@mobassir I would do a google search on this for sure! If you ask for my opinion, I would think that if you went for a gtx 1080 you would get similar or slightly faster model training than the Kaggle kernels'. For big datasets, which can't be read directly to memory, a fast CPU with multiple cores (perhaps hexacore) could be good. The GPU memory, 8 GB (which I have on my RTX 2070) I sometimes find a bit limited (but it's OK).
", "votes": 1, "replies": [] } ] }, { "id": 637876, "author_name": "Yifeng (Ethan) Zou", "author_url": "", "post_date": "2019-10-01T10:35:12.100000", "content": "I tried to use tf.train.experimental.enable_mixed_precision_graph_rewrite with slightly larger batch size. Now EfficientNetB0 takes <2hr per epoch instead of ~3.5hr. It should have been even faster because I read DCM files and resize the image on the fly and there's CPU bottleneck with 2 GPUs running simultaneously.
The interesting topic! I'm struggling now with slowness. So I subscribe here. :)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "636801": "To quote @taindow from his kernel: \"Using mixed precision along with efficientnet-b0 and a little bit of pre-processing, a single pass of the entire 670k image dataset should take approx. 45m (at 224x224 resolution).\"