I have been struggling with Memory issues and slow speed when using EfficientNet. After reading and roaming on dark web I found this GitHub disccusion.
https://github.com/lukemelas/EfficientNet-PyTorch/issues/18
\n\nTLDR: if you want to lower your memory and fit more images in the batch do following:
\n\n1) Go to site package (or clone git hub repo)\n2) Go to utils and modify following line (https://github.com/lukemelas/EfficientNet-PyTorch/blob/de40cbfec8244a6ddbb367fd491d700ecc2eef85/efficientnet_pytorch/utils.py#L39)
\n\nfrom:\n\ndef relu_fn(x):\n \"\"\" Swish activation function \"\"\"\n return x * torch.sigmoid(x)\n
to:\n```\nsigmoid = torch.nn.Sigmoid()\nclass Swish(torch.autograd.Function):\n @staticmethod\n def forward(ctx, i):\n result = i * sigmoid(i)\n ctx.save_for_backward(i)\n return result
\n\n@staticmethod\ndef backward(ctx, grad_output):\n i = ctx.saved_variables[0]\n sigmoid_i = sigmoid(i)\n return grad_output * (sigmoid_i * (1 + i * (1 - sigmoid_i)))\nswish = Swish.apply
\n\nclass Swish_module(nn.Module):\n def forward(self, x):\n return swish(x)
\n\nswish_layer = Swish_module()\ndef relu_fn(x):\n \"\"\" Swish activation function \"\"\"\n # return x * torch.sigmoid(x)\n return swish_layer(x)\n```
\n\nEnjoy your 20-30% Memory reduction
", "messages": [ { "id": 641275, "postDate": "2019-10-04T14:53:28.593Z", "content": "I have been struggling with Memory issues and slow speed when using EfficientNet. After reading and roaming on dark web I found this GitHub disccusion.
https://github.com/lukemelas/EfficientNet-PyTorch/issues/18
\n\nTLDR: if you want to lower your memory and fit more images in the batch do following:
\n\n1) Go to site package (or clone git hub repo)\n2) Go to utils and modify following line (https://github.com/lukemelas/EfficientNet-PyTorch/blob/de40cbfec8244a6ddbb367fd491d700ecc2eef85/efficientnet_pytorch/utils.py#L39)
\n\nfrom:\n\ndef relu_fn(x):\n \"\"\" Swish activation function \"\"\"\n return x * torch.sigmoid(x)\n
to:\n```\nsigmoid = torch.nn.Sigmoid()\nclass Swish(torch.autograd.Function):\n @staticmethod\n def forward(ctx, i):\n result = i * sigmoid(i)\n ctx.save_for_backward(i)\n return result
\n\n@staticmethod\ndef backward(ctx, grad_output):\n i = ctx.saved_variables[0]\n sigmoid_i = sigmoid(i)\n return grad_output * (sigmoid_i * (1 + i * (1 - sigmoid_i)))\nswish = Swish.apply
\n\nclass Swish_module(nn.Module):\n def forward(self, x):\n return swish(x)
\n\nswish_layer = Swish_module()\ndef relu_fn(x):\n \"\"\" Swish activation function \"\"\"\n # return x * torch.sigmoid(x)\n return swish_layer(x)\n```
\n\nEnjoy your 20-30% Memory reduction
", "rawMarkdown": "I have been struggling with Memory issues and slow speed when using `EfficientNet`. After reading and roaming on dark web I found this GitHub disccusion.\n\nhttps://github.com/lukemelas/EfficientNet-PyTorch/issues/18\n\nTLDR: if you want to lower your memory and fit more images in the batch do following:\n\n1) Go to site package (or clone git hub repo)\n2) Go to utils and modify following line (https://github.com/lukemelas/EfficientNet-PyTorch/blob/de40cbfec8244a6ddbb367fd491d700ecc2eef85/efficientnet_pytorch/utils.py#L39)\n\nfrom:\n```\ndef relu_fn(x):\n \"\"\" Swish activation function \"\"\"\n return x * torch.sigmoid(x)\n```\n\nto:\n```\nsigmoid = torch.nn.Sigmoid()\nclass Swish(torch.autograd.Function):\n @staticmethod\n def forward(ctx, i):\n result = i * sigmoid(i)\n ctx.save_for_backward(i)\n return result\n\n @staticmethod\n def backward(ctx, grad_output):\n i = ctx.saved_variables[0]\n sigmoid_i = sigmoid(i)\n return grad_output * (sigmoid_i * (1 + i * (1 - sigmoid_i)))\n\nswish = Swish.apply\n\nclass Swish_module(nn.Module):\n def forward(self, x):\n return swish(x)\n\nswish_layer = Swish_module()\ndef relu_fn(x):\n \"\"\" Swish activation function \"\"\"\n # return x * torch.sigmoid(x)\n return swish_layer(x)\n```\n\nEnjoy your 20-30% Memory reduction ", "votes": 79 }, { "id": 659090, "postDate": "2019-10-27T04:13:06.253Z", "content": "Jeremy Howard has been on the case with another implementation of Swish: https://twitter.com/jeremyphoward/status/1188304165971218432?s=20
", "rawMarkdown": "Jeremy Howard has been on the case with another implementation of Swish: https://twitter.com/jeremyphoward/status/1188304165971218432?s=20 ", "votes": 1, "replies": [ { "id": 660391, "postDate": "2019-10-29T04:12:45.147Z", "content": "I think updated library has this correction my training tine reduced from 52 to 39 minutes and model.summary says memory efficient swish
", "rawMarkdown": "I think updated library has this correction my training tine reduced from 52 to 39 minutes and model.summary says memory efficient swish " } ] }, { "id": 641815, "postDate": "2019-10-05T06:30:50.910Z", "content": "Yes it did lower my efficientnet memory usage from 9100mb to 7600mb with exactly the same setting.\nThanks for sharing!
", "rawMarkdown": "Yes it did lower my efficientnet memory usage from 9100mb to 7600mb with exactly the same setting.\nThanks for sharing!", "votes": 1 }, { "id": 644320, "postDate": "2019-10-08T16:01:18.153Z", "content": "I just made experiment with original and corrected EfficientNet's. \nThat's my results on V100 GPU:\n- original EffNet b2 takes 15.75Gb and 17.2 min to train one epoch;\n- corrected EffNet b2 takes 13.5Gb and 18.3 min to train one epoch.
", "rawMarkdown": "I just made experiment with original and corrected EfficientNet's. \nThat's my results on V100 GPU:\n- original EffNet b2 takes 15.75Gb and 17.2 min to train one epoch;\n- corrected EffNet b2 takes 13.5Gb and 18.3 min to train one epoch.", "votes": 2, "replies": [ { "id": 644325, "postDate": "2019-10-08T16:03:28.077Z", "content": "Interesting. Same batch size? Have you tried it more than once? perhaps there is variance?
", "rawMarkdown": "Interesting. Same batch size? Have you tried it more than once? perhaps there is variance?" }, { "id": 644336, "postDate": "2019-10-08T16:30:36.730Z", "content": "all the same parameters except proposed correction. \nfor sure it isnt variance, because i averaged across some folds.
", "rawMarkdown": "all the same parameters except proposed correction. \nfor sure it isnt variance, because i averaged across some folds." }, { "id": 644353, "postDate": "2019-10-08T17:07:00.363Z", "content": "I have similar results. However, if you adjust BS to fill all the memory GPU has, you will gain a small speed-up.
", "rawMarkdown": "I have similar results. However, if you adjust BS to fill all the memory GPU has, you will gain a small speed-up." } ] }, { "id": 642275, "postDate": "2019-10-05T19:41:32.397Z", "content": "Can anyone explain why the original function was using so much more memory?
\n\nPS. Thanks DrHB, gonna try this :)
", "rawMarkdown": "Can anyone explain why the original function was using so much more memory? \n\nPS. Thanks DrHB, gonna try this :)", "votes": 2, "replies": [ { "id": 642388, "postDate": "2019-10-06T01:39:56.843Z", "content": "Thanks to the author!\nI believe that when using x * torch.sigmoid(x), the autograd will remember both input to torch.sigmoid and the two inputs to torch.mul as separate tensors. There are two graph nodes: sigmoid and multiplication. This can be optimized if we define Swish as a separate graph node with its own gradient computation.\nSo, the main reason is that autograd is not very smart :)
thanks for your sharing.\nWill this modification potentially affect the final training result?\nIf not, the 20-30% memory reduction is dope!
", "rawMarkdown": "thanks for your sharing.\nWill this modification potentially affect the final training result?\nIf not, the 20-30% memory reduction is dope!", "votes": 2 }, { "id": 644209, "postDate": "2019-10-08T13:44:27.510Z", "content": "I made a fork with a fix for convenience purposes: https://github.com/hokmund/EfficientNet-PyTorch\nYou are welcome to use it via pip install git+https://github.com/hokmund/EfficientNet-PyTorch
However, it is still slower and more memory-consuming than the original TF implementation.
", "rawMarkdown": "I made a fork with a fix for convenience purposes: https://github.com/hokmund/EfficientNet-PyTorch\nYou are welcome to use it via `pip install git+https://github.com/hokmund/EfficientNet-PyTorch`\n\nHowever, it is still slower and more memory-consuming than the original TF implementation." }, { "id": 643452, "postDate": "2019-10-07T14:18:36.080Z", "content": "This might be a silly question, but how would you do this on a Linux VM on GCP? Specific commands?
", "rawMarkdown": "This might be a silly question, but how would you do this on a Linux VM on GCP? Specific commands?", "replies": [ { "id": 644255, "postDate": "2019-10-08T14:37:04.230Z", "content": "First you need to find path of the site packages directory. If you run pip install efficientnet-pytorch command, then you will find exact location of site packages directory. From there, you will find efficientnet_pytorch package directory and under that directory, the utils.py file is located. On my GCP instance, the utils.py file is located here -
/opt/anaconda3/lib/python3.7/site-packages/efficientnet_pytorch/utils.py
Once you find utils.py file location, open the file and replace the code mentioned in the original post.
relu_fn definition [using terminal]If you are comfortable using command line editor, then open the utils.py file using the following command (otherwise skip to the next section) -
\nvi /path/to/site-packages/efficientnet_pytorch/utils.py\n
relu_fn definition [from Jupyter notebook]If you prefer to use Jupyter environment, you can run the following command from a cell -
\n\n\n%load /path/to/site-packages/efficientnet_pytorch/utils.py\n
The utils.py file content will be loaded to the notebook cell. Edit the code and then run the cell. Finally to overwrite the code to the utils.py file, add the %%writefile cell magic command at the top of the code cell and run the cell again -
\n%%writefile /path/to/site-packages/efficientnet_pytorch/utils.py\n// utils.py file content goes here\n
Nice work.
\n\n\n\n\nWill this modification potentially affect the final training result?
\n
It shouldn't. That looks like a perfectly valid implementation of the gradient for Swish, probably the same thing PyTorch does when applying the chain rule, so should be identical.
\n\nIt will likely perform a little slower as it needs more CUDA kernel launches but the increased batch sizes should mean it still performs better than the original.
", "rawMarkdown": "Nice work.\n\n> Will this modification potentially affect the final training result?\n\nIt shouldn't. That looks like a perfectly valid implementation of the gradient for Swish, probably the same thing PyTorch does when applying the chain rule, so should be identical.\n\nIt will likely perform a little slower as it needs more CUDA kernel launches but the increased batch sizes should mean it still performs better than the original." }, { "id": 643280, "postDate": "2019-10-07T11:18:30.323Z", "content": "@DrHb when we were using effnet in aptos,we dint face this issue any time...\nWhat is different now out here
", "rawMarkdown": "@DrHb when we were using effnet in aptos,we dint face this issue any time...\nWhat is different now out here" }, { "id": 642966, "postDate": "2019-10-06T21:28:19.907Z", "content": "Intresting idea! Great Work...!! Thanks @drhabib
", "rawMarkdown": "Intresting idea! Great Work...!! Thanks @drhabib " }, { "id": 641437, "postDate": "2019-10-04T17:29:00.133Z", "content": "Very Helpful Share,...\nThanks @drhabib
", "rawMarkdown": "Very Helpful Share,...\nThanks @drhabib ", "votes": 1 }, { "id": 641378, "postDate": "2019-10-04T16:23:24.483Z", "content": "Thank you very much for sharing! :-D
", "rawMarkdown": "Thank you very much for sharing! :-D", "votes": 1 }, { "id": 644648, "postDate": "2019-10-09T06:14:32.710Z", "content": "Thank you for sharing! Very useful !
", "rawMarkdown": "Thank you for sharing! Very useful !" } ], "comments": [ { "id": 659090, "author_name": "datasaurus", "author_url": "", "post_date": "2019-10-27T04:13:06.253000", "content": "Jeremy Howard has been on the case with another implementation of Swish: https://twitter.com/jeremyphoward/status/1188304165971218432?s=20
", "votes": 1, "replies": [ { "id": 660391, "author_name": "Jaideep", "author_url": "", "post_date": "2019-10-29T04:12:45.147000", "content": "I think updated library has this correction my training tine reduced from 52 to 39 minutes and model.summary says memory efficient swish
", "votes": 0, "replies": [] } ] }, { "id": 641815, "author_name": "Qishen Ha", "author_url": "", "post_date": "2019-10-05T06:30:50.910000", "content": "Yes it did lower my efficientnet memory usage from 9100mb to 7600mb with exactly the same setting.\nThanks for sharing!
", "votes": 1, "replies": [] }, { "id": 644320, "author_name": "Ivan V.", "author_url": "", "post_date": "2019-10-08T16:01:18.153000", "content": "I just made experiment with original and corrected EfficientNet's. \nThat's my results on V100 GPU:\n- original EffNet b2 takes 15.75Gb and 17.2 min to train one epoch;\n- corrected EffNet b2 takes 13.5Gb and 18.3 min to train one epoch.
", "votes": 2, "replies": [ { "id": 644325, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-08T16:03:28.077000", "content": "Interesting. Same batch size? Have you tried it more than once? perhaps there is variance?
", "votes": 0, "replies": [] }, { "id": 644336, "author_name": "Ivan V.", "author_url": "", "post_date": "2019-10-08T16:30:36.730000", "content": "all the same parameters except proposed correction. \nfor sure it isnt variance, because i averaged across some folds.
", "votes": 0, "replies": [] }, { "id": 644353, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-10-08T17:07:00.363000", "content": "I have similar results. However, if you adjust BS to fill all the memory GPU has, you will gain a small speed-up.
", "votes": 0, "replies": [] } ] }, { "id": 642275, "author_name": "Tom Aindow", "author_url": "", "post_date": "2019-10-05T19:41:32.397000", "content": "Can anyone explain why the original function was using so much more memory?
\n\nPS. Thanks DrHB, gonna try this :)
", "votes": 2, "replies": [ { "id": 642388, "author_name": "Ruslan Baynazarov", "author_url": "", "post_date": "2019-10-06T01:39:56.843000", "content": "Thanks to the author!\nI believe that when using x * torch.sigmoid(x), the autograd will remember both input to torch.sigmoid and the two inputs to torch.mul as separate tensors. There are two graph nodes: sigmoid and multiplication. This can be optimized if we define Swish as a separate graph node with its own gradient computation.\nSo, the main reason is that autograd is not very smart :)
thanks for your sharing.\nWill this modification potentially affect the final training result?\nIf not, the 20-30% memory reduction is dope!
", "votes": 2, "replies": [] }, { "id": 644209, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2019-10-08T13:44:27.510000", "content": "I made a fork with a fix for convenience purposes: https://github.com/hokmund/EfficientNet-PyTorch\nYou are welcome to use it via pip install git+https://github.com/hokmund/EfficientNet-PyTorch
However, it is still slower and more memory-consuming than the original TF implementation.
", "votes": 0, "replies": [] }, { "id": 643452, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-10-07T14:18:36.080000", "content": "This might be a silly question, but how would you do this on a Linux VM on GCP? Specific commands?
", "votes": 0, "replies": [ { "id": 644255, "author_name": "Atikur Rahman", "author_url": "", "post_date": "2019-10-08T14:37:04.230000", "content": "First you need to find path of the site packages directory. If you run pip install efficientnet-pytorch command, then you will find exact location of site packages directory. From there, you will find efficientnet_pytorch package directory and under that directory, the utils.py file is located. On my GCP instance, the utils.py file is located here -
/opt/anaconda3/lib/python3.7/site-packages/efficientnet_pytorch/utils.py
Once you find utils.py file location, open the file and replace the code mentioned in the original post.
relu_fn definition [using terminal]If you are comfortable using command line editor, then open the utils.py file using the following command (otherwise skip to the next section) -
\nvi /path/to/site-packages/efficientnet_pytorch/utils.py\n
relu_fn definition [from Jupyter notebook]If you prefer to use Jupyter environment, you can run the following command from a cell -
\n\n\n%load /path/to/site-packages/efficientnet_pytorch/utils.py\n
The utils.py file content will be loaded to the notebook cell. Edit the code and then run the cell. Finally to overwrite the code to the utils.py file, add the %%writefile cell magic command at the top of the code cell and run the cell again -
\n%%writefile /path/to/site-packages/efficientnet_pytorch/utils.py\n// utils.py file content goes here\n
Nice work.
\n\n\n\n\nWill this modification potentially affect the final training result?
\n
It shouldn't. That looks like a perfectly valid implementation of the gradient for Swish, probably the same thing PyTorch does when applying the chain rule, so should be identical.
\n\nIt will likely perform a little slower as it needs more CUDA kernel launches but the increased batch sizes should mean it still performs better than the original.
", "votes": 0, "replies": [] }, { "id": 643280, "author_name": "Jaideep", "author_url": "", "post_date": "2019-10-07T11:18:30.323000", "content": "@DrHb when we were using effnet in aptos,we dint face this issue any time...\nWhat is different now out here
", "votes": 0, "replies": [] }, { "id": 642966, "author_name": "Evgeny Shtepin", "author_url": "", "post_date": "2019-10-06T21:28:19.907000", "content": "Intresting idea! Great Work...!! Thanks @drhabib
", "votes": 0, "replies": [] }, { "id": 641437, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-10-04T17:29:00.133000", "content": "Very Helpful Share,...\nThanks @drhabib
", "votes": 1, "replies": [] }, { "id": 641378, "author_name": "Michael Pieler", "author_url": "", "post_date": "2019-10-04T16:23:24.483000", "content": "Thank you very much for sharing! :-D
", "votes": 1, "replies": [] }, { "id": 644648, "author_name": "seefun", "author_url": "", "post_date": "2019-10-09T06:14:32.710000", "content": "Thank you for sharing! Very useful !
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "641275": "I have been struggling with Memory issues and slow speed when using `EfficientNet`. After reading and roaming on dark web I found this GitHub disccusion.\n\nhttps://github.com/lukemelas/EfficientNet-PyTorch/issues/18\n\nTLDR: if you want to lower your memory and fit more images in the batch do following:\n\n1) Go to site package (or clone git hub repo)\n2) Go to utils and modify following line (https://github.com/lukemelas/EfficientNet-PyTorch/blob/de40cbfec8244a6ddbb367fd491d700ecc2eef85/efficientnet_pytorch/utils.py#L39)\n\nfrom:\n```\ndef relu_fn(x):\n \"\"\" Swish activation function \"\"\"\n return x * torch.sigmoid(x)\n```\n\nto:\n```\nsigmoid = torch.nn.Sigmoid()\nclass Swish(torch.autograd.Function):\n @staticmethod\n def forward(ctx, i):\n result = i * sigmoid(i)\n ctx.save_for_backward(i)\n return result\n\n @staticmethod\n def backward(ctx, grad_output):\n i = ctx.saved_variables[0]\n sigmoid_i = sigmoid(i)\n return grad_output * (sigmoid_i * (1 + i * (1 - sigmoid_i)))\n\nswish = Swish.apply\n\nclass Swish_module(nn.Module):\n def forward(self, x):\n return swish(x)\n\nswish_layer = Swish_module()\ndef relu_fn(x):\n \"\"\" Swish activation function \"\"\"\n # return x * torch.sigmoid(x)\n return swish_layer(x)\n```\n\nEnjoy your 20-30% Memory reduction ", "659090": "Jeremy Howard has been on the case with another implementation of Swish: https://twitter.com/jeremyphoward/status/1188304165971218432?s=20 ", "641815": "Yes it did lower my efficientnet memory usage from 9100mb to 7600mb with exactly the same setting.\nThanks for sharing!", "644320": "I just made experiment with original and corrected EfficientNet's. \nThat's my results on V100 GPU:\n- original EffNet b2 takes 15.75Gb and 17.2 min to train one epoch;\n- corrected EffNet b2 takes 13.5Gb and 18.3 min to train one epoch.", "642275": "Can anyone explain why the original function was using so much more memory? \n\nPS. Thanks DrHB, gonna try this :)", "641293": "thanks for your sharing.\nWill this modification potentially affect the final training result?\nIf not, the 20-30% memory reduction is dope!", "644209": "I made a fork with a fix for convenience purposes: https://github.com/hokmund/EfficientNet-PyTorch\nYou are welcome to use it via `pip install git+https://github.com/hokmund/EfficientNet-PyTorch`\n\nHowever, it is still slower and more memory-consuming than the original TF implementation.", "643452": "This might be a silly question, but how would you do this on a Linux VM on GCP? Specific commands?", "643307": "Nice work.\n\n> Will this modification potentially affect the final training result?\n\nIt shouldn't. That looks like a perfectly valid implementation of the gradient for Swish, probably the same thing PyTorch does when applying the chain rule, so should be identical.\n\nIt will likely perform a little slower as it needs more CUDA kernel launches but the increased batch sizes should mean it still performs better than the original.", "643280": "@DrHb when we were using effnet in aptos,we dint face this issue any time...\nWhat is different now out here", "642966": "Intresting idea! Great Work...!! Thanks @drhabib ", "641437": "Very Helpful Share,...\nThanks @drhabib ", "641378": "Thank you very much for sharing! :-D", "644648": "Thank you for sharing! Very useful !" } }