![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F554091%2F04a87006f36961b8291e419050248e4c%2FScreen%202020-10-15%20at%2010.55.44.jpg?generation=1602752237934324&alt=media\")
I'm using the this 256x256 JPEGs dataset with cv2.imread. I realised that the limiting factor on my training speed is the data loader (no augmentations) so I'm assuming it's IO speed. Any tips to speed this up?
Hi @alexandersoare, you can try a library called TurboJPEG to read JPEG images. It should be faster than cv2.imread() from OpenCV. You can read more on how to implement it here: https://www.learnopencv.com/efficient-image-loading/
Hi @kozodoi - how can I install turbojpeg in a kaggle notebook?
", "rawMarkdown": "Hi @kozodoi - how can I install turbojpeg in a kaggle notebook?" }, { "id": 1050524, "postDate": "2020-10-15T13:57:09.570Z", "content": "@cascadenite In a notebook with the Internet enabled you can use:
\n!pip install -U git+git://github.com/lilohuang/PyTurboJPEG.git
See here for more details.
", "rawMarkdown": "@cascadenite In a notebook with the Internet enabled you can use:\n`!pip install -U git+git://github.com/lilohuang/PyTurboJPEG.git`\n\n See [here](https://github.com/lilohuang/PyTurboJPEG) for more details.\n\n\n\n", "votes": 1 }, { "id": 1050834, "postDate": "2020-10-15T19:13:19.603Z", "content": "got it - thanks! That's reduced the training time by about 20% (only 1hr15/epoch for Effnet0 now - obviously still far too slow but an improvement so thank you)
", "rawMarkdown": "got it - thanks! That's reduced the training time by about 20% (only 1hr15/epoch for Effnet0 now - obviously still far too slow but an improvement so thank you)" }, { "id": 1055968, "postDate": "2020-10-21T10:05:19.523Z", "content": "@cascadenite
\ndid u get this error
Unable to locate turbojpeg library automatically. You may specify the turbojpeg library path manually.\ne.g. jpeg = TurboJPEG(lib_pat\nh)
\nhow to resolve it..
@jaideepvalani See the comment of @lndosu above
", "rawMarkdown": "@jaideepvalani See the comment of @lndosu above" }, { "id": 1056178, "postDate": "2020-10-21T13:41:36.920Z", "content": "@alexandersoare yeah i saw.. does this reduces the image quality TUrbo Jpeg ?
", "rawMarkdown": "@alexandersoare yeah i saw.. does this reduces the image quality TUrbo Jpeg ?\n" }, { "id": 1059321, "postDate": "2020-10-24T23:26:51.287Z", "content": "It's too late for me to use TurboJPEG in this competition, but I'll def use it in others.. :-).
", "rawMarkdown": "It's too late for me to use TurboJPEG in this competition, but I'll def use it in others.. :-)." }, { "id": 1059567, "postDate": "2020-10-25T08:24:10.030Z", "content": "@jaideepvalani would feel weird if it did reduce the image quality. Something that should be printed on the box
", "rawMarkdown": "@jaideepvalani would feel weird if it did reduce the image quality. Something that should be printed on the box" } ] }, { "id": 1048518, "postDate": "2020-10-13T15:09:20.553Z", "content": "I'm using the this 256x256 JPEGs dataset with cv2.imread. I realised that the limiting factor on my training speed is the data loader (no augmentations) so I'm assuming it's IO speed. Any tips to speed this up?
Hi Alexander, I’m having the same issue. I increased my number of workers and set pin_memory=True. That increased speed by about 40%.
", "rawMarkdown": "Hi Alexander, I’m having the same issue. I increased my number of workers and set pin_memory=True. That increased speed by about 40%.", "votes": 2 }, { "id": 1048735, "postDate": "2020-10-13T18:40:29.123Z", "content": "You can try to use TFRecords dataset provided by organizers https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/189545.
", "rawMarkdown": "You can try to use TFRecords dataset provided by organizers https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/189545.", "votes": 2 }, { "id": 1050981, "postDate": "2020-10-16T02:22:17.107Z", "content": "Hi@Nikita Kozodoi,Thanks for the great sharing.
\nI get the following error in kaggle notebookHow can I use TurboJPEG?
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4289810%2Fe90b9869ea1672b976a6939cbd73bf22%2F2020-10-16%2011.18.25.png?generation=1602814737399905&alt=media\")
!apt-get update
\n!apt-get install libturbojpeg
\n!pip install -U git+git://github.com/lilohuang/PyTurboJPEG.git
You have to install libjpeg-turbo.
", "rawMarkdown": "`!apt-get update`\n`!apt-get install libturbojpeg`\n`!pip install -U git+git://github.com/lilohuang/PyTurboJPEG.git`\n\nYou have to install libjpeg-turbo.", "votes": 1 }, { "id": 1051096, "postDate": "2020-10-16T06:54:47.070Z", "content": "@Yohan
\nThank you so much!
\nThanks to you, I was able to solve the error.
Hi @alexandersoare, you can try a library called TurboJPEG to read JPEG images. It should be faster than cv2.imread() from OpenCV. You can read more on how to implement it here: https://www.learnopencv.com/efficient-image-loading/
Hi @kozodoi - how can I install turbojpeg in a kaggle notebook?
", "votes": 0, "replies": [] }, { "id": 1050524, "author_name": "Nikita Kozodoi", "author_url": "", "post_date": "2020-10-15T13:57:09.570000", "content": "@cascadenite In a notebook with the Internet enabled you can use:
\n!pip install -U git+git://github.com/lilohuang/PyTurboJPEG.git
See here for more details.
", "votes": 1, "replies": [] }, { "id": 1050834, "author_name": "Mark P", "author_url": "", "post_date": "2020-10-15T19:13:19.603000", "content": "got it - thanks! That's reduced the training time by about 20% (only 1hr15/epoch for Effnet0 now - obviously still far too slow but an improvement so thank you)
", "votes": 0, "replies": [] }, { "id": 1055968, "author_name": "Jaideep", "author_url": "", "post_date": "2020-10-21T10:05:19.523000", "content": "@cascadenite
\ndid u get this error
Unable to locate turbojpeg library automatically. You may specify the turbojpeg library path manually.\ne.g. jpeg = TurboJPEG(lib_pat\nh)
\nhow to resolve it..
@jaideepvalani See the comment of @lndosu above
", "votes": 0, "replies": [] }, { "id": 1056178, "author_name": "Jaideep", "author_url": "", "post_date": "2020-10-21T13:41:36.920000", "content": "@alexandersoare yeah i saw.. does this reduces the image quality TUrbo Jpeg ?
", "votes": 0, "replies": [] }, { "id": 1059321, "author_name": "عثمان", "author_url": "", "post_date": "2020-10-24T23:26:51.287000", "content": "It's too late for me to use TurboJPEG in this competition, but I'll def use it in others.. :-).
", "votes": 0, "replies": [] }, { "id": 1059567, "author_name": "Alexander Soare", "author_url": "", "post_date": "2020-10-25T08:24:10.030000", "content": "@jaideepvalani would feel weird if it did reduce the image quality. Something that should be printed on the box
", "votes": 0, "replies": [] } ] }, { "id": 1049855, "author_name": "Mark P", "author_url": "", "post_date": "2020-10-14T20:52:43.927000", "content": "Hi Alexander, I’m having the same issue. I increased my number of workers and set pin_memory=True. That increased speed by about 40%.
", "votes": 2, "replies": [] }, { "id": 1048735, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2020-10-13T18:40:29.123000", "content": "You can try to use TFRecords dataset provided by organizers https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/189545.
", "votes": 2, "replies": [] }, { "id": 1050981, "author_name": "SHINO", "author_url": "", "post_date": "2020-10-16T02:22:17.107000", "content": "Hi@Nikita Kozodoi,Thanks for the great sharing.
\nI get the following error in kaggle notebookHow can I use TurboJPEG?
!apt-get update
\n!apt-get install libturbojpeg
\n!pip install -U git+git://github.com/lilohuang/PyTurboJPEG.git
You have to install libjpeg-turbo.
", "votes": 1, "replies": [] }, { "id": 1051096, "author_name": "SHINO", "author_url": "", "post_date": "2020-10-16T06:54:47.070000", "content": "@Yohan
\nThank you so much!
\nThanks to you, I was able to solve the error.