![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9052057%2F0615d039c6168c022ea0a41659a1cca9%2FScreenshot%202022-12-21%202.05.06%20PM.png?generation=1671660329546306&alt=media\")
https://www.kaggle.com/code/kaggleqrdl/fast-jpeg-lossless-decoding
\nedit:
\nMay have jumped the gun a bit =D
\nStill looking for good cython bindings
\nPerf improvements only seem to be 2-3x. I'm not sure cython will help much, but we'll see. Working on them now.
", "messages": [ { "id": 2072236, "postDate": "2022-12-21T22:05:54.137Z", "content": "
https://www.kaggle.com/code/kaggleqrdl/fast-jpeg-lossless-decoding
\nedit:
\nMay have jumped the gun a bit =D
\nStill looking for good cython bindings
\nPerf improvements only seem to be 2-3x. I'm not sure cython will help much, but we'll see. Working on them now.
", "rawMarkdown": "\n\nhttps://www.kaggle.com/code/kaggleqrdl/fast-jpeg-lossless-decoding\n\nedit:\n\n\nMay have jumped the gun a bit =D\n\nStill looking for good cython bindings \n\nPerf improvements only seem to be 2-3x. I'm not sure cython will help much, but we'll see. Working on them now.", "votes": 8 }, { "id": 2072242, "postDate": "2022-12-21T22:14:14.900Z", "content": "public the notebook please, thank you
", "rawMarkdown": "public the notebook please, thank you", "votes": 1, "replies": [ { "id": 2072247, "postDate": "2022-12-21T22:20:48.837Z", "content": "it is done.
", "rawMarkdown": "it is done." } ] }, { "id": 2074311, "postDate": "2022-12-24T01:25:08.430Z", "content": "I've made some more gcc optimizations here, about 25% faster. More to be done. Will look into proper cython bindings and find out where the bottleneck is on kaggle machines. It looks like the turbojpeg guy didn't do any as his reasonable take is that there isn't much demand for this rather outdated and rarely used standard.
", "rawMarkdown": "I've made some more gcc optimizations here, about 25% faster. More to be done. Will look into proper cython bindings and find out where the bottleneck is on kaggle machines. It looks like the turbojpeg guy didn't do any as his reasonable take is that there isn't much demand for this rather outdated and rarely used standard." }, { "id": 2072960, "postDate": "2022-12-22T14:48:35.520Z", "content": "Ok, apologies, I may have overhyped this. I saw very good results on my local server (which is about 20 years old) which led me to believe incorrectly that this was performing very well. My local and rather ancient CPUs do appear to be faster than kaggle, though not by the orders of magnitude which would explain the difference in performance. More investigation is required. I'm going to revert back to the author of the lossless jpeg as well and ask him about benchmarking.
\nThat said, it does appear to be somewhat faster, but certainly not 10x.
", "rawMarkdown": "Ok, apologies, I may have overhyped this. I saw very good results on my local server (which is about 20 years old) which led me to believe incorrectly that this was performing very well. My local and rather ancient CPUs do appear to be faster than kaggle, though not by the orders of magnitude which would explain the difference in performance. More investigation is required. I'm going to revert back to the author of the lossless jpeg as well and ask him about benchmarking.\n\nThat said, it does appear to be somewhat faster, but certainly not 10x." }, { "id": 2072344, "postDate": "2022-12-22T01:50:29.760Z", "content": "import cv2\nimport re \nre.findall('.*jpeg.*',cv2.getBuildInformation()) \nOut[36]: \n[' 3rdparty dependencies: libprotobuf ade ittnotify libjpeg-turbo libwebp libtiff libopenjp2 IlmImf quirc ippiw ippicv',\n ' JPEG: libjpeg-turbo (ver 2.1.2-62)']\nThese libraries don't seem to link against shared objects, which makes re-using them a pain.
\nI'll do a quick cython binding, it's pretty basic stuff. Hopefully shouldn't take more than 60 minutes or so
\nedit: actually i'll do a quick test in all c to make sure it's worthwhile.
", "rawMarkdown": "These libraries don't seem to link against shared objects, which makes re-using them a pain.\n\nI'll do a quick cython binding, it's pretty basic stuff. Hopefully shouldn't take more than 60 minutes or so\n\nedit: actually i'll do a quick test in all c to make sure it's worthwhile. ", "votes": 1 } ] }, { "id": 2072343, "postDate": "2022-12-22T01:38:29.670Z", "content": "time to learn some simple c ode for python binding … maybe someone can do a quick course on this (e.g. live code video conference)
", "rawMarkdown": "time to learn some simple c ode for python binding ... maybe someone can do a quick course on this (e.g. live code video conference)" }, { "id": 2072268, "postDate": "2022-12-21T23:11:37.937Z", "content": "I found https://github.com/lilohuang/PyTurboJPEG, let's see how it preforms.
", "rawMarkdown": "I found https://github.com/lilohuang/PyTurboJPEG, let's see how it preforms.", "replies": [ { "id": 2072270, "postDate": "2022-12-21T23:14:28.967Z", "content": "No, that won't work. Those APIs are not supported. My notebook is the only way until someone rolls their own cython.
\nTo be honest, it's not immediately necessary and will only provide marginal utility. os.system / imread is very fast.
", "rawMarkdown": "No, that won't work. Those APIs are not supported. My notebook is the only way until someone rolls their own cython. \n\nTo be honest, it's not immediately necessary and will only provide marginal utility. os.system / imread is very fast.", "votes": 2, "replies": [ { "id": 2072278, "postDate": "2022-12-21T23:25:50.700Z", "content": "Thanks for the response, I found the thread on the issue (Anyone wondering check out https://github.com/libjpeg-turbo/libjpeg-turbo/issues/402#issuecomment-1362011578).
\nAny speedups will be helpful with the number of images we have. I expect cutting out IO would save some more time. This is a cool find, thanks for posting.
", "rawMarkdown": "Thanks for the response, I found the thread on the issue (Anyone wondering check out https://github.com/libjpeg-turbo/libjpeg-turbo/issues/402#issuecomment-1362011578).\n\nAny speedups will be helpful with the number of images we have. I expect cutting out IO would save some more time. This is a cool find, thanks for posting.", "votes": 1 }, { "id": 2072279, "postDate": "2022-12-21T23:29:06.167Z", "content": "Yes, I believe this will be a massive win. The turbojpeg dude is GOAT
", "rawMarkdown": "Yes, I believe this will be a massive win. The turbojpeg dude is GOAT", "votes": 1 } ] } ] } ], "comments": [ { "id": 2072242, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-12-21T22:14:14.900000", "content": "public the notebook please, thank you
", "votes": 1, "replies": [ { "id": 2072247, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-21T22:20:48.837000", "content": "it is done.
", "votes": 0, "replies": [] } ] }, { "id": 2074311, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-24T01:25:08.430000", "content": "I've made some more gcc optimizations here, about 25% faster. More to be done. Will look into proper cython bindings and find out where the bottleneck is on kaggle machines. It looks like the turbojpeg guy didn't do any as his reasonable take is that there isn't much demand for this rather outdated and rarely used standard.
", "votes": 0, "replies": [] }, { "id": 2072960, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-22T14:48:35.520000", "content": "Ok, apologies, I may have overhyped this. I saw very good results on my local server (which is about 20 years old) which led me to believe incorrectly that this was performing very well. My local and rather ancient CPUs do appear to be faster than kaggle, though not by the orders of magnitude which would explain the difference in performance. More investigation is required. I'm going to revert back to the author of the lossless jpeg as well and ask him about benchmarking.
\nThat said, it does appear to be somewhat faster, but certainly not 10x.
", "votes": 0, "replies": [] }, { "id": 2072344, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-22T01:50:29.760000", "content": "import cv2\nimport re \nre.findall('.*jpeg.*',cv2.getBuildInformation()) \nOut[36]: \n[' 3rdparty dependencies: libprotobuf ade ittnotify libjpeg-turbo libwebp libtiff libopenjp2 IlmImf quirc ippiw ippicv',\n ' JPEG: libjpeg-turbo (ver 2.1.2-62)']\nThese libraries don't seem to link against shared objects, which makes re-using them a pain.
\nI'll do a quick cython binding, it's pretty basic stuff. Hopefully shouldn't take more than 60 minutes or so
\nedit: actually i'll do a quick test in all c to make sure it's worthwhile.
", "votes": 1, "replies": [] } ] }, { "id": 2072343, "author_name": "hengck23", "author_url": "", "post_date": "2022-12-22T01:38:29.670000", "content": "time to learn some simple c ode for python binding … maybe someone can do a quick course on this (e.g. live code video conference)
", "votes": 0, "replies": [] }, { "id": 2072268, "author_name": "outwrest", "author_url": "", "post_date": "2022-12-21T23:11:37.937000", "content": "I found https://github.com/lilohuang/PyTurboJPEG, let's see how it preforms.
", "votes": 0, "replies": [ { "id": 2072270, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-21T23:14:28.967000", "content": "No, that won't work. Those APIs are not supported. My notebook is the only way until someone rolls their own cython.
\nTo be honest, it's not immediately necessary and will only provide marginal utility. os.system / imread is very fast.
", "votes": 2, "replies": [ { "id": 2072278, "author_name": "outwrest", "author_url": "", "post_date": "2022-12-21T23:25:50.700000", "content": "Thanks for the response, I found the thread on the issue (Anyone wondering check out https://github.com/libjpeg-turbo/libjpeg-turbo/issues/402#issuecomment-1362011578).
\nAny speedups will be helpful with the number of images we have. I expect cutting out IO would save some more time. This is a cool find, thanks for posting.
", "votes": 1, "replies": [] }, { "id": 2072279, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2022-12-21T23:29:06.167000", "content": "Yes, I believe this will be a massive win. The turbojpeg dude is GOAT
", "votes": 1, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2072236": "\n\nhttps://www.kaggle.com/code/kaggleqrdl/fast-jpeg-lossless-decoding\n\nedit:\n\n\nMay have jumped the gun a bit =D\n\nStill looking for good cython bindings \n\nPerf improvements only seem to be 2-3x. I'm not sure cython will help much, but we'll see. Working on them now.", "2072242": "public the notebook please, thank you", "2074311": "I've made some more gcc optimizations here, about 25% faster. More to be done. Will look into proper cython bindings and find out where the bottleneck is on kaggle machines. It looks like the turbojpeg guy didn't do any as his reasonable take is that there isn't much demand for this rather outdated and rarely used standard.", "2072960": "Ok, apologies, I may have overhyped this. I saw very good results on my local server (which is about 20 years old) which led me to believe incorrectly that this was performing very well. My local and rather ancient CPUs do appear to be faster than kaggle, though not by the orders of magnitude which would explain the difference in performance. More investigation is required. I'm going to revert back to the author of the lossless jpeg as well and ask him about benchmarking.\n\nThat said, it does appear to be somewhat faster, but certainly not 10x.", "2072344": "```\nimport cv2\nimport re \nre.findall('.*jpeg.*',cv2.getBuildInformation()) \nOut[36]: \n[' 3rdparty dependencies: libprotobuf ade ittnotify libjpeg-turbo libwebp libtiff libopenjp2 IlmImf quirc ippiw ippicv',\n ' JPEG: libjpeg-turbo (ver 2.1.2-62)']\n\n```", "2072343": "time to learn some simple c ode for python binding ... maybe someone can do a quick course on this (e.g. live code video conference)", "2072268": "I found https://github.com/lilohuang/PyTurboJPEG, let's see how it preforms." } }