Thanks @theoviel , @tivfrvqhs5 , @christofhenkel , @outwrest for your effort and introducion to DALI. DALI is one of the best things I have learned in this competition. Based on the work they published before, I wrapped all the data reading, image decoding, preprocessing process(dali and customized), data generation, into a custom data iterator in a fully aligned Pytorch style.
\nTalk is cheap. Show you the code. It's modified based on the public notebook of @outwrest .
\nSome brief introduction: Dali pipeline should always contain 2 or 3 steps.
\nfn.external_source or a custom iterator.In fact, in the third step we can implement any operations that need to be done outside of DALI pipe, such as numpy, opencv, yolo, and aligning to pytrorch style, by inheriting DALIGenericIterator and plugging the operations into the custom data iterator. It's CustomDALIGenericIterator in the code. After that, j2k_iter and jll_iter in this code have the exact same APIs as Pytorch Dataloader, which is easier to read and use.
j2k_iter = (length=(J2Ki), pipelines=, yolo_model=yolo_model)\n in (j2k_iter):\n = (x)\n ...\n\njll_iter = (length=(JLLi), pipelines=, yolo_model=yolo_model)\n in (jll_iter):\n = (x)\n ...\nThanks @theoviel , @tivfrvqhs5 , @christofhenkel , @outwrest for your effort and introducion to DALI. DALI is one of the best things I have learned in this competition. Based on the work they published before, I wrapped all the data reading, image decoding, preprocessing process(dali and customized), data generation, into a custom data iterator in a fully aligned Pytorch style.
\nTalk is cheap. Show you the code. It's modified based on the public notebook of @outwrest .
\nSome brief introduction: Dali pipeline should always contain 2 or 3 steps.
\nfn.external_source or a custom iterator.In fact, in the third step we can implement any operations that need to be done outside of DALI pipe, such as numpy, opencv, yolo, and aligning to pytrorch style, by inheriting DALIGenericIterator and plugging the operations into the custom data iterator. It's CustomDALIGenericIterator in the code. After that, j2k_iter and jll_iter in this code have the exact same APIs as Pytorch Dataloader, which is easier to read and use.
j2k_iter = (length=(J2Ki), pipelines=, yolo_model=yolo_model)\n in (j2k_iter):\n = (x)\n ...\n\njll_iter = (length=(JLLi), pipelines=, yolo_model=yolo_model)\n in (jll_iter):\n = (x)\n ...\nHey, that's super cool! DALI is awesome, there is something to learn! This is my first time developing a pipeline and I was learning alongside everyone you mentioned (they're all awesome ❤️❤️). I stopped competing in this competition and I kinda regret it. I just posted one of my WIP notebooks where I was testing 2xGPU with DALI, one big addition left is adding JLL GPU decoding but it should be another full end2end with GPU apply_windowing operation. With your additions, it can be even better (and faster 👀)!
\nI hope to use DALI more often in future competitions. 😆
\nAnyways, congratulations on the competition and great notebook! 🎉🎉🎉🎉
", "rawMarkdown": "Hey, that's super cool! DALI is awesome, there is something to learn! This is my first time developing a pipeline and I was learning alongside everyone you mentioned (they're all awesome ❤️❤️). I stopped competing in this competition and I kinda regret it. I just posted [one of my WIP notebooks](https://www.kaggle.com/code/outwrest/2xt4-dali) where I was testing 2xGPU with DALI, one big addition left is adding JLL GPU decoding but it should be another full end2end with GPU apply_windowing operation. With your additions, it can be even better (and faster 👀)! \n\nI hope to use DALI more often in future competitions. 😆\n\nAnyways, congratulations on the competition and great notebook! 🎉🎉🎉🎉", "votes": 1, "replies": [ { "id": 2164693, "postDate": "2023-03-01T18:41:27.853Z", "content": "Also, my original public notebook looks so bad, I should've refactored it before making it public. Hope it wasn't bad to read 😂😂
", "rawMarkdown": "Also, my original public notebook looks so bad, I should've refactored it before making it public. Hope it wasn't bad to read 😂😂", "votes": 1, "replies": [ { "id": 2165051, "postDate": "2023-03-02T00:44:02.570Z", "content": "Thank you. Your original notebook is already very good. Look forward to see you in next competition.
", "rawMarkdown": "Thank you. Your original notebook is already very good. Look forward to see you in next competition.", "votes": 1 } ] } ] }, { "id": 2165045, "postDate": "2023-03-02T00:36:24.427Z", "content": "i use https://github.com/kornia/kornia for gpu agumention.
\nit use augmentation as a nn.Module
Thank you for posting! What are the benefits of using DALI? Is it faster or capable to handle larger batches? Thank you!
Hi @jiweiliu , the main reason that DALI is widely used in this competition is that it can decode jpeg2000 images with a much faster speed.
", "rawMarkdown": "Hi @jiweiliu , the main reason that DALI is widely used in this competition is that it can decode jpeg2000 images with a much faster speed.", "votes": 1 } ] }, { "id": 2164261, "postDate": "2023-03-01T12:55:55.803Z", "content": "Very useful and practical! Thanks!
", "rawMarkdown": "Very useful and practical! Thanks!" } ], "comments": [ { "id": 2164686, "author_name": "outwrest", "author_url": "", "post_date": "2023-03-01T18:39:09.837000", "content": "Hey, that's super cool! DALI is awesome, there is something to learn! This is my first time developing a pipeline and I was learning alongside everyone you mentioned (they're all awesome ❤️❤️). I stopped competing in this competition and I kinda regret it. I just posted one of my WIP notebooks where I was testing 2xGPU with DALI, one big addition left is adding JLL GPU decoding but it should be another full end2end with GPU apply_windowing operation. With your additions, it can be even better (and faster 👀)!
\nI hope to use DALI more often in future competitions. 😆
\nAnyways, congratulations on the competition and great notebook! 🎉🎉🎉🎉
", "votes": 1, "replies": [ { "id": 2164693, "author_name": "outwrest", "author_url": "", "post_date": "2023-03-01T18:41:27.853000", "content": "Also, my original public notebook looks so bad, I should've refactored it before making it public. Hope it wasn't bad to read 😂😂
", "votes": 1, "replies": [ { "id": 2165051, "author_name": "ForcewithMe", "author_url": "", "post_date": "2023-03-02T00:44:02.570000", "content": "Thank you. Your original notebook is already very good. Look forward to see you in next competition.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2165045, "author_name": "hengck23", "author_url": "", "post_date": "2023-03-02T00:36:24.427000", "content": "i use https://github.com/kornia/kornia for gpu agumention.
\nit use augmentation as a nn.Module
Thank you for posting! What are the benefits of using DALI? Is it faster or capable to handle larger batches? Thank you!
Hi @jiweiliu , the main reason that DALI is widely used in this competition is that it can decode jpeg2000 images with a much faster speed.
", "votes": 1, "replies": [] } ] }, { "id": 2164261, "author_name": "Fatemeh.V.Younesi", "author_url": "", "post_date": "2023-03-01T12:55:55.803000", "content": "Very useful and practical! Thanks!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2162527": "Thanks @theoviel , @tivfrvqhs5 , @christofhenkel , @outwrest for your effort and introducion to DALI. DALI is one of the best things I have learned in this competition. Based on the work they published before, I wrapped all the data reading, image decoding, preprocessing process(dali and customized), data generation, into a custom data iterator **in a fully aligned Pytorch style**. \n\nTalk is cheap. Show you [the code](https://www.kaggle.com/code/forcewithme/rsna-elegant-end2end-dali-infer-in-pytorch-style?scriptVersionId=120573402). It's modified based on the public notebook of @outwrest .\n\nSome brief introduction: Dali pipeline should always contain 2 or 3 steps. \n\n1.\tDesign an iterator to retrieve and read raw data in batches. You can use `fn.external_source` or a custom iterator.\n2.\tImplement a DALI pipeline, and perform all DALI operations in the pipeline, such as decoding, Pad, Resize, etc. of images.\n3.\tImplement a DALIGenericIterator, or customize your data iterator (optional).\n\nIn fact, in the third step **we can implement any operations** that need to be done outside of `DALI pipe`, such as `numpy`, `opencv`, `yolo`, and aligning to pytrorch style, by inheriting DALIGenericIterator and plugging the operations into the custom data iterator. It's `CustomDALIGenericIterator` in [the code](https://www.kaggle.com/code/forcewithme/rsna-elegant-end2end-dali-infer-in-pytorch-style?scriptVersionId=120573402). After that, `j2k_iter` and `jll_iter` in this code have **the exact same APIs as Pytorch Dataloader**, which is easier to read and use.\n\n```\n\nj2k_iter = CustomDALIGenericIterator(length=len(J2Ki), pipelines=[j2k_pipe], yolo_model=yolo_model)\nfor x in tqdm(j2k_iter):\n y = model(x)\n ...\n\njll_iter = CustomDALIGenericIterator(length=len(JLLi), pipelines=[jll_pipe], yolo_model=yolo_model)\nfor x in tqdm(jll_iter):\n y = model(x)\n ...\n\n```", "2164686": "Hey, that's super cool! DALI is awesome, there is something to learn! This is my first time developing a pipeline and I was learning alongside everyone you mentioned (they're all awesome ❤️❤️). I stopped competing in this competition and I kinda regret it. I just posted [one of my WIP notebooks](https://www.kaggle.com/code/outwrest/2xt4-dali) where I was testing 2xGPU with DALI, one big addition left is adding JLL GPU decoding but it should be another full end2end with GPU apply_windowing operation. With your additions, it can be even better (and faster 👀)! \n\nI hope to use DALI more often in future competitions. 😆\n\nAnyways, congratulations on the competition and great notebook! 🎉🎉🎉🎉", "2165045": "i use https://github.com/kornia/kornia for gpu agumention.\nit use augmentation as a nn.Module", "2162908": "Thank you for posting! What are the benefits of using `DALI`? Is it faster or capable to handle larger batches? Thank you!", "2164261": "Very useful and practical! Thanks!" } }