When using Pillow package to read and process (mostly resizing) images, 2 big files (1 in train folder and 1 in other folder) can trigger the message that tells you the notebook is trying to allocate more memory than what's available (i.e. 16GB) and the notebooks restarts.
This notebook BloodClots🔬: EDA🔎 & load WSI + prune background provided a solution to avoid the out of memory issue using pyvips package. (scroll to the bottom of this notebook where you can see image processing scripts)
Compared to PIL (pillow package), pyvips package has its own limitations such as not providing sampling options in its resize and thumbnail functions.
Since Kaggle notebook environment does not come with pyvips, you have to install it yourself. Installing the pyvips package can be a bit troublesome to make it work in the submission notebook when the internet has to be set offline.
Fortunately, this amazing notebook tells you how to install the package hassle free: How to use pyvips offline. In fact, you can use this notebook directly by following these steps:
\n+ Add data button on the upper right corner of your submission notebookNotebook Output Filespyvips in the search box and click searchAdd!conda install /kaggle/input/how-to-use-pyvips-offline/*.tar.bz2 to a cell and run itimport pyvips to import the packageWhen using Pillow package to read and process (mostly resizing) images, 2 big files (1 in train folder and 1 in other folder) can trigger the message that tells you the notebook is trying to allocate more memory than what's available (i.e. 16GB) and the notebooks restarts.
This notebook BloodClots🔬: EDA🔎 & load WSI + prune background provided a solution to avoid the out of memory issue using pyvips package. (scroll to the bottom of this notebook where you can see image processing scripts)
Compared to PIL (pillow package), pyvips package has its own limitations such as not providing sampling options in its resize and thumbnail functions.
Since Kaggle notebook environment does not come with pyvips, you have to install it yourself. Installing the pyvips package can be a bit troublesome to make it work in the submission notebook when the internet has to be set offline.
Fortunately, this amazing notebook tells you how to install the package hassle free: How to use pyvips offline. In fact, you can use this notebook directly by following these steps:
\n+ Add data button on the upper right corner of your submission notebookNotebook Output Filespyvips in the search box and click searchAdd!conda install /kaggle/input/how-to-use-pyvips-offline/*.tar.bz2 to a cell and run itimport pyvips to import the packageThanks for the information, this is a highly useful post for almost all image based assignments here and at work as well!
", "rawMarkdown": "Thanks for the information, this is a highly useful post for almost all image based assignments here and at work as well!" } ], "comments": [ { "id": 1873535, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-07-27T16:49:31.043000", "content": "Thanks for the information, this is a highly useful post for almost all image based assignments here and at work as well!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1872929": "When using `Pillow` package to read and process (mostly resizing) images, 2 big files (1 in train folder and 1 in other folder) can trigger the message that tells you the notebook is trying to allocate more memory than what's available (i.e. 16GB) and the notebooks restarts. \n\nThis notebook [BloodClots🔬: EDA🔎 & load WSI + prune background](https://www.kaggle.com/code/jirkaborovec/bloodclots-eda-load-wsi-prune-background?scriptVersionId=101797769) provided a solution to avoid the **out of memory** issue using `pyvips` package. (scroll to the bottom of this notebook where you can see image processing scripts)\n\nCompared to `PIL` (pillow package), `pyvips` package has its own limitations such as not providing sampling options in its `resize` and `thumbnail` functions. \n\n\nSince Kaggle notebook environment does not come with `pyvips`, you have to install it yourself. Installing the `pyvips` package can be a bit troublesome to make it work in the submission notebook when the internet has to be set *offline*.\n\nFortunately, this amazing notebook tells you how to install the package hassle free: [How to use pyvips offline](https://www.kaggle.com/code/analokamus/how-to-use-pyvips-offline). In fact, you can use this notebook directly by following these steps:\n - click `+ Add data` button on the upper right corner of your submission notebook\n - click on `Notebook Output Files`\n - enter `pyvips` in the search box and click search\n - you will see that **How to use pyvips offline** notebook on the top of the list, click `Add`\n - copy `!conda install /kaggle/input/how-to-use-pyvips-offline/*.tar.bz2` to a cell and run it\n - now you can run `import pyvips` to import the package\n", "1873535": "Thanks for the information, this is a highly useful post for almost all image based assignments here and at work as well!" } }