As title suggests, and same topic was on official Discord channel, I can't submit because of out of memory error.
\nI tried some method like these
However, all of them threw the same error.
\nDoes anyone know how to deal with it?
As title suggests, and same topic was on official Discord channel, I can't submit because of out of memory error.
\nI tried some method like these
However, all of them threw the same error.
\nDoes anyone know how to deal with it?
Choosing the most suitable batch size is crucial for both computational efficiency and model performance. It's always a balancing act, as a larger batch might speed up the training process but may also consume more memory. On the other hand, a smaller batch may provide a more refined gradient estimate but might slow down the overall training.
\nFurthermore, rather than pre-loading all the images into the CPU's memory, which can be memory-intensive and lead to potential slowdowns, it's more practical and efficient to fetch and read the images dynamically as they're needed during the training process. This approach not only conserves memory but also facilitates the handling of large datasets that might not fit into the system's memory all at once. By adopting a dynamic reading strategy, one can seamlessly work with extensive datasets without the need for excessive computational resources.
", "rawMarkdown": "Choosing the most suitable batch size is crucial for both computational efficiency and model performance. It's always a balancing act, as a larger batch might speed up the training process but may also consume more memory. On the other hand, a smaller batch may provide a more refined gradient estimate but might slow down the overall training. \n\nFurthermore, rather than pre-loading all the images into the CPU's memory, which can be memory-intensive and lead to potential slowdowns, it's more practical and efficient to fetch and read the images dynamically as they're needed during the training process. This approach not only conserves memory but also facilitates the handling of large datasets that might not fit into the system's memory all at once. By adopting a dynamic reading strategy, one can seamlessly work with extensive datasets without the need for excessive computational resources." }, { "id": 2490941, "postDate": "2023-10-21T08:02:19.807Z", "content": "I had the same issue, but I then just decided to use batch_size=1, num_workers=1; although, it's quite slow. How did you solve it?
", "rawMarkdown": "I had the same issue, but I then just decided to use batch_size=1, num_workers=1; although, it's quite slow. How did you solve it?" } ], "comments": [ { "id": 2492874, "author_name": "dhinesh", "author_url": "", "post_date": "2023-10-23T01:53:14.043000", "content": "Choosing the most suitable batch size is crucial for both computational efficiency and model performance. It's always a balancing act, as a larger batch might speed up the training process but may also consume more memory. On the other hand, a smaller batch may provide a more refined gradient estimate but might slow down the overall training.
\nFurthermore, rather than pre-loading all the images into the CPU's memory, which can be memory-intensive and lead to potential slowdowns, it's more practical and efficient to fetch and read the images dynamically as they're needed during the training process. This approach not only conserves memory but also facilitates the handling of large datasets that might not fit into the system's memory all at once. By adopting a dynamic reading strategy, one can seamlessly work with extensive datasets without the need for excessive computational resources.
", "votes": 0, "replies": [] }, { "id": 2490941, "author_name": "samu2505", "author_url": "", "post_date": "2023-10-21T08:02:19.807000", "content": "I had the same issue, but I then just decided to use batch_size=1, num_workers=1; although, it's quite slow. How did you solve it?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2478503": "As title suggests, and same topic was on official Discord channel, I can't submit because of out of memory error.\nI tried some method like these\n- load test datas online using dataloader\n- image resize and save to local before inferencing\n- use CPU, not GPU\n- slice test images into smaller portion using pytorch-toolbelt\n\nHowever, all of them threw the same error.\nDoes anyone know how to deal with it?", "2492874": "Choosing the most suitable batch size is crucial for both computational efficiency and model performance. It's always a balancing act, as a larger batch might speed up the training process but may also consume more memory. On the other hand, a smaller batch may provide a more refined gradient estimate but might slow down the overall training. \n\nFurthermore, rather than pre-loading all the images into the CPU's memory, which can be memory-intensive and lead to potential slowdowns, it's more practical and efficient to fetch and read the images dynamically as they're needed during the training process. This approach not only conserves memory but also facilitates the handling of large datasets that might not fit into the system's memory all at once. By adopting a dynamic reading strategy, one can seamlessly work with extensive datasets without the need for excessive computational resources.", "2490941": "I had the same issue, but I then just decided to use batch_size=1, num_workers=1; although, it's quite slow. How did you solve it?" } }