The new memory boost for GPU notebooks may be of interest for anyone interested in working with the larger images at full resolution.
", "messages": [ { "id": 2487918, "postDate": "2023-10-18T21:20:57.113Z", "content": "The new memory boost for GPU notebooks may be of interest for anyone interested in working with the larger images at full resolution.
", "rawMarkdown": "[The new memory boost for GPU notebooks](https://www.kaggle.com/discussions/product-feedback/448251) may be of interest for anyone interested in working with the larger images at full resolution.", "votes": 33 }, { "id": 2518347, "postDate": "2023-11-09T08:14:21.567Z", "content": "Thanks, the memory boost is very helpful for processing these large images.
\nI'm still running into resource problems though. I need tiled pyramidal TIFF for my code work. I've changed my feature extraction pipeline to convert one PNG to TIFF at a time, extract features and then delete the TIFF file. Unfortunately with LSW compression on the TIFF this is too slow to run in 12 hours on the complete test set. So I tried multi-processing, but it still takes over 12 hours. So I went to Packbits compression, which makes it fast enough, and the individual files small enough for this to work for each file individually. But now I run up against this 96 GB disk utilization limit. Seems like even if I delete my temporary files, the still count against that limit.
\nIs there a way to increase this temporary storage limit for this competition. I've spent countless hours now trying to work around the resource limits. Of course I could crop the images, but that will degrade model quality.
", "rawMarkdown": "Thanks, the memory boost is very helpful for processing these large images.\n\nI'm still running into resource problems though. I need tiled pyramidal TIFF for my code work. I've changed my feature extraction pipeline to convert one PNG to TIFF at a time, extract features and then delete the TIFF file. Unfortunately with LSW compression on the TIFF this is too slow to run in 12 hours on the complete test set. So I tried multi-processing, but it still takes over 12 hours. So I went to Packbits compression, which makes it fast enough, and the individual files small enough for this to work for each file individually. But now I run up against this 96 GB disk utilization limit. Seems like even if I delete my temporary files, the still count against that limit.\n\nIs there a way to increase this temporary storage limit for this competition. I've spent countless hours now trying to work around the resource limits. Of course I could crop the images, but that will degrade model quality.", "votes": 1, "replies": [ { "id": 2526238, "postDate": "2023-11-15T16:48:48.493Z", "content": "@sohier: Any news on this?
", "rawMarkdown": "@sohier: Any news on this? " } ] }, { "id": 2521739, "postDate": "2023-11-12T03:00:34.680Z", "content": "Thanks for the info!
", "rawMarkdown": "Thanks for the info!" }, { "id": 2496251, "postDate": "2023-10-23T21:22:05.660Z", "content": "Thanks for the info!
", "rawMarkdown": "Thanks for the info!" } ], "comments": [ { "id": 2518347, "author_name": "DanielT", "author_url": "", "post_date": "2023-11-09T08:14:21.567000", "content": "Thanks, the memory boost is very helpful for processing these large images.
\nI'm still running into resource problems though. I need tiled pyramidal TIFF for my code work. I've changed my feature extraction pipeline to convert one PNG to TIFF at a time, extract features and then delete the TIFF file. Unfortunately with LSW compression on the TIFF this is too slow to run in 12 hours on the complete test set. So I tried multi-processing, but it still takes over 12 hours. So I went to Packbits compression, which makes it fast enough, and the individual files small enough for this to work for each file individually. But now I run up against this 96 GB disk utilization limit. Seems like even if I delete my temporary files, the still count against that limit.
\nIs there a way to increase this temporary storage limit for this competition. I've spent countless hours now trying to work around the resource limits. Of course I could crop the images, but that will degrade model quality.
", "votes": 1, "replies": [ { "id": 2526238, "author_name": "DanielT", "author_url": "", "post_date": "2023-11-15T16:48:48.493000", "content": "@sohier: Any news on this?
", "votes": 0, "replies": [] } ] }, { "id": 2521739, "author_name": "Yanjie Qiu", "author_url": "", "post_date": "2023-11-12T03:00:34.680000", "content": "Thanks for the info!
", "votes": 0, "replies": [] }, { "id": 2496251, "author_name": "dmk2050", "author_url": "", "post_date": "2023-10-23T21:22:05.660000", "content": "Thanks for the info!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2487918": "[The new memory boost for GPU notebooks](https://www.kaggle.com/discussions/product-feedback/448251) may be of interest for anyone interested in working with the larger images at full resolution.", "2518347": "Thanks, the memory boost is very helpful for processing these large images.\n\nI'm still running into resource problems though. I need tiled pyramidal TIFF for my code work. I've changed my feature extraction pipeline to convert one PNG to TIFF at a time, extract features and then delete the TIFF file. Unfortunately with LSW compression on the TIFF this is too slow to run in 12 hours on the complete test set. So I tried multi-processing, but it still takes over 12 hours. So I went to Packbits compression, which makes it fast enough, and the individual files small enough for this to work for each file individually. But now I run up against this 96 GB disk utilization limit. Seems like even if I delete my temporary files, the still count against that limit.\n\nIs there a way to increase this temporary storage limit for this competition. I've spent countless hours now trying to work around the resource limits. Of course I could crop the images, but that will degrade model quality.", "2521739": "Thanks for the info!", "2496251": "Thanks for the info!" } }