{ "id": 428423, "title": "Data Processing", "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/428423", "author_name": "RAHUL CHOUDHARY", "post_date": "2023-08-01T10:49:28.222000", "votes": 2, "comment_count": 2, "views": 0, "content": "

The dataset is huge .I cannot fit this complete data in my computer and process it. How to go about it

", "messages": [ { "id": 2368729, "postDate": "2023-08-01T10:49:28.223Z", "content": "

The dataset is huge .I cannot fit this complete data in my computer and process it. How to go about it

", "rawMarkdown": "The dataset is huge .I cannot fit this complete data in my computer and process it. How to go about it", "votes": 1 }, { "id": 2372669, "postDate": "2023-08-03T20:42:21.247Z", "content": "

Hi Rahul, check this dataset from @shashwatraman that most of the people is currently using and it only takes 12GB instead of 450GB like the original.

\n

Here you will find all images already processed in ash-color (which makes contrails better perceived in images) and only taking 3 channels out of the 8 bands, the 5th frame which is the target out of 8 time frames and these images are converted to float16 taking half of the memory but keeping almost all the info. Every image also contains its label btw, and there are all train and validation images.

", "rawMarkdown": "Hi Rahul, check this [dataset](https://www.kaggle.com/datasets/shashwatraman/contrails-images-ash-color) from @shashwatraman that most of the people is currently using and it only takes **12GB** instead of **450GB** like the original. \n\nHere you will find all images already processed in ash-color (which makes contrails better perceived in images) and only taking 3 channels out of the 8 bands, the 5th frame which is the target out of 8 time frames and these images are converted to float16 taking half of the memory but keeping almost all the info. Every image also contains its label btw, and there are all train and validation images. " }, { "id": 2368797, "postDate": "2023-08-01T11:30:34.360Z", "content": "

You can process the data here on Kaggle from a Python(/R) notebook. There are many examples in code section.

", "rawMarkdown": "You can process the data here on Kaggle from a Python(/R) notebook. There are many examples in code section." } ], "comments": [ { "id": 2372669, "author_name": "Enric Domingo", "author_url": "", "post_date": "2023-08-03T20:42:21.247000", "content": "

Hi Rahul, check this dataset from @shashwatraman that most of the people is currently using and it only takes 12GB instead of 450GB like the original.

\n

Here you will find all images already processed in ash-color (which makes contrails better perceived in images) and only taking 3 channels out of the 8 bands, the 5th frame which is the target out of 8 time frames and these images are converted to float16 taking half of the memory but keeping almost all the info. Every image also contains its label btw, and there are all train and validation images.

", "votes": 0, "replies": [] }, { "id": 2368797, "author_name": "David Slavíček", "author_url": "", "post_date": "2023-08-01T11:30:34.360000", "content": "

You can process the data here on Kaggle from a Python(/R) notebook. There are many examples in code section.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2368729": "The dataset is huge .I cannot fit this complete data in my computer and process it. How to go about it", "2372669": "Hi Rahul, check this [dataset](https://www.kaggle.com/datasets/shashwatraman/contrails-images-ash-color) from @shashwatraman that most of the people is currently using and it only takes **12GB** instead of **450GB** like the original. \n\nHere you will find all images already processed in ash-color (which makes contrails better perceived in images) and only taking 3 channels out of the 8 bands, the 5th frame which is the target out of 8 time frames and these images are converted to float16 taking half of the memory but keeping almost all the info. Every image also contains its label btw, and there are all train and validation images. ", "2368797": "You can process the data here on Kaggle from a Python(/R) notebook. There are many examples in code section." } }