Hi,
\nData loading for this dataset has been slow. I think the primary cause is, for each example - target pair we have to read 10+ npy files. This in my opinion is the bottle-neck. To reduce this high number of IO operations, I have created a fork of the dataset, that reduces the IO by a factor of 10.
forked dataset
\npart1 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-1
\npart2 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-2
\npart3 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-3
\npart4 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-4
\npart5 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-5
Features of this dataset:
\nDisadvantages:
\nHow to use - https://www.kaggle.com/code/soumyadeepkhandual/superfast-dataloading
\n** The dataset is split over 5 parts because of the 20GB limit, each part contains 2000 example-target pairs saved in npz file. All parts have been added.
\nIf you find any mistakes, please let me know. If the dataset was helpful, upvote, so that people can find this out.
", "messages": [ { "id": 2259314, "postDate": "2023-05-14T21:06:32.553Z", "content": "Hi,
\nData loading for this dataset has been slow. I think the primary cause is, for each example - target pair we have to read 10+ npy files. This in my opinion is the bottle-neck. To reduce this high number of IO operations, I have created a fork of the dataset, that reduces the IO by a factor of 10.
forked dataset
\npart1 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-1
\npart2 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-2
\npart3 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-3
\npart4 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-4
\npart5 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-5
Features of this dataset:
\nDisadvantages:
\nHow to use - https://www.kaggle.com/code/soumyadeepkhandual/superfast-dataloading
\n** The dataset is split over 5 parts because of the 20GB limit, each part contains 2000 example-target pairs saved in npz file. All parts have been added.
\nIf you find any mistakes, please let me know. If the dataset was helpful, upvote, so that people can find this out.
", "rawMarkdown": "Hi,\nData loading for this dataset has been slow. I think the primary cause is, for each example - target pair we have to read 10+ npy files. This in my opinion is the bottle-neck. To reduce this high number of IO operations, I have created a fork of the dataset, that reduces the IO by a factor of 10.\n\nforked dataset \npart1 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-1\npart2 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-2\npart3 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-3\npart4 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-4\npart5 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-5\n\nFeatures of this dataset:\n- about 55% of the original dataset did not contain contrails which have been removed here.\n- each example target pair is saved in a single npz file, instead of 10 npy files, making it load much faster with a lot less cpu usage, since number of IO operation are 10 times less.\n- Examples stored in float16. This saves a lot of space. The loss of information from using float16 instead of float32 (mean absolute reconstruction error) was less than 0.019%.\n- All Examples have been normalized as per channel mean and std.\n\nDisadvantages:\n- Individual human mask have been omitted.\n- The examples without contrails have been omitted\n- Information lost (less than 0.019%) due to the use of float16 \n\nHow to use - https://www.kaggle.com/code/soumyadeepkhandual/superfast-dataloading\n\n** The dataset is split over 5 parts because of the 20GB limit, each part contains 2000 example-target pairs saved in npz file. All parts have been added.\n\nIf you find any mistakes, please let me know. If the dataset was helpful, upvote, so that people can find this out.", "votes": 3 }, { "id": 2260135, "postDate": "2023-05-15T13:35:36.873Z", "content": "Concatenating the arrays and changing the dtype is in my opinion very usefull. But i don't think that it's a good idea to remove images without contrails. These images still contain information (that this is no contrail) and the test dataset will have the same label distribution
", "rawMarkdown": "Concatenating the arrays and changing the dtype is in my opinion very usefull. But i don't think that it's a good idea to remove images without contrails. These images still contain information (that this is no contrail) and the test dataset will have the same label distribution" }, { "id": 2259605, "postDate": "2023-05-15T05:47:47.917Z", "content": "The shape of the tensor is [1, 8, 9, 256, 256]. How can you use this dataset to train a model?
", "rawMarkdown": "The shape of the tensor is [1, 8, 9, 256, 256]. How can you use this dataset to train a model? ", "replies": [ { "id": 2259615, "postDate": "2023-05-15T06:01:40.277Z", "content": "Output of the dataloader is of shape (Batch_size, TimeFrame, Bands, H, W). So i my case i used batch size 1 for demonstration purpose, you can change as per your choice. There are 8 time frames for each band. There are total 9 bands and the image shape is 256 x 256. This makes the shape (1, 8, 9, 256, 256).
", "rawMarkdown": "Output of the dataloader is of shape (Batch_size, TimeFrame, Bands, H, W). So i my case i used batch size 1 for demonstration purpose, you can change as per your choice. There are 8 time frames for each band. There are total 9 bands and the image shape is 256 x 256. This makes the shape (1, 8, 9, 256, 256)." }, { "id": 2260131, "postDate": "2023-05-15T13:32:51.863Z", "content": "CNN's work with 4D Datasets (3D for image channels and one dimension for batches). the 5th dimension is the time dimension, so you probably have to train a RNN-CNN combination
", "rawMarkdown": "CNN's work with 4D Datasets (3D for image channels and one dimension for batches). the 5th dimension is the time dimension, so you probably have to train a RNN-CNN combination" } ] }, { "id": 2259498, "postDate": "2023-05-15T03:30:16.067Z", "content": "Waiting for you updating and thanks for you sharing.
", "rawMarkdown": "Waiting for you updating and thanks for you sharing.", "replies": [ { "id": 2259654, "postDate": "2023-05-15T06:42:36.763Z", "content": "Hey, All parts have been added. How to use : check this notebook
", "rawMarkdown": "Hey, All parts have been added. How to use : check this [notebook](https://www.kaggle.com/code/soumyadeepkhandual/superfast-dataloading) " } ] } ], "comments": [ { "id": 2260135, "author_name": "Patchef", "author_url": "", "post_date": "2023-05-15T13:35:36.873000", "content": "Concatenating the arrays and changing the dtype is in my opinion very usefull. But i don't think that it's a good idea to remove images without contrails. These images still contain information (that this is no contrail) and the test dataset will have the same label distribution
", "votes": 0, "replies": [] }, { "id": 2259605, "author_name": "Harsha Arya", "author_url": "", "post_date": "2023-05-15T05:47:47.917000", "content": "The shape of the tensor is [1, 8, 9, 256, 256]. How can you use this dataset to train a model?
", "votes": 0, "replies": [ { "id": 2259615, "author_name": "Soumyadeep Khandual", "author_url": "", "post_date": "2023-05-15T06:01:40.277000", "content": "Output of the dataloader is of shape (Batch_size, TimeFrame, Bands, H, W). So i my case i used batch size 1 for demonstration purpose, you can change as per your choice. There are 8 time frames for each band. There are total 9 bands and the image shape is 256 x 256. This makes the shape (1, 8, 9, 256, 256).
", "votes": 0, "replies": [] }, { "id": 2260131, "author_name": "Patchef", "author_url": "", "post_date": "2023-05-15T13:32:51.863000", "content": "CNN's work with 4D Datasets (3D for image channels and one dimension for batches). the 5th dimension is the time dimension, so you probably have to train a RNN-CNN combination
", "votes": 0, "replies": [] } ] }, { "id": 2259498, "author_name": "Dewei Chen", "author_url": "", "post_date": "2023-05-15T03:30:16.067000", "content": "Waiting for you updating and thanks for you sharing.
", "votes": 0, "replies": [ { "id": 2259654, "author_name": "Soumyadeep Khandual", "author_url": "", "post_date": "2023-05-15T06:42:36.763000", "content": "Hey, All parts have been added. How to use : check this notebook
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2259314": "Hi,\nData loading for this dataset has been slow. I think the primary cause is, for each example - target pair we have to read 10+ npy files. This in my opinion is the bottle-neck. To reduce this high number of IO operations, I have created a fork of the dataset, that reduces the IO by a factor of 10.\n\nforked dataset \npart1 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-1\npart2 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-2\npart3 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-3\npart4 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-4\npart5 - https://www.kaggle.com/datasets/soumyadeepkhandual/google-contrails-normalized-float16-part-5\n\nFeatures of this dataset:\n- about 55% of the original dataset did not contain contrails which have been removed here.\n- each example target pair is saved in a single npz file, instead of 10 npy files, making it load much faster with a lot less cpu usage, since number of IO operation are 10 times less.\n- Examples stored in float16. This saves a lot of space. The loss of information from using float16 instead of float32 (mean absolute reconstruction error) was less than 0.019%.\n- All Examples have been normalized as per channel mean and std.\n\nDisadvantages:\n- Individual human mask have been omitted.\n- The examples without contrails have been omitted\n- Information lost (less than 0.019%) due to the use of float16 \n\nHow to use - https://www.kaggle.com/code/soumyadeepkhandual/superfast-dataloading\n\n** The dataset is split over 5 parts because of the 20GB limit, each part contains 2000 example-target pairs saved in npz file. All parts have been added.\n\nIf you find any mistakes, please let me know. If the dataset was helpful, upvote, so that people can find this out.", "2260135": "Concatenating the arrays and changing the dtype is in my opinion very usefull. But i don't think that it's a good idea to remove images without contrails. These images still contain information (that this is no contrail) and the test dataset will have the same label distribution", "2259605": "The shape of the tensor is [1, 8, 9, 256, 256]. How can you use this dataset to train a model? ", "2259498": "Waiting for you updating and thanks for you sharing." } }