The train dataset is 180GB, how are we supposed to read it in, to then convert to parquet and load in as polar or dask, if we only have 20gb of ram? It fully saturates the ram and crashes before it can convert to parquet. I am using Kaggle notebook here, my personal comp is too weak for this, and I don't wan to pay for cloud computing.
", "messages": [ { "id": 2772324, "postDate": "2024-04-24T16:51:35.593Z", "content": "I processed the huge dataset and saved it in a more friendly format here. It was split in 17 parts with 625k rows each. So you can easily load a subset and play with it.
", "rawMarkdown": "I processed the huge dataset and saved it in a more friendly format [here](https://www.kaggle.com/datasets/titericz/leap-dataset-giba). It was split in 17 parts with 625k rows each. So you can easily load a subset and play with it.", "votes": 5, "replies": [ { "id": 2781855, "postDate": "2024-04-29T03:21:17.130Z", "content": "OSError: Could not open Parquet input source '<Buffer>': Metadata contains Thrift LogicalType that is not recognized
could not load the data
", "rawMarkdown": "`OSError: Could not open Parquet input source 'Of course you can. I'm loading it here: https://www.kaggle.com/code/titericz/giba-baseline-xgboost
", "rawMarkdown": "Of course you can. I'm loading it here: https://www.kaggle.com/code/titericz/giba-baseline-xgboost", "votes": 1 } ] } ] }, { "id": 2770787, "postDate": "2024-04-24T03:05:17.057Z", "content": "@rehankhandaya perhaps reading in chunks will help you, both pandas and polars offer chunked reading in of files.
\nI agree with you that using a 196GB csv file as a single training set was sub-optimal. I hope such issues could be handled better by hosts and Kaggle going ahead.
\nFor this purpose, you can also use the TPU, but be mindful of the weekly quotas herewith!
The train dataset is 180GB, how are we supposed to read it in, to then convert to parquet and load in as polar or dask, if we only have 20gb of ram? It fully saturates the ram and crashes before it can convert to parquet. I am using Kaggle notebook here, my personal comp is too weak for this, and I don't wan to pay for cloud computing.
", "rawMarkdown": "The train dataset is 180GB, how are we supposed to read it in, to then convert to parquet and load in as polar or dask, if we only have 20gb of ram? It fully saturates the ram and crashes before it can convert to parquet. I am using Kaggle notebook here, my personal comp is too weak for this, and I don't wan to pay for cloud computing.", "votes": 2 } ], "comments": [ { "id": 2772324, "author_name": "Giba", "author_url": "", "post_date": "2024-04-24T16:51:35.593000", "content": "I processed the huge dataset and saved it in a more friendly format here. It was split in 17 parts with 625k rows each. So you can easily load a subset and play with it.
", "votes": 5, "replies": [ { "id": 2781855, "author_name": "jewinv", "author_url": "", "post_date": "2024-04-29T03:21:17.130000", "content": "OSError: Could not open Parquet input source '<Buffer>': Metadata contains Thrift LogicalType that is not recognized
could not load the data
", "votes": 0, "replies": [ { "id": 2783143, "author_name": "Giba", "author_url": "", "post_date": "2024-04-29T15:38:33.707000", "content": "Of course you can. I'm loading it here: https://www.kaggle.com/code/titericz/giba-baseline-xgboost
", "votes": 1, "replies": [] } ] } ] }, { "id": 2770787, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-04-24T03:05:17.057000", "content": "@rehankhandaya perhaps reading in chunks will help you, both pandas and polars offer chunked reading in of files.
\nI agree with you that using a 196GB csv file as a single training set was sub-optimal. I hope such issues could be handled better by hosts and Kaggle going ahead.
\nFor this purpose, you can also use the TPU, but be mindful of the weekly quotas herewith!