{ "id": 495316, "title": "Sharing some starter artefacts for convenience", "url": "/competitions/leap-atmospheric-physics-ai-climsim/discussion/495316", "author_name": "Ravi Ramakrishnan", "post_date": "2024-04-20T14:24:14.585000", "votes": 5, "comment_count": 0, "views": 0, "content": "

Hello everyone,

\n

I am sure that this competition is likely to posit significant challenges for data collation and wrangling considering the large size of the training data. I have thereby prepared 7.5 million rows of data as below-

\n
    \n
  1. I collated a dataset with batches of 5000 rows per chunk and 2.5 million records per folder
    2. \n
  2. I created 2 folders in the dataset, totally amounting to 5 million records in the dataset here
    3. \n
\n

One may simply concatenate the requisite number of batches/ chunks as per his/ her requirements and proceed as a starter material. Please note that all the chunks preserve the train data structure with some datatype shrinkages.

\n

I also augmented the dataset with a starter kernel with additional 2.5 million records amounting to 7.5 million record collection, all in batches of 5000 per chunk. All files are named as Train_Batch{i}.parquet, *i ranging from 0-499 per folder/ kernel output.
\nEach folder is approximately 6.2 GB large, so use with caution.

\n

I also extended the idea discussed in this kernel to include this using polars and on my 7.5 million train data subset to yield a starter result. The submitted kernel is here for perusal. This kernel scores 0.17117 on the leaderboard.

\n

I purposely used polars for wrangling to illustrate the efficacy of the library herewith.

\n

Additionally, I have saved the sample submission file in parquet in my starter dataset, reducing the time taken to import and use it too hereby. Please feel free to use these elements in your pipelines too!

\n

Best wishes!

", "messages": [ { "id": 2763410, "postDate": "2024-04-20T14:24:14.587Z", "content": "

Hello everyone,

\n

I am sure that this competition is likely to posit significant challenges for data collation and wrangling considering the large size of the training data. I have thereby prepared 7.5 million rows of data as below-

\n
    \n
  1. I collated a dataset with batches of 5000 rows per chunk and 2.5 million records per folder
    2. \n
  2. I created 2 folders in the dataset, totally amounting to 5 million records in the dataset here
    3. \n
\n

One may simply concatenate the requisite number of batches/ chunks as per his/ her requirements and proceed as a starter material. Please note that all the chunks preserve the train data structure with some datatype shrinkages.

\n

I also augmented the dataset with a starter kernel with additional 2.5 million records amounting to 7.5 million record collection, all in batches of 5000 per chunk. All files are named as Train_Batch{i}.parquet, *i ranging from 0-499 per folder/ kernel output.
\nEach folder is approximately 6.2 GB large, so use with caution.

\n

I also extended the idea discussed in this kernel to include this using polars and on my 7.5 million train data subset to yield a starter result. The submitted kernel is here for perusal. This kernel scores 0.17117 on the leaderboard.

\n

I purposely used polars for wrangling to illustrate the efficacy of the library herewith.

\n

Additionally, I have saved the sample submission file in parquet in my starter dataset, reducing the time taken to import and use it too hereby. Please feel free to use these elements in your pipelines too!

\n

Best wishes!

", "rawMarkdown": "Hello everyone,\n\nI am sure that this competition is likely to posit significant challenges for data collation and wrangling considering the large size of the training data. I have thereby prepared 7.5 million rows of data as below-\n1. I collated a dataset with batches of 5000 rows per chunk and 2.5 million records per folder\n2. I created 2 folders in the dataset, totally amounting to 5 million records in the dataset [here](https://www.kaggle.com/datasets/ravi20076/climsim2024starterv1)\n\nOne may simply concatenate the requisite number of batches/ chunks as per his/ her requirements and proceed as a starter material. Please note that all the chunks preserve the train data structure with some datatype shrinkages. \n\nI also augmented the dataset with a [starter kernel](https://www.kaggle.com/code/ravi20076/climsim2024-starterdata-v1) with additional 2.5 million records amounting to 7.5 million record collection, all in batches of 5000 per chunk. All files are named as **Train_Batch{i}.parquet**, *i ranging from 0-499 per folder/ kernel output. \nEach folder is approximately **6.2 GB large**, so use with caution. \n\nI also extended the idea discussed in [this kernel](https://www.kaggle.com/code/asarvazyan/leap-predict-the-mean) to include this using polars and on my 7.5 million train data subset to yield a starter result. The submitted kernel is [here](https://www.kaggle.com/code/ravi20076/climsim-starter-v2) for perusal. This kernel scores **0.17117** on the leaderboard. \n\nI purposely used polars for wrangling to illustrate the efficacy of the library herewith. \n\nAdditionally, I have saved the sample submission file in parquet in my starter dataset, reducing the time taken to import and use it too hereby. Please feel free to use these elements in your pipelines too!\n\nBest wishes!", "votes": 5 } ], "comments": [], "raw_markdown_by_id": { "2763410": "Hello everyone,\n\nI am sure that this competition is likely to posit significant challenges for data collation and wrangling considering the large size of the training data. I have thereby prepared 7.5 million rows of data as below-\n1. I collated a dataset with batches of 5000 rows per chunk and 2.5 million records per folder\n2. I created 2 folders in the dataset, totally amounting to 5 million records in the dataset [here](https://www.kaggle.com/datasets/ravi20076/climsim2024starterv1)\n\nOne may simply concatenate the requisite number of batches/ chunks as per his/ her requirements and proceed as a starter material. Please note that all the chunks preserve the train data structure with some datatype shrinkages. \n\nI also augmented the dataset with a [starter kernel](https://www.kaggle.com/code/ravi20076/climsim2024-starterdata-v1) with additional 2.5 million records amounting to 7.5 million record collection, all in batches of 5000 per chunk. All files are named as **Train_Batch{i}.parquet**, *i ranging from 0-499 per folder/ kernel output. \nEach folder is approximately **6.2 GB large**, so use with caution. \n\nI also extended the idea discussed in [this kernel](https://www.kaggle.com/code/asarvazyan/leap-predict-the-mean) to include this using polars and on my 7.5 million train data subset to yield a starter result. The submitted kernel is [here](https://www.kaggle.com/code/ravi20076/climsim-starter-v2) for perusal. This kernel scores **0.17117** on the leaderboard. \n\nI purposely used polars for wrangling to illustrate the efficacy of the library herewith. \n\nAdditionally, I have saved the sample submission file in parquet in my starter dataset, reducing the time taken to import and use it too hereby. Please feel free to use these elements in your pipelines too!\n\nBest wishes!" } }