{
  "id": 506853,
  "title": "Has anyone managed to download climSim data *in Kaggle*?",
  "url": "/competitions/leap-atmospheric-physics-ai-climsim/discussion/506853",
  "author_name": "greySnow",
  "post_date": "2024-05-23T13:17:11.605000",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I fail miserably…<br>\nAny ideas on how to make it work?<br>\nA working example on Kaggle would be extremely helpful.<br>\n<a href=\"https://www.kaggle.com/shlomoron/leap-download-data\" target=\"_blank\">My fail notebook</a></p>\n<p>EDIT:<br>\nApparently (well, obviously) we need to change the script to download from HF.<br>\n~sigh, this is going to take some work.</p>",
  "messages": [
    {
      "id": 2831018,
      "postDate": "2024-05-23T13:17:11.607Z",
      "content": "<p>I fail miserably…<br>\nAny ideas on how to make it work?<br>\nA working example on Kaggle would be extremely helpful.<br>\n<a href=\"https://www.kaggle.com/shlomoron/leap-download-data\" target=\"_blank\">My fail notebook</a></p>\n<p>EDIT:<br>\nApparently (well, obviously) we need to change the script to download from HF.<br>\n~sigh, this is going to take some work.</p>",
      "rawMarkdown": "I fail miserably...\nAny ideas on how to make it work?\nA working example on Kaggle would be extremely helpful.\n[My fail notebook](https://www.kaggle.com/shlomoron/leap-download-data)\n\nEDIT:\nApparently (well, obviously) we need to change the script to download from HF.\n~sigh, this is going to take some work.",
      "votes": 2
    },
    {
      "id": 2831254,
      "postDate": "2024-05-23T15:32:24.700Z",
      "content": "<p>I have started to upload the tfrecords of the full data, look here for the first block: <a href=\"https://www.kaggle.com/datasets/abiolatti/leap-complete-training-data-0001\" target=\"_blank\">https://www.kaggle.com/datasets/abiolatti/leap-complete-training-data-0001</a></p>",
      "rawMarkdown": "I have started to upload the tfrecords of the full data, look here for the first block: https://www.kaggle.com/datasets/abiolatti/leap-complete-training-data-0001",
      "replies": [
        {
          "id": 2831368,
          "postDate": "2024-05-23T16:33:52.127Z",
          "rawMarkdown": "",
          "isDeleted": true
        },
        {
          "id": 2831379,
          "postDate": "2024-05-23T16:37:28.530Z",
          "content": "<p>I saw. Thank you for your hard work. But this is the kind of work I will have to do myself to be sure I got everything right haha.</p>",
          "rawMarkdown": "I saw. Thank you for your hard work. But this is the kind of work I will have to do myself to be sure I got everything right haha.",
          "replies": [
            {
              "id": 2831416,
              "postDate": "2024-05-23T17:06:58.127Z",
              "content": "<p>I started from this: <a href=\"https://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py\" target=\"_blank\">https://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py</a></p>",
              "rawMarkdown": "I started from this: https://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py"
            },
            {
              "id": 2832856,
              "postDate": "2024-05-23T22:29:01.150Z",
              "content": "<p>Yes me too…problem is we need to change it to download from HF, it does not work straight out of the box haha</p>",
              "rawMarkdown": "Yes me too...problem is we need to change it to download from HF, it does not work straight out of the box haha"
            },
            {
              "id": 2832881,
              "postDate": "2024-05-23T23:18:37.653Z",
              "content": "<p>This worked for me (example to download year 0001 month 03)</p>\n<pre><code>huggingface_hub.snapshot_download(\n    repo_id=,\n    allow_patterns=,\n    cache_dir=,\n    local_dir=,\n    repo_type=,\n)\n</code></pre>",
              "rawMarkdown": "This worked for me (example to download year 0001 month 03)\n\n```python\nhuggingface_hub.snapshot_download(\n    repo_id=\"LEAP/ClimSim_low-res\",\n    allow_patterns=\"train/0001-03/*0001-*-*-*.nc\",\n    cache_dir=None,\n    local_dir=\".\",\n    repo_type=\"dataset\",\n)\n```",
              "votes": 3
            }
          ]
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2831254,
      "author_name": "Amedeo Biolatti",
      "author_url": "",
      "post_date": "2024-05-23T15:32:24.700000",
      "content": "<p>I have started to upload the tfrecords of the full data, look here for the first block: <a href=\"https://www.kaggle.com/datasets/abiolatti/leap-complete-training-data-0001\" target=\"_blank\">https://www.kaggle.com/datasets/abiolatti/leap-complete-training-data-0001</a></p>",
      "votes": 0,
      "replies": [
        {
          "id": 2831368,
          "author_name": "",
          "author_url": "",
          "post_date": "2024-05-23T16:33:52.127000",
          "content": "",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2831379,
          "author_name": "greySnow",
          "author_url": "",
          "post_date": "2024-05-23T16:37:28.530000",
          "content": "<p>I saw. Thank you for your hard work. But this is the kind of work I will have to do myself to be sure I got everything right haha.</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2831416,
              "author_name": "Amedeo Biolatti",
              "author_url": "",
              "post_date": "2024-05-23T17:06:58.127000",
              "content": "<p>I started from this: <a href=\"https://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py\" target=\"_blank\">https://github.com/leap-stc/ClimSim/blob/main/for_kaggle_users.py</a></p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2832856,
              "author_name": "greySnow",
              "author_url": "",
              "post_date": "2024-05-23T22:29:01.150000",
              "content": "<p>Yes me too…problem is we need to change it to download from HF, it does not work straight out of the box haha</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2832881,
              "author_name": "Amedeo Biolatti",
              "author_url": "",
              "post_date": "2024-05-23T23:18:37.653000",
              "content": "<p>This worked for me (example to download year 0001 month 03)</p>\n<pre><code>huggingface_hub.snapshot_download(\n    repo_id=,\n    allow_patterns=,\n    cache_dir=,\n    local_dir=,\n    repo_type=,\n)\n</code></pre>",
              "votes": 3,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2831018": "I fail miserably...\nAny ideas on how to make it work?\nA working example on Kaggle would be extremely helpful.\n[My fail notebook](https://www.kaggle.com/shlomoron/leap-download-data)\n\nEDIT:\nApparently (well, obviously) we need to change the script to download from HF.\n~sigh, this is going to take some work.",
    "2831254": "I have started to upload the tfrecords of the full data, look here for the first block: https://www.kaggle.com/datasets/abiolatti/leap-complete-training-data-0001"
  }
}