{
  "id": 359485,
  "title": "Tips for speeding up data loading?",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/359485",
  "author_name": "Marius ",
  "post_date": "2022-10-12T08:36:47.490000",
  "votes": 0,
  "comment_count": 3,
  "views": 0,
  "content": "<p>The biggest bottleneck in my model is the actual loading of the image volumes. I already saved them as resized numpy arrays to disc, but It's still taking ages to load them in a DataLoader with num_workers&gt;0. </p>\n<p>Do you have any tips for speeding up the loading process? </p>\n<p>I feel like the CPU is the reason here. All of my CPUs are constantly on 100% in kernel threads.</p>",
  "messages": [
    {
      "id": 1985657,
      "postDate": "2022-10-13T13:28:47.730Z",
      "content": "<p>Are you talking about training or at test time?  For training, I have found that hdf5 files are quite fast to load when stored as 3D arrays.<br>\nFor test time, it is more complicated, because you will run out of temporary storage space, depending on how you store them (uint8, 1 channel or 3, etc.).  I found using jpeg compression worked well and could be stored/loaded quickly.  If you are using TensorFlow, TFRecords can store jpeg records serially efficiently.</p>",
      "rawMarkdown": "Are you talking about training or at test time?  For training, I have found that hdf5 files are quite fast to load when stored as 3D arrays.\nFor test time, it is more complicated, because you will run out of temporary storage space, depending on how you store them (uint8, 1 channel or 3, etc.).  I found using jpeg compression worked well and could be stored/loaded quickly.  If you are using TensorFlow, TFRecords can store jpeg records serially efficiently.\n",
      "votes": 1,
      "replies": [
        {
          "id": 1988569,
          "postDate": "2022-10-15T12:04:26.313Z",
          "content": "<p>I'm more talking about training time. The actual dataloading from disc from the saved numpy arrays is super fast with around 50 ms. But my dataloader is somewhat slow. Using 7 CPU cores and a batch size = 28, loading one batch takes over 2 mins. I feel like collating the batches is what slows me down, I don't know why </p>",
          "rawMarkdown": "I'm more talking about training time. The actual dataloading from disc from the saved numpy arrays is super fast with around 50 ms. But my dataloader is somewhat slow. Using 7 CPU cores and a batch size = 28, loading one batch takes over 2 mins. I feel like collating the batches is what slows me down, I don't know why "
        },
        {
          "id": 1998814,
          "postDate": "2022-10-21T21:51:04.770Z",
          "content": "<p>Using the <code>CacheDataset</code> from Monai actually made a huge difference for me…</p>",
          "rawMarkdown": "Using the `CacheDataset` from Monai actually made a huge difference for me..."
        }
      ]
    },
    {
      "id": 1983798,
      "postDate": "2022-10-12T08:36:47.490Z",
      "content": "<p>The biggest bottleneck in my model is the actual loading of the image volumes. I already saved them as resized numpy arrays to disc, but It's still taking ages to load them in a DataLoader with num_workers&gt;0. </p>\n<p>Do you have any tips for speeding up the loading process? </p>\n<p>I feel like the CPU is the reason here. All of my CPUs are constantly on 100% in kernel threads.</p>",
      "rawMarkdown": "The biggest bottleneck in my model is the actual loading of the image volumes. I already saved them as resized numpy arrays to disc, but It's still taking ages to load them in a DataLoader with num_workers>0. \n\nDo you have any tips for speeding up the loading process? \n\nI feel like the CPU is the reason here. All of my CPUs are constantly on 100% in kernel threads."
    }
  ],
  "comments": [
    {
      "id": 1985657,
      "author_name": "SolverWorld",
      "author_url": "",
      "post_date": "2022-10-13T13:28:47.730000",
      "content": "<p>Are you talking about training or at test time?  For training, I have found that hdf5 files are quite fast to load when stored as 3D arrays.<br>\nFor test time, it is more complicated, because you will run out of temporary storage space, depending on how you store them (uint8, 1 channel or 3, etc.).  I found using jpeg compression worked well and could be stored/loaded quickly.  If you are using TensorFlow, TFRecords can store jpeg records serially efficiently.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1988569,
          "author_name": "Marius ",
          "author_url": "",
          "post_date": "2022-10-15T12:04:26.313000",
          "content": "<p>I'm more talking about training time. The actual dataloading from disc from the saved numpy arrays is super fast with around 50 ms. But my dataloader is somewhat slow. Using 7 CPU cores and a batch size = 28, loading one batch takes over 2 mins. I feel like collating the batches is what slows me down, I don't know why </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1998814,
          "author_name": "Marius ",
          "author_url": "",
          "post_date": "2022-10-21T21:51:04.770000",
          "content": "<p>Using the <code>CacheDataset</code> from Monai actually made a huge difference for me…</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1985657": "Are you talking about training or at test time?  For training, I have found that hdf5 files are quite fast to load when stored as 3D arrays.\nFor test time, it is more complicated, because you will run out of temporary storage space, depending on how you store them (uint8, 1 channel or 3, etc.).  I found using jpeg compression worked well and could be stored/loaded quickly.  If you are using TensorFlow, TFRecords can store jpeg records serially efficiently.\n",
    "1983798": "The biggest bottleneck in my model is the actual loading of the image volumes. I already saved them as resized numpy arrays to disc, but It's still taking ages to load them in a DataLoader with num_workers>0. \n\nDo you have any tips for speeding up the loading process? \n\nI feel like the CPU is the reason here. All of my CPUs are constantly on 100% in kernel threads."
  }
}