{
  "id": 375418,
  "title": "How to load the dcm images efficiently",
  "url": "/competitions/rsna-breast-cancer-detection/discussion/375418",
  "author_name": "Ridhesh Goti",
  "post_date": "2023-01-01T14:16:42.938000",
  "votes": 2,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>How could I load all the training dcm images efficiently and rapidly? </p>\n<p>Thanks,</p>",
  "messages": [
    {
      "id": 2082387,
      "postDate": "2023-01-01T14:16:42.940Z",
      "content": "<p>Hi everyone,</p>\n<p>How could I load all the training dcm images efficiently and rapidly? </p>\n<p>Thanks,</p>",
      "rawMarkdown": "Hi everyone,\n\n  How could I load all the training dcm images efficiently and rapidly? \n\nThanks,",
      "votes": 2
    },
    {
      "id": 2087914,
      "postDate": "2023-01-05T22:18:19.487Z",
      "content": "<p>I believe the best bet it is read all the dicom images and store it in a different format maybe .jpeg or .png. <br>\nReading DICOM matrix looks computational expensive.</p>\n<p>I am using this already converted public dataset to train and validate: <a href=\"https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs\" target=\"_blank\">https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs</a></p>",
      "rawMarkdown": "I believe the best bet it is read all the dicom images and store it in a different format maybe .jpeg or .png. \nReading DICOM matrix looks computational expensive.\n\nI am using this already converted public dataset to train and validate: https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs",
      "replies": [
        {
          "id": 2088393,
          "postDate": "2023-01-06T09:32:37.817Z",
          "content": "<p>Thanks. I also stored the pixel data in the.pkl file but it took me 5 hours to run. </p>",
          "rawMarkdown": "Thanks. I also stored the pixel data in the.pkl file but it took me 5 hours to run. ",
          "replies": [
            {
              "id": 2091808,
              "postDate": "2023-01-08T18:52:02.167Z",
              "content": "<p>is it a single .pkl file or one per image? if it is single .pkl then again system can't load all the data into memory during prediction. I will do it per file or as a batch</p>",
              "rawMarkdown": "is it a single .pkl file or one per image? if it is single .pkl then again system can't load all the data into memory during prediction. I will do it per file or as a batch"
            },
            {
              "id": 2092192,
              "postDate": "2023-01-09T07:52:52.070Z",
              "content": "<p>No  I stored it in one single file. After storing this file, I don't have any problems regarding the memory overflow problem. I can load the pixel data within a minute. If you want to have a look please look at this which I created <a href=\"https://www.kaggle.com/datasets/ridheshgoti/output\" target=\"_blank\">https://www.kaggle.com/datasets/ridheshgoti/output</a>. Just for your information, this dataset was processed with some image modifications like the crop.</p>",
              "rawMarkdown": "No  I stored it in one single file. After storing this file, I don't have any problems regarding the memory overflow problem. I can load the pixel data within a minute. If you want to have a look please look at this which I created https://www.kaggle.com/datasets/ridheshgoti/output. Just for your information, this dataset was processed with some image modifications like the crop."
            }
          ]
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2087914,
      "author_name": "dhinesh",
      "author_url": "",
      "post_date": "2023-01-05T22:18:19.487000",
      "content": "<p>I believe the best bet it is read all the dicom images and store it in a different format maybe .jpeg or .png. <br>\nReading DICOM matrix looks computational expensive.</p>\n<p>I am using this already converted public dataset to train and validate: <a href=\"https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs\" target=\"_blank\">https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs</a></p>",
      "votes": 0,
      "replies": [
        {
          "id": 2088393,
          "author_name": "Ridhesh Goti",
          "author_url": "",
          "post_date": "2023-01-06T09:32:37.817000",
          "content": "<p>Thanks. I also stored the pixel data in the.pkl file but it took me 5 hours to run. </p>",
          "votes": 0,
          "replies": [
            {
              "id": 2091808,
              "author_name": "dhinesh",
              "author_url": "",
              "post_date": "2023-01-08T18:52:02.167000",
              "content": "<p>is it a single .pkl file or one per image? if it is single .pkl then again system can't load all the data into memory during prediction. I will do it per file or as a batch</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2092192,
              "author_name": "Ridhesh Goti",
              "author_url": "",
              "post_date": "2023-01-09T07:52:52.070000",
              "content": "<p>No  I stored it in one single file. After storing this file, I don't have any problems regarding the memory overflow problem. I can load the pixel data within a minute. If you want to have a look please look at this which I created <a href=\"https://www.kaggle.com/datasets/ridheshgoti/output\" target=\"_blank\">https://www.kaggle.com/datasets/ridheshgoti/output</a>. Just for your information, this dataset was processed with some image modifications like the crop.</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2082387": "Hi everyone,\n\n  How could I load all the training dcm images efficiently and rapidly? \n\nThanks,",
    "2087914": "I believe the best bet it is read all the dicom images and store it in a different format maybe .jpeg or .png. \nReading DICOM matrix looks computational expensive.\n\nI am using this already converted public dataset to train and validate: https://www.kaggle.com/datasets/radek1/rsna-mammography-images-as-pngs"
  }
}