{
  "id": 429266,
  "title": "Hi all! May have found something which speeds up reading dicom files (1.6x faster)",
  "url": "/competitions/rsna-2023-abdominal-trauma-detection/discussion/429266",
  "author_name": "Art. Berz.",
  "post_date": "2023-08-04T18:40:11.208000",
  "votes": 22,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I'm sort of new to this particular kind of competition, so I'm having to do lots of research on a variety of different parts of the process (only learned what a data pipeline was recently). I stumbled upon a kaggle post referencing 'dicomsdl' while trying to reduce the memory usage / increase speed of a certain function I created to process a segment of dicom files (roughly 32000). <br>\nIn my personal testing of .pixelData property (.pixel_array in Pydicom), I noticed a change in function execution time from about 9 minutes (547s) using Pydicom to about 6 minutes (340s) using Dicomsdl . There was a similar drop in memory usage, 22.3gb using Pydicom to 16.5gb using Dicomsdl. </p>\n<p>Link to original kaggle discussion in which this was discovered: <a href=\"url\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369684#2057282</a></p>\n<p>Link to code notebook showing speed tests: <a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/remekkinas/fast-dicom-processing-1-6-2x-faster/notebook</a></p>\n<p>Note: .pixelData appears to return arrays in float32 type by default, be sure to pass 'storedvalue = True' if that is undesirable, like this:<br>\n<code>myarray = dicomsdl.open(your_file_path_here).pixelData(storedvalue = True)</code></p>\n<p>Hope this helps, good luck in the competition! 🙂</p>",
  "messages": [
    {
      "id": 2374184,
      "postDate": "2023-08-04T18:40:11.207Z",
      "content": "<p>I'm sort of new to this particular kind of competition, so I'm having to do lots of research on a variety of different parts of the process (only learned what a data pipeline was recently). I stumbled upon a kaggle post referencing 'dicomsdl' while trying to reduce the memory usage / increase speed of a certain function I created to process a segment of dicom files (roughly 32000). <br>\nIn my personal testing of .pixelData property (.pixel_array in Pydicom), I noticed a change in function execution time from about 9 minutes (547s) using Pydicom to about 6 minutes (340s) using Dicomsdl . There was a similar drop in memory usage, 22.3gb using Pydicom to 16.5gb using Dicomsdl. </p>\n<p>Link to original kaggle discussion in which this was discovered: <a href=\"url\" target=\"_blank\">https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369684#2057282</a></p>\n<p>Link to code notebook showing speed tests: <a href=\"url\" target=\"_blank\">https://www.kaggle.com/code/remekkinas/fast-dicom-processing-1-6-2x-faster/notebook</a></p>\n<p>Note: .pixelData appears to return arrays in float32 type by default, be sure to pass 'storedvalue = True' if that is undesirable, like this:<br>\n<code>myarray = dicomsdl.open(your_file_path_here).pixelData(storedvalue = True)</code></p>\n<p>Hope this helps, good luck in the competition! 🙂</p>",
      "rawMarkdown": "I'm sort of new to this particular kind of competition, so I'm having to do lots of research on a variety of different parts of the process (only learned what a data pipeline was recently). I stumbled upon a kaggle post referencing 'dicomsdl' while trying to reduce the memory usage / increase speed of a certain function I created to process a segment of dicom files (roughly 32000). \nIn my personal testing of .pixelData property (.pixel_array in Pydicom), I noticed a change in function execution time from about 9 minutes (547s) using Pydicom to about 6 minutes (340s) using Dicomsdl . There was a similar drop in memory usage, 22.3gb using Pydicom to 16.5gb using Dicomsdl. \n\nLink to original kaggle discussion in which this was discovered: [https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369684#2057282](url)\n\nLink to code notebook showing speed tests: [https://www.kaggle.com/code/remekkinas/fast-dicom-processing-1-6-2x-faster/notebook](url)\n\nNote: .pixelData appears to return arrays in float32 type by default, be sure to pass 'storedvalue = True' if that is undesirable, like this:\n`myarray = dicomsdl.open(your_file_path_here).pixelData(storedvalue = True)`\n\nHope this helps, good luck in the competition! 🙂",
      "votes": 21
    },
    {
      "id": 2374505,
      "postDate": "2023-08-05T03:37:50.290Z",
      "content": "<p>thank you so much that would be so helpful</p>",
      "rawMarkdown": "thank you so much that would be so helpful",
      "votes": 1
    },
    {
      "id": 2419976,
      "postDate": "2023-09-02T09:56:36.297Z",
      "content": "<p>Thank you!</p>",
      "rawMarkdown": "Thank you!"
    },
    {
      "id": 2374455,
      "postDate": "2023-08-05T02:03:41.640Z",
      "content": "<p>Thanks for sharing!</p>",
      "rawMarkdown": "Thanks for sharing!",
      "votes": 1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2374505,
      "author_name": "golnaz ahmadvand",
      "author_url": "",
      "post_date": "2023-08-05T03:37:50.290000",
      "content": "<p>thank you so much that would be so helpful</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2419976,
      "author_name": "Gyula Maloveczky4",
      "author_url": "",
      "post_date": "2023-09-02T09:56:36.297000",
      "content": "<p>Thank you!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2374455,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-08-05T02:03:41.640000",
      "content": "<p>Thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2374184": "I'm sort of new to this particular kind of competition, so I'm having to do lots of research on a variety of different parts of the process (only learned what a data pipeline was recently). I stumbled upon a kaggle post referencing 'dicomsdl' while trying to reduce the memory usage / increase speed of a certain function I created to process a segment of dicom files (roughly 32000). \nIn my personal testing of .pixelData property (.pixel_array in Pydicom), I noticed a change in function execution time from about 9 minutes (547s) using Pydicom to about 6 minutes (340s) using Dicomsdl . There was a similar drop in memory usage, 22.3gb using Pydicom to 16.5gb using Dicomsdl. \n\nLink to original kaggle discussion in which this was discovered: [https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/369684#2057282](url)\n\nLink to code notebook showing speed tests: [https://www.kaggle.com/code/remekkinas/fast-dicom-processing-1-6-2x-faster/notebook](url)\n\nNote: .pixelData appears to return arrays in float32 type by default, be sure to pass 'storedvalue = True' if that is undesirable, like this:\n`myarray = dicomsdl.open(your_file_path_here).pixelData(storedvalue = True)`\n\nHope this helps, good luck in the competition! 🙂",
    "2374505": "thank you so much that would be so helpful",
    "2419976": "Thank you!",
    "2374455": "Thanks for sharing!"
  }
}