{
  "id": 114650,
  "title": "What's wrong in this?",
  "url": "/competitions/rsna-intracranial-hemorrhage-detection/discussion/114650",
  "author_name": "NitinKshatriya",
  "post_date": "2019-10-28T09:44:23.120000",
  "votes": 0,
  "comment_count": 2,
  "views": 0,
  "content": "<p>By going through various kernel, I came up with this data generator. But running this for full data makes kernel run out of RAM. </p>\n\n<p>`\nclass DataGen(Sequence):\n    def <strong>init</strong>(self, images_path,labels=None,cols=None,batch_size=64,shuffle=False):\n            #images_path: series from train containing path of each image\n            #labels: Dataframe with values of \n            #\"any\",\"epidural\",\"intraparenchymal\",\"intraventricular\",\"subarachnoid\",\"subdural\"\n            #cols: [\"any\",\"epidural\",\"intraparenchymal\",\"intraventricular\",\"subarachnoid\",\"subdural\"]</p>\n\n<pre><code>    self.images_path = images_path\n    self.n = len(self.images_path)\n    self.labels = labels\n    self.cols = cols\n    self.batch_size = batch_size\n    self.img_size = img_size\n    self.shuffle = shuffle\n    self.augment = augment\n    self.on_epoch_end()\n\ndef __len__(self):\n    return int(np.ceil(self.n / self.batch_size))\n\ndef on_epoch_end(self):\n    self.indexes = np.arange(len(self.images_path))\n    if self.shuffle:\n        np.random.shuffle(self.indexes)\n\ndef load_image(self, image_path):\n    data = pydicom.dcmread(image_path)\n    window_center, window_width, intercept, slope = get_windowing(data)\n    image = pydicom.read_file(image_path).pixel_array.astype('float32')\n    image = join_windows(image, intercept, slope)\n    return image\n\ndef __getitem__(self, index):\n    indexes = self.indexes[index * self.batch_size : (index+1) * self.batch_size]\n    labels = np.array([self.labels[self.cols].iloc[k] for k in indexes])\n    images = np.array([self.load_image(self.images_path.iloc[k]) for k in indexes])\n\n    if self.augment:\n        images = self.augmentor(images)\n    return images, labels\n\ndef augmentor(self, images):\n    pass`\n</code></pre>\n\n<p>`</p>",
  "messages": [
    {
      "id": 659931,
      "postDate": "2019-10-28T13:15:05.513Z",
      "content": "<p>Try to reduce the batchsize</p>",
      "rawMarkdown": "Try to reduce the batchsize",
      "replies": [
        {
          "id": 659938,
          "postDate": "2019-10-28T13:23:44.913Z",
          "content": "<p>even after reducing the batch size sometime i am getting the same issue. And for some cases for which it's running, it's geving 1 epoch time as 25-70 hours which kind of looks alot.</p>",
          "rawMarkdown": "even after reducing the batch size sometime i am getting the same issue. And for some cases for which it's running, it's geving 1 epoch time as 25-70 hours which kind of looks alot."
        }
      ]
    },
    {
      "id": 659814,
      "postDate": "2019-10-28T09:44:23.120Z",
      "content": "<p>By going through various kernel, I came up with this data generator. But running this for full data makes kernel run out of RAM. </p>\n\n<p>`\nclass DataGen(Sequence):\n    def <strong>init</strong>(self, images_path,labels=None,cols=None,batch_size=64,shuffle=False):\n            #images_path: series from train containing path of each image\n            #labels: Dataframe with values of \n            #\"any\",\"epidural\",\"intraparenchymal\",\"intraventricular\",\"subarachnoid\",\"subdural\"\n            #cols: [\"any\",\"epidural\",\"intraparenchymal\",\"intraventricular\",\"subarachnoid\",\"subdural\"]</p>\n\n<pre><code>    self.images_path = images_path\n    self.n = len(self.images_path)\n    self.labels = labels\n    self.cols = cols\n    self.batch_size = batch_size\n    self.img_size = img_size\n    self.shuffle = shuffle\n    self.augment = augment\n    self.on_epoch_end()\n\ndef __len__(self):\n    return int(np.ceil(self.n / self.batch_size))\n\ndef on_epoch_end(self):\n    self.indexes = np.arange(len(self.images_path))\n    if self.shuffle:\n        np.random.shuffle(self.indexes)\n\ndef load_image(self, image_path):\n    data = pydicom.dcmread(image_path)\n    window_center, window_width, intercept, slope = get_windowing(data)\n    image = pydicom.read_file(image_path).pixel_array.astype('float32')\n    image = join_windows(image, intercept, slope)\n    return image\n\ndef __getitem__(self, index):\n    indexes = self.indexes[index * self.batch_size : (index+1) * self.batch_size]\n    labels = np.array([self.labels[self.cols].iloc[k] for k in indexes])\n    images = np.array([self.load_image(self.images_path.iloc[k]) for k in indexes])\n\n    if self.augment:\n        images = self.augmentor(images)\n    return images, labels\n\ndef augmentor(self, images):\n    pass`\n</code></pre>\n\n<p>`</p>",
      "rawMarkdown": "By going through various kernel, I came up with this data generator. But running this for full data makes kernel run out of RAM. \n\n\n`\nclass DataGen(Sequence):\n    def __init__(self, images_path,labels=None,cols=None,batch_size=64,shuffle=False):\n            #images_path: series from train containing path of each image\n            #labels: Dataframe with values of \n            #\"any\",\"epidural\",\"intraparenchymal\",\"intraventricular\",\"subarachnoid\",\"subdural\"\n            #cols: [\"any\",\"epidural\",\"intraparenchymal\",\"intraventricular\",\"subarachnoid\",\"subdural\"]\n\n        self.images_path = images_path\n        self.n = len(self.images_path)\n        self.labels = labels\n        self.cols = cols\n        self.batch_size = batch_size\n        self.img_size = img_size\n        self.shuffle = shuffle\n        self.augment = augment\n        self.on_epoch_end()\n            \n    def __len__(self):\n        return int(np.ceil(self.n / self.batch_size))\n    \n    def on_epoch_end(self):\n        self.indexes = np.arange(len(self.images_path))\n        if self.shuffle:\n            np.random.shuffle(self.indexes)\n    \n    def load_image(self, image_path):\n        data = pydicom.dcmread(image_path)\n        window_center, window_width, intercept, slope = get_windowing(data)\n        image = pydicom.read_file(image_path).pixel_array.astype('float32')\n        image = join_windows(image, intercept, slope)\n        return image\n        \n    def __getitem__(self, index):\n        indexes = self.indexes[index * self.batch_size : (index+1) * self.batch_size]\n        labels = np.array([self.labels[self.cols].iloc[k] for k in indexes])\n        images = np.array([self.load_image(self.images_path.iloc[k]) for k in indexes])\n        \n        if self.augment:\n            images = self.augmentor(images)\n        return images, labels\n    \n    def augmentor(self, images):\n        pass`\n\n`"
    }
  ],
  "comments": [
    {
      "id": 659931,
      "author_name": "Felipe Loque",
      "author_url": "",
      "post_date": "2019-10-28T13:15:05.513000",
      "content": "<p>Try to reduce the batchsize</p>",
      "votes": 0,
      "replies": [
        {
          "id": 659938,
          "author_name": "NitinKshatriya",
          "author_url": "",
          "post_date": "2019-10-28T13:23:44.913000",
          "content": "<p>even after reducing the batch size sometime i am getting the same issue. And for some cases for which it's running, it's geving 1 epoch time as 25-70 hours which kind of looks alot.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "659931": "Try to reduce the batchsize",
    "659814": "By going through various kernel, I came up with this data generator. But running this for full data makes kernel run out of RAM. \n\n\n`\nclass DataGen(Sequence):\n    def __init__(self, images_path,labels=None,cols=None,batch_size=64,shuffle=False):\n            #images_path: series from train containing path of each image\n            #labels: Dataframe with values of \n            #\"any\",\"epidural\",\"intraparenchymal\",\"intraventricular\",\"subarachnoid\",\"subdural\"\n            #cols: [\"any\",\"epidural\",\"intraparenchymal\",\"intraventricular\",\"subarachnoid\",\"subdural\"]\n\n        self.images_path = images_path\n        self.n = len(self.images_path)\n        self.labels = labels\n        self.cols = cols\n        self.batch_size = batch_size\n        self.img_size = img_size\n        self.shuffle = shuffle\n        self.augment = augment\n        self.on_epoch_end()\n            \n    def __len__(self):\n        return int(np.ceil(self.n / self.batch_size))\n    \n    def on_epoch_end(self):\n        self.indexes = np.arange(len(self.images_path))\n        if self.shuffle:\n            np.random.shuffle(self.indexes)\n    \n    def load_image(self, image_path):\n        data = pydicom.dcmread(image_path)\n        window_center, window_width, intercept, slope = get_windowing(data)\n        image = pydicom.read_file(image_path).pixel_array.astype('float32')\n        image = join_windows(image, intercept, slope)\n        return image\n        \n    def __getitem__(self, index):\n        indexes = self.indexes[index * self.batch_size : (index+1) * self.batch_size]\n        labels = np.array([self.labels[self.cols].iloc[k] for k in indexes])\n        images = np.array([self.load_image(self.images_path.iloc[k]) for k in indexes])\n        \n        if self.augment:\n            images = self.augmentor(images)\n        return images, labels\n    \n    def augmentor(self, images):\n        pass`\n\n`"
  }
}