{
  "id": 359223,
  "title": "My torch Dataset uses too much RAM and is pretty slow",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/359223",
  "author_name": "osman bayram",
  "post_date": "2022-10-11T10:02:58.507000",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I want to build a torch 3d dataset class that will be used for pretty basic CNN models. But for some reason, it uses too much RAM. But i am only storing the shape of the tensors in a list.</p>\n<ul>\n<li>And also If you have any other improvements on this i would love to hear them. </li>\n</ul>\n<p>Thanks :)</p>\n<pre><code>!pip install torchio\nfrom torchvision import transforms\nfrom tqdm import tqdm\nimport time\nfrom torch.utils.data import Dataset, DataLoader\nimport torch\nimport pandas as pd\nfrom glob import glob\nimport torchio\n\ndataset_path = \"../input/rsna-2022-cervical-spine-fracture-detection/\"\n\nclass RSNADataset(Dataset):\n\n    def __init__(self, subset='train', transform=None):\n        self.transform = transform\n        self.df = pd.read_csv(dataset_path + subset + \".csv\")\n        self.train = subset == 'train'\n\n        print('Glob started, here we go :(') # it took 260 seconds first time :/\n        start_time = time.time()\n        self.image_paths = glob(dataset_path + subset + '_images/**/**')\n        print(f'Glob took {time.time() - start_time:.2f} seconds')\n\n\n        self.target_cols = ['C1','C2','C3','C4','C5','C6','C7','patient_overall']\n\n    def __getitem__(self, index):\n        \"\"\"\n        Prepare 1 batch of 3d images and labels\n        \"\"\"\n        row = self.df.iloc[index]\n\n        image_paths = [i for i in self.image_paths if row['StudyInstanceUID'] in i]\n\n        images = []\n\n        for path in image_paths:\n            images.append(torchio.data.image.read_image(path)[0])\n\n        images = torch.stack(images)\n\n        # apply augmentation\n        if self.transform:\n            images = self.transform(images)\n\n        # get labels\n        if self.train:\n            labels = torch.tensor(row[self.target_cols]).float()\n            return images, labels\n        else:\n            return images\n\n    def __len__(self):\n        return len(self.df)\n\ntrain_ds = RSNADataset(subset=\"train\")\n\nimage_counts = [train_ds[i][0].shape for i in tqdm(range(len(train_ds)))]\ntrain_ds[0][0].shape\n</code></pre>",
  "messages": [
    {
      "id": 1982694,
      "postDate": "2022-10-11T15:03:08.317Z",
      "content": "<p>Try using garbage collector or some other ram-clearing library to clear your memory after returning the images. A lot of times what happens is the runtime runs out of RAM even by loading many images, whether or not you are saving them anywhere. I had a similar problem with the Mayo Clinic Competition from last week.</p>",
      "rawMarkdown": "Try using garbage collector or some other ram-clearing library to clear your memory after returning the images. A lot of times what happens is the runtime runs out of RAM even by loading many images, whether or not you are saving them anywhere. I had a similar problem with the Mayo Clinic Competition from last week.",
      "votes": 1
    },
    {
      "id": 1982183,
      "postDate": "2022-10-11T10:02:58.507Z",
      "content": "<p>I want to build a torch 3d dataset class that will be used for pretty basic CNN models. But for some reason, it uses too much RAM. But i am only storing the shape of the tensors in a list.</p>\n<ul>\n<li>And also If you have any other improvements on this i would love to hear them. </li>\n</ul>\n<p>Thanks :)</p>\n<pre><code>!pip install torchio\nfrom torchvision import transforms\nfrom tqdm import tqdm\nimport time\nfrom torch.utils.data import Dataset, DataLoader\nimport torch\nimport pandas as pd\nfrom glob import glob\nimport torchio\n\ndataset_path = \"../input/rsna-2022-cervical-spine-fracture-detection/\"\n\nclass RSNADataset(Dataset):\n\n    def __init__(self, subset='train', transform=None):\n        self.transform = transform\n        self.df = pd.read_csv(dataset_path + subset + \".csv\")\n        self.train = subset == 'train'\n\n        print('Glob started, here we go :(') # it took 260 seconds first time :/\n        start_time = time.time()\n        self.image_paths = glob(dataset_path + subset + '_images/**/**')\n        print(f'Glob took {time.time() - start_time:.2f} seconds')\n\n\n        self.target_cols = ['C1','C2','C3','C4','C5','C6','C7','patient_overall']\n\n    def __getitem__(self, index):\n        \"\"\"\n        Prepare 1 batch of 3d images and labels\n        \"\"\"\n        row = self.df.iloc[index]\n\n        image_paths = [i for i in self.image_paths if row['StudyInstanceUID'] in i]\n\n        images = []\n\n        for path in image_paths:\n            images.append(torchio.data.image.read_image(path)[0])\n\n        images = torch.stack(images)\n\n        # apply augmentation\n        if self.transform:\n            images = self.transform(images)\n\n        # get labels\n        if self.train:\n            labels = torch.tensor(row[self.target_cols]).float()\n            return images, labels\n        else:\n            return images\n\n    def __len__(self):\n        return len(self.df)\n\ntrain_ds = RSNADataset(subset=\"train\")\n\nimage_counts = [train_ds[i][0].shape for i in tqdm(range(len(train_ds)))]\ntrain_ds[0][0].shape\n</code></pre>",
      "rawMarkdown": "I want to build a torch 3d dataset class that will be used for pretty basic CNN models. But for some reason, it uses too much RAM. But i am only storing the shape of the tensors in a list.\n\n- And also If you have any other improvements on this i would love to hear them. \n\nThanks :)\n\n```python\n!pip install torchio\nfrom torchvision import transforms\nfrom tqdm import tqdm\nimport time\nfrom torch.utils.data import Dataset, DataLoader\nimport torch\nimport pandas as pd\nfrom glob import glob\nimport torchio\n\ndataset_path = \"../input/rsna-2022-cervical-spine-fracture-detection/\"\n\nclass RSNADataset(Dataset):\n    \n    def __init__(self, subset='train', transform=None):\n        self.transform = transform\n        self.df = pd.read_csv(dataset_path + subset + \".csv\")\n        self.train = subset == 'train'\n        \n        print('Glob started, here we go :(') # it took 260 seconds first time :/\n        start_time = time.time()\n        self.image_paths = glob(dataset_path + subset + '_images/**/**')\n        print(f'Glob took {time.time() - start_time:.2f} seconds')\n        \n        \n        self.target_cols = ['C1','C2','C3','C4','C5','C6','C7','patient_overall']\n        \n    def __getitem__(self, index):\n        \"\"\"\n        Prepare 1 batch of 3d images and labels\n        \"\"\"\n        row = self.df.iloc[index]\n        \n        image_paths = [i for i in self.image_paths if row['StudyInstanceUID'] in i]\n        \n        images = []\n        \n        for path in image_paths:\n            images.append(torchio.data.image.read_image(path)[0])\n\n        images = torch.stack(images)\n\n        # apply augmentation\n        if self.transform:\n            images = self.transform(images)\n\n        # get labels\n        if self.train:\n            labels = torch.tensor(row[self.target_cols]).float()\n            return images, labels\n        else:\n            return images\n\n    def __len__(self):\n        return len(self.df)\n\ntrain_ds = RSNADataset(subset=\"train\")\n\nimage_counts = [train_ds[i][0].shape for i in tqdm(range(len(train_ds)))]\ntrain_ds[0][0].shape\n```\n",
      "votes": 2
    }
  ],
  "comments": [
    {
      "id": 1982694,
      "author_name": "Aaditya Agnihotri",
      "author_url": "",
      "post_date": "2022-10-11T15:03:08.317000",
      "content": "<p>Try using garbage collector or some other ram-clearing library to clear your memory after returning the images. A lot of times what happens is the runtime runs out of RAM even by loading many images, whether or not you are saving them anywhere. I had a similar problem with the Mayo Clinic Competition from last week.</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1982694": "Try using garbage collector or some other ram-clearing library to clear your memory after returning the images. A lot of times what happens is the runtime runs out of RAM even by loading many images, whether or not you are saving them anywhere. I had a similar problem with the Mayo Clinic Competition from last week.",
    "1982183": "I want to build a torch 3d dataset class that will be used for pretty basic CNN models. But for some reason, it uses too much RAM. But i am only storing the shape of the tensors in a list.\n\n- And also If you have any other improvements on this i would love to hear them. \n\nThanks :)\n\n```python\n!pip install torchio\nfrom torchvision import transforms\nfrom tqdm import tqdm\nimport time\nfrom torch.utils.data import Dataset, DataLoader\nimport torch\nimport pandas as pd\nfrom glob import glob\nimport torchio\n\ndataset_path = \"../input/rsna-2022-cervical-spine-fracture-detection/\"\n\nclass RSNADataset(Dataset):\n    \n    def __init__(self, subset='train', transform=None):\n        self.transform = transform\n        self.df = pd.read_csv(dataset_path + subset + \".csv\")\n        self.train = subset == 'train'\n        \n        print('Glob started, here we go :(') # it took 260 seconds first time :/\n        start_time = time.time()\n        self.image_paths = glob(dataset_path + subset + '_images/**/**')\n        print(f'Glob took {time.time() - start_time:.2f} seconds')\n        \n        \n        self.target_cols = ['C1','C2','C3','C4','C5','C6','C7','patient_overall']\n        \n    def __getitem__(self, index):\n        \"\"\"\n        Prepare 1 batch of 3d images and labels\n        \"\"\"\n        row = self.df.iloc[index]\n        \n        image_paths = [i for i in self.image_paths if row['StudyInstanceUID'] in i]\n        \n        images = []\n        \n        for path in image_paths:\n            images.append(torchio.data.image.read_image(path)[0])\n\n        images = torch.stack(images)\n\n        # apply augmentation\n        if self.transform:\n            images = self.transform(images)\n\n        # get labels\n        if self.train:\n            labels = torch.tensor(row[self.target_cols]).float()\n            return images, labels\n        else:\n            return images\n\n    def __len__(self):\n        return len(self.df)\n\ntrain_ds = RSNADataset(subset=\"train\")\n\nimage_counts = [train_ds[i][0].shape for i in tqdm(range(len(train_ds)))]\ntrain_ds[0][0].shape\n```\n"
  }
}