{"cells":[{"metadata":{},"cell_type":"markdown","source":"Dataset URL: https://www.kaggle.com/guiferviz/rsna_stage1_png_128\n\n# Preparing dataset\n\nThe DICOM format is so cool, but I prefer normal images :)\n\nWith 156GB (compressed) it is very difficult to work with the resources of the vast majority of the mortals.\nThis notebook shows you how to scale down all the images and create a new dataset easier to deal with.\nEven with the best computing resources, I don't think it's necessary to use the original size to get good accuracy.\n\nIMPORTANT: In this notebook runs in a subset of the data, so don't use the generated output. That is because the Kaggle notebook runs out of space if you use all the examples. If you want to run this by yourself you should run it in a different machine or opening the next notebook https://colab.research.google.com/gist/guiferviz/50912a681776d5afe012b1a9259bd637/resize-dataset.ipynb in Google Colab. If you try to unzip the data in Google Colab you will also run out of space, so I've used the amazing tool *fuse-zip* to mount the zip and work with the files in it without extracting any of those.\n\nSome code taken from:\n* https://www.kaggle.com/omission/eda-view-dicom-images-with-correct-windowing\n* https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/109649#latest-631701\n\n# Constants"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Desired output size.\nRESIZED_WIDTH, RESIZED_HEIGHT = 128, 128\n\nOUTPUT_FORMAT = \"png\"\n\nOUTPUT_DIR = \"output\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import glob\n\nimport joblib\n\nimport numpy as np\n\nimport PIL\n\nimport pydicom\n\nimport tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Get images paths"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data_dir = \"../input/rsna-intracranial-hemorrhage-detection\"\n!ls {data_dir}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dir = \"stage_1_train_images\"\ntrain_paths = glob.glob(f\"{data_dir}/{train_dir}/*.dcm\")\ntest_dir = \"stage_1_test_images\"\ntest_paths = glob.glob(f\"{data_dir}/{test_dir}/*.dcm\")\nlen(train_paths), len(test_paths)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocess all data\n\nFirst declare a bunch of useful functions."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_first_of_dicom_field_as_int(x):\n    if type(x) == pydicom.multival.MultiValue:\n        return int(x[0])\n    return int(x)\n\ndef get_id(img_dicom):\n    return str(img_dicom.SOPInstanceUID)\n\ndef get_metadata_from_dicom(img_dicom):\n    metadata = {\n        \"window_center\": img_dicom.WindowCenter,\n        \"window_width\": img_dicom.WindowWidth,\n        \"intercept\": img_dicom.RescaleIntercept,\n        \"slope\": img_dicom.RescaleSlope,\n    }\n    return {k: get_first_of_dicom_field_as_int(v) for k, v in metadata.items()}\n\ndef window_image(img, window_center, window_width, intercept, slope):\n    img = img * slope + intercept\n    img_min = window_center - window_width // 2\n    img_max = window_center + window_width // 2\n    img[img < img_min] = img_min\n    img[img > img_max] = img_max\n    return img \n\ndef resize(img, new_w, new_h):\n    img = PIL.Image.fromarray(img.astype(np.int8), mode=\"L\")\n    return img.resize((new_w, new_h), resample=PIL.Image.BICUBIC)\n\ndef save_img(img_pil, subfolder, name):\n    img_pil.save(f\"{OUTPUT_DIR}/{subfolder}/{name}.{OUTPUT_FORMAT}\")\n\ndef normalize_minmax(img):\n    mi, ma = img.min(), img.max()\n    return (img - mi) / (ma - mi)\n\ndef prepare_image(img_path):\n    img_dicom = pydicom.read_file(img_path)\n    img_id = get_id(img_dicom)\n    metadata = get_metadata_from_dicom(img_dicom)\n    img = window_image(img_dicom.pixel_array, **metadata)\n    img = normalize_minmax(img) * 255\n    img_pil = resize(img, RESIZED_WIDTH, RESIZED_HEIGHT)\n    return img_id, img_pil\n\ndef prepare_and_save(img_path, subfolder):\n    try:\n        l.error(\"loading eso\")\n        img_id, img_pil = prepare_image(img_path)\n        save_img(img_pil, subfolder, img_id)\n    except KeyboardInterrupt:\n        # Rais interrupt exception so we can stop the cell execution\n        # without shutting down the kernel.\n        raise\n    except:\n        l.error(f\"Error processing the image: {img_path}\")\n\ndef prepare_images(imgs_path, subfolder):\n    for i in tqdm.tqdm(imgs_path):\n        prepare_and_save(i, subfolder)\nimport logging as l\ndef prepare_images_njobs(img_paths, subfolder, n_jobs=-1):\n    joblib.Parallel(n_jobs=n_jobs)(joblib.delayed(prepare_and_save)(i, subfolder) for i in tqdm.tqdm(img_paths))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir -p {OUTPUT_DIR}/{train_dir}\n!mkdir -p {OUTPUT_DIR}/{test_dir}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# Running on the first 100 files of train and set!!!\nprepare_images_njobs(train_paths[:100], train_dir)\nprepare_images_njobs(test_paths[:100], test_dir)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load converted images\n\nLet's test that everything is ok!"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_output_path = glob.glob(f\"{OUTPUT_DIR}/{train_dir}/*\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_path = train_output_path[0]\nPIL.Image.open(img_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Comments and future work\n\nKeep in mind that you may want to normalize the downsampled images before feeding them into a neural network, the values are between 0 and 255.\n\nFinally, a series of ideas/future work/open questions:\n* Maybe we can let the algorithm to optimize the window size and width.\n* Create differents datasets with different sizes and try to find the best training time and memory footprint vs accuracy.\n* Try to crop the images. Black margins are too big. Not sure if this change of scale can affect the algorithm. If it does not affect it will allow to use smaller images with the same details.\n* Can we use '(0020, 0032) Image Position (Patient)' and '(0020, 0037) Image Orientation (Patient)' to rotate and crop the images? For example, the image with \"ID_c03cdcb55\" has a big rotation."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}