{"cells":[{"metadata":{},"cell_type":"markdown","source":"# RSNA Intracranial Hemorrhage Detection"},{"metadata":{},"cell_type":"markdown","source":"This kernel is meant to do the pre-processing on the DICOM files. Firstly to read the DICOM metadata to extract all the information. Secondly to normalize data using equal binarization, zero mean and unit variance of images depending on the group where they belong (according to 'BitsStored' and 'PixelRepresentation')\n\n### Acknowledgment\nAcknowledgment to the following kernels which helped me built this:\n* [DON'T see like a radiologist! (fastai)](https://www.kaggle.com/jhoward/don-t-see-like-a-radiologist-fastai)\n* [Creating a metadata DataFrame (fastai)](https://www.kaggle.com/jhoward/creating-a-metadata-dataframe-fastai)\n* [Cleaning the data for rapid prototyping (fastai)](https://www.kaggle.com/jhoward/cleaning-the-data-for-rapid-prototyping-fastai)\n* [RSNA Intracranial: Simple DenseNet in Keras](https://www.kaggle.com/xhlulu/rsna-intracranial-simple-densenet-in-keras)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nfrom scipy import interpolate\nimport pandas as pd \nimport os\nimport shutil\nimport warnings\n\nimport pydicom\nimport cv2\n\nfrom tqdm import tqdm_notebook\n\n!pip install swifter\nimport swifter\nfrom joblib import Parallel, delayed\nimport multiprocessing\n\nfrom sklearn.model_selection import train_test_split\n\nfrom matplotlib import pyplot as plt\nimport matplotlib.patches as patches\nfrom imageio import imwrite\n%matplotlib inline ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('display.max_columns', None)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Paths of raw data\nBASE_PATH = '/kaggle/input/rsna-intracranial-hemorrhage-detection/'\nTRAIN_DIR = BASE_PATH + 'stage_1_train_images/'\nSUB_DIR = BASE_PATH + 'stage_1_test_images/'\n\n# Paths to save images after pre-processing and to be used for training\nTRAIN_PNG = '/kaggle/tmp/train/'\nTEST_PNG = '/kaggle/tmp/test/'\nSUB_PNG = '/kaggle/tmp/sub/'\n\n# Path to save CSV files corresponents to the images saved\nCSV_DIR = './csv/'\n\n# Classes to classify in the multi-label training\nCLASSES = ['any', 'epidural', 'intraparenchymal', 'intraventricular', 'subarachnoid', 'subdural']\n\n# As the training dataset is huge we could only train on a part of the whole training directory\nFRACTION_TRAINING = 0.2\n\n# Image size which Xception model was trained\nimg_size = (299, 299)\n\n# Minimum area that a training/validation image need to contain of bran\nUSE_MIN_AREA = False # Currently disabled\nMIN_AREA = 200*200 # From an image of 512*512 \n\nNUM_CORES = multiprocessing.cpu_count()\n\nSEED = 42\nnp.random.seed(seed=SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load CSV\nLoading CSV files and split 'ID' column to extract 'filename' and 'type'. In order to extract labels and set them with one binary column per label we used a pivot table."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def load_csv(path):\n    df = pd.read_csv(path)\n    df['filename'] = df['ID'].apply(lambda st: \"ID_\" + st.split('_')[1] + \".png\")\n    df['type'] = df['ID'].apply(lambda st: st.split('_')[2])\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_df = load_csv(BASE_PATH + 'stage_1_train.csv')\ndata_df.drop_duplicates(['filename','type'], inplace=True)\ndata_df = data_df.pivot('filename', 'type', 'Label').reset_index() # Extract Labels\nsub_df = load_csv(BASE_PATH + 'stage_1_sample_submission.csv')\nsub_df = pd.DataFrame(sub_df.filename.unique(), columns=['filename'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Data shape: {data_df.shape}\")\ndata_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Submission shape: {sub_df.shape}\")\nsub_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Extract metadata from DICOM files\nCode to transform metadata from DICOM files to DataFrame"},{"metadata":{"trusted":true},"cell_type":"code","source":"# There are 3 groups, but we will create a fourth for Others:\n# 1) Bits Stored 16bits\n# 2) Bits Stored 12bits - Pixel Representation 0\n# 3) Bits Stored 12bits - Pixel Representation 1\n# -1) Others (in case new data appears)\ndef _subgroup(res):\n    if res['BitsStored'] == 16:\n        res['SubGroup'] = 1\n    elif res['BitsStored'] == 12 and res['PixelRepresentation'] == 0:\n        res['SubGroup'] = 2\n    elif res['BitsStored'] == 12 and res['PixelRepresentation'] == 1:\n        res['SubGroup'] = 3\n    else:\n        res['SubGroup'] = -1\n\ndef _cast_dicom_special(x):\n    cls = type(x)\n    if not cls.__module__.startswith('pydicom'): return x\n    return cls.__base__(x)\n\n# Extract data from values\ndef _split_elem(res,k,v):\n    if not isinstance(v,pydicom.multival.MultiValue): return\n    res[f'Multi{k}'] = 1\n    for i,o in enumerate(v): res[f'{k}{\"\" if i==0 else i}']=o\n\n# Transform DICOM data to dictionary\ndef as_dict(dcm, px_summ=True):\n    pxdata = (0x7fe0,0x0010)\n    vals = [dcm[o] for o in dcm.keys() if o != pxdata]\n    its = [(v.keyword,v.value) for v in vals]\n    res = dict(its)\n    res['fname'] = dcm.filename\n    for k,v in its: _split_elem(res,k,v)\n    _subgroup(res)\n    if not px_summ: return res\n    stats = 'min','max','mean','std'\n    try:\n        pxs = dcm.pixel_array\n        for f in stats: res['img_'+f] = getattr(pxs,f)()\n    except Exception as e:\n        for f in stats: res['img_'+f] = 0\n        print(res,e)\n    for k in res: res[k] = _cast_dicom_special(res[k])\n    return res\n\n# Function used in apply function to fill row with DICOM data\ndef fill_row(row, px_summ = True):\n    row_dict = as_dict(pydicom.dcmread(row.paths), px_summ)\n    for key in row_dict:\n        row[key] = row_dict[key]\n\ndef get_all_metadata(load_dir, px_summ = True, n_sample = None, dcm_dir = None):\n    if dcm_dir is None:\n        dcm_dir = load_dir\n    filenames = os.listdir(load_dir)\n    if not n_sample is None:\n        filenames = np.random.choice(filenames, size = n_sample, replace = False)\n    dcm_paths = [dcm_dir + filename[:-3] + 'dcm' for filename in filenames]\n    filenames = [filename[:-3] + 'png' for filename in filenames.copy()]\n    sample_dict = as_dict(pydicom.dcmread(dcm_paths[0]), px_summ = True)\n    sample_dict['paths'] = dcm_paths[0]\n    columns = list(sample_dict.keys())\n    metadata_df = pd.DataFrame({'paths' : dcm_paths},columns = columns, index = filenames)\n    metadata_df.swifter.apply(fill_row, axis=1, px_summ = px_summ)\n    return metadata_df\n\ndef get_file_metadata(load_dir, filename, px_summ = True):\n    dcm_path = load_dir + filename[:-3] + 'dcm'\n    filename = filename[:-3] + 'png'\n    sample_dict = as_dict(pydicom.dcmread(dcm_path), px_summ)\n    sample_dict['paths'] = dcm_path\n    columns = list(sample_dict.keys())\n    metadata_df = pd.DataFrame(sample_dict, columns = columns, index = [filename])\n    return metadata_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# To big to be stored all in-memory (we only could stored 50.000 files)\n# train_meta = get_all_metadata(TRAIN_DIR, px_summ = False, n_sample = 50000)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Pre-processing\nThe pre-processing applied here consists on several steps:\n\n1) DICOM images should be rescaled according to its 'Rescale Intercept' and 'Rescale Slope'\n\n2) A process try to find the brain part and crop the image to only show this part (idea from [Cleaning the data for rapid prototyping (fastai)](https://www.kaggle.com/jhoward/cleaning-the-data-for-rapid-prototyping-fastai)). Besides only images with a minimum of brain are will be used for training/validation\n\n3) As pixels values are unequally distributed, they are remapped using an interpolation from equally distributed bins (more detail on [DON'T see like a radiologist! (fastai)](https://www.kaggle.com/jhoward/don-t-see-like-a-radiologist-fastai))\n\n4) Zero mean and unit variance normalization\n\n5) Resize images to the specified size\n\nSteps 3 and 4 are using bins, means and stds from samples of the same group. Hence, the final PNG images will be remapped using a particular distributed bins depending on the subgroup (extract from DICOM metadata) and using a specific mean and std also depending on the subgroup."},{"metadata":{"trusted":true},"cell_type":"code","source":"# From https://radiopaedia.org/articles/windowing-ct\ndicom_windows = {\n    'brain' : (80,40),\n    'subdural':(200,80),\n    'stroke':(8,32),\n    'brain_bone':(2800,600),\n    'brain_soft':(375,40),\n    'lungs':(1500,-600),\n    'mediastinum':(350,50),\n    'abdomen_soft':(400,50),\n    'liver':(150,30),\n    'spine_soft':(250,50),\n    'spine_bone':(1800,400)\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_brain(image, n_top_areas = 5, max_scale_diff = 3, plot = False):\n    image = normalize_img(image, use_min_max = True)\n    if (image.max() - image.min()) == 0 or np.isnan(image.max()) or np.isnan(image.min()):\n        raise ValueError('Empty image')\n    gray = np.uint8(image * 255)\n    blur = cv2.blur(gray, (5, 5)) # blur the image\n    # Detect edges using Canny\n    #canny_output = cv2.Canny(blur, threshold, threshold * 2)\n    # Find contours\n    contours, _ = cv2.findContours(blur, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)\n\n    # Cycle through contours and add area to array\n    areas = []\n    for c in contours:\n        areas.append(cv2.contourArea(c))\n\n    # Sort array of areas by size\n    sorted_areas = sorted(zip(areas, contours), key=lambda x: x[0], reverse=True)\n    biggest_area = sorted_areas[0][0]\n\n    # Approximate contours to polygons + get bounding rects and circles\n    contours_poly = []\n    boundRect = []\n    min_dist_to_center = np.inf\n    best_contour_idx = 0\n    for i, c in enumerate(sorted_areas):\n        # Only treat contours which are in top 5 and less than 'max_scale_diff' times smaller than the biggest one\n        if c[0] > 0 and i < n_top_areas and biggest_area/c[0] < max_scale_diff:\n            contour_poly = cv2.approxPolyDP(c[1], 3, True)\n            contours_poly.append(contour_poly)\n            boundRect.append(cv2.boundingRect(contour_poly))\n            center, _ = cv2.minEnclosingCircle(contour_poly)\n\n            # Calculate distance from contour center to center of image\n            dist = (gray.shape[0]//2 - center[0])**2 + (gray.shape[1]//2 - center[1])**2\n            if min_dist_to_center > dist:\n                best_contour_idx = i\n                min_dist_to_center = dist\n        else:\n            break\n\n    # Get boundaries of the Rectangle which includes the contour\n    x,y,w,h = boundRect[best_contour_idx]\n    # Crop the image\n    cropped = image[y:y+h,x:x+w]\n    # Pad needed pixels\n    final_image = pad_square(cropped)\n    \n    # Show three images (original, cropped, final)\n    if plot:\n        fig=plt.figure(figsize  = (10,30))    \n        ax = fig.add_subplot(1, 3, 1)\n        plt.imshow(image)\n        ax.add_patch(patches.Rectangle(\n            (x, y),\n            w,\n            h,\n            fill=False      # remove background\n         )) \n        fig.add_subplot(1, 3, 2)\n        plt.imshow(cropped)\n        fig.add_subplot(1, 3, 3)\n        plt.imshow(final_image)\n        plt.show()\n    return final_image, sorted_areas[best_contour_idx]\n\ndef pad_square(x):\n    r,c = x.shape\n    d = (c-r)/2\n    pl,pr,pt,pb = 0,0,0,0\n    if d>0: pt,pd = int(np.floor( d)),int(np.ceil( d))        \n    else:   pl,pr = int(np.floor(-d)),int(np.ceil(-d))\n    return np.pad(x, ((pt,pb),(pl,pr)), 'minimum')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_first_of_dicom_field_as_int(x):\n    #get x[0] as in int is x is a 'pydicom.multival.MultiValue', otherwise get int(x)\n    if type(x) == pydicom.multival.MultiValue:\n        return int(x[0])\n    else:\n        return int(x)\n\ndef get_windowing(data):\n    dicom_fields = [data[('0028','1050')].value, #window center\n                    data[('0028','1051')].value, #window width\n                    data[('0028','1052')].value, #intercept\n                    data[('0028','1053')].value] #slope\n    return [get_first_of_dicom_field_as_int(x) for x in dicom_fields]\n\ndef get_subgroups(data):\n    # There are 3 groups, but we will create a fourth for Others:\n    # 1) Bits Stored 16bits\n    # 2) Bits Stored 12bits - Pixel Representation 0\n    # 3) Bits Stored 12bits - Pixel Representation 1\n    # -1) Others (in case new data appears)\n    dicom_fields = [data[('0028', '0101')].value, #Bits Stored\n                    data[('0028', '0103')].value] #Pixel Representation\n    dicom_values = [get_first_of_dicom_field_as_int(x) for x in dicom_fields]\n    if dicom_values[0] == 16:\n        return 1\n    elif dicom_values[0] == 12 and dicom_values[1] == 0:\n        return 2\n    elif dicom_values[0] == 12 and dicom_values[1] == 1:\n        return 3\n    else:\n        return -1\n\n# According to https://www.kaggle.com/jhoward/cleaning-the-data-for-rapid-prototyping-fastai\ndef correct_dcm(dcm):\n    x = dcm.pixel_array + 1000\n    px_mode = 4096\n    x[x>=px_mode] = x[x>=px_mode] - px_mode\n    dcm.PixelData = x.tobytes()\n    dcm.RescaleIntercept = -1000\n    return dcm.pixel_array, dcm.RescaleIntercept\n\ndef get_freqhist_bins(dcm_img, n_bins = 100):\n    imsd = np.sort(dcm_img.reshape(-1))\n    t = np.concatenate([[0.001],\n                       np.arange(n_bins).astype(np.float64)/n_bins+(1/2/n_bins),\n                       [0.999]])\n    t = (len(imsd)*t).astype(np.int64)\n    return np.unique(imsd[t])\n\ndef get_dcm_img(path, window_type = 'brain'):\n    # Read and scale of DICOM images according to its metadata\n    dcm = pydicom.dcmread(path)\n    window_center, window_width, intercept, slope = get_windowing(dcm)\n    group = get_subgroups(dcm)   \n    \n    if group == 2 and (int(intercept) > -100):\n        dcm_img, intercept = correct_dcm(dcm)\n    \n    dcm_img = dcm.pixel_array.astype(np.float64) \n    dcm_img = dcm_img * slope + intercept\n    \n    min_px = dicom_windows[window_type][1] - dicom_windows[window_type][0]//2\n    max_px = dicom_windows[window_type][1] + dicom_windows[window_type][0]//2\n    if min_px is not None: dcm_img[dcm_img<min_px] = min_px\n    if max_px is not None: dcm_img[dcm_img>max_px] = max_px\n    if (dcm_img.max() - dcm_img.min()) == 0:\n        dcm_img[:, :] = 0\n        warnings.warn('Empty image from path: ' + path, UserWarning)\n    \n    return dcm_img, group\n\ndef interpolate_img(dcm_img, bins = None, n_bins = 100):\n    # Equal distribution of intensity\n    if bins is None: \n        bins = get_freqhist_bins(dcm_img, n_bins)\n    \n    return np.clip(interpolate.interp1d(bins, np.linspace(0., 1., len(bins)), fill_value=\"extrapolate\")(dcm_img.flatten()).reshape(dcm_img.shape), 0., 1.)\n\ndef normalize_img(dcm_img, mean = None, std = None, use_min_max = False):\n    # Normalization to zero mean and unit variance\n    if use_min_max:\n        if (dcm_img.max() - dcm_img.min()) != 0:\n            return (dcm_img - dcm_img.min()) / (dcm_img.max() - dcm_img.min())\n        else:\n            return dcm_img\n    else:\n        if mean is None: \n            mean = dcm_img.mean()\n\n        if std is None: \n            std = dcm_img.std()\n        return (dcm_img - mean) / std\n    \n\ndef preprocess_dicom(path, x, y, bins = None, n_bins = 100, mean = None, std = None, use_min_max = False, remove_empty = False): \n    dcm_img, group = get_dcm_img(path)\n    \n    # Crop image to show only the brain part (only posible if the image is not empyt)\n    try:\n        isEmpty = False\n        dcm_img, area = crop_brain(dcm_img)\n    except ValueError as e:\n        isEmpty = True\n        area = 0\n        print(\"DICOM image from \", path, \" is not treated because gave the following error: \", e)\n    finally:\n        if isEmpty and remove_empty:\n            return None, None\n        else:\n            # If distributed by groups (different than -1) then use only the values of the group\n            if group != -1:\n                if type(bins) == dict:\n                    bins = bins[group]\n                if type(mean) == dict:\n                    mean = mean[group]\n                if type(std) == dict:\n                    std = std[group]\n\n            dcm_img = interpolate_img(dcm_img, bins, n_bins)\n            dcm_img = normalize_img(dcm_img, mean, std, use_min_max)\n\n            # Rescale to the defined image size\n            if dcm_img.shape != (x, y):\n                dcm_img = cv2.resize(dcm_img, (x, y), interpolation=cv2.INTER_NEAREST)\n\n            return dcm_img, area","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Comparison of images using crop process"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"n_samples = 10\nfilenames = data_df.sample(n_samples)['filename']\npng_paths = [TRAIN_DIR + filename for filename in filenames]\nfor path in png_paths:\n    path = path.replace('.png', '.dcm')\n    image, _ = get_dcm_img(path)\n    print(path)\n    _,_ = crop_brain(image, plot = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Example of an image before and after the treatment"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"filename = 'ID_9d9cc6b01.dcm'\nsample_path = TRAIN_DIR + filename\n\nprint('Before preprocessing')\nimg, _ = get_dcm_img(sample_path)\npx = img.flatten()\nplt.hist(px, bins=40)\nplt.title('Histogram pixel values')\nplt.show()\nplt.imshow(img)\nplt.show()\n\nprint('After preprocessing')\nimg, area = preprocess_dicom(sample_path, img_size[0], img_size[1], use_min_max = True)\npx = img.flatten()\nplt.hist(px, bins=40)\nplt.title('Histogram pixel values')\nplt.show()\nplt.imshow(img)\nplt.show()\n\n# Labels of example\ndisplay(data_df[data_df['filename'] == filename[:-3] + 'png'])\n\n# Metadata from example\nget_file_metadata(TRAIN_DIR, filename)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sampling groups\n\nIn order to stablish a proper normalization process, means for bins, mean and std have to be choose. For this it has been split by groups according to the DICOM metadata."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Samples from each group are extracted by trying to find an specific number of samples per group\n# This is done due to memory limitations of have all training metadata in a DataFrame\ndef sample_groups(load_dir, samples_per_group = 5, max_trys = 1000):\n    filenames = os.listdir(load_dir)\n    filenames_groups = {1 : [], 2 : [], 3 : []}\n    for group in tqdm_notebook([1,2,3], desc = 'Group sample'):\n        count_samples = 0\n        for _ in tqdm_notebook(range(max_trys), desc = 'Try'):\n            filename = np.random.choice(filenames, size = 1, replace = False)[0]\n            sample_group = get_subgroups(pydicom.dcmread(load_dir + filename))\n            if sample_group == group:\n                filenames_groups[group].append(load_dir + filename)\n                count_samples += 1\n                if count_samples >= samples_per_group:\n                    break\n    return filenames_groups\n\n# For each group it is computed:\n# Firstly mean of equally distributed bin\n# Secondly mean of mean pixels values and mean of std pixel values using the previous bin mean\ndef sample_bins_mean_std(load_dir, samples_per_group = 5, max_trys = 1000, n_bins = 100):\n    bins_mean = {}\n    mean = {}\n    std = {}\n    groups_paths = sample_groups(TRAIN_DIR, samples_per_group = samples_per_group, max_trys = max_trys)\n    for group in [1,2,3]:    \n        # Do not proceed if there is no images\n        if len(groups_paths[group]) == 0:\n            bins_mean[group] = []\n            mean[group] = np.nan\n            std[group] = np.nan\n            continue\n                    \n        filenames = groups_paths[group]\n    \n        dcm_img_array = []\n        for filename in tqdm_notebook(filenames, desc = 'Calc bins'):\n            dcm_img, _ = get_dcm_img(filename)\n            dcm_img_array.append(dcm_img)\n            #bins_array.append(get_freqhist_bins(dcm_img, n_bins))\n\n        #bins_mean[group] = np.array(bins_array).mean(axis = 0)\n        bins_mean[group] = get_freqhist_bins(np.array(dcm_img_array).reshape(-1), n_bins)\n    \n        dcm_img_array = []\n        for filename in tqdm_notebook(filenames, desc = 'Calc mean & std'):\n            dcm_img, _ = get_dcm_img(filename)\n            dcm_img_array.append(interpolate_img(dcm_img, bins_mean[group], n_bins))\n    \n        mean[group] = np.array(dcm_img_array).flatten().mean()\n        std[group] = np.array(dcm_img_array).flatten().std()\n    \n    return bins_mean, mean, std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bins_mean, mean, std = sample_bins_mean_std(TRAIN_DIR, samples_per_group = 5, max_trys = 5000, n_bins = 100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train/Validation split\nAs a mesure to limit size of training due to memory limitations, it has been set a FRACTION_TRAINING value to only use a part of the training.\n\nDue to the imbalance on between classes here it is needed to use a stratified split."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df, _ = train_test_split(data_df, train_size = FRACTION_TRAINING, stratify = data_df[CLASSES], random_state = SEED)\ntrain_df, test_df = train_test_split(train_df, test_size = 0.1, stratify = train_df[CLASSES], random_state = SEED)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"DICOM files are here treat, to finally be saved as PNG for in the future be used in training"},{"metadata":{"trusted":true},"cell_type":"code","source":"def save_and_resize(load_dir, filenames, img_size, bins = None, mean = None, std = None, use_min_max = False, remove_small = False, remove_empty = False, save_dir = '', save_on_zip = True, zip_name = 'output'):\n    if not os.path.exists(save_dir):\n        os.makedirs(save_dir)    \n    \n    png_paths = [load_dir + filename for filename in filenames]\n    #for png_path in tqdm_notebook(png_paths):\n    #    print(png_path)\n    #    process_save_and_resize(png_path, img_size, bins, mean, std, use_min_max, remove_small, remove_empty, save_dir)\n    Parallel(n_jobs=NUM_CORES)(delayed(process_save_and_resize)(png_path, img_size, bins, mean, std, use_min_max, remove_small, remove_empty, save_dir) for png_path in tqdm_notebook(png_paths))\n    \n    # Save images in ZIP file\n    if save_on_zip:\n        shutil.make_archive(zip_name, 'zip', save_dir)\n    \ndef process_save_and_resize(png_path, img_size, bins, mean, std, use_min_max, remove_small, remove_empty, save_dir):\n    path = png_path.replace('.png', '.dcm')\n    filename = png_path[len(png_path) - png_path[::-1].find(\"/\") : ]\n    new_path = save_dir + filename\n    img, area = preprocess_dicom(path, img_size[0], img_size[1], bins = bins, mean = mean, std = std, use_min_max = use_min_max)\n    # Do not save image in case that:\n    # 1 - it is empty and want to remove empty\n    # 2 - It has an small brain area and want to remove these images\n    if (remove_empty and img is None) or (remove_small and USE_MIN_AREA and area < MIN_AREA):\n        pass # Do not save image\n    else:\n        #image.imsave(new_path, img)\n        imwrite(new_path, img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#save_and_resize(TRAIN_DIR, train_df['filename'], img_size, bins = bins_mean, mean = mean, std = std, save_dir = TRAIN_PNG)\nsave_and_resize(TRAIN_DIR, train_df['filename'], img_size, bins = bins_mean, use_min_max = True, remove_small = True, remove_empty = True, save_dir = TRAIN_PNG, save_on_zip = True, zip_name = 'train')\n\n#save_and_resize(TRAIN_DIR, test_df['filename'], img_size, bins = bins_mean, mean = mean, std = std, save_dir = TEST_PNG)\nsave_and_resize(TRAIN_DIR, test_df['filename'], img_size, bins = bins_mean, use_min_max = True, remove_small = True, remove_empty = True, save_dir = TEST_PNG, save_on_zip = True, zip_name = 'test')\n\n#save_and_resize(SUB_DIR, sub_df['filename'], img_size, bins = bins_mean, mean = mean, std = std, save_dir = SUB_PNG)\nsave_and_resize(SUB_DIR, sub_df['filename'], img_size, bins = bins_mean, use_min_max = True, remove_small = False, remove_empty = False, save_dir = SUB_PNG, save_on_zip = True, zip_name = 'sub')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some images are not written down to the PNG directory due to be empty images or with not enough relevant data. However, we have to make sure that the DataFrames don't contain these images any more."},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_files(df, png_dir):\n    df['check_in_dir'] = df['filename'].apply(lambda filename : os.path.exists(png_dir + filename))\n    return df[df['check_in_dir'] == True].drop('check_in_dir', axis = 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As this kernel is used by other kernels, csv files are created to have listed the data used here."},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists(CSV_DIR):\n    os.makedirs(CSV_DIR)\n\ntrain_df = check_files(train_df, TRAIN_PNG)\ntest_df = check_files(test_df, TEST_PNG)\nsub_df = check_files(sub_df, SUB_PNG)\n    \ntrain_df.to_csv(CSV_DIR + 'train_df.csv', index = None, header=True)\ntest_df.to_csv(CSV_DIR + 'test_df.csv', index = None, header=True)\nsub_df.to_csv(CSV_DIR + 'sub_df.csv', index = None, header=True)\n\ntrain_meta_df = get_all_metadata(TRAIN_PNG, dcm_dir = TRAIN_DIR)\ntrain_meta_df.to_csv(CSV_DIR + 'train_meta_df.csv', header=True)\n\ntest_meta_df = get_all_metadata(TEST_PNG, dcm_dir = TRAIN_DIR)\ntest_meta_df.to_csv(CSV_DIR + 'test_meta_df.csv', header=True)\n\nsub_meta_df = get_all_metadata(SUB_PNG, dcm_dir = SUB_DIR)\nsub_meta_df.to_csv(CSV_DIR + 'sub_meta_df.csv', header=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"shutil.rmtree(TRAIN_PNG)\nshutil.rmtree(TEST_PNG)\nshutil.rmtree(SUB_PNG)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}