{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Writing TFRecords with DICOM Metadata\n \nThis Notebook creates Tensorflow Records for the Pulmonary Embolism competition.\n\nIt is heavily derived from [https://www.kaggle.com/cdeotte/how-to-create-tfrecords] Courtesy of Chris Deotte.\n\nPlease let me know if there are sections of the Notebook where additional credit should be given.\n\n\nTFRecords can be used by CPU, GPU and TPU. To use them with a TPU I usually make them public. There are ways to use them when they are private, but there is some configuration involved that I haven't tackled.\n\nThere are two ways to get these TFRecords into a dataset:\n\n1. If you run this notebook in interactive mode, download the output and then upload to a dataset.\n\n2. If you run in Commit mode, you can send the output files directly to a dataset. \n\n\nThis notebook stores Images, targets and metadata from the DICOM files in the TFRecords. They can be used in models that combine Image and Metadata into one model.\n\nFirst, we import our typical libraries (probably more than I need for this notebook).\n\nPaths point to the train or test dataset and a location to store the TFRec files. They might take some adjustment if you are running outside of Notebooks. There is a flag that should allow you to run them for both train and test data.\n\nDue to the size of the data, you can specify a start record and a size of the extract. Note that this is based on a sorted order (StudyInstanceUID and SOPInstanceUID), so as not to rely on the order of the train/test file or the directories.\n\nTo break things up, we specify a start position in the train file list and a number of files to get.\n\n10,000 images result in about 250 MB. So I estimate 45 GB of data.\n\nReasonable block sizes would be between 50,000 and 100,000 images. 100,000 might get close to 9 hours of processing time.\n\nProcessing in notebooks will hit disk space and time limitations.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np, pandas as pd, os\nimport re, math, random, csv, cv2\nimport pydicom\nimport tensorflow as tf\nfrom skimage.transform import resize\n\n!conda install -y -c conda-forge gdcm\n\ntrain_or_test = 'train'\n\nmypath = '../input/rsna-str-pulmonary-embolism-detection'\nimages_folder =    mypath+'/'+train_or_test\ntfrec_folder = ''\n\ncsv_file = mypath + '/' + train_or_test + '.csv'\n\n# change this prefix _a_ -> _b_, etc for each time you run this routine\nprefix = train_or_test + '_tfrec_version1_a_-'\n\n# for processing, break the Train data into smaller chunks.\n\ndf_start = 100000\n\n#change to 50000 for typical processing. df_size = 1000 just for testing\ndf_size = 1000\n\nmetadata_file = 'metadata_' + train_or_test + '_' + str(df_start) + '.csv'\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# First we will extract the DICOM Metadata.\n\nThis routine reads a DICOM directory and gets the Metadata from all the images.\n\nI use try/except blocks since there is no guarantee that a DICOM Tag exists in every file. Pydicom also lets you explicitly check for a tag before reading it.\n\nI provide default values if the tag doesn't exist.\n\nI write these out to a file, which can be Downloaded. I'll use it as we create TFRecords.\n\nI break out Tags that contain arrays into separate elements. For some, I just take the first element."},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_DICOM_attributes(folder):\n    images = list(os.listdir(os.path.join(folder)))\n    for image in images:\n        image_name = image.split(\".\")[0]\n        dicom_file_path = os.path.join(folder,image)\n        ds = pydicom.read_file(dicom_file_path)\n        myStudyInstanceUID = ds.StudyInstanceUID.strip()\n        mySeriesInstanceUID = ds.SeriesInstanceUID.strip()\n        mySOPInstanceUID = ds.SOPInstanceUID.strip()\n        slicethickness = ds.SliceThickness\n        rows = ds.Rows\n        columns = ds.Columns\n        pixelspacing0 = ds.PixelSpacing[0]\n        pixelspacing1 = ds.PixelSpacing[1]\n        wc = ds.WindowCenter\n        if hasattr(wc, \"__len__\"):\n            wc = wc[0]\n        ww = ds.WindowWidth\n        if hasattr(ww, \"__len__\"):\n            ww = ww[0]\n        ri = ds.RescaleIntercept\n        rs = ds.RescaleSlope\n        try:\n            pp = ds.PatientPosition\n        except:\n            pp = 'FFS'\n        try:\n            ipp0 = ds.ImagePositionPatient[0]\n        except:\n            ipp0 = 0\n        try:\n            ipp1 = ds.ImagePositionPatient[1]\n        except:\n            ipp1 = 0\n        try:\n            ipp2 = ds.ImagePositionPatient[2]\n        except:\n            ipp2 = 0\n        try:\n            io0 = ds.ImageOrientationPatient[0]\n        except:\n            io0 = 0\n        try:\n            io1 = ds.ImageOrientationPatient[1]\n        except:\n            io1 = 0\n        try:\n            io2 = ds.ImageOrientationPatient[2]\n        except:\n            io2 = 0\n        try:\n            io3 = ds.ImageOrientationPatient[3]\n        except:\n            io3 = 0\n        try:\n            io4 = ds.ImageOrientationPatient[4]\n        except:\n            io4 = 0\n        try:\n            io5 = ds.ImageOrientationPatient[5]\n        except:\n            io5 = 0\n        try:\n            inum = ds.InstanceNumber\n        except:\n            inum = 0\n        try:\n            kvp = ds.KVP\n        except:\n            kvp = 0\n        try:\n            tc = ds.XRayTubeCurrent\n        except:\n            tc = 0\n        try:\n            exposure = ds.Exposure\n        except:\n            exposure = 0\n\n        mystring= myStudyInstanceUID + ',' + \\\n            mySeriesInstanceUID + ',' + \\\n            mySOPInstanceUID + ',' + \\\n            str(slicethickness) + ',' + \\\n            str(pixelspacing0) + ',' + \\\n            str(pixelspacing1) + ',' + \\\n            str(wc) + ',' + \\\n            str(ww) + ',' + \\\n            str(ri) + ',' + \\\n            str(rs) + ',' + \\\n            str(rows) + ',' + \\\n            str(columns) + ',' + \\\n            str(pp) + ',' + \\\n            str(ipp0) + ',' + \\\n            str(ipp1) + ',' + \\\n            str(ipp2) + ',' + \\\n            str(io0) + ',' + \\\n            str(io1) + ',' + \\\n            str(io2) + ',' + \\\n            str(io3) + ',' + \\\n            str(io4) + ',' + \\\n            str(io5) + ',' + \\\n            str(inum) + ',' + \\\n            str(kvp) + ',' + \\\n            str(tc) + ',' + \\\n            str(exposure) + '\\n'\n        file1.write(mystring)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We sort the csv file and images directory list so they are in the same order.\n\nThen we look through the patients and add their DICOM attributes to a metadata file. I read through the metadata first so that I can do post-processing on the metadata within each patient before I write the TFRecords (not present in this notebook). In practice I captured all the metadata once and just reference a file of the full metadata. In this stand-alone notebook, I build it on the fly. Also, for the private test data, you will need to build it on the fly anyway.\n\nIt would be more efficient to combine those two steps and cache the DICOM data, calculate on the patient level and then write the TFRecords."},{"metadata":{"trusted":true},"cell_type":"code","source":"file1 = open(metadata_file,'w')\nfile1.write('StudyInstanceUID,SeriesInstanceUID,SOPInstanceUID,dcm_slice_thickness,dcm_pixel_spacing0,dcm_pixel_spacing1,dcm_window_center,dcm_window_width,dcm_rescale_intercept,dcm_rescale_slope,dcm_rows,dcm_columns,dcm_patient_position,dcm_image_position_patient0,dcm_image_position_patient1,dcm_image_position_patient2,dcm_image_orientation0,dcm_image_orientation1,dcm_image_orientation2,dcm_image_orientation3,dcm_image_orientation4,dcm_image_orientation5,dcm_instance_number,dcm_kvp,dcm_xray_tube_current,dcm_exposure\\n')\n\n#mydirs = os.listdir(images_folder)\n#mydirs.sort()\n\nprint('only reading limited records', df_size)\ndf = pd.read_csv(csv_file)\ndf = df.sort_values(by=['StudyInstanceUID','SOPInstanceUID'])\ndf = df[df_start:(df_start+df_size)].reset_index(drop=True)\n\ndf_temp = df\ndf_patients = df_temp['StudyInstanceUID'].unique()\npatient_count = len(df_patients)\nprint('only reading limited patients', patient_count)\n\ncount = 0\n#for mydir in mydirs[0:patient_count]:\nfor mydir in df_patients:\n    SeriesUID = os.listdir(images_folder+'/'+mydir)\n    extract_DICOM_attributes(images_folder+'/'+mydir+'/'+SeriesUID[0])\n    count = count + 1\n    if count//100 == count/100:\n        print('patient count processed',count)\nfile1.close()\n\nmd = pd.read_csv(metadata_file)\ndf_count = len(md)\nprint('image count (adjusted to get full exams)',df_count)\nprint('done')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we include standard Tensorflow routines for different datatypes. I think this is straight from the Tensorflow examples."},{"metadata":{"trusted":true},"cell_type":"code","source":"def _bytes_feature(value):\n  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n  if isinstance(value, type(tf.constant(0))):\n    value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n  \"\"\"Returns a float_list from a float / double.\"\"\"\n  return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n  \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we have a custom routine that lists all the variables that we want to store in the Tensorflow records.\n\nNote that the first variable is the image itself. The rest are from the train.csv file or the DICOM files."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"def serialize_example(feature0, feature1, feature2, feature3, feature4, feature5, feature6,\\\n    feature7, feature8, feature9, feature10,\\\n    feature11, feature12, feature13, feature14, feature15, feature16, feature17, \\\n    feature18, feature19, feature20, feature21, feature22, feature23, feature24, \\\n    feature25, feature26, feature27, feature28, feature29, feature30, feature31):\n  feature = {\n      'image': _bytes_feature(feature0),\n      'StudyInstanceUID': _bytes_feature(feature1),\n      'SeriesInstanceUID': _bytes_feature(feature2),\n      'SOPInstanceUID': _bytes_feature(feature3),\n      'pe_present_on_image': _int64_feature(feature4),\n      'negative_exam_for_pe': _int64_feature(feature5),\n      'qa_motion':  _int64_feature(feature6),\n      'qa_contrast':  _int64_feature(feature7),\n      'flow_artifact':  _int64_feature(feature8),\n      'rv_lv_ratio_gte_1':  _int64_feature(feature9),\n      'rv_lv_ratio_lt_1':  _int64_feature(feature10),\n      'leftsided_pe': _int64_feature(feature11),\n      'chronic_pe':  _int64_feature(feature12),\n      'true_filling_defect_not_pe':  _int64_feature(feature13),\n      'rightsided_pe': _int64_feature(feature14),\n      'acute_and_chronic_pe': _int64_feature(feature15),\n      'central_pe': _int64_feature(feature16),\n      'indeterminate': _int64_feature(feature17),\n      'dcm_slice_thickness': _float_feature(feature18),\n      'dcm_pixel_spacing0': _float_feature(feature19),\n      'dcm_patient_position': _bytes_feature(feature20),\n      'dcm_image_position_patient2': _float_feature(feature21),\n      'dcm_image_orientation0': _float_feature(feature22),\n      'dcm_image_orientation1': _float_feature(feature23),\n      'dcm_image_orientation2': _float_feature(feature24),\n      'dcm_image_orientation3': _float_feature(feature25),\n      'dcm_image_orientation4': _float_feature(feature26),\n      'dcm_image_orientation5': _float_feature(feature27),\n      'dcm_instance_number': _int64_feature(feature28),\n      'dcm_kvp': _int64_feature(feature29),\n      'dcm_xray_tube_current': _int64_feature(feature30),\n      'dcm_exposure': _int64_feature(feature31)\n  }\n  example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n  return example_proto.SerializeToString()\n\ndef serialize_example_image_only(feature0,feature1):\n  feature = {\n      'image': _bytes_feature(feature0),\n      'SOPInstanceUID': _bytes_feature(feature1),\n  }\n  example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n  return example_proto.SerializeToString()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The window and level function is a pretty standard routine. (I'm note sure who claims original authorship.)\n\nThe DICOM data comes in Hounsfield units, which are being converted to grayscale. The range is roughly -1000 to 3000, so we pick a subset of the range that is clinically appropriate to our task.\n\nI read the Window/Level (also called Window/Center) from the DICOM file. For this Competition, I am actually plugging in a standard W/C rather than using the DICOM entries. But I keep the code in case I want to use the actual DICOM values.\n\nNote that WindowCenter and WindowLevel can be multi-valued. I just get the first entry.\n\nI get the image data from the DICOM file within a \"try:\" expression. Some of the image data requires a library GDCM for decompression. This is tricky to install (see Melanoma contest for a wide range of attempts). At least in the Train Dataset, they are not that common, so I just replace them with a blank image.\n\nI then define a \"crop\" function. CT images have a lot of space on the edges and a pulmonary embolism will never be outside the lungs. How much to crop is based on trial and error."},{"metadata":{"trusted":true},"cell_type":"code","source":"def window_image(img, window_center,window_width, intercept, slope):\n    window_width = abs(window_width)\n    img = (img*slope +intercept)\n    img_min = window_center - window_width//2\n    img_max = window_center + window_width//2\n    img[img<img_min] = img_min\n    img[img>img_max] = img_max\n    img = img - np.min(img)\n    img = img / np.max(img)\n    img = (img*255.0).astype('uint8')\n    return img \n\n\ndef dicom_window(dcm_data):\n    if 'RescaleIntercept' in dcm_data:\n        RI = dcm_data.RescaleIntercept\n    else:\n        RI = -1024\n    if 'RescaleSlope' in dcm_data:\n        RS = dcm_data.RescaleSlope\n    else:\n        RS = 1\n    if 'WindowCenter' in dcm_data:\n        WC = dcm_data.WindowCenter\n        if hasattr(WC, \"__len__\"):\n            WC = WC[0]\n    else:\n        WC = 40\n    if 'WindowWidth' in dcm_data:\n        WW = dcm_data.WindowWidth\n        if hasattr(WW, \"__len__\"):\n            WW = WW[0]\n    else:\n        WW = 400\n# get image from DICOM file. Sometimes error due to missing GDCM library.\n    try:\n        img = dcm_data.pixel_array\n    except:\n        print('DICOM Image error (usually GDCM library)',dcm_data.SOPInstanceUID)\n        img = np.zeros(shape=(512,512))\n# Here I replace the above parameters with a standard PE specific Window/Level. Idea courtesy of Ian Pan \n# Comment out these two lines if you want to use the W/L stored in the DICOM files.\n    WC = 100\n    WW = 700\n    img = window_image(img,WC,WW,RI,RS)\n    return(img)\n\ndef crop_center(img,cropx,cropy):\n    y,x = img.shape\n    startx = x//2-(cropx//2)\n    starty = y//2-(cropy//2)    \n    return img[starty:starty+cropy,startx:startx+cropx]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Because of the size of the dataset, I break it into manageable chunks. Typically 100,000 images at a time. Not perfect, because it could duplicate patients across TFRecords.\n\nI read train.csv and the new metadata file and merge them."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('only reading limited records', df_size)\n\n#sort csv file because we might not have all Metadata created.\ndf = pd.read_csv(csv_file)\ndf = df.sort_values(by=['StudyInstanceUID','SOPInstanceUID'])\ndf = df[df_start:(df_start+df_size)].reset_index(drop=True)\ndf['name'] = df['StudyInstanceUID'] + '/' + df['SeriesInstanceUID'] + '/' + df['SOPInstanceUID'] + '.dcm'\ndf['name2'] = df['StudyInstanceUID'] + '/' + df['SOPInstanceUID']\n\n#add dummy placeholder columns for non-existent test metadata. \nif train_or_test == 'test':\n    df = df.assign(pe_present_on_image = 0)\n    df = df.assign(negative_exam_for_pe = 0)\n    df = df.assign(qa_motion = 0)\n    df = df.assign(qa_contrast = 0)\n    df = df.assign(flow_artifact = 0)\n    df = df.assign(rv_lv_ratio_gte_1 = 0)\n    df = df.assign(rv_lv_ratio_lt_1 = 0)\n    df = df.assign(leftsided_pe = 0)\n    df = df.assign(chronic_pe = 0)\n    df = df.assign(true_filling_defect_not_pe = 0)\n    df = df.assign(rightsided_pe = 0)\n    df = df.assign(acute_and_chronic_pe = 0)\n    df = df.assign(central_pe = 0)\n    df = df.assign(indeterminate = 0)\n\nmd = pd.read_csv(metadata_file)\nmd['name2'] = md['StudyInstanceUID'] + '/' + md['SOPInstanceUID']\ndel md['StudyInstanceUID']\ndel md['SeriesInstanceUID']\ndel md['SOPInstanceUID']\n\ndf = pd.merge(df,md,on='name2')\n\nfilenames = df['name'].copy()\n\nprint(train_or_test, 'count',len(filenames))                        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Write TFRecords\n\nThe main routine to read the DICOM images and train.csv data and write the TFRecords. Written in 10,000 record chunks.\n\nAll the images in this competition seem to be 512x512, but I check anyway and resize if necessary.\n\nThen I crop to 400x400 because there is a lot of \"air\" on the edge of the patients and pulmonary embolism is never in the soft tissues of the chest. Value is just be some trial and error and hasn't been proven to be useful.\n\nThen I resize to 256 x 256 for my final images.\n\nI leave the images as Gray-Scale. Ian Pan (in his separate JPEG Dataset) uses the color channels to store three different Windows/Levels. I'm not sure if that takes up more disk storage. Probably a good idea, but I haven't implemented it yet.\n\nI'll change the Gray-Scale images to RGB during training/testing since I use Models that are pretrained on color images.\n\nImage errors related to decompression errors requiring the GDCM library result in blank images, but do not interfere with processing. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#optional shuffle for randomization\n#random.shuffle(filenames)\n\nSIZE = 10000\nIMGS = filenames\nCT = len(IMGS)//SIZE + int(len(IMGS)%SIZE!=0)\nfor j in range(CT):\n    print(); print('Writing TFRecord %i of %i...'%(j,CT))\n    CT2 = min(SIZE,len(IMGS)-j*SIZE)\n    with tf.io.TFRecordWriter(tfrec_folder + prefix + '%.2i-%i.tfrec'%(j,CT2)) as writer:\n        for k in range(CT2):\n            thisimg = SIZE*j+k\n            thisfile = IMGS[thisimg]\n            ds = pydicom.read_file(images_folder+'/'+thisfile)\n            img = dicom_window(ds)\n            if img.shape != [512,512]:\n                img = resize(img, (512,512), \\\n                   mode='reflect', anti_aliasing=True,preserve_range=True)\n            img = crop_center(img,400,400)\n            img = resize(img, (256,256), \\\n                   mode='reflect', anti_aliasing=True,preserve_range=True)\n            img = cv2.imencode('.jpg', img, (cv2.IMWRITE_JPEG_QUALITY, 94))[1].tostring()\n            name = IMGS[SIZE*j+k].split('.')[0]\n            \n            example = serialize_example(\n                img,\n                str.encode(df.loc[df['name']==thisfile, 'StudyInstanceUID'].iloc[0]),\n                str.encode(df.loc[df['name']==thisfile, 'SeriesInstanceUID'].iloc[0]),\n                str.encode(df.loc[df['name']==thisfile, 'SOPInstanceUID'].iloc[0]),\n                df.loc[df['name']==thisfile, 'pe_present_on_image'].iloc[0],\n                df.loc[df['name']==thisfile, 'negative_exam_for_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'qa_motion'].iloc[0],\n                df.loc[df['name']==thisfile, 'qa_contrast'].iloc[0],\n                df.loc[df['name']==thisfile, 'flow_artifact'].iloc[0],\n                df.loc[df['name']==thisfile, 'rv_lv_ratio_gte_1'].iloc[0],\n                df.loc[df['name']==thisfile, 'rv_lv_ratio_lt_1'].iloc[0],\n                df.loc[df['name']==thisfile, 'leftsided_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'chronic_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'true_filling_defect_not_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'rightsided_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'acute_and_chronic_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'central_pe'].iloc[0],\n                df.loc[df['name']==thisfile, 'indeterminate'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_slice_thickness'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_pixel_spacing0'].iloc[0],\n                str.encode(df.loc[df['name']==thisfile, 'dcm_patient_position'].iloc[0]),\n                df.loc[df['name']==thisfile, 'dcm_image_position_patient2'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation0'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation1'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation2'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation3'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation4'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_image_orientation5'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_instance_number'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_kvp'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_xray_tube_current'].iloc[0],\n                df.loc[df['name']==thisfile, 'dcm_exposure'].iloc[0]\n            )\n            writer.write(example)\n            if k%1000==0: print(k,', ',end='')\nprint('done')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Now we should test our files by reading a sample back.\n\nKeeping some header code (not needed in this notebook), so that this section can be a standalone notebook also.\n\nIf you use this section in a separate notebook, you'll need to update the location of the TFRecords to point to your stored DataSet."},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf \nfrom tensorflow import keras \nfrom tensorflow.keras.utils import Sequence \nimport tensorflow.keras.backend as K \nimport tensorflow.keras.backend \nimport tensorflow.keras.layers as L\n\n# adjust to train or test if broken out to separate notebookd\n#train_or_test = 'train'\n#train_or_test = 'test'\n\n# adjust to your directory if not working directory. Will need \"/\"\ntfrec_folder = ''\n\n#if accessing through a Dataset, enter name here:\nmy_GCS_PATH = 'your TFRecs Dataset here'\n\nDEVICE = \"TPU\"\n\nif DEVICE == \"TPU\":\n    print(\"connecting to TPU...\")\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n        print('Running on TPU ', tpu.master())\n    except ValueError:\n        print(\"Could not connect to TPU\")\n        tpu = None\n\nif tpu:\n    try:\n        print(\"initializing  TPU ...\")\n        tf.config.experimental_connect_to_cluster(tpu)\n        tf.tpu.experimental.initialize_tpu_system(tpu)\n        strategy = tf.distribute.experimental.TPUStrategy(tpu)\n        print(\"TPU initialized\")\n    except _:\n        print(\"failed to initialize TPU\")\nelse:\n    DEVICE = \"GPU\"\nif DEVICE != \"TPU\": \n    print(\"Using default strategy for CPU and single GPU\") \n    strategy = tf.distribute.get_strategy()\n\nif DEVICE == \"GPU\": \n    print(\"Num GPUs Available: \", len(tf.config.experimental.list_physical_devices('GPU')))\n\nAUTO = tf.data.experimental.AUTOTUNE\nREPLICAS = strategy.num_replicas_in_sync\n\nCFG = dict(\n    batch_size        = 32,\n    read_size         = 256, \n    crop_size         = 256, \n    net_size          = 256)\n\nimport os, gc, random, cv2, csv\nimport numpy as np\nfrom numpy.random import seed \nimport pandas as pd\nfrom skimage import measure\nfrom skimage.transform import resize \nfrom sklearn.metrics import roc_auc_score\nimport tensorflow as tf \n#from tensorflow import keras \n#from tensorflow.keras.utils import Sequence \n#from matplotlib import pyplot as plt \n#import matplotlib.pyplot as plt \n#import matplotlib.image as mpimg \nimport pydicom\nimport PIL \nfrom tqdm import tqdm\n\nfrom kaggle_datasets import KaggleDatasets \nmypath = '../input/metadata' \n\n#If using the TPU, you have to use a different path.\n\nif DEVICE == 'TPU': \n    GCS_PATH = KaggleDatasets().get_gcs_path(my_GCS_PATH) \nelse: \n    GCS_PATH = ''\n    \ntfrec_filter = train_or_test + '*.tfrec'\nprint(tfrec_filter)\n\ntfrec_files = np.sort(np.array(tf.io.gfile.glob(tfrec_filter)))\nprint('Number of files', len(tfrec_files))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Routines here to \n    1. Load images and metadata.\n    2. Augment the images (rotate, intensity, etc). Not needed here, but might be used in a training model."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def read_labeled_tfrecord_metadata(example):\n    tfrec_format = {\n        'image'                        : tf.io.FixedLenFeature([], tf.string),\n        'StudyInstanceUID'             : tf.io.FixedLenFeature([], tf.string),\n        'SeriesInstanceUID'            : tf.io.FixedLenFeature([], tf.string),\n        'SOPInstanceUID'               : tf.io.FixedLenFeature([], tf.string),\n        'pe_present_on_image'          : tf.io.FixedLenFeature([], tf.int64),\n        'negative_exam_for_pe'         : tf.io.FixedLenFeature([], tf.int64),\n        'qa_motion'                    : tf.io.FixedLenFeature([], tf.int64),\n        'qa_contrast'                  : tf.io.FixedLenFeature([], tf.int64),\n        'flow_artifact'                : tf.io.FixedLenFeature([], tf.int64),\n        'rv_lv_ratio_gte_1'            : tf.io.FixedLenFeature([], tf.int64),\n        'rv_lv_ratio_lt_1'             : tf.io.FixedLenFeature([], tf.int64),\n        'leftsided_pe'                 : tf.io.FixedLenFeature([], tf.int64),\n        'chronic_pe'                   : tf.io.FixedLenFeature([], tf.int64),\n        'true_filling_defect_not_pe'   : tf.io.FixedLenFeature([], tf.int64),\n        'rightsided_pe'                : tf.io.FixedLenFeature([], tf.int64),\n        'acute_and_chronic_pe'         : tf.io.FixedLenFeature([], tf.int64),\n        'central_pe'                   : tf.io.FixedLenFeature([], tf.int64),\n        'indeterminate'                : tf.io.FixedLenFeature([], tf.int64),\n        'dcm_slice_thickness'          : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_pixel_spacing0'           : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_patient_position'         : tf.io.FixedLenFeature([], tf.string),\n        'dcm_image_position_patient2'  : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation0'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation1'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation2'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation3'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation4'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_image_orientation5'       : tf.io.FixedLenFeature([], tf.float32),\n        'dcm_kvp'                      : tf.io.FixedLenFeature([], tf.int64),\n        'dcm_instance_number'          : tf.io.FixedLenFeature([], tf.int64),\n        'dcm_xray_tube_current'        : tf.io.FixedLenFeature([], tf.int64),\n        'dcm_exposure'                 : tf.io.FixedLenFeature([], tf.int64),\n    }           \n    example = tf.io.parse_single_example(example, tfrec_format)\n    metadata = [example['dcm_slice_thickness'],\n                example['dcm_pixel_spacing0']\n# for some reason I cannot return the dcm_instance_number\n#                example['dcm_instance_number']\n               ]\n    target = [example['pe_present_on_image'],\n              example['negative_exam_for_pe'],\n              example['rv_lv_ratio_gte_1'],\n              example['rv_lv_ratio_lt_1'],\n              example['leftsided_pe'],\n              example['chronic_pe'],\n              example['rightsided_pe'],\n              example['acute_and_chronic_pe'],\n              example['central_pe'],\n              example['indeterminate']]\n\n    return example['image'], metadata, target\n\ndef prepare_image(img, cfg=None, augment=True):    \n    img = tf.image.decode_jpeg(img, channels=1)\n#    img = tf.image.resize(img, [cfg['read_size'], cfg['read_size']])\n    img = tf.cast(img, tf.float32) / 255.0\n    \n#    if augment:\n#        img = tf.image.random_crop(img, [cfg['crop_size'], cfg['crop_size'], 3])\n#        img = tf.image.random_flip_left_right(img)\n#        img = tf.image.random_hue(img, 0.01)\n#        img = tf.image.random_saturation(img, 0.7, 1.3)\n#        img = tf.image.random_contrast(img, 0.8, 1.2)\n#        img = tf.image.random_brightness(img, 0.1)\n#    else:\n#        img = tf.image.central_crop(img, cfg['crop_size'] / cfg['read_size'])\n                                   \n    img = tf.image.resize(img, [cfg['net_size'], cfg['net_size']])\n    img = tf.image.grayscale_to_rgb(img, name=None)\n    img = tf.reshape(img, [cfg['net_size'], cfg['net_size'], 3])\n    return img\n\ndef get_dataset_metadata(files, cfg, augment = False, shuffle = False, repeat = False, \n                labeled=True, return_image_names=True):\n    \n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=AUTO)\n    ds = ds.cache()\n    \n    if repeat:\n        ds = ds.repeat()\n    \n    if shuffle: \n        ds = ds.shuffle(1024*8)\n        opt = tf.data.Options()\n        opt.experimental_deterministic = False\n        ds = ds.with_options(opt)\n        \n    if labeled: \n        ds = ds.map(read_labeled_tfrecord_metadata, num_parallel_calls=AUTO)\n    else:\n        ds = ds.map(lambda example: read_unlabeled_tfrecord_metadata(example, return_image_names), \n                    num_parallel_calls=AUTO)      \n    \n    ds = ds.map(lambda img, metadata, imgname_or_label: (prepare_image(img, augment=augment, cfg=cfg), metadata, imgname_or_label), \\\n                num_parallel_calls=AUTO)\n    ds = ds.map(lambda img, metadata, imgname_or_label: (tuple([img, metadata]), imgname_or_label), num_parallel_calls=AUTO)\n    ds = ds.batch(cfg['batch_size'] * REPLICAS)\n    ds = ds.prefetch(AUTO)\n    return ds\nprint(\"done\") \n\ndef show_dataset_metadata(thumb_size, cols, rows, ds):\n    mosaic = PIL.Image.new(mode='RGB', size=(thumb_size*cols + (cols-1), \n                                             thumb_size*rows + (rows-1)))\n    for idx, data in enumerate(iter(ds)):\n#        img, metadata, target_or_imgid = data\n        inputs, target_or_imgid = data\n        img = inputs[0]\n        metadata = inputs[1]\n#        print(metadata)\n        ix  = idx % cols\n        iy  = idx // cols\n        img = np.clip(img.numpy() * 255, 0, 255).astype(np.uint8)\n        img = PIL.Image.fromarray(img)\n        img = img.resize((thumb_size, thumb_size), resample=PIL.Image.BILINEAR)\n        mosaic.paste(img, (ix*thumb_size + ix, \n                           iy*thumb_size + iy))\n    display(mosaic)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load a sample of images and display."},{"metadata":{"trusted":true},"cell_type":"code","source":"files_sample = np.sort(np.array(tfrec_files))\n\nds = get_dataset_metadata(files_sample, CFG, shuffle=True,augment=True, labeled=True).unbatch().take(12*5)   \nshow_dataset_metadata(256, 2, 5, ds)\nprint(\"done\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}