{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Pulmonary Embolism - How to Create TFRecords\n\nIn this notebook, we will create TFRecords that contains basic information to train TensorFlow models. \n\n### Overview\nThe images are CT scans (axial slices) that were pre-processed using different windows. Each channel is a different window. Resolution size 512x512.\n\n- RED channel / LUNG window / level=-600, width=1500\n- GREEN channel / PE window / level=100, width=700\n- BLUE channel / MEDIASTINAL window / level=40, width=400\n\n\n\n### Acknowledgements\nThese images were made using [Ian Pan's](https://www.kaggle.com/vaillant) pre-processing found [here](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182930)"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# LOAD LIBRARIES\nimport numpy as np, pandas as pd, os\nimport matplotlib.pyplot as plt, cv2\nimport tensorflow as tf, re, math\nfrom glob import glob\nimport os\nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fold0 = [glob(x + '/*.jpg') for x in glob('../input/pe-train-512x512-fold-0-batch-*')]\nfold1 = [glob(x + '/*.jpg') for x in glob('../input/pe-train-512x512-fold-1-batch-*')]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"LABEL_COLUMNS = ['negative_exam_for_pe', 'indeterminate', 'chronic_pe', 'acute_and_chronic_pe', 'central_pe', 'leftsided_pe', 'rightsided_pe', 'rv_lv_ratio_gte_1', 'rv_lv_ratio_lt_1']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Write TFRecords - Train\nAll the code below comes from Tensorflow's docs [here](https://www.tensorflow.org/tutorials/load_data/tfrecord)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def _bytes_feature(value):\n  \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n  if isinstance(value, type(tf.constant(0))):\n    value = value.numpy() # BytesList won't unpack a string from an EagerTensor.\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\ndef _float_feature(value):\n  \"\"\"Returns a float_list from a float / double.\"\"\"\n  return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))\n\ndef _int64_feature(value):\n  \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def serialize_example(feature0, feature1, feature2):\n  feature = {\n      'image': _bytes_feature(feature0),\n      'image_name': _bytes_feature(feature1),\n      'target': _bytes_feature(feature2)\n  }\n  example_proto = tf.train.Example(features=tf.train.Features(feature=feature))\n  return example_proto.SerializeToString()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def write_file(fold, tfrec_name='', file_size=100):\n    with tf.io.TFRecordWriter(f'{tfrec_name}.tfrec') as writer:\n        for k, image_path in enumerate(fold):\n            image_name = image_path.split('/')[-1].split('.')[0]\n            target = df[df['SOPInstanceUID'] == image_name][LABEL_COLUMNS].values[0]\n            img = cv2.imread(image_path)\n            img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)\n            img = cv2.imencode('.jpg', img, (cv2.IMWRITE_JPEG_QUALITY, 94))[1].tostring()\n\n            example = serialize_example(\n                img, \n                str.encode(image_name), \n                tf.io.serialize_tensor(np.array(target, dtype=np.uint8)))\n            writer.write(example)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def shard_files(file_paths, k=0, f=0):\n    file_size = 1000\n    path_array = []\n    for num, fp in enumerate(file_paths):\n        path_array.append(fp)\n        if (num+1)%file_size==0:\n            print(f'Sharding files {num+1}')\n            write_file(path_array, tfrec_name=f'train-{f}{k:02}-{(num+1)//file_size}', file_size=file_size)\n            path_array = [] # reset the array\n    # write remaining files\n    if len(path_array) > 0:\n        print('\\nSharding remainder: ', len(path_array))\n        write_file(path_array, tfrec_name=f'train-{f}{k:02}-{(num+1)//file_size+1}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Notes for Pulmonary Embolism TFRecord creation: \n1. enter a number from k = 0-19, where k is the batch number\n2. once you reach 19 for fold0, change fold0 to fold1 and f=0 to f=1\n3. you must complete 40 notebooks to create TFRecords for Pulmonary Embolism dataset\n4. once the notebook is done executing, create a \"new dataset\" using the button below\n5. make sure the dataset is public in order to use the TFRecord through Kaggle's notebooks"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nk = 0\nf = 0\nshard_files(fold0[k], k, f)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}