{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1 style=\"color:green\">What, Why and How?</h1>\n\n<h3 style=\"color:gray\">Pulomonary Embolism</h3>\n\n![](https://upload.wikimedia.org/wikipedia/commons/7/77/SaddlePE.PNG)\n\nPulmonary embolism (PE) is a blockage of an artery in the lungs by a substance that has moved from elsewhere in the body through the bloodstream (embolism). Symptoms of a PE may include shortness of breath, chest pain particularly upon breathing in, and coughing up blood. Symptoms of a blood clot in the leg may also be present, such as a red, warm, swollen, and painful leg. Signs of a PE include low blood oxygen levels, rapid breathing, rapid heart rate, and sometimes a mild fever. Severe cases can lead to passing out, abnormally low blood pressure, and sudden death.\n\n<h3 style=\"color:gray\">How it happens?</h3>\n\nPE usually results from a blood clot in the leg that travels to the lung. The risk of blood clots is increased by cancer, prolonged bed rest, smoking, stroke, certain genetic conditions, estrogen-based medication, pregnancy, obesity, and after some types of surgery. A small proportion of cases are due to the embolization of air, fat, or amniotic fluid.\n\n<strong>In this competition, we are predicting the existence and characteristics of pulmonary embolisms.</strong>"},{"metadata":{},"cell_type":"markdown","source":"<strong style=\"color:red\">If you liked this notebook, don't forget to leave an upvote or a comment!</strong>"},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"! pip install -q dabl","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport pydicom as dcm\nimport glob\nimport random\nimport warnings\nfrom tqdm.notebook import tqdm\nfrom colorama import Fore, Style\nimport os\n\nimport dabl\n\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.graph_objs as go\nfrom plotly.offline import iplot\n\nimport matplotlib.animation as animation\nfrom matplotlib.widgets import Slider\nfrom IPython.display import HTML, Image\n\nwarnings.simplefilter(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def cout(string: str, color: str) -> str:\n    \"\"\"\n    Prints a string in the required color\n    \"\"\"\n    print(color+string+Style.RESET_ALL)\n    \ndef read_image(filename: str) -> np.ndarray:\n    \"\"\"\n    Read a DICOM Image File and return it (as a numpy array)\n    \"\"\"\n    img = dcm.dcmread(filename).pixel_array\n    img[img == -2000] = 0\n    return img\n\ndef plot_dicom(image_list, rows=5, cols=4, cmap='jet', is_train=True):\n    fig = plt.figure(figsize=(12, 12))\n    if is_train:\n        plt.title(f\"DICOM Images from Training Set\")\n    else:\n        plt.title(f\"DICOM Images from Testing Set\")\n    img_count = 0\n    for i in range(1, rows*cols+1):\n        filename = image_list[img_count]\n        image = read_image(filename)\n        fig.add_subplot(rows, cols, i)\n        plt.grid(False)\n        plt.imshow(image, cmap=cmap)\n        img_count += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/rsna-str-pulmonary-embolism-detection/train.csv\")\ntest = pd.read_csv(\"../input/rsna-str-pulmonary-embolism-detection/test.csv\")\nsub = pd.read_csv(\"../input/rsna-str-pulmonary-embolism-detection/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%%time\ntrain_files = glob.glob(\"../input/rsna-str-pulmonary-embolism-detection/train/*/*/*.dcm\")\ntest_files = glob.glob(\"../input/rsna-str-pulmonary-embolism-detection/test/*/*/*.dcm\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h1 style=\"color:blue\">Exploratory Data Analysis</h1>\n\nLet's start with EDA. We'll cover both training and testing sets."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"cout(f\"Total Number of DICOM Images in Training Set: {len(train_files)}\", Fore.GREEN)\ncout(f\"Total Number of DICOM Images in Testing Set:  {len(test_files)}\", Fore.YELLOW)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2 style=\"color:aqua\">Peaking at the DataFrames</h2>\n\nLet's start by taking a look at the dataframes of train, test and submission sets."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.head()   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2 style=\"color:aqua\">Feature Columns</h2>\nAt the hindsight it may appear that there are no features other than the images themselves, but we are provided with 4 Features that aren't used for prediction but only for helping our predictions."},{"metadata":{},"cell_type":"markdown","source":"These features are:\n\n* **qa_motion** - Indicates whether radiologists noted an issue with motion in the study.\n* **qa_contrast** - Indicates whether radiologists noted an issue with contrast in the study.\n* **true_filling_defect_not_pe** - Indicates a defect that is NOT PE.\n* **flow_artifact**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"features = train[['qa_motion', 'qa_contrast', 'true_filling_defect_not_pe', 'flow_artifact']]\nfeatures.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:yellow\">Issue with Motion? (qa_motion)</h3>\nThis feature indicates if there was an issue with the motion noted by radiologists in the study."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"vals = features['qa_motion'].value_counts().tolist()\nidx = ['No Issue', 'Issue']\nfig = px.pie(\n    values=vals,\n    names=idx,\n    title='Issue with Motion in Studies',\n    color_discrete_sequence=['blue', 'cyan']\n)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:yellow\">Issue with Contrast? (qa_contrast)</h3>\nThis feature indicates if there was an issue with the contrast noted by radiologists in the study."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"vals = features['qa_contrast'].value_counts().tolist()\nidx = ['No Issue', 'Issue']\nfig = px.pie(\n    values=vals,\n    names=idx,\n    title='Issue with Contrast in Studies',\n    color_discrete_sequence=['gold', 'yellow']\n)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:yellow\">Indicates if a Defect isn't PE? (true_filling_defect_not_pe)</h3>\nThis feature indicates if the defect is a PE or not."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"vals = features['true_filling_defect_not_pe'].value_counts().tolist()\nidx = ['Defect not PE', 'Defect is PE']\nfig = px.pie(\n    values=vals,\n    names=idx,\n    title='Is defect PE or Not',\n    color_discrete_sequence=['black', 'gray']\n)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:yellow\">flow_artifact</h3>"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"vals = features['flow_artifact'].value_counts().tolist()\nidx = ['Is an Artifact', 'Is not an Artifact']\nfig = px.pie(\n    values=vals,\n    names=idx,\n    title='Flow Artifact',\n)\niplot(fig)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note: I couldn't find what `flow_artifact` feature is about, please correct me if I am wrong about it."},{"metadata":{},"cell_type":"markdown","source":"<h2 style=\"color:aqua\">Target Columns</h2>\n\nThe columns that remain after we disclude the feature columns are target columns. Here are the details of what different target columns mean:\n\n* **negative_exam_for_pe** - exam-level, whether there are any images in the study that have PE present.\n* **rv_lv_ratio_gte_1** - exam-level, indicates whether the RV/LV ratio present in the study is >= 1\n* **rv_lv_ratio_lt_1** - exam-level, indicates whether the RV/LV ratio present in the study is < 1\n* **leftsided_pe** - exam-level, indicates that there is PE present on the left side of the images in the study\n* **chronic_pe** - exam-level, indicates that the PE in the study is chronic\n* **rightsided_pe** - exam-level, indicates that there is PE present on the right side of the images in the study\n* **acute_and_chronic_pe** - exam-level, indicates that the PE present in the study is both acute AND chronic\n* **central_pe** - exam-level, indicates that there is PE present in the center of the images in the study\n* **indeterminate** -exam-level, indicates that while the study is not negative for PE, an ultimate set of exam-level labels could not be created, due to QA issues"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"targets = train[['negative_exam_for_pe', 'rv_lv_ratio_gte_1', 'rv_lv_ratio_lt_1', 'leftsided_pe', 'chronic_pe', 'rightsided_pe', 'acute_and_chronic_pe', 'central_pe', 'indeterminate']]\ntargets.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:yellow\">DABL Plot</h3>\n\nLet's plot some of the target columns using DABL"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"dabl.plot(targets, target_col='negative_exam_for_pe')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"dabl.plot(targets, target_col='rv_lv_ratio_gte_1')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dabl.plot(targets, 'rightsided_pe')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2 style=\"color:aqua\">DICOM Image Data Analysis</h2>\n\nLet's now start doing the DICOM Image Data Analysis."},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:yellow\">Inspect the Images</h3>\n\nLet's start by just taking a look at a few images from training and testing sets"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Plot any-20 training images\ntrain_imgs = train_files[:20]\nplot_dicom(image_list=train_imgs, is_train=True, cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Plot any-20 testing images\ntest_imgs = test_files[:20]\nplot_dicom(image_list=test_imgs, is_train=False, cmap='gray')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3 style=\"color:yellow\">Lung CT Animation</h3>\n\nFinally, let's make Animation of different lung scan slices of a particular Study.\n\n[This Notebook](https://www.kaggle.com/avloss/eda-with-animation) helped me in making the animation."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%capture\n\nids = \"../input/rsna-str-pulmonary-embolism-detection/train/6897fa9de148/2bfbb7fd2e8b/\"\n\nimg_datas = []\nfor im in os.listdir(ids):\n    meta = dcm.dcmread(os.path.join(ids, im))\n    srl = meta.InstanceNumber\n    data = meta.pixel_array\n    data[data == -2000] = 0\n    img_datas.append((srl, data))\n    \nimg_datas.sort()\nims = []\nfig = plt.figure()\nfor gg in img_datas:\n    img_ = plt.imshow(gg[1], cmap='jet', animated=True)\n    plt.axis(\"off\")\n    ims.append([img_])\n\nani = animation.ArtistAnimation(fig, ims, interval=1000//24, blit=False, repeat_delay=1000)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"HTML(ani.to_jshtml())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}