{"cells":[{"metadata":{},"cell_type":"markdown","source":"**UPDATE:** this notebok has been verified by competitions hosts. They will be using it to check the top 10 private leaderboard submissions to verify compliance with the logical consistency of labels requirements after the end of the contest. The official copy of version 3 of this notebook that will be used by the hosts is available [here](https://www.kaggle.com/anthracene/host-confirmed-label-consistency-check).\n\n# OVERVIEW\n\nOne of the unique rules of this competition is a special requirement regarding the label hierarchy consistency. We predict nine exam-level and one image-level label, where some of the labels are conflicting and must adhere to a specific hirearchy displayed on the image below.\n\n![hierarchy](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F115173%2Fa2a5ee66b5799274141dd547cc3ea466%2FPE%20figure.jpg?generation=1599575183749576&alt=media)\n\nAccording to the data description page: \n\n> Winning submissions will be inspected to ensure label predictions adhere to the expected label hierarchy defined by the diagram on the Data page. The metric intends to heavily penalize submissions which mis-predict in this manner, however due to the complexity of predictions at both image and study levels and as an extra precaution, the host will verify that prospective winners have not made conflicting label predictions. The requirements which submissions will be held to are [specified by the host in this post](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473).\n\nThis notebook develops a function that checks label consistency in a submission file to make sure it adheres to the competition rules."},{"metadata":{},"cell_type":"markdown","source":"# VERSION HISTORY\n\n- v1: first version of the notebook\n- v2: corrected handling of `prediction == 0.5` for some labels as noted by [@anthracene](https://www.kaggle.com/anthracene) in the comments\n- v3: wrapped consistency checks into `check_consitency()` function\n- v4: text adjustments (no code changes)\n- v5: added rule 1d (see the [discussion](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473]))"},{"metadata":{},"cell_type":"markdown","source":"# CHECKING CONSISTENCY\n\n## PREPARATION"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# LIBRARIES\nimport pandas as pd\nimport numpy as np","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# IMPORT DATA\ntrain = pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/train.csv')\ntest  = pd.read_csv('../input/rsna-str-pulmonary-embolism-detection/test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's import one of the public kernel submission files to check it for the label consistency requirements. As an example, I am using the `submission.csv` file produced by [@seraphwedd18](https://www.kaggle.com/seraphwedd18) in [this kernel](https://www.kaggle.com/seraphwedd18/pe-detection-with-keras-model-creation/output?scriptVersionId=42782514&select=submission.csv). "},{"metadata":{"trusted":true},"cell_type":"code","source":"# IMPORT EXAMPLE SUBMISSION\nsub = pd.read_csv('../input/pe-detection-with-keras-model-creation/submission.csv')\nsub.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## SUBMISSION TRANSFORMATION"},{"metadata":{},"cell_type":"markdown","source":"First, we transform submission from the long to the wide format similar to the `train.csv` file to make it easier to check for rules. This involves two steps:\n1. Reshaping exam-level predictions to wide format.\n2. Merging image-level predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# EXAM LEVEL\nfor i in test['StudyInstanceUID'].unique():\n\n    df_tmp = sub.loc[sub.id.str.contains(i, regex = False)].reset_index(drop = True)\n    df_tmp['StudyInstanceUID'] = df_tmp['id'].str.split('_').str[0]\n    df_tmp['label_type']       = df_tmp['id'].str.split('_').str[1:].apply(lambda x: '_'.join(x))\n    del df_tmp['id']\n    \n    if i == test['StudyInstanceUID'].unique()[0]:\n        df = df_tmp.copy()\n    else:\n        df = pd.concat([df, df_tmp], axis = 0)\n        \ndf_exam = df.pivot(index = 'StudyInstanceUID', columns = 'label_type', values = 'label')\ndf_exam.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# IMAGE LEVEL\ndf_image = sub.loc[sub.id.isin(test.SOPInstanceUID)].reset_index(drop = True)\ndf_image = df_image.merge(test, how = 'left', left_on = 'id', right_on = 'SOPInstanceUID')\ndf_image.rename(columns = {\"label\": \"pe_present_on_image\"}, inplace = True)\ndel df_image['id']\ndf_image.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# MERGER\ndf = df_exam.merge(df_image, how = 'left', on = 'StudyInstanceUID')\nids    = ['StudyInstanceUID', 'SeriesInstanceUID', 'SOPInstanceUID']\nlabels = [c for c in df.columns if c not in ids]\ndf = df[ids + labels]\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## FORMALIZING CONSISTENCY RULES\n\nThe rules below represent the label consistency requirements outlined by [@anthracene](https://www.kaggle.com/anthracene) in [this discussion topic](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473). I encourage you to read the topic before inspecting the rules below.\n\nWe implement the following two groups of rules on the exam level. The first group specifies conflicting characteristics of PE if it is detected on any of the images in the exam. The second group makes sure that if there are no images with detected PE in the exam, we do not predict any of the PE charactersitcs to be present.\n\n1. If there is at least one image per `StudyInstanceUID` with `pe_present_on_image` > 0.5, then:\n    - either `rv_lv_ratio_lt_1` or `rv_lv_ratio_gte_1` must have p > 0.5; both cannot have p > 0.5.\n    - at least one of `central_pe`, `rightsided_pe` and `leftsided_pe` must have p > 0.5; multiple having p > 0.5 is allowed.\n    - `acute_and_chronic_pe` and `chronic_pe`: only one of them can have p > 0.5; neither having p > 0.5 is allowed.\n    - both `indeterminate` and `negative_exam_for_pe` can not have p > 0.5.\n2. If there are no images per `StudyInstanceUID` with `pe_present_on_image` > 0.5, then:\n    - either `indeterminate` or `negative_exam_for_pe` must have p > 0.5; both cannot have p > 0.5.\n    - all positive-related labels: `rv_lv_ratio_lt_1`, `rv_lv_ratio_gte_1`, `central_pe`, `rightsided_pe`, `leftsided_pe`, `acute_and_chronic_pe` and `chronic_pe` must have p < 0.5."},{"metadata":{},"cell_type":"markdown","source":"## CHECKING CONSISTENCY RULES\n\nLet's start by checking if there is at least one image predicted as positive (`pe_present_on_image > 0.5`) in an exam and splitting our submission data into positive and negative exams."},{"metadata":{"trusted":true},"cell_type":"code","source":"# SPLIT NEGATIVE AND POSITIVE EXAMS\n\ndf['positive_images_in_exam'] = df['StudyInstanceUID'].map(df.groupby(['StudyInstanceUID']).pe_present_on_image.max())\n\ndf_pos = df.loc[df.positive_images_in_exam >  0.5]\ndf_neg = df.loc[df.positive_images_in_exam <= 0.5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can check label consistency rules separately for positive and negative exams. We will identify rows that do not satisfy any of the requirements and merge them into a data frame representing the inconsistent predictions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# CHECKING CONSISTENCY OF POSITIVE EXAM LABELS\n\nrule1a = df_pos.loc[((df_pos.rv_lv_ratio_lt_1  >  0.5)  & \n                     (df_pos.rv_lv_ratio_gte_1 >  0.5)) | \n                    ((df_pos.rv_lv_ratio_lt_1  <= 0.5)  & \n                     (df_pos.rv_lv_ratio_gte_1 <= 0.5))].reset_index(drop = True)\nrule1a['broken_rule'] = '1a'\n\nrule1b = df_pos.loc[(df_pos.central_pe    <= 0.5) & \n                    (df_pos.rightsided_pe <= 0.5) & \n                    (df_pos.leftsided_pe  <= 0.5)].reset_index(drop = True)\nrule1b['broken_rule'] = '1b'\n\nrule1c = df_pos.loc[(df_pos.acute_and_chronic_pe > 0.5) & \n                    (df_pos.chronic_pe           > 0.5)].reset_index(drop = True)\nrule1c['broken_rule'] = '1c'\n\nrule1d = df_pos.loc[(df_pos.indeterminate        > 0.5) | \n                    (df_pos.negative_exam_for_pe > 0.5)].reset_index(drop = True)\nrule1d['broken_rule'] = '1d'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CHECKING CONSISTENCY OF NEGATIVE EXAM LABELS\n\nrule2a = df_neg.loc[((df_neg.indeterminate        >  0.5)  & \n                     (df_neg.negative_exam_for_pe >  0.5)) | \n                    ((df_neg.indeterminate        <= 0.5)  & \n                     (df_neg.negative_exam_for_pe <= 0.5))].reset_index(drop = True)\nrule2a['broken_rule'] = '2a'\n\nrule2b = df_neg.loc[(df_neg.rv_lv_ratio_lt_1     > 0.5) | \n                    (df_neg.rv_lv_ratio_gte_1    > 0.5) |\n                    (df_neg.central_pe           > 0.5) | \n                    (df_neg.rightsided_pe        > 0.5) | \n                    (df_neg.leftsided_pe         > 0.5) |\n                    (df_neg.acute_and_chronic_pe > 0.5) | \n                    (df_neg.chronic_pe           > 0.5)].reset_index(drop = True)\nrule2b['broken_rule'] = '2b'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# MERGING INCONSISTENT PREDICTIONS\nerrors = pd.concat([rule1a, rule1b, rule1c, rule2a, rule2b], axis = 0)\nerrors['broken_rule'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, the submission file does not have any errors and satisfies the consistency requirements!"},{"metadata":{},"cell_type":"markdown","source":"# WRAPPING IN A FUNCTION\n\nThe function below wraps the previous code blocks into a function that can be applied to a submission file."},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_consistency(sub, test):\n    \n    '''\n    Checks label consistency and returns the errors\n    \n    Args:\n    sub   = submission dataframe (pandas)\n    test  = test.csv dataframe (pandas)\n    '''\n    \n    # EXAM LEVEL\n    for i in test['StudyInstanceUID'].unique():\n        df_tmp = sub.loc[sub.id.str.contains(i, regex = False)].reset_index(drop = True)\n        df_tmp['StudyInstanceUID'] = df_tmp['id'].str.split('_').str[0]\n        df_tmp['label_type']       = df_tmp['id'].str.split('_').str[1:].apply(lambda x: '_'.join(x))\n        del df_tmp['id']\n        if i == test['StudyInstanceUID'].unique()[0]:\n            df = df_tmp.copy()\n        else:\n            df = pd.concat([df, df_tmp], axis = 0)\n    df_exam = df.pivot(index = 'StudyInstanceUID', columns = 'label_type', values = 'label')\n    \n    # IMAGE LEVEL\n    df_image = sub.loc[sub.id.isin(test.SOPInstanceUID)].reset_index(drop = True)\n    df_image = df_image.merge(test, how = 'left', left_on = 'id', right_on = 'SOPInstanceUID')\n    df_image.rename(columns = {\"label\": \"pe_present_on_image\"}, inplace = True)\n    del df_image['id']\n    \n    # MERGER\n    df = df_exam.merge(df_image, how = 'left', on = 'StudyInstanceUID')\n    ids    = ['StudyInstanceUID', 'SeriesInstanceUID', 'SOPInstanceUID']\n    labels = [c for c in df.columns if c not in ids]\n    df = df[ids + labels]\n    \n    # SPLIT NEGATIVE AND POSITIVE EXAMS\n    df['positive_images_in_exam'] = df['StudyInstanceUID'].map(df.groupby(['StudyInstanceUID']).pe_present_on_image.max())\n    df_pos = df.loc[df.positive_images_in_exam >  0.5]\n    df_neg = df.loc[df.positive_images_in_exam <= 0.5]\n    \n    # CHECKING CONSISTENCY OF POSITIVE EXAM LABELS\n    rule1a = df_pos.loc[((df_pos.rv_lv_ratio_lt_1  >  0.5)  & \n                         (df_pos.rv_lv_ratio_gte_1 >  0.5)) | \n                        ((df_pos.rv_lv_ratio_lt_1  <= 0.5)  & \n                         (df_pos.rv_lv_ratio_gte_1 <= 0.5))].reset_index(drop = True)\n    rule1a['broken_rule'] = '1a'\n    rule1b = df_pos.loc[(df_pos.central_pe    <= 0.5) & \n                        (df_pos.rightsided_pe <= 0.5) & \n                        (df_pos.leftsided_pe  <= 0.5)].reset_index(drop = True)\n    rule1b['broken_rule'] = '1b'\n    rule1c = df_pos.loc[(df_pos.acute_and_chronic_pe > 0.5) & \n                        (df_pos.chronic_pe           > 0.5)].reset_index(drop = True)\n    rule1c['broken_rule'] = '1c'\n    rule1d = df_pos.loc[(df_pos.indeterminate        > 0.5) | \n                        (df_pos.negative_exam_for_pe > 0.5)].reset_index(drop = True)\n    rule1d['broken_rule'] = '1d'\n\n    # CHECKING CONSISTENCY OF NEGATIVE EXAM LABELS\n    rule2a = df_neg.loc[((df_neg.indeterminate        >  0.5)  & \n                         (df_neg.negative_exam_for_pe >  0.5)) | \n                        ((df_neg.indeterminate        <= 0.5)  & \n                         (df_neg.negative_exam_for_pe <= 0.5))].reset_index(drop = True)\n    rule2a['broken_rule'] = '2a'\n    rule2b = df_neg.loc[(df_neg.rv_lv_ratio_lt_1     > 0.5) | \n                        (df_neg.rv_lv_ratio_gte_1    > 0.5) |\n                        (df_neg.central_pe           > 0.5) | \n                        (df_neg.rightsided_pe        > 0.5) | \n                        (df_neg.leftsided_pe         > 0.5) |\n                        (df_neg.acute_and_chronic_pe > 0.5) | \n                        (df_neg.chronic_pe           > 0.5)].reset_index(drop = True)\n    rule2b['broken_rule'] = '2b'\n    \n    # MERGING INCONSISTENT PREDICTIONS\n    errors = pd.concat([rule1a, rule1b, rule1c, rule1d, rule2a, rule2b], axis = 0)\n    \n    # OUTPUT\n    print('Found', len(errors), 'inconsistent predictions')\n    return errors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CHECK\nerrors = check_consistency(sub, test)\nerrors.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}