{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":45867,"databundleVersionId":6924515,"sourceType":"competition"},{"sourceId":6661702,"sourceType":"datasetVersion","datasetId":3844162}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"raw","source":"Simple Keras Training, only using the thumbnail images\nCheck it out if yoy like it please consider to upvote this notebook\n\nAnd the inference notebook can be found here: https://www.kaggle.com/code/pjmathematician/ubco-keras-cnn-baseline-thumbnails-inference","metadata":{}},{"cell_type":"markdown","source":"# **Ovarian Cancer Classification and Outlier Detection with CNNs**","metadata":{}},{"cell_type":"markdown","source":"**INTRODUCTION**","metadata":{}},{"cell_type":"markdown","source":"Ovarian cancer is a type of cancer that begins in the ovaries. \nThe ovaries — each about the size of an almond — produce eggs (ova) as well as \nthe hormones estrogen and progesterone. Ovarian cancer is a growth of cells that \nforms in the ovaries. The cells multiply quickly and can invade and destroy healthy\nbody tissue.9 May, 2023\nMayo Clinic\nMedical center in Rochester, Minnesota\nMayo Clinic is a nonprofit American academic medical center focused on\nintegrated health care, education, and research. Wikipedia","metadata":{}},{"cell_type":"markdown","source":"\nCancerous epithelial tumors are called carcinomas. About 85% to 90% of malignant ovarian \ncancers are epithelial ovarian carcinomas. These tumor cells have several features\n(when looked at in the lab) that can be used to classify epithelial ovarian carcinomas \ninto different types.11 Apr 2018\n\nhttps://www.cancer.org/cancer/types/ovarian-cancer/about/what-is-ovarian-cancer.html#:~:text=Malignant%20epithelial%20ovarian%20tumors,ovarian%20carcinomas%20into%20different%20types.","metadata":{}},{"cell_type":"markdown","source":"**DESCRIPTION**","metadata":{}},{"cell_type":"markdown","source":"Description\nOvarian carcinoma is the most lethal cancer of the female reproductive system. There are five common subtypes of ovarian cancer: high-grade serous carcinoma, clear-cell ovarian carcinoma, endometrioid, low-grade serous, and mucinous carcinoma. Additionally, there are several rare subtypes (\"Outliers\"). These are all characterized by distinct cellular morphologies, etiologies, molecular and genetic profiles, and clinical attributes. Subtype-specific treatment approaches are gaining prominence, though first requires subtype identification, a process that could be improved with data science.\n\nCurrently, ovarian cancer diagnosis relies on pathologists to assess subtypes. However, this presents several challenges, including disagreements between observers and the reproducibility of diagnostics. Furthermore, underserved communities often lack access to specialist pathologists, and even well-developed communities face a shortage of pathologists with expertise in gynecologic malignancies.\n\nDeep learning models have exhibited remarkable proficiency in analyzing histopathology images. Yet challenges still exist, such as the need for a significant amount of training data, ideally from a single source. Technical, ethical, and financial constraints, as well as confidentiality concerns, make training a challenge. In this competition, you will have access to the most extensive and diverse ovarian cancer dataset of histopathology images from more than 20 centers across four continents.","metadata":{}},{"cell_type":"markdown","source":"**OBJECTIVE**","metadata":{}},{"cell_type":"markdown","source":"1.The goal of the UBC Ovarian Cancer subtypE classification and outlier detectioN (UBC-OCEAN) competition is to classify ovarian cancer subtypes. \n\n2.To build a model trained on the world's most extensive ovarian cancer dataset of histopathology images obtained from more than 20 medical centers.","metadata":{}},{"cell_type":"markdown","source":"**Resizing the Dataset**: https://www.kaggle.com/datasets/utm529fg/ubc-reduced-png-2964x2964/data","metadata":{"execution":{"iopub.status.busy":"2023-10-11T12:07:31.733202Z","iopub.execute_input":"2023-10-11T12:07:31.733554Z","iopub.status.idle":"2023-10-11T12:07:31.739863Z","shell.execute_reply.started":"2023-10-11T12:07:31.733526Z","shell.execute_reply":"2023-10-11T12:07:31.738485Z"}}},{"cell_type":"code","source":"### Importing Libraries to be use\nimport os\n\nimport pandas as pd\nimport numpy as np\nimport math\nimport cv2\nfrom glob import glob #\nimport itertools\nimport seaborn as sns\nfrom tqdm.auto import tqdm\nimport matplotlib.pyplot as plt\nsns.set(color_codes =True)\n%matplotlib inline\n\nfrom skimage import io\nfrom skimage.color import rgb2gray\nfrom skimage.transform import rescale, resize, downscale_local_mean\n\n\nfrom sklearn.preprocessing import LabelEncoder, OneHotEncoder\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\n\nimport matplotlib.pyplot as plt\n\n### From flow import the following\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential,Model # Sequential api for sequential model \nfrom tensorflow.keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, GlobalMaxPooling2D # Importing different layers \nfrom tensorflow.keras.layers import BatchNormalization, Activation, Input, LeakyReLU\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.utils import to_categorical # To perform one-hot encoding \nfrom tensorflow.keras import losses, optimizers\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout\nfrom keras.utils import to_categorical\nprint(\"Tensorflow version \" + tf.__version__)\n\nimport math\n\nrc = {\n    \"axes.facecolor\": \"#ffaaa5\",\n    \"figure.facecolor\": \"#ffaaa5\",\n    \"axes.edgecolor\": \"#000000\",\n    \"grid.color\": \"#EBEBE7\",\n    \"font.family\": \"serif\",\n    \"axes.labelcolor\": \"#000000\",\n    \"xtick.color\": \"#000000\",\n    \"ytick.color\": \"#000000\",\n    \"grid.alpha\": 0.4\n}\n\nsns.set(rc=rc)\n\nfrom colorama import Style, Fore\nred = Style.BRIGHT + Fore.RED\nblu = Style.BRIGHT + Fore.BLUE\nmgt = Style.BRIGHT + Fore.MAGENTA\ngld = Style.BRIGHT + Fore.YELLOW\nres = Style.RESET_ALL","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.353468Z","iopub.execute_input":"2023-12-02T12:43:38.353846Z","iopub.status.idle":"2023-12-02T12:43:38.378458Z","shell.execute_reply.started":"2023-12-02T12:43:38.353819Z","shell.execute_reply":"2023-12-02T12:43:38.377455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CFG:\n    TRAIN_THUMBNAILS_PATH = \"/kaggle/input/UBC-OCEAN/train_thumbnails\"\n    TRAIN_FULL_PATH = \"/kaggle/input/UBC-OCEAN/train_images\"\n    TEST_THUMBNAILS_PATH = \"/kaggle/input/UBC-OCEAN/test_thumbnails\"\n    TEST_FULL_PATH = \"/kaggle/input/UBC-OCEAN/test_images\"","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.380377Z","iopub.execute_input":"2023-12-02T12:43:38.380672Z","iopub.status.idle":"2023-12-02T12:43:38.386348Z","shell.execute_reply.started":"2023-12-02T12:43:38.380639Z","shell.execute_reply":"2023-12-02T12:43:38.385472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = pd.read_csv('/kaggle/input/UBC-OCEAN/train.csv')\ndata.head().style.set_properties(**{'background-color':'blue','color':'white','border-color':'#8b8c8c'})","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.420232Z","iopub.execute_input":"2023-12-02T12:43:38.420584Z","iopub.status.idle":"2023-12-02T12:43:38.4449Z","shell.execute_reply.started":"2023-12-02T12:43:38.420555Z","shell.execute_reply":"2023-12-02T12:43:38.443981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.tail()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.447138Z","iopub.execute_input":"2023-12-02T12:43:38.447808Z","iopub.status.idle":"2023-12-02T12:43:38.461715Z","shell.execute_reply.started":"2023-12-02T12:43:38.447773Z","shell.execute_reply":"2023-12-02T12:43:38.460684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Exploratory Data Analysis**","metadata":{}},{"cell_type":"code","source":"sns.countplot(data['is_tma'])\nplt.xticks(rotation='vertical')","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.463763Z","iopub.execute_input":"2023-12-02T12:43:38.464147Z","iopub.status.idle":"2023-12-02T12:43:38.718031Z","shell.execute_reply.started":"2023-12-02T12:43:38.464114Z","shell.execute_reply":"2023-12-02T12:43:38.717067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.info()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.719407Z","iopub.execute_input":"2023-12-02T12:43:38.719731Z","iopub.status.idle":"2023-12-02T12:43:38.740904Z","shell.execute_reply.started":"2023-12-02T12:43:38.719704Z","shell.execute_reply":"2023-12-02T12:43:38.739916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Dataset has five(5) and five hundred and thirtyeight row","metadata":{}},{"cell_type":"code","source":"data.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.743872Z","iopub.execute_input":"2023-12-02T12:43:38.744218Z","iopub.status.idle":"2023-12-02T12:43:38.756298Z","shell.execute_reply.started":"2023-12-02T12:43:38.744186Z","shell.execute_reply":"2023-12-02T12:43:38.755273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.tail(10)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.757734Z","iopub.execute_input":"2023-12-02T12:43:38.758167Z","iopub.status.idle":"2023-12-02T12:43:38.770934Z","shell.execute_reply.started":"2023-12-02T12:43:38.758129Z","shell.execute_reply":"2023-12-02T12:43:38.769964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.shape","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.772183Z","iopub.execute_input":"2023-12-02T12:43:38.772461Z","iopub.status.idle":"2023-12-02T12:43:38.780575Z","shell.execute_reply.started":"2023-12-02T12:43:38.772437Z","shell.execute_reply":"2023-12-02T12:43:38.779581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.size","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.782069Z","iopub.execute_input":"2023-12-02T12:43:38.782378Z","iopub.status.idle":"2023-12-02T12:43:38.790775Z","shell.execute_reply.started":"2023-12-02T12:43:38.782347Z","shell.execute_reply":"2023-12-02T12:43:38.789925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.dtypes","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.792085Z","iopub.execute_input":"2023-12-02T12:43:38.792412Z","iopub.status.idle":"2023-12-02T12:43:38.801812Z","shell.execute_reply.started":"2023-12-02T12:43:38.792367Z","shell.execute_reply":"2023-12-02T12:43:38.800795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aside the image_id we have image_width and image_height as integers while, is_tma is boolen","metadata":{}},{"cell_type":"code","source":"data.isna()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.803124Z","iopub.execute_input":"2023-12-02T12:43:38.803423Z","iopub.status.idle":"2023-12-02T12:43:38.824211Z","shell.execute_reply.started":"2023-12-02T12:43:38.803399Z","shell.execute_reply":"2023-12-02T12:43:38.822761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.isna().values.any()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.825733Z","iopub.execute_input":"2023-12-02T12:43:38.826103Z","iopub.status.idle":"2023-12-02T12:43:38.833639Z","shell.execute_reply.started":"2023-12-02T12:43:38.82606Z","shell.execute_reply":"2023-12-02T12:43:38.832683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No missing value found","metadata":{}},{"cell_type":"code","source":"type(data.is_tma)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.835205Z","iopub.execute_input":"2023-12-02T12:43:38.83599Z","iopub.status.idle":"2023-12-02T12:43:38.843578Z","shell.execute_reply.started":"2023-12-02T12:43:38.835953Z","shell.execute_reply":"2023-12-02T12:43:38.842541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"type(data.image_width)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.844774Z","iopub.execute_input":"2023-12-02T12:43:38.845153Z","iopub.status.idle":"2023-12-02T12:43:38.852872Z","shell.execute_reply.started":"2023-12-02T12:43:38.845127Z","shell.execute_reply":"2023-12-02T12:43:38.851825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"type(data.image_height)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.858241Z","iopub.execute_input":"2023-12-02T12:43:38.85856Z","iopub.status.idle":"2023-12-02T12:43:38.865733Z","shell.execute_reply.started":"2023-12-02T12:43:38.858533Z","shell.execute_reply":"2023-12-02T12:43:38.864618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[data['image_id']==data['image_id'].max()]['image_height']","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.867228Z","iopub.execute_input":"2023-12-02T12:43:38.868027Z","iopub.status.idle":"2023-12-02T12:43:38.877998Z","shell.execute_reply.started":"2023-12-02T12:43:38.867971Z","shell.execute_reply":"2023-12-02T12:43:38.876933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data[data['image_id']==data['image_id'].max()]['image_width']","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.879373Z","iopub.execute_input":"2023-12-02T12:43:38.879787Z","iopub.status.idle":"2023-12-02T12:43:38.889857Z","shell.execute_reply.started":"2023-12-02T12:43:38.87975Z","shell.execute_reply":"2023-12-02T12:43:38.889055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.describe()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.891045Z","iopub.execute_input":"2023-12-02T12:43:38.891359Z","iopub.status.idle":"2023-12-02T12:43:38.913722Z","shell.execute_reply.started":"2023-12-02T12:43:38.891335Z","shell.execute_reply":"2023-12-02T12:43:38.912789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.describe().T","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.914897Z","iopub.execute_input":"2023-12-02T12:43:38.915282Z","iopub.status.idle":"2023-12-02T12:43:38.939692Z","shell.execute_reply.started":"2023-12-02T12:43:38.915248Z","shell.execute_reply":"2023-12-02T12:43:38.938853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(data[\"image_width\"])\nplt.title('image_width')\nplt.show()\nQ3 = data[\"image_width\"].quantile(0.75)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:38.940666Z","iopub.execute_input":"2023-12-02T12:43:38.940916Z","iopub.status.idle":"2023-12-02T12:43:39.588592Z","shell.execute_reply.started":"2023-12-02T12:43:38.940894Z","shell.execute_reply":"2023-12-02T12:43:39.587634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(data[\"image_height\"])\nplt.title('image_height')\nplt.show()\nQ3 = data[\"image_height\"].quantile(0.75)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:39.590164Z","iopub.execute_input":"2023-12-02T12:43:39.590882Z","iopub.status.idle":"2023-12-02T12:43:40.217923Z","shell.execute_reply.started":"2023-12-02T12:43:39.590838Z","shell.execute_reply":"2023-12-02T12:43:40.216781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.boxplot(data['image_width'])\nplt.title('image_width')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:40.219597Z","iopub.execute_input":"2023-12-02T12:43:40.219938Z","iopub.status.idle":"2023-12-02T12:43:40.523248Z","shell.execute_reply.started":"2023-12-02T12:43:40.219909Z","shell.execute_reply":"2023-12-02T12:43:40.522087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nplt.boxplot(data['image_height'])\nplt.title('image_height')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:40.524733Z","iopub.execute_input":"2023-12-02T12:43:40.525136Z","iopub.status.idle":"2023-12-02T12:43:40.7711Z","shell.execute_reply.started":"2023-12-02T12:43:40.525101Z","shell.execute_reply":"2023-12-02T12:43:40.769955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nsns.countplot(data['is_tma'])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:40.77248Z","iopub.execute_input":"2023-12-02T12:43:40.772875Z","iopub.status.idle":"2023-12-02T12:43:40.991815Z","shell.execute_reply.started":"2023-12-02T12:43:40.772837Z","shell.execute_reply":"2023-12-02T12:43:40.990604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nsns.pairplot(data,vars=['image_width', 'image_height', 'image_id'], hue='is_tma')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:40.993199Z","iopub.execute_input":"2023-12-02T12:43:40.993615Z","iopub.status.idle":"2023-12-02T12:43:45.521664Z","shell.execute_reply.started":"2023-12-02T12:43:40.993572Z","shell.execute_reply":"2023-12-02T12:43:45.520667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Class Distribution\nclass_distribution = data['label'].value_counts()\nprint(class_distribution)\n\n# TMA Distribution\ntma_distribution = data['is_tma'].value_counts()\nprint(tma_distribution)\n\n# Correlation between Image Dimensions\ncorrelation = data[['image_width', 'image_height']].corr()\nprint(correlation)\n\n# Visualization\nplt.figure(figsize=(9, 4))\nsns.scatterplot(x='image_width', y='image_height', data=data, hue='label')\nplt.title('Scatter plot of Image Dimensions', fontsize = 14, fontweight = 'bold', color = 'darkblue')\nplt.savefig('Scatter plot of Image Dimensions.png')\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:45.522884Z","iopub.execute_input":"2023-12-02T12:43:45.523208Z","iopub.status.idle":"2023-12-02T12:43:46.344357Z","shell.execute_reply.started":"2023-12-02T12:43:45.52318Z","shell.execute_reply":"2023-12-02T12:43:46.343391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for label in ['HGSC', 'CC', 'EC', 'LGSC', 'MC']:\n    data_tmp = data[data['label']==label]\n    image_id_list = list(data_tmp[data_tmp['is_tma']]['image_id'])#+list(df_tmp[~df_tmp['is_tma']].sample(1)['image_id'])\n    plt.figure(figsize=(20.0, 6.0))\n    for i in range(len(image_id_list)):\n        image_id = image_id_list[i]\n        plt.subplot(1, 5, i+1)\n        plt.title(f'image_id:{image_id}  ({label})', fontsize=14)\n        io.imshow(f'/kaggle/input/UBC-OCEAN/train_images/{image_id}.png')\n        plt.tick_params(labelbottom=False, labelleft=False, labelright=False, labeltop=False, bottom=False, left=False, right=False, top=False)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:43:46.345736Z","iopub.execute_input":"2023-12-02T12:43:46.346047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# HGSC","metadata":{}},{"cell_type":"code","source":"def show_krim(label):\n    df_tmp = data[data['label']==label]\n    image_id_list = list(data[~data['is_tma']]['image_id'])\n    i = 0\n    for image_id in image_id_list:\n        if i == 0:\n            plt.figure(figsize=(10.0, 6.0))\n\n        plt.subplot(1, 5, i + 1)\n        plt.title(f'image_id:{image_id} ({label})', fontsize=12)\n        io.imshow(f'/kaggle/input/ubc-reduced-png-2964x2964/{image_id}.png')\n        plt.tick_params(labelbottom=False, labelleft=False, labelright=False, labeltop=False, bottom=False, left=False, right=False, top=False)\n        \n        i += 1\n        if (i == 5) | (image_id == image_id_list[-1]):\n            plt.show()\n            i = 0","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_krim('HGSC')","metadata":{"execution":{"iopub.status.idle":"2023-12-02T12:55:41.38542Z","shell.execute_reply.started":"2023-12-02T12:44:37.200677Z","shell.execute_reply":"2023-12-02T12:55:41.384594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CC","metadata":{}},{"cell_type":"code","source":"show_krim('CC')","metadata":{"execution":{"iopub.status.busy":"2023-12-02T12:55:41.386981Z","iopub.execute_input":"2023-12-02T12:55:41.38768Z","iopub.status.idle":"2023-12-02T13:05:47.253355Z","shell.execute_reply.started":"2023-12-02T12:55:41.387646Z","shell.execute_reply":"2023-12-02T13:05:47.252362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"EC","metadata":{}},{"cell_type":"code","source":"show_krim('EC')","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:05:47.254519Z","iopub.execute_input":"2023-12-02T13:05:47.254793Z","iopub.status.idle":"2023-12-02T13:15:54.011505Z","shell.execute_reply.started":"2023-12-02T13:05:47.25477Z","shell.execute_reply":"2023-12-02T13:15:54.010526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"LGSC","metadata":{}},{"cell_type":"code","source":"show_krim('LGSC')","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:15:54.012956Z","iopub.execute_input":"2023-12-02T13:15:54.013636Z","iopub.status.idle":"2023-12-02T13:26:00.826233Z","shell.execute_reply.started":"2023-12-02T13:15:54.0136Z","shell.execute_reply":"2023-12-02T13:26:00.82515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"MC","metadata":{}},{"cell_type":"code","source":"show_krim('MC')","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:26:00.827634Z","iopub.execute_input":"2023-12-02T13:26:00.828031Z","iopub.status.idle":"2023-12-02T13:36:08.098171Z","shell.execute_reply.started":"2023-12-02T13:26:00.827978Z","shell.execute_reply":"2023-12-02T13:36:08.097146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(8, 6))\nsns.heatmap(correlation, annot=True, cmap='coolwarm', fmt=\".2f\")\nplt.title('Correlation Heatmap', fontsize = 12, fontweight = 'bold', color = 'darkblue')\nplt.savefig('Correlation Heatmap.png')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:36:08.099439Z","iopub.execute_input":"2023-12-02T13:36:08.099747Z","iopub.status.idle":"2023-12-02T13:36:08.468718Z","shell.execute_reply.started":"2023-12-02T13:36:08.099721Z","shell.execute_reply":"2023-12-02T13:36:08.467732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12, 4))\nsns.barplot(x=class_distribution.index, y=class_distribution.values)\nplt.title('Class Distribution', fontsize=12, fontweight='bold', color='darkblue')\nplt.xlabel('Class Label', fontsize=12, fontweight='bold', color='darkblue')\nplt.ylabel('Count', fontsize=12, fontweight='bold', color='darkblue')\nplt.savefig('Class Distribution.png')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:36:08.469764Z","iopub.execute_input":"2023-12-02T13:36:08.470044Z","iopub.status.idle":"2023-12-02T13:36:08.810024Z","shell.execute_reply.started":"2023-12-02T13:36:08.470003Z","shell.execute_reply":"2023-12-02T13:36:08.809012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"classes = data['label'].unique().tolist()\nclasses.append(\"Other\")\nclasses","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:36:08.811282Z","iopub.execute_input":"2023-12-02T13:36:08.811584Z","iopub.status.idle":"2023-12-02T13:36:08.818256Z","shell.execute_reply.started":"2023-12-02T13:36:08.811557Z","shell.execute_reply":"2023-12-02T13:36:08.817293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"###Lets us define propabality which is to be use later when we are going to preidct random choice\nproba = [0.4,0.2,0.1,0.075,0.075,0.15]","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:36:08.819488Z","iopub.execute_input":"2023-12-02T13:36:08.819812Z","iopub.status.idle":"2023-12-02T13:36:08.827338Z","shell.execute_reply.started":"2023-12-02T13:36:08.81977Z","shell.execute_reply":"2023-12-02T13:36:08.826343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"###loading thumbnails (is_tma == False), and resize them into (128,128)\n\n\n\nimages = []\nlabels = []\nfor (i, row) in tqdm(data.iterrows(), total = 538):\n    if row['is_tma']:\n        continue\n    img = io.imread(os.path.join(CFG.TRAIN_THUMBNAILS_PATH, str(row['image_id'])+\"_thumbnail.png\"))\n    img = rgb2gray(img)\n    img = resize(img, (128,128), anti_aliasing=False)\n    img = img.reshape(128,128,1)\n    images.append(img)\n    labels.append(row['label'])","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:36:08.828497Z","iopub.execute_input":"2023-12-02T13:36:08.828955Z","iopub.status.idle":"2023-12-02T13:38:34.629557Z","shell.execute_reply.started":"2023-12-02T13:36:08.828923Z","shell.execute_reply":"2023-12-02T13:38:34.628639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"images = np.array(images)\nlabels = np.array(labels)\n\nX = images.copy()\n\nencoder = OneHotEncoder(sparse_output = False)\nlabel_encoded = encoder.fit_transform(labels.reshape(-1,1), )\nY = label_encoded\n\nmodel = Sequential()\n\nmodel.add(Conv2D(filters=64, kernel_size=3,strides=(2,1), padding='same', activation='relu', input_shape=(128,128,1)))\nmodel.add(Flatten())\nmodel.add(Dense(64, activation='relu'))\nmodel.add(Dense(32, activation='relu'))\nmodel.add(Dense(5, activation='softmax'))\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n\nhistory = model.fit(X, Y, batch_size=8, epochs=10)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:38:34.630999Z","iopub.execute_input":"2023-12-02T13:38:34.631313Z","iopub.status.idle":"2023-12-02T13:38:51.688773Z","shell.execute_reply.started":"2023-12-02T13:38:34.631288Z","shell.execute_reply":"2023-12-02T13:38:51.687945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.save(\"model.h5\")","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:38:51.690096Z","iopub.execute_input":"2023-12-02T13:38:51.690378Z","iopub.status.idle":"2023-12-02T13:38:52.513627Z","shell.execute_reply.started":"2023-12-02T13:38:51.690355Z","shell.execute_reply":"2023-12-02T13:38:52.51245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The training data is not provided while submitting the notebook for testing, thus the Sample submission code provided will not work\n\nPlease refer to this notebook for inference.","metadata":{}},{"cell_type":"code","source":"test_data = pd.read_csv(\"/kaggle/input/UBC-OCEAN/test.csv\")\ntest_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:38:52.517705Z","iopub.execute_input":"2023-12-02T13:38:52.518613Z","iopub.status.idle":"2023-12-02T13:38:52.531809Z","shell.execute_reply.started":"2023-12-02T13:38:52.518583Z","shell.execute_reply":"2023-12-02T13:38:52.530909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.random.seed(40)\nthumbnails = os.listdir(CFG.TEST_THUMBNAILS_PATH)\npreds = []\nfor (i, row) in tqdm(test_data.iterrows()):\n    if str(row['image_id'])+\"_thumbnail.png\" not in thumbnails:\n        print(row['id'])\n        preds.append(np.random.choice(classes, p = proba))\n    img = io.imread(os.path.join(CFG.TEST_THUMBNAILS_PATH, str(row['image_id'])+\"_thumbnail.png\"))\n    img = rgb2gray(img)\n    img = resize(img, (128,128), anti_aliasing=False)\n    img = img.reshape(128,128,1)\n    prediction = encoder.inverse_transform(model.predict(np.array([img])))[0][0]\n    preds.append(prediction)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:38:52.538354Z","iopub.execute_input":"2023-12-02T13:38:52.538634Z","iopub.status.idle":"2023-12-02T13:38:53.013543Z","shell.execute_reply.started":"2023-12-02T13:38:52.538611Z","shell.execute_reply":"2023-12-02T13:38:53.012624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = pd.read_csv(\"/kaggle/input/UBC-OCEAN/sample_submission.csv\")\nsample_submission['label'] = preds\nsample_submission","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:38:53.014941Z","iopub.execute_input":"2023-12-02T13:38:53.015326Z","iopub.status.idle":"2023-12-02T13:38:53.02993Z","shell.execute_reply.started":"2023-12-02T13:38:53.01529Z","shell.execute_reply":"2023-12-02T13:38:53.029051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission.to_csv('submission.csv', index = False)","metadata":{"execution":{"iopub.status.busy":"2023-12-02T13:38:53.031235Z","iopub.execute_input":"2023-12-02T13:38:53.03188Z","iopub.status.idle":"2023-12-02T13:38:53.038961Z","shell.execute_reply.started":"2023-12-02T13:38:53.031846Z","shell.execute_reply":"2023-12-02T13:38:53.038178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}