{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Problem state\n\nOvarian carcinoma is the most lethal cancer of the female reproductive system. There are five common subtypes of ovarian cancer: high-grade serous carcinoma, clear-cell ovarian carcinoma, endometrioid, low-grade serous, and mucinous carcinoma. Additionally, there are several rare subtypes (\"Outliers\"). These are all characterized by distinct cellular morphologies, etiologies, molecular and genetic profiles, and clinical attributes. Subtype-specific treatment approaches are gaining prominence, though first requires subtype identification, a process that could be improved with data science.\n\nCurrently, ovarian cancer diagnosis relies on pathologists to assess subtypes. However, this presents several challenges, including disagreements between observers and the reproducibility of diagnostics. Furthermore, underserved communities often lack access to specialist pathologists, and even well-developed communities face a shortage of pathologists with expertise in gynecologic malignancies.","metadata":{}},{"cell_type":"markdown","source":"# Goal\n\nThe goal of the competition is to identify subtypes of <a href='https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8774015/'>ovarian cancer</a>: CC, EC, HGSC, LGSC, MC using AI/ML/DL.","metadata":{}},{"cell_type":"markdown","source":"## Load libs","metadata":{}},{"cell_type":"code","source":"import os, warnings\nimport matplotlib.pyplot as plt\nfrom matplotlib import gridspec\nimport pandas as pd\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.keras.preprocessing import image_dataset_from_directory\n\n# Reproducability\ndef set_seed(seed=31415):\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    os.environ['TF_DETERMINISTIC_OPS'] = '1'\nset_seed(31415)\n\n# Set Matplotlib defaults\nplt.rc('figure', autolayout=True)\nplt.rc('axes', labelweight='bold', labelsize='large',\n       titleweight='bold', titlesize=18, titlepad=10)\nplt.rc('image', cmap='magma')\nwarnings.filterwarnings(\"ignore\") # to clean up output cells\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-10-17T11:13:44.737447Z","iopub.execute_input":"2023-10-17T11:13:44.737812Z","iopub.status.idle":"2023-10-17T11:13:53.718008Z","shell.execute_reply.started":"2023-10-17T11:13:44.737779Z","shell.execute_reply":"2023-10-17T11:13:53.71702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load data","metadata":{}},{"cell_type":"code","source":"train=pd.read_csv('/kaggle/input/UBC-OCEAN/train.csv')\ntest=pd.read_csv('/kaggle/input/UBC-OCEAN/test.csv')","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:53.720019Z","iopub.execute_input":"2023-10-17T11:13:53.720908Z","iopub.status.idle":"2023-10-17T11:13:53.746104Z","shell.execute_reply.started":"2023-10-17T11:13:53.720869Z","shell.execute_reply":"2023-10-17T11:13:53.745299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:53.747547Z","iopub.execute_input":"2023-10-17T11:13:53.747965Z","iopub.status.idle":"2023-10-17T11:13:53.77541Z","shell.execute_reply.started":"2023-10-17T11:13:53.74793Z","shell.execute_reply":"2023-10-17T11:13:53.774443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"*not nulls, perfect!*","metadata":{}},{"cell_type":"code","source":"test.info()","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:53.777813Z","iopub.execute_input":"2023-10-17T11:13:53.778678Z","iopub.status.idle":"2023-10-17T11:13:53.787423Z","shell.execute_reply.started":"2023-10-17T11:13:53.778645Z","shell.execute_reply":"2023-10-17T11:13:53.786424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. What is the target ratio?","metadata":{}},{"cell_type":"code","source":"train.label.value_counts().plot(kind='pie', autopct=\"%.1f%%\")","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:53.788935Z","iopub.execute_input":"2023-10-17T11:13:53.789542Z","iopub.status.idle":"2023-10-17T11:13:54.093676Z","shell.execute_reply.started":"2023-10-17T11:13:53.78951Z","shell.execute_reply":"2023-10-17T11:13:54.092784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**As expected, we have an unbalanced multi-classification problem**","metadata":{}},{"cell_type":"markdown","source":"# 2. What is the size of the images?","metadata":{}},{"cell_type":"code","source":"train.image_width.describe()","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:54.095062Z","iopub.execute_input":"2023-10-17T11:13:54.095571Z","iopub.status.idle":"2023-10-17T11:13:54.112533Z","shell.execute_reply.started":"2023-10-17T11:13:54.09554Z","shell.execute_reply":"2023-10-17T11:13:54.111549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.image_height.describe()","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:54.113931Z","iopub.execute_input":"2023-10-17T11:13:54.114426Z","iopub.status.idle":"2023-10-17T11:13:54.126495Z","shell.execute_reply.started":"2023-10-17T11:13:54.114395Z","shell.execute_reply":"2023-10-17T11:13:54.125434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.groupby('label')[['image_width','image_height']].describe().T","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:54.128065Z","iopub.execute_input":"2023-10-17T11:13:54.128684Z","iopub.status.idle":"2023-10-17T11:13:54.186653Z","shell.execute_reply.started":"2023-10-17T11:13:54.128652Z","shell.execute_reply":"2023-10-17T11:13:54.185667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Images are of adequate size**","metadata":{}},{"cell_type":"markdown","source":"# 3. What's the ratio of tma?\n\n<a link =\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2628443/\">Support material about TMA</a>","metadata":{}},{"cell_type":"code","source":"train.is_tma.value_counts().plot(kind='pie', autopct=\"%.1f%%\")","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:54.188272Z","iopub.execute_input":"2023-10-17T11:13:54.188815Z","iopub.status.idle":"2023-10-17T11:13:54.418779Z","shell.execute_reply.started":"2023-10-17T11:13:54.188784Z","shell.execute_reply":"2023-10-17T11:13:54.417809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We must not forget that this information can help with analysis, but it is not in the test, and accordingly it cannot be the feature for the model**","metadata":{}},{"cell_type":"markdown","source":"# 4. Review images by each labels","metadata":{}},{"cell_type":"code","source":"train_is_tma_false=train[train['is_tma']==False]","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:54.422133Z","iopub.execute_input":"2023-10-17T11:13:54.423175Z","iopub.status.idle":"2023-10-17T11:13:54.431254Z","shell.execute_reply.started":"2023-10-17T11:13:54.423141Z","shell.execute_reply":"2023-10-17T11:13:54.430014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_is_tma_false.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:54.432989Z","iopub.execute_input":"2023-10-17T11:13:54.434185Z","iopub.status.idle":"2023-10-17T11:13:54.450364Z","shell.execute_reply.started":"2023-10-17T11:13:54.434154Z","shell.execute_reply":"2023-10-17T11:13:54.449402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Images from train_thumbnails folder","metadata":{}},{"cell_type":"code","source":"by_label=train_is_tma_false.groupby('label')","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:54.451712Z","iopub.execute_input":"2023-10-17T11:13:54.452341Z","iopub.status.idle":"2023-10-17T11:13:54.457277Z","shell.execute_reply.started":"2023-10-17T11:13:54.45231Z","shell.execute_reply":"2023-10-17T11:13:54.456265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\npath = \"/kaggle/input/UBC-OCEAN/train_thumbnails\"\nj=1\nfor label in train['label'].unique():\n    plt.subplot(2,3,j)\n    image = plt.imread(path+'/'+str(train.iloc[by_label.groups[label][0]]['image_id'])+'_thumbnail.png')\n    image = plt.imshow(image)\n    plt.title(f\"Label:{label}\")\n    j+=1","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:13:54.458683Z","iopub.execute_input":"2023-10-17T11:13:54.459593Z","iopub.status.idle":"2023-10-17T11:14:05.114269Z","shell.execute_reply.started":"2023-10-17T11:13:54.459563Z","shell.execute_reply":"2023-10-17T11:14:05.113444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Images from train_images folder","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\npath = \"/kaggle/input/UBC-OCEAN/train_images\"\nj=1\nfor label in train['label'].unique():\n    plt.subplot(2,3,j)\n    image = plt.imread(path+'/'+str(train.iloc[by_label.groups[label][0]]['image_id'])+'.png')\n    image = plt.imshow(image)\n    plt.title(f\"Label:{label}\")\n    j+=1","metadata":{"execution":{"iopub.status.busy":"2023-10-17T11:14:05.11521Z","iopub.execute_input":"2023-10-17T11:14:05.115506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}