{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Feature Clustering based Discriminatory Tile Feaure Extractor training - an MIL approach\n<br>\n\nAn MIL approach is selected for given classification problem of ischemic stroke type. Specifically a 2-step methodology is used:\n\n1) Train a feature extractor that can discriminate between critical tiles from the two classes.\n\n2) Aggregate tile level features to image slide level features and train a classifier for final inference.\n\nThis notebook contains step 1) details, with step 2) summarized at the end.\n\n\nThe main problem that the MIL approach faces is that of having tiles with slide level target labels for training an instance based or feature aggregation based classifier for slide class prediction. Especially for noisy data, like the one in this competition, it is critical to address this issue carefully. In this context, a clustering based discrimination between tiles from the two classes is implemented, mostly inspired by the idea in [this](https://arxiv.org/pdf/2206.08861.pdf) paper. The main idea is that we try to assign positive and negative pseudo labels to tiles based on their distance from the competing class tile level features. In addition, a critical vs non-critical tile discrimination is also implemented using smallest distance as well as out-of-class tiles.\n\nWe know that CE class slide will have at least one positive CE tile and LAA class will have all negative CE tiles - and vice-versa. So, CE class features are clustered (using kmeans) along with OTHER (out-of-class) features and LAA tiles are scored on their distance from these clusters. The farthest tiles are positively labelled (meaning they represent discriminatory signal for LAA class) and closest to competing class is negatively labelled (meaning they have negligible signal for LAA class). Same is done for the CE tiles against LAA+OTHER feature clusters. The pseudo labels are assigned as:\n\n- Original target labels for all tiles: CE - [0,1], LAA - [1,0], OTHER - [0,0]\n- Top 20% (farthest) distance tiles for CE are labelled [0,1], bottom 10% (closest) distance tiles for CE are labelled [0,0] -- similar process for LAA class tiles pseudo label assignment.\n- OTHER tiles and [0,0] pseudo labelled tiles represent non-critical tile targets and help discriminate between tiles that help predict a class and those that do not.\n- Distance scores are based on minimum euclidean from any competing class (+OTHER) cluster center.\n\n\n<br>\nFinally, from the trained network, a feature extractor is saved and used for creating tile feature embeddings, which are ultilized for training a slide image level classifier.\n","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \n\nimport cv2\nimport tifffile as tifi\nfrom tqdm.notebook import tqdm\n\nimport tensorflow as tf\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.applications import EfficientNetB0\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn import metrics\n\nfrom scipy.spatial import distance\nfrom scipy.special import softmax\n\nimport matplotlib.pylab as plt\nimport seaborn as sns\n\nimport gc\nimport os\nimport warnings\nwarnings.filterwarnings(\"ignore\")\ngc.enable()\ntf.version.VERSION","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:48:30.184367Z","iopub.execute_input":"2022-10-08T18:48:30.18491Z","iopub.status.idle":"2022-10-08T18:48:36.30714Z","shell.execute_reply.started":"2022-10-08T18:48:30.184801Z","shell.execute_reply":"2022-10-08T18:48:36.306135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Selection\n\nTiles have been created at slide image down scale factor of 6, and selected via two steps:\n\n1) A simple threshold for image array elements unique values and counts - to remove tiles that have large bakcground or monochrome regions.\n\n2) Tiles from step 1) are run through EfficientNetB0 with ImageNet weights and scored. Top 50-60% scoring tiles are selected for final training.\n\nFor train-validation split, a 90-10 split case is presented here. Random sampling of CE class images is used to address the class imbalance. LAA class already contains extra augmented tiles, created by padding slide image and offsetting tile location in image by tile_size/2 in both directions.","metadata":{}},{"cell_type":"code","source":"df_train_data = pd.read_csv('../input/mayo-clinic-strip-ai/train.csv')\ndf_tiles = pd.read_csv('../input/fork-of-ds6-select-tiles-folds-mayo-clinic-strip-a/selected_tiles.csv')","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:48:40.755499Z","iopub.execute_input":"2022-10-08T18:48:40.756737Z","iopub.status.idle":"2022-10-08T18:48:40.820225Z","shell.execute_reply.started":"2022-10-08T18:48:40.756698Z","shell.execute_reply":"2022-10-08T18:48:40.819115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dfout = df_train_data['label'].replace({'LAA':0, 'CE':1}).to_numpy()\ndfstrat = df_train_data[['label', 'center_id']]\ndf_train1, df_valid, df1, y_valid = train_test_split(df_train_data, dfout, test_size=0.1, \n                                                         stratify=dfstrat)\n\nprint(df_train1.shape, df_valid.shape)\nprint(df_train1['label'].value_counts(), df_valid['label'].value_counts())\n\n# print(df_train1['center_id'].value_counts(), df_valid['center_id'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:48:42.160947Z","iopub.execute_input":"2022-10-08T18:48:42.161381Z","iopub.status.idle":"2022-10-08T18:48:42.195064Z","shell.execute_reply.started":"2022-10-08T18:48:42.161345Z","shell.execute_reply":"2022-10-08T18:48:42.194083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dce = df_train1[df_train1['label'] == 'CE']\ndlaa = df_train1[df_train1['label'] == 'LAA']\n\n## -- adding augmented image row for laa class (augmented tiles already created) --\ndf_aug = dlaa.sample(frac=1)\nfor i,filename in enumerate(tqdm(df_aug['image_id'])):\n    \n    if len(df_tiles[df_tiles['image_id'] == filename]) > 0:\n        dat = df_aug.iloc[[i]]\n        dat['image_id'] = filename+'a'\n        dlaa = pd.concat([dlaa, dat], axis=0, ignore_index=True)\n\ndf_train = pd.concat([dce.sample(frac=0.85), dlaa], axis=0, ignore_index=True)\nprint(df_train.shape)\nprint(df_train['label'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:48:43.591499Z","iopub.execute_input":"2022-10-08T18:48:43.592163Z","iopub.status.idle":"2022-10-08T18:48:44.266055Z","shell.execute_reply.started":"2022-10-08T18:48:43.592128Z","shell.execute_reply":"2022-10-08T18:48:44.265214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## -- training data tiles selection from all tiles, including augumented images --\n\ndf_train_feat = pd.DataFrame(data=None, columns=df_tiles.columns)\n\nfor i,filename in enumerate(tqdm(df_train['image_id'])):\n    dat = df_tiles[df_tiles['image_id'] == filename]\n    df_train_feat = pd.concat([df_train_feat, dat], axis=0, ignore_index=True)\n\nprint(df_train_feat.shape)\nprint(df_train_feat['label'].value_counts())\n# df_train_feat.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:48:46.759657Z","iopub.execute_input":"2022-10-08T18:48:46.760054Z","iopub.status.idle":"2022-10-08T18:48:49.307495Z","shell.execute_reply.started":"2022-10-08T18:48:46.760019Z","shell.execute_reply":"2022-10-08T18:48:49.306536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## -- validation data tiles selection from all tiles, excluding augumented images --\n\ndf_valid_feat = pd.DataFrame(data=None, columns=df_tiles.columns)\n\nfor i,filename in enumerate(tqdm(df_valid['image_id'])):\n    dat = df_tiles[df_tiles['image_id'] == filename]\n    df_valid_feat = pd.concat([df_valid_feat, dat], axis=0, ignore_index=True)\n        \nprint(df_valid_feat.shape)\nprint(df_valid_feat['label'].value_counts())\n# df_valid_feat.head()","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:48:50.348698Z","iopub.execute_input":"2022-10-08T18:48:50.349062Z","iopub.status.idle":"2022-10-08T18:48:50.629391Z","shell.execute_reply.started":"2022-10-08T18:48:50.349029Z","shell.execute_reply":"2022-10-08T18:48:50.628439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## -- dataframes of all tiles used for every clustering step --\n\ndf_ce = df_train_feat[df_train_feat['label'] == 'CE']\ndf_laa = df_train_feat[df_train_feat['label'] == 'LAA']\ndf_other = df_tiles[df_tiles['label'] == 'Other']\nprint(df_ce.shape, df_laa.shape, df_other.shape)","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:48:53.33476Z","iopub.execute_input":"2022-10-08T18:48:53.335112Z","iopub.status.idle":"2022-10-08T18:48:53.350498Z","shell.execute_reply.started":"2022-10-08T18:48:53.335082Z","shell.execute_reply":"2022-10-08T18:48:53.349453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model and Training\n\n## Model definition\nPretrained EfficientNetB0 model with ImageNet weights is selected. The last FC classification head is trained using *BinaryCrossEntropy* loss and *sigmoid* activation. *Softmax* is actively avoided because we want the model to have the ability to output a low probability for both classes in case tile is deemed non-critical or not discriminatory enough.\n\nFrom the main model, a feature extractor is derived from input layer to the n-1 layer of shape (1,1280) for creating features from tiles to be used for clustering.\n\n<br>\n\n## Training\nTraining commences with slide level labels for all tiles - clustering, pseudo labelling and top-bottom tile pick triggered for subsequent epochs. A random sampling of 0.7 fraction is implemented to select tiles from top and bottom scoring tile pool to avoid same extreme scoring tiles from creating overfit problems.\n\n\nIt was observed from [this](https://arxiv.org/pdf/1902.06543.pdf) paper that adjusting hue of the image, even a little bit, resulted in better chance of predicting unseen strains better. So a random hue augmentation is implemented, based on stretching/compressing the hue channel of each tile randomly by a factor betweem (0.85, 1.15) - *hue_augument* function, applied to each tile before training.\n","metadata":{}},{"cell_type":"code","source":"TILE_SIZE = 224\nNUM_CLASSES = 2\n\n## -- TF EfficientNetB0 model --\ninputs = tf.keras.Input(shape=(TILE_SIZE, TILE_SIZE, 3))\n\nmodel_frozen = EfficientNetB0(include_top=False, input_tensor=inputs, weights=\"imagenet\")\nmodel_frozen.trainable = True\n\nx = tf.keras.layers.GlobalAveragePooling2D(name=\"avg_pool\")(model_frozen.output)\n\ntop_dropout_rate = 0.2\nx = tf.keras.layers.Dropout(top_dropout_rate, name=\"top_dropout\")(x)\noutputs = tf.keras.layers.Dense(NUM_CLASSES, activation=\"sigmoid\", name=\"pred\")(x)\n\nmodel = tf.keras.Model(inputs, outputs, name=\"EffB0_clust_feat_pseudo_label_Mayo\")\n\noptimizer = tf.keras.optimizers.Adam(learning_rate=8e-04)\nmodel.compile(optimizer=optimizer,\n              loss='binary_crossentropy', \n              metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:49:15.781853Z","iopub.execute_input":"2022-10-08T18:49:15.782229Z","iopub.status.idle":"2022-10-08T18:49:17.52655Z","shell.execute_reply.started":"2022-10-08T18:49:15.782195Z","shell.execute_reply":"2022-10-08T18:49:17.52563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def hue_augument(img):\n    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)\n    f = np.random.uniform(low=0.85, high=1.15, size=1)[0]\n    hsv[:,:,0] = hsv[:,:,0]*f\n    img_aug = cv2.cvtColor(hsv, cv2.COLOR_HSV2BGR)\n    return img_aug\n\ndef loss_of_log(targ, pred):\n    ll = 0\n    for i in range(len(targ)):\n        ll += -(targ[i][0]*np.log(pred[i][0]+1e-7) + targ[i][1]*np.log(pred[i][1]+1e-7))\n        \n    return ll/len(targ)","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:49:19.501598Z","iopub.execute_input":"2022-10-08T18:49:19.501966Z","iopub.status.idle":"2022-10-08T18:49:19.509415Z","shell.execute_reply.started":"2022-10-08T18:49:19.501933Z","shell.execute_reply":"2022-10-08T18:49:19.508377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## -- defining feature extractor and training --\ndef training(df, bts=64, train=False):\n    \n    n = 8000\n    if train == True:\n        for k in range(len(df)//n+1):\n            dff = df.iloc[k*n : min((k+1)*n, len(df))]\n\n            gc.collect()\n            ## -- reading tiles for training set with hue augumentation --\n            x_dat = np.zeros((len(dff),TILE_SIZE,TILE_SIZE,3), dtype=np.uint8)\n            y_dat = np.zeros((len(dff),2))\n            smpl_wts = np.ones(len(dff))\n\n            for i,f_path in enumerate(dff['path']):\n                im = cv2.imread(f_path)\n                x_dat[i] = hue_augument(im)\n\n            for i,labl in enumerate(dff['label']):\n                if labl == 'LAA':\n                    y_dat[i,0] = 1\n                    smpl_wts[i] = 1.05\n                elif labl == 'CE':\n                    y_dat[i,1] = 1\n                    smpl_wts[i] = 0.9\n                                     \n            history = model.fit(x_dat, y_dat, \n                                batch_size=bts,\n                                sample_weight=smpl_wts,\n                                epochs=1)     \n            gc.collect()\n        \n    feat_ext = tf.keras.Model(inputs=model.inputs,\n                               outputs=model.get_layer(name=\"avg_pool\").output, \n                              name='feature_extractor'\n                             )\n    wt = model.get_layer(name='pred').get_weights()\n    return wt, feat_ext\n\n## -- creating features for clustering -- \ndef featurize(df, feat_ext, status_bar=True):\n    fsize = feat_ext.layers[-1].get_output_at(0).get_shape().as_list()[1]\n    features = np.zeros((len(df),fsize))\n\n    if status_bar == True:\n        for i,path in enumerate(tqdm(df['path'])):\n            tile = cv2.imread(path)\n            features[i] = feat_ext.predict(tile.reshape(-1,TILE_SIZE,TILE_SIZE,3))\n    else:\n        for i,path in enumerate(df['path']):\n            tile = cv2.imread(path)\n            features[i] = feat_ext.predict(tile.reshape(-1,TILE_SIZE,TILE_SIZE,3))\n    \n    return features","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:49:21.082044Z","iopub.execute_input":"2022-10-08T18:49:21.082424Z","iopub.status.idle":"2022-10-08T18:49:21.098932Z","shell.execute_reply.started":"2022-10-08T18:49:21.082373Z","shell.execute_reply":"2022-10-08T18:49:21.097912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def pseudo_label_pick(df, claas, pp, np):\n    df_pos = pd.DataFrame(data=None, columns=df.columns)\n    df_neg = pd.DataFrame(data=None, columns=df.columns)\n    for f,filename in enumerate(df_train['image_id']):\n        \n        if df_train['label'].iloc[f] == claas:\n            dff = df[df['image_id'] == filename]\n            if len(dff) < 10:\n                pick_p = 2\n                pick_n = 1\n                samp_frac = 1\n            else:\n                pick_p = int(len(dff)*pp)\n                pick_n = int(len(dff)*np)\n                samp_frac = 1\n                \n            df_p = dff.sort_values(by=['distance_min'], ascending=False).iloc[:pick_p]\n            df_pos = pd.concat([df_pos, df_p.sample(frac=samp_frac)], axis=0, ignore_index=True)\n            \n            if len(dff) > 2:\n                df_n = dff.sort_values(by=['distance_min'], ascending=True).iloc[:pick_n]\n                df_n['label'] = 'negative'\n                df_neg = pd.concat([df_neg, df_n.sample(frac=samp_frac)], axis=0, ignore_index=True)\n            \n    return df_pos, df_neg","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:49:23.405712Z","iopub.execute_input":"2022-10-08T18:49:23.406257Z","iopub.status.idle":"2022-10-08T18:49:23.421355Z","shell.execute_reply.started":"2022-10-08T18:49:23.406217Z","shell.execute_reply":"2022-10-08T18:49:23.418876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Evaluation \n\nThe model is trained with tile level labels, which keep chaning based on feature cluster distances. The features themselves keep changing based on each training epoch, while there is no image level aggregaton during training to guide the training process. To address this issue, a simple evaluation metric is implemented to ascertain the training performance of the network and stop training.\n\nA maxpooling AUC score is used - validation set AUC scores at image level, obtained from maximum class predictions probabilities for each class from any tile within the image. AUC was used as a metric because the purpose of the feature extractor is to be able to differentiate between the two classes, and log loss is only to monitor if one class is overfitting. Training is stopped when validation AUC scores reached a good value after at least a few clustering rounds.","metadata":{}},{"cell_type":"code","source":"def feat_to_logit(df, features, wt):\n    df['CE'] = 0.5\n    df['LAA'] = 0.5\n    for i,ft in enumerate(features):\n        pred = tf.keras.activations.sigmoid(np.dot(ft,wt[0]) + wt[1]).numpy()\n        df['LAA'].iloc[i] = pred[0]\n        df['CE'].iloc[i] = pred[1]\n        \n    return df\n\ndef maxpooling(df, df_feat):\n    df['LAA_max'] = 0.5\n    df['CE_max'] = 0.5\n    for f,filename in enumerate(df['image_id']):\n        dff = df_feat[df_feat['image_id'] == filename]\n        if len(dff) > 0:\n            df['LAA_max'].iloc[f] = dff['LAA'].to_numpy().max()\n            df['CE_max'].iloc[f] = dff['CE'].to_numpy().max()\n        \n    return df","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:49:26.23152Z","iopub.execute_input":"2022-10-08T18:49:26.232226Z","iopub.status.idle":"2022-10-08T18:49:26.242618Z","shell.execute_reply.started":"2022-10-08T18:49:26.23219Z","shell.execute_reply":"2022-10-08T18:49:26.241476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_cnn = pd.concat([df_ce.sample(frac=1),\n                    df_laa.sample(frac=1), \n                    df_other.sample(frac=0.7)], axis=0, ignore_index=True)\ndf_cnn = df_cnn.sample(frac=1)\nprint(df_cnn.shape)","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:49:27.712188Z","iopub.execute_input":"2022-10-08T18:49:27.712578Z","iopub.status.idle":"2022-10-08T18:49:27.732644Z","shell.execute_reply.started":"2022-10-08T18:49:27.712543Z","shell.execute_reply":"2022-10-08T18:49:27.731392Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LL_valid = []\nLL_ce = []\nLL_laa = []\nAUC_valid = []\n\nclusters = 12\niters = 3","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:49:40.989149Z","iopub.execute_input":"2022-10-08T18:49:40.989563Z","iopub.status.idle":"2022-10-08T18:49:40.995342Z","shell.execute_reply.started":"2022-10-08T18:49:40.989526Z","shell.execute_reply":"2022-10-08T18:49:40.994292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Main training loop -- train -> create features from extractor -> cluster LAA+Other and CE+Other -> score CE class feature on euclidean distance from LAA+Other clusters and vice-versa -> pick tiles based on score with pseudo labelling -> repeat","metadata":{}},{"cell_type":"code","source":"for itr in range(iters):\n\n    print(f'\\n --- ITERATION : {itr+1} ---')\n        \n    bts = 64\n    if itr == 0:\n        bts = 120\n        \n    wt, feat_ext = training(df_cnn, bts=bts, train=True)\n    \n    \n    ## -- tile feature embedding --\n    \n    print('Other -- feature embedding...')\n    features_other = featurize(df_other, feat_ext)\n    print('LAA -- feature embedding...')\n    features_laa = featurize(df_laa, feat_ext)\n    print('CE -- feature embedding...')\n    features_ce = featurize(df_ce, feat_ext)\n\n\n    ## -- negative feature cluster data points --\n    \n    feat_laa_clust = np.concatenate((features_laa, features_other), axis=0)\n    feat_ce_clust = np.concatenate((features_ce, features_other), axis=0)\n\n\n    ## -- clustering --\n    \n    print('Clustering...')\n    kmeans_laa = KMeans(n_clusters=clusters, random_state=0, init='k-means++').fit(feat_laa_clust)\n    pca = PCA(n_components=2)\n    laa_pca = pca.fit_transform(feat_laa_clust)\n    df_laa_clusters = pd.DataFrame({'d1': laa_pca[:,0], 'd2': laa_pca[:,1], 'cluster': kmeans_laa.labels_})\n    uniq, counts = np.unique(kmeans_laa.labels_, return_counts=True)\n    print(f'LAA cluster counts -- {counts}')\n\n    kmeans_ce = KMeans(n_clusters=clusters, random_state=0, init='k-means++').fit(feat_ce_clust)\n    pca = PCA(n_components=2)\n    ce_pca = pca.fit_transform(feat_ce_clust)\n    df_ce_clusters = pd.DataFrame({'d1': ce_pca[:,0], 'd2': ce_pca[:,1], 'cluster': kmeans_ce.labels_})\n    uniq, counts = np.unique(kmeans_ce.labels_, return_counts=True)\n    print(f'CE cluster counts -- {counts}')\n\n    \n    ## -- plotting clusters in 2D space --\n    \n    fig, axes = plt.subplots(ncols=2, nrows=1, figsize=(20, 6))\n    axes[0].set_title('LAA clustering')\n    sns.scatterplot(data=df_laa_clusters, x=\"d1\", y=\"d2\", hue=\"cluster\", palette=\"deep\", ax=axes[0])\n\n    axes[1].set_title('CE clustering')\n    sns.scatterplot(data=df_ce_clusters, x=\"d1\", y=\"d2\", hue=\"cluster\", palette=\"deep\", ax=axes[1])\n    plt.show()\n\n\n    ## -- scoring against negative cluster - distance based scoring --\n    \n    dist_ce_min = np.zeros(len(features_ce))\n    for j,feat in enumerate(features_ce):\n        d = np.zeros(clusters)\n        for k,cc in enumerate(kmeans_laa.cluster_centers_):\n            d[k] = distance.euclidean(feat, cc)\n        dist_ce_min[j] = d.min()\n    df_ce['distance_min'] = dist_ce_min\n\n    \n    dist_laa_min = np.zeros(len(features_laa))\n    for j,feat in enumerate(features_laa):\n        d = np.zeros(clusters)\n        for k,cc in enumerate(kmeans_ce.cluster_centers_):\n            d[k] = distance.euclidean(feat, cc)\n        dist_laa_min[j] = d.min()\n    df_laa['distance_min'] = dist_laa_min\n\n    \n    ## -- picking top and bottom % distance based tiles as positive and negative --\n\n    if itr > 0:\n        df_ce_old = pd.concat([df_ce_pos, df_ce_neg], axis=0, ignore_index=True)\n        df_laa_old = pd.concat([df_laa_pos, df_laa_neg], axis=0, ignore_index=True)\n        \n    df_ce_pos, df_ce_neg = pseudo_label_pick(df_ce, 'CE', 0.2, 0.1)\n    df_laa_pos, df_laa_neg = pseudo_label_pick(df_laa, 'LAA', 0.2, 0.1)\n    \n    df_cnn = pd.concat([df_laa_pos.sample(frac=0.7),\n                        df_laa_neg.sample(frac=0.7), \n                        df_ce_pos.sample(frac=0.7), \n                        df_ce_neg.sample(frac=0.7), \n                        df_other.sample(frac=0.3)], axis=0, ignore_index=True)\n    df_cnn = df_cnn.sample(frac=1)\n    print(f'\\nData selected for next training iteration: {len(df_cnn)}')\n    \n    if itr > 0:\n        df_ce_new = pd.concat([df_ce_pos, df_ce_neg], axis=0, ignore_index=True)\n        df_laa_new = pd.concat([df_laa_pos, df_laa_neg], axis=0, ignore_index=True)\n        \n        n1 = len([x for x in df_ce_new['path'].to_list() if x not in df_ce_old['path'].to_list()])\n        n2 = len([x for x in df_laa_new['path'].to_list() if x not in df_laa_old['path'].to_list()])\n        print(f'Diff -- {n1+n2} new tiles picked')\n       \n    ### ___________________________\n    ### ------ EVALUATION ---------\n    ### ---------------------------\n\n    ## -- VALIDATION data evaluation metrics --\n\n    print('\\nValidation tiles -- feature embedding...')\n    features_valid = featurize(df_valid_feat, feat_ext)\n\n    df_valid_feat = feat_to_logit(df_valid_feat, features_valid, wt)\n    df_valid = maxpooling(df_valid, df_valid_feat)\n    \n    targ = to_categorical(df_valid['label'].replace({'LAA':0, 'CE':1}).to_numpy())\n    y_prob = df_valid[['LAA_max', 'CE_max']].to_numpy()\n    \n    dfc = df_valid[df_valid['label'] == 'CE']\n    yc = dfc[['LAA_max', 'CE_max']].to_numpy()\n    tc = np.zeros(yc.shape)\n    tc[:,1] = 1\n\n    dfl = df_valid[df_valid['label'] == 'LAA']\n    yl = dfl[['LAA_max', 'CE_max']].to_numpy()\n    tl = np.zeros(yl.shape)\n    tl[:,0] = 1\n\n    print('\\n -- MODEL : VALIDATION DATA METRICS --')\n    print('Maxpooling log loss: ', round(loss_of_log(targ, y_prob),4))\n    print('\\nlog loss CE: ', round(loss_of_log(tc, yc),4))\n    LL_ce.append(loss_of_log(tc, yc))\n    print('log loss LAA: ', round(loss_of_log(tl, yl),4))\n    LL_laa.append(loss_of_log(tl, yl))\n\n    y_cls = np.zeros(y_prob.shape)\n    for i,y in enumerate(y_prob):\n        if y[0] >= y[1]:\n            y_cls[i] = [1,0]\n        else:\n            y_cls[i] = [0,1]\n    print('Maxpooling AUC: ', round(metrics.roc_auc_score(targ, y_cls),4))\n    LL_valid.append(loss_of_log(targ, y_prob))\n    AUC_valid.append(metrics.roc_auc_score(targ, y_cls))\n    \n    gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-10-08T18:49:43.02213Z","iopub.execute_input":"2022-10-08T18:49:43.022532Z","iopub.status.idle":"2022-10-08T19:42:39.297534Z","shell.execute_reply.started":"2022-10-08T18:49:43.022497Z","shell.execute_reply":"2022-10-08T19:42:39.296468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nl = len(LL_valid)\nfig, axes = plt.subplots(ncols=2, nrows=1, figsize=(20, 6))\naxes[0].set_title('Log Loss')\naxes[0].plot(np.arange(nl), LL_valid[0:nl], label='mean')\naxes[0].plot(np.arange(nl), LL_ce[0:nl], label='CE')\naxes[0].plot(np.arange(nl), LL_laa[0:nl], label='LAA')\naxes[0].set_xlabel('epochs')\naxes[0].set_ylabel('log loss')\naxes[0].legend()\n\naxes[1].set_title('AUC score')\naxes[1].plot(np.arange(nl), AUC_valid[0:nl], label='valid')\naxes[1].set_xlabel('epochs')\naxes[1].set_ylabel('auc score')\naxes[1].legend()\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-10-08T20:09:06.058148Z","iopub.execute_input":"2022-10-08T20:09:06.058565Z","iopub.status.idle":"2022-10-08T20:09:06.409328Z","shell.execute_reply.started":"2022-10-08T20:09:06.058529Z","shell.execute_reply":"2022-10-08T20:09:06.408359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feat_ext.save('./')\nnp.save('pred_weights.npy', wt, allow_pickle=True)","metadata":{"execution":{"iopub.status.busy":"2022-10-08T20:09:11.73948Z","iopub.execute_input":"2022-10-08T20:09:11.739834Z","iopub.status.idle":"2022-10-08T20:09:41.642141Z","shell.execute_reply.started":"2022-10-08T20:09:11.739803Z","shell.execute_reply":"2022-10-08T20:09:41.641134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Aggregation and Classifier\n\nThe saved feature extractor provides tile level features that need to be aggregated to a slide level before classification. This is achieved by attention based aggreagation with attention to the tile feature with maximum logits, adapted from [this](https://arxiv.org/pdf/2011.08939.pdf) paper. A random forest classifier was trained on the aggregated features, which included max logits for the two classes are extra features. The aggreagation and inference portions can be found in [this notebook](https://www.kaggle.com/code/icemantd/new-predict-mil-logit-features-mayo-strip-ai/notebook).","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}