{
  "id": 465455,
  "title": "10th place solution",
  "url": "/competitions/UBC-OCEAN/discussion/465455",
  "author_name": "Gunes Evitan",
  "post_date": "2024-01-04T10:19:21.027000",
  "votes": 28,
  "comment_count": 19,
  "views": 0,
  "content": "<p>This was an interesting competition and I would like to thank my teammate <a href=\"https://www.kaggle.com/samfc10\" target=\"_blank\">@samfc10</a> and everyone involved with the organization of it.</p>\n<p>This is a simple textbook solution that heavily relies on external TMA data and strong labels. There is nothing special or novel in this pipeline.</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/gunesevitan/ubc-ocean-inference\" target=\"_blank\">Inference</a></li>\n<li><a href=\"https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started\" target=\"_blank\">libvips/pyvips Installation and Getting Started</a></li>\n<li><a href=\"https://www.kaggle.com/code/gunesevitan/ubc-ocean-jpeg-dataset-pipeline\" target=\"_blank\">UBC-OCEAN - JPEG Dataset Pipeline</a></li>\n<li><a href=\"https://www.kaggle.com/code/gunesevitan/ubc-ocean-eda\" target=\"_blank\">UBC-OCEAN - EDA</a></li>\n<li><a href=\"https://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset\" target=\"_blank\">UBC-OCEAN - Dataset</a></li>\n<li><a href=\"https://github.com/gunesevitan/ubc-ovarian-cancer-subtype-classification-and-outlier-detection\" target=\"_blank\">GitHub Repository</a></li>\n</ul>\n<h2>1. Raw Dataset</h2>\n<h3>WSI</h3>\n<p>Masks of WSIs are resized to thumbnail sizes. Tiles of WSIs and masks are extracted from their thumbnails with stride of 384 and they are padded to 512. A MaxViT Tiny FPN model is trained on those padded tiles and masks. Segmentation model outputs are activated with sigmoid and 3x TTA (horizontal, vertical and diagonal flip) are applied after the activation.</p>\n<p>Final segmentation mask prediction is blocky since the model was trained on tiles and merged later.</p>\n<p><img src=\"https://i.ibb.co/jg24x1H/Screenshot-from-2024-01-04-09-28-01.png\" alt=\"seg1\"></p>\n<p>Segmentation mask predictions are cast to 8-bit integer and upsampled to original WSI size with nearest neighbor interpolation.</p>\n<p><img src=\"https://i.ibb.co/ZHjtfmY/Screenshot-from-2024-01-04-09-31-42.png\" alt=\"seg2\"></p>\n<ul>\n<li>WSI and their mask predictions are cropped maximum number of times with stride of 1024.</li>\n<li>Crops are sorted based on their mask areas in descending order</li>\n<li>Top 16 crops are taken and WSI label is assigned to them</li>\n</ul>\n<h3>TMA</h3>\n<p>Rows and columns with low standard deviation are dropped on TMAs with the function below. The purpose of this preprocessing is removing white regions and making WSIs and TMAs as similar as possible. Using higher values of threshold were dropping areas in the tissue region so the standard deviation threshold is set to 10.</p>\n<pre><code> ():\n\n    \n\n    vertical_stds = image.std(axis=(, ))\n    horizontal_stds = image.std(axis=(, ))\n    cropped_image = image[vertical_stds &gt; threshold, :, :]\n    cropped_image = cropped_image[:, horizontal_stds &gt; threshold, :]\n\n     cropped_image\n</code></pre>\n<p><img src=\"https://i.ibb.co/8jCyhgG/4134-crop.png\" alt=\"seg2\"></p>\n<h2>2. Validation</h2>\n<p>Multi-label stratified kfold is used as the cross-validation scheme. Dataset is split into 5 folds. <code>label</code> and <code>is_tma</code> columns are used for stratification.</p>\n<h2>3. Models</h2>\n<p>EfficientNetV2 small model is used as the backbone with a regular classification head.</p>\n<h2>4. Training</h2>\n<p>CrossEntropyLoss with class weights are used as the loss function. Class weights are calculated as n / n ith class.</p>\n<p>AdamW optimizer is used with 0.0001 learning rate. Cosine annealing scheduler is used with 0.00001 minimum learning rate.</p>\n<p>AMP is also used for faster training and regularization.</p>\n<p>Each fold is trained for 15 epochs and epochs with the highest balanced accuracy are selected.</p>\n<p>Training transforms are:</p>\n<ul>\n<li>Resize TMAs to size 1024 (WSI crops are already 1024 sized)</li>\n<li>Magnification normalization (resize WSI to 512 and resize it back to 1024 with a random chance)</li>\n<li>Horizontal flip</li>\n<li>Vertical flip</li>\n<li>Random 90-degree rotation</li>\n<li>Shift scale rotate with 45-degree rotations and mild shift/scale augmentation</li>\n<li>Color jitter with strong hue and saturation</li>\n<li>Channel shuffle</li>\n<li>Gaussian blur</li>\n<li>Coarse dropout (cutout)</li>\n<li>ImageNet normalization</li>\n</ul>\n<h2>5. Inference</h2>\n<p>5 folds of EfficientNetV2 small model are used in the inference pipeline. Average of 5 folds are taken after predicting with each model.</p>\n<p>3x TTA (horizontal, vertical and diagonal flip) are applied and average of predictions are taken.</p>\n<p>16 crops are extracted for each WSI and average of their predictions are taken.</p>\n<p>The average pooling order for a single image is:</p>\n<ul>\n<li>Predict original and flipped images, activate predictions with softmax and average</li>\n<li>Predict with all folds and average</li>\n<li>Predict all crops and average if WSI </li>\n</ul>\n<h2>6. Change of Direction</h2>\n<p>The model had 86.70 OOF score (TMA: 84, WSI: 86.59) at that point but the LB score was 0.47 (private 0.52/32th-42th) which was very low.</p>\n<p><img src=\"https://i.ibb.co/tQRgZd0/wsi-confusion-matrix.png\" alt=\"wsi_confusion_matrix1\"></p>\n<p><img src=\"https://i.ibb.co/YQPDY2D/tma-confusion-matrix.png\" alt=\"tma_confusion_matrix1\"></p>\n<p><img src=\"https://i.ibb.co/zhsGR9x/confusion-matrix.png\" alt=\"confusion_matrix1\"></p>\n<p>I noticed some people were getting better LB scores with worse OOF scores and I was stuck at 0.47 for a while. I had worked on Optiver competition for 2 weeks and came back. I decided to dedicate my time to finding external data because breaking the entire pipeline and starting from scratch didn't make sense.</p>\n<h2>7. External Data</h2>\n<h3>UBC Ocean</h3>\n<p>The most obvious one is the test set image that is classified as HGSC confidently. 16 crops are extracted from that image and HGSC label is assigned to them.</p>\n<h3>Stanford Tissue Microarray Database</h3>\n<p>134 ovarian cancer TMAs are downloaded from <a href=\"https://tma.im/cgi-bin/viewArrayBlockList.pl\" target=\"_blank\">here</a>.</p>\n<p>Classes are converted with this mapping</p>\n<pre><code> = {\n      ovary spindle cell fibroma  ovary': ,\n     papillary serous': ,\n     endometrioid': ,\n     precursor  lymphoblastic': ,\n     adeno': ,\n     clear cell': ,\n     mucinous': ,\n     adeno mucinous': ,\n     dysgerminoma': \n}\n</code></pre>\n<h3>kztymsrjx9</h3>\n<p>This dataset is downloaded from <a href=\"https://data.mendeley.com/datasets/kztymsrjx9/1\" target=\"_blank\">here</a>. HGSC label is assigned to images in the Serous directory. Images in the Non_Cancerous directory are not used. 398 ovarian cancer TMAs are found here.</p>\n<h3>tissuearray.com</h3>\n<p>Screenshots of high resolution previews are taken from <a href=\"https://www.tissuearray.com/tissue-arrays/Ovary\" target=\"_blank\">here</a>. 1221 ovarian cancer TMAs are found here.</p>\n<h3>usbiolab.com</h3>\n<p>Screenshots of high resolution previews are taken from <a href=\"https://usbiolab.com/tissue-array/product/ovary\" target=\"_blank\">here</a>. 440 ovarian cancer TMAs are found here.</p>\n<h3>proteinatlas.org</h3>\n<p>Images are downloaded from <a href=\"https://www.proteinatlas.org/search/prognostic:ovarian+cancer;Favorable+AND+sort_by:prognostic+ovarian+cancer\" target=\"_blank\">here</a>. 376 ovarian cancer TMAs are found here.</p>\n<h3>Summary</h3>\n<p>Those were the sources where I found the external data.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Images</th>\n<th>Type</th>\n<th>HGSC</th>\n<th>EC</th>\n<th>CC</th>\n<th>LGSC</th>\n<th>MC</th>\n<th>Other</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>UBC Ocean Public Test</td>\n<td>16</td>\n<td>WSI</td>\n<td>16</td>\n<td>0</td>\n<td>0</td>\n<td>0</td>\n<td>0</td>\n<td>0</td>\n</tr>\n<tr>\n<td>Stanford Tissue Microarray Database</td>\n<td>134</td>\n<td>TMA</td>\n<td>37</td>\n<td>11</td>\n<td>4</td>\n<td>0</td>\n<td>4</td>\n<td>78</td>\n</tr>\n<tr>\n<td>kztymsrjx9</td>\n<td>398</td>\n<td>TMA</td>\n<td>100</td>\n<td>98</td>\n<td>100</td>\n<td>0</td>\n<td>100</td>\n<td>0</td>\n</tr>\n<tr>\n<td>tissuearray.com</td>\n<td>1221</td>\n<td>TMA</td>\n<td>348</td>\n<td>39</td>\n<td>24</td>\n<td>140</td>\n<td>100</td>\n<td>570</td>\n</tr>\n<tr>\n<td>usbiolab.com</td>\n<td>440</td>\n<td>TMA</td>\n<td>124</td>\n<td>40</td>\n<td>29</td>\n<td>89</td>\n<td>68</td>\n<td>90</td>\n</tr>\n<tr>\n<td>proteinatlas.org</td>\n<td>376</td>\n<td>TMA</td>\n<td>25</td>\n<td>155</td>\n<td>0</td>\n<td>63</td>\n<td>133</td>\n<td>0</td>\n</tr>\n</tbody>\n</table>\n<h2>8. Final Iteration</h2>\n<p>Final dataset (including 16 crops per WSI) label distribution was like this</p>\n<ul>\n<li>HGSC: 4127</li>\n<li>EC: 2252</li>\n<li>CC: 1666</li>\n<li>MC: 1066</li>\n<li>LGSC: 969</li>\n<li>Other: 738</li>\n</ul>\n<p>and image type distribution was like this</p>\n<ul>\n<li>WSI (16x 1024 crops): 8224</li>\n<li>TMA: 2594</li>\n</ul>\n<p>All the external data are concatenated to each fold's training sets. Validation sets are not changed in order to get comparable results. OOF score is decreased from 86.70 to 83.85 but LB score jumped to 0.54. I thought this jump was related to Other class but the improvement wasn't good enough. That's when I thought private test set could have more Other classes which is very likely of Kaggle competitions. Twist of this competition was predicting TMAs and Other so private test set would likely have more of them. I decided to trust LB and selected a submission with the highest LB score. That submission scored 0.54 on public and 0.58 on private.</p>",
  "messages": [
    {
      "id": 2586603,
      "postDate": "2024-01-04T10:19:21.027Z",
      "content": "<p>This was an interesting competition and I would like to thank my teammate <a href=\"https://www.kaggle.com/samfc10\" target=\"_blank\">@samfc10</a> and everyone involved with the organization of it.</p>\n<p>This is a simple textbook solution that heavily relies on external TMA data and strong labels. There is nothing special or novel in this pipeline.</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/gunesevitan/ubc-ocean-inference\" target=\"_blank\">Inference</a></li>\n<li><a href=\"https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started\" target=\"_blank\">libvips/pyvips Installation and Getting Started</a></li>\n<li><a href=\"https://www.kaggle.com/code/gunesevitan/ubc-ocean-jpeg-dataset-pipeline\" target=\"_blank\">UBC-OCEAN - JPEG Dataset Pipeline</a></li>\n<li><a href=\"https://www.kaggle.com/code/gunesevitan/ubc-ocean-eda\" target=\"_blank\">UBC-OCEAN - EDA</a></li>\n<li><a href=\"https://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset\" target=\"_blank\">UBC-OCEAN - Dataset</a></li>\n<li><a href=\"https://github.com/gunesevitan/ubc-ovarian-cancer-subtype-classification-and-outlier-detection\" target=\"_blank\">GitHub Repository</a></li>\n</ul>\n<h2>1. Raw Dataset</h2>\n<h3>WSI</h3>\n<p>Masks of WSIs are resized to thumbnail sizes. Tiles of WSIs and masks are extracted from their thumbnails with stride of 384 and they are padded to 512. A MaxViT Tiny FPN model is trained on those padded tiles and masks. Segmentation model outputs are activated with sigmoid and 3x TTA (horizontal, vertical and diagonal flip) are applied after the activation.</p>\n<p>Final segmentation mask prediction is blocky since the model was trained on tiles and merged later.</p>\n<p><img src=\"https://i.ibb.co/jg24x1H/Screenshot-from-2024-01-04-09-28-01.png\" alt=\"seg1\"></p>\n<p>Segmentation mask predictions are cast to 8-bit integer and upsampled to original WSI size with nearest neighbor interpolation.</p>\n<p><img src=\"https://i.ibb.co/ZHjtfmY/Screenshot-from-2024-01-04-09-31-42.png\" alt=\"seg2\"></p>\n<ul>\n<li>WSI and their mask predictions are cropped maximum number of times with stride of 1024.</li>\n<li>Crops are sorted based on their mask areas in descending order</li>\n<li>Top 16 crops are taken and WSI label is assigned to them</li>\n</ul>\n<h3>TMA</h3>\n<p>Rows and columns with low standard deviation are dropped on TMAs with the function below. The purpose of this preprocessing is removing white regions and making WSIs and TMAs as similar as possible. Using higher values of threshold were dropping areas in the tissue region so the standard deviation threshold is set to 10.</p>\n<pre><code> ():\n\n    \n\n    vertical_stds = image.std(axis=(, ))\n    horizontal_stds = image.std(axis=(, ))\n    cropped_image = image[vertical_stds &gt; threshold, :, :]\n    cropped_image = cropped_image[:, horizontal_stds &gt; threshold, :]\n\n     cropped_image\n</code></pre>\n<p><img src=\"https://i.ibb.co/8jCyhgG/4134-crop.png\" alt=\"seg2\"></p>\n<h2>2. Validation</h2>\n<p>Multi-label stratified kfold is used as the cross-validation scheme. Dataset is split into 5 folds. <code>label</code> and <code>is_tma</code> columns are used for stratification.</p>\n<h2>3. Models</h2>\n<p>EfficientNetV2 small model is used as the backbone with a regular classification head.</p>\n<h2>4. Training</h2>\n<p>CrossEntropyLoss with class weights are used as the loss function. Class weights are calculated as n / n ith class.</p>\n<p>AdamW optimizer is used with 0.0001 learning rate. Cosine annealing scheduler is used with 0.00001 minimum learning rate.</p>\n<p>AMP is also used for faster training and regularization.</p>\n<p>Each fold is trained for 15 epochs and epochs with the highest balanced accuracy are selected.</p>\n<p>Training transforms are:</p>\n<ul>\n<li>Resize TMAs to size 1024 (WSI crops are already 1024 sized)</li>\n<li>Magnification normalization (resize WSI to 512 and resize it back to 1024 with a random chance)</li>\n<li>Horizontal flip</li>\n<li>Vertical flip</li>\n<li>Random 90-degree rotation</li>\n<li>Shift scale rotate with 45-degree rotations and mild shift/scale augmentation</li>\n<li>Color jitter with strong hue and saturation</li>\n<li>Channel shuffle</li>\n<li>Gaussian blur</li>\n<li>Coarse dropout (cutout)</li>\n<li>ImageNet normalization</li>\n</ul>\n<h2>5. Inference</h2>\n<p>5 folds of EfficientNetV2 small model are used in the inference pipeline. Average of 5 folds are taken after predicting with each model.</p>\n<p>3x TTA (horizontal, vertical and diagonal flip) are applied and average of predictions are taken.</p>\n<p>16 crops are extracted for each WSI and average of their predictions are taken.</p>\n<p>The average pooling order for a single image is:</p>\n<ul>\n<li>Predict original and flipped images, activate predictions with softmax and average</li>\n<li>Predict with all folds and average</li>\n<li>Predict all crops and average if WSI </li>\n</ul>\n<h2>6. Change of Direction</h2>\n<p>The model had 86.70 OOF score (TMA: 84, WSI: 86.59) at that point but the LB score was 0.47 (private 0.52/32th-42th) which was very low.</p>\n<p><img src=\"https://i.ibb.co/tQRgZd0/wsi-confusion-matrix.png\" alt=\"wsi_confusion_matrix1\"></p>\n<p><img src=\"https://i.ibb.co/YQPDY2D/tma-confusion-matrix.png\" alt=\"tma_confusion_matrix1\"></p>\n<p><img src=\"https://i.ibb.co/zhsGR9x/confusion-matrix.png\" alt=\"confusion_matrix1\"></p>\n<p>I noticed some people were getting better LB scores with worse OOF scores and I was stuck at 0.47 for a while. I had worked on Optiver competition for 2 weeks and came back. I decided to dedicate my time to finding external data because breaking the entire pipeline and starting from scratch didn't make sense.</p>\n<h2>7. External Data</h2>\n<h3>UBC Ocean</h3>\n<p>The most obvious one is the test set image that is classified as HGSC confidently. 16 crops are extracted from that image and HGSC label is assigned to them.</p>\n<h3>Stanford Tissue Microarray Database</h3>\n<p>134 ovarian cancer TMAs are downloaded from <a href=\"https://tma.im/cgi-bin/viewArrayBlockList.pl\" target=\"_blank\">here</a>.</p>\n<p>Classes are converted with this mapping</p>\n<pre><code> = {\n      ovary spindle cell fibroma  ovary': ,\n     papillary serous': ,\n     endometrioid': ,\n     precursor  lymphoblastic': ,\n     adeno': ,\n     clear cell': ,\n     mucinous': ,\n     adeno mucinous': ,\n     dysgerminoma': \n}\n</code></pre>\n<h3>kztymsrjx9</h3>\n<p>This dataset is downloaded from <a href=\"https://data.mendeley.com/datasets/kztymsrjx9/1\" target=\"_blank\">here</a>. HGSC label is assigned to images in the Serous directory. Images in the Non_Cancerous directory are not used. 398 ovarian cancer TMAs are found here.</p>\n<h3>tissuearray.com</h3>\n<p>Screenshots of high resolution previews are taken from <a href=\"https://www.tissuearray.com/tissue-arrays/Ovary\" target=\"_blank\">here</a>. 1221 ovarian cancer TMAs are found here.</p>\n<h3>usbiolab.com</h3>\n<p>Screenshots of high resolution previews are taken from <a href=\"https://usbiolab.com/tissue-array/product/ovary\" target=\"_blank\">here</a>. 440 ovarian cancer TMAs are found here.</p>\n<h3>proteinatlas.org</h3>\n<p>Images are downloaded from <a href=\"https://www.proteinatlas.org/search/prognostic:ovarian+cancer;Favorable+AND+sort_by:prognostic+ovarian+cancer\" target=\"_blank\">here</a>. 376 ovarian cancer TMAs are found here.</p>\n<h3>Summary</h3>\n<p>Those were the sources where I found the external data.</p>\n<table>\n<thead>\n<tr>\n<th></th>\n<th>Images</th>\n<th>Type</th>\n<th>HGSC</th>\n<th>EC</th>\n<th>CC</th>\n<th>LGSC</th>\n<th>MC</th>\n<th>Other</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>UBC Ocean Public Test</td>\n<td>16</td>\n<td>WSI</td>\n<td>16</td>\n<td>0</td>\n<td>0</td>\n<td>0</td>\n<td>0</td>\n<td>0</td>\n</tr>\n<tr>\n<td>Stanford Tissue Microarray Database</td>\n<td>134</td>\n<td>TMA</td>\n<td>37</td>\n<td>11</td>\n<td>4</td>\n<td>0</td>\n<td>4</td>\n<td>78</td>\n</tr>\n<tr>\n<td>kztymsrjx9</td>\n<td>398</td>\n<td>TMA</td>\n<td>100</td>\n<td>98</td>\n<td>100</td>\n<td>0</td>\n<td>100</td>\n<td>0</td>\n</tr>\n<tr>\n<td>tissuearray.com</td>\n<td>1221</td>\n<td>TMA</td>\n<td>348</td>\n<td>39</td>\n<td>24</td>\n<td>140</td>\n<td>100</td>\n<td>570</td>\n</tr>\n<tr>\n<td>usbiolab.com</td>\n<td>440</td>\n<td>TMA</td>\n<td>124</td>\n<td>40</td>\n<td>29</td>\n<td>89</td>\n<td>68</td>\n<td>90</td>\n</tr>\n<tr>\n<td>proteinatlas.org</td>\n<td>376</td>\n<td>TMA</td>\n<td>25</td>\n<td>155</td>\n<td>0</td>\n<td>63</td>\n<td>133</td>\n<td>0</td>\n</tr>\n</tbody>\n</table>\n<h2>8. Final Iteration</h2>\n<p>Final dataset (including 16 crops per WSI) label distribution was like this</p>\n<ul>\n<li>HGSC: 4127</li>\n<li>EC: 2252</li>\n<li>CC: 1666</li>\n<li>MC: 1066</li>\n<li>LGSC: 969</li>\n<li>Other: 738</li>\n</ul>\n<p>and image type distribution was like this</p>\n<ul>\n<li>WSI (16x 1024 crops): 8224</li>\n<li>TMA: 2594</li>\n</ul>\n<p>All the external data are concatenated to each fold's training sets. Validation sets are not changed in order to get comparable results. OOF score is decreased from 86.70 to 83.85 but LB score jumped to 0.54. I thought this jump was related to Other class but the improvement wasn't good enough. That's when I thought private test set could have more Other classes which is very likely of Kaggle competitions. Twist of this competition was predicting TMAs and Other so private test set would likely have more of them. I decided to trust LB and selected a submission with the highest LB score. That submission scored 0.54 on public and 0.58 on private.</p>",
      "rawMarkdown": "This was an interesting competition and I would like to thank my teammate @samfc10 and everyone involved with the organization of it.\n\nThis is a simple textbook solution that heavily relies on external TMA data and strong labels. There is nothing special or novel in this pipeline.\n\n* [Inference](https://www.kaggle.com/code/gunesevitan/ubc-ocean-inference)\n* [libvips/pyvips Installation and Getting Started](https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started)\n* [UBC-OCEAN - JPEG Dataset Pipeline](https://www.kaggle.com/code/gunesevitan/ubc-ocean-jpeg-dataset-pipeline)\n* [UBC-OCEAN - EDA](https://www.kaggle.com/code/gunesevitan/ubc-ocean-eda)\n* [UBC-OCEAN - Dataset](https://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset)\n* [GitHub Repository](https://github.com/gunesevitan/ubc-ovarian-cancer-subtype-classification-and-outlier-detection)\n\n## 1. Raw Dataset\n\n### WSI\n\nMasks of WSIs are resized to thumbnail sizes. Tiles of WSIs and masks are extracted from their thumbnails with stride of 384 and they are padded to 512. A MaxViT Tiny FPN model is trained on those padded tiles and masks. Segmentation model outputs are activated with sigmoid and 3x TTA (horizontal, vertical and diagonal flip) are applied after the activation.\n\nFinal segmentation mask prediction is blocky since the model was trained on tiles and merged later.\n\n![seg1](https://i.ibb.co/jg24x1H/Screenshot-from-2024-01-04-09-28-01.png)\n\nSegmentation mask predictions are cast to 8-bit integer and upsampled to original WSI size with nearest neighbor interpolation.\n\n![seg2](https://i.ibb.co/ZHjtfmY/Screenshot-from-2024-01-04-09-31-42.png)\n\n* WSI and their mask predictions are cropped maximum number of times with stride of 1024.\n* Crops are sorted based on their mask areas in descending order\n* Top 16 crops are taken and WSI label is assigned to them\n\n### TMA\n\nRows and columns with low standard deviation are dropped on TMAs with the function below. The purpose of this preprocessing is removing white regions and making WSIs and TMAs as similar as possible. Using higher values of threshold were dropping areas in the tissue region so the standard deviation threshold is set to 10.\n\n```\ndef drop_low_std(image, threshold):\n\n    \"\"\"\n    Drop rows and columns that are below the given standard deviation threshold\n\n    Parameters\n    ----------\n    image: numpy.ndarray of shape (height, width, 3)\n        Image array\n\n    threshold: int\n        Standard deviation threshold\n\n    Returns\n    -------\n    image: numpy.ndarray of shape (cropped_height, cropped_width, 3)\n        Cropped image array\n    \"\"\"\n\n    vertical_stds = image.std(axis=(1, 2))\n    horizontal_stds = image.std(axis=(0, 2))\n    cropped_image = image[vertical_stds > threshold, :, :]\n    cropped_image = cropped_image[:, horizontal_stds > threshold, :]\n\n    return cropped_image\n```\n\n![seg2](https://i.ibb.co/8jCyhgG/4134-crop.png)\n\n## 2. Validation\n\nMulti-label stratified kfold is used as the cross-validation scheme. Dataset is split into 5 folds. `label` and `is_tma` columns are used for stratification.\n\n## 3. Models\n\nEfficientNetV2 small model is used as the backbone with a regular classification head.\n\n## 4. Training\n\nCrossEntropyLoss with class weights are used as the loss function. Class weights are calculated as n / n ith class.\n\nAdamW optimizer is used with 0.0001 learning rate. Cosine annealing scheduler is used with 0.00001 minimum learning rate.\n\nAMP is also used for faster training and regularization.\n\nEach fold is trained for 15 epochs and epochs with the highest balanced accuracy are selected.\n\nTraining transforms are:\n\n* Resize TMAs to size 1024 (WSI crops are already 1024 sized)\n* Magnification normalization (resize WSI to 512 and resize it back to 1024 with a random chance)\n* Horizontal flip\n* Vertical flip\n* Random 90-degree rotation\n* Shift scale rotate with 45-degree rotations and mild shift/scale augmentation\n* Color jitter with strong hue and saturation\n* Channel shuffle\n* Gaussian blur\n* Coarse dropout (cutout)\n* ImageNet normalization\n\n## 5. Inference\n\n5 folds of EfficientNetV2 small model are used in the inference pipeline. Average of 5 folds are taken after predicting with each model.\n\n3x TTA (horizontal, vertical and diagonal flip) are applied and average of predictions are taken.\n\n16 crops are extracted for each WSI and average of their predictions are taken.\n\nThe average pooling order for a single image is:\n* Predict original and flipped images, activate predictions with softmax and average\n* Predict with all folds and average\n* Predict all crops and average if WSI \n\n## 6. Change of Direction\n\nThe model had 86.70 OOF score (TMA: 84, WSI: 86.59) at that point but the LB score was 0.47 (private 0.52/32th-42th) which was very low.\n\n![wsi_confusion_matrix1](https://i.ibb.co/tQRgZd0/wsi-confusion-matrix.png)\n\n![tma_confusion_matrix1](https://i.ibb.co/YQPDY2D/tma-confusion-matrix.png)\n\n![confusion_matrix1](https://i.ibb.co/zhsGR9x/confusion-matrix.png)\n\nI noticed some people were getting better LB scores with worse OOF scores and I was stuck at 0.47 for a while. I had worked on Optiver competition for 2 weeks and came back. I decided to dedicate my time to finding external data because breaking the entire pipeline and starting from scratch didn't make sense.\n\n## 7. External Data\n\n### UBC Ocean\nThe most obvious one is the test set image that is classified as HGSC confidently. 16 crops are extracted from that image and HGSC label is assigned to them.\n\n### Stanford Tissue Microarray Database\n\n134 ovarian cancer TMAs are downloaded from [here](https://tma.im/cgi-bin/viewArrayBlockList.pl).\n\nClasses are converted with this mapping\n\n```\nCLASS_MAPPING = {\n    'fibroma of ovary spindle cell fibroma of ovary': 'Other',\n    'carcinoma papillary serous': 'HGSC',\n    'carcinoma endometrioid': 'EC',\n    'lymphoma precursor B lymphoblastic': 'Other',\n    'carcinoma adeno': 'HGSC',\n    'carcinoma clear cell': 'CC',\n    'carcinoma mucinous': 'MC',\n    'carcinoma adeno mucinous': 'MC',\n    'seminoma dysgerminoma': 'Other'\n}\n```\n\n### kztymsrjx9\n\nThis dataset is downloaded from [here](https://data.mendeley.com/datasets/kztymsrjx9/1). HGSC label is assigned to images in the Serous directory. Images in the Non_Cancerous directory are not used. 398 ovarian cancer TMAs are found here.\n\n### tissuearray.com\n\nScreenshots of high resolution previews are taken from [here](https://www.tissuearray.com/tissue-arrays/Ovary). 1221 ovarian cancer TMAs are found here.\n\n### usbiolab.com\n\nScreenshots of high resolution previews are taken from [here](https://usbiolab.com/tissue-array/product/ovary). 440 ovarian cancer TMAs are found here.\n\n### proteinatlas.org\n\nImages are downloaded from [here](https://www.proteinatlas.org/search/prognostic:ovarian+cancer;Favorable+AND+sort_by:prognostic+ovarian+cancer). 376 ovarian cancer TMAs are found here.\n\n### Summary\n\nThose were the sources where I found the external data.\n\n|                                     | Images | Type | HGSC | EC  | CC  | LGSC | MC  | Other |\n|-------------------------------------|--------|------|------|-----|-----|------|-----|-------|\n| UBC Ocean Public Test               | 16     | WSI  | 16   | 0   | 0   | 0    | 0   | 0     |\n| Stanford Tissue Microarray Database | 134    | TMA  | 37   | 11  | 4   | 0    | 4   | 78    |\n| kztymsrjx9                          | 398    | TMA  | 100  | 98  | 100 | 0    | 100 | 0     |\n| tissuearray.com                     | 1221   | TMA  | 348  | 39  | 24  | 140  | 100 | 570   |\n| usbiolab.com                        | 440    | TMA  | 124  | 40  | 29  | 89   | 68  | 90    |\n| proteinatlas.org                    | 376    | TMA  | 25   | 155 | 0   | 63   | 133 | 0     |\n\n## 8. Final Iteration\n\nFinal dataset (including 16 crops per WSI) label distribution was like this\n\n* HGSC: 4127\n* EC: 2252\n* CC: 1666\n* MC: 1066\n* LGSC: 969\n* Other: 738\n\nand image type distribution was like this\n\n* WSI (16x 1024 crops): 8224\n* TMA: 2594\n\nAll the external data are concatenated to each fold's training sets. Validation sets are not changed in order to get comparable results. OOF score is decreased from 86.70 to 83.85 but LB score jumped to 0.54. I thought this jump was related to Other class but the improvement wasn't good enough. That's when I thought private test set could have more Other classes which is very likely of Kaggle competitions. Twist of this competition was predicting TMAs and Other so private test set would likely have more of them. I decided to trust LB and selected a submission with the highest LB score. That submission scored 0.54 on public and 0.58 on private.\n",
      "votes": 28
    },
    {
      "id": 2587467,
      "postDate": "2024-01-04T19:05:56.603Z",
      "content": "<p>Congratulations!. You've found <code>Other</code> class data, we've searched some for a long time without any real success</p>",
      "rawMarkdown": "Congratulations!. You've found `Other` class data, we've searched some for a long time without any real success",
      "votes": 1,
      "replies": [
        {
          "id": 2587815,
          "postDate": "2024-01-05T03:50:01.830Z",
          "content": "<p>Thanks, congrats on your strong finish too! I think I wouldn't finish in top 20 without the Other data.</p>",
          "rawMarkdown": "Thanks, congrats on your strong finish too! I think I wouldn't finish in top 20 without the Other data.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2586886,
      "postDate": "2024-01-04T13:14:40.933Z",
      "content": "<p>The site you posted is very helpful. thank you!</p>\n<p>Also, was there a score higher than your private score that was not ultimately selected?</p>",
      "rawMarkdown": "The site you posted is very helpful. thank you!\n\nAlso, was there a score higher than your private score that was not ultimately selected?\n\n",
      "votes": 1,
      "replies": [
        {
          "id": 2586902,
          "postDate": "2024-01-04T13:26:10.203Z",
          "content": "<p>There were two more submissions with higher private score but both of them were 0.58 too. I think those 0.58 subs are pretty close to each other.</p>\n<p><img src=\"https://i.ibb.co/98ddh08/Screenshot-from-2024-01-04-16-24-46.png\" alt=\"1\"></p>",
          "rawMarkdown": "There were two more submissions with higher private score but both of them were 0.58 too. I think those 0.58 subs are pretty close to each other.\n\n![1](https://i.ibb.co/98ddh08/Screenshot-from-2024-01-04-16-24-46.png)",
          "votes": 1,
          "replies": [
            {
              "id": 2586927,
              "postDate": "2024-01-04T13:34:48.403Z",
              "content": "<p>Thank you! Happy new year</p>",
              "rawMarkdown": "Thank you! Happy new year",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2586694,
      "postDate": "2024-01-04T10:51:22.853Z",
      "content": "<p>Congratulations for the gold!</p>",
      "rawMarkdown": "Congratulations for the gold!",
      "votes": 1,
      "replies": [
        {
          "id": 2586699,
          "postDate": "2024-01-04T10:55:33.553Z",
          "content": "<p>Thanks so much</p>",
          "rawMarkdown": "Thanks so much"
        }
      ]
    },
    {
      "id": 2586634,
      "postDate": "2024-01-04T10:29:30.523Z",
      "content": "<p>\"The model had 86.70 OOF score (TMA: 84, WSI: 86.59) at that point but the LB score was 0.47 (private 0.52/32th-42th) which was very low.\" My model have OOF score about 86.4, TMA ~ 8x, same with you. My public LB is was 0.5-0.52 (private 0.5 - 0.52). I was stuck in 0.51 more than month. PB only TMA = 0.1 and PB only WSI = 0.42, but Private only TMA = 0.33-0.36 and private only WSI 0.18-0.15 :((</p>",
      "rawMarkdown": "\"The model had 86.70 OOF score (TMA: 84, WSI: 86.59) at that point but the LB score was 0.47 (private 0.52/32th-42th) which was very low.\" My model have OOF score about 86.4, TMA ~ 8x, same with you. My public LB is was 0.5-0.52 (private 0.5 - 0.52). I was stuck in 0.51 more than month. PB only TMA = 0.1 and PB only WSI = 0.42, but Private only TMA = 0.33-0.36 and private only WSI 0.18-0.15 :((",
      "votes": 1,
      "replies": [
        {
          "id": 2586701,
          "postDate": "2024-01-04T10:56:23.047Z",
          "content": "<p>Looks like we were on the same boat. I still don't understand how some teams broke that 0.5x barrier.</p>",
          "rawMarkdown": "Looks like we were on the same boat. I still don't understand how some teams broke that 0.5x barrier.",
          "votes": 1,
          "replies": [
            {
              "id": 2586852,
              "postDate": "2024-01-04T12:41:25.497Z",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> , Did you use the same weights for both WSI and TMA? I think TMA and WSI didn't share any similar feature. Hence , unless it's a large transformer-based model pretrained on large tissues dataset, general CNN will badly overfit.</p>",
              "rawMarkdown": "Hi @gunesevitan , Did you use the same weights for both WSI and TMA? I think TMA and WSI didn't share any similar feature. Hence , unless it's a large transformer-based model pretrained on large tissues dataset, general CNN will badly overfit."
            },
            {
              "id": 2586863,
              "postDate": "2024-01-04T12:49:53.140Z",
              "content": "<p>Do you mean weights for WSIs and TMAs? I didn't use any weights for that. I assigned 6 class weights for cross entropy loss and the weights are </p>\n<ul>\n<li>HGSC: 2.621275</li>\n<li>EC: 4.803730</li>\n<li>CC: 6.493397</li>\n<li>LGSC: 11.164087</li>\n<li>MC: 10.148218</li>\n<li>Other: 14.658537</li>\n</ul>",
              "rawMarkdown": "Do you mean weights for WSIs and TMAs? I didn't use any weights for that. I assigned 6 class weights for cross entropy loss and the weights are \n* HGSC: 2.621275\n* EC: 4.803730\n* CC: 6.493397\n* LGSC: 11.164087\n* MC: 10.148218\n* Other: 14.658537"
            },
            {
              "id": 2586870,
              "postDate": "2024-01-04T13:00:49.627Z",
              "content": "<p>I mean did you train one model on both WSI and TMA. Or trained two models on WSI, and TMA, respectively? </p>",
              "rawMarkdown": "I mean did you train one model on both WSI and TMA. Or trained two models on WSI, and TMA, respectively? "
            },
            {
              "id": 2586876,
              "postDate": "2024-01-04T13:02:41.313Z",
              "content": "<p>I used a single model for both image types. When I used all competition and external data for validation, WSI and TMA scores were very similar so I don't think it was overfitting.</p>",
              "rawMarkdown": "I used a single model for both image types. When I used all competition and external data for validation, WSI and TMA scores were very similar so I don't think it was overfitting.",
              "votes": 1
            },
            {
              "id": 2586880,
              "postDate": "2024-01-04T13:05:36.953Z",
              "content": "<p>But I think in essence, WSI and TMA doesn't share any similar feature, so training on both of them with a small model is very dangerous. </p>",
              "rawMarkdown": "But I think in essence, WSI and TMA doesn't share any similar feature, so training on both of them with a small model is very dangerous. ",
              "votes": 2
            },
            {
              "id": 2586881,
              "postDate": "2024-01-04T13:06:12.793Z",
              "content": "<p>Maybe you can try to retrain a model for TMA only and see if it can get a better scores on LB.  Your external data for TMA is enough for training a good model.</p>",
              "rawMarkdown": "Maybe you can try to retrain a model for TMA only and see if it can get a better scores on LB.  Your external data for TMA is enough for training a good model.",
              "votes": 1
            },
            {
              "id": 2586907,
              "postDate": "2024-01-04T13:28:06.077Z",
              "content": "<p>I really don't have time to do it but I open sourced my external data if someone wants to do ablation studies with it.</p>\n<p><a href=\"https://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset?select=datasets\" target=\"_blank\">https://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset?select=datasets</a></p>",
              "rawMarkdown": "I really don't have time to do it but I open sourced my external data if someone wants to do ablation studies with it.\n\nhttps://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset?select=datasets",
              "votes": 2
            },
            {
              "id": 2586924,
              "postDate": "2024-01-04T13:34:27.440Z",
              "content": "<p>I think this will be very helpful to anyone studying pathology slides. Thank you!</p>",
              "rawMarkdown": "I think this will be very helpful to anyone studying pathology slides. Thank you!",
              "votes": 2
            }
          ]
        }
      ]
    },
    {
      "id": 2586842,
      "postDate": "2024-01-04T12:36:38.793Z",
      "content": "<p>Wow! there are so many external dataset . I tried but I can't find many. 😅</p>",
      "rawMarkdown": "Wow! there are so many external dataset . I tried but I can't find many. 😅",
      "votes": 2,
      "replies": [
        {
          "id": 2586848,
          "postDate": "2024-01-04T12:40:23.857Z",
          "content": "<p>Yeah I have mad googling skills lol. Btw congrats on becoming a GM!</p>",
          "rawMarkdown": "Yeah I have mad googling skills lol. Btw congrats on becoming a GM!",
          "votes": 2
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 2587467,
      "author_name": "MPWARE",
      "author_url": "",
      "post_date": "2024-01-04T19:05:56.603000",
      "content": "<p>Congratulations!. You've found <code>Other</code> class data, we've searched some for a long time without any real success</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2587815,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2024-01-05T03:50:01.830000",
          "content": "<p>Thanks, congrats on your strong finish too! I think I wouldn't finish in top 20 without the Other data.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2586886,
      "author_name": "devchopin",
      "author_url": "",
      "post_date": "2024-01-04T13:14:40.933000",
      "content": "<p>The site you posted is very helpful. thank you!</p>\n<p>Also, was there a score higher than your private score that was not ultimately selected?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2586902,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2024-01-04T13:26:10.203000",
          "content": "<p>There were two more submissions with higher private score but both of them were 0.58 too. I think those 0.58 subs are pretty close to each other.</p>\n<p><img src=\"https://i.ibb.co/98ddh08/Screenshot-from-2024-01-04-16-24-46.png\" alt=\"1\"></p>",
          "votes": 1,
          "replies": [
            {
              "id": 2586927,
              "author_name": "devchopin",
              "author_url": "",
              "post_date": "2024-01-04T13:34:48.403000",
              "content": "<p>Thank you! Happy new year</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2586694,
      "author_name": "turkenm",
      "author_url": "",
      "post_date": "2024-01-04T10:51:22.853000",
      "content": "<p>Congratulations for the gold!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2586699,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2024-01-04T10:55:33.553000",
          "content": "<p>Thanks so much</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2586634,
      "author_name": "Quan Vu",
      "author_url": "",
      "post_date": "2024-01-04T10:29:30.523000",
      "content": "<p>\"The model had 86.70 OOF score (TMA: 84, WSI: 86.59) at that point but the LB score was 0.47 (private 0.52/32th-42th) which was very low.\" My model have OOF score about 86.4, TMA ~ 8x, same with you. My public LB is was 0.5-0.52 (private 0.5 - 0.52). I was stuck in 0.51 more than month. PB only TMA = 0.1 and PB only WSI = 0.42, but Private only TMA = 0.33-0.36 and private only WSI 0.18-0.15 :((</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2586701,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2024-01-04T10:56:23.047000",
          "content": "<p>Looks like we were on the same boat. I still don't understand how some teams broke that 0.5x barrier.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2586852,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2024-01-04T12:41:25.497000",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> , Did you use the same weights for both WSI and TMA? I think TMA and WSI didn't share any similar feature. Hence , unless it's a large transformer-based model pretrained on large tissues dataset, general CNN will badly overfit.</p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2586863,
              "author_name": "Gunes Evitan",
              "author_url": "",
              "post_date": "2024-01-04T12:49:53.140000",
              "content": "<p>Do you mean weights for WSIs and TMAs? I didn't use any weights for that. I assigned 6 class weights for cross entropy loss and the weights are </p>\n<ul>\n<li>HGSC: 2.621275</li>\n<li>EC: 4.803730</li>\n<li>CC: 6.493397</li>\n<li>LGSC: 11.164087</li>\n<li>MC: 10.148218</li>\n<li>Other: 14.658537</li>\n</ul>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2586870,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2024-01-04T13:00:49.627000",
              "content": "<p>I mean did you train one model on both WSI and TMA. Or trained two models on WSI, and TMA, respectively? </p>",
              "votes": 0,
              "replies": []
            },
            {
              "id": 2586876,
              "author_name": "Gunes Evitan",
              "author_url": "",
              "post_date": "2024-01-04T13:02:41.313000",
              "content": "<p>I used a single model for both image types. When I used all competition and external data for validation, WSI and TMA scores were very similar so I don't think it was overfitting.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2586880,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2024-01-04T13:05:36.953000",
              "content": "<p>But I think in essence, WSI and TMA doesn't share any similar feature, so training on both of them with a small model is very dangerous. </p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 2586881,
              "author_name": "ForcewithMe",
              "author_url": "",
              "post_date": "2024-01-04T13:06:12.793000",
              "content": "<p>Maybe you can try to retrain a model for TMA only and see if it can get a better scores on LB.  Your external data for TMA is enough for training a good model.</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2586907,
              "author_name": "Gunes Evitan",
              "author_url": "",
              "post_date": "2024-01-04T13:28:06.077000",
              "content": "<p>I really don't have time to do it but I open sourced my external data if someone wants to do ablation studies with it.</p>\n<p><a href=\"https://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset?select=datasets\" target=\"_blank\">https://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset?select=datasets</a></p>",
              "votes": 2,
              "replies": []
            },
            {
              "id": 2586924,
              "author_name": "devchopin",
              "author_url": "",
              "post_date": "2024-01-04T13:34:27.440000",
              "content": "<p>I think this will be very helpful to anyone studying pathology slides. Thank you!</p>",
              "votes": 2,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2586842,
      "author_name": "ForcewithMe",
      "author_url": "",
      "post_date": "2024-01-04T12:36:38.793000",
      "content": "<p>Wow! there are so many external dataset . I tried but I can't find many. 😅</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2586848,
          "author_name": "Gunes Evitan",
          "author_url": "",
          "post_date": "2024-01-04T12:40:23.857000",
          "content": "<p>Yeah I have mad googling skills lol. Btw congrats on becoming a GM!</p>",
          "votes": 2,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2586603": "This was an interesting competition and I would like to thank my teammate @samfc10 and everyone involved with the organization of it.\n\nThis is a simple textbook solution that heavily relies on external TMA data and strong labels. There is nothing special or novel in this pipeline.\n\n* [Inference](https://www.kaggle.com/code/gunesevitan/ubc-ocean-inference)\n* [libvips/pyvips Installation and Getting Started](https://www.kaggle.com/code/gunesevitan/libvips-pyvips-installation-and-getting-started)\n* [UBC-OCEAN - JPEG Dataset Pipeline](https://www.kaggle.com/code/gunesevitan/ubc-ocean-jpeg-dataset-pipeline)\n* [UBC-OCEAN - EDA](https://www.kaggle.com/code/gunesevitan/ubc-ocean-eda)\n* [UBC-OCEAN - Dataset](https://www.kaggle.com/datasets/gunesevitan/ubc-ocean-dataset)\n* [GitHub Repository](https://github.com/gunesevitan/ubc-ovarian-cancer-subtype-classification-and-outlier-detection)\n\n## 1. Raw Dataset\n\n### WSI\n\nMasks of WSIs are resized to thumbnail sizes. Tiles of WSIs and masks are extracted from their thumbnails with stride of 384 and they are padded to 512. A MaxViT Tiny FPN model is trained on those padded tiles and masks. Segmentation model outputs are activated with sigmoid and 3x TTA (horizontal, vertical and diagonal flip) are applied after the activation.\n\nFinal segmentation mask prediction is blocky since the model was trained on tiles and merged later.\n\n![seg1](https://i.ibb.co/jg24x1H/Screenshot-from-2024-01-04-09-28-01.png)\n\nSegmentation mask predictions are cast to 8-bit integer and upsampled to original WSI size with nearest neighbor interpolation.\n\n![seg2](https://i.ibb.co/ZHjtfmY/Screenshot-from-2024-01-04-09-31-42.png)\n\n* WSI and their mask predictions are cropped maximum number of times with stride of 1024.\n* Crops are sorted based on their mask areas in descending order\n* Top 16 crops are taken and WSI label is assigned to them\n\n### TMA\n\nRows and columns with low standard deviation are dropped on TMAs with the function below. The purpose of this preprocessing is removing white regions and making WSIs and TMAs as similar as possible. Using higher values of threshold were dropping areas in the tissue region so the standard deviation threshold is set to 10.\n\n```\ndef drop_low_std(image, threshold):\n\n    \"\"\"\n    Drop rows and columns that are below the given standard deviation threshold\n\n    Parameters\n    ----------\n    image: numpy.ndarray of shape (height, width, 3)\n        Image array\n\n    threshold: int\n        Standard deviation threshold\n\n    Returns\n    -------\n    image: numpy.ndarray of shape (cropped_height, cropped_width, 3)\n        Cropped image array\n    \"\"\"\n\n    vertical_stds = image.std(axis=(1, 2))\n    horizontal_stds = image.std(axis=(0, 2))\n    cropped_image = image[vertical_stds > threshold, :, :]\n    cropped_image = cropped_image[:, horizontal_stds > threshold, :]\n\n    return cropped_image\n```\n\n![seg2](https://i.ibb.co/8jCyhgG/4134-crop.png)\n\n## 2. Validation\n\nMulti-label stratified kfold is used as the cross-validation scheme. Dataset is split into 5 folds. `label` and `is_tma` columns are used for stratification.\n\n## 3. Models\n\nEfficientNetV2 small model is used as the backbone with a regular classification head.\n\n## 4. Training\n\nCrossEntropyLoss with class weights are used as the loss function. Class weights are calculated as n / n ith class.\n\nAdamW optimizer is used with 0.0001 learning rate. Cosine annealing scheduler is used with 0.00001 minimum learning rate.\n\nAMP is also used for faster training and regularization.\n\nEach fold is trained for 15 epochs and epochs with the highest balanced accuracy are selected.\n\nTraining transforms are:\n\n* Resize TMAs to size 1024 (WSI crops are already 1024 sized)\n* Magnification normalization (resize WSI to 512 and resize it back to 1024 with a random chance)\n* Horizontal flip\n* Vertical flip\n* Random 90-degree rotation\n* Shift scale rotate with 45-degree rotations and mild shift/scale augmentation\n* Color jitter with strong hue and saturation\n* Channel shuffle\n* Gaussian blur\n* Coarse dropout (cutout)\n* ImageNet normalization\n\n## 5. Inference\n\n5 folds of EfficientNetV2 small model are used in the inference pipeline. Average of 5 folds are taken after predicting with each model.\n\n3x TTA (horizontal, vertical and diagonal flip) are applied and average of predictions are taken.\n\n16 crops are extracted for each WSI and average of their predictions are taken.\n\nThe average pooling order for a single image is:\n* Predict original and flipped images, activate predictions with softmax and average\n* Predict with all folds and average\n* Predict all crops and average if WSI \n\n## 6. Change of Direction\n\nThe model had 86.70 OOF score (TMA: 84, WSI: 86.59) at that point but the LB score was 0.47 (private 0.52/32th-42th) which was very low.\n\n![wsi_confusion_matrix1](https://i.ibb.co/tQRgZd0/wsi-confusion-matrix.png)\n\n![tma_confusion_matrix1](https://i.ibb.co/YQPDY2D/tma-confusion-matrix.png)\n\n![confusion_matrix1](https://i.ibb.co/zhsGR9x/confusion-matrix.png)\n\nI noticed some people were getting better LB scores with worse OOF scores and I was stuck at 0.47 for a while. I had worked on Optiver competition for 2 weeks and came back. I decided to dedicate my time to finding external data because breaking the entire pipeline and starting from scratch didn't make sense.\n\n## 7. External Data\n\n### UBC Ocean\nThe most obvious one is the test set image that is classified as HGSC confidently. 16 crops are extracted from that image and HGSC label is assigned to them.\n\n### Stanford Tissue Microarray Database\n\n134 ovarian cancer TMAs are downloaded from [here](https://tma.im/cgi-bin/viewArrayBlockList.pl).\n\nClasses are converted with this mapping\n\n```\nCLASS_MAPPING = {\n    'fibroma of ovary spindle cell fibroma of ovary': 'Other',\n    'carcinoma papillary serous': 'HGSC',\n    'carcinoma endometrioid': 'EC',\n    'lymphoma precursor B lymphoblastic': 'Other',\n    'carcinoma adeno': 'HGSC',\n    'carcinoma clear cell': 'CC',\n    'carcinoma mucinous': 'MC',\n    'carcinoma adeno mucinous': 'MC',\n    'seminoma dysgerminoma': 'Other'\n}\n```\n\n### kztymsrjx9\n\nThis dataset is downloaded from [here](https://data.mendeley.com/datasets/kztymsrjx9/1). HGSC label is assigned to images in the Serous directory. Images in the Non_Cancerous directory are not used. 398 ovarian cancer TMAs are found here.\n\n### tissuearray.com\n\nScreenshots of high resolution previews are taken from [here](https://www.tissuearray.com/tissue-arrays/Ovary). 1221 ovarian cancer TMAs are found here.\n\n### usbiolab.com\n\nScreenshots of high resolution previews are taken from [here](https://usbiolab.com/tissue-array/product/ovary). 440 ovarian cancer TMAs are found here.\n\n### proteinatlas.org\n\nImages are downloaded from [here](https://www.proteinatlas.org/search/prognostic:ovarian+cancer;Favorable+AND+sort_by:prognostic+ovarian+cancer). 376 ovarian cancer TMAs are found here.\n\n### Summary\n\nThose were the sources where I found the external data.\n\n|                                     | Images | Type | HGSC | EC  | CC  | LGSC | MC  | Other |\n|-------------------------------------|--------|------|------|-----|-----|------|-----|-------|\n| UBC Ocean Public Test               | 16     | WSI  | 16   | 0   | 0   | 0    | 0   | 0     |\n| Stanford Tissue Microarray Database | 134    | TMA  | 37   | 11  | 4   | 0    | 4   | 78    |\n| kztymsrjx9                          | 398    | TMA  | 100  | 98  | 100 | 0    | 100 | 0     |\n| tissuearray.com                     | 1221   | TMA  | 348  | 39  | 24  | 140  | 100 | 570   |\n| usbiolab.com                        | 440    | TMA  | 124  | 40  | 29  | 89   | 68  | 90    |\n| proteinatlas.org                    | 376    | TMA  | 25   | 155 | 0   | 63   | 133 | 0     |\n\n## 8. Final Iteration\n\nFinal dataset (including 16 crops per WSI) label distribution was like this\n\n* HGSC: 4127\n* EC: 2252\n* CC: 1666\n* MC: 1066\n* LGSC: 969\n* Other: 738\n\nand image type distribution was like this\n\n* WSI (16x 1024 crops): 8224\n* TMA: 2594\n\nAll the external data are concatenated to each fold's training sets. Validation sets are not changed in order to get comparable results. OOF score is decreased from 86.70 to 83.85 but LB score jumped to 0.54. I thought this jump was related to Other class but the improvement wasn't good enough. That's when I thought private test set could have more Other classes which is very likely of Kaggle competitions. Twist of this competition was predicting TMAs and Other so private test set would likely have more of them. I decided to trust LB and selected a submission with the highest LB score. That submission scored 0.54 on public and 0.58 on private.\n",
    "2587467": "Congratulations!. You've found `Other` class data, we've searched some for a long time without any real success",
    "2586886": "The site you posted is very helpful. thank you!\n\nAlso, was there a score higher than your private score that was not ultimately selected?\n\n",
    "2586694": "Congratulations for the gold!",
    "2586634": "\"The model had 86.70 OOF score (TMA: 84, WSI: 86.59) at that point but the LB score was 0.47 (private 0.52/32th-42th) which was very low.\" My model have OOF score about 86.4, TMA ~ 8x, same with you. My public LB is was 0.5-0.52 (private 0.5 - 0.52). I was stuck in 0.51 more than month. PB only TMA = 0.1 and PB only WSI = 0.42, but Private only TMA = 0.33-0.36 and private only WSI 0.18-0.15 :((",
    "2586842": "Wow! there are so many external dataset . I tried but I can't find many. 😅"
  }
}