{ "id": 465415, "title": "[Our Solution] UBC-OCEAN", "url": "/competitions/UBC-OCEAN/discussion/465415", "author_name": "Zijie Fang", "post_date": "2024-01-04T06:42:11.218000", "votes": 10, "comment_count": 6, "views": 0, "content": "

TL;DR

\n

We modified a multiple instance learning (MIL) framework, ABMIL[1], with 224x224-shaped patches under 20x magnification to predict the subtypes. The patch features are extracted by CTransPath [2]. To predict the outliers, synthesized WSIs and TMAs are generated using patches in the stroma and necrosis category.

\n

Preprocessing for MIL

\n\n

MIL model for ID

\n\n

OOD synthesis for WSI

\n\n

OOD classification for TMA

\n\n

Producing the final submission

\n

The probabilities produced by the two MIL models and the ViT-S are nan-averaged to get the final prediction labels.

\n

References

\n

[1] M. Ilse, J. M. Tomczak, and M. Welling, ‘Attention-based Deep Multiple Instance Learning’, in International Conference on Machine Learning, PMLR, Jul. 2018, pp. 2127–2136.
\n[2] X. Wang et al., ‘Transformer-based unsupervised contrastive learning for histopathological image classification’, Medical Image Analysis, vol. 81, p. 102559, Oct. 2022, doi: 10.1016/j.media.2022.102559.

", "messages": [ { "id": 2586370, "postDate": "2024-01-04T06:42:11.220Z", "content": "

TL;DR

\n

We modified a multiple instance learning (MIL) framework, ABMIL[1], with 224x224-shaped patches under 20x magnification to predict the subtypes. The patch features are extracted by CTransPath [2]. To predict the outliers, synthesized WSIs and TMAs are generated using patches in the stroma and necrosis category.

\n

Preprocessing for MIL

\n\n

MIL model for ID

\n\n

OOD synthesis for WSI

\n\n

OOD classification for TMA

\n\n

Producing the final submission

\n

The probabilities produced by the two MIL models and the ViT-S are nan-averaged to get the final prediction labels.

\n

References

\n

[1] M. Ilse, J. M. Tomczak, and M. Welling, ‘Attention-based Deep Multiple Instance Learning’, in International Conference on Machine Learning, PMLR, Jul. 2018, pp. 2127–2136.
\n[2] X. Wang et al., ‘Transformer-based unsupervised contrastive learning for histopathological image classification’, Medical Image Analysis, vol. 81, p. 102559, Oct. 2022, doi: 10.1016/j.media.2022.102559.

", "rawMarkdown": "\n\n## TL;DR\nWe modified a multiple instance learning (MIL) framework, ABMIL[1], with 224x224-shaped patches under 20x magnification to predict the subtypes. The patch features are extracted by CTransPath [2]. To predict the outliers, synthesized WSIs and TMAs are generated using patches in the stroma and necrosis category.\n\n## Preprocessing for MIL\n* WSIs and TMAs are splitted into 224x224 shaped patches under 20x (which means the TMAs should be 2x downsampled as they are 40x). \n\n* Patches with large backgrounds are discarded.\n\n* The pretrained model CTransPath is utilized to extract instance (patch) features.\n\n* During feature extraction, data augmentation including brightness, contract, color, color and sharpness adjustment are utilized for each WSI/TMA. Data augmentation is not utilized in testing.\n\n## MIL model for ID\n* We choose the ABMIL framework as a baseline of the MIL model. To model the dependencies between different patches, an 1D convolution block is added before the attention block, which contributes to ~0.04 improvement on the public LB.\n\n* Other MIL frameworks such as CLAM, TransMIL, and DTFDMIL are tried, but with no performance improvement.\n\n## OOD synthesis for WSI\n* We randomly choose patches masked as stroma and necrosis to synthesize the Other category, and train another MIL model using ABMIL for six class classification (5 ID + 1 OOD)，\n\n* ~0.04 improvement compared with no 'Other'\n\n## OOD classification for TMA\n* We split the masked patches in WSIs under 10x magnification and produce 512x512 patches and train a ViT-S for TMA classification. For tumor patches, we assign them subtype labels. For stroma and necrosis patches, we treat them as Others. For TMAs, one 2048x2048 patch is center cropped, and downsample to 512x512 to produce 10x patches.\n\n* ~0.01 improvement on public LB\n\n## Producing the final submission\nThe probabilities produced by the two MIL models and the ViT-S are nan-averaged to get the final prediction labels.\n\n## References\n[1] M. Ilse, J. M. Tomczak, and M. Welling, ‘Attention-based Deep Multiple Instance Learning’, in International Conference on Machine Learning, PMLR, Jul. 2018, pp. 2127–2136.\n[2] X. Wang et al., ‘Transformer-based unsupervised contrastive learning for histopathological image classification’, Medical Image Analysis, vol. 81, p. 102559, Oct. 2022, doi: 10.1016/j.media.2022.102559.\n\n\n\n\n", "votes": 10 }, { "id": 2586408, "postDate": "2024-01-04T07:24:15.427Z", "content": "

Hi @zijiefang , it seems that our solution shares some key similarity. Congratulations on your first gold medal.

", "rawMarkdown": "Hi @zijiefang , it seems that our solution shares some key similarity. Congratulations on your first gold medal.", "votes": 1, "replies": [ { "id": 2586412, "postDate": "2024-01-04T07:32:01.997Z", "content": "

Thank you! And congratulations for your becoming a GM!

", "rawMarkdown": "Thank you! And congratulations for your becoming a GM!", "votes": 1 } ] }, { "id": 2586485, "postDate": "2024-01-04T08:49:54.787Z", "content": "

Great work! Is there a public implementation of ABMIL and CTransPath we could use in the future?

", "rawMarkdown": "Great work! Is there a public implementation of ABMIL and CTransPath we could use in the future?", "replies": [ { "id": 2586604, "postDate": "2024-01-04T10:19:26.130Z", "content": "

Of course! we just adopted the official implementation of CTransPath at https://github.com/Xiyue-Wang/TransPath/blob/main/ctran.py.

\n

For ABMIL, its official implementation is patch-based (i.e., the input are RGB images rather than features), we made a slight modification on it.

\n
 (nn.Module):\n     ():\n        (ABMILConv, self).__init__()\n        self.L, self.D = ,  \n        self.K = \n\n        fc = [nn.Linear(input_dim, self.L), nn.ReLU()]\n\n         dropout:\n            fc.append(nn.Dropout(dropout))\n\n        self.fc = nn.Sequential(*fc)\n        self.conv = nn.Conv1d(in_channels=self.L, out_channels=self.L, kernel_size=kernel_size, padding=kernel_size//) \n\n        self.attention = nn.Sequential(\n            nn.Linear(self.L, self.D),\n            nn.Tanh(),\n            nn.Dropout(dropout)  dropout  nn.Identity(),\n            nn.Linear(self.D, self.K)\n        )\n\n        self.classifier = nn.Linear(self.L*self.K, n_classes)\n    ():\n        \n        ...\n
", "rawMarkdown": "Of course! we just adopted the official implementation of CTransPath at https://github.com/Xiyue-Wang/TransPath/blob/main/ctran.py.\n\nFor ABMIL, its [official implementation](https://github.com/AMLab-Amsterdam/AttentionDeepMIL/blob/master/model.py) is patch-based (i.e., the input are RGB images rather than features), we made a slight modification on it.\n\n```python\nclass ABMILConv(nn.Module):\n def __init__(self, n_classes, input_dim, kernel_size=3, dropout=0.25):\n super(ABMILConv, self).__init__()\n self.L, self.D = 512, 256 # can be changed\n self.K = 1\n\n fc = [nn.Linear(input_dim, self.L), nn.ReLU()]\n\n if dropout:\n fc.append(nn.Dropout(dropout))\n \n self.fc = nn.Sequential(*fc)\n self.conv = nn.Conv1d(in_channels=self.L, out_channels=self.L, kernel_size=kernel_size, padding=kernel_size//2) # 1D Conv implementation, which is different from the original paper\n\n self.attention = nn.Sequential(\n nn.Linear(self.L, self.D),\n nn.Tanh(),\n nn.Dropout(dropout) if dropout else nn.Identity(),\n nn.Linear(self.D, self.K)\n )\n\n self.classifier = nn.Linear(self.L*self.K, n_classes)\n def forward(self, x):\n # Original forward function, adding self.conv after self.fc and before self.attention\n ...\n```", "votes": 1 }, { "id": 2586619, "postDate": "2024-01-04T10:23:09.853Z", "content": "

Besides, thank you for the excellent training and inference code, helped us a lot during the competition! :)

", "rawMarkdown": "Besides, thank you for the excellent training and inference code, helped us a lot during the competition! :)", "votes": 1, "replies": [ { "id": 2586796, "postDate": "2024-01-04T12:08:45.687Z", "content": "

Thank you, just feeling sorry that I could not push any MIL due to lack of time :(

", "rawMarkdown": "Thank you, just feeling sorry that I could not push any MIL due to lack of time :(" } ] } ] } ], "comments": [ { "id": 2586408, "author_name": "ForcewithMe", "author_url": "", "post_date": "2024-01-04T07:24:15.427000", "content": "

Hi @zijiefang , it seems that our solution shares some key similarity. Congratulations on your first gold medal.

", "votes": 1, "replies": [ { "id": 2586412, "author_name": "Zijie Fang", "author_url": "", "post_date": "2024-01-04T07:32:01.997000", "content": "

Thank you! And congratulations for your becoming a GM!

", "votes": 1, "replies": [] } ] }, { "id": 2586485, "author_name": "Jirka", "author_url": "", "post_date": "2024-01-04T08:49:54.787000", "content": "

Great work! Is there a public implementation of ABMIL and CTransPath we could use in the future?

", "votes": 0, "replies": [ { "id": 2586604, "author_name": "Zijie Fang", "author_url": "", "post_date": "2024-01-04T10:19:26.130000", "content": "

Of course! we just adopted the official implementation of CTransPath at https://github.com/Xiyue-Wang/TransPath/blob/main/ctran.py.

\n

For ABMIL, its official implementation is patch-based (i.e., the input are RGB images rather than features), we made a slight modification on it.

\n
 (nn.Module):\n     ():\n        (ABMILConv, self).__init__()\n        self.L, self.D = ,  \n        self.K = \n\n        fc = [nn.Linear(input_dim, self.L), nn.ReLU()]\n\n         dropout:\n            fc.append(nn.Dropout(dropout))\n\n        self.fc = nn.Sequential(*fc)\n        self.conv = nn.Conv1d(in_channels=self.L, out_channels=self.L, kernel_size=kernel_size, padding=kernel_size//) \n\n        self.attention = nn.Sequential(\n            nn.Linear(self.L, self.D),\n            nn.Tanh(),\n            nn.Dropout(dropout)  dropout  nn.Identity(),\n            nn.Linear(self.D, self.K)\n        )\n\n        self.classifier = nn.Linear(self.L*self.K, n_classes)\n    ():\n        \n        ...\n
", "votes": 1, "replies": [] }, { "id": 2586619, "author_name": "Zijie Fang", "author_url": "", "post_date": "2024-01-04T10:23:09.853000", "content": "

Besides, thank you for the excellent training and inference code, helped us a lot during the competition! :)

", "votes": 1, "replies": [ { "id": 2586796, "author_name": "Jirka", "author_url": "", "post_date": "2024-01-04T12:08:45.687000", "content": "

Thank you, just feeling sorry that I could not push any MIL due to lack of time :(

", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2586370": "\n\n## TL;DR\nWe modified a multiple instance learning (MIL) framework, ABMIL[1], with 224x224-shaped patches under 20x magnification to predict the subtypes. The patch features are extracted by CTransPath [2]. To predict the outliers, synthesized WSIs and TMAs are generated using patches in the stroma and necrosis category.\n\n## Preprocessing for MIL\n* WSIs and TMAs are splitted into 224x224 shaped patches under 20x (which means the TMAs should be 2x downsampled as they are 40x). \n\n* Patches with large backgrounds are discarded.\n\n* The pretrained model CTransPath is utilized to extract instance (patch) features.\n\n* During feature extraction, data augmentation including brightness, contract, color, color and sharpness adjustment are utilized for each WSI/TMA. Data augmentation is not utilized in testing.\n\n## MIL model for ID\n* We choose the ABMIL framework as a baseline of the MIL model. To model the dependencies between different patches, an 1D convolution block is added before the attention block, which contributes to ~0.04 improvement on the public LB.\n\n* Other MIL frameworks such as CLAM, TransMIL, and DTFDMIL are tried, but with no performance improvement.\n\n## OOD synthesis for WSI\n* We randomly choose patches masked as stroma and necrosis to synthesize the Other category, and train another MIL model using ABMIL for six class classification (5 ID + 1 OOD)，\n\n* ~0.04 improvement compared with no 'Other'\n\n## OOD classification for TMA\n* We split the masked patches in WSIs under 10x magnification and produce 512x512 patches and train a ViT-S for TMA classification. For tumor patches, we assign them subtype labels. For stroma and necrosis patches, we treat them as Others. For TMAs, one 2048x2048 patch is center cropped, and downsample to 512x512 to produce 10x patches.\n\n* ~0.01 improvement on public LB\n\n## Producing the final submission\nThe probabilities produced by the two MIL models and the ViT-S are nan-averaged to get the final prediction labels.\n\n## References\n[1] M. Ilse, J. M. Tomczak, and M. Welling, ‘Attention-based Deep Multiple Instance Learning’, in International Conference on Machine Learning, PMLR, Jul. 2018, pp. 2127–2136.\n[2] X. Wang et al., ‘Transformer-based unsupervised contrastive learning for histopathological image classification’, Medical Image Analysis, vol. 81, p. 102559, Oct. 2022, doi: 10.1016/j.media.2022.102559.\n\n\n\n\n", "2586408": "Hi @zijiefang , it seems that our solution shares some key similarity. Congratulations on your first gold medal.", "2586485": "Great work! Is there a public implementation of ABMIL and CTransPath we could use in the future?" } }