{ "id": 364837, "title": "4th place solution, CSN is all you need for 3D", "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/364837", "author_name": "Selim Seferbekov", "post_date": "2022-11-08T14:28:16.586000", "votes": 34, "comment_count": 4, "views": 0, "content": "

I started working on this challenge really late, made first sub 10 days before deadline, so could not run a lot of experiments and used codebase/tricks from GI tract segmentation challenge and NASA comet detection challenge.

\n

TLDR

\n

Two stage approach, 3D segmentation + 3D classification for each vertebra crop

\n

The main trick

\n

A lot of failed attempts to use 3D networks for classification are related to overfitting . That's the case for pure 3D convolutions which usually are not needed and we can easily work with architectures that achieve state of the art performance on video/action classification datasets.
\nSo the options are:

\n\n

From https://paperswithcode.com/sota/action-classification-on-kinetics-400 it is clearly seen that the best convolutional architecture for video classifcation in 2022 is still ir-CSN-152. Even though transformers achieve higher scores on Kinetics datasets they don't perform as good on small datasets like RSNA and CSNs are the best for small sized datasets and if one needs fast and accurate 3D segmentation and/or classification.

\n

I used mmaction2's implementation of CSNs.

\n

Segmentation

\n\n

Training:

\n\n

Classification

\n\n

Training

\n\n

Code

\n", "messages": [ { "id": 2021867, "postDate": "2022-11-08T14:28:16.587Z", "content": "

I started working on this challenge really late, made first sub 10 days before deadline, so could not run a lot of experiments and used codebase/tricks from GI tract segmentation challenge and NASA comet detection challenge.

\n

TLDR

\n

Two stage approach, 3D segmentation + 3D classification for each vertebra crop

\n

The main trick

\n

A lot of failed attempts to use 3D networks for classification are related to overfitting . That's the case for pure 3D convolutions which usually are not needed and we can easily work with architectures that achieve state of the art performance on video/action classification datasets.
\nSo the options are:

\n\n

From https://paperswithcode.com/sota/action-classification-on-kinetics-400 it is clearly seen that the best convolutional architecture for video classifcation in 2022 is still ir-CSN-152. Even though transformers achieve higher scores on Kinetics datasets they don't perform as good on small datasets like RSNA and CSNs are the best for small sized datasets and if one needs fast and accurate 3D segmentation and/or classification.

\n

I used mmaction2's implementation of CSNs.

\n

Segmentation

\n\n

Training:

\n\n

Classification

\n\n

Training

\n\n

Code

\n", "rawMarkdown": "I started working on this challenge really late, made first sub 10 days before deadline, so could not run a lot of experiments and used codebase/tricks from [GI tract segmentation challenge](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation) and NASA comet detection challenge.\n\n###TLDR\nTwo stage approach, 3D segmentation + 3D classification for each vertebra crop\n\n###The main trick\nA lot of failed attempts to use 3D networks for classification are related to overfitting . That's the case for pure 3D convolutions which usually are not needed and we can easily work with architectures that achieve state of the art performance on video/action classification datasets. \nSo the options are:\n- inflate some state of the art 2d nets, like effnets but start using 3Ds from 2nd or 3rd block . It works but still tends to overfit and imagenet pretraining is not the best in this case\n- use 2D networks with LSTM/ConvLSTM/Transformer heads. It works and allows to use a lot of architectures, the main issue here is that 3D nature is considered only on the latest stage.\n- use pretrained networks from ig65m/kinetics datasest. That gave the best results for me. \n\n\n\n \nFrom https://paperswithcode.com/sota/action-classification-on-kinetics-400 it is clearly seen that the best convolutional architecture for video classifcation in 2022 is still [ir-CSN-152](https://arxiv.org/abs/1904.02811). Even though transformers achieve higher scores on Kinetics datasets they don't perform as good on small datasets like RSNA and CSNs are the best for small sized datasets and if one needs fast and accurate 3D segmentation and/or classification.\n\nI used mmaction2's implementation of CSNs.\n\n### Segmentation\n- UNet like multiclass segmentor\n- encoder: **ir-CSN-50**\n- decoder: standard unet decoder with nn upsampling but with [(2+1)d convolutions](https://arxiv.org/abs/1711.11248v3)\n- pure 3d convolutions in decoder lead to NaNs in amp training\n\n**Training**:\n\n- 2x subsampling(just slice ::2) by z axis\n- 2x linear downsampled images\n- memory mapping to reduce IO/CPU overhead\n- AdamW + wd, cosine LR annealing\n- Loss: focal-jaccard optimized loss for faster multiclass jaccard computation\n- 2D augs, ReplayCompose in Albumentations library, lightweight geometric augs + hflip\n\n### Classification\n\n- 4 folds of ir(ip)-CSN-152 with global max pooling\n- multilabel 8 classes, if less than 30% of vertebra is visible then make vertebra's label 0 and recompute overall label, otherwise keep the same as provided\n\n**Training**\n\n- 3 channel input (img, img, integer encoded segmentation mask\n- 2x subsampling(just slice ::2) by z axis\n- using 40 slices(80 in original data) around each vertebra\n- crops were resized to 256x256\n- BCE loss\n- target metric as validation\n- AdamW + wd, cosine LR annealing\n- Augmentations: 2D augs, replay compose in Albumentations library, flips, rotations, geometric. Needed to make more augs compared to segmentation pipeline\n\n### Code\n- inference kernel and weights: https://www.kaggle.com/code/selimsef/rsna-csn-segmentor-classifier\n- github code for training: https://github.com/selimsef/rsna_cervical_fracture/", "votes": 34 }, { "id": 2060896, "postDate": "2022-12-10T14:36:34.997Z", "content": "

@selimsef, could you explain why the input of the classifiers is in the form of 2x images + masks? Below is the part of the code from DatasetCrops.__getitem__() I am asking about.

\n
images = np.concatenate([images, images, masks], axis=-)\n
", "rawMarkdown": "@selimsef, could you explain why the input of the classifiers is in the form of 2x images + masks? Below is the part of the code from `DatasetCrops.__getitem__()` I am asking about.\n\n```py\nimages = np.concatenate([images, images, masks], axis=-1)\n```", "replies": [ { "id": 2062151, "postDate": "2022-12-11T19:12:39.093Z", "content": "

@mariuszwisniewski I used integer encoded predicted segmentation mask as an additional input. That's not the best approach but it was the first thing (combined with 3D crop around each vertebra) that worked for me. The other approach is to just use binary masks.
\nWhy not just img + mask? Was a bit lazy to recompute the first conv weights to 2 channels.

", "rawMarkdown": "@mariuszwisniewski I used integer encoded predicted segmentation mask as an additional input. That's not the best approach but it was the first thing (combined with 3D crop around each vertebra) that worked for me. The other approach is to just use binary masks.\nWhy not just img + mask? Was a bit lazy to recompute the first conv weights to 2 channels.", "votes": 2 } ] }, { "id": 2024863, "postDate": "2022-11-10T20:00:33.797Z", "content": "

Congratulations and thanks for sharing your code (RSNA CSN segmentor & classifier ), GitHub and solution too.

", "rawMarkdown": "Congratulations and thanks for sharing your code (RSNA CSN segmentor & classifier ), GitHub and solution too." }, { "id": 2022199, "postDate": "2022-11-08T19:48:36.620Z", "content": "

Wow, awesome result with just 10 days of active submissions @selimsef! I appreciate your diligence and efforts. Congratulations!!
\nWishing you the best!!
\nI appreciate the detailed approach note too, this is informative..

", "rawMarkdown": "Wow, awesome result with just 10 days of active submissions @selimsef! I appreciate your diligence and efforts. Congratulations!!\nWishing you the best!! \nI appreciate the detailed approach note too, this is informative.." } ], "comments": [ { "id": 2060896, "author_name": "Mariusz Wiśniewski", "author_url": "", "post_date": "2022-12-10T14:36:34.997000", "content": "

@selimsef, could you explain why the input of the classifiers is in the form of 2x images + masks? Below is the part of the code from DatasetCrops.__getitem__() I am asking about.

\n
images = np.concatenate([images, images, masks], axis=-)\n
", "votes": 0, "replies": [ { "id": 2062151, "author_name": "Selim Seferbekov", "author_url": "", "post_date": "2022-12-11T19:12:39.093000", "content": "

@mariuszwisniewski I used integer encoded predicted segmentation mask as an additional input. That's not the best approach but it was the first thing (combined with 3D crop around each vertebra) that worked for me. The other approach is to just use binary masks.
\nWhy not just img + mask? Was a bit lazy to recompute the first conv weights to 2 channels.

", "votes": 2, "replies": [] } ] }, { "id": 2024863, "author_name": "Marília Prata", "author_url": "", "post_date": "2022-11-10T20:00:33.797000", "content": "

Congratulations and thanks for sharing your code (RSNA CSN segmentor & classifier ), GitHub and solution too.

", "votes": 0, "replies": [] }, { "id": 2022199, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-11-08T19:48:36.620000", "content": "

Wow, awesome result with just 10 days of active submissions @selimsef! I appreciate your diligence and efforts. Congratulations!!
\nWishing you the best!!
\nI appreciate the detailed approach note too, this is informative..

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2021867": "I started working on this challenge really late, made first sub 10 days before deadline, so could not run a lot of experiments and used codebase/tricks from [GI tract segmentation challenge](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation) and NASA comet detection challenge.\n\n###TLDR\nTwo stage approach, 3D segmentation + 3D classification for each vertebra crop\n\n###The main trick\nA lot of failed attempts to use 3D networks for classification are related to overfitting . That's the case for pure 3D convolutions which usually are not needed and we can easily work with architectures that achieve state of the art performance on video/action classification datasets. \nSo the options are:\n- inflate some state of the art 2d nets, like effnets but start using 3Ds from 2nd or 3rd block . It works but still tends to overfit and imagenet pretraining is not the best in this case\n- use 2D networks with LSTM/ConvLSTM/Transformer heads. It works and allows to use a lot of architectures, the main issue here is that 3D nature is considered only on the latest stage.\n- use pretrained networks from ig65m/kinetics datasest. That gave the best results for me. \n\n\n\n \nFrom https://paperswithcode.com/sota/action-classification-on-kinetics-400 it is clearly seen that the best convolutional architecture for video classifcation in 2022 is still [ir-CSN-152](https://arxiv.org/abs/1904.02811). Even though transformers achieve higher scores on Kinetics datasets they don't perform as good on small datasets like RSNA and CSNs are the best for small sized datasets and if one needs fast and accurate 3D segmentation and/or classification.\n\nI used mmaction2's implementation of CSNs.\n\n### Segmentation\n- UNet like multiclass segmentor\n- encoder: **ir-CSN-50**\n- decoder: standard unet decoder with nn upsampling but with [(2+1)d convolutions](https://arxiv.org/abs/1711.11248v3)\n- pure 3d convolutions in decoder lead to NaNs in amp training\n\n**Training**:\n\n- 2x subsampling(just slice ::2) by z axis\n- 2x linear downsampled images\n- memory mapping to reduce IO/CPU overhead\n- AdamW + wd, cosine LR annealing\n- Loss: focal-jaccard optimized loss for faster multiclass jaccard computation\n- 2D augs, ReplayCompose in Albumentations library, lightweight geometric augs + hflip\n\n### Classification\n\n- 4 folds of ir(ip)-CSN-152 with global max pooling\n- multilabel 8 classes, if less than 30% of vertebra is visible then make vertebra's label 0 and recompute overall label, otherwise keep the same as provided\n\n**Training**\n\n- 3 channel input (img, img, integer encoded segmentation mask\n- 2x subsampling(just slice ::2) by z axis\n- using 40 slices(80 in original data) around each vertebra\n- crops were resized to 256x256\n- BCE loss\n- target metric as validation\n- AdamW + wd, cosine LR annealing\n- Augmentations: 2D augs, replay compose in Albumentations library, flips, rotations, geometric. Needed to make more augs compared to segmentation pipeline\n\n### Code\n- inference kernel and weights: https://www.kaggle.com/code/selimsef/rsna-csn-segmentor-classifier\n- github code for training: https://github.com/selimsef/rsna_cervical_fracture/", "2060896": "@selimsef, could you explain why the input of the classifiers is in the form of 2x images + masks? Below is the part of the code from `DatasetCrops.__getitem__()` I am asking about.\n\n```py\nimages = np.concatenate([images, images, masks], axis=-1)\n```", "2024863": "Congratulations and thanks for sharing your code (RSNA CSN segmentor & classifier ), GitHub and solution too.", "2022199": "Wow, awesome result with just 10 days of active submissions @selimsef! I appreciate your diligence and efforts. Congratulations!!\nWishing you the best!! \nI appreciate the detailed approach note too, this is informative.." } }