{ "id": 350859, "title": "[placeholder] 3d transformer benchmark results", "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/350859", "author_name": "hengck23", "post_date": "2022-09-07T11:44:40.933000", "votes": 58, "comment_count": 11, "views": 0, "content": "

to be updated ….

\n

baseline model:

\n

\"Selection-176\"

\n

approximate plan:

\n
    \n
  1. design a framework with use of aux label (segmentation and box)
    2. \n
  2. compare results of 3d transformer (including pyramid vit), 3d cnn and 3d hybrid and others (2D 2.5D, LSTM, etc …)
    3. \n
  3. apply patch pretraining to transformer
    4. \n
  4. apply human in the loop active learning (click point annotation)
    5. \n
\n

why use transformer?

\n\n

reference paper:
\n\"Selection-177\"

\n
\n

the nice thing about transformer is that i just need to change the vision tokenization layer for 2d input to 3d input. The rest of the transformer layers can remains unchanged. Hence i can easily modify any of the existing open source 2d vision transformer code. the pretrained weights for 2d also works for 3d image data.

\n

you can think that the below is approximately true :
\nconv3d(x,w) = conv2d(x.mean(0), w.mean(0))

\n

just like
\nconv2d(x_10x10_patch,w_10x10_patch) = conv2d(x_resize_to_5x5, w_resize_to_5x5)

", "messages": [ { "id": 1929822, "postDate": "2022-09-07T11:44:40.933Z", "content": "

to be updated ….

\n

baseline model:

\n

\"Selection-176\"

\n

approximate plan:

\n
    \n
  1. design a framework with use of aux label (segmentation and box)
    2. \n
  2. compare results of 3d transformer (including pyramid vit), 3d cnn and 3d hybrid and others (2D 2.5D, LSTM, etc …)
    3. \n
  3. apply patch pretraining to transformer
    4. \n
  4. apply human in the loop active learning (click point annotation)
    5. \n
\n

why use transformer?

\n\n

reference paper:
\n\"Selection-177\"

\n
\n

the nice thing about transformer is that i just need to change the vision tokenization layer for 2d input to 3d input. The rest of the transformer layers can remains unchanged. Hence i can easily modify any of the existing open source 2d vision transformer code. the pretrained weights for 2d also works for 3d image data.

\n

you can think that the below is approximately true :
\nconv3d(x,w) = conv2d(x.mean(0), w.mean(0))

\n

just like
\nconv2d(x_10x10_patch,w_10x10_patch) = conv2d(x_resize_to_5x5, w_resize_to_5x5)

", "rawMarkdown": "to be updated ....\n\nbaseline model:\n\n\"Selection-176\"\n\napproximate plan:\n\n1. design a framework with use of aux label (segmentation and box)\n2. compare results of 3d transformer (including pyramid vit), 3d cnn and 3d hybrid and others (2D 2.5D, LSTM, etc ...)\n3. apply patch pretraining to transformer \n4. apply human in the loop active learning (click point annotation) \n\n\nwhy use transformer?\n- the receptive field is large: e.g. you need to see the whole cervical spine (C1 to C8) to know which is the \"section of interest (e.g. Cn)\n- there is only \"one layout/structure\". the body face the same direction, etc... position encoding works well as each body parts are fairly similar in both appearance (i.e. can be tokenized) and location(i.e. can be encoded by pos encoding).\n- because of the above, there is great chance that pretraining can boost performance\n\n\nreference paper:\n\"Selection-177\"\n\n\n---\n\nthe nice thing about transformer is that i just need to change the vision tokenization layer for 2d input to 3d input. The rest of the transformer layers can remains unchanged. Hence i can easily modify any of the existing open source 2d vision transformer code. the pretrained weights for 2d also works for 3d image data.\n\nyou can think that the below is approximately true :\nconv3d(x,w) = conv2d(x.mean(0), w.mean(0)) \n\njust like\nconv2d(x\\_10x10_patch,w\\_10x10_patch) = conv2d(x\\_resize_to_5x5, w\\_resize_to_5x5) \n", "votes": 56 }, { "id": 1935256, "postDate": "2022-09-12T01:25:33.787Z", "content": "

initial 3d transformer results

\n

i modified mix transformer as follows. as a proof of concept, i trained 3d cervical spine segmentation with the image token as aux loss.
\ni haven trained the cls token yet.

\n

using persistent cache of monai, it took 60 min to train with all 87 segment samples in about 60 epochs for a train loss 0.023 (multi-class segmentation cross entropy). as a reference simple resnet183d-unet takes about 35 min for the same setting.

\n

train = 8x 256x256x256 crops at spacing dim =0.6,0.6,0.6
\ntest = varying single input size without cropping

\n

code to be released soon

\n

\"Selection-223\"
\n\"Selection-222\"

", "rawMarkdown": "initial 3d transformer results\n\ni modified mix transformer as follows. as a proof of concept, i trained 3d cervical spine segmentation with the image token as aux loss.\ni haven trained the cls token yet.\n\nusing persistent cache of monai, it took 60 min to train with all 87 segment samples in about 60 epochs for a train loss 0.023 (multi-class segmentation cross entropy). as a reference simple resnet183d-unet takes about 35 min for the same setting.\n\ntrain = 8x 256x256x256 crops at spacing dim =0.6,0.6,0.6\ntest = varying single input size without cropping\n\ncode to be released soon\n\n\"Selection-223\"\n\"Selection-222\"", "votes": 8, "replies": [ { "id": 1936451, "postDate": "2022-09-12T18:52:29.597Z", "content": "

some updated results

\n

\"Selection-241\"
\n\"Selection-243\"
\n\"Selection-242\"

", "rawMarkdown": "some updated results\n\n\"Selection-241\" \n\"Selection-243\"\n\"Selection-242\"", "votes": 1 }, { "id": 1936491, "postDate": "2022-09-12T19:26:02.427Z", "content": "

type of cervical-spine-fractures:

\n

https://www.youtube.com/watch?v=6itvmj7y5TE
\nthis is good. it shows you how to read from Ct scan

\n

others:
\nhttps://www.youtube.com/watch?v=jY2lXySPiAw
\nhttps://www.youtube.com/watch?v=mU75SnzPlbc
\nhttps://www.nuemblog.com/blog/cervical-spine-fractures
\nhttps://teachmesurgery.com/orthopaedic/spine/cervical-fracture/

", "rawMarkdown": "type of cervical-spine-fractures:\n\nhttps://www.youtube.com/watch?v=6itvmj7y5TE\nthis is good. it shows you how to read from Ct scan\n\n\nothers:\nhttps://www.youtube.com/watch?v=jY2lXySPiAw\nhttps://www.youtube.com/watch?v=mU75SnzPlbc\nhttps://www.nuemblog.com/blog/cervical-spine-fractures\nhttps://teachmesurgery.com/orthopaedic/spine/cervical-fracture/", "votes": 1 } ] }, { "id": 1929857, "postDate": "2022-09-07T11:59:13.193Z", "content": "

how to train on high resolution:

\n

stage one

\n\n

stage two

\n\n
\n

how to prevent overfitting when apply transformer on small data?

\n\n

paper for training transformer from scratch:

\n

[1] Training Vision Transformers with Only 2040 Images
\nhttps://arxiv.org/abs/2201.10728

\n

[2] WHEN VISION TRANSFORMERS OUTPERFORM RESNETS WITHOUT PRE-TRAINING OR STRONG DATA AUGMENTATIONS
\nhttps://arxiv.org/pdf/2106.01548.pdf
\nrelated: Towards Efficient and Scalable Sharpness-Aware Minimization

\n

[3] Efficient Training of Visual Transformers with Small Datasets
\nhttps://openreview.net/pdf?id=AJofO-OFT40

\n

[4] Training Object Detectors from Scratch: An Empirical Study in the Era of Vision Transformer
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Hong_Training_Object_Detectors_From_Scratch_An_Empirical_Study_in_the_CVPR_2022_paper.pdf

", "rawMarkdown": "how to train on high resolution:\n\nstage one\n- ignore cls. train toke loss in 2d crops (e.g. 196x196x196)\n\nstage two\n- freeze some initial layers. train both cls and token loss\n\n\n---\n\nhow to prevent overfitting when apply transformer on small data?\n- assume transformer has N layers. Output from intermediate layer e.g. 3,8,12,N. For each of the output, apply token loss. the loss weights for different output layers can be high to low.\n\npaper for training transformer from scratch:\n\n[1] Training Vision Transformers with Only 2040 Images \nhttps://arxiv.org/abs/2201.10728\n\n[2] WHEN VISION TRANSFORMERS OUTPERFORM RESNETS WITHOUT PRE-TRAINING OR STRONG DATA AUGMENTATIONS\nhttps://arxiv.org/pdf/2106.01548.pdf\nrelated: Towards Efficient and Scalable Sharpness-Aware Minimization\n\n[3] Efficient Training of Visual Transformers with Small Datasets\nhttps://openreview.net/pdf?id=AJofO-OFT40\n\n[4] Training Object Detectors from Scratch: An Empirical Study in the Era of Vision Transformer\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Hong_Training_Object_Detectors_From_Scratch_An_Empirical_Study_in_the_CVPR_2022_paper.pdf \n", "votes": 2 }, { "id": 1929838, "postDate": "2022-09-07T11:49:53.713Z", "content": "

background reading:

\n

[1] Deep Sequential Learning for Cervical Spine Fracture Detection on Computed Tomography Imaging
\nhttps://arxiv.org/abs/2010.13336

\n

[2] CT Cervical Spine Fracture Detection Using a Convolutional Neural Network
\nhttp://www.ajnr.org/content/ajnr/42/7/1341.full.pdf

\n

[3] Reinventing 2D Convolutions for 3D Medical Images
\nhttps://www.arxiv-vanity.com/papers/1911.10477/

\n

[4] A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs
\nhttps://www.nature.com/articles/s41467-021-21311-3.pdf

\n

[5] Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.pdf

\n

list of 3d transformer
\nhttps://github.com/lahoud/3d-vision-transformers

\n
\n

Detecting pelvic fracture on 3D-CT using deep convolutional neural networks with multi-orientated slab images
\nhttps://www.nature.com/articles/s41598-021-91144-z

\n

Automatic Vertebra Localization and Identification in CT by Spine Rectification and Anatomically-constrained Optimization -cvpr2021
\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Wang_Automatic_Vertebra_Localization_and_Identification_in_CT_by_Spine_Rectification_CVPR_2021_paper.pdf

", "rawMarkdown": "background reading:\n\n[1] Deep Sequential Learning for Cervical Spine Fracture Detection on Computed Tomography Imaging \nhttps://arxiv.org/abs/2010.13336\n\n[2] CT Cervical Spine Fracture Detection Using a Convolutional Neural Network \nhttp://www.ajnr.org/content/ajnr/42/7/1341.full.pdf \n\n[3] Reinventing 2D Convolutions for 3D Medical Images\nhttps://www.arxiv-vanity.com/papers/1911.10477/\n\n[4] A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs\nhttps://www.nature.com/articles/s41467-021-21311-3.pdf\n\n[5] Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.pdf\n\nlist of 3d transformer\nhttps://github.com/lahoud/3d-vision-transformers\n\n---\n\nDetecting pelvic fracture on 3D-CT using deep convolutional neural networks with multi-orientated slab images\nhttps://www.nature.com/articles/s41598-021-91144-z\n\nAutomatic Vertebra Localization and Identification in CT by Spine Rectification and Anatomically-constrained Optimization -cvpr2021 \nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Wang_Automatic_Vertebra_Localization_and_Identification_in_CT_by_Spine_Rectification_CVPR_2021_paper.pdf ", "votes": 2, "replies": [ { "id": 1930358, "postDate": "2022-09-07T19:08:00.150Z", "content": "

i am very interested in using point annotation to help fracture detection.
\nbecause it is easy to mark the unlabelled CT spine scans (we just need to identify the correct spine section)

\n

here is another paper:
\n[4a] A New Window Loss Function for Bone Fracture Detection and Localization in X-ray Images with Point-based Annotation
\nhttps://www.catalyzex.com/paper/arxiv:2012.04066

\n

\"Selection-170\"
\n\"Selection-180\"
\n\"Selection-178\"

\n

with more annotation, we can get better public/private board ranking

\n
\n

prostateX MRI 3d dataset is also using point annotation for classification

\n

https://towardsdatascience.com/3d-cnn-classification-of-prostate-tumour-on-multi-parametric-mri-sequences-prostatex-2-cced525394bb

", "rawMarkdown": "i am very interested in using point annotation to help fracture detection.\nbecause it is easy to mark the unlabelled CT spine scans (we just need to identify the correct spine section)\n\nhere is another paper:\n[4a] A New Window Loss Function for Bone Fracture Detection and Localization in X-ray Images with Point-based Annotation\nhttps://www.catalyzex.com/paper/arxiv:2012.04066\n\n\"Selection-170\"\n\"Selection-180\"\n\"Selection-178\"\n\nwith more annotation, we can get better public/private board ranking\n\n---\n\nprostateX MRI 3d dataset is also using point annotation for classification\n\nhttps://towardsdatascience.com/3d-cnn-classification-of-prostate-tumour-on-multi-parametric-mri-sequences-prostatex-2-cced525394bb", "votes": 2 }, { "id": 1930391, "postDate": "2022-09-07T20:09:13.843Z", "content": "

external dataset:
\n(assuming that self supervised learning of other spline can help cervical-spine?)

\n

[1] CTSpine1K: A Large-Scale Dataset for Spinal Vertebrae Segmentation in Computed Tomography
\nhttps://arxiv.org/pdf/2105.14711.pdf

\n

[2] A Vertebral Segmentation Dataset with Fracture Grading
\nhttps://pubs.rsna.org/doi/epdf/10.1148/ryai.2020190138

\n

http://spineweb.digitalimaginggroup.ca/
\nhttps://verse2019.grand-challenge.org/
\nhttps://verse2020.grand-challenge.org
\nhttps://github.com/anjany/verse

\n

http://lit.fe.uni-lj.si/xVertSeg/overview.php#12

\n

https://biomedia.doc.ic.ac.uk/data/spine/
\nhttps://github.com/jfm15/SpineFinder

\n

extra label:
\nhttps://anduin.bonescreen.de/
\nAnduin is a freely available research tool to segment vertebrae in a CT scan, uploaded as NIFTI data.

\n
\n

from other kagglers:

\n

credits goes to @ipythonx

\n

https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/341140

\n\n

https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340612#1911268

\n", "rawMarkdown": "external dataset:\n(assuming that self supervised learning of other spline can help cervical-spine?)\n\n[1] CTSpine1K: A Large-Scale Dataset for Spinal Vertebrae Segmentation in Computed Tomography\nhttps://arxiv.org/pdf/2105.14711.pdf\n\n[2] A Vertebral Segmentation Dataset with Fracture Grading\nhttps://pubs.rsna.org/doi/epdf/10.1148/ryai.2020190138\n\nhttp://spineweb.digitalimaginggroup.ca/\nhttps://verse2019.grand-challenge.org/\nhttps://verse2020.grand-challenge.org\nhttps://github.com/anjany/verse\n\nhttp://lit.fe.uni-lj.si/xVertSeg/overview.php#12\n\nhttps://biomedia.doc.ic.ac.uk/data/spine/\nhttps://github.com/jfm15/SpineFinder\n\nextra label: \nhttps://anduin.bonescreen.de/\nAnduin is a freely available research tool to segment vertebrae in a CT scan, uploaded as NIFTI data.\n\n---\nfrom other kagglers:\n\ncredits goes to @ipythonx\n\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/341140\n- https://www.radimagenet.com/\n- https://github.com/BMEII-AI/RadImageNet\n\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340612#1911268\n- https://arxiv.org/pdf/2208.05868.pdf\n- https://github.com/wasserth/TotalSegmentator\n- https://zenodo.org/record/6802614#.YwWGAXZByHs", "votes": 1 }, { "id": 1931118, "postDate": "2022-09-08T13:08:13.307Z", "content": "

when it comes to semi-supervised learning in medical problems, kaggle can consider use of medical records.

\n

we can extract weak labels from medical records.
\ne.g. the doctor may indicate the presence of xxx fracture or no abnormality in the CT scan.

\n

this requires additional knowledge of NLP in understanding medical records.

", "rawMarkdown": "when it comes to semi-supervised learning in medical problems, kaggle can consider use of medical records.\n\nwe can extract weak labels from medical records.\ne.g. the doctor may indicate the presence of xxx fracture or no abnormality in the CT scan. \n\nthis requires additional knowledge of NLP in understanding medical records. " }, { "id": 1937400, "postDate": "2022-09-13T13:14:53.343Z", "content": "

is radiomic Features useful?

\n

these are texture/histogram features like Gray level co-occurence matrices, etc

\n", "rawMarkdown": "is radiomic Features useful?\n\nthese are texture/histogram features like Gray level co-occurence matrices, etc\n\n- Prediction of the Acuity of Vertebral Compression Fractures on CT Using Radiologic and Radiomic Features \n- Artificial intelligence‑based radiomics on computed tomography of lumbar spine in subjects with fragility vertebral fractures \nhttps://link.springer.com/content/pdf/10.1007/s40618-022-01837-z.pdf" } ] }, { "id": 1937328, "postDate": "2022-09-13T12:22:48.747Z", "content": "

if you want to read about commercial system, here is one:

\n
\n

AI for C-Spine Fractures: Aidoc Sets the Pace With 3rd FDA
\nhttps://www.prnewswire.com/news-releases/ai-for-c-spine-fractures-aidoc-sets-the-pace-with-3rd-fda-clearance-in-9-months-300865221.html

\n
\n

paper:
\nhttps://www.aidoc.com/blog/clinical-impact-ai-triaging-c-spine-fractures/
\nCT Cervical Spine Fracture Detection Using a Convolutional Neural Network

\n

\"The cervical spine fracture detection model consists of 2 stages:
\na region proposal stage and a false-positive reduction stage. The
\nfirst stage is a 3D fully convolutional deep neural network. The
\narchitecture is based on the Residual Network architecture, which
\nconsists of repeated blocks of several convolutional layers with skip
\nconnections between them, and is followed by a pooling layer that
\nreduces the dimensions of the output. This network is trained on
\nsegmented scans and produces a 3D segmentation map. The
\nmodel was trained from scratch, with no pretraining from addi-
\ntional datasets. From the segmentation map, region proposals are
\nextracted and passed as input to the second stage of the algorithm.
\nThe second stage classifies each region as positive or negative. Two
\nsets of features are extracted from each region, fused together, and
\nused for the final decision. The first are learned features from a
\nmultilayered, classification head that receives the features from the
\nlast layer of the 3D segmentation network for the proposed regions
\nas input. The second are nonlearned engineered features obtained
\nfrom traditional image-processing methods that operate on the
\nproposed regions. These features are combined through an addi-
\ntional neural network, which classifies each proposal as a fracture
\nor not\"

\n
\n

https://pubmed.ncbi.nlm.nih.gov/34117018/
\nDiagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Cervical Spine Fractures
\nerror analysis
\n\" Aidoc and the neuroradiologist's interpretation were concordant in 91.5% of cases. Aidoc correctly identified 67 of 122 fractures (54.9%) with 106 false-positive flagged studies. Diagnostic performance was calculated as the following: sensitivity, 54.9% (95% CI, 45.7%-63.9%); specificity, 94.1% (95% CI, 92.9%-95.1%); positive predictive value, 38.7% (95% CI, 33.1%-44.7%); and negative predictive value, 96.8% (95% CI, 96.2%-97.4%). Worsened performance was observed in the detection of chronic fractures; differences in diagnostic performance were not altered by study indication or patient characteristics. \"

", "rawMarkdown": "if you want to read about commercial system, here is one:\n\n---\n\nAI for C-Spine Fractures: Aidoc Sets the Pace With 3rd FDA\nhttps://www.prnewswire.com/news-releases/ai-for-c-spine-fractures-aidoc-sets-the-pace-with-3rd-fda-clearance-in-9-months-300865221.html\n\n---\n\npaper:\nhttps://www.aidoc.com/blog/clinical-impact-ai-triaging-c-spine-fractures/\nCT Cervical Spine Fracture Detection Using a Convolutional Neural Network\n\n\"The cervical spine fracture detection model consists of 2 stages:\na region proposal stage and a false-positive reduction stage. The\nfirst stage is a 3D fully convolutional deep neural network. The\narchitecture is based on the Residual Network architecture, which\nconsists of repeated blocks of several convolutional layers with skip\nconnections between them, and is followed by a pooling layer that\nreduces the dimensions of the output. This network is trained on\nsegmented scans and produces a 3D segmentation map. The\nmodel was trained from scratch, with no pretraining from addi-\ntional datasets. From the segmentation map, region proposals are\nextracted and passed as input to the second stage of the algorithm.\nThe second stage classifies each region as positive or negative. Two\nsets of features are extracted from each region, fused together, and\nused for the final decision. The first are learned features from a\nmultilayered, classification head that receives the features from the\nlast layer of the 3D segmentation network for the proposed regions\nas input. The second are nonlearned engineered features obtained\nfrom traditional image-processing methods that operate on the\nproposed regions. These features are combined through an addi-\ntional neural network, which classifies each proposal as a fracture\nor not\"\n\n\n---\n\nhttps://pubmed.ncbi.nlm.nih.gov/34117018/\nDiagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Cervical Spine Fractures \nerror analysis\n\" Aidoc and the neuroradiologist's interpretation were concordant in 91.5% of cases. Aidoc correctly identified 67 of 122 fractures (54.9%) with 106 false-positive flagged studies. Diagnostic performance was calculated as the following: sensitivity, 54.9% (95% CI, 45.7%-63.9%); specificity, 94.1% (95% CI, 92.9%-95.1%); positive predictive value, 38.7% (95% CI, 33.1%-44.7%); and negative predictive value, 96.8% (95% CI, 96.2%-97.4%). Worsened performance was observed in the detection of chronic fractures; differences in diagnostic performance were not altered by study indication or patient characteristics. \"\n\n" }, { "id": 1932904, "postDate": "2022-09-10T05:33:11.193Z", "content": "

Very insightful content from you as always. I will try on it on the data to see the result.

", "rawMarkdown": "Very insightful content from you as always. I will try on it on the data to see the result." } ], "comments": [ { "id": 1935256, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-12T01:25:33.787000", "content": "

initial 3d transformer results

\n

i modified mix transformer as follows. as a proof of concept, i trained 3d cervical spine segmentation with the image token as aux loss.
\ni haven trained the cls token yet.

\n

using persistent cache of monai, it took 60 min to train with all 87 segment samples in about 60 epochs for a train loss 0.023 (multi-class segmentation cross entropy). as a reference simple resnet183d-unet takes about 35 min for the same setting.

\n

train = 8x 256x256x256 crops at spacing dim =0.6,0.6,0.6
\ntest = varying single input size without cropping

\n

code to be released soon

\n

\"Selection-223\"
\n\"Selection-222\"

", "votes": 8, "replies": [ { "id": 1936451, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-12T18:52:29.597000", "content": "

some updated results

\n

\"Selection-241\"
\n\"Selection-243\"
\n\"Selection-242\"

", "votes": 1, "replies": [] }, { "id": 1936491, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-12T19:26:02.427000", "content": "

type of cervical-spine-fractures:

\n

https://www.youtube.com/watch?v=6itvmj7y5TE
\nthis is good. it shows you how to read from Ct scan

\n

others:
\nhttps://www.youtube.com/watch?v=jY2lXySPiAw
\nhttps://www.youtube.com/watch?v=mU75SnzPlbc
\nhttps://www.nuemblog.com/blog/cervical-spine-fractures
\nhttps://teachmesurgery.com/orthopaedic/spine/cervical-fracture/

", "votes": 1, "replies": [] } ] }, { "id": 1929857, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-07T11:59:13.193000", "content": "

how to train on high resolution:

\n

stage one

\n\n

stage two

\n\n
\n

how to prevent overfitting when apply transformer on small data?

\n\n

paper for training transformer from scratch:

\n

[1] Training Vision Transformers with Only 2040 Images
\nhttps://arxiv.org/abs/2201.10728

\n

[2] WHEN VISION TRANSFORMERS OUTPERFORM RESNETS WITHOUT PRE-TRAINING OR STRONG DATA AUGMENTATIONS
\nhttps://arxiv.org/pdf/2106.01548.pdf
\nrelated: Towards Efficient and Scalable Sharpness-Aware Minimization

\n

[3] Efficient Training of Visual Transformers with Small Datasets
\nhttps://openreview.net/pdf?id=AJofO-OFT40

\n

[4] Training Object Detectors from Scratch: An Empirical Study in the Era of Vision Transformer
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Hong_Training_Object_Detectors_From_Scratch_An_Empirical_Study_in_the_CVPR_2022_paper.pdf

", "votes": 2, "replies": [] }, { "id": 1929838, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-07T11:49:53.713000", "content": "

background reading:

\n

[1] Deep Sequential Learning for Cervical Spine Fracture Detection on Computed Tomography Imaging
\nhttps://arxiv.org/abs/2010.13336

\n

[2] CT Cervical Spine Fracture Detection Using a Convolutional Neural Network
\nhttp://www.ajnr.org/content/ajnr/42/7/1341.full.pdf

\n

[3] Reinventing 2D Convolutions for 3D Medical Images
\nhttps://www.arxiv-vanity.com/papers/1911.10477/

\n

[4] A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs
\nhttps://www.nature.com/articles/s41467-021-21311-3.pdf

\n

[5] Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis
\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.pdf

\n

list of 3d transformer
\nhttps://github.com/lahoud/3d-vision-transformers

\n
\n

Detecting pelvic fracture on 3D-CT using deep convolutional neural networks with multi-orientated slab images
\nhttps://www.nature.com/articles/s41598-021-91144-z

\n

Automatic Vertebra Localization and Identification in CT by Spine Rectification and Anatomically-constrained Optimization -cvpr2021
\nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Wang_Automatic_Vertebra_Localization_and_Identification_in_CT_by_Spine_Rectification_CVPR_2021_paper.pdf

", "votes": 2, "replies": [ { "id": 1930358, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-07T19:08:00.150000", "content": "

i am very interested in using point annotation to help fracture detection.
\nbecause it is easy to mark the unlabelled CT spine scans (we just need to identify the correct spine section)

\n

here is another paper:
\n[4a] A New Window Loss Function for Bone Fracture Detection and Localization in X-ray Images with Point-based Annotation
\nhttps://www.catalyzex.com/paper/arxiv:2012.04066

\n

\"Selection-170\"
\n\"Selection-180\"
\n\"Selection-178\"

\n

with more annotation, we can get better public/private board ranking

\n
\n

prostateX MRI 3d dataset is also using point annotation for classification

\n

https://towardsdatascience.com/3d-cnn-classification-of-prostate-tumour-on-multi-parametric-mri-sequences-prostatex-2-cced525394bb

", "votes": 2, "replies": [] }, { "id": 1930391, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-07T20:09:13.843000", "content": "

external dataset:
\n(assuming that self supervised learning of other spline can help cervical-spine?)

\n

[1] CTSpine1K: A Large-Scale Dataset for Spinal Vertebrae Segmentation in Computed Tomography
\nhttps://arxiv.org/pdf/2105.14711.pdf

\n

[2] A Vertebral Segmentation Dataset with Fracture Grading
\nhttps://pubs.rsna.org/doi/epdf/10.1148/ryai.2020190138

\n

http://spineweb.digitalimaginggroup.ca/
\nhttps://verse2019.grand-challenge.org/
\nhttps://verse2020.grand-challenge.org
\nhttps://github.com/anjany/verse

\n

http://lit.fe.uni-lj.si/xVertSeg/overview.php#12

\n

https://biomedia.doc.ic.ac.uk/data/spine/
\nhttps://github.com/jfm15/SpineFinder

\n

extra label:
\nhttps://anduin.bonescreen.de/
\nAnduin is a freely available research tool to segment vertebrae in a CT scan, uploaded as NIFTI data.

\n
\n

from other kagglers:

\n

credits goes to @ipythonx

\n

https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/341140

\n\n

https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/340612#1911268

\n", "votes": 1, "replies": [] }, { "id": 1931118, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-08T13:08:13.307000", "content": "

when it comes to semi-supervised learning in medical problems, kaggle can consider use of medical records.

\n

we can extract weak labels from medical records.
\ne.g. the doctor may indicate the presence of xxx fracture or no abnormality in the CT scan.

\n

this requires additional knowledge of NLP in understanding medical records.

", "votes": 0, "replies": [] }, { "id": 1937400, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-13T13:14:53.343000", "content": "

is radiomic Features useful?

\n

these are texture/histogram features like Gray level co-occurence matrices, etc

\n", "votes": 0, "replies": [] } ] }, { "id": 1937328, "author_name": "hengck23", "author_url": "", "post_date": "2022-09-13T12:22:48.747000", "content": "

if you want to read about commercial system, here is one:

\n
\n

AI for C-Spine Fractures: Aidoc Sets the Pace With 3rd FDA
\nhttps://www.prnewswire.com/news-releases/ai-for-c-spine-fractures-aidoc-sets-the-pace-with-3rd-fda-clearance-in-9-months-300865221.html

\n
\n

paper:
\nhttps://www.aidoc.com/blog/clinical-impact-ai-triaging-c-spine-fractures/
\nCT Cervical Spine Fracture Detection Using a Convolutional Neural Network

\n

\"The cervical spine fracture detection model consists of 2 stages:
\na region proposal stage and a false-positive reduction stage. The
\nfirst stage is a 3D fully convolutional deep neural network. The
\narchitecture is based on the Residual Network architecture, which
\nconsists of repeated blocks of several convolutional layers with skip
\nconnections between them, and is followed by a pooling layer that
\nreduces the dimensions of the output. This network is trained on
\nsegmented scans and produces a 3D segmentation map. The
\nmodel was trained from scratch, with no pretraining from addi-
\ntional datasets. From the segmentation map, region proposals are
\nextracted and passed as input to the second stage of the algorithm.
\nThe second stage classifies each region as positive or negative. Two
\nsets of features are extracted from each region, fused together, and
\nused for the final decision. The first are learned features from a
\nmultilayered, classification head that receives the features from the
\nlast layer of the 3D segmentation network for the proposed regions
\nas input. The second are nonlearned engineered features obtained
\nfrom traditional image-processing methods that operate on the
\nproposed regions. These features are combined through an addi-
\ntional neural network, which classifies each proposal as a fracture
\nor not\"

\n
\n

https://pubmed.ncbi.nlm.nih.gov/34117018/
\nDiagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Cervical Spine Fractures
\nerror analysis
\n\" Aidoc and the neuroradiologist's interpretation were concordant in 91.5% of cases. Aidoc correctly identified 67 of 122 fractures (54.9%) with 106 false-positive flagged studies. Diagnostic performance was calculated as the following: sensitivity, 54.9% (95% CI, 45.7%-63.9%); specificity, 94.1% (95% CI, 92.9%-95.1%); positive predictive value, 38.7% (95% CI, 33.1%-44.7%); and negative predictive value, 96.8% (95% CI, 96.2%-97.4%). Worsened performance was observed in the detection of chronic fractures; differences in diagnostic performance were not altered by study indication or patient characteristics. \"

", "votes": 0, "replies": [] }, { "id": 1932904, "author_name": "Rahul Pandey", "author_url": "", "post_date": "2022-09-10T05:33:11.193000", "content": "

Very insightful content from you as always. I will try on it on the data to see the result.

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1929822": "to be updated ....\n\nbaseline model:\n\n\"Selection-176\"\n\napproximate plan:\n\n1. design a framework with use of aux label (segmentation and box)\n2. compare results of 3d transformer (including pyramid vit), 3d cnn and 3d hybrid and others (2D 2.5D, LSTM, etc ...)\n3. apply patch pretraining to transformer \n4. apply human in the loop active learning (click point annotation) \n\n\nwhy use transformer?\n- the receptive field is large: e.g. you need to see the whole cervical spine (C1 to C8) to know which is the \"section of interest (e.g. Cn)\n- there is only \"one layout/structure\". the body face the same direction, etc... position encoding works well as each body parts are fairly similar in both appearance (i.e. can be tokenized) and location(i.e. can be encoded by pos encoding).\n- because of the above, there is great chance that pretraining can boost performance\n\n\nreference paper:\n\"Selection-177\"\n\n\n---\n\nthe nice thing about transformer is that i just need to change the vision tokenization layer for 2d input to 3d input. The rest of the transformer layers can remains unchanged. Hence i can easily modify any of the existing open source 2d vision transformer code. the pretrained weights for 2d also works for 3d image data.\n\nyou can think that the below is approximately true :\nconv3d(x,w) = conv2d(x.mean(0), w.mean(0)) \n\njust like\nconv2d(x\\_10x10_patch,w\\_10x10_patch) = conv2d(x\\_resize_to_5x5, w\\_resize_to_5x5) \n", "1935256": "initial 3d transformer results\n\ni modified mix transformer as follows. as a proof of concept, i trained 3d cervical spine segmentation with the image token as aux loss.\ni haven trained the cls token yet.\n\nusing persistent cache of monai, it took 60 min to train with all 87 segment samples in about 60 epochs for a train loss 0.023 (multi-class segmentation cross entropy). as a reference simple resnet183d-unet takes about 35 min for the same setting.\n\ntrain = 8x 256x256x256 crops at spacing dim =0.6,0.6,0.6\ntest = varying single input size without cropping\n\ncode to be released soon\n\n\"Selection-223\"\n\"Selection-222\"", "1929857": "how to train on high resolution:\n\nstage one\n- ignore cls. train toke loss in 2d crops (e.g. 196x196x196)\n\nstage two\n- freeze some initial layers. train both cls and token loss\n\n\n---\n\nhow to prevent overfitting when apply transformer on small data?\n- assume transformer has N layers. Output from intermediate layer e.g. 3,8,12,N. For each of the output, apply token loss. the loss weights for different output layers can be high to low.\n\npaper for training transformer from scratch:\n\n[1] Training Vision Transformers with Only 2040 Images \nhttps://arxiv.org/abs/2201.10728\n\n[2] WHEN VISION TRANSFORMERS OUTPERFORM RESNETS WITHOUT PRE-TRAINING OR STRONG DATA AUGMENTATIONS\nhttps://arxiv.org/pdf/2106.01548.pdf\nrelated: Towards Efficient and Scalable Sharpness-Aware Minimization\n\n[3] Efficient Training of Visual Transformers with Small Datasets\nhttps://openreview.net/pdf?id=AJofO-OFT40\n\n[4] Training Object Detectors from Scratch: An Empirical Study in the Era of Vision Transformer\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Hong_Training_Object_Detectors_From_Scratch_An_Empirical_Study_in_the_CVPR_2022_paper.pdf \n", "1929838": "background reading:\n\n[1] Deep Sequential Learning for Cervical Spine Fracture Detection on Computed Tomography Imaging \nhttps://arxiv.org/abs/2010.13336\n\n[2] CT Cervical Spine Fracture Detection Using a Convolutional Neural Network \nhttp://www.ajnr.org/content/ajnr/42/7/1341.full.pdf \n\n[3] Reinventing 2D Convolutions for 3D Medical Images\nhttps://www.arxiv-vanity.com/papers/1911.10477/\n\n[4] A scalable physician-level deep learning algorithm detects universal trauma on pelvic radiographs\nhttps://www.nature.com/articles/s41467-021-21311-3.pdf\n\n[5] Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis\nhttps://openaccess.thecvf.com/content/CVPR2022/papers/Tang_Self-Supervised_Pre-Training_of_Swin_Transformers_for_3D_Medical_Image_Analysis_CVPR_2022_paper.pdf\n\nlist of 3d transformer\nhttps://github.com/lahoud/3d-vision-transformers\n\n---\n\nDetecting pelvic fracture on 3D-CT using deep convolutional neural networks with multi-orientated slab images\nhttps://www.nature.com/articles/s41598-021-91144-z\n\nAutomatic Vertebra Localization and Identification in CT by Spine Rectification and Anatomically-constrained Optimization -cvpr2021 \nhttps://openaccess.thecvf.com/content/CVPR2021/papers/Wang_Automatic_Vertebra_Localization_and_Identification_in_CT_by_Spine_Rectification_CVPR_2021_paper.pdf ", "1937328": "if you want to read about commercial system, here is one:\n\n---\n\nAI for C-Spine Fractures: Aidoc Sets the Pace With 3rd FDA\nhttps://www.prnewswire.com/news-releases/ai-for-c-spine-fractures-aidoc-sets-the-pace-with-3rd-fda-clearance-in-9-months-300865221.html\n\n---\n\npaper:\nhttps://www.aidoc.com/blog/clinical-impact-ai-triaging-c-spine-fractures/\nCT Cervical Spine Fracture Detection Using a Convolutional Neural Network\n\n\"The cervical spine fracture detection model consists of 2 stages:\na region proposal stage and a false-positive reduction stage. The\nfirst stage is a 3D fully convolutional deep neural network. The\narchitecture is based on the Residual Network architecture, which\nconsists of repeated blocks of several convolutional layers with skip\nconnections between them, and is followed by a pooling layer that\nreduces the dimensions of the output. This network is trained on\nsegmented scans and produces a 3D segmentation map. The\nmodel was trained from scratch, with no pretraining from addi-\ntional datasets. From the segmentation map, region proposals are\nextracted and passed as input to the second stage of the algorithm.\nThe second stage classifies each region as positive or negative. Two\nsets of features are extracted from each region, fused together, and\nused for the final decision. The first are learned features from a\nmultilayered, classification head that receives the features from the\nlast layer of the 3D segmentation network for the proposed regions\nas input. The second are nonlearned engineered features obtained\nfrom traditional image-processing methods that operate on the\nproposed regions. These features are combined through an addi-\ntional neural network, which classifies each proposal as a fracture\nor not\"\n\n\n---\n\nhttps://pubmed.ncbi.nlm.nih.gov/34117018/\nDiagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Cervical Spine Fractures \nerror analysis\n\" Aidoc and the neuroradiologist's interpretation were concordant in 91.5% of cases. Aidoc correctly identified 67 of 122 fractures (54.9%) with 106 false-positive flagged studies. Diagnostic performance was calculated as the following: sensitivity, 54.9% (95% CI, 45.7%-63.9%); specificity, 94.1% (95% CI, 92.9%-95.1%); positive predictive value, 38.7% (95% CI, 33.1%-44.7%); and negative predictive value, 96.8% (95% CI, 96.2%-97.4%). Worsened performance was observed in the detection of chronic fractures; differences in diagnostic performance were not altered by study indication or patient characteristics. \"\n\n", "1932904": "Very insightful content from you as always. I will try on it on the data to see the result." } }