{ "id": 362607, "title": "1st Place Solution", "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/362607", "author_name": "Qishen Ha", "post_date": "2022-10-28T03:58:02.851000", "votes": 222, "comment_count": 87, "views": 0, "content": "

Thanks to the organizers and congrats to all the winners and those who worked hard to develop new pipelines and stuck with it until the end of the competition.

\n

This is a very interesting competition, because we can think of many different ways to approach this dataset. Therefore the most important thing for this competition is to develop a reasonable pipeline, followed by optimization of the model.

\n

Code

\n

https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/362787

\n

Summary

\n

I designed a 2-stage pipeline to deal with this problem.

\n

stage1: 3D semantic segmentation -> stage2: 2.5D w/ LSTM classification.

\n

In addition, there are 2 different types of classification models in stage2.

\n

3D Semantic Segmentation

\n

For 3D semantic segmentation, we only have 87 samples w/ 3d mask in the dataset, but it's sufficient to train 3D semantic segmentation models with good performance.

\n

I use 128x128x128 input, to train resnet18d or efficientnet v2s + unet model, for segmenting C[1-7] vertebraes (7ch output).

\n

After the training was completed, I predicted 3d masks for each vertebrae for all 2k samples in the training set.

\n

Here is an example of predicted masks of C[1-7] vertebrae. Center slice of x, y, z dimension view, from left to right.
\n​
\n\"\"

\n

Prepare Data for Classification

\n

Next step is to prepare data for classification.

\n

First using the predicted 3D mask for each vertebrae, we can crop out 7 vertebraes from a single original 3d image (there might be multiple vertebraes shown in a single crop, but It's fine). At this moment, we cropped 2k * 7 = 14k samples and for each sample there is only one single binary label.

\n

Then for each vertebrae sample, I extracted 15 slices evenly by z-dimension, and for each slice, I further extracted +-2 adjacent slices to form an image with 5 channels. E.g if a 3D vertebrae sample have a shape of (128, 128, 30), I extracted 0th, 2nd, 4th, 6th….26th, 28th slices, then for example for the 2nd one, I use 0th~4th slices to form a 5-channel image.

\n

In addition, I added the predicted mask of corresponding vertebrae as the 6th channel to each image, as a way to exclude the effect of having multiple vertebraes in a single sample.

\n

Here is an example of one slice of a single vertebrae, and its predicted mask (with augmentations). We can see that the left half of the vertebrae in the image do not belong to the vertebrae specified by this crop.
\n​
\n\"\"

\n

2.5D + LSTM Classification

\n

We now have 14k 3D training samples of vertebrae. Theoretically the easiest way to deal with this data is to train 3D CNN on it. But unfortunately this method does not work. Training a 3D CNN on this data did not give me satisfactory results.

\n

So I backed off and chose the 2.5D approach. Here 2.5D means that each 2D slice in a vertebrae sample has the information of several adjacent slices, so it is written 2.5D. But the model is a normal 2D CNN with 5-channels input.

\n

The structure of this model is that, I first input 15 slices from a single sample into a 2D CNN, extracted out features of each slice, and then follow it with an LSTM model. So that the whole model can learn the features of the whole vertebrae. I call it type1 model ↓

\n

​
\n\"\"

\n

This model structure above, while being able to train a single vertebrae for fracture, does not able to train the patient as a whole for the presence of a fracture. So I designed another model.

\n

The second classification model is basically the same as the one above, except that it treats a patient as one training sample (the model above treats a vertebrae as one training sample). This model is fed with 7x15 2D images at the same time, so that it has the ability to learn patient_overall labels. I call it type2 model ↓

\n

​
\n\"\"

\n

However, the disadvantage of this model is that it takes up too much GPU memory and therefore can only use small backbones (Imagine a model with batch_size = 1 that has to be trained on 105 images at one time, it is insane).

\n

Final Submission

\n

3D Seg

\n\n

2.5D Cls

\n\n

The submission time is 7.5 hours.

\n

Thanks to timm library for having so good implementation of those models. I always using it.

\n

Acknowledge

\n

In this competition, more than half of my models are trained on Z8G4 Workstation with dual A6000 GPU from Z by HP.
\nI would say I couldn't have achieved this without this workstation, thanks a lot!

", "messages": [ { "id": 2007128, "postDate": "2022-10-28T03:58:02.853Z", "content": "

Thanks to the organizers and congrats to all the winners and those who worked hard to develop new pipelines and stuck with it until the end of the competition.

\n

This is a very interesting competition, because we can think of many different ways to approach this dataset. Therefore the most important thing for this competition is to develop a reasonable pipeline, followed by optimization of the model.

\n

Code

\n

https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/362787

\n

Summary

\n

I designed a 2-stage pipeline to deal with this problem.

\n

stage1: 3D semantic segmentation -> stage2: 2.5D w/ LSTM classification.

\n

In addition, there are 2 different types of classification models in stage2.

\n

3D Semantic Segmentation

\n

For 3D semantic segmentation, we only have 87 samples w/ 3d mask in the dataset, but it's sufficient to train 3D semantic segmentation models with good performance.

\n

I use 128x128x128 input, to train resnet18d or efficientnet v2s + unet model, for segmenting C[1-7] vertebraes (7ch output).

\n

After the training was completed, I predicted 3d masks for each vertebrae for all 2k samples in the training set.

\n

Here is an example of predicted masks of C[1-7] vertebrae. Center slice of x, y, z dimension view, from left to right.
\n​
\n\"\"

\n

Prepare Data for Classification

\n

Next step is to prepare data for classification.

\n

First using the predicted 3D mask for each vertebrae, we can crop out 7 vertebraes from a single original 3d image (there might be multiple vertebraes shown in a single crop, but It's fine). At this moment, we cropped 2k * 7 = 14k samples and for each sample there is only one single binary label.

\n

Then for each vertebrae sample, I extracted 15 slices evenly by z-dimension, and for each slice, I further extracted +-2 adjacent slices to form an image with 5 channels. E.g if a 3D vertebrae sample have a shape of (128, 128, 30), I extracted 0th, 2nd, 4th, 6th….26th, 28th slices, then for example for the 2nd one, I use 0th~4th slices to form a 5-channel image.

\n

In addition, I added the predicted mask of corresponding vertebrae as the 6th channel to each image, as a way to exclude the effect of having multiple vertebraes in a single sample.

\n

Here is an example of one slice of a single vertebrae, and its predicted mask (with augmentations). We can see that the left half of the vertebrae in the image do not belong to the vertebrae specified by this crop.
\n​
\n\"\"

\n

2.5D + LSTM Classification

\n

We now have 14k 3D training samples of vertebrae. Theoretically the easiest way to deal with this data is to train 3D CNN on it. But unfortunately this method does not work. Training a 3D CNN on this data did not give me satisfactory results.

\n

So I backed off and chose the 2.5D approach. Here 2.5D means that each 2D slice in a vertebrae sample has the information of several adjacent slices, so it is written 2.5D. But the model is a normal 2D CNN with 5-channels input.

\n

The structure of this model is that, I first input 15 slices from a single sample into a 2D CNN, extracted out features of each slice, and then follow it with an LSTM model. So that the whole model can learn the features of the whole vertebrae. I call it type1 model ↓

\n

​
\n\"\"

\n

This model structure above, while being able to train a single vertebrae for fracture, does not able to train the patient as a whole for the presence of a fracture. So I designed another model.

\n

The second classification model is basically the same as the one above, except that it treats a patient as one training sample (the model above treats a vertebrae as one training sample). This model is fed with 7x15 2D images at the same time, so that it has the ability to learn patient_overall labels. I call it type2 model ↓

\n

​
\n\"\"

\n

However, the disadvantage of this model is that it takes up too much GPU memory and therefore can only use small backbones (Imagine a model with batch_size = 1 that has to be trained on 105 images at one time, it is insane).

\n

Final Submission

\n

3D Seg

\n\n

2.5D Cls

\n\n

The submission time is 7.5 hours.

\n

Thanks to timm library for having so good implementation of those models. I always using it.

\n

Acknowledge

\n

In this competition, more than half of my models are trained on Z8G4 Workstation with dual A6000 GPU from Z by HP.
\nI would say I couldn't have achieved this without this workstation, thanks a lot!

", "rawMarkdown": "Thanks to the organizers and congrats to all the winners and those who worked hard to develop new pipelines and stuck with it until the end of the competition. \n\nThis is a very interesting competition, because we can think of many different ways to approach this dataset. Therefore the most important thing for this competition is to develop a reasonable pipeline, followed by optimization of the model. \n\n# Code\n\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/362787\n\n# Summary\n\nI designed a 2-stage pipeline to deal with this problem.\n\nstage1: 3D semantic segmentation -> stage2: 2.5D w/ LSTM classification.\n\nIn addition, there are 2 different types of classification models in stage2.\n\n# 3D Semantic Segmentation\n\nFor 3D semantic segmentation, we only have 87 samples w/ 3d mask in the dataset, but it's sufficient to train 3D semantic segmentation models with good performance. \n\nI use 128x128x128 input, to train resnet18d or efficientnet v2s + unet model, for segmenting C[1-7] vertebraes (7ch output).\n\nAfter the training was completed, I predicted 3d masks for each vertebrae for all 2k samples in the training set.\n\nHere is an example of predicted masks of C[1-7] vertebrae. Center slice of x, y, z dimension view, from left to right.\n​\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2F6389a0864805b2f51c12dfe64c590cd5%2F1.png?generation=1666929046271861&alt=media)\n \n# Prepare Data for Classification\n\nNext step is to prepare data for classification.\n\nFirst using the predicted 3D mask for each vertebrae, we can crop out 7 vertebraes from a single original 3d image (there might be multiple vertebraes shown in a single crop, but It's fine). At this moment, we cropped 2k * 7 = 14k samples and for each sample there is only one single binary label.\n\nThen for each vertebrae sample, I extracted 15 slices evenly by z-dimension, and for each slice, I further extracted +-2 adjacent slices to form an image with 5 channels. E.g if a 3D vertebrae sample have a shape of (128, 128, 30), I extracted 0th, 2nd, 4th, 6th....26th, 28th slices, then for example for the 2nd one, I use 0th~4th slices to form a 5-channel image.\n \nIn addition, I added the predicted mask of corresponding vertebrae as the 6th channel to each image, as a way to exclude the effect of having multiple vertebraes in a single sample.\n\nHere is an example of one slice of a single vertebrae, and its predicted mask (with augmentations). We can see that the left half of the vertebrae in the image do not belong to the vertebrae specified by this crop.\n​\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2Fca4e40db1dc08930b7db99416f122bb0%2F2.png?generation=1666929152961988&alt=media)\n\n\n# 2.5D + LSTM Classification\n\nWe now have 14k 3D training samples of vertebrae. Theoretically the easiest way to deal with this data is to train 3D CNN on it. But unfortunately this method does not work. Training a 3D CNN on this data did not give me satisfactory results.\n\nSo I backed off and chose the 2.5D approach. Here 2.5D means that each 2D slice in a vertebrae sample has the information of several adjacent slices, so it is written 2.5D. But the model is a normal 2D CNN with 5-channels input.\n \nThe structure of this model is that, I first input 15 slices from a single sample into a 2D CNN, extracted out features of each slice, and then follow it with an LSTM model. So that the whole model can learn the features of the whole vertebrae. I call it type1 model ↓\n\n​\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2F15ff7ec369c300c4f8d8f5e3df64b071%2F3.png?generation=1666929243244770&alt=media)\n\n\nThis model structure above, while being able to train a single vertebrae for fracture, does not able to train the patient as a whole for the presence of a fracture. So I designed another model.\n\nThe second classification model is basically the same as the one above, except that it treats a patient as one training sample (the model above treats a vertebrae as one training sample). This model is fed with 7x15 2D images at the same time, so that it has the ability to learn patient_overall labels. I call it type2 model ↓\n\n​\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2Feecde695d666d4be6cb27791fa813ac0%2Fv2-0c70d96faae895d030d7b7f023570598_1440w.png?generation=1666929301847778&alt=media)\n\nHowever, the disadvantage of this model is that it takes up too much GPU memory and therefore can only use small backbones (Imagine a model with batch_size = 1 that has to be trained on 105 images at one time, it is insane). \n\n# Final Submission\n\n3D Seg\n\n* 5fold resnet 18d unet (128x128x128)\n* 5fold effv2 s (128x128x128)\n\n2.5D Cls\n* Type1 5fold effv2s (512x512)\n* Type1 5fold convnext tiny (384x384) \n\n* Type2 5fold convnext nano (512x512)\n* Type2 2fold convnext pico (512x512)\n* Type2 2fold convnext tiny (384x384)\n* Type2 2fold nfnet l0 (384x384)\n\nThe submission time is 7.5 hours.\n\nThanks to timm library for having so good implementation of those models. I always using it.\n\n# Acknowledge\n\nIn this competition, more than half of my models are trained on Z8G4 Workstation with dual A6000 GPU from Z by HP.\nI would say I couldn't have achieved this without this workstation, thanks a lot!\n", "votes": 222 }, { "id": 2062717, "postDate": "2022-12-12T10:21:16.837Z", "content": "

Hi, @haqishen , These are two architectures of 1st solution stage2 model that i plot:

\n

stage2 type1
\n\"\"

\n

stage2 type2
\n\"\"

", "rawMarkdown": "Hi, @haqishen , These are two architectures of 1st solution stage2 model that i plot:\n\nstage2 type1\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1214223%2F1fe26279c72b0db9bbded65124248693%2Fcsfd21.drawio.png?generation=1670840366506679&alt=media)\n\nstage2 type2\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1214223%2F686d682d11fdf2ddda1d064df7dae20f%2Fcsfd22.drawio.png?generation=1670840376652832&alt=media) ", "votes": 14, "replies": [ { "id": 2093940, "postDate": "2023-01-10T13:25:12.323Z", "content": "

Good work!

", "rawMarkdown": "Good work!", "votes": 1 } ] }, { "id": 2063746, "postDate": "2022-12-13T09:15:27.897Z", "content": "

Hi, @haqishen , This is the architecture of 1st solution stage1 model that i plot:
\n\"\"

", "rawMarkdown": "Hi, @haqishen , This is the architecture of 1st solution stage1 model that i plot:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1214223%2F6fba73aec5622d1f425d06a6478fdf62%2Fstage1.png?generation=1670922925813034&alt=media)", "votes": 8 }, { "id": 2012708, "postDate": "2022-11-01T11:04:42.380Z", "content": "

Congratulations and thanks for sharing your code, very well deserved !

\n\n

Again, thanks for your time and congratulations 👍

", "rawMarkdown": "Congratulations and thanks for sharing your code, very well deserved !\n\n- Did you try using a transformer rather than LSTM? You think LSTM in this kind of tasks still outperform transformers?\n- I was really surprised by your end2end training of these models, we tried that as well but succeeded only by first training the encoder... We will certainly have to learn how to train them end2end. How long does it take to train one of these models?\n- How important would you say mixup is in this kind of problem? Did you try without it?\n- Which performance had a single model? How was the improvement just by doing the 20 or so ensembles in classification?\n- The transformation in which you do a permutation of slices within the vertebrae and of vertebrates in the type2 model helped your training? Does this mean LSTM has no idea what is up and down (as transformers do with positional encoding)?\n\nAgain, thanks for your time and congratulations 👍", "votes": 5, "replies": [ { "id": 2014276, "postDate": "2022-11-02T13:30:11.823Z", "content": "

Thx.
\nGood questions!

\n
    \n
  1. I didn't try transformers here. The sequence is not too long…
    2. \n
  2. Final model takes 12h~24h and little experiments takes 6~12h.
    3. \n
  3. To avoid overfitting, mixup is one of the choses.
    4. \n
  4. Ensemble make CV score around ~0.02 better.
    5. \n
  5. Also kind of avoiding overfit, nothing special behind it.
    6. \n
", "rawMarkdown": "Thx.\nGood questions!\n1. I didn't try transformers here. The sequence is not too long...\n2. Final model takes 12h~24h and little experiments takes 6~12h.\n3. To avoid overfitting, mixup is one of the choses.\n4. Ensemble make CV score around ~0.02 better.\n5. Also kind of avoiding overfit, nothing special behind it.", "votes": 6 } ] }, { "id": 2007185, "postDate": "2022-10-28T04:43:49.267Z", "content": "

congratulations! I will replicate your solution and prepare for the next.
\nWhich idea boosted LB from 0.35 to ~0.2?

", "rawMarkdown": "congratulations! I will replicate your solution and prepare for the next.\nWhich idea boosted LB from 0.35 to ~0.2?", "votes": 3, "replies": [ { "id": 2007201, "postDate": "2022-10-28T05:03:49.380Z", "content": "

Thx. The 0.35 submission is just kind of POC of my ideas. Nothing change in pipeline afterward.

", "rawMarkdown": "Thx. The 0.35 submission is just kind of POC of my ideas. Nothing change in pipeline afterward." }, { "id": 2007213, "postDate": "2022-10-28T05:20:01.583Z", "content": "

I made a type1 model without crop by segmentation, but LB stopped at 0.41. Did the croppping improved your score significantly ?

", "rawMarkdown": "I made a type1 model without crop by segmentation, but LB stopped at 0.41. Did the croppping improved your score significantly ?" }, { "id": 2007229, "postDate": "2022-10-28T05:31:28.207Z", "content": "

Haven't tried no crop

", "rawMarkdown": "Haven't tried no crop", "votes": 1 }, { "id": 2007324, "postDate": "2022-10-28T06:49:17.790Z", "content": "

No crop was significantly lower scoring for me in cv (using AUC as a metric), but I am unsure that it matters too much because lstm(s) just might be able to cover some of the score up..

", "rawMarkdown": "No crop was significantly lower scoring for me in cv (using AUC as a metric), but I am unsure that it matters too much because lstm(s) just might be able to cover some of the score up..", "votes": 1 } ] }, { "id": 2007155, "postDate": "2022-10-28T04:15:35.640Z", "content": "

Thanks @haqishen for sharing and congrats to your solo Champion!
\nYour stage 2 models are brilliant! Could I ask how much gain it is for type 2 model compared with using 7 type 1 models and then calculate patient_overall?

\n

BTW, I also could not build a good 3d model.

", "rawMarkdown": "Thanks @haqishen for sharing and congrats to your solo Champion!\nYour stage 2 models are brilliant! Could I ask how much gain it is for type 2 model compared with using 7 type 1 models and then calculate `patient_overall`?\n\nBTW, I also could not build a good 3d model.", "votes": 3, "replies": [ { "id": 2007167, "postDate": "2022-10-28T04:24:52.653Z", "content": "

Type2 make the CV of patient_overall 0.02 better, so for whole CV 0.01 or so.

", "rawMarkdown": "Type2 make the CV of patient_overall 0.02 better, so for whole CV 0.01 or so.", "votes": 2 }, { "id": 2007299, "postDate": "2022-10-28T06:28:30.603Z", "content": "
\n

BTW, I also could not build a good 3d model.

\n
\n

same to me 😭

", "rawMarkdown": "> BTW, I also could not build a good 3d model.\n\nsame to me 😭" }, { "id": 2007405, "postDate": "2022-10-28T07:36:31.943Z", "content": "

BTW, we also started from 3D model using monai. But it didnt work.
\nThen we shift to 2 stage solution

", "rawMarkdown": "BTW, we also started from 3D model using monai. But it didnt work.\nThen we shift to 2 stage solution", "votes": 1 } ] }, { "id": 2007242, "postDate": "2022-10-28T05:44:38.323Z", "content": "

congratulations!

", "rawMarkdown": "congratulations! ", "votes": 1 }, { "id": 2007219, "postDate": "2022-10-28T05:24:43.813Z", "content": "

Amazing solution. congratulations !

", "rawMarkdown": "Amazing solution. congratulations !", "votes": 1 }, { "id": 2007203, "postDate": "2022-10-28T05:05:40.273Z", "content": "

Congratulation! very nice solution😋

", "rawMarkdown": "Congratulation! very nice solution😋", "votes": 1 }, { "id": 2007184, "postDate": "2022-10-28T04:43:37.123Z", "content": "

Wonderful approach @haqishen, your stage 2 model approach is the highlight of the overall post. Keep up the great work and hearty congratulations for the solo gold!

", "rawMarkdown": "Wonderful approach @haqishen, your stage 2 model approach is the highlight of the overall post. Keep up the great work and hearty congratulations for the solo gold! ", "votes": 1 }, { "id": 2007142, "postDate": "2022-10-28T04:09:18.003Z", "content": "

Congratulations on winning, Amazing Solution!

", "rawMarkdown": "Congratulations on winning, Amazing Solution!", "votes": 1, "replies": [ { "id": 2007159, "postDate": "2022-10-28T04:19:51.430Z", "content": "

Congrats to your solo gold!
\nI knew it when I first saw the LB, well deserved!

", "rawMarkdown": "Congrats to your solo gold!\nI knew it when I first saw the LB, well deserved!", "votes": 1 } ] }, { "id": 2009132, "postDate": "2022-10-29T17:18:46.833Z", "content": "

Congrats! Awesome solution.

\n

I told my teammate @yeeseng back in mid August that you would be #1. It's the only thing I predicted well for this competition. 😅

\n

For the type1 model, if those are only contributing to individual c level score, how are the patient scores calculated? And how much was gain do you estimate ensembling type1+type2 vs type2 alone?

", "rawMarkdown": "Congrats! Awesome solution.\n\nI told my teammate @yeeseng back in mid August that you would be #1. It's the only thing I predicted well for this competition. 😅\n\nFor the type1 model, if those are only contributing to individual c level score, how are the patient scores calculated? And how much was gain do you estimate ensembling type1+type2 vs type2 alone?", "votes": 2, "replies": [ { "id": 2009762, "postDate": "2022-10-30T09:42:31.560Z", "content": "

haha thx, I've just published my inference code with some small models and you can take a look at it.
\nI think the type1+type2 ensemble should have made the score 0.01~0.02 better.

", "rawMarkdown": "haha thx, I've just published my inference code with some small models and you can take a look at it.\nI think the type1+type2 ensemble should have made the score 0.01~0.02 better.", "votes": 2 }, { "id": 2011468, "postDate": "2022-10-31T15:30:57.780Z", "content": "

Great thank you!

", "rawMarkdown": "Great thank you!" } ] }, { "id": 2007132, "postDate": "2022-10-28T04:02:16.323Z", "content": "

I'm very interested if anyone has got good performence with 3D classification, if yes please leave a comment here!

", "rawMarkdown": "I'm very interested if anyone has got good performence with 3D classification, if yes please leave a comment here!", "votes": 2, "replies": [ { "id": 2007304, "postDate": "2022-10-28T06:30:02.443Z", "content": "

Would be very interested too. My 3D models were overfitting like crazy on the validation set.

", "rawMarkdown": "Would be very interested too. My 3D models were overfitting like crazy on the validation set.", "votes": 1 } ] }, { "id": 2056901, "postDate": "2022-12-06T14:50:41.003Z", "content": "

congratulations!

", "rawMarkdown": "congratulations!" }, { "id": 2020127, "postDate": "2022-11-07T07:44:49.070Z", "content": "

Congratulations @haqishen. Nice work👍

", "rawMarkdown": "Congratulations @haqishen. Nice work👍" }, { "id": 2018271, "postDate": "2022-11-05T14:53:16.980Z", "content": "

Big Congrats!

", "rawMarkdown": "Big Congrats! " }, { "id": 2018120, "postDate": "2022-11-05T12:24:11.213Z", "content": "

congratulations !!
\ncan you send to me the test train plz !
\nsalemtorkia5@gmail.com

", "rawMarkdown": "congratulations !! \ncan you send to me the test train plz ! \nsalemtorkia5@gmail.com\n" }, { "id": 2018030, "postDate": "2022-11-05T10:28:26.853Z", "content": "

Congratulations! I have recently changed my field and started spending more time on Kaggle. I will replicate the code to study for the upcoming competitions.

", "rawMarkdown": "Congratulations! I have recently changed my field and started spending more time on Kaggle. I will replicate the code to study for the upcoming competitions." }, { "id": 2016381, "postDate": "2022-11-04T00:37:47.740Z", "content": "

Congrats! Awesome solution.

", "rawMarkdown": "Congrats! Awesome solution." }, { "id": 2015998, "postDate": "2022-11-03T17:22:43.043Z", "content": "

Thank you for sharing this and congratulations !

", "rawMarkdown": "Thank you for sharing this and congratulations !" }, { "id": 2014783, "postDate": "2022-11-02T20:40:59.077Z", "content": "

Congratulations! And thanks for sharing the code and the architecture.

", "rawMarkdown": "Congratulations! And thanks for sharing the code and the architecture." }, { "id": 2014702, "postDate": "2022-11-02T19:05:20.370Z", "content": "

Congratulations on your achievement of first place.
\nI learnt alot and though not the same solution given my lack of resources I'm glad you achieved your results based on the direction of travel to my own solution (kfold, segmentation, Classification)
\nWell done!

", "rawMarkdown": "Congratulations on your achievement of first place. \nI learnt alot and though not the same solution given my lack of resources I'm glad you achieved your results based on the direction of travel to my own solution (kfold, segmentation, Classification)\nWell done!" }, { "id": 2014052, "postDate": "2022-11-02T09:25:46.587Z", "content": "

congratulations!

", "rawMarkdown": "congratulations!" }, { "id": 2013418, "postDate": "2022-11-01T22:01:25.103Z", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!" }, { "id": 2012622, "postDate": "2022-11-01T09:57:51.557Z", "content": "

Congratulations!!

", "rawMarkdown": "Congratulations!!" }, { "id": 2012058, "postDate": "2022-11-01T04:20:54.483Z", "content": "

Thanks for sharing. In the Final Submission you have listed two 3D seg models and 6 2.5D classifiers. How did you aggregate their results?

", "rawMarkdown": "Thanks for sharing. In the Final Submission you have listed two 3D seg models and 6 2.5D classifiers. How did you aggregate their results?", "replies": [ { "id": 2014278, "postDate": "2022-11-02T13:30:58.450Z", "content": "

check my inference code!

", "rawMarkdown": "check my inference code!", "votes": 1 } ] }, { "id": 2011302, "postDate": "2022-10-31T13:38:03.543Z", "content": "

Congratulations! Nice solution

", "rawMarkdown": "Congratulations! Nice solution" }, { "id": 2011151, "postDate": "2022-10-31T11:52:06.570Z", "content": "

Congrats! Awesome solution.

", "rawMarkdown": "Congrats! Awesome solution." }, { "id": 2010947, "postDate": "2022-10-31T09:17:46.867Z", "content": "

Congratulations! Thanks for diving into your approach!

", "rawMarkdown": "Congratulations! Thanks for diving into your approach!\n" }, { "id": 2010096, "postDate": "2022-10-30T13:58:15.083Z", "content": "

Great solution. congratulations!

", "rawMarkdown": "Great solution. congratulations!" }, { "id": 2009730, "postDate": "2022-10-30T08:41:13.727Z", "content": "

congratulations! Thanks for sharing this amazing solution.

", "rawMarkdown": "congratulations! Thanks for sharing this amazing solution. " }, { "id": 2009716, "postDate": "2022-10-30T08:26:54.757Z", "content": "

A good idea

", "rawMarkdown": "A good idea" }, { "id": 2009488, "postDate": "2022-10-30T04:11:49.093Z", "content": "

Congratulation!

", "rawMarkdown": "Congratulation!" }, { "id": 2009045, "postDate": "2022-10-29T15:33:37.103Z", "content": "

Amazing solution. congratulations !

", "rawMarkdown": "Amazing solution. congratulations !" }, { "id": 2009040, "postDate": "2022-10-29T15:22:51.583Z", "content": "

WOW congratulations!

", "rawMarkdown": "WOW congratulations!" }, { "id": 2008633, "postDate": "2022-10-29T08:13:57.673Z", "content": "

Congratulations & Thanks for the Solution

", "rawMarkdown": "Congratulations & Thanks for the Solution" }, { "id": 2008542, "postDate": "2022-10-29T07:05:51.583Z", "content": "

nice work! I'd like to study your solution!

", "rawMarkdown": "nice work! I'd like to study your solution!" }, { "id": 2008513, "postDate": "2022-10-29T06:27:22.977Z", "content": "

Congratulations & Thanks for the Solution

", "rawMarkdown": "Congratulations & Thanks for the Solution" }, { "id": 2008388, "postDate": "2022-10-29T02:28:11.437Z", "content": "

congratulations!

", "rawMarkdown": "congratulations!" }, { "id": 2008138, "postDate": "2022-10-28T19:17:05.583Z", "content": "

One quick question to the Champion:
\nSince you worked with slices in z-axis, 2-d segmentation with default axial slices would also fit to your pipeline, right?
\nIf this is true, based on your choice, can we say that 3-d segmentation yields better results in time/resources or accuracy?

\n

Congrats again!

", "rawMarkdown": "One quick question to the Champion:\nSince you worked with slices in z-axis, 2-d segmentation with default axial slices would also fit to your pipeline, right?\nIf this is true, based on your choice, can we say that 3-d segmentation yields better results in time/resources or accuracy?\n\nCongrats again!", "replies": [ { "id": 2008719, "postDate": "2022-10-29T09:26:17.193Z", "content": "

Thx. I haven't tried 2d seg but I don't think 2d seg would work here because z-axis information is too important.

", "rawMarkdown": "Thx. I haven't tried 2d seg but I don't think 2d seg would work here because z-axis information is too important.", "votes": 3 } ] }, { "id": 2007783, "postDate": "2022-10-28T13:39:48.993Z", "content": "

nice solution

", "rawMarkdown": "nice solution" }, { "id": 2007778, "postDate": "2022-10-28T13:36:22.727Z", "content": "

Congratulations! You are great

", "rawMarkdown": "Congratulations! You are great" }, { "id": 2007618, "postDate": "2022-10-28T11:12:54.760Z", "content": "

Congratulations!
\nThis might be a newbie question. I understand that the models were trained in a workstation. The code for training has to be made public or uploaded to a Kaggle kernel before the submission or just the model weights?

\n

Thanks

", "rawMarkdown": "Congratulations!\nThis might be a newbie question. I understand that the models were trained in a workstation. The code for training has to be made public or uploaded to a Kaggle kernel before the submission or just the model weights?\n\nThanks ", "replies": [ { "id": 2007766, "postDate": "2022-10-28T13:24:16.577Z", "content": "

Thx. Nothing should be made public before submission.
\nJust upload model weight and write inference code using kernels.

", "rawMarkdown": "Thx. Nothing should be made public before submission.\nJust upload model weight and write inference code using kernels.", "votes": 1 }, { "id": 2007878, "postDate": "2022-10-28T15:01:54.570Z", "content": "

Thanks for the answer

", "rawMarkdown": "Thanks for the answer " } ] }, { "id": 2007596, "postDate": "2022-10-28T10:57:52.930Z", "content": "

Congratulations! Great solution. Did you compare mask channel vs. no mask channel? In my experiments, it did not really help.

", "rawMarkdown": "Congratulations! Great solution. Did you compare mask channel vs. no mask channel? In my experiments, it did not really help. ", "replies": [ { "id": 2007661, "postDate": "2022-10-28T12:02:23.783Z", "content": "

Thx! Didn't try that.

", "rawMarkdown": "Thx! Didn't try that." } ] }, { "id": 2007580, "postDate": "2022-10-28T10:26:16.657Z", "content": "

Wow thank you for sharing your amazing idea! And Congratulations!!!

", "rawMarkdown": "Wow thank you for sharing your amazing idea! And Congratulations!!!" }, { "id": 2007571, "postDate": "2022-10-28T10:15:48.687Z", "content": "

Congratulations! Thank you for sharing the amazing solution!

", "rawMarkdown": "Congratulations! Thank you for sharing the amazing solution!" }, { "id": 2007495, "postDate": "2022-10-28T09:14:34.413Z", "content": "

Congratulations! I suspect this would have been hard to achieve on Kaggle's environment. At least I often ran out of ram/time.

", "rawMarkdown": "Congratulations! I suspect this would have been hard to achieve on Kaggle's environment. At least I often ran out of ram/time. " }, { "id": 2007442, "postDate": "2022-10-28T08:14:17.637Z", "content": "

Big congrats on your well deserved win.
\nAnd with such an efficiency!
\n(meaning: if there would be another leaderboard by: log(submissioncount/lbscore) loss, you had won that too! :-)

", "rawMarkdown": "Big congrats on your well deserved win.\nAnd with such an efficiency!\n(meaning: if there would be another leaderboard by: `log(submissioncount/lbscore)` loss, you had won that too! :-)", "replies": [ { "id": 2007450, "postDate": "2022-10-28T08:25:02.310Z", "content": "

Thx. Just chose to trust CV.

", "rawMarkdown": "Thx. Just chose to trust CV.", "votes": 2 } ] }, { "id": 2007390, "postDate": "2022-10-28T07:20:27.867Z", "content": "

Amazing solution!!! Congratulations on the solo win, well deserved

", "rawMarkdown": "Amazing solution!!! Congratulations on the solo win, well deserved" }, { "id": 2007313, "postDate": "2022-10-28T06:40:32.400Z", "content": "

Congratulations!
\nDo you train 2.5D models in end-to-end manner or in 2 steps (train 2d CNNs and extract features and then train LSTMs)?
\nI, as well as others I believe, would be really appreciated if you share us the code to learn more detail.

", "rawMarkdown": "Congratulations!\nDo you train 2.5D models in end-to-end manner or in 2 steps (train 2d CNNs and extract features and then train LSTMs)?\nI, as well as others I believe, would be really appreciated if you share us the code to learn more detail.", "replies": [ { "id": 2007439, "postDate": "2022-10-28T08:13:45.613Z", "content": "

Thx. It's end2end.

", "rawMarkdown": "Thx. It's end2end.", "votes": 1 }, { "id": 2007704, "postDate": "2022-10-28T12:42:14.100Z", "content": "

Congratulations @haqishen! Have you tried in 2 steps to see the difference? I tried both ways but unfortunately I don't have enough hardware to end2end and I couldn't make 2 steps work properly. I'd like to know if there is a significant difference in the score

", "rawMarkdown": "Congratulations @haqishen! Have you tried in 2 steps to see the difference? I tried both ways but unfortunately I don't have enough hardware to end2end and I couldn't make 2 steps work properly. I'd like to know if there is a significant difference in the score", "votes": 1 }, { "id": 2007769, "postDate": "2022-10-28T13:26:32.480Z", "content": "

Thx. I tried from very beginning and it's not as good as end2end then I sticked to end2end.

", "rawMarkdown": "Thx. I tried from very beginning and it's not as good as end2end then I sticked to end2end.", "votes": 1 } ] }, { "id": 2007241, "postDate": "2022-10-28T05:42:23.480Z", "content": "

Congratulations and thank you for sharing the idea. May I ask that how do you know the submission time?

", "rawMarkdown": "Congratulations and thank you for sharing the idea. May I ask that how do you know the submission time? ", "replies": [ { "id": 2007302, "postDate": "2022-10-28T06:29:07.100Z", "content": "

Thx. Haha just check sub page frequently…

", "rawMarkdown": "Thx. Haha just check sub page frequently...", "votes": 2 } ] }, { "id": 2007214, "postDate": "2022-10-28T05:21:38.550Z", "content": "

Thanks for sharing this innovative solution and congratulation for the 1st position. 👍💪 A question about the model 2: How LSTM knows to which Cx[1-7] should the fracture probability is assigned? It does not have a strong clue about the indexing (or maybe it does as the number of slices per vertebra is constant?!) so it may forget its decision history for the vertebras that it observes sooner than the others. I would guess that bidirectional LSTM may help but not entirely. Would u please comment on these? How adding indices you think it will help LSTM to do the job better (or it is not necessary to add indices)?

", "rawMarkdown": "Thanks for sharing this innovative solution and congratulation for the 1st position. 👍💪 A question about the model 2: How LSTM knows to which Cx[1-7] should the fracture probability is assigned? It does not have a strong clue about the indexing (or maybe it does as the number of slices per vertebra is constant?!) so it may forget its decision history for the vertebras that it observes sooner than the others. I would guess that bidirectional LSTM may help but not entirely. Would u please comment on these? How adding indices you think it will help LSTM to do the job better (or it is not necessary to add indices)? ", "replies": [ { "id": 2007231, "postDate": "2022-10-28T05:33:45.903Z", "content": "

Thx.
\nI'm sorry I don't understand your question. What do you mean by indices

", "rawMarkdown": "Thx.\nI'm sorry I don't understand your question. What do you mean by indices" }, { "id": 2007271, "postDate": "2022-10-28T06:09:15.363Z", "content": "

By indexing, I mean the position embedding. Like this one (32nd in this competition): https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/362593
\nLet me rephrase my question with an example: In type2 model, 105 feature vectors are fed to LSTM. Imagine that vectors 50 to 60 include the features corresponding to a fractured vertebra. (1) How does LSTM understand that C3 is the fractured vertebra? (2) Will vectors at higher indices (i.e., 60 to 105 corresponding to intact vertebra) will cause that LSTM modifies its final decision about C3?

", "rawMarkdown": "By indexing, I mean the position embedding. Like this one (32nd in this competition): https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/362593\nLet me rephrase my question with an example: In type2 model, 105 feature vectors are fed to LSTM. Imagine that vectors 50 to 60 include the features corresponding to a fractured vertebra. (1) How does LSTM understand that C3 is the fractured vertebra? (2) Will vectors at higher indices (i.e., 60 to 105 corresponding to intact vertebra) will cause that LSTM modifies its final decision about C3? " }, { "id": 2007447, "postDate": "2022-10-28T08:23:21.347Z", "content": "

I didn't use position embedding. For (1) the idea is quite simple, just use the same label for all 15 slices from a vertebrae. For (2) it's complicated, because for individual vertebrae the type1 models are better. There are probably many reasons for this.

", "rawMarkdown": "I didn't use position embedding. For (1) the idea is quite simple, just use the same label for all 15 slices from a vertebrae. For (2) it's complicated, because for individual vertebrae the type1 models are better. There are probably many reasons for this." }, { "id": 2007470, "postDate": "2022-10-28T08:44:38.600Z", "content": "

thanks a lot

", "rawMarkdown": "thanks a lot" } ] }, { "id": 2007202, "postDate": "2022-10-28T05:04:11.830Z", "content": "

Congratulations on winning and the beautiful solution! How much adding the mask as a 6th channel helped the results? did you also try to further crop the masked area?
\nAlso did you use any windowing on the image?

", "rawMarkdown": "Congratulations on winning and the beautiful solution! How much adding the mask as a 6th channel helped the results? did you also try to further crop the masked area?\nAlso did you use any windowing on the image?\n", "replies": [ { "id": 2007218, "postDate": "2022-10-28T05:24:43.310Z", "content": "

Thx.
\nHaven't tried.
\nYes but not working.
\nNo. Just simply norm to 0~1

", "rawMarkdown": "Thx.\nHaven't tried.\nYes but not working.\nNo. Just simply norm to 0~1", "votes": 1 } ] }, { "id": 2017594, "postDate": "2022-11-05T01:29:41.513Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 2018443, "postDate": "2022-11-05T17:36:29.150Z", "content": "

Thanks for your kind words ;)

", "rawMarkdown": "Thanks for your kind words ;)", "votes": 1 } ] }, { "id": 2015466, "postDate": "2022-11-03T09:47:45.990Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2012159, "postDate": "2022-11-01T05:47:44.183Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2872521, "postDate": "2024-06-14T21:11:41.697Z", "content": "

Wow, very exciting solution! Thanks!

", "rawMarkdown": "Wow, very exciting solution! Thanks!" }, { "id": 2016737, "postDate": "2022-11-04T07:49:57.023Z", "content": "

Hey, that's a perfect solution. Thanks!

", "rawMarkdown": "Hey, that's a perfect solution. Thanks!" }, { "id": 2007999, "postDate": "2022-10-28T16:28:29.393Z", "content": "

Congrats! And thanks for sharing!

", "rawMarkdown": "Congrats! And thanks for sharing! " } ], "comments": [ { "id": 2062717, "author_name": "README", "author_url": "", "post_date": "2022-12-12T10:21:16.837000", "content": "

Hi, @haqishen , These are two architectures of 1st solution stage2 model that i plot:

\n

stage2 type1
\n\"\"

\n

stage2 type2
\n\"\"

", "votes": 14, "replies": [ { "id": 2093940, "author_name": "Qishen Ha", "author_url": "", "post_date": "2023-01-10T13:25:12.323000", "content": "

Good work!

", "votes": 1, "replies": [] } ] }, { "id": 2063746, "author_name": "README", "author_url": "", "post_date": "2022-12-13T09:15:27.897000", "content": "

Hi, @haqishen , This is the architecture of 1st solution stage1 model that i plot:
\n\"\"

", "votes": 8, "replies": [] }, { "id": 2012708, "author_name": "Victor Salvia", "author_url": "", "post_date": "2022-11-01T11:04:42.380000", "content": "

Congratulations and thanks for sharing your code, very well deserved !

\n\n

Again, thanks for your time and congratulations 👍

", "votes": 5, "replies": [ { "id": 2014276, "author_name": "Qishen Ha", "author_url": "", "post_date": "2022-11-02T13:30:11.823000", "content": "

Thx.
\nGood questions!

\n
    \n
  1. I didn't try transformers here. The sequence is not too long…
    2. \n
  2. Final model takes 12h~24h and little experiments takes 6~12h.
    3. \n
  3. To avoid overfitting, mixup is one of the choses.
    4. \n
  4. Ensemble make CV score around ~0.02 better.
    5. \n
  5. Also kind of avoiding overfit, nothing special behind it.
    6. \n
", "votes": 6, "replies": [] } ] }, { "id": 2007185, "author_name": "patriot", "author_url": "", "post_date": "2022-10-28T04:43:49.267000", "content": "

congratulations! I will replicate your solution and prepare for the next.
\nWhich idea boosted LB from 0.35 to ~0.2?

", "votes": 3, "replies": [ { "id": 2007201, "author_name": "Qishen Ha", "author_url": "", "post_date": "2022-10-28T05:03:49.380000", "content": "

Thx. The 0.35 submission is just kind of POC of my ideas. Nothing change in pipeline afterward.

", "votes": 0, "replies": [] }, { "id": 2007213, "author_name": "patriot", "author_url": "", "post_date": "2022-10-28T05:20:01.583000", "content": "

I made a type1 model without crop by segmentation, but LB stopped at 0.41. Did the croppping improved your score significantly ?

", "votes": 0, "replies": [] }, { "id": 2007229, "author_name": "Qishen Ha", "author_url": "", "post_date": "2022-10-28T05:31:28.207000", "content": "

Haven't tried no crop

", "votes": 1, "replies": [] }, { "id": 2007324, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-10-28T06:49:17.790000", "content": "

No crop was significantly lower scoring for me in cv (using AUC as a metric), but I am unsure that it matters too much because lstm(s) just might be able to cover some of the score up..

", "votes": 1, "replies": [] } ] }, { "id": 2007155, "author_name": "Yiheng Wang", "author_url": "", "post_date": "2022-10-28T04:15:35.640000", "content": "

Thanks @haqishen for sharing and congrats to your solo Champion!
\nYour stage 2 models are brilliant! Could I ask how much gain it is for type 2 model compared with using 7 type 1 models and then calculate patient_overall?

\n

BTW, I also could not build a good 3d model.

", "votes": 3, "replies": [ { "id": 2007167, "author_name": "Qishen Ha", "author_url": "", "post_date": "2022-10-28T04:24:52.653000", "content": "

Type2 make the CV of patient_overall 0.02 better, so for whole CV 0.01 or so.

", "votes": 2, "replies": [] }, { "id": 2007299, "author_name": "yu4u", "author_url": "", "post_date": "2022-10-28T06:28:30.603000", "content": "
\n

BTW, I also could not build a good 3d model.

\n
\n

same to me 😭

", "votes": 0, "replies": [] }, { "id": 2007405, "author_name": "RihanPiggy", "author_url": "", "post_date": "2022-10-28T07:36:31.943000", "content": "

BTW, we also started from 3D model using monai. But it didnt work.
\nThen we shift to 2 stage solution

", "votes": 1, "replies": [] } ] }, { "id": 2007242, "author_name": "shigengtian", "author_url": "", "post_date": "2022-10-28T05:44:38.323000", "content": "

congratulations!

", "votes": 1, "replies": [] }, { "id": 2007219, "author_name": "olivepicker", "author_url": "", "post_date": "2022-10-28T05:24:43.813000", "content": "

Amazing solution. congratulations !

", "votes": 1, "replies": [] }, { "id": 2007203, "author_name": "README", "author_url": "", "post_date": "2022-10-28T05:05:40.273000", "content": "

Congratulation! very nice solution😋

", "votes": 1, "replies": [] }, { "id": 2007184, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-10-28T04:43:37.123000", "content": "

Wonderful approach @haqishen, your stage 2 model approach is the highlight of the overall post. Keep up the great work and hearty congratulations for the solo gold!

", "votes": 1, "replies": [] }, { "id": 2007142, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-10-28T04:09:18.003000", "content": "

Congratulations on winning, Amazing Solution!

", "votes": 1, "replies": [ { "id": 2007159, "author_name": "Qishen Ha", "author_url": "", "post_date": "2022-10-28T04:19:51.430000", "content": "

Congrats to your solo gold!
\nI knew it when I first saw the LB, well deserved!

", "votes": 1, "replies": [] } ] }, { "id": 2009132, "author_name": "Jesse", "author_url": "", "post_date": "2022-10-29T17:18:46.833000", "content": "

Congrats! Awesome solution.

\n

I told my teammate @yeeseng back in mid August that you would be #1. It's the only thing I predicted well for this competition. 😅

\n

For the type1 model, if those are only contributing to individual c level score, how are the patient scores calculated? And how much was gain do you estimate ensembling type1+type2 vs type2 alone?

", "votes": 2, "replies": [ { "id": 2009762, "author_name": "Qishen Ha", "author_url": "", "post_date": "2022-10-30T09:42:31.560000", "content": "

haha thx, I've just published my inference code with some small models and you can take a look at it.
\nI think the type1+type2 ensemble should have made the score 0.01~0.02 better.

", "votes": 2, "replies": [] }, { "id": 2011468, "author_name": "Jesse", "author_url": "", "post_date": "2022-10-31T15:30:57.780000", "content": "

Great thank you!

", "votes": 0, "replies": [] } ] }, { "id": 2007132, "author_name": "Qishen Ha", "author_url": "", "post_date": "2022-10-28T04:02:16.323000", "content": "

I'm very interested if anyone has got good performence with 3D classification, if yes please leave a comment here!

", "votes": 2, "replies": [ { "id": 2007304, "author_name": "Marius ", "author_url": "", "post_date": "2022-10-28T06:30:02.443000", "content": "

Would be very interested too. My 3D models were overfitting like crazy on the validation set.

", "votes": 1, "replies": [] } ] }, { "id": 2056901, "author_name": "liron", "author_url": "", "post_date": "2022-12-06T14:50:41.003000", "content": "

congratulations!

", "votes": 0, "replies": [] }, { "id": 2020127, "author_name": "Konika Rani", "author_url": "", "post_date": "2022-11-07T07:44:49.070000", "content": "

Congratulations @haqishen. Nice work👍

", "votes": 0, "replies": [] }, { "id": 2018271, "author_name": "LuckyLuke1", "author_url": "", "post_date": "2022-11-05T14:53:16.980000", "content": "

Big Congrats!

", "votes": 0, "replies": [] }, { "id": 2018120, "author_name": "Torkia Salem", "author_url": "", "post_date": "2022-11-05T12:24:11.213000", "content": "

congratulations !!
\ncan you send to me the test train plz !
\nsalemtorkia5@gmail.com

", "votes": 0, "replies": [] }, { "id": 2018030, "author_name": "Toyan Ünal", "author_url": "", "post_date": "2022-11-05T10:28:26.853000", "content": "

Congratulations! I have recently changed my field and started spending more time on Kaggle. I will replicate the code to study for the upcoming competitions.

", "votes": 0, "replies": [] }, { "id": 2016381, "author_name": "Shrijayan", "author_url": "", "post_date": "2022-11-04T00:37:47.740000", "content": "

Congrats! Awesome solution.

", "votes": 0, "replies": [] }, { "id": 2015998, "author_name": "Quadeer Shaikh", "author_url": "", "post_date": "2022-11-03T17:22:43.043000", "content": "

Thank you for sharing this and congratulations !

", "votes": 0, "replies": [] }, { "id": 2014783, "author_name": "Adrien Heinzlé", "author_url": "", "post_date": "2022-11-02T20:40:59.077000", "content": "

Congratulations! And thanks for sharing the code and the architecture.

", "votes": 0, "replies": [] }, { "id": 2014702, "author_name": "RogerTaylor", "author_url": "", "post_date": "2022-11-02T19:05:20.370000", "content": "

Congratulations on your achievement of first place.
\nI learnt alot and though not the same solution given my lack of resources I'm glad you achieved your results based on the direction of travel to my own solution (kfold, segmentation, Classification)
\nWell done!

", "votes": 0, "replies": [] }, { "id": 2014052, "author_name": "Dariusz Trzeciak", "author_url": "", "post_date": "2022-11-02T09:25:46.587000", "content": "

congratulations!

", "votes": 0, "replies": [] }, { "id": 2013418, "author_name": "Dogaska", "author_url": "", "post_date": "2022-11-01T22:01:25.103000", "content": "

Congratulations!

", "votes": 0, "replies": [] }, { "id": 2012622, "author_name": "dropstone34", "author_url": "", "post_date": "2022-11-01T09:57:51.557000", "content": "

Congratulations!!

", "votes": 0, "replies": [] }, { "id": 2012058, "author_name": "MarHenKa", "author_url": "", "post_date": "2022-11-01T04:20:54.483000", "content": "

Thanks for sharing. In the Final Submission you have listed two 3D seg models and 6 2.5D classifiers. How did you aggregate their results?

", "votes": 0, "replies": [ { "id": 2014278, "author_name": "Qishen Ha", "author_url": "", "post_date": "2022-11-02T13:30:58.450000", "content": "

check my inference code!

", "votes": 1, "replies": [] } ] }, { "id": 2011302, "author_name": "luizhtc", "author_url": "", "post_date": "2022-10-31T13:38:03.543000", "content": "

Congratulations! Nice solution

", "votes": 0, "replies": [] }, { "id": 2011151, "author_name": "Syed Mustafa Raza", "author_url": "", "post_date": "2022-10-31T11:52:06.570000", "content": "

Congrats! Awesome solution.

", "votes": 0, "replies": [] }, { "id": 2010947, "author_name": "aspiring", "author_url": "", "post_date": "2022-10-31T09:17:46.867000", "content": "

Congratulations! Thanks for diving into your approach!

", "votes": 0, "replies": [] }, { "id": 2010096, "author_name": "Ryan", "author_url": "", "post_date": "2022-10-30T13:58:15.083000", "content": "

Great solution. congratulations!

", "votes": 0, "replies": [] }, { "id": 2009730, "author_name": "", "author_url": "", "post_date": "2022-10-30T08:41:13.727000", "content": "", "votes": 0, "replies": [] }, { "id": 2009716, "author_name": "", "author_url": "", "post_date": "2022-10-30T08:26:54.757000", "content": "", "votes": 0, "replies": [] }, { "id": 2009488, "author_name": "", "author_url": "", "post_date": "2022-10-30T04:11:49.093000", "content": "", "votes": 0, "replies": [] }, { "id": 2009045, "author_name": "", "author_url": "", "post_date": "2022-10-29T15:33:37.103000", "content": "", "votes": 0, "replies": [] }, { "id": 2009040, "author_name": "", "author_url": "", "post_date": "2022-10-29T15:22:51.583000", "content": "", "votes": 0, "replies": [] }, { "id": 2008633, "author_name": "", "author_url": "", "post_date": "2022-10-29T08:13:57.673000", "content": "", "votes": 0, "replies": [] }, { "id": 2008542, "author_name": "", "author_url": "", "post_date": "2022-10-29T07:05:51.583000", "content": "", "votes": 0, "replies": [] }, { "id": 2008513, "author_name": "", "author_url": "", "post_date": "2022-10-29T06:27:22.977000", "content": "", "votes": 0, "replies": [] }, { "id": 2008388, "author_name": "", "author_url": "", "post_date": "2022-10-29T02:28:11.437000", "content": "", "votes": 0, "replies": [] }, { "id": 2008138, "author_name": "", "author_url": "", "post_date": "2022-10-28T19:17:05.583000", "content": "", "votes": 0, "replies": [ { "id": 2008719, "author_name": "", "author_url": "", "post_date": "2022-10-29T09:26:17.193000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 2007783, "author_name": "", "author_url": "", "post_date": "2022-10-28T13:39:48.993000", "content": "", "votes": 0, "replies": [] }, { "id": 2007778, "author_name": "", "author_url": "", "post_date": "2022-10-28T13:36:22.727000", "content": "", "votes": 0, "replies": [] }, { "id": 2007618, "author_name": "", "author_url": "", "post_date": "2022-10-28T11:12:54.760000", "content": "", "votes": 0, "replies": [ { "id": 2007766, "author_name": "", "author_url": "", "post_date": "2022-10-28T13:24:16.577000", "content": "", "votes": 1, "replies": [] }, { "id": 2007878, "author_name": "", "author_url": "", "post_date": "2022-10-28T15:01:54.570000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2007596, "author_name": "", "author_url": "", "post_date": "2022-10-28T10:57:52.930000", "content": "", "votes": 0, "replies": [ { "id": 2007661, "author_name": "", "author_url": "", "post_date": "2022-10-28T12:02:23.783000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2007580, "author_name": "", "author_url": "", "post_date": "2022-10-28T10:26:16.657000", "content": "", "votes": 0, "replies": [] }, { "id": 2007571, "author_name": "", "author_url": "", "post_date": "2022-10-28T10:15:48.687000", "content": "", "votes": 0, "replies": [] }, { "id": 2007495, "author_name": "", "author_url": "", "post_date": "2022-10-28T09:14:34.413000", "content": "", "votes": 0, "replies": [] }, { "id": 2007442, "author_name": "", "author_url": "", "post_date": "2022-10-28T08:14:17.637000", "content": "", "votes": 0, "replies": [ { "id": 2007450, "author_name": "", "author_url": "", "post_date": "2022-10-28T08:25:02.310000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 2007390, "author_name": "", "author_url": "", "post_date": "2022-10-28T07:20:27.867000", "content": "", "votes": 0, "replies": [] }, { "id": 2007313, "author_name": "", "author_url": "", "post_date": "2022-10-28T06:40:32.400000", "content": "", "votes": 0, "replies": [ { "id": 2007439, "author_name": "", "author_url": "", "post_date": "2022-10-28T08:13:45.613000", "content": "", "votes": 1, "replies": [] }, { "id": 2007704, "author_name": "", "author_url": "", "post_date": "2022-10-28T12:42:14.100000", "content": "", "votes": 1, "replies": [] }, { "id": 2007769, "author_name": "", "author_url": "", "post_date": "2022-10-28T13:26:32.480000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2007241, "author_name": "", "author_url": "", "post_date": "2022-10-28T05:42:23.480000", "content": "", "votes": 0, "replies": [ { "id": 2007302, "author_name": "", "author_url": "", "post_date": "2022-10-28T06:29:07.100000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 2007214, "author_name": "", "author_url": "", "post_date": "2022-10-28T05:21:38.550000", "content": "", "votes": 0, "replies": [ { "id": 2007231, "author_name": "", "author_url": "", "post_date": "2022-10-28T05:33:45.903000", "content": "", "votes": 0, "replies": [] }, { "id": 2007271, "author_name": "", "author_url": "", "post_date": "2022-10-28T06:09:15.363000", "content": "", "votes": 0, "replies": [] }, { "id": 2007447, "author_name": "", "author_url": "", "post_date": "2022-10-28T08:23:21.347000", "content": "", "votes": 0, "replies": [] }, { "id": 2007470, "author_name": "", "author_url": "", "post_date": "2022-10-28T08:44:38.600000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2007202, "author_name": "", "author_url": "", "post_date": "2022-10-28T05:04:11.830000", "content": "", "votes": 0, "replies": [ { "id": 2007218, "author_name": "", "author_url": "", "post_date": "2022-10-28T05:24:43.310000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2017594, "author_name": "", "author_url": "", "post_date": "2022-11-05T01:29:41.513000", "content": "", "votes": 1, "replies": [ { "id": 2018443, "author_name": "", "author_url": "", "post_date": "2022-11-05T17:36:29.150000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 2015466, "author_name": "", "author_url": "", "post_date": "2022-11-03T09:47:45.990000", "content": "", "votes": 0, "replies": [] }, { "id": 2012159, "author_name": "", "author_url": "", "post_date": "2022-11-01T05:47:44.183000", "content": "", "votes": 0, "replies": [] }, { "id": 2872521, "author_name": "", "author_url": "", "post_date": "2024-06-14T21:11:41.697000", "content": "", "votes": 0, "replies": [] }, { "id": 2016737, "author_name": "", "author_url": "", "post_date": "2022-11-04T07:49:57.023000", "content": "", "votes": 0, "replies": [] }, { "id": 2007999, "author_name": "", "author_url": "", "post_date": "2022-10-28T16:28:29.393000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2007128": "Thanks to the organizers and congrats to all the winners and those who worked hard to develop new pipelines and stuck with it until the end of the competition. \n\nThis is a very interesting competition, because we can think of many different ways to approach this dataset. Therefore the most important thing for this competition is to develop a reasonable pipeline, followed by optimization of the model. \n\n# Code\n\nhttps://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/362787\n\n# Summary\n\nI designed a 2-stage pipeline to deal with this problem.\n\nstage1: 3D semantic segmentation -> stage2: 2.5D w/ LSTM classification.\n\nIn addition, there are 2 different types of classification models in stage2.\n\n# 3D Semantic Segmentation\n\nFor 3D semantic segmentation, we only have 87 samples w/ 3d mask in the dataset, but it's sufficient to train 3D semantic segmentation models with good performance. \n\nI use 128x128x128 input, to train resnet18d or efficientnet v2s + unet model, for segmenting C[1-7] vertebraes (7ch output).\n\nAfter the training was completed, I predicted 3d masks for each vertebrae for all 2k samples in the training set.\n\nHere is an example of predicted masks of C[1-7] vertebrae. Center slice of x, y, z dimension view, from left to right.\n​\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2F6389a0864805b2f51c12dfe64c590cd5%2F1.png?generation=1666929046271861&alt=media)\n \n# Prepare Data for Classification\n\nNext step is to prepare data for classification.\n\nFirst using the predicted 3D mask for each vertebrae, we can crop out 7 vertebraes from a single original 3d image (there might be multiple vertebraes shown in a single crop, but It's fine). At this moment, we cropped 2k * 7 = 14k samples and for each sample there is only one single binary label.\n\nThen for each vertebrae sample, I extracted 15 slices evenly by z-dimension, and for each slice, I further extracted +-2 adjacent slices to form an image with 5 channels. E.g if a 3D vertebrae sample have a shape of (128, 128, 30), I extracted 0th, 2nd, 4th, 6th....26th, 28th slices, then for example for the 2nd one, I use 0th~4th slices to form a 5-channel image.\n \nIn addition, I added the predicted mask of corresponding vertebrae as the 6th channel to each image, as a way to exclude the effect of having multiple vertebraes in a single sample.\n\nHere is an example of one slice of a single vertebrae, and its predicted mask (with augmentations). We can see that the left half of the vertebrae in the image do not belong to the vertebrae specified by this crop.\n​\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2Fca4e40db1dc08930b7db99416f122bb0%2F2.png?generation=1666929152961988&alt=media)\n\n\n# 2.5D + LSTM Classification\n\nWe now have 14k 3D training samples of vertebrae. Theoretically the easiest way to deal with this data is to train 3D CNN on it. But unfortunately this method does not work. Training a 3D CNN on this data did not give me satisfactory results.\n\nSo I backed off and chose the 2.5D approach. Here 2.5D means that each 2D slice in a vertebrae sample has the information of several adjacent slices, so it is written 2.5D. But the model is a normal 2D CNN with 5-channels input.\n \nThe structure of this model is that, I first input 15 slices from a single sample into a 2D CNN, extracted out features of each slice, and then follow it with an LSTM model. So that the whole model can learn the features of the whole vertebrae. I call it type1 model ↓\n\n​\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2F15ff7ec369c300c4f8d8f5e3df64b071%2F3.png?generation=1666929243244770&alt=media)\n\n\nThis model structure above, while being able to train a single vertebrae for fracture, does not able to train the patient as a whole for the presence of a fracture. So I designed another model.\n\nThe second classification model is basically the same as the one above, except that it treats a patient as one training sample (the model above treats a vertebrae as one training sample). This model is fed with 7x15 2D images at the same time, so that it has the ability to learn patient_overall labels. I call it type2 model ↓\n\n​\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2Feecde695d666d4be6cb27791fa813ac0%2Fv2-0c70d96faae895d030d7b7f023570598_1440w.png?generation=1666929301847778&alt=media)\n\nHowever, the disadvantage of this model is that it takes up too much GPU memory and therefore can only use small backbones (Imagine a model with batch_size = 1 that has to be trained on 105 images at one time, it is insane). \n\n# Final Submission\n\n3D Seg\n\n* 5fold resnet 18d unet (128x128x128)\n* 5fold effv2 s (128x128x128)\n\n2.5D Cls\n* Type1 5fold effv2s (512x512)\n* Type1 5fold convnext tiny (384x384) \n\n* Type2 5fold convnext nano (512x512)\n* Type2 2fold convnext pico (512x512)\n* Type2 2fold convnext tiny (384x384)\n* Type2 2fold nfnet l0 (384x384)\n\nThe submission time is 7.5 hours.\n\nThanks to timm library for having so good implementation of those models. I always using it.\n\n# Acknowledge\n\nIn this competition, more than half of my models are trained on Z8G4 Workstation with dual A6000 GPU from Z by HP.\nI would say I couldn't have achieved this without this workstation, thanks a lot!\n", "2062717": "Hi, @haqishen , These are two architectures of 1st solution stage2 model that i plot:\n\nstage2 type1\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1214223%2F1fe26279c72b0db9bbded65124248693%2Fcsfd21.drawio.png?generation=1670840366506679&alt=media)\n\nstage2 type2\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1214223%2F686d682d11fdf2ddda1d064df7dae20f%2Fcsfd22.drawio.png?generation=1670840376652832&alt=media) ", "2063746": "Hi, @haqishen , This is the architecture of 1st solution stage1 model that i plot:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1214223%2F6fba73aec5622d1f425d06a6478fdf62%2Fstage1.png?generation=1670922925813034&alt=media)", "2012708": "Congratulations and thanks for sharing your code, very well deserved !\n\n- Did you try using a transformer rather than LSTM? You think LSTM in this kind of tasks still outperform transformers?\n- I was really surprised by your end2end training of these models, we tried that as well but succeeded only by first training the encoder... We will certainly have to learn how to train them end2end. How long does it take to train one of these models?\n- How important would you say mixup is in this kind of problem? Did you try without it?\n- Which performance had a single model? How was the improvement just by doing the 20 or so ensembles in classification?\n- The transformation in which you do a permutation of slices within the vertebrae and of vertebrates in the type2 model helped your training? Does this mean LSTM has no idea what is up and down (as transformers do with positional encoding)?\n\nAgain, thanks for your time and congratulations 👍", "2007185": "congratulations! I will replicate your solution and prepare for the next.\nWhich idea boosted LB from 0.35 to ~0.2?", "2007155": "Thanks @haqishen for sharing and congrats to your solo Champion!\nYour stage 2 models are brilliant! Could I ask how much gain it is for type 2 model compared with using 7 type 1 models and then calculate `patient_overall`?\n\nBTW, I also could not build a good 3d model.", "2007242": "congratulations! ", "2007219": "Amazing solution. congratulations !", "2007203": "Congratulation! very nice solution😋", "2007184": "Wonderful approach @haqishen, your stage 2 model approach is the highlight of the overall post. Keep up the great work and hearty congratulations for the solo gold! ", "2007142": "Congratulations on winning, Amazing Solution!", "2009132": "Congrats! Awesome solution.\n\nI told my teammate @yeeseng back in mid August that you would be #1. It's the only thing I predicted well for this competition. 😅\n\nFor the type1 model, if those are only contributing to individual c level score, how are the patient scores calculated? And how much was gain do you estimate ensembling type1+type2 vs type2 alone?", "2007132": "I'm very interested if anyone has got good performence with 3D classification, if yes please leave a comment here!", "2056901": "congratulations!", "2020127": "Congratulations @haqishen. Nice work👍", "2018271": "Big Congrats! ", "2018120": "congratulations !! \ncan you send to me the test train plz ! \nsalemtorkia5@gmail.com\n", "2018030": "Congratulations! I have recently changed my field and started spending more time on Kaggle. I will replicate the code to study for the upcoming competitions.", "2016381": "Congrats! Awesome solution.", "2015998": "Thank you for sharing this and congratulations !", "2014783": "Congratulations! And thanks for sharing the code and the architecture.", "2014702": "Congratulations on your achievement of first place. \nI learnt alot and though not the same solution given my lack of resources I'm glad you achieved your results based on the direction of travel to my own solution (kfold, segmentation, Classification)\nWell done!", "2014052": "congratulations!", "2013418": "Congratulations!", "2012622": "Congratulations!!", "2012058": "Thanks for sharing. In the Final Submission you have listed two 3D seg models and 6 2.5D classifiers. How did you aggregate their results?", "2011302": "Congratulations! Nice solution", "2011151": "Congrats! Awesome solution.", "2010947": "Congratulations! Thanks for diving into your approach!\n", "2010096": "Great solution. congratulations!", "2009730": "congratulations! Thanks for sharing this amazing solution. ", "2009716": "A good idea", "2009488": "Congratulation!", "2009045": "Amazing solution. congratulations !", "2009040": "WOW congratulations!", "2008633": "Congratulations & Thanks for the Solution", "2008542": "nice work! I'd like to study your solution!", "2008513": "Congratulations & Thanks for the Solution", "2008388": "congratulations!", "2008138": "One quick question to the Champion:\nSince you worked with slices in z-axis, 2-d segmentation with default axial slices would also fit to your pipeline, right?\nIf this is true, based on your choice, can we say that 3-d segmentation yields better results in time/resources or accuracy?\n\nCongrats again!", "2007783": "nice solution", "2007778": "Congratulations! You are great", "2007618": "Congratulations!\nThis might be a newbie question. I understand that the models were trained in a workstation. The code for training has to be made public or uploaded to a Kaggle kernel before the submission or just the model weights?\n\nThanks ", "2007596": "Congratulations! Great solution. Did you compare mask channel vs. no mask channel? In my experiments, it did not really help. ", "2007580": "Wow thank you for sharing your amazing idea! And Congratulations!!!", "2007571": "Congratulations! Thank you for sharing the amazing solution!", "2007495": "Congratulations! I suspect this would have been hard to achieve on Kaggle's environment. At least I often ran out of ram/time. ", "2007442": "Big congrats on your well deserved win.\nAnd with such an efficiency!\n(meaning: if there would be another leaderboard by: `log(submissioncount/lbscore)` loss, you had won that too! :-)", "2007390": "Amazing solution!!! Congratulations on the solo win, well deserved", "2007313": "Congratulations!\nDo you train 2.5D models in end-to-end manner or in 2 steps (train 2d CNNs and extract features and then train LSTMs)?\nI, as well as others I believe, would be really appreciated if you share us the code to learn more detail.", "2007241": "Congratulations and thank you for sharing the idea. May I ask that how do you know the submission time? ", "2007214": "Thanks for sharing this innovative solution and congratulation for the 1st position. 👍💪 A question about the model 2: How LSTM knows to which Cx[1-7] should the fracture probability is assigned? It does not have a strong clue about the indexing (or maybe it does as the number of slices per vertebra is constant?!) so it may forget its decision history for the vertebras that it observes sooner than the others. I would guess that bidirectional LSTM may help but not entirely. Would u please comment on these? How adding indices you think it will help LSTM to do the job better (or it is not necessary to add indices)? ", "2007202": "Congratulations on winning and the beautiful solution! How much adding the mask as a 6th channel helped the results? did you also try to further crop the masked area?\nAlso did you use any windowing on the image?\n", "2017594": "", "2015466": "", "2012159": "", "2872521": "Wow, very exciting solution! Thanks!", "2016737": "Hey, that's a perfect solution. Thanks!", "2007999": "Congrats! And thanks for sharing! " } }