![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F281652%2Ff6a15edbc81d867c0faecd0894d6aa27%2Fpe.png?generation=1603756913344030&alt=media\")
Update: code available at https://github.com/i-pan/kaggle-rsna-pe
\nCongratulations to all the participants and the winners. Special congrats to prize winners @osciiart and @jpbremer who are on track to be physician GMs. This was a tough, compute-heavy challenge given the large amount of data and short timeframe. My setup was 4 24 GB Quadro RTX 6000 GPUs. I always feel guilty during competitions like these since I have the luxury of strong compute. Models were trained using DDP in PyTorch 1.6 with automatic mixed precision.
\nEven though the results aren't final yet and my submission may be removed for violating the label consistency requirements (hopefully my heuristic for fixing those worked!), I still wanted to share my solution.
\n
Here is a schematic outlining my solution. It has a lot of moving parts, so I apologize for the lengthy summary.
\nLast year's RSNA Intracranial Hemorrhage Detection challenge shared a lot of similarities with this year's challenge. Most of the top solutions combined 2D CNN feature extraction with sequence modeling. The backbone of my solution also relied on a similar setup.
\nI first trained ResNeSt50 on 2D images. Images were windowed using the PE-specific window (WL=100, WW=700) that I mentioned in one of my initial posts on DICOM processing. Each \"image\" was 3 channels, with each channel representing an individual slice. Thus the 2D image was a stack of 3 continuous slices. The targets were the 7 PE-related labels (i.e., excluding the RV/LV ratio labels). Note: most of the PE-related labels were exam-level labels. However, I just assigned the exam labels to each slice positive for PE (negative slices had all zeros), with the understanding that there would be label noise. Also, I predicted the labels for the middle slice among the 3 slices in the image. Models were trained using vanilla binary cross-entropy loss (BCEWithLogitsLoss in PyTorch), 512x512 with 448x448 random crops (single center crop during inference), RandAugment data augmentation, batch size 128, 5 epochs, 2500 steps per epoch, RAdam optimizer with cosine annealing learning rate scheduler. I used generalized mean pooling and reduced the final feature vector to 512-D. Mean loss was around 0.06 (AUC 0.95-0.96 for slice-wise prediction of PE vs. no PE). Features were then extracted for all slices.
Many of last year's solutions used LSTMs/GRUs as the sequence model of choice. For this competition, I used huggingface transformers (specifically, the Transformer class from transformers.modeling_distilbert). I used one 4-layer transformer to produce slice-wise pe_present_on_image predictions and another transformer to predict exam-wise PE-labels. Sequence length was 512 during training, padded/truncated as necessary. During inference, I used the sequence without modifications.
Important point: Images from non-PE exams do not contribute to the loss. At first, I was training the slice-wise transformer on all exams. Then, I decided to train these models on positive exams only. This lowered my CV by about 0.01-0.02. I used a custom weighted loss where I weighted the loss from each example by the proportion of positive PE slices (as described in the metric), though I'm still not sure I wrote it correctly.
\nThe exam-level transformer was trained using a weighted BCE loss based on the competition label weights. Exam-level validation losses ranged from 0.15-0.17.
\nTo add some variety into my modeling, I then trained a time-distributed CNN, which is just another way of saying I stacked a transformer on top of a CNN feature extraction backbone and trained end-to-end.
\nBut before doing that, I performed inference using the slice-wise transformer model to get PE scores for every slice (5-fold OOF predictions). Then, when training the TD-CNN, I only trained on the top 30% of slices from each exam, sorted by PE score. These were trained on 3D volumes of size 32x416x416 cropped to 32x364x364 using the same windowing strategy (WL=100, WW=700) in batches of 16.
\nI initialized the CNN backbone and the transformer head with trained models from steps 1 and 2 to help with convergence. I forced all the batch normalization layers in the backbone to eval mode as well- this prevents the running mean and variance in each layer from updating and only trains the coefficients. Exam-level validation losses ranged from 0.15-0.17, similar to step 2.
RV/LV ratio is a significant portion of the loss. I hand-labeled slices with heart in 1,000 CT scans and trained a model (EfficientNet-B1 pruned, AUC 0.998, 256x256->224x224 crops) to classify heart slices in each CT scan. I did this because I felt that by focusing a model on the heart, I could get better, more consistent results across scans. It actually wasn't hard to label 1,000 scans- probably a full day's worth of work. I just needed to find the top and bottom heart slices; everything in between thus must also contain the heart.
\nI trained a 3D CNN to classify RV/LV ratio. Specifically, I used a 101-layer channel separated network, pretrained on 65 million Instagram videos (https://arxiv.org/abs/1904.02811, https://github.com/facebookresearch/VMZ).
\nThis model was trained only on heart slices from each exam. Also, it was only trained on positive exams. This is because RV/LV ratio was not labeled for negative exams- both RV/LV labels are 0. Thus, it didn't make sense to me to try and train my model on the entire dataset's labels directly. Models were trained using a weighted BCE loss using the competition weights. I resized the input to 64x256x256->64x224x224 crops and used mediastinal window (WL=50, WW=350). AUC for RV/LV ratio > 1 was about 0.85. The validation loss ranged from 0.44-0.48. I then used these models to extract 2048-D features from each exam.
\nThe challenge here was that the likelihood of an exam being PE-positive greatly influenced the RV/LV labels. From step 2, I calculated 5-fold OOF exam-level predictions for all exams. Then, I trained a linear model that took as input the concatenation of the 2048-D 3D CNN feature and the 7 PE exam labels. This linear model was trained across all exams using the labels directly. This way, the model could take into account the imaging features from the scan but also adjust the predictions based on the likelihood of PE. Validation losses ranged from 0.22-0.25.
\nI didn't validate my entire pipeline that often due to time constraints, instead choosing to focus on the individual component losses and optimizing each one as well as I could. I did periodically do a sanity-check on 200 single-fold exams to make sure that my entire pipeline was working, and my final validation loss was 0.183 for my 0.150 public LB submission. I'm still not confident that I implemented the metric correctly, but I was seeing good correlation (0.159/0.195->0.156/0.188-> 0.150/0.183).
\nAt the end, I applied a function to enforce label consistency requirements for each exam. There was about 0.001 change in CV after applying this function. I have some thoughts about this requirement which maybe I'll save for another post.
\nOverall, I used these models:
\n-2x ResNeSt50 feature extractors
\n-6x exam transformers (3 for each extractor)
\n-6x slice transformers
\n-5x ResNeSt50 TD-CNN
\n-1x EfficientNet-B1 pruned heart slice classifier
\n-5x ip-CSN-101 3D CNN RV/LV feature extractor
\n-5x RV/LV linear model
Inference took about 7-8 hours for the entire test set.
\nCode to follow once I clean it up.
\nThings that didn't work as well:
\n-3D CNN for PE exam-level prediction
\n-Pseudolabeling RV/LV ratio for negative exams
\n-Using logits/probabilities instead of features for second-stage model
\n-TD-CNN for RV/LV ratio prediction
\n-Training feature extractor on positive exams only to improve slice-wise PE prediction
Update: code available at https://github.com/i-pan/kaggle-rsna-pe
\nCongratulations to all the participants and the winners. Special congrats to prize winners @osciiart and @jpbremer who are on track to be physician GMs. This was a tough, compute-heavy challenge given the large amount of data and short timeframe. My setup was 4 24 GB Quadro RTX 6000 GPUs. I always feel guilty during competitions like these since I have the luxury of strong compute. Models were trained using DDP in PyTorch 1.6 with automatic mixed precision.
\nEven though the results aren't final yet and my submission may be removed for violating the label consistency requirements (hopefully my heuristic for fixing those worked!), I still wanted to share my solution.
\n
Here is a schematic outlining my solution. It has a lot of moving parts, so I apologize for the lengthy summary.
\nLast year's RSNA Intracranial Hemorrhage Detection challenge shared a lot of similarities with this year's challenge. Most of the top solutions combined 2D CNN feature extraction with sequence modeling. The backbone of my solution also relied on a similar setup.
\nI first trained ResNeSt50 on 2D images. Images were windowed using the PE-specific window (WL=100, WW=700) that I mentioned in one of my initial posts on DICOM processing. Each \"image\" was 3 channels, with each channel representing an individual slice. Thus the 2D image was a stack of 3 continuous slices. The targets were the 7 PE-related labels (i.e., excluding the RV/LV ratio labels). Note: most of the PE-related labels were exam-level labels. However, I just assigned the exam labels to each slice positive for PE (negative slices had all zeros), with the understanding that there would be label noise. Also, I predicted the labels for the middle slice among the 3 slices in the image. Models were trained using vanilla binary cross-entropy loss (BCEWithLogitsLoss in PyTorch), 512x512 with 448x448 random crops (single center crop during inference), RandAugment data augmentation, batch size 128, 5 epochs, 2500 steps per epoch, RAdam optimizer with cosine annealing learning rate scheduler. I used generalized mean pooling and reduced the final feature vector to 512-D. Mean loss was around 0.06 (AUC 0.95-0.96 for slice-wise prediction of PE vs. no PE). Features were then extracted for all slices.
Many of last year's solutions used LSTMs/GRUs as the sequence model of choice. For this competition, I used huggingface transformers (specifically, the Transformer class from transformers.modeling_distilbert). I used one 4-layer transformer to produce slice-wise pe_present_on_image predictions and another transformer to predict exam-wise PE-labels. Sequence length was 512 during training, padded/truncated as necessary. During inference, I used the sequence without modifications.
Important point: Images from non-PE exams do not contribute to the loss. At first, I was training the slice-wise transformer on all exams. Then, I decided to train these models on positive exams only. This lowered my CV by about 0.01-0.02. I used a custom weighted loss where I weighted the loss from each example by the proportion of positive PE slices (as described in the metric), though I'm still not sure I wrote it correctly.
\nThe exam-level transformer was trained using a weighted BCE loss based on the competition label weights. Exam-level validation losses ranged from 0.15-0.17.
\nTo add some variety into my modeling, I then trained a time-distributed CNN, which is just another way of saying I stacked a transformer on top of a CNN feature extraction backbone and trained end-to-end.
\nBut before doing that, I performed inference using the slice-wise transformer model to get PE scores for every slice (5-fold OOF predictions). Then, when training the TD-CNN, I only trained on the top 30% of slices from each exam, sorted by PE score. These were trained on 3D volumes of size 32x416x416 cropped to 32x364x364 using the same windowing strategy (WL=100, WW=700) in batches of 16.
\nI initialized the CNN backbone and the transformer head with trained models from steps 1 and 2 to help with convergence. I forced all the batch normalization layers in the backbone to eval mode as well- this prevents the running mean and variance in each layer from updating and only trains the coefficients. Exam-level validation losses ranged from 0.15-0.17, similar to step 2.
RV/LV ratio is a significant portion of the loss. I hand-labeled slices with heart in 1,000 CT scans and trained a model (EfficientNet-B1 pruned, AUC 0.998, 256x256->224x224 crops) to classify heart slices in each CT scan. I did this because I felt that by focusing a model on the heart, I could get better, more consistent results across scans. It actually wasn't hard to label 1,000 scans- probably a full day's worth of work. I just needed to find the top and bottom heart slices; everything in between thus must also contain the heart.
\nI trained a 3D CNN to classify RV/LV ratio. Specifically, I used a 101-layer channel separated network, pretrained on 65 million Instagram videos (https://arxiv.org/abs/1904.02811, https://github.com/facebookresearch/VMZ).
\nThis model was trained only on heart slices from each exam. Also, it was only trained on positive exams. This is because RV/LV ratio was not labeled for negative exams- both RV/LV labels are 0. Thus, it didn't make sense to me to try and train my model on the entire dataset's labels directly. Models were trained using a weighted BCE loss using the competition weights. I resized the input to 64x256x256->64x224x224 crops and used mediastinal window (WL=50, WW=350). AUC for RV/LV ratio > 1 was about 0.85. The validation loss ranged from 0.44-0.48. I then used these models to extract 2048-D features from each exam.
\nThe challenge here was that the likelihood of an exam being PE-positive greatly influenced the RV/LV labels. From step 2, I calculated 5-fold OOF exam-level predictions for all exams. Then, I trained a linear model that took as input the concatenation of the 2048-D 3D CNN feature and the 7 PE exam labels. This linear model was trained across all exams using the labels directly. This way, the model could take into account the imaging features from the scan but also adjust the predictions based on the likelihood of PE. Validation losses ranged from 0.22-0.25.
\nI didn't validate my entire pipeline that often due to time constraints, instead choosing to focus on the individual component losses and optimizing each one as well as I could. I did periodically do a sanity-check on 200 single-fold exams to make sure that my entire pipeline was working, and my final validation loss was 0.183 for my 0.150 public LB submission. I'm still not confident that I implemented the metric correctly, but I was seeing good correlation (0.159/0.195->0.156/0.188-> 0.150/0.183).
\nAt the end, I applied a function to enforce label consistency requirements for each exam. There was about 0.001 change in CV after applying this function. I have some thoughts about this requirement which maybe I'll save for another post.
\nOverall, I used these models:
\n-2x ResNeSt50 feature extractors
\n-6x exam transformers (3 for each extractor)
\n-6x slice transformers
\n-5x ResNeSt50 TD-CNN
\n-1x EfficientNet-B1 pruned heart slice classifier
\n-5x ip-CSN-101 3D CNN RV/LV feature extractor
\n-5x RV/LV linear model
Inference took about 7-8 hours for the entire test set.
\nCode to follow once I clean it up.
\nThings that didn't work as well:
\n-3D CNN for PE exam-level prediction
\n-Pseudolabeling RV/LV ratio for negative exams
\n-Using logits/probabilities instead of features for second-stage model
\n-TD-CNN for RV/LV ratio prediction
\n-Training feature extractor on positive exams only to improve slice-wise PE prediction
Congratulations! Great solution write up. Thank you. So many ingenuities squeezed into one diagram.
\nI have just one question. How did you find time to do all that as an intern in a hospital as busy as yours?
", "rawMarkdown": "Congratulations! Great solution write up. Thank you. So many ingenuities squeezed into one diagram.\n\nI have just one question. How did you find time to do all that as an intern in a hospital as busy as yours?", "votes": 5, "replies": [ { "id": 1061927, "postDate": "2020-10-27T12:45:34.570Z", "content": "Congrats Ian and Yee ! Both very impressive results. Go Radiology go !
\nAbout your question Yee, I stopped thinking years ago that Ian is a human like us.
", "rawMarkdown": "Congrats Ian and Yee ! Both very impressive results. Go Radiology go !\n \nAbout your question Yee, I stopped thinking years ago that Ian is a human like us.\n", "votes": 2 }, { "id": 1061940, "postDate": "2020-10-27T13:00:57.603Z", "content": "\n\nI stopped thinking years ago that Ian is a human like us.
\n
haha. Agree.
", "rawMarkdown": "> I stopped thinking years ago that Ian is a human like us.\n\nhaha. Agree.", "votes": 1 }, { "id": 1061973, "postDate": "2020-10-27T13:26:36.360Z", "content": "Thanks, congrats on your strong finish as well! After the initial setup, it's just a matter of running experiments, most of which is just waiting for models to finish training. At the beginning of the challenge, I laid out my plan for what I was going to do and just focused on executing it. I didn't deviate from that initial plan very much. Fortunately it worked out.
", "rawMarkdown": "Thanks, congrats on your strong finish as well! After the initial setup, it's just a matter of running experiments, most of which is just waiting for models to finish training. At the beginning of the challenge, I laid out my plan for what I was going to do and just focused on executing it. I didn't deviate from that initial plan very much. Fortunately it worked out. ", "votes": 7 }, { "id": 1062099, "postDate": "2020-10-27T15:13:10.787Z", "content": "Ian, should have figured this out years ago in bone age, but you are truly on another plane. Echoing Alexandre's sentiment. Glad you're a rad.
", "rawMarkdown": "Ian, should have figured this out years ago in bone age, but you are truly on another plane. Echoing Alexandre's sentiment. Glad you're a rad.", "votes": 1 }, { "id": 1064377, "postDate": "2020-10-30T03:52:25.100Z", "content": "Congrats @vaillant just a thought whoch I couldn't test ,using segmented lung as one channel instead of windowed one
", "rawMarkdown": "Congrats @vaillant just a thought whoch I couldn't test ,using segmented lung as one channel instead of windowed one " } ] }, { "id": 1117660, "postDate": "2020-12-18T10:09:45.573Z", "content": "Ian, this is amazing and inspiring. I had a look at the first place solution and that sent me down several rabbit holes, and I'm only getting to look at this for the first time now. I suspect this will send me on a 3 month detour!
\nI'm hoping I could get your insights on something. There are a lot of specific choices in this setup, and I noticed your comment below about having planned this at the start, and not deviating much from it. In light of that, it would be super helpful to get your thoughts on which specific choices you are most convinced made a strong difference, and which you are still unsure of.
", "rawMarkdown": "Ian, this is amazing and inspiring. I had a look at the first place solution and that sent me down several rabbit holes, and I'm only getting to look at this for the first time now. I suspect this will send me on a 3 month detour!\n\nI'm hoping I could get your insights on something. There are a lot of specific choices in this setup, and I noticed your comment below about having planned this at the start, and not deviating much from it. In light of that, it would be super helpful to get your thoughts on which specific choices you are most convinced made a strong difference, and which you are still unsure of.", "votes": 1 }, { "id": 1061385, "postDate": "2020-10-27T01:18:25.680Z", "content": "Excellent work- thanks for sharing your solution. I'm going to come back and read this a few more times!
", "rawMarkdown": "Excellent work- thanks for sharing your solution. I'm going to come back and read this a few more times!", "votes": 1 }, { "id": 1062116, "postDate": "2020-10-27T15:30:21.920Z", "content": "It's a great honor for me to have such words from you. Congratulations!
", "rawMarkdown": "It's a great honor for me to have such words from you. Congratulations!", "votes": 2 }, { "id": 1064518, "postDate": "2020-10-30T07:57:47.043Z", "content": "Wow, what a useful notebook. I'm using it for further education.
", "rawMarkdown": "Wow, what a useful notebook. I'm using it for further education." }, { "id": 1061566, "postDate": "2020-10-27T04:56:01.413Z", "content": "congratulations @vaillant
", "rawMarkdown": "congratulations @vaillant ", "votes": -1 }, { "id": 1065349, "postDate": "2020-10-31T07:33:36.190Z", "content": "it is very meaningful result
", "rawMarkdown": "it is very meaningful result" }, { "id": 1065040, "postDate": "2020-10-30T19:25:25.150Z", "content": "Wow, that was a nice documentation. Really its useful. And my deep Congratulations to winner..!
", "rawMarkdown": "Wow, that was a nice documentation. Really its useful. And my deep Congratulations to winner..!" }, { "id": 1064731, "postDate": "2020-10-30T13:14:39.067Z", "content": "Yeah this one is really very impressive
", "rawMarkdown": "Yeah this one is really very impressive " }, { "id": 1064460, "postDate": "2020-10-30T06:54:29.423Z", "content": "Good job! Interesting pipeline
", "rawMarkdown": "Good job! Interesting pipeline" }, { "id": 1064208, "postDate": "2020-10-29T20:50:47.477Z", "content": "Congratulations, and Thanks for sharing :)
", "rawMarkdown": "Congratulations, and Thanks for sharing :)" }, { "id": 1064098, "postDate": "2020-10-29T17:35:57.350Z", "content": "Congratulations
", "rawMarkdown": "Congratulations " }, { "id": 1063855, "postDate": "2020-10-29T12:22:37.437Z", "content": "congrats!!!
", "rawMarkdown": "congrats!!!" }, { "id": 1063565, "postDate": "2020-10-29T04:28:47.043Z", "content": "Well done!
", "rawMarkdown": "Well done!" }, { "id": 1063366, "postDate": "2020-10-28T19:35:38.857Z", "content": "Congrats , great job !
", "rawMarkdown": "Congrats , great job !" }, { "id": 1063307, "postDate": "2020-10-28T18:14:33.700Z", "content": "I am jealous of your setup. :p
\nJoke aside, very well done. You said:
\n\nInference took about 7-8 hours for the entire test set.
\n
Do you have a rough estimate of the training time? Many thanks and well done again!
", "rawMarkdown": "I am jealous of your setup. :p\nJoke aside, very well done. You said:\n\n> Inference took about 7-8 hours for the entire test set. \n\nDo you have a rough estimate of the training time? Many thanks and well done again!", "replies": [ { "id": 1063415, "postDate": "2020-10-28T21:58:09.563Z", "content": "The feature extractors took about 5.5 hours to train. TD-CNNs took about 7.5-8 hours to train. The 3D CNN took about 30 minutes, and the transformers were very fast- just a few minutes.
", "rawMarkdown": "The feature extractors took about 5.5 hours to train. TD-CNNs took about 7.5-8 hours to train. The 3D CNN took about 30 minutes, and the transformers were very fast- just a few minutes. ", "votes": 2 } ] }, { "id": 1062769, "postDate": "2020-10-28T06:58:43.090Z", "content": "congrats!!
", "rawMarkdown": "congrats!!" }, { "id": 1062633, "postDate": "2020-10-28T03:54:24.493Z", "content": "Wow.That's an amazing pipeline. Congratulations!
", "rawMarkdown": "Wow.That's an amazing pipeline. Congratulations!\n" }, { "id": 1062596, "postDate": "2020-10-28T02:45:00.693Z", "content": "Provisionally 2nd place now! Thanks for a very informational write and congratulations! ๐
", "rawMarkdown": "Provisionally 2nd place now! Thanks for a very informational write and congratulations! ๐" }, { "id": 1062111, "postDate": "2020-10-27T15:27:15.050Z", "content": "Congratulations! Great solution write up. Thank you.
", "rawMarkdown": "Congratulations! Great solution write up. Thank you." }, { "id": 1061680, "postDate": "2020-10-27T07:52:23.510Z", "content": "Congrats, that is an impressive solution. And thanks for your JPEG contribution too.
", "rawMarkdown": "Congrats, that is an impressive solution. And thanks for your JPEG contribution too." }, { "id": 1061675, "postDate": "2020-10-27T07:43:35.543Z", "content": "I'm trully impressed by your consistency at getting strong results in medical imaging competitions.
\nThe fact that you're a physician probably plays a role, but that's even more impressive to me since it means that you already have a really busy schedule.
\nAnyways, very elegant solution, I hope you don't get removed for that 3rd place is truly deserved.
", "rawMarkdown": "I'm trully impressed by your consistency at getting strong results in medical imaging competitions. \n\nThe fact that you're a physician probably plays a role, but that's even more impressive to me since it means that you already have a really busy schedule. \n\nAnyways, very elegant solution, I hope you don't get removed for that 3rd place is truly deserved.", "replies": [ { "id": 1063416, "postDate": "2020-10-28T22:00:14.010Z", "content": "Thank you! When I started out on Kaggle, I felt like my medical imaging domain knowledge would give me a surefire edge in these competitions. Kagglers are too smart though, so at best it's only a slight advantage.
", "rawMarkdown": "Thank you! When I started out on Kaggle, I felt like my medical imaging domain knowledge would give me a surefire edge in these competitions. Kagglers are too smart though, so at best it's only a slight advantage. ", "votes": 2 } ] }, { "id": 1061580, "postDate": "2020-10-27T05:17:56.357Z", "content": "Congratulations and thank you for your solution.
\nI have a question about feature extraction. Would you tell me the experiments with other models if you do (efficientnet, ResNext, densenetโฆ)? ResNeSt50 is better than others? or ensemble didn't work?
", "rawMarkdown": "Congratulations and thank you for your solution. \n\nI have a question about feature extraction. Would you tell me the experiments with other models if you do (efficientnet, ResNext, densenet...)? ResNeSt50 is better than others? or ensemble didn't work?", "replies": [ { "id": 1061978, "postDate": "2020-10-27T13:29:00.607Z", "content": "I tried initially with EfficientNet-B4 but in the past I've found that ResNet-like architectures worked better for tasks relying on feature extraction, so I tried ResNeSt which worked better (0.01 decrease in loss).
\nI also tried EfficientNet-B3 pruned at lower resolution to increase inference speed but this was significantly worse (0.02 increase in loss).
", "rawMarkdown": "I tried initially with EfficientNet-B4 but in the past I've found that ResNet-like architectures worked better for tasks relying on feature extraction, so I tried ResNeSt which worked better (0.01 decrease in loss). \n\nI also tried EfficientNet-B3 pruned at lower resolution to increase inference speed but this was significantly worse (0.02 increase in loss). ", "votes": 2 } ] }, { "id": 1061488, "postDate": "2020-10-27T03:16:04.190Z", "content": "Great @vaillant
", "rawMarkdown": "Great @vaillant " }, { "id": 1061457, "postDate": "2020-10-27T02:48:58.723Z", "content": "thought about hand labeling the heart earlier, but then i gave up :(
", "rawMarkdown": "thought about hand labeling the heart earlier, but then i gave up :(", "replies": [ { "id": 1061970, "postDate": "2020-10-27T13:24:38.727Z", "content": "Seems like it didn't make too much of a difference. ๐
", "rawMarkdown": "Seems like it didn't make too much of a difference. ๐ " } ] }, { "id": 1061448, "postDate": "2020-10-27T02:37:24.063Z", "content": "congratulations @vaillant !!
\nand thanks for sharing the images and help throughout the competition ๐
Congratulations Ian!
", "rawMarkdown": "Congratulations Ian!" }, { "id": 1061439, "postDate": "2020-10-27T02:32:32.847Z", "content": "Congratulations Ian! Really amazing approach for formulating this problem in a single end-to-end model. You combined amazing clinical/problem insights with cutting edge architecture. I hope institutions can leverage this to build better tools for PE triage! -- Also a double congrats for doing this during your intern year :)
", "rawMarkdown": "Congratulations Ian! Really amazing approach for formulating this problem in a single end-to-end model. You combined amazing clinical/problem insights with cutting edge architecture. I hope institutions can leverage this to build better tools for PE triage! -- Also a double congrats for doing this during your intern year :)" }, { "id": 1892427, "postDate": "2022-08-10T05:00:14.293Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1062145, "postDate": "2020-10-27T15:51:28.563Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1061377, "author_name": "Yee Ng", "author_url": "", "post_date": "2020-10-27T01:03:18.117000", "content": "Congratulations! Great solution write up. Thank you. So many ingenuities squeezed into one diagram.
\nI have just one question. How did you find time to do all that as an intern in a hospital as busy as yours?
", "votes": 5, "replies": [ { "id": 1061927, "author_name": "Alexandre Cadrin-Chรชnevert", "author_url": "", "post_date": "2020-10-27T12:45:34.570000", "content": "Congrats Ian and Yee ! Both very impressive results. Go Radiology go !
\nAbout your question Yee, I stopped thinking years ago that Ian is a human like us.
", "votes": 2, "replies": [] }, { "id": 1061940, "author_name": "Yee Ng", "author_url": "", "post_date": "2020-10-27T13:00:57.603000", "content": "\n\nI stopped thinking years ago that Ian is a human like us.
\n
haha. Agree.
", "votes": 1, "replies": [] }, { "id": 1061973, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-27T13:26:36.360000", "content": "Thanks, congrats on your strong finish as well! After the initial setup, it's just a matter of running experiments, most of which is just waiting for models to finish training. At the beginning of the challenge, I laid out my plan for what I was going to do and just focused on executing it. I didn't deviate from that initial plan very much. Fortunately it worked out.
", "votes": 7, "replies": [] }, { "id": 1062099, "author_name": "Jesse", "author_url": "", "post_date": "2020-10-27T15:13:10.787000", "content": "Ian, should have figured this out years ago in bone age, but you are truly on another plane. Echoing Alexandre's sentiment. Glad you're a rad.
", "votes": 1, "replies": [] }, { "id": 1064377, "author_name": "Jaideep", "author_url": "", "post_date": "2020-10-30T03:52:25.100000", "content": "Congrats @vaillant just a thought whoch I couldn't test ,using segmented lung as one channel instead of windowed one
", "votes": 0, "replies": [] } ] }, { "id": 1117660, "author_name": "Alexander Soare", "author_url": "", "post_date": "2020-12-18T10:09:45.573000", "content": "Ian, this is amazing and inspiring. I had a look at the first place solution and that sent me down several rabbit holes, and I'm only getting to look at this for the first time now. I suspect this will send me on a 3 month detour!
\nI'm hoping I could get your insights on something. There are a lot of specific choices in this setup, and I noticed your comment below about having planned this at the start, and not deviating much from it. In light of that, it would be super helpful to get your thoughts on which specific choices you are most convinced made a strong difference, and which you are still unsure of.
", "votes": 1, "replies": [] }, { "id": 1061385, "author_name": "Rob Mulla", "author_url": "", "post_date": "2020-10-27T01:18:25.680000", "content": "Excellent work- thanks for sharing your solution. I'm going to come back and read this a few more times!
", "votes": 1, "replies": [] }, { "id": 1062116, "author_name": "OsciiArt", "author_url": "", "post_date": "2020-10-27T15:30:21.920000", "content": "It's a great honor for me to have such words from you. Congratulations!
", "votes": 2, "replies": [] }, { "id": 1064518, "author_name": "Albert Azeri", "author_url": "", "post_date": "2020-10-30T07:57:47.043000", "content": "Wow, what a useful notebook. I'm using it for further education.
", "votes": 0, "replies": [] }, { "id": 1061566, "author_name": "Akash Singh", "author_url": "", "post_date": "2020-10-27T04:56:01.413000", "content": "congratulations @vaillant
", "votes": -1, "replies": [] }, { "id": 1065349, "author_name": "Jiwwwooong", "author_url": "", "post_date": "2020-10-31T07:33:36.190000", "content": "it is very meaningful result
", "votes": 0, "replies": [] }, { "id": 1065040, "author_name": "Shrivas@Amrith", "author_url": "", "post_date": "2020-10-30T19:25:25.150000", "content": "Wow, that was a nice documentation. Really its useful. And my deep Congratulations to winner..!
", "votes": 0, "replies": [] }, { "id": 1064731, "author_name": "Aman Jain", "author_url": "", "post_date": "2020-10-30T13:14:39.067000", "content": "Yeah this one is really very impressive
", "votes": 0, "replies": [] }, { "id": 1064460, "author_name": "whoami", "author_url": "", "post_date": "2020-10-30T06:54:29.423000", "content": "Good job! Interesting pipeline
", "votes": 0, "replies": [] }, { "id": 1064208, "author_name": "Dreamer", "author_url": "", "post_date": "2020-10-29T20:50:47.477000", "content": "Congratulations, and Thanks for sharing :)
", "votes": 0, "replies": [] }, { "id": 1064098, "author_name": "Mohammed Saket", "author_url": "", "post_date": "2020-10-29T17:35:57.350000", "content": "Congratulations
", "votes": 0, "replies": [] }, { "id": 1063855, "author_name": "Yasser Radouani", "author_url": "", "post_date": "2020-10-29T12:22:37.437000", "content": "congrats!!!
", "votes": 0, "replies": [] }, { "id": 1063565, "author_name": "Issac Antony", "author_url": "", "post_date": "2020-10-29T04:28:47.043000", "content": "Well done!
", "votes": 0, "replies": [] }, { "id": 1063366, "author_name": "Emirhan KALEM", "author_url": "", "post_date": "2020-10-28T19:35:38.857000", "content": "Congrats , great job !
", "votes": 0, "replies": [] }, { "id": 1063307, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-10-28T18:14:33.700000", "content": "I am jealous of your setup. :p
\nJoke aside, very well done. You said:
\n\nInference took about 7-8 hours for the entire test set.
\n
Do you have a rough estimate of the training time? Many thanks and well done again!
", "votes": 0, "replies": [ { "id": 1063415, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-28T21:58:09.563000", "content": "The feature extractors took about 5.5 hours to train. TD-CNNs took about 7.5-8 hours to train. The 3D CNN took about 30 minutes, and the transformers were very fast- just a few minutes.
", "votes": 2, "replies": [] } ] }, { "id": 1062769, "author_name": "yui coiuyu", "author_url": "", "post_date": "2020-10-28T06:58:43.090000", "content": "congrats!!
", "votes": 0, "replies": [] }, { "id": 1062633, "author_name": "Ronaldo S.A. Batista", "author_url": "", "post_date": "2020-10-28T03:54:24.493000", "content": "Wow.That's an amazing pipeline. Congratulations!
", "votes": 0, "replies": [] }, { "id": 1062596, "author_name": "Amrit Virdee", "author_url": "", "post_date": "2020-10-28T02:45:00.693000", "content": "Provisionally 2nd place now! Thanks for a very informational write and congratulations! ๐
", "votes": 0, "replies": [] }, { "id": 1062111, "author_name": "NANDINI SINGH 05", "author_url": "", "post_date": "2020-10-27T15:27:15.050000", "content": "Congratulations! Great solution write up. Thank you.
", "votes": 0, "replies": [] }, { "id": 1061680, "author_name": "Vee", "author_url": "", "post_date": "2020-10-27T07:52:23.510000", "content": "Congrats, that is an impressive solution. And thanks for your JPEG contribution too.
", "votes": 0, "replies": [] }, { "id": 1061675, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-10-27T07:43:35.543000", "content": "I'm trully impressed by your consistency at getting strong results in medical imaging competitions.
\nThe fact that you're a physician probably plays a role, but that's even more impressive to me since it means that you already have a really busy schedule.
\nAnyways, very elegant solution, I hope you don't get removed for that 3rd place is truly deserved.
", "votes": 0, "replies": [ { "id": 1063416, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-28T22:00:14.010000", "content": "Thank you! When I started out on Kaggle, I felt like my medical imaging domain knowledge would give me a surefire edge in these competitions. Kagglers are too smart though, so at best it's only a slight advantage.
", "votes": 2, "replies": [] } ] }, { "id": 1061580, "author_name": "Youngno Yoon", "author_url": "", "post_date": "2020-10-27T05:17:56.357000", "content": "Congratulations and thank you for your solution.
\nI have a question about feature extraction. Would you tell me the experiments with other models if you do (efficientnet, ResNext, densenetโฆ)? ResNeSt50 is better than others? or ensemble didn't work?
", "votes": 0, "replies": [ { "id": 1061978, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-27T13:29:00.607000", "content": "I tried initially with EfficientNet-B4 but in the past I've found that ResNet-like architectures worked better for tasks relying on feature extraction, so I tried ResNeSt which worked better (0.01 decrease in loss).
\nI also tried EfficientNet-B3 pruned at lower resolution to increase inference speed but this was significantly worse (0.02 increase in loss).
", "votes": 2, "replies": [] } ] }, { "id": 1061488, "author_name": "Jaideep", "author_url": "", "post_date": "2020-10-27T03:16:04.190000", "content": "Great @vaillant
", "votes": 0, "replies": [] }, { "id": 1061457, "author_name": "DatNT", "author_url": "", "post_date": "2020-10-27T02:48:58.723000", "content": "thought about hand labeling the heart earlier, but then i gave up :(
", "votes": 0, "replies": [ { "id": 1061970, "author_name": "Ian Pan", "author_url": "", "post_date": "2020-10-27T13:24:38.727000", "content": "Seems like it didn't make too much of a difference. ๐
", "votes": 0, "replies": [] } ] }, { "id": 1061448, "author_name": "Kamal Das", "author_url": "", "post_date": "2020-10-27T02:37:24.063000", "content": "congratulations @vaillant !!
\nand thanks for sharing the images and help throughout the competition ๐
Congratulations Ian!
", "votes": 0, "replies": [] }, { "id": 1061439, "author_name": "aksg87", "author_url": "", "post_date": "2020-10-27T02:32:32.847000", "content": "Congratulations Ian! Really amazing approach for formulating this problem in a single end-to-end model. You combined amazing clinical/problem insights with cutting edge architecture. I hope institutions can leverage this to build better tools for PE triage! -- Also a double congrats for doing this during your intern year :)
", "votes": 0, "replies": [] }, { "id": 1892427, "author_name": "", "author_url": "", "post_date": "2022-08-10T05:00:14.293000", "content": "", "votes": 0, "replies": [] }, { "id": 1062145, "author_name": "", "author_url": "", "post_date": "2020-10-27T15:51:28.563000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1061321": "Update: code available at https://github.com/i-pan/kaggle-rsna-pe\n\nCongratulations to all the participants and the winners. Special congrats to prize winners @osciiart and @jpbremer who are on track to be physician GMs. This was a tough, compute-heavy challenge given the large amount of data and short timeframe. My setup was 4 24 GB Quadro RTX 6000 GPUs. I always feel guilty during competitions like these since I have the luxury of strong compute. Models were trained using DDP in PyTorch 1.6 with automatic mixed precision. \n\nEven though the results aren't final yet and my submission may be removed for violating the label consistency requirements (hopefully my heuristic for fixing those worked!), I still wanted to share my solution.\n\n\n\nHere is a schematic outlining my solution. It has a lot of moving parts, so I apologize for the lengthy summary. \n\n# Step 1: Feature Extraction\n\nLast year's RSNA Intracranial Hemorrhage Detection challenge shared a lot of similarities with this year's challenge. Most of the top solutions combined 2D CNN feature extraction with sequence modeling. The backbone of my solution also relied on a similar setup. \n\nI first trained ResNeSt50 on 2D images. Images were windowed using the PE-specific window (WL=100, WW=700) that I mentioned in one of my initial posts on DICOM processing. Each \"image\" was 3 channels, with each channel representing an individual slice. Thus the 2D image was a stack of 3 continuous slices. The targets were the 7 PE-related labels (i.e., excluding the RV/LV ratio labels). Note: most of the PE-related labels were exam-level labels. However, I just assigned the exam labels to each slice positive for PE (negative slices had all zeros), with the understanding that there would be label noise. Also, I predicted the labels for the middle slice among the 3 slices in the image. Models were trained using vanilla binary cross-entropy loss (`BCEWithLogitsLoss` in PyTorch), 512x512 with 448x448 random crops (single center crop during inference), RandAugment data augmentation, batch size 128, 5 epochs, 2500 steps per epoch, RAdam optimizer with cosine annealing learning rate scheduler. I used generalized mean pooling and reduced the final feature vector to 512-D. Mean loss was around 0.06 (AUC 0.95-0.96 for slice-wise prediction of PE vs. no PE). Features were then extracted for all slices. \n\n# Step 2: Sequence Modeling\n\nMany of last year's solutions used LSTMs/GRUs as the sequence model of choice. For this competition, I used `huggingface` transformers (specifically, the `Transformer` class from `transformers.modeling_distilbert`). I used one 4-layer transformer to produce slice-wise `pe_present_on_image` predictions and another transformer to predict exam-wise PE-labels. Sequence length was 512 during training, padded/truncated as necessary. During inference, I used the sequence without modifications. \n\nImportant point: **Images from non-PE exams do not contribute to the loss.** At first, I was training the slice-wise transformer on all exams. Then, I decided to train these models on positive exams only. This lowered my CV by about 0.01-0.02. I used a custom weighted loss where I weighted the loss from each example by the proportion of positive PE slices (as described in the metric), though I'm still not sure I wrote it correctly. \n\nThe exam-level transformer was trained using a weighted BCE loss based on the competition label weights. Exam-level validation losses ranged from 0.15-0.17. \n\n# Step 3: Time-Distributed CNN\n\nTo add some variety into my modeling, I then trained a time-distributed CNN, which is just another way of saying I stacked a transformer on top of a CNN feature extraction backbone and trained end-to-end. \n\nBut before doing that, I performed inference using the slice-wise transformer model to get PE scores for every slice (5-fold OOF predictions). Then, when training the TD-CNN, I only trained on the top 30% of slices from each exam, sorted by PE score. These were trained on 3D volumes of size 32x416x416 cropped to 32x364x364 using the same windowing strategy (WL=100, WW=700) in batches of 16. \n\nI initialized the CNN backbone and the transformer head with trained models from steps 1 and 2 to help with convergence. I forced all the batch normalization layers in the backbone to `eval` mode as well- this prevents the running mean and variance in each layer from updating and only trains the coefficients. Exam-level validation losses ranged from 0.15-0.17, similar to step 2. \n\n# Step 4: Heart Slice Prediction\n\nRV/LV ratio is a significant portion of the loss. I hand-labeled slices with heart in 1,000 CT scans and trained a model (EfficientNet-B1 pruned, AUC 0.998, 256x256->224x224 crops) to classify heart slices in each CT scan. I did this because I felt that by focusing a model on the heart, I could get better, more consistent results across scans. It actually wasn't hard to label 1,000 scans- probably a full day's worth of work. I just needed to find the top and bottom heart slices; everything in between thus must also contain the heart.\n\n# Step 5: RV/LV 3D CNN\n\nI trained a 3D CNN to classify RV/LV ratio. Specifically, I used a 101-layer channel separated network, pretrained on 65 million Instagram videos (https://arxiv.org/abs/1904.02811, https://github.com/facebookresearch/VMZ). \n\nThis model was trained only on heart slices from each exam. Also, it was only trained on positive exams. This is because RV/LV ratio was not labeled for negative exams- both RV/LV labels are 0. Thus, it didn't make sense to me to try and train my model on the entire dataset's labels directly. Models were trained using a weighted BCE loss using the competition weights. I resized the input to 64x256x256->64x224x224 crops and used mediastinal window (WL=50, WW=350). AUC for RV/LV ratio > 1 was about 0.85. The validation loss ranged from 0.44-0.48. I then used these models to extract 2048-D features from each exam.\n\nThe challenge here was that the likelihood of an exam being PE-positive greatly influenced the RV/LV labels. From step 2, I calculated 5-fold OOF exam-level predictions for all exams. Then, I trained a linear model that took as input the concatenation of the 2048-D 3D CNN feature and the 7 PE exam labels. This linear model was trained across **all exams** using the labels directly. This way, the model could take into account the imaging features from the scan but also adjust the predictions based on the likelihood of PE. Validation losses ranged from 0.22-0.25. \n\nI didn't validate my entire pipeline that often due to time constraints, instead choosing to focus on the individual component losses and optimizing each one as well as I could. I did periodically do a sanity-check on 200 single-fold exams to make sure that my entire pipeline was working, and my final validation loss was 0.183 for my 0.150 public LB submission. I'm still not confident that I implemented the metric correctly, but I was seeing good correlation (0.159/0.195->0.156/0.188-> 0.150/0.183).\n\nAt the end, I applied a function to enforce label consistency requirements for each exam. There was about 0.001 change in CV after applying this function. I have some thoughts about this requirement which maybe I'll save for another post. \n\n# Final Models\n\nOverall, I used these models:\n-2x ResNeSt50 feature extractors\n-6x exam transformers (3 for each extractor)\n-6x slice transformers\n-5x ResNeSt50 TD-CNN\n-1x EfficientNet-B1 pruned heart slice classifier\n-5x ip-CSN-101 3D CNN RV/LV feature extractor\n-5x RV/LV linear model \n\nInference took about 7-8 hours for the entire test set. \n\nCode to follow once I clean it up. \n\nThings that didn't work as well:\n-3D CNN for PE exam-level prediction\n-Pseudolabeling RV/LV ratio for negative exams\n-Using logits/probabilities instead of features for second-stage model\n-TD-CNN for RV/LV ratio prediction\n-Training feature extractor on positive exams only to improve slice-wise PE prediction ", "1061377": "Congratulations! Great solution write up. Thank you. So many ingenuities squeezed into one diagram.\n\nI have just one question. How did you find time to do all that as an intern in a hospital as busy as yours?", "1117660": "Ian, this is amazing and inspiring. I had a look at the first place solution and that sent me down several rabbit holes, and I'm only getting to look at this for the first time now. I suspect this will send me on a 3 month detour!\n\nI'm hoping I could get your insights on something. There are a lot of specific choices in this setup, and I noticed your comment below about having planned this at the start, and not deviating much from it. In light of that, it would be super helpful to get your thoughts on which specific choices you are most convinced made a strong difference, and which you are still unsure of.", "1061385": "Excellent work- thanks for sharing your solution. I'm going to come back and read this a few more times!", "1062116": "It's a great honor for me to have such words from you. Congratulations!", "1064518": "Wow, what a useful notebook. I'm using it for further education.", "1061566": "congratulations @vaillant ", "1065349": "it is very meaningful result", "1065040": "Wow, that was a nice documentation. Really its useful. And my deep Congratulations to winner..!", "1064731": "Yeah this one is really very impressive ", "1064460": "Good job! Interesting pipeline", "1064208": "Congratulations, and Thanks for sharing :)", "1064098": "Congratulations ", "1063855": "congrats!!!", "1063565": "Well done!", "1063366": "Congrats , great job !", "1063307": "I am jealous of your setup. :p\nJoke aside, very well done. You said:\n\n> Inference took about 7-8 hours for the entire test set. \n\nDo you have a rough estimate of the training time? Many thanks and well done again!", "1062769": "congrats!!", "1062633": "Wow.That's an amazing pipeline. Congratulations!\n", "1062596": "Provisionally 2nd place now! Thanks for a very informational write and congratulations! ๐", "1062111": "Congratulations! Great solution write up. Thank you.", "1061680": "Congrats, that is an impressive solution. And thanks for your JPEG contribution too.", "1061675": "I'm trully impressed by your consistency at getting strong results in medical imaging competitions. \n\nThe fact that you're a physician probably plays a role, but that's even more impressive to me since it means that you already have a really busy schedule. \n\nAnyways, very elegant solution, I hope you don't get removed for that 3rd place is truly deserved.", "1061580": "Congratulations and thank you for your solution. \n\nI have a question about feature extraction. Would you tell me the experiments with other models if you do (efficientnet, ResNext, densenet...)? ResNeSt50 is better than others? or ensemble didn't work?", "1061488": "Great @vaillant ", "1061457": "thought about hand labeling the heart earlier, but then i gave up :(", "1061448": "congratulations @vaillant !!\nand thanks for sharing the images and help throughout the competition ๐", "1061441": "Congratulations Ian!", "1061439": "Congratulations Ian! Really amazing approach for formulating this problem in a single end-to-end model. You combined amazing clinical/problem insights with cutting edge architecture. I hope institutions can leverage this to build better tools for PE triage! -- Also a double congrats for doing this during your intern year :)", "1892427": "", "1062145": "" } }