{ "id": 193460, "title": "7th place - EfficientNet, Transformer and a 2nd opinion [edited]", "url": "/competitions/rsna-str-pulmonary-embolism-detection/discussion/193460", "author_name": "yuval reina", "post_date": "2020-10-27T06:50:57.040000", "votes": 54, "comment_count": 7, "views": 0, "content": "

I used a two stage model.

\n
    \n
  1. A EfficientNet (B5, B3) used for feature extraction per image
    2. \n
  2. A transformer used per series to predict the series related classes and the 'PE Present on Image' per image
    3. \n
\n

This is the 2nd competition I use such network and an extensive description of this model can be found here

\n

The targets for the EfficientNet where:

\n\n

The loss was weighted BCE - the weights reflecting the competitions matric weights.
\nI used flip, rotate, random resize/crop, mean/std shift as augmentation.
\nI also use trainable 3 windows to convert the CT image to jpeg (WSO )

\n

The transformer was a 4 layer encoder (using Pytorch's transformer encoder module). Where the relative and absolute places of the images in the series were embedded and added to the features vectors (as is done for positional embedding in NLP transformers such as BERT)
\nThe loss function reflected the competition's matric.

\n

This model gave an LB of 0.166

\n

Ensembling improved the LB to 0.162, but I could only ensemble 2 models in the time frame (I didn't use the public/private LB trick that can add 25%). to improve this I used a 2nd opinion mechanism - instead of using a 2nd model to inference all the data again, I did what an MD will do, I chose only series where the results where the most uncertain (near 0.5), inference them with another model and ensembled - I did this 3 times for ~ 30-40% of the data each time gaining an equivalence of ensembling 4 models => LB 0.157

\n

As a last step, I checked if any prediction meet the competition's Label Consistency Requirement and if it didn't, made the minimal changes needed to meet the requirements

\n", "messages": [ { "id": 1061641, "postDate": "2020-10-27T06:50:57.040Z", "content": "

I used a two stage model.

\n
    \n
  1. A EfficientNet (B5, B3) used for feature extraction per image
    2. \n
  2. A transformer used per series to predict the series related classes and the 'PE Present on Image' per image
    3. \n
\n

This is the 2nd competition I use such network and an extensive description of this model can be found here

\n

The targets for the EfficientNet where:

\n\n

The loss was weighted BCE - the weights reflecting the competitions matric weights.
\nI used flip, rotate, random resize/crop, mean/std shift as augmentation.
\nI also use trainable 3 windows to convert the CT image to jpeg (WSO )

\n

The transformer was a 4 layer encoder (using Pytorch's transformer encoder module). Where the relative and absolute places of the images in the series were embedded and added to the features vectors (as is done for positional embedding in NLP transformers such as BERT)
\nThe loss function reflected the competition's matric.

\n

This model gave an LB of 0.166

\n

Ensembling improved the LB to 0.162, but I could only ensemble 2 models in the time frame (I didn't use the public/private LB trick that can add 25%). to improve this I used a 2nd opinion mechanism - instead of using a 2nd model to inference all the data again, I did what an MD will do, I chose only series where the results where the most uncertain (near 0.5), inference them with another model and ensembled - I did this 3 times for ~ 30-40% of the data each time gaining an equivalence of ensembling 4 models => LB 0.157

\n

As a last step, I checked if any prediction meet the competition's Label Consistency Requirement and if it didn't, made the minimal changes needed to meet the requirements

\n", "rawMarkdown": "I used a two stage model.\n1. A EfficientNet (B5, B3) used for feature extraction per image\n2. A transformer used per series to predict the series related classes and the 'PE Present on Image' per image\n\nThis is the 2nd competition I use such network and an extensive description of this model can be found [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/181830) \n\nThe targets for the EfficientNet where: \n* The original targets for images where PE Present on Image = 1\n* 0 for every other image. Except the Intermediate target which remained the same.\n\nThe loss was weighted BCE - the weights reflecting the competitions matric weights.\nI used flip, rotate, random resize/crop, mean/std shift as augmentation.\nI also use trainable 3 windows to convert the CT image to jpeg ([WSO](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117480) )\n\nThe transformer was a 4 layer encoder (using Pytorch's transformer encoder module). Where the relative and absolute places of the images in the series were embedded and added to the features vectors (as is done for positional embedding in NLP transformers such as BERT)\nThe loss function reflected the competition's matric.\n\nThis model gave an **LB of 0.166** \n\nEnsembling improved the **LB to 0.162**, but I could only ensemble 2 models in the time frame (I didn't use the public/private LB trick that can add 25%). to improve this I used a **2nd opinion mechanism** - instead of using a 2nd model to inference all the data again, I did what an MD will do, I chose only series where the results where the most uncertain (near 0.5), inference them with another model and ensembled - I did this 3 times for ~ 30-40% of the data each time gaining an equivalence of ensembling 4 models => **LB 0.157**\n\nAs a last step, I checked if any prediction meet the competition's [Label Consistency Requirement ](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473) and if it didn't, made the minimal changes needed to meet the requirements\n\n* The full code can be found in [git](https://github.com/yuval6957/RSNA2020_final.git)\n\n* The inference code can be found [in this notebook](https://www.kaggle.com/yuval6967/rsna2020-inference-2nd-op-final)\n\n* The models' weights are [in this](https://www.kaggle.com/yuval6967/rsna2020-models) public dataset\n\n* A more detailed description can be found [in this documentation](https://github.com/yuval6957/RSNA2020_final/blob/master/Documentation.md) and [this presentation](https://github.com/yuval6957/RSNA2020_final/blob/master/RSNA2020%20presentation.pdf)\n\n* [This is a video](https://www.youtube.com/watch?v=hVgIawktZgs) which present this solution\n ", "votes": 54 }, { "id": 1063923, "postDate": "2020-10-29T14:03:17.273Z", "content": "

Thanks for sharing and good explanation!
\nRecently, we are seeing a lot of possibilities of transformer into vision tasks.
\nThis solution will be a good example.

\n

Congrats and thanks again. @yuval6967

", "rawMarkdown": "Thanks for sharing and good explanation!\nRecently, we are seeing a lot of possibilities of transformer into vision tasks.\nThis solution will be a good example.\n\nCongrats and thanks again. @yuval6967 ", "votes": 1 }, { "id": 1061835, "postDate": "2020-10-27T11:14:50.790Z", "content": "

Congratulations on your medal +1

", "rawMarkdown": "Congratulations on your medal +1", "votes": 1 }, { "id": 1061728, "postDate": "2020-10-27T09:13:07.880Z", "content": "

Congrats on 8th place and solo gold medal @yuval6967 and thanks for sharing solution

", "rawMarkdown": "Congrats on 8th place and solo gold medal @yuval6967 and thanks for sharing solution", "votes": 1 }, { "id": 1061704, "postDate": "2020-10-27T08:30:04.373Z", "content": "
\n

2nd opinion mechanism - instead of using a 2nd model to inference all the data again, I did what an MD will do, I chose only series where the results where the most uncertain (near 0.5), inference them with another model and ensembled - I did this 3 times for ~ 30-40% of the data each time gaining an equivalence of ensembling 4 models => LB 0.157

\n
\n

Just wow. Thanks for this technique.

", "rawMarkdown": "> 2nd opinion mechanism - instead of using a 2nd model to inference all the data again, I did what an MD will do, I chose only series where the results where the most uncertain (near 0.5), inference them with another model and ensembled - I did this 3 times for ~ 30-40% of the data each time gaining an equivalence of ensembling 4 models => LB 0.157\n\nJust wow. Thanks for this technique.", "votes": 1 }, { "id": 1061691, "postDate": "2020-10-27T08:03:36.050Z", "content": "

Nice solution, thank you for sharing and congratulation 👍

", "rawMarkdown": "Nice solution, thank you for sharing and congratulation 👍", "votes": 1 }, { "id": 1061658, "postDate": "2020-10-27T07:20:28.500Z", "content": "

@yuval6967 Congratulations!

", "rawMarkdown": "@yuval6967 Congratulations!", "votes": 1 }, { "id": 1061644, "postDate": "2020-10-27T06:58:38.027Z", "content": "

Thank for sharing and congratulations @yuval6967 !

", "rawMarkdown": "Thank for sharing and congratulations @yuval6967 !", "votes": 1 } ], "comments": [ { "id": 1063923, "author_name": "Heroseo", "author_url": "", "post_date": "2020-10-29T14:03:17.273000", "content": "

Thanks for sharing and good explanation!
\nRecently, we are seeing a lot of possibilities of transformer into vision tasks.
\nThis solution will be a good example.

\n

Congrats and thanks again. @yuval6967

", "votes": 1, "replies": [] }, { "id": 1061835, "author_name": "Eisa", "author_url": "", "post_date": "2020-10-27T11:14:50.790000", "content": "

Congratulations on your medal +1

", "votes": 1, "replies": [] }, { "id": 1061728, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-10-27T09:13:07.880000", "content": "

Congrats on 8th place and solo gold medal @yuval6967 and thanks for sharing solution

", "votes": 1, "replies": [] }, { "id": 1061704, "author_name": "sin", "author_url": "", "post_date": "2020-10-27T08:30:04.373000", "content": "
\n

2nd opinion mechanism - instead of using a 2nd model to inference all the data again, I did what an MD will do, I chose only series where the results where the most uncertain (near 0.5), inference them with another model and ensembled - I did this 3 times for ~ 30-40% of the data each time gaining an equivalence of ensembling 4 models => LB 0.157

\n
\n

Just wow. Thanks for this technique.

", "votes": 1, "replies": [] }, { "id": 1061691, "author_name": "", "author_url": "", "post_date": "2020-10-27T08:03:36.050000", "content": "

Nice solution, thank you for sharing and congratulation 👍

", "votes": 1, "replies": [] }, { "id": 1061658, "author_name": "Alexey Kachalov", "author_url": "", "post_date": "2020-10-27T07:20:28.500000", "content": "

@yuval6967 Congratulations!

", "votes": 1, "replies": [] }, { "id": 1061644, "author_name": "DungNB", "author_url": "", "post_date": "2020-10-27T06:58:38.027000", "content": "

Thank for sharing and congratulations @yuval6967 !

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1061641": "I used a two stage model.\n1. A EfficientNet (B5, B3) used for feature extraction per image\n2. A transformer used per series to predict the series related classes and the 'PE Present on Image' per image\n\nThis is the 2nd competition I use such network and an extensive description of this model can be found [here](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/181830) \n\nThe targets for the EfficientNet where: \n* The original targets for images where PE Present on Image = 1\n* 0 for every other image. Except the Intermediate target which remained the same.\n\nThe loss was weighted BCE - the weights reflecting the competitions matric weights.\nI used flip, rotate, random resize/crop, mean/std shift as augmentation.\nI also use trainable 3 windows to convert the CT image to jpeg ([WSO](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117480) )\n\nThe transformer was a 4 layer encoder (using Pytorch's transformer encoder module). Where the relative and absolute places of the images in the series were embedded and added to the features vectors (as is done for positional embedding in NLP transformers such as BERT)\nThe loss function reflected the competition's matric.\n\nThis model gave an **LB of 0.166** \n\nEnsembling improved the **LB to 0.162**, but I could only ensemble 2 models in the time frame (I didn't use the public/private LB trick that can add 25%). to improve this I used a **2nd opinion mechanism** - instead of using a 2nd model to inference all the data again, I did what an MD will do, I chose only series where the results where the most uncertain (near 0.5), inference them with another model and ensembled - I did this 3 times for ~ 30-40% of the data each time gaining an equivalence of ensembling 4 models => **LB 0.157**\n\nAs a last step, I checked if any prediction meet the competition's [Label Consistency Requirement ](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/183473) and if it didn't, made the minimal changes needed to meet the requirements\n\n* The full code can be found in [git](https://github.com/yuval6957/RSNA2020_final.git)\n\n* The inference code can be found [in this notebook](https://www.kaggle.com/yuval6967/rsna2020-inference-2nd-op-final)\n\n* The models' weights are [in this](https://www.kaggle.com/yuval6967/rsna2020-models) public dataset\n\n* A more detailed description can be found [in this documentation](https://github.com/yuval6957/RSNA2020_final/blob/master/Documentation.md) and [this presentation](https://github.com/yuval6957/RSNA2020_final/blob/master/RSNA2020%20presentation.pdf)\n\n* [This is a video](https://www.youtube.com/watch?v=hVgIawktZgs) which present this solution\n ", "1063923": "Thanks for sharing and good explanation!\nRecently, we are seeing a lot of possibilities of transformer into vision tasks.\nThis solution will be a good example.\n\nCongrats and thanks again. @yuval6967 ", "1061835": "Congratulations on your medal +1", "1061728": "Congrats on 8th place and solo gold medal @yuval6967 and thanks for sharing solution", "1061704": "> 2nd opinion mechanism - instead of using a 2nd model to inference all the data again, I did what an MD will do, I chose only series where the results where the most uncertain (near 0.5), inference them with another model and ensembled - I did this 3 times for ~ 30-40% of the data each time gaining an equivalence of ensembling 4 models => LB 0.157\n\nJust wow. Thanks for this technique.", "1061691": "Nice solution, thank you for sharing and congratulation 👍", "1061658": "@yuval6967 Congratulations!", "1061644": "Thank for sharing and congratulations @yuval6967 !" } }