I want to thank my amazing team @aerdem4 @drhabib @cateek and @proletheus. This was a very challenging competition and I had a great time collaborating as a team and discussing ideas. Also thanks to the host and winners! I trust the top solutions will positively benefit physicians and those impacted by pulmonary embolism.
\nEven though our best submission was based off the from the \"baseline\" kernel that was released a week before the deadline, it's a little bittersweet. It's disappointing that such a high scoring kernel was released so late- I know it frustrated many teams (ours included). Our team was working on other end-to-end and 3D based models. I took on the role of improving the baseline as our backup plan.
\nThis turned into more of an engineering challenge, as we had to balance the limited inference time with such a large amount of data- without exceeding GPU or local memory. It was also challenging because inference time seemed to vary randomly. A submission might complete in 7 hours but then a nearly identical submission would go over 9 hours and fail.
\nDifferences between our solution and the public kernel:
\nWhat didn't work:
\nOur solution also meets the required criteria and shouldn't have conflicting label predictions. Our best submission ignoring these restrictions would've given us a private LB score of 0.177 but we didn't select it to stay within the rules.
\nI'd also like to personally thank Z by HP and NVIDIA for providing me the Z8 desktop which I put to good use on this challenge.
", "messages": [ { "id": 1061353, "postDate": "2020-10-27T00:38:28.420Z", "content": "I want to thank my amazing team @aerdem4 @drhabib @cateek and @proletheus. This was a very challenging competition and I had a great time collaborating as a team and discussing ideas. Also thanks to the host and winners! I trust the top solutions will positively benefit physicians and those impacted by pulmonary embolism.
\nEven though our best submission was based off the from the \"baseline\" kernel that was released a week before the deadline, it's a little bittersweet. It's disappointing that such a high scoring kernel was released so late- I know it frustrated many teams (ours included). Our team was working on other end-to-end and 3D based models. I took on the role of improving the baseline as our backup plan.
\nThis turned into more of an engineering challenge, as we had to balance the limited inference time with such a large amount of data- without exceeding GPU or local memory. It was also challenging because inference time seemed to vary randomly. A submission might complete in 7 hours but then a nearly identical submission would go over 9 hours and fail.
\nDifferences between our solution and the public kernel:
\nWhat didn't work:
\nOur solution also meets the required criteria and shouldn't have conflicting label predictions. Our best submission ignoring these restrictions would've given us a private LB score of 0.177 but we didn't select it to stay within the rules.
\nI'd also like to personally thank Z by HP and NVIDIA for providing me the Z8 desktop which I put to good use on this challenge.
", "rawMarkdown": "I want to thank my amazing team @aerdem4 @drhabib @cateek and @proletheus. This was a very challenging competition and I had a great time collaborating as a team and discussing ideas. Also thanks to the host and winners! I trust the top solutions will positively benefit physicians and those impacted by pulmonary embolism.\n\nEven though our best submission was based off the from the \"baseline\" kernel that was released a week before the deadline, it's a little bittersweet. It's disappointing that such a high scoring kernel was released so late- I know it frustrated many teams (ours included). Our team was working on other end-to-end and 3D based models. I took on the role of improving the baseline as our backup plan.\n\nThis turned into more of an engineering challenge, as we had to balance the limited inference time with such a large amount of data- without exceeding GPU or local memory. It was also challenging because inference time seemed to vary randomly. A submission might complete in 7 hours but then a nearly identical submission would go over 9 hours and fail.\n\nDifferences between our solution and the public kernel:\n- Trained a b6 in place of b0 for the stage 1 models.\n- Changed the inference loop to bag predictions from all 10 (5 folds x 2) stage 1 models.\n- Bagged 5x predictions from stage 2.\n- Changed the code so that it would only predict the private test during inference and used offline calculated public test predictions.\n- Modified the code to only loop through the dataloader once instead of twice for stage 1 predictions.\n- Ahmet used some magic to tweak the stage 2 model to get some added boost, including modifying the loss function.\n\nWhat didn't work:\n- ResNext101 in stage 1 models, although it had a slightly better CV score.\n- Parallel GRU + LSTM for stage 2 model.\n\nOur solution also meets the required criteria and shouldn't have conflicting label predictions. Our best submission ignoring these restrictions would've given us a private LB score of 0.177 but we didn't select it to stay within the rules.\n\nI'd also like to personally thank Z by HP and NVIDIA for providing me the Z8 desktop which I put to good use on this challenge.", "votes": 21 }, { "id": 1061498, "postDate": "2020-10-27T03:31:39.893Z", "content": "Some nice tweaks there Rob and team. Congrats on the nice finish.
", "rawMarkdown": "Some nice tweaks there Rob and team. Congrats on the nice finish.", "votes": 3, "replies": [ { "id": 1061504, "postDate": "2020-10-27T03:37:51.923Z", "content": "Thanks @watzisname - Congrats to you on a solid finish too!
", "rawMarkdown": "Thanks @watzisname - Congrats to you on a solid finish too!", "votes": 3 } ] }, { "id": 1063687, "postDate": "2020-10-29T08:00:13.807Z", "content": "Congratulations! 👍
", "rawMarkdown": "Congratulations! 👍", "votes": 1 }, { "id": 1063328, "postDate": "2020-10-28T18:43:31.100Z", "content": "@robikscube This is 28th place solution.😉
", "rawMarkdown": "@robikscube This is 28th place solution.😉", "votes": 1, "replies": [ { "id": 1063405, "postDate": "2020-10-28T21:27:13.500Z", "content": "Let's wait till LB's finalized🙂
", "rawMarkdown": "Let's wait till LB's finalized🙂", "votes": 1 } ] }, { "id": 1061391, "postDate": "2020-10-27T01:25:18.207Z", "content": "Congrats Rob and team. I'm surprised that you could gain so much LB by not enforcing rules (0.195 up to 0.177). For me, i only gain +- 0.002. Do you know why the difference is so large?
\nThat is a clever trick only predicting private test. I didn't try training a EffB6 thinking that it would never commit in time but it looks like you got a B6 to work. Nice job.
", "rawMarkdown": "Congrats Rob and team. I'm surprised that you could gain so much LB by not enforcing rules (0.195 up to 0.177). For me, i only gain +- 0.002. Do you know why the difference is so large?\n\nThat is a clever trick only predicting private test. I didn't try training a EffB6 thinking that it would never commit in time but it looks like you got a B6 to work. Nice job.", "votes": 1, "replies": [ { "id": 1061401, "postDate": "2020-10-27T01:44:41.570Z", "content": "Thanks Chris! Removing the rules gave us a pretty big bump on LB ~0.01 consistently. I'm not sure exactly why other teams didn't experience this. The 0.177 sub seems like it might have been extra lucky too, the submission is version 24 here if you want to take a closer look.
", "rawMarkdown": "Thanks Chris! Removing the rules gave us a pretty big bump on LB ~0.01 consistently. I'm not sure exactly why other teams didn't experience this. The 0.177 sub seems like it might have been extra lucky too, [the submission is version 24 here if you want to take a closer look]( https://www.kaggle.com/robikscube/b0b6-cnn-gru-baseline-stage2-train-inference?scriptVersionId=45499507).", "votes": 1 }, { "id": 1061483, "postDate": "2020-10-27T03:12:09.590Z", "content": "Not sure if its true, but i just read here
\n\n\nI just did a simple experiment. You can get 0.208private score and 0.214 public score with Yeh’s public notebook by taking the post-processing away. So that’s why so many 0.208 in private score.
\n
I'll need to look at that public notebook's post process. Perhaps it isn't the most efficient way to enforce post process.
", "rawMarkdown": "Not sure if its true, but i just read [here][1]\n> I just did a simple experiment. You can get 0.208private score and 0.214 public score with Yeh’s public notebook by taking the post-processing away. So that’s why so many 0.208 in private score.\n\nI'll need to look at that public notebook's post process. Perhaps it isn't the most efficient way to enforce post process.\n\n[1]: https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/193414#1061470", "votes": 1 }, { "id": 1061489, "postDate": "2020-10-27T03:16:06.177Z", "content": "I wonder if my PP (which only affects LB by 0.002) is different? My PP applies the following rules
\nOne and only one of negative, ideterminate, positive
\nRight, left, central
\nRV RL ratio
\nInteresting. Does your submission pass the check_label_consistency function tests?
Yes. I apply my rules above and then it passes the check_label_consistency tests
If you send me your post processing function I could run it on our best sub and see what LB score we get.
", "rawMarkdown": "If you send me your post processing function I could run it on our best sub and see what LB score we get.", "votes": 1 }, { "id": 1062631, "postDate": "2020-10-28T03:51:57.680Z", "content": "I just ran my post process function on the public \"baseline\" notebook.
\nSo something is inefficient with the PP in the notebook.
", "rawMarkdown": "I just ran my post process function on the public \"baseline\" notebook. \n* Without PP, it achieves public LB 0.214 \n* With my PP, it achieves LB 0.215\n* With the PP contained in the notebook it achieves LB 0.233. \n\nSo something is inefficient with the PP in the notebook.", "votes": 1 } ] } ], "comments": [ { "id": 1061498, "author_name": "Vee", "author_url": "", "post_date": "2020-10-27T03:31:39.893000", "content": "Some nice tweaks there Rob and team. Congrats on the nice finish.
", "votes": 3, "replies": [ { "id": 1061504, "author_name": "Rob Mulla", "author_url": "", "post_date": "2020-10-27T03:37:51.923000", "content": "Thanks @watzisname - Congrats to you on a solid finish too!
", "votes": 3, "replies": [] } ] }, { "id": 1063687, "author_name": "Tim Marston", "author_url": "", "post_date": "2020-10-29T08:00:13.807000", "content": "Congratulations! 👍
", "votes": 1, "replies": [] }, { "id": 1063328, "author_name": "Alexey Kachalov", "author_url": "", "post_date": "2020-10-28T18:43:31.100000", "content": "@robikscube This is 28th place solution.😉
", "votes": 1, "replies": [ { "id": 1063405, "author_name": "Eek The Cat", "author_url": "", "post_date": "2020-10-28T21:27:13.500000", "content": "Let's wait till LB's finalized🙂
", "votes": 1, "replies": [] } ] }, { "id": 1061391, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-10-27T01:25:18.207000", "content": "Congrats Rob and team. I'm surprised that you could gain so much LB by not enforcing rules (0.195 up to 0.177). For me, i only gain +- 0.002. Do you know why the difference is so large?
\nThat is a clever trick only predicting private test. I didn't try training a EffB6 thinking that it would never commit in time but it looks like you got a B6 to work. Nice job.
", "votes": 1, "replies": [ { "id": 1061401, "author_name": "Rob Mulla", "author_url": "", "post_date": "2020-10-27T01:44:41.570000", "content": "Thanks Chris! Removing the rules gave us a pretty big bump on LB ~0.01 consistently. I'm not sure exactly why other teams didn't experience this. The 0.177 sub seems like it might have been extra lucky too, the submission is version 24 here if you want to take a closer look.
", "votes": 1, "replies": [] }, { "id": 1061483, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-10-27T03:12:09.590000", "content": "Not sure if its true, but i just read here
\n\n\nI just did a simple experiment. You can get 0.208private score and 0.214 public score with Yeh’s public notebook by taking the post-processing away. So that’s why so many 0.208 in private score.
\n
I'll need to look at that public notebook's post process. Perhaps it isn't the most efficient way to enforce post process.
", "votes": 1, "replies": [] }, { "id": 1061489, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-10-27T03:16:06.177000", "content": "I wonder if my PP (which only affects LB by 0.002) is different? My PP applies the following rules
\nOne and only one of negative, ideterminate, positive
\nRight, left, central
\nRV RL ratio
\nInteresting. Does your submission pass the check_label_consistency function tests?
Yes. I apply my rules above and then it passes the check_label_consistency tests
If you send me your post processing function I could run it on our best sub and see what LB score we get.
", "votes": 1, "replies": [] }, { "id": 1062631, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-10-28T03:51:57.680000", "content": "I just ran my post process function on the public \"baseline\" notebook.
\nSo something is inefficient with the PP in the notebook.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1061353": "I want to thank my amazing team @aerdem4 @drhabib @cateek and @proletheus. This was a very challenging competition and I had a great time collaborating as a team and discussing ideas. Also thanks to the host and winners! I trust the top solutions will positively benefit physicians and those impacted by pulmonary embolism.\n\nEven though our best submission was based off the from the \"baseline\" kernel that was released a week before the deadline, it's a little bittersweet. It's disappointing that such a high scoring kernel was released so late- I know it frustrated many teams (ours included). Our team was working on other end-to-end and 3D based models. I took on the role of improving the baseline as our backup plan.\n\nThis turned into more of an engineering challenge, as we had to balance the limited inference time with such a large amount of data- without exceeding GPU or local memory. It was also challenging because inference time seemed to vary randomly. A submission might complete in 7 hours but then a nearly identical submission would go over 9 hours and fail.\n\nDifferences between our solution and the public kernel:\n- Trained a b6 in place of b0 for the stage 1 models.\n- Changed the inference loop to bag predictions from all 10 (5 folds x 2) stage 1 models.\n- Bagged 5x predictions from stage 2.\n- Changed the code so that it would only predict the private test during inference and used offline calculated public test predictions.\n- Modified the code to only loop through the dataloader once instead of twice for stage 1 predictions.\n- Ahmet used some magic to tweak the stage 2 model to get some added boost, including modifying the loss function.\n\nWhat didn't work:\n- ResNext101 in stage 1 models, although it had a slightly better CV score.\n- Parallel GRU + LSTM for stage 2 model.\n\nOur solution also meets the required criteria and shouldn't have conflicting label predictions. Our best submission ignoring these restrictions would've given us a private LB score of 0.177 but we didn't select it to stay within the rules.\n\nI'd also like to personally thank Z by HP and NVIDIA for providing me the Z8 desktop which I put to good use on this challenge.", "1061498": "Some nice tweaks there Rob and team. Congrats on the nice finish.", "1063687": "Congratulations! 👍", "1063328": "@robikscube This is 28th place solution.😉", "1061391": "Congrats Rob and team. I'm surprised that you could gain so much LB by not enforcing rules (0.195 up to 0.177). For me, i only gain +- 0.002. Do you know why the difference is so large?\n\nThat is a clever trick only predicting private test. I didn't try training a EffB6 thinking that it would never commit in time but it looks like you got a B6 to work. Nice job." } }