Hello to all participants, organizers and people who are here to learn from the successful solutions or to exchange opinions
\n\nThis was an interesting competition with a lot of challenge even for the experimented computer vision programmers. The main challenge was the fact that the provided data was limited and noisy\nI am glad with the result, this being the 3rd consecutive Kaggle medal (after gold at Bengali and bronze at M5).\nI will mention some of the things I tried here and a comment about how that implemented technique worked for me.
\n\nPre-processing techniques\na) Tiles design\n- 256x256x36 tiles (best results)\n- 256x256x49 tiles (a lot of white space and image was too big to sustained with the number of provided training data)\n- 128x128x144 tiles (lower results than the 256x256x36 tiles, probably some patterns are interupting by making the slides smaller)\n- 512x512x9 tiles (also lower results than 256x256x36, the tiles being so big they had a lot of white space when select them)\nAdditional comment: 256x256x36 seemed to be the sweet point between making the tiles too big where they will have a lot of empty pixels and 128x128x144 where the tiles patterns are interupted
\n\nb) Data selection\nDue to noisy labels I design a system to eliminate data which had the biggest probability of being labeled wrong. I used the best 5 fold ensable to predict on training data, average the prediction and eliminate data where the difference between prediction and real label was biggest than a specific threhold\nThe thresholds tested were:\n- 2 (137 data eliminated) \n- 3 (39 data eliminated)\n- 4 (7 data eliminated)
\n\nResults comment: Best CV results were obtain where I eliminated data where abs(prediction-truth labels)>3 where 39 data were eliminated
\n\nModel arhitectures\n- Efficientnet B0\n- Efficientnet B1\n- Efficientnet B2\n- Efficientnet B3\n- Efficientnet B4\n- SE_Resnext50
\n\nResult comment: Best result was on B2 arhitecture, B3-B4 lead to overfit and the rest of them simply did not worked for me
\n\nData Augmentation\n- Transpose\n- Vertical Flip\n- Horizontal Flip\n- RandomRotate90\n- Rotate at a random angle\n- Shift starting position pad when making tiles
\n\nResult comment: All the augmentation were made at 2 levels: tile level + after the tile ensamble\nIn the best results I used: Transpose + Vertical Flip + Horizontal Flip
\n\nOptimizer + scheduler\n- Adam + GradualWarmupScheduler + CosineAnnealingLR\n- RangerLars + GradualWarmupScheduler + CosineAnnealingLR\n- RangerLars + OneCycleLR\n- Adam + GradualWarmupScheduler + ReduceLROnPlateau
\n\nResult comment: Although I tried to use RangerLars in a variety of combination of schedulers, best result was obtain with the good old Adam (Adam + GradualWarmupScheduler + CosineAnnealingLR)
\n\nTTA\nTTA composed of:\n- Transpose\n- Vertical Flip\n- Horizontal Flip\n- Shift starting position pad when making tiles
\n\nResult comment: Usually I recommend to use on TTA similar techniques with what you are using on training augmentation, otherwise the prediction will see images that were not taught by the model. This is why I used:Transpose + Vertical + Horizontal Flip
\n\nCongratulations to all participants and to the organizers !!!
\n\nSee you on the next computer vision competition !
", "messages": [ { "id": 942099, "postDate": "2020-07-23T15:24:27.037Z", "content": "Hello to all participants, organizers and people who are here to learn from the successful solutions or to exchange opinions
\n\nThis was an interesting competition with a lot of challenge even for the experimented computer vision programmers. The main challenge was the fact that the provided data was limited and noisy\nI am glad with the result, this being the 3rd consecutive Kaggle medal (after gold at Bengali and bronze at M5).\nI will mention some of the things I tried here and a comment about how that implemented technique worked for me.
\n\nPre-processing techniques\na) Tiles design\n- 256x256x36 tiles (best results)\n- 256x256x49 tiles (a lot of white space and image was too big to sustained with the number of provided training data)\n- 128x128x144 tiles (lower results than the 256x256x36 tiles, probably some patterns are interupting by making the slides smaller)\n- 512x512x9 tiles (also lower results than 256x256x36, the tiles being so big they had a lot of white space when select them)\nAdditional comment: 256x256x36 seemed to be the sweet point between making the tiles too big where they will have a lot of empty pixels and 128x128x144 where the tiles patterns are interupted
\n\nb) Data selection\nDue to noisy labels I design a system to eliminate data which had the biggest probability of being labeled wrong. I used the best 5 fold ensable to predict on training data, average the prediction and eliminate data where the difference between prediction and real label was biggest than a specific threhold\nThe thresholds tested were:\n- 2 (137 data eliminated) \n- 3 (39 data eliminated)\n- 4 (7 data eliminated)
\n\nResults comment: Best CV results were obtain where I eliminated data where abs(prediction-truth labels)>3 where 39 data were eliminated
\n\nModel arhitectures\n- Efficientnet B0\n- Efficientnet B1\n- Efficientnet B2\n- Efficientnet B3\n- Efficientnet B4\n- SE_Resnext50
\n\nResult comment: Best result was on B2 arhitecture, B3-B4 lead to overfit and the rest of them simply did not worked for me
\n\nData Augmentation\n- Transpose\n- Vertical Flip\n- Horizontal Flip\n- RandomRotate90\n- Rotate at a random angle\n- Shift starting position pad when making tiles
\n\nResult comment: All the augmentation were made at 2 levels: tile level + after the tile ensamble\nIn the best results I used: Transpose + Vertical Flip + Horizontal Flip
\n\nOptimizer + scheduler\n- Adam + GradualWarmupScheduler + CosineAnnealingLR\n- RangerLars + GradualWarmupScheduler + CosineAnnealingLR\n- RangerLars + OneCycleLR\n- Adam + GradualWarmupScheduler + ReduceLROnPlateau
\n\nResult comment: Although I tried to use RangerLars in a variety of combination of schedulers, best result was obtain with the good old Adam (Adam + GradualWarmupScheduler + CosineAnnealingLR)
\n\nTTA\nTTA composed of:\n- Transpose\n- Vertical Flip\n- Horizontal Flip\n- Shift starting position pad when making tiles
\n\nResult comment: Usually I recommend to use on TTA similar techniques with what you are using on training augmentation, otherwise the prediction will see images that were not taught by the model. This is why I used:Transpose + Vertical + Horizontal Flip
\n\nCongratulations to all participants and to the organizers !!!
\n\nSee you on the next computer vision competition !
", "rawMarkdown": "Hello to all participants, organizers and people who are here to learn from the successful solutions or to exchange opinions\n\nThis was an interesting competition with a lot of challenge even for the experimented computer vision programmers. The main challenge was the fact that the provided data was limited and noisy\nI am glad with the result, this being the 3rd consecutive Kaggle medal (after gold at Bengali and bronze at M5).\nI will mention some of the things I tried here and a comment about how that implemented technique worked for me.\n\n**Pre-processing techniques**\na) Tiles design\n- 256x256x36 tiles *(best results)*\n- 256x256x49 tiles *(a lot of white space and image was too big to sustained with the number of provided training data)*\n- 128x128x144 tiles *(lower results than the 256x256x36 tiles, probably some patterns are interupting by making the slides smaller)*\n- 512x512x9 tiles *(also lower results than 256x256x36, the tiles being so big they had a lot of white space when select them)*\nAdditional comment: *256x256x36 seemed to be the sweet point between making the tiles too big where they will have a lot of empty pixels and 128x128x144 where the tiles patterns are interupted*\n\nb) Data selection\nDue to noisy labels I design a system to eliminate data which had the biggest probability of being labeled wrong. I used the best 5 fold ensable to predict on training data, average the prediction and eliminate data where the difference between prediction and real label was biggest than a specific threhold\nThe thresholds tested were:\n- 2 (137 data eliminated) \n- 3 (39 data eliminated)\n- 4 (7 data eliminated)\n\nResults comment: *Best CV results were obtain where I eliminated data where abs(prediction-truth labels)>3 where 39 data were eliminated*\n\n**Model arhitectures**\n- Efficientnet B0\n- Efficientnet B1\n- Efficientnet B2\n- Efficientnet B3\n- Efficientnet B4\n- SE_Resnext50\n\nResult comment: *Best result was on B2 arhitecture, B3-B4 lead to overfit and the rest of them simply did not worked for me*\n\n**Data Augmentation**\n- Transpose\n- Vertical Flip\n- Horizontal Flip\n- RandomRotate90\n- Rotate at a random angle\n- Shift starting position pad when making tiles\n\nResult comment: *All the augmentation were made at 2 levels: tile level + after the tile ensamble\nIn the best results I used: Transpose + Vertical Flip + Horizontal Flip*\n\n**Optimizer + scheduler**\n- Adam + GradualWarmupScheduler + CosineAnnealingLR\n- RangerLars + GradualWarmupScheduler + CosineAnnealingLR\n- RangerLars + OneCycleLR\n- Adam + GradualWarmupScheduler + ReduceLROnPlateau\n\nResult comment: Although I tried to use RangerLars in a variety of combination of schedulers, best result was obtain with the good old Adam (Adam + GradualWarmupScheduler + CosineAnnealingLR)\n\n\n**TTA**\nTTA composed of:\n- Transpose\n- Vertical Flip\n- Horizontal Flip\n- Shift starting position pad when making tiles\n\nResult comment: *Usually I recommend to use on TTA similar techniques with what you are using on training augmentation, otherwise the prediction will see images that were not taught by the model. This is why I used:Transpose + Vertical + Horizontal Flip*\n\n\nCongratulations to all participants and to the organizers !!!\n\nSee you on the next computer vision competition !", "votes": 13 }, { "id": 946783, "postDate": "2020-07-26T20:26:27.433Z", "content": "Congratulations, your solutions always seem to exhibit solid computer vision fundamentals.
", "rawMarkdown": "Congratulations, your solutions always seem to exhibit solid computer vision fundamentals.", "votes": 1, "replies": [ { "id": 947996, "postDate": "2020-07-27T15:45:12.437Z", "content": "Thank you @yousof9 . It is my profession, this is what I do where I work
", "rawMarkdown": "Thank you @yousof9 . It is my profession, this is what I do where I work", "votes": 1 } ] }, { "id": 944230, "postDate": "2020-07-25T00:10:11.480Z", "content": "@vladvdv Thanks for solution sharing and the kind words under my solution topic :D! I want to consult you a question.๐\nIf I use some lr-scheduler, such as onecycle and cosineannealing for training a pre-set value of epoch(assume 30). After ending training, the scheduler already reached the min-lr. But I find the model isn't converge well. Is there any proper ways to reuse these model weights for further training? I try set a small lr(magnitude about 1e-06), but the result isn't so good. Thanks in advance :D
", "rawMarkdown": "@vladvdv Thanks for solution sharing and the kind words under my solution topic :D! I want to consult you a question.๐\nIf I use some lr-scheduler, such as onecycle and cosineannealing for training a pre-set value of epoch(assume 30). After ending training, the scheduler already reached the min-lr. But I find the model isn't converge well. Is there any proper ways to reuse these model weights for further training? I try set a small lr(magnitude about 1e-06), but the result isn't so good. Thanks in advance :D", "votes": 1, "replies": [ { "id": 944638, "postDate": "2020-07-25T08:14:48.737Z", "content": "@cnzengshiyuan . After you trained a model with onecycle or cosineannealing the final lr at the end is very small. If you want to retrain this model after you finish the first training you have 2 possibilities:\n1) Use the same lr like the end of the first training session or even smaller and it should work but it will take some time to see improvements (small lr -> very small steps)\n2) Use a bigger lr but this is very risky. It can drag you out of the local minimum you have found instead of further exploring it and the loss will increase. Another possibility is to increase a little bit the lr and on the retrain come with a cosine anneling, to somehow simulate the CosineAnnealingWarmRestarts
", "rawMarkdown": "@cnzengshiyuan . After you trained a model with onecycle or cosineannealing the final lr at the end is very small. If you want to retrain this model after you finish the first training you have 2 possibilities:\n1) Use the same lr like the end of the first training session or even smaller and it should work but it will take some time to see improvements (small lr -> very small steps)\n2) Use a bigger lr but this is very risky. It can drag you out of the local minimum you have found instead of further exploring it and the loss will increase. Another possibility is to increase a little bit the lr and on the retrain come with a cosine anneling, to somehow simulate the CosineAnnealingWarmRestarts \n", "votes": 1 }, { "id": 944857, "postDate": "2020-07-25T11:49:34.363Z", "content": "@vladvdv , OK, I will try and do some experiments further. Thanks for your advice :D
", "rawMarkdown": "@vladvdv , OK, I will try and do some experiments further. Thanks for your advice :D", "votes": 1 }, { "id": 945214, "postDate": "2020-07-25T16:55:16.713Z", "content": "My pleasure, good luck with the experiments !
", "rawMarkdown": "My pleasure, good luck with the experiments !", "votes": 1 }, { "id": 945639, "postDate": "2020-07-26T03:39:13.760Z", "content": "Thanks ! ๐ค ๐ค ๐ค
", "rawMarkdown": "Thanks ! ๐ค ๐ค ๐ค ", "votes": 1 } ] }, { "id": 942159, "postDate": "2020-07-23T16:03:09.400Z", "content": "Interesting solution - yours is a Silver medal solution, not a bronze one.
\n\nAre you focussing only on Computer vision competitions?
", "rawMarkdown": "Interesting solution - yours is a Silver medal solution, not a bronze one.\n\nAre you focussing only on Computer vision competitions?", "votes": 2, "replies": [ { "id": 942171, "postDate": "2020-07-23T16:08:34.927Z", "content": "@kurianbenoy My bad, I corrected the title now. Thanks for the observation\nComputer vision is my best area of expertise but I sometimes participate on others also if they seem interesting. Any new thing that I can learn is a useful thing
", "rawMarkdown": "@kurianbenoy My bad, I corrected the title now. Thanks for the observation\nComputer vision is my best area of expertise but I sometimes participate on others also if they seem interesting. Any new thing that I can learn is a useful thing", "votes": 2 }, { "id": 942356, "postDate": "2020-07-23T17:53:49.410Z", "content": "Congrats๐
", "rawMarkdown": "Congrats๐ " }, { "id": 942505, "postDate": "2020-07-23T19:30:36.523Z", "content": "Thank you @spidyweb
", "rawMarkdown": "Thank you @spidyweb " } ] } ], "comments": [ { "id": 946783, "author_name": "Yousef Rabi", "author_url": "", "post_date": "2020-07-26T20:26:27.433000", "content": "Congratulations, your solutions always seem to exhibit solid computer vision fundamentals.
", "votes": 1, "replies": [ { "id": 947996, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-07-27T15:45:12.437000", "content": "Thank you @yousof9 . It is my profession, this is what I do where I work
", "votes": 1, "replies": [] } ] }, { "id": 944230, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-07-25T00:10:11.480000", "content": "@vladvdv Thanks for solution sharing and the kind words under my solution topic :D! I want to consult you a question.๐\nIf I use some lr-scheduler, such as onecycle and cosineannealing for training a pre-set value of epoch(assume 30). After ending training, the scheduler already reached the min-lr. But I find the model isn't converge well. Is there any proper ways to reuse these model weights for further training? I try set a small lr(magnitude about 1e-06), but the result isn't so good. Thanks in advance :D
", "votes": 1, "replies": [ { "id": 944638, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-07-25T08:14:48.737000", "content": "@cnzengshiyuan . After you trained a model with onecycle or cosineannealing the final lr at the end is very small. If you want to retrain this model after you finish the first training you have 2 possibilities:\n1) Use the same lr like the end of the first training session or even smaller and it should work but it will take some time to see improvements (small lr -> very small steps)\n2) Use a bigger lr but this is very risky. It can drag you out of the local minimum you have found instead of further exploring it and the loss will increase. Another possibility is to increase a little bit the lr and on the retrain come with a cosine anneling, to somehow simulate the CosineAnnealingWarmRestarts
", "votes": 1, "replies": [] }, { "id": 944857, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-07-25T11:49:34.363000", "content": "@vladvdv , OK, I will try and do some experiments further. Thanks for your advice :D
", "votes": 1, "replies": [] }, { "id": 945214, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-07-25T16:55:16.713000", "content": "My pleasure, good luck with the experiments !
", "votes": 1, "replies": [] }, { "id": 945639, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-07-26T03:39:13.760000", "content": "Thanks ! ๐ค ๐ค ๐ค
", "votes": 1, "replies": [] } ] }, { "id": 942159, "author_name": "Kurian Benoy", "author_url": "", "post_date": "2020-07-23T16:03:09.400000", "content": "Interesting solution - yours is a Silver medal solution, not a bronze one.
\n\nAre you focussing only on Computer vision competitions?
", "votes": 2, "replies": [ { "id": 942171, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-07-23T16:08:34.927000", "content": "@kurianbenoy My bad, I corrected the title now. Thanks for the observation\nComputer vision is my best area of expertise but I sometimes participate on others also if they seem interesting. Any new thing that I can learn is a useful thing
", "votes": 2, "replies": [] }, { "id": 942356, "author_name": "Rajeev", "author_url": "", "post_date": "2020-07-23T17:53:49.410000", "content": "Congrats๐
", "votes": 0, "replies": [] }, { "id": 942505, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-07-23T19:30:36.523000", "content": "Thank you @spidyweb
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "942099": "Hello to all participants, organizers and people who are here to learn from the successful solutions or to exchange opinions\n\nThis was an interesting competition with a lot of challenge even for the experimented computer vision programmers. The main challenge was the fact that the provided data was limited and noisy\nI am glad with the result, this being the 3rd consecutive Kaggle medal (after gold at Bengali and bronze at M5).\nI will mention some of the things I tried here and a comment about how that implemented technique worked for me.\n\n**Pre-processing techniques**\na) Tiles design\n- 256x256x36 tiles *(best results)*\n- 256x256x49 tiles *(a lot of white space and image was too big to sustained with the number of provided training data)*\n- 128x128x144 tiles *(lower results than the 256x256x36 tiles, probably some patterns are interupting by making the slides smaller)*\n- 512x512x9 tiles *(also lower results than 256x256x36, the tiles being so big they had a lot of white space when select them)*\nAdditional comment: *256x256x36 seemed to be the sweet point between making the tiles too big where they will have a lot of empty pixels and 128x128x144 where the tiles patterns are interupted*\n\nb) Data selection\nDue to noisy labels I design a system to eliminate data which had the biggest probability of being labeled wrong. I used the best 5 fold ensable to predict on training data, average the prediction and eliminate data where the difference between prediction and real label was biggest than a specific threhold\nThe thresholds tested were:\n- 2 (137 data eliminated) \n- 3 (39 data eliminated)\n- 4 (7 data eliminated)\n\nResults comment: *Best CV results were obtain where I eliminated data where abs(prediction-truth labels)>3 where 39 data were eliminated*\n\n**Model arhitectures**\n- Efficientnet B0\n- Efficientnet B1\n- Efficientnet B2\n- Efficientnet B3\n- Efficientnet B4\n- SE_Resnext50\n\nResult comment: *Best result was on B2 arhitecture, B3-B4 lead to overfit and the rest of them simply did not worked for me*\n\n**Data Augmentation**\n- Transpose\n- Vertical Flip\n- Horizontal Flip\n- RandomRotate90\n- Rotate at a random angle\n- Shift starting position pad when making tiles\n\nResult comment: *All the augmentation were made at 2 levels: tile level + after the tile ensamble\nIn the best results I used: Transpose + Vertical Flip + Horizontal Flip*\n\n**Optimizer + scheduler**\n- Adam + GradualWarmupScheduler + CosineAnnealingLR\n- RangerLars + GradualWarmupScheduler + CosineAnnealingLR\n- RangerLars + OneCycleLR\n- Adam + GradualWarmupScheduler + ReduceLROnPlateau\n\nResult comment: Although I tried to use RangerLars in a variety of combination of schedulers, best result was obtain with the good old Adam (Adam + GradualWarmupScheduler + CosineAnnealingLR)\n\n\n**TTA**\nTTA composed of:\n- Transpose\n- Vertical Flip\n- Horizontal Flip\n- Shift starting position pad when making tiles\n\nResult comment: *Usually I recommend to use on TTA similar techniques with what you are using on training augmentation, otherwise the prediction will see images that were not taught by the model. This is why I used:Transpose + Vertical + Horizontal Flip*\n\n\nCongratulations to all participants and to the organizers !!!\n\nSee you on the next computer vision competition !", "946783": "Congratulations, your solutions always seem to exhibit solid computer vision fundamentals.", "944230": "@vladvdv Thanks for solution sharing and the kind words under my solution topic :D! I want to consult you a question.๐\nIf I use some lr-scheduler, such as onecycle and cosineannealing for training a pre-set value of epoch(assume 30). After ending training, the scheduler already reached the min-lr. But I find the model isn't converge well. Is there any proper ways to reuse these model weights for further training? I try set a small lr(magnitude about 1e-06), but the result isn't so good. Thanks in advance :D", "942159": "Interesting solution - yours is a Silver medal solution, not a bronze one.\n\nAre you focussing only on Computer vision competitions?" } }