![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3149580%2Fb58bb38f6702dc75e667348cd0b90857%2FRUSH.jpg?generation=1588124257762795&alt=media\")
Hi everybody \nThese days I was excited to see another computer vision competition on Kaggle.\nFirst I would like to thank the organizers for give us the opportunity to test our computer vision skills again in a very interesting and useful competition\nSo, let's get into it.\nMy firsts experiments were with 2 model architectures: densenet121 and seresnet50
\n\nInitial conclusions:\nAugmentation\nWe have a wide range of usefull augmentation methods for these types of images, I started with: \n- Noise augmentation (GaussNoise, MultiplicativeNoise)\n- Affine Transforms (scale, rot angle, shear angle, translations and flipping horizontal and vertical)\n- CoarseDropout (max_holes=4, max_height=30, max_width=30)\nFor now the results were best with the following probabilities for augmenting class ( general_noise_aug_prob =0.1, affine_transform_prob=0.3, coarse_dropout_prob=0.2, flip_prob= 0.2)\nTODO next:\n- augment with mixup (in this context I don't think that cutmix will be useful, you can cut an irrelevant part of the image and put it on another, so the model would not see what makes that image a 3 or a 4 if you cut the part that looks like 1 and put it next to a labeled 1 image)\n- try a more hardcore augmentation probabilities combined with more training epochs
\n\nImage input size\n- Current resolution is 512x512. From my tests it is clear that improving the resolution helps a lot, the tests made on 300x300 images or lower showed a incapacity of the model to learn as it does on 512x512
\n\nTODO next:\n- try bigger resolution\n- use the segmentation masks to make a smart crop and use only useful region
\n\nModel architectures\n- For now, it seems that seresnet50 is a little bit better than densenet121 tested in the same condition.
\n\nTODO next:\n- Benchmark: a larger densenet like densenet161 and also se_resnext101_32x4d
\n\nOther aspects:\nWhen you use a large architecture like the ones mentioned before combined with a large image size there are 2 aspects that are not so great : \n- time use for training (especially that after 50 epochs the model still hasn't reach it's peak) doesn't allow us too much experimentation options\n- smaller batch size to fit in the RAM of the GPU. Using a good GPU like 2080TI I still have to use batch sizes like 4 of 6 in the mentioned conditions. Although batch size is a controversial aspect with a lot and pro and cons, making the optimizer step at every 4-6 images in this context can be a little bit to jumpy for my taste
\n\nI have made public my inference kernel (Inference Kernel) that can be used as a template very easy for anybody. You can easily add different models each with their own folds and use custom weights for blending them. Also, a TTA flag is enable for whoever wants to experiment.\nI will keep you up to date with my experiments and if I have time I will make public my training kernel where you can see other tricks that I used
\n\nGood luck to everybody
\n\nLater edit: The public training kernel in online now\nThis base allows you to
\n\nThere are a lot of knobs to tweek in order to personalize it:
\n\nhttps://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds
", "messages": [ { "id": 822242, "postDate": "2020-04-26T19:23:08.090Z", "content": "Hi everybody \nThese days I was excited to see another computer vision competition on Kaggle.\nFirst I would like to thank the organizers for give us the opportunity to test our computer vision skills again in a very interesting and useful competition\nSo, let's get into it.\nMy firsts experiments were with 2 model architectures: densenet121 and seresnet50
\n\nInitial conclusions:\nAugmentation\nWe have a wide range of usefull augmentation methods for these types of images, I started with: \n- Noise augmentation (GaussNoise, MultiplicativeNoise)\n- Affine Transforms (scale, rot angle, shear angle, translations and flipping horizontal and vertical)\n- CoarseDropout (max_holes=4, max_height=30, max_width=30)\nFor now the results were best with the following probabilities for augmenting class ( general_noise_aug_prob =0.1, affine_transform_prob=0.3, coarse_dropout_prob=0.2, flip_prob= 0.2)\nTODO next:\n- augment with mixup (in this context I don't think that cutmix will be useful, you can cut an irrelevant part of the image and put it on another, so the model would not see what makes that image a 3 or a 4 if you cut the part that looks like 1 and put it next to a labeled 1 image)\n- try a more hardcore augmentation probabilities combined with more training epochs
\n\nImage input size\n- Current resolution is 512x512. From my tests it is clear that improving the resolution helps a lot, the tests made on 300x300 images or lower showed a incapacity of the model to learn as it does on 512x512
\n\nTODO next:\n- try bigger resolution\n- use the segmentation masks to make a smart crop and use only useful region
\n\nModel architectures\n- For now, it seems that seresnet50 is a little bit better than densenet121 tested in the same condition.
\n\nTODO next:\n- Benchmark: a larger densenet like densenet161 and also se_resnext101_32x4d
\n\nOther aspects:\nWhen you use a large architecture like the ones mentioned before combined with a large image size there are 2 aspects that are not so great : \n- time use for training (especially that after 50 epochs the model still hasn't reach it's peak) doesn't allow us too much experimentation options\n- smaller batch size to fit in the RAM of the GPU. Using a good GPU like 2080TI I still have to use batch sizes like 4 of 6 in the mentioned conditions. Although batch size is a controversial aspect with a lot and pro and cons, making the optimizer step at every 4-6 images in this context can be a little bit to jumpy for my taste
\n\nI have made public my inference kernel (Inference Kernel) that can be used as a template very easy for anybody. You can easily add different models each with their own folds and use custom weights for blending them. Also, a TTA flag is enable for whoever wants to experiment.\nI will keep you up to date with my experiments and if I have time I will make public my training kernel where you can see other tricks that I used
\n\nGood luck to everybody
\n\nLater edit: The public training kernel in online now\nThis base allows you to
\n\nThere are a lot of knobs to tweek in order to personalize it:
\n\nhttps://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds
", "rawMarkdown": "Hi everybody \nThese days I was excited to see another computer vision competition on Kaggle.\nFirst I would like to thank the organizers for give us the opportunity to test our computer vision skills again in a very interesting and useful competition\nSo, let's get into it.\nMy firsts experiments were with 2 model architectures: densenet121 and seresnet50\n\nInitial conclusions:\n**Augmentation**\nWe have a wide range of usefull augmentation methods for these types of images, I started with: \n- Noise augmentation (GaussNoise, MultiplicativeNoise)\n- Affine Transforms (scale, rot angle, shear angle, translations and flipping horizontal and vertical)\n- CoarseDropout (max_holes=4, max_height=30, max_width=30)\nFor now the results were best with the following probabilities for augmenting class ( general_noise_aug_prob =0.1, affine_transform_prob=0.3, coarse_dropout_prob=0.2, flip_prob= 0.2)\nTODO next:\n- augment with mixup (in this context I don't think that cutmix will be useful, you can cut an irrelevant part of the image and put it on another, so the model would not see what makes that image a 3 or a 4 if you cut the part that looks like 1 and put it next to a labeled 1 image)\n- try a more hardcore augmentation probabilities combined with more training epochs\n\n\n**Image input size**\n- Current resolution is 512x512. From my tests it is clear that improving the resolution helps a lot, the tests made on 300x300 images or lower showed a incapacity of the model to learn as it does on 512x512 \n\nTODO next:\n- try bigger resolution\n- use the segmentation masks to make a smart crop and use only useful region\n\n\n**Model architectures**\n- For now, it seems that seresnet50 is a little bit better than densenet121 tested in the same condition.\n\nTODO next:\n- Benchmark: a larger densenet like densenet161 and also se_resnext101_32x4d\n\n\n**Other aspects:**\nWhen you use a large architecture like the ones mentioned before combined with a large image size there are 2 aspects that are not so great : \n- time use for training (especially that after 50 epochs the model still hasn't reach it's peak) doesn't allow us too much experimentation options\n- smaller batch size to fit in the RAM of the GPU. Using a good GPU like 2080TI I still have to use batch sizes like 4 of 6 in the mentioned conditions. Although batch size is a controversial aspect with a lot and pro and cons, making the optimizer step at every 4-6 images in this context can be a little bit to jumpy for my taste \n\nI have made public my inference kernel ([Inference Kernel](https://www.kaggle.com/vladvdv/pytorch-inference-multiple-models-and-folds)) that can be used as a template very easy for anybody. You can easily add different models each with their own folds and use custom weights for blending them. Also, a TTA flag is enable for whoever wants to experiment.\nI will keep you up to date with my experiments and if I have time I will make public my training kernel where you can see other tricks that I used\n\nGood luck to everybody \n\n**Later edit**: The public training kernel in online now\nThis base allows you to\n\n- understand how pytorch and computer vision works\n- be competitive in the competition\n- gain experience by trying easy to implement different customizations\n\nThere are a lot of knobs to tweek in order to personalize it:\n\n- a lot of augmentation techniques\n- choose image size\n- choose the desired pretrained model\n- customize the arhitecture by add more dense layers or any other layer types\n- change number of folds or the spliting\n\nhttps://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds", "votes": 26 }, { "id": 824805, "postDate": "2020-04-28T15:57:51.547Z", "content": "This is fantastic. Thanks for always sharing your insights and your work. It is an inspiration.
", "rawMarkdown": "This is fantastic. Thanks for always sharing your insights and your work. It is an inspiration.", "votes": 1, "replies": [ { "id": 824823, "postDate": "2020-04-28T16:07:32.843Z", "content": "I am glad that I can help. Good luck !
", "rawMarkdown": "I am glad that I can help. Good luck ! " } ] }, { "id": 823630, "postDate": "2020-04-27T19:12:49.103Z", "content": "Hey thanks for all this info much appreciated! May i ask how many epochs are you training for? For me when im training longer im getting a higher CV and validation loss is still decreasing but this is resulting in a lower LB
", "rawMarkdown": "Hey thanks for all this info much appreciated! May i ask how many epochs are you training for? For me when im training longer im getting a higher CV and validation loss is still decreasing but this is resulting in a lower LB", "votes": 1, "replies": [ { "id": 824419, "postDate": "2020-04-28T11:00:05.780Z", "content": "Hi @yannmajewski . I am training somewhere between 70-100 epochs, depending on the augmentation that I am designing.\nHow is your QWK on validation ? still increasing of plateauing ?
", "rawMarkdown": "Hi @yannmajewski . I am training somewhere between 70-100 epochs, depending on the augmentation that I am designing.\nHow is your QWK on validation ? still increasing of plateauing ?" } ] }, { "id": 823443, "postDate": "2020-04-27T16:35:46.257Z", "content": "Hi, @vladvdv thanks a lot for these tips, they are great especially for people like me that are learning about image segmentation and this type of domain. \nI was wondering when will you post your training code.
\n\nThanks again.
", "rawMarkdown": "Hi, @vladvdv thanks a lot for these tips, they are great especially for people like me that are learning about image segmentation and this type of domain. \nI was wondering when will you post your training code.\n\nThanks again.", "votes": 1, "replies": [ { "id": 824416, "postDate": "2020-04-28T10:58:19.877Z", "content": "@oscarrangel Now It's online and public
", "rawMarkdown": "@oscarrangel Now It's online and public" }, { "id": 824442, "postDate": "2020-04-28T11:12:06.373Z", "content": "Thanks you bro
", "rawMarkdown": "Thanks you bro" } ] }, { "id": 822751, "postDate": "2020-04-27T05:31:44.187Z", "content": "@vladvdv May I ask if your image augmentations are performed on the training images or the masks as well? Thanks and great post, always looking out for your inputs.
", "rawMarkdown": "@vladvdv May I ask if your image augmentations are performed on the training images or the masks as well? Thanks and great post, always looking out for your inputs.", "votes": 1, "replies": [ { "id": 822958, "postDate": "2020-04-27T09:29:27.440Z", "content": "Hi @reighns ,\nThe augmentation mentioned in the post are performed just on the train images. For now I am not using the masks, I am just performing a classification based on the images
", "rawMarkdown": "Hi @reighns ,\nThe augmentation mentioned in the post are performed just on the train images. For now I am not using the masks, I am just performing a classification based on the images" } ] }, { "id": 822421, "postDate": "2020-04-26T22:18:03.973Z", "content": "Thanks for sharing this post!
", "rawMarkdown": "Thanks for sharing this post!", "votes": 1, "replies": [ { "id": 822961, "postDate": "2020-04-27T09:30:02.870Z", "content": "It is my pleasure @mpanfil . I hope people will gain good information from it and also share any good ideas
", "rawMarkdown": "It is my pleasure @mpanfil . I hope people will gain good information from it and also share any good ideas " } ] }, { "id": 822312, "postDate": "2020-04-26T20:22:44.687Z", "content": "Awesome post, thanks for sharing your results so far
", "rawMarkdown": "Awesome post, thanks for sharing your results so far", "votes": 1, "replies": [ { "id": 822357, "postDate": "2020-04-26T20:46:37.143Z", "content": "Thank you for the kind words
", "rawMarkdown": "Thank you for the kind words" } ] }, { "id": 822307, "postDate": "2020-04-26T20:20:05.270Z", "content": "Glad to see you in another CV comp! Can't wait to learn a lot from you! Your sharing is always appreciated.
\n\nI have been doing similar experiments with seresnext50 however the model keeps overfitting. Have you ran into this issue? If so how have you gone about fixing it?
", "rawMarkdown": "Glad to see you in another CV comp! Can't wait to learn a lot from you! Your sharing is always appreciated.\n\nI have been doing similar experiments with seresnext50 however the model keeps overfitting. Have you ran into this issue? If so how have you gone about fixing it?", "votes": 1, "replies": [ { "id": 822356, "postDate": "2020-04-26T20:46:13.310Z", "content": "Glad to see you here @greatgamedota . I appreciate your kind words and I hope Panda Challenge will be a successful competition for both of us
\n\nIf you are overfitting you have several options you can use:\nFirst, I would try using harder image augmenting so your model can generalize better. What did you use so far ?\nAlso, do you use multiple dense layers after the output of the seresnext50 ? If so, try to eliminate most of them, they are making your architecture bigger than your data can support it.
", "rawMarkdown": "Glad to see you here @greatgamedota . I appreciate your kind words and I hope Panda Challenge will be a successful competition for both of us\n\nIf you are overfitting you have several options you can use:\nFirst, I would try using harder image augmenting so your model can generalize better. What did you use so far ?\nAlso, do you use multiple dense layers after the output of the seresnext50 ? If so, try to eliminate most of them, they are making your architecture bigger than your data can support it.", "votes": 2 }, { "id": 822362, "postDate": "2020-04-26T20:51:17.473Z", "content": "Likewise! I tried harder augmentation like mixup and cutmix and got 5fold CV of .684 however only .48 on LB. Those same models also had 2 dense layers which may have undid the augmentation, I will continue experimenting thanks! I will also try the augmentations you recommended.
", "rawMarkdown": "Likewise! I tried harder augmentation like mixup and cutmix and got 5fold CV of .684 however only .48 on LB. Those same models also had 2 dense layers which may have undid the augmentation, I will continue experimenting thanks! I will also try the augmentations you recommended." }, { "id": 823469, "postDate": "2020-04-27T16:57:51.553Z", "content": "Thanks to @vladvdv I started to use 512x512 and the model is generalizing better, now I am going to try the augmentation that he is recommending, thanks for all the tips, bro @vladvdv
", "rawMarkdown": "Thanks to @vladvdv I started to use 512x512 and the model is generalizing better, now I am going to try the augmentation that he is recommending, thanks for all the tips, bro @vladvdv ", "votes": 1 }, { "id": 824423, "postDate": "2020-04-28T11:00:54.790Z", "content": "@oscarrangel It is my pleasure, good luck !
", "rawMarkdown": "@oscarrangel It is my pleasure, good luck !" } ] }, { "id": 824385, "postDate": "2020-04-28T10:35:36.973Z", "content": "I made a public the training kernel.\nThis is a solid starting point for any kaggler in this competition. This base allows you to
\n\nThere are a lot of knobs to tweek in order to personalize it:
\n\nhttps://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds
\n\nHave fun !
", "rawMarkdown": "I made a public the training kernel.\nThis is a solid starting point for any kaggler in this competition. This base allows you to\n\n- understand how pytorch and computer vision works\n- be competitive in the competition\n- gain experience by trying easy to implement different customizations\n\nThere are a lot of knobs to tweek in order to personalize it:\n\n- a lot of augmentation techniques\n- choose image size\n- choose the desired pretrained model\n- customize the arhitecture by add more dense layers or any other layer types\n- change number of folds or the spliting\n\n[https://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds](https://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds)\n\nHave fun ! ", "votes": 2 }, { "id": 825409, "postDate": "2020-04-29T01:38:36.173Z", "content": "
Happy to see you here ;)\nWhat'more, I wanna ask you a question.\nI noticed the model you used in the kernel will cost almost 30mins(train+valid), but I use fp16 to train my efficientnet-b0(I wanna use small model to do more experiments) will cost 14mins(train+valid) and if I use full-precision to train will cost also 14mins(train+valid), it makes me confused... However, here is iafoss use resnext-32x4d-ssl use fp16 to train just cost ~7mins(train+valid). Is my model normal? My arch is: efficientb0-pool-drop-lin(1 layer), bs=32, resolution=64x64\nThanks in advance 😄
", "rawMarkdown": "\n\nHappy to see you here ;)\nWhat'more, I wanna ask you a question.\nI noticed the model you used in the kernel will cost almost 30mins(train+valid), but I use fp16 to train my efficientnet-b0(I wanna use small model to do more experiments) will cost 14mins(train+valid) and if I use full-precision to train will cost also 14mins(train+valid), it makes me confused... However, [here is iafoss use resnext-32x4d-ssl](https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb/notebook) use fp16 to train just cost ~7mins(train+valid). Is my model normal? My arch is: efficientb0-pool-drop-lin(1 layer), bs=32, resolution=64x64\nThanks in advance 😄 ", "replies": [ { "id": 825806, "postDate": "2020-04-29T09:01:27.873Z", "content": "Hi @cnzengshiyuan ,\nWell normally on fp16 you should have about half of the normal training time, but this is only pure training, another time and resources consuming part is the data preprocessing, in our case the augmentation, the augmentation part time and processing needs remains the same if you are using fp32 or fp16, so the real scalability is not just cut in half the time, if a fp32 takes 20 min, a fp16 can take 15-16 mins if you have hardcore augmentation. But in your case, if it takes exactly the same time it is something wrong somewhere, be sure you are not forgetting something in the code
", "rawMarkdown": "Hi @cnzengshiyuan ,\nWell normally on fp16 you should have about half of the normal training time, but this is only pure training, another time and resources consuming part is the data preprocessing, in our case the augmentation, the augmentation part time and processing needs remains the same if you are using fp32 or fp16, so the real scalability is not just cut in half the time, if a fp32 takes 20 min, a fp16 can take 15-16 mins if you have hardcore augmentation. But in your case, if it takes exactly the same time it is something wrong somewhere, be sure you are not forgetting something in the code ", "votes": 2 }, { "id": 825879, "postDate": "2020-04-29T09:51:51.567Z", "content": "Thank you! Heard what you said, I more convinced that the time-costing is actually abnormal, I will check it carefully.(still debuging :D)
\n\nUPDATE:\nI think I had solved the time-costing problem. But I'm not sure whether this way is the cause or not. But the part I changed is only what I show below.
\n\nChange\n\nimg_info = openslide.OpenSlide(img_path) # downsample 1, 4, 16\nimg = img_info.read_region((0,0), img_info.level_count-1, img_info.level_dimensions[-1]).convert('RGB')\n\nTo\n\nimg = skimage.io.MultiImage(img_path)[-1]\n\nThe morning I use the openslide as I wrote and ~14min/epoch. And I think I may try the skimage.io and ~3.5min/epoch. I noticed there is a kernel to compare them, but I didn't realize the gap will so huge...\nModel is the same as I mentioned above(efficientnetb0-pool-dropout-lin)
The only augmentation I would suggest right now will be random rotate, it makes the model go on learning without overfitting and gets better result, the rest of the augmentations have to be tried and differentiate on data preprocessing and pipeline of the model.
", "rawMarkdown": "The only augmentation I would suggest right now will be random rotate, it makes the model go on learning without overfitting and gets better result, the rest of the augmentations have to be tried and differentiate on data preprocessing and pipeline of the model." }, { "id": 823150, "postDate": "2020-04-27T12:58:30.920Z", "content": "Greatly appreciate this overview!
\n\nI am new to this type of data, but most of the research I have reviewed speaks to the importance of sampling from the WSI's. It seems like from your work you opted for full images just lower resolution. Would you mind speaking to how you made this choice? Trying to expand my understadning of this domain. Thanks!
", "rawMarkdown": "Greatly appreciate this overview! \n\nI am new to this type of data, but most of the research I have reviewed speaks to the importance of sampling from the WSI's. It seems like from your work you opted for full images just lower resolution. Would you mind speaking to how you made this choice? Trying to expand my understadning of this domain. Thanks!", "replies": [ { "id": 823202, "postDate": "2020-04-27T13:39:50.420Z", "content": "From what I see, there are 3 options to use a image with the data provided:\na) use a resized image of a full image\nb) use the mask where you have it and crop with the help of the masks the interested region\nc) eliminate the white space and use only the colored region
\n\nEach of the options above has advantagesand and disadvantages.\n- version a can lose a lot of details in the process of resizing\n- version b learns the algorithm to see high quality and zoom on the interesting region but on test set we do not have any mask that we can use, so it will see another type of image\n- version c sounds perfect on paper but on real life can lead to problems related to aspect ratio differences from image to image when cropping then resize, is somehow discutable and need to be tested.\nI started with a and my next test will be on c
", "rawMarkdown": "From what I see, there are 3 options to use a image with the data provided:\na) use a resized image of a full image\nb) use the mask where you have it and crop with the help of the masks the interested region\nc) eliminate the white space and use only the colored region\n\nEach of the options above has advantagesand and disadvantages.\n- version a can lose a lot of details in the process of resizing\n- version b learns the algorithm to see high quality and zoom on the interesting region but on test set we do not have any mask that we can use, so it will see another type of image\n- version c sounds perfect on paper but on real life can lead to problems related to aspect ratio differences from image to image when cropping then resize, is somehow discutable and need to be tested.\nI started with a and my next test will be on c", "votes": 2 }, { "id": 823210, "postDate": "2020-04-27T13:44:40.943Z", "content": "Awesome thanks!
", "rawMarkdown": "Awesome thanks!" } ] }, { "id": 823012, "postDate": "2020-04-27T10:26:27.303Z", "content": "Thanks @vladvdv . I have a question. Suppose I try augmentations A, B, and C individually (not Compose) on images with supposing resolution 128x128 and I get the CV order as \"using only A < using only B < using only C\", will this CV order remain same after I increase the resolution of images suppose to 512x512?
", "rawMarkdown": "Thanks @vladvdv . I have a question. Suppose I try augmentations A, B, and C individually (not Compose) on images with supposing resolution 128x128 and I get the CV order as \"using only A < using only B < using only C\", will this CV order remain same after I increase the resolution of images suppose to 512x512? ", "replies": [ { "id": 823053, "postDate": "2020-04-27T11:17:10.633Z", "content": "This is a very good question. From my experience it depends on the augmentation techniques used. For example, if you use CoarseDropout (maxholes=4, maxheight=10, maxwidth=10) at 128x128 resolution it will not have the same effect on 512x512 resolution, holes will be much smaller compared with the image and will not help so much on generalizing, so you will have to increase the maxheight and maxwidth proportional to the scaling factor if you want to have a similar impact. \nAt blurring augmentation, when you are having smaller resolutions the edges are not so sharp and you have to use a smaller blur factor so you won't blur to much and the model would not understand nothing from the image, but on higher resolution you have sharper edges, your blur coefficient can be a little higher. At affine transforms, the rotation, translation and flips generally keeps the same effect on different image scale.\nSo, when you increase the resolution, some augmenting techniques parameters needs to be modified in order to have the same effect, if you modify them accordingly, there are big changes that the proportion of importance in the techniques ranking will be kept
", "rawMarkdown": "This is a very good question. From my experience it depends on the augmentation techniques used. For example, if you use CoarseDropout (maxholes=4, maxheight=10, maxwidth=10) at 128x128 resolution it will not have the same effect on 512x512 resolution, holes will be much smaller compared with the image and will not help so much on generalizing, so you will have to increase the maxheight and maxwidth proportional to the scaling factor if you want to have a similar impact. \nAt blurring augmentation, when you are having smaller resolutions the edges are not so sharp and you have to use a smaller blur factor so you won't blur to much and the model would not understand nothing from the image, but on higher resolution you have sharper edges, your blur coefficient can be a little higher. At affine transforms, the rotation, translation and flips generally keeps the same effect on different image scale.\nSo, when you increase the resolution, some augmenting techniques parameters needs to be modified in order to have the same effect, if you modify them accordingly, there are big changes that the proportion of importance in the techniques ranking will be kept ", "votes": 3 }, { "id": 823080, "postDate": "2020-04-27T11:36:52.100Z", "content": "Wow, great answer. Thanks for the quick response as well. All the best.
", "rawMarkdown": "Wow, great answer. Thanks for the quick response as well. All the best. ", "votes": 1 }, { "id": 825808, "postDate": "2020-04-29T09:01:52.047Z", "content": "I am glad I can help. All the best !
", "rawMarkdown": "I am glad I can help. All the best !" } ] } ], "comments": [ { "id": 824805, "author_name": "Tom M", "author_url": "", "post_date": "2020-04-28T15:57:51.547000", "content": "This is fantastic. Thanks for always sharing your insights and your work. It is an inspiration.
", "votes": 1, "replies": [ { "id": 824823, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-28T16:07:32.843000", "content": "I am glad that I can help. Good luck !
", "votes": 0, "replies": [] } ] }, { "id": 823630, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-04-27T19:12:49.103000", "content": "Hey thanks for all this info much appreciated! May i ask how many epochs are you training for? For me when im training longer im getting a higher CV and validation loss is still decreasing but this is resulting in a lower LB
", "votes": 1, "replies": [ { "id": 824419, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-28T11:00:05.780000", "content": "Hi @yannmajewski . I am training somewhere between 70-100 epochs, depending on the augmentation that I am designing.\nHow is your QWK on validation ? still increasing of plateauing ?
", "votes": 0, "replies": [] } ] }, { "id": 823443, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-04-27T16:35:46.257000", "content": "Hi, @vladvdv thanks a lot for these tips, they are great especially for people like me that are learning about image segmentation and this type of domain. \nI was wondering when will you post your training code.
\n\nThanks again.
", "votes": 1, "replies": [ { "id": 824416, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-28T10:58:19.877000", "content": "@oscarrangel Now It's online and public
", "votes": 0, "replies": [] }, { "id": 824442, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-04-28T11:12:06.373000", "content": "Thanks you bro
", "votes": 0, "replies": [] } ] }, { "id": 822751, "author_name": "gao-hongnan", "author_url": "", "post_date": "2020-04-27T05:31:44.187000", "content": "@vladvdv May I ask if your image augmentations are performed on the training images or the masks as well? Thanks and great post, always looking out for your inputs.
", "votes": 1, "replies": [ { "id": 822958, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-27T09:29:27.440000", "content": "Hi @reighns ,\nThe augmentation mentioned in the post are performed just on the train images. For now I am not using the masks, I am just performing a classification based on the images
", "votes": 0, "replies": [] } ] }, { "id": 822421, "author_name": "Monika Panfil", "author_url": "", "post_date": "2020-04-26T22:18:03.973000", "content": "Thanks for sharing this post!
", "votes": 1, "replies": [ { "id": 822961, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-27T09:30:02.870000", "content": "It is my pleasure @mpanfil . I hope people will gain good information from it and also share any good ideas
", "votes": 0, "replies": [] } ] }, { "id": 822312, "author_name": "Matt", "author_url": "", "post_date": "2020-04-26T20:22:44.687000", "content": "Awesome post, thanks for sharing your results so far
", "votes": 1, "replies": [ { "id": 822357, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-26T20:46:37.143000", "content": "Thank you for the kind words
", "votes": 0, "replies": [] } ] }, { "id": 822307, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-04-26T20:20:05.270000", "content": "Glad to see you in another CV comp! Can't wait to learn a lot from you! Your sharing is always appreciated.
\n\nI have been doing similar experiments with seresnext50 however the model keeps overfitting. Have you ran into this issue? If so how have you gone about fixing it?
", "votes": 1, "replies": [ { "id": 822356, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-26T20:46:13.310000", "content": "Glad to see you here @greatgamedota . I appreciate your kind words and I hope Panda Challenge will be a successful competition for both of us
\n\nIf you are overfitting you have several options you can use:\nFirst, I would try using harder image augmenting so your model can generalize better. What did you use so far ?\nAlso, do you use multiple dense layers after the output of the seresnext50 ? If so, try to eliminate most of them, they are making your architecture bigger than your data can support it.
", "votes": 2, "replies": [] }, { "id": 822362, "author_name": "GreatGameDota", "author_url": "", "post_date": "2020-04-26T20:51:17.473000", "content": "Likewise! I tried harder augmentation like mixup and cutmix and got 5fold CV of .684 however only .48 on LB. Those same models also had 2 dense layers which may have undid the augmentation, I will continue experimenting thanks! I will also try the augmentations you recommended.
", "votes": 0, "replies": [] }, { "id": 823469, "author_name": "TheStoneMX", "author_url": "", "post_date": "2020-04-27T16:57:51.553000", "content": "Thanks to @vladvdv I started to use 512x512 and the model is generalizing better, now I am going to try the augmentation that he is recommending, thanks for all the tips, bro @vladvdv
", "votes": 1, "replies": [] }, { "id": 824423, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-28T11:00:54.790000", "content": "@oscarrangel It is my pleasure, good luck !
", "votes": 0, "replies": [] } ] }, { "id": 824385, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-28T10:35:36.973000", "content": "I made a public the training kernel.\nThis is a solid starting point for any kaggler in this competition. This base allows you to
\n\nThere are a lot of knobs to tweek in order to personalize it:
\n\nhttps://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds
\n\nHave fun !
", "votes": 2, "replies": [] }, { "id": 825409, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-04-29T01:38:36.173000", "content": "
Happy to see you here ;)\nWhat'more, I wanna ask you a question.\nI noticed the model you used in the kernel will cost almost 30mins(train+valid), but I use fp16 to train my efficientnet-b0(I wanna use small model to do more experiments) will cost 14mins(train+valid) and if I use full-precision to train will cost also 14mins(train+valid), it makes me confused... However, here is iafoss use resnext-32x4d-ssl use fp16 to train just cost ~7mins(train+valid). Is my model normal? My arch is: efficientb0-pool-drop-lin(1 layer), bs=32, resolution=64x64\nThanks in advance 😄
", "votes": 0, "replies": [ { "id": 825806, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-29T09:01:27.873000", "content": "Hi @cnzengshiyuan ,\nWell normally on fp16 you should have about half of the normal training time, but this is only pure training, another time and resources consuming part is the data preprocessing, in our case the augmentation, the augmentation part time and processing needs remains the same if you are using fp32 or fp16, so the real scalability is not just cut in half the time, if a fp32 takes 20 min, a fp16 can take 15-16 mins if you have hardcore augmentation. But in your case, if it takes exactly the same time it is something wrong somewhere, be sure you are not forgetting something in the code
", "votes": 2, "replies": [] }, { "id": 825879, "author_name": "Shiyuan Zeng", "author_url": "", "post_date": "2020-04-29T09:51:51.567000", "content": "Thank you! Heard what you said, I more convinced that the time-costing is actually abnormal, I will check it carefully.(still debuging :D)
\n\nUPDATE:\nI think I had solved the time-costing problem. But I'm not sure whether this way is the cause or not. But the part I changed is only what I show below.
\n\nChange\n\nimg_info = openslide.OpenSlide(img_path) # downsample 1, 4, 16\nimg = img_info.read_region((0,0), img_info.level_count-1, img_info.level_dimensions[-1]).convert('RGB')\n\nTo\n\nimg = skimage.io.MultiImage(img_path)[-1]\n\nThe morning I use the openslide as I wrote and ~14min/epoch. And I think I may try the skimage.io and ~3.5min/epoch. I noticed there is a kernel to compare them, but I didn't realize the gap will so huge...\nModel is the same as I mentioned above(efficientnetb0-pool-dropout-lin)
The only augmentation I would suggest right now will be random rotate, it makes the model go on learning without overfitting and gets better result, the rest of the augmentations have to be tried and differentiate on data preprocessing and pipeline of the model.
", "votes": 0, "replies": [] }, { "id": 823150, "author_name": "Zac Dannelly", "author_url": "", "post_date": "2020-04-27T12:58:30.920000", "content": "Greatly appreciate this overview!
\n\nI am new to this type of data, but most of the research I have reviewed speaks to the importance of sampling from the WSI's. It seems like from your work you opted for full images just lower resolution. Would you mind speaking to how you made this choice? Trying to expand my understadning of this domain. Thanks!
", "votes": 0, "replies": [ { "id": 823202, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-27T13:39:50.420000", "content": "From what I see, there are 3 options to use a image with the data provided:\na) use a resized image of a full image\nb) use the mask where you have it and crop with the help of the masks the interested region\nc) eliminate the white space and use only the colored region
\n\nEach of the options above has advantagesand and disadvantages.\n- version a can lose a lot of details in the process of resizing\n- version b learns the algorithm to see high quality and zoom on the interesting region but on test set we do not have any mask that we can use, so it will see another type of image\n- version c sounds perfect on paper but on real life can lead to problems related to aspect ratio differences from image to image when cropping then resize, is somehow discutable and need to be tested.\nI started with a and my next test will be on c
", "votes": 2, "replies": [] }, { "id": 823210, "author_name": "Zac Dannelly", "author_url": "", "post_date": "2020-04-27T13:44:40.943000", "content": "Awesome thanks!
", "votes": 0, "replies": [] } ] }, { "id": 823012, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-04-27T10:26:27.303000", "content": "Thanks @vladvdv . I have a question. Suppose I try augmentations A, B, and C individually (not Compose) on images with supposing resolution 128x128 and I get the CV order as \"using only A < using only B < using only C\", will this CV order remain same after I increase the resolution of images suppose to 512x512?
", "votes": 0, "replies": [ { "id": 823053, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-27T11:17:10.633000", "content": "This is a very good question. From my experience it depends on the augmentation techniques used. For example, if you use CoarseDropout (maxholes=4, maxheight=10, maxwidth=10) at 128x128 resolution it will not have the same effect on 512x512 resolution, holes will be much smaller compared with the image and will not help so much on generalizing, so you will have to increase the maxheight and maxwidth proportional to the scaling factor if you want to have a similar impact. \nAt blurring augmentation, when you are having smaller resolutions the edges are not so sharp and you have to use a smaller blur factor so you won't blur to much and the model would not understand nothing from the image, but on higher resolution you have sharper edges, your blur coefficient can be a little higher. At affine transforms, the rotation, translation and flips generally keeps the same effect on different image scale.\nSo, when you increase the resolution, some augmenting techniques parameters needs to be modified in order to have the same effect, if you modify them accordingly, there are big changes that the proportion of importance in the techniques ranking will be kept
", "votes": 3, "replies": [] }, { "id": 823080, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-04-27T11:36:52.100000", "content": "Wow, great answer. Thanks for the quick response as well. All the best.
", "votes": 1, "replies": [] }, { "id": 825808, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-04-29T09:01:52.047000", "content": "I am glad I can help. All the best !
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "822242": "Hi everybody \nThese days I was excited to see another computer vision competition on Kaggle.\nFirst I would like to thank the organizers for give us the opportunity to test our computer vision skills again in a very interesting and useful competition\nSo, let's get into it.\nMy firsts experiments were with 2 model architectures: densenet121 and seresnet50\n\nInitial conclusions:\n**Augmentation**\nWe have a wide range of usefull augmentation methods for these types of images, I started with: \n- Noise augmentation (GaussNoise, MultiplicativeNoise)\n- Affine Transforms (scale, rot angle, shear angle, translations and flipping horizontal and vertical)\n- CoarseDropout (max_holes=4, max_height=30, max_width=30)\nFor now the results were best with the following probabilities for augmenting class ( general_noise_aug_prob =0.1, affine_transform_prob=0.3, coarse_dropout_prob=0.2, flip_prob= 0.2)\nTODO next:\n- augment with mixup (in this context I don't think that cutmix will be useful, you can cut an irrelevant part of the image and put it on another, so the model would not see what makes that image a 3 or a 4 if you cut the part that looks like 1 and put it next to a labeled 1 image)\n- try a more hardcore augmentation probabilities combined with more training epochs\n\n\n**Image input size**\n- Current resolution is 512x512. From my tests it is clear that improving the resolution helps a lot, the tests made on 300x300 images or lower showed a incapacity of the model to learn as it does on 512x512 \n\nTODO next:\n- try bigger resolution\n- use the segmentation masks to make a smart crop and use only useful region\n\n\n**Model architectures**\n- For now, it seems that seresnet50 is a little bit better than densenet121 tested in the same condition.\n\nTODO next:\n- Benchmark: a larger densenet like densenet161 and also se_resnext101_32x4d\n\n\n**Other aspects:**\nWhen you use a large architecture like the ones mentioned before combined with a large image size there are 2 aspects that are not so great : \n- time use for training (especially that after 50 epochs the model still hasn't reach it's peak) doesn't allow us too much experimentation options\n- smaller batch size to fit in the RAM of the GPU. Using a good GPU like 2080TI I still have to use batch sizes like 4 of 6 in the mentioned conditions. Although batch size is a controversial aspect with a lot and pro and cons, making the optimizer step at every 4-6 images in this context can be a little bit to jumpy for my taste \n\nI have made public my inference kernel ([Inference Kernel](https://www.kaggle.com/vladvdv/pytorch-inference-multiple-models-and-folds)) that can be used as a template very easy for anybody. You can easily add different models each with their own folds and use custom weights for blending them. Also, a TTA flag is enable for whoever wants to experiment.\nI will keep you up to date with my experiments and if I have time I will make public my training kernel where you can see other tricks that I used\n\nGood luck to everybody \n\n**Later edit**: The public training kernel in online now\nThis base allows you to\n\n- understand how pytorch and computer vision works\n- be competitive in the competition\n- gain experience by trying easy to implement different customizations\n\nThere are a lot of knobs to tweek in order to personalize it:\n\n- a lot of augmentation techniques\n- choose image size\n- choose the desired pretrained model\n- customize the arhitecture by add more dense layers or any other layer types\n- change number of folds or the spliting\n\nhttps://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds", "824805": "This is fantastic. Thanks for always sharing your insights and your work. It is an inspiration.", "823630": "Hey thanks for all this info much appreciated! May i ask how many epochs are you training for? For me when im training longer im getting a higher CV and validation loss is still decreasing but this is resulting in a lower LB", "823443": "Hi, @vladvdv thanks a lot for these tips, they are great especially for people like me that are learning about image segmentation and this type of domain. \nI was wondering when will you post your training code.\n\nThanks again.", "822751": "@vladvdv May I ask if your image augmentations are performed on the training images or the masks as well? Thanks and great post, always looking out for your inputs.", "822421": "Thanks for sharing this post!", "822312": "Awesome post, thanks for sharing your results so far", "822307": "Glad to see you in another CV comp! Can't wait to learn a lot from you! Your sharing is always appreciated.\n\nI have been doing similar experiments with seresnext50 however the model keeps overfitting. Have you ran into this issue? If so how have you gone about fixing it?", "824385": "I made a public the training kernel.\nThis is a solid starting point for any kaggler in this competition. This base allows you to\n\n- understand how pytorch and computer vision works\n- be competitive in the competition\n- gain experience by trying easy to implement different customizations\n\nThere are a lot of knobs to tweek in order to personalize it:\n\n- a lot of augmentation techniques\n- choose image size\n- choose the desired pretrained model\n- customize the arhitecture by add more dense layers or any other layer types\n- change number of folds or the spliting\n\n[https://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds](https://www.kaggle.com/vladvdv/pytorch-training-customizable-kernel-with-5-folds)\n\nHave fun ! ", "825409": "\n\nHappy to see you here ;)\nWhat'more, I wanna ask you a question.\nI noticed the model you used in the kernel will cost almost 30mins(train+valid), but I use fp16 to train my efficientnet-b0(I wanna use small model to do more experiments) will cost 14mins(train+valid) and if I use full-precision to train will cost also 14mins(train+valid), it makes me confused... However, [here is iafoss use resnext-32x4d-ssl](https://www.kaggle.com/iafoss/panda-concat-tile-pooling-starter-0-79-lb/notebook) use fp16 to train just cost ~7mins(train+valid). Is my model normal? My arch is: efficientb0-pool-drop-lin(1 layer), bs=32, resolution=64x64\nThanks in advance 😄 ", "823733": "The only augmentation I would suggest right now will be random rotate, it makes the model go on learning without overfitting and gets better result, the rest of the augmentations have to be tried and differentiate on data preprocessing and pipeline of the model.", "823150": "Greatly appreciate this overview! \n\nI am new to this type of data, but most of the research I have reviewed speaks to the importance of sampling from the WSI's. It seems like from your work you opted for full images just lower resolution. Would you mind speaking to how you made this choice? Trying to expand my understadning of this domain. Thanks!", "823012": "Thanks @vladvdv . I have a question. Suppose I try augmentations A, B, and C individually (not Compose) on images with supposing resolution 128x128 and I get the CV order as \"using only A < using only B < using only C\", will this CV order remain same after I increase the resolution of images suppose to 512x512? " } }