{ "id": 414344, "title": "some successful/fail experiment", "url": "/competitions/google-research-identify-contrails-reduce-global-warming/discussion/414344", "author_name": "lyu", "post_date": "2023-06-01T09:59:58.307000", "votes": 112, "comment_count": 26, "views": 0, "content": "

Hello everyone, I have been participating in this competition for almost a week now. This week, I have conducted a lot of experiments, from the initial confusion about how to calculate CV to the subsequent parameter tuning, training models, and so on.

\n

Here, I would like to share some of my experimental records. I find some experimental results very strange, and I hope everyone can communicate together!

\n

My first baseline model is unet+efficientnet-b0, with a score of around 610 for both cv and lb, and all the following experiments were conducted based on this.

\n
    \n

  1. ALL data augmentation do not works for me. It's vert strange. I have tried simple flipping, rotating, and other operations, as well as advanced methods such as mixup, which can cause a decrease in my CV and LB

    2. \n

  2. I tried to fuse images from multiple times to create a 2.5D image, but cv droped…

    3. \n

  3. I replaced efficientnet with mit, but the cv was even worse. eficientnet seems to performe very well in this competition

    4. \n

  4. I simply tried models such as Segformer and Deeplabv3, the results were even worse. but, I did not perform detailed parameter tuning on them. This competition is like those medical image segmentation tasks in Kaggle, where simple unet can achieve good results…

    5. \n

  5. A larger backbone can bring great improvements, similarly, a larger resolution can also bring great improvements. With a larger backbone and a larger resolution, my baseline can be improved to 650+

    6. \n
\n

6.The improvement that pseudo labels is also very significant. I simply tried some methods of pseudo labels and the trained model has a score of nearly 670

\n
    \n
  1. Post processing is very important, and an appropriate threshold generally increases 2% -3%. And I think maybe we can consider implementing this paragraph in the post processing , but currently I don't have a very suitable idea
    2. \n
\n
\n

Some key labeling guidance:
\n Contrails must contain at least 10 pixels
\n At some time in their life, Contrails must be at least 3x longer than they are wide
\n Contrails must either appear suddenly or enter from the sides of the image
\n Contrails should be visible in at least two image

\n
\n
    \n

  1. I have tried many losses, bce and dice achieve the best results, but the combination of the two has not improved😟

    2. \n

  2. Adding auxiliary classification header, failed

    3. \n

  3. As the experiment progresses, my CV seems to be somewhat inconsistent with LB…

    4. \n
\n

Is anyone willing to share their model and scores？

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
cvlb
584589
607604
639643
643665
648673
653675
653670
656682
657678
\n
\n

update:
\nso upset…I already get 0.682 lb. these days, I get new 0.681cv, but lb only 0.680…
\nAlthough I know it's because there are too few images in lb, the fact that the score hasn't improved is still very disappointing.

", "messages": [ { "id": 2283509, "postDate": "2023-06-01T09:59:58.307Z", "content": "

Hello everyone, I have been participating in this competition for almost a week now. This week, I have conducted a lot of experiments, from the initial confusion about how to calculate CV to the subsequent parameter tuning, training models, and so on.

\n

Here, I would like to share some of my experimental records. I find some experimental results very strange, and I hope everyone can communicate together!

\n

My first baseline model is unet+efficientnet-b0, with a score of around 610 for both cv and lb, and all the following experiments were conducted based on this.

\n
    \n

  1. ALL data augmentation do not works for me. It's vert strange. I have tried simple flipping, rotating, and other operations, as well as advanced methods such as mixup, which can cause a decrease in my CV and LB

    2. \n

  2. I tried to fuse images from multiple times to create a 2.5D image, but cv droped…

    3. \n

  3. I replaced efficientnet with mit, but the cv was even worse. eficientnet seems to performe very well in this competition

    4. \n

  4. I simply tried models such as Segformer and Deeplabv3, the results were even worse. but, I did not perform detailed parameter tuning on them. This competition is like those medical image segmentation tasks in Kaggle, where simple unet can achieve good results…

    5. \n

  5. A larger backbone can bring great improvements, similarly, a larger resolution can also bring great improvements. With a larger backbone and a larger resolution, my baseline can be improved to 650+

    6. \n
\n

6.The improvement that pseudo labels is also very significant. I simply tried some methods of pseudo labels and the trained model has a score of nearly 670

\n
    \n
  1. Post processing is very important, and an appropriate threshold generally increases 2% -3%. And I think maybe we can consider implementing this paragraph in the post processing , but currently I don't have a very suitable idea
    2. \n
\n
\n

Some key labeling guidance:
\n Contrails must contain at least 10 pixels
\n At some time in their life, Contrails must be at least 3x longer than they are wide
\n Contrails must either appear suddenly or enter from the sides of the image
\n Contrails should be visible in at least two image

\n
\n
    \n

  1. I have tried many losses, bce and dice achieve the best results, but the combination of the two has not improved😟

    2. \n

  2. Adding auxiliary classification header, failed

    3. \n

  3. As the experiment progresses, my CV seems to be somewhat inconsistent with LB…

    4. \n
\n

Is anyone willing to share their model and scores？

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
cvlb
584589
607604
639643
643665
648673
653675
653670
656682
657678
\n
\n

update:
\nso upset…I already get 0.682 lb. these days, I get new 0.681cv, but lb only 0.680…
\nAlthough I know it's because there are too few images in lb, the fact that the score hasn't improved is still very disappointing.

", "rawMarkdown": "Hello everyone, I have been participating in this competition for almost a week now. This week, I have conducted a lot of experiments, from the initial confusion about how to calculate CV to the subsequent parameter tuning, training models, and so on.\n\nHere, I would like to share some of my experimental records. I find some experimental results very strange, and I hope everyone can communicate together!\n\nMy first baseline model is unet+efficientnet-b0, with a score of around 610 for both cv and lb, and all the following experiments were conducted based on this.\n\n1. ALL data augmentation do not works for me. It's vert strange. I have tried simple flipping, rotating, and other operations, as well as advanced methods such as mixup, which can cause a decrease in my CV and LB\n\n2. I tried to fuse images from multiple times to create a 2.5D image, but cv droped...\n\n3. I replaced efficientnet with mit, but the cv was even worse. eficientnet seems to performe very well in this competition\n\n4. I simply tried models such as Segformer and Deeplabv3, the results were even worse. but, I did not perform detailed parameter tuning on them. This competition is like those medical image segmentation tasks in Kaggle, where simple unet can achieve good results...\n\n5. A larger backbone can bring great improvements, similarly, a larger resolution can also bring great improvements. With a larger backbone and a larger resolution, my baseline can be improved to 650+\n\n6.The improvement that pseudo labels is also very significant. I simply tried some methods of pseudo labels and the trained model has a score of nearly 670\n\n7. Post processing is very important, and an appropriate threshold generally increases 2% -3%. And I think maybe we can consider implementing this paragraph in the post processing , but currently I don't have a very suitable idea\n>Some key labeling guidance:\nContrails must contain at least 10 pixels\nAt some time in their life, Contrails must be at least 3x longer than they are wide\nContrails must either appear suddenly or enter from the sides of the image\nContrails should be visible in at least two image\n\n8. I have tried many losses, bce and dice achieve the best results, but the combination of the two has not improved😟\n\n9. Adding auxiliary classification header, failed\n\n10. As the experiment progresses, my CV seems to be somewhat inconsistent with LB...\n\nIs anyone willing to share their model and scores？\n\n| cv | lb |\n| --- | --- |\n| 584 | 589 |\n| 607| 604 |\n| 639 | 643 |\n| 643 | 665 |\n| 648 | 673 |\n| 653 | 675 |\n| 653 | 670 |\n| 656 | 682 |\n| 657 | 678 |\n\n__________________________________________________________________\nupdate:\nso upset...I already get 0.682 lb. these days, I get new 0.681cv, but lb only 0.680...\nAlthough I know it's because there are too few images in lb, the fact that the score hasn't improved is still very disappointing.", "votes": 110 }, { "id": 2283543, "postDate": "2023-06-01T10:35:30.830Z", "content": "

First of all thank you @zhuwanglju so much for writing so much details about your solution when you are already Number 1 in the current public LB . It takes lot of courage to do so . Therefore let me also write my details and ask few questions

\n

baseline model is unet+effb0 , 256 size - LB .643

\n
    \n

  1. I have always used hard spatial and color augmentations . I have not tried with minimal augmentation . Mixup gives low CV and LB . I dont have a reason why .

    2. \n

  2. I tried to fuse frames from the images with 2.5D , my CV dropped and hence I didnt pursue it for now.

    3. \n

  3. MIT , Convnext , Densenet etc , drops CV and also LB .

    4. \n
\n

4.DeeplabV3 + and Deeplabv3plus gives lower CV , I have not tested on LB . Unet works well in my experiment

\n

5.Larger backbones are better to certain extent . B1 is better than B0 , but B2 was worse for me than B0 and B1 .So for me not all bigger backbone works .

\n

6.Have not tried Pseudolabel . Are you trying pseudolabel by prediction on train and valid data ? (You need not answer if this gives away very special tricks )

\n

7.Have not tried post processing

\n

8.Adding auxiliary classifier reduces the CV and LB .

\n

9.CV and LB is not consistent for me always . I have a question about your dice score . I use the public notebook one and the CV is always quite high .
\nI see you are using the below . I am not sure if you are already passing mask after applying sigmoid and threshold ? or passing only the raw logits ? Are you only passing the positive images into this ? Any guidance would be helpful .
\ndef dice_score(y_p, y_t, smooth=1e-6):
\n i = torch.sum(y_p * y_t, dim=(2, 3))
\n u = torch.sum(y_p, dim=(2, 3)) + torch.sum(y_t, dim=(2, 3))
\n score = (2 * i + smooth)/(u + smooth)
\n return torch.mean(score)

\n
    \n
  1. I am using combination of losses . I have not tried only BCE .
    2. \n
\n

11.I am using 256 size images , I saw couple of experiment that 512 images are better . But since it has a longer training time , I got impatient with it for now and only testing with 256 size . Single model score is 0.661 .

", "rawMarkdown": "First of all thank you @zhuwanglju so much for writing so much details about your solution when you are already Number 1 in the current public LB . It takes lot of courage to do so . Therefore let me also write my details and ask few questions \n\nbaseline model is unet+effb0 , 256 size - LB .643\n\n1. I have always used hard spatial and color augmentations . I have not tried with minimal augmentation . Mixup gives low CV and LB . I dont have a reason why . \n\n2. I tried to fuse frames from the images with 2.5D , my CV dropped and hence I didnt pursue it for now. \n\n3. MIT , Convnext , Densenet etc , drops CV and also LB .\n\n4.DeeplabV3 + and Deeplabv3plus gives lower CV , I have not tested on LB . Unet works well in my experiment \n\n5.Larger backbones are better to certain extent . B1 is better than B0 , but B2 was worse for me than B0 and B1 .So for me not all bigger backbone works . \n\n6.Have not tried Pseudolabel . Are you trying pseudolabel by prediction on train and valid data ? (You need not answer if this gives away very special tricks )\n\n7.Have not tried post processing \n\n8.Adding auxiliary classifier reduces the CV and LB . \n\n9.CV and LB is not consistent for me always . I have a question about your dice score . I use the public notebook one and the CV is always quite high . \nI see you are using the below . I am not sure if you are already passing mask after applying sigmoid and threshold ? or passing only the raw logits ? Are you only passing the positive images into this ? Any guidance would be helpful . \ndef dice_score(y_p, y_t, smooth=1e-6):\n i = torch.sum(y_p * y_t, dim=(2, 3))\n u = torch.sum(y_p, dim=(2, 3)) + torch.sum(y_t, dim=(2, 3))\n score = (2 * i + smooth)/(u + smooth)\n return torch.mean(score)\n\n10. I am using combination of losses . I have not tried only BCE . \n\n11.I am using 256 size images , I saw couple of experiment that 512 images are better . But since it has a longer training time , I got impatient with it for now and only testing with 256 size . Single model score is 0.661 . ", "votes": 17, "replies": [ { "id": 2283603, "postDate": "2023-06-01T11:36:40.223Z", "content": "

Thank you for your so detailed reply! Your notebook also helped me a lot in the early stages! The competition time is still long, and I just want to learn more from it.💪

\n

some reply:
\nI didn't use the dice function above. What I meant was that if we use dice function like this, our cv will be higher, easily exceeding 0.7.

\n

my pseudo label is simply predict train and valid, and add some easy post-processing.

", "rawMarkdown": "Thank you for your so detailed reply! Your [notebook ](https://www.kaggle.com/code/phoenix9032/inference-unet-effnetb0-on-ash-v2-baseline)also helped me a lot in the early stages! The competition time is still long, and I just want to learn more from it.💪\n\nsome reply:\nI didn't use the dice function above. What I meant was that if we use dice function like this, our cv will be higher, easily exceeding 0.7. \n\nmy pseudo label is simply predict train and valid, and add some easy post-processing. ", "votes": 5, "replies": [ { "id": 2284008, "postDate": "2023-06-01T17:00:09.323Z", "content": "

can you share your dice function

", "rawMarkdown": "can you share your dice function", "votes": -2 }, { "id": 2284077, "postDate": "2023-06-01T17:49:05.030Z", "content": "

Thanks @zhuwanglju .. Got it .

", "rawMarkdown": "Thanks @zhuwanglju .. Got it ." }, { "id": 2287540, "postDate": "2023-06-04T16:03:16.757Z", "content": "

If you got dice function then share, please.

", "rawMarkdown": "If you got dice function then share, please.\n", "votes": -1 } ] }, { "id": 2284230, "postDate": "2023-06-01T20:26:08.207Z", "content": "

A baseline with Unet and effnet b0 using 256 as size with a score of 0.643 is crazy! With the same setup (I am using segmentation models pytorch implementation of unet and effnetb0 as backbone) my highest LB score is 0.538. You said that you use a combination of losses. May I ask what kind of losses you combined?

", "rawMarkdown": "A baseline with Unet and effnet b0 using 256 as size with a score of 0.643 is crazy! With the same setup (I am using segmentation models pytorch implementation of unet and effnetb0 as backbone) my highest LB score is 0.538. You said that you use a combination of losses. May I ask what kind of losses you combined?", "votes": 6 } ] }, { "id": 2283521, "postDate": "2023-06-01T10:09:47.343Z", "content": "

I face same problems, augmentations did not help at all and scaling x2 helps and low threshold also helps

", "rawMarkdown": "I face same problems, augmentations did not help at all and scaling x2 helps and low threshold also helps", "votes": 11, "replies": [ { "id": 2283605, "postDate": "2023-06-01T11:37:51.667Z", "content": "

thanks for your reply. We have the same situation😂

", "rawMarkdown": "thanks for your reply. We have the same situation😂", "votes": 4, "replies": [ { "id": 2321784, "postDate": "2023-06-28T21:44:15.937Z", "content": "

@maksimovka How are you scaling up the size of the images by x2?

\n

-

\n

Just found the answer. In the preprint, the authors \"perform bilinear resampling to enlarge the input image\". Looks likes this is the default interpolation method in torchvision.

\n

ex. torchvision.transforms.Resize((size, size))

", "rawMarkdown": "@maksimovka How are you scaling up the size of the images by x2? \n\n-\n\nJust found the answer. In the preprint, the authors \"perform bilinear resampling to enlarge the input image\". Looks likes this is the default interpolation method in torchvision.\n\nex. `torchvision.transforms.Resize((size, size))`", "votes": 1 }, { "id": 2330205, "postDate": "2023-07-04T19:39:30.577Z", "content": "

Yes, I have used the same

", "rawMarkdown": "Yes, I have used the same", "votes": 1 } ] } ] }, { "id": 2283623, "postDate": "2023-06-01T11:58:13.747Z", "content": "

Hi everyone,

\n

My initial approach was U-Net(Vgg-16 then Efficientnet-b7) with 12 channels as output used in combination with CrossEntropy loss. This pushed my score to public lb 5 in the beginning stages of the competition. I then changed the channels to 1 and applied BinaryCrossEntropy loss and it performed worse. I shifted towards segmentation models (Pytorch) with efficientnet b7. Small gains in dice_loss was directly co-related with the public LB.

\n

But I found something interesting and made a post about it, deleted it and now I am posting again (Apologies for this, I should have given learning a higher priority over lb).

\n

dice_loss - 0.4447, iou_score - 0.4002 ( VS ) dice_loss - 0.4475, iou_score - 0.4044

\n

The former brought me to public lb position 13 with score of 0.629 and the latter to public lb position 12 with score of 0.638. Seems like IOU is more of an important metric in conjunction with dice_loss.

\n

So far i have been playing with a single model and trying to squeeze as much as I can before trying something diverse for future ensembles later on in the competition.
\nI have only covered things that are different from my approach to your approach. Things like augmentation, small model etc didnt do the trick for me either.

\n

Thanks for the post and happy kaggling guys.

\n

If any1 is interested in teaming up with me, I am all open to a transparent relation of learning and winning together.

", "rawMarkdown": "Hi everyone,\n\nMy initial approach was U-Net(Vgg-16 then Efficientnet-b7) with 12 channels as output used in combination with CrossEntropy loss. This pushed my score to public lb 5 in the beginning stages of the competition. I then changed the channels to 1 and applied BinaryCrossEntropy loss and it performed worse. I shifted towards segmentation models (Pytorch) with efficientnet b7. Small gains in dice_loss was directly co-related with the public LB.\n\nBut I found something interesting and made a post about it, deleted it and now I am posting again (Apologies for this, I should have given learning a higher priority over lb).\n\ndice_loss - 0.4447, iou_score - 0.4002 ( VS ) dice_loss - 0.4475, iou_score - 0.4044\n\nThe former brought me to public lb position 13 with score of 0.629 and the latter to public lb position 12 with score of 0.638. Seems like IOU is more of an important metric in conjunction with dice_loss. \n\nSo far i have been playing with a single model and trying to squeeze as much as I can before trying something diverse for future ensembles later on in the competition.\nI have only covered things that are different from my approach to your approach. Things like augmentation, small model etc didnt do the trick for me either. \n\nThanks for the post and happy kaggling guys.\n\nIf any1 is interested in teaming up with me, I am all open to a transparent relation of learning and winning together.", "votes": 5, "replies": [ { "id": 2319640, "postDate": "2023-06-27T08:38:00.237Z", "content": "

I'm a little bit surprised IoU can work on a segmentation task👀

", "rawMarkdown": "I'm a little bit surprised IoU can work on a segmentation task👀", "replies": [ { "id": 2329963, "postDate": "2023-07-04T16:17:31.273Z", "content": "

@fuckvenkatraman, I was just as surprised as you are. I was following the enclosed notebook. I have seen some good results by taking IOU into consideration apart from DICE Co-efficient alone. Let me know your thoughts.

\n
https:thub.comsegmentation_models.pytorchastercars%segmentation%(camvid).ipynb\n
", "rawMarkdown": "@fuckvenkatraman, I was just as surprised as you are. I was following the enclosed notebook. I have seen some good results by taking IOU into consideration apart from DICE Co-efficient alone. Let me know your thoughts. \n```\nhttps://github.com/qubvel/segmentation_models.pytorch/blob/master/examples/cars%20segmentation%20(camvid).ipynb\n```" } ] } ] }, { "id": 2283993, "postDate": "2023-06-01T16:46:25.240Z", "content": "

Hey Lyu,

\n

thanks for your very valuable insights! Currently I am also pursuing Unet+effnetb0 from Segmentation Models Pytorch.
\nI also experimented with different augmentations (also with my earlier DeepLabV3+ Model) and none of them worked for me. Flipping or Rotating always decreased the LB and CV scores…
\nI have two further questions regarding your training loop / hyperparameters:
\n1.) Do you guys also use the train and validation folders for train/val split or do you modify them in any other way?
\n2.) What callbacks do you use? I experimented with ReduceLROnPlateau as well as LRScheduler with LR decay after each epoch. Also I used a LinearWarmup, but it seems like none of these made any difference regarding my CV and LB Scores.

\n

Good Luck!

", "rawMarkdown": "Hey Lyu,\n\nthanks for your very valuable insights! Currently I am also pursuing Unet+effnetb0 from Segmentation Models Pytorch.\nI also experimented with different augmentations (also with my earlier DeepLabV3+ Model) and none of them worked for me. Flipping or Rotating always decreased the LB and CV scores... \nI have two further questions regarding your training loop / hyperparameters:\n1.) Do you guys also use the train and validation folders for train/val split or do you modify them in any other way?\n2.) What callbacks do you use? I experimented with ReduceLROnPlateau as well as LRScheduler with LR decay after each epoch. Also I used a LinearWarmup, but it seems like none of these made any difference regarding my CV and LB Scores.\n\nGood Luck!", "votes": 1, "replies": [ { "id": 2284382, "postDate": "2023-06-02T01:53:26.113Z", "content": "

reply :
\n1.yes, using the train and validation folders for train/val use the train and validation folders for train/val
\n2.I have not try different lr-scheduler ,just use CosineAnnealingLR

", "rawMarkdown": "reply :\n1.yes, using the train and validation folders for train/val use the train and validation folders for train/val \n2.I have not try different lr-scheduler ,just use CosineAnnealingLR", "votes": 3 } ] }, { "id": 2286917, "postDate": "2023-06-04T00:54:06.890Z", "content": "

Unet+B0,…B6 gets LB0.60-0.62. Albu augments rot90, hv-flips. Could not find other augmentations which improved LB. Interestingly, averaging multiple folds makes only a tiny LB improvement.

", "rawMarkdown": "Unet+B0,...B6 gets LB0.60-0.62. Albu augments rot90, hv-flips. Could not find other augmentations which improved LB. Interestingly, averaging multiple folds makes only a tiny LB improvement.", "votes": 2, "replies": [ { "id": 2287597, "postDate": "2023-06-04T17:35:45.977Z", "content": "

I think you should train on full data for a single type of model and then try ensembling different backbones. It should bring a big boost.

", "rawMarkdown": "I think you should train on full data for a single type of model and then try ensembling different backbones. It should bring a big boost.", "votes": 5 } ] }, { "id": 2283912, "postDate": "2023-06-01T15:24:42.597Z", "content": "

As I said in this thread, the public LB is based on just 264-292 samples. That is why CV is not consistent with LB.

", "rawMarkdown": "As I said [in this thread](https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/413068#2276374), the public LB is based on just 264-292 samples. That is why CV is not consistent with LB.", "votes": 1 }, { "id": 2283636, "postDate": "2023-06-01T12:11:22.070Z", "content": "

is segformer allowed here?

", "rawMarkdown": "is segformer allowed here?", "votes": -3 }, { "id": 2364678, "postDate": "2023-07-29T14:18:05.213Z", "content": "

thank you all for sharing! useful for me

", "rawMarkdown": "thank you all for sharing! useful for me" }, { "id": 2331806, "postDate": "2023-07-05T19:32:05.337Z", "content": "

@zhuwanglju I have a question about point 6 - you have used pseudo labels to relabel the training data?

", "rawMarkdown": "@zhuwanglju I have a question about point 6 - you have used pseudo labels to relabel the training data?", "replies": [ { "id": 2332939, "postDate": "2023-07-06T14:35:13.197Z", "content": "

just predict the unlabeled train data.

", "rawMarkdown": "just predict the unlabeled train data.", "replies": [ { "id": 2332965, "postDate": "2023-07-06T15:01:45.323Z", "content": "

Where did you get unlabeled train data from?

", "rawMarkdown": "Where did you get unlabeled train data from?" } ] } ] }, { "id": 2286919, "postDate": "2023-06-04T01:00:27.617Z", "content": "

Has anyone tried to train only on images with non-zero labels? On my todo list.

", "rawMarkdown": "Has anyone tried to train only on images with non-zero labels? On my todo list.", "replies": [ { "id": 2287593, "postDate": "2023-06-04T17:33:55.787Z", "content": "

Ya I tried this, it has a worse score than training on all images.

", "rawMarkdown": "Ya I tried this, it has a worse score than training on all images.", "votes": 6 } ] }, { "id": 2283870, "postDate": "2023-06-01T14:51:18.327Z", "content": "

Very interesting, thank you all for sharing

", "rawMarkdown": "Very interesting, thank you all for sharing" } ], "comments": [ { "id": 2283543, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2023-06-01T10:35:30.830000", "content": "

First of all thank you @zhuwanglju so much for writing so much details about your solution when you are already Number 1 in the current public LB . It takes lot of courage to do so . Therefore let me also write my details and ask few questions

\n

baseline model is unet+effb0 , 256 size - LB .643

\n
    \n

  1. I have always used hard spatial and color augmentations . I have not tried with minimal augmentation . Mixup gives low CV and LB . I dont have a reason why .

    2. \n

  2. I tried to fuse frames from the images with 2.5D , my CV dropped and hence I didnt pursue it for now.

    3. \n

  3. MIT , Convnext , Densenet etc , drops CV and also LB .

    4. \n
\n

4.DeeplabV3 + and Deeplabv3plus gives lower CV , I have not tested on LB . Unet works well in my experiment

\n

5.Larger backbones are better to certain extent . B1 is better than B0 , but B2 was worse for me than B0 and B1 .So for me not all bigger backbone works .

\n

6.Have not tried Pseudolabel . Are you trying pseudolabel by prediction on train and valid data ? (You need not answer if this gives away very special tricks )

\n

7.Have not tried post processing

\n

8.Adding auxiliary classifier reduces the CV and LB .

\n

9.CV and LB is not consistent for me always . I have a question about your dice score . I use the public notebook one and the CV is always quite high .
\nI see you are using the below . I am not sure if you are already passing mask after applying sigmoid and threshold ? or passing only the raw logits ? Are you only passing the positive images into this ? Any guidance would be helpful .
\ndef dice_score(y_p, y_t, smooth=1e-6):
\n i = torch.sum(y_p * y_t, dim=(2, 3))
\n u = torch.sum(y_p, dim=(2, 3)) + torch.sum(y_t, dim=(2, 3))
\n score = (2 * i + smooth)/(u + smooth)
\n return torch.mean(score)

\n
    \n
  1. I am using combination of losses . I have not tried only BCE .
    2. \n
\n

11.I am using 256 size images , I saw couple of experiment that 512 images are better . But since it has a longer training time , I got impatient with it for now and only testing with 256 size . Single model score is 0.661 .

", "votes": 17, "replies": [ { "id": 2283603, "author_name": "lyu", "author_url": "", "post_date": "2023-06-01T11:36:40.223000", "content": "

Thank you for your so detailed reply! Your notebook also helped me a lot in the early stages! The competition time is still long, and I just want to learn more from it.💪

\n

some reply:
\nI didn't use the dice function above. What I meant was that if we use dice function like this, our cv will be higher, easily exceeding 0.7.

\n

my pseudo label is simply predict train and valid, and add some easy post-processing.

", "votes": 5, "replies": [ { "id": 2284008, "author_name": "Arunodhayan", "author_url": "", "post_date": "2023-06-01T17:00:09.323000", "content": "

can you share your dice function

", "votes": -2, "replies": [] }, { "id": 2284077, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2023-06-01T17:49:05.030000", "content": "

Thanks @zhuwanglju .. Got it .

", "votes": 0, "replies": [] }, { "id": 2287540, "author_name": "Md Hasan Shahriar", "author_url": "", "post_date": "2023-06-04T16:03:16.757000", "content": "

If you got dice function then share, please.

", "votes": -1, "replies": [] } ] }, { "id": 2284230, "author_name": "Jan H", "author_url": "", "post_date": "2023-06-01T20:26:08.207000", "content": "

A baseline with Unet and effnet b0 using 256 as size with a score of 0.643 is crazy! With the same setup (I am using segmentation models pytorch implementation of unet and effnetb0 as backbone) my highest LB score is 0.538. You said that you use a combination of losses. May I ask what kind of losses you combined?

", "votes": 6, "replies": [] } ] }, { "id": 2283521, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2023-06-01T10:09:47.343000", "content": "

I face same problems, augmentations did not help at all and scaling x2 helps and low threshold also helps

", "votes": 11, "replies": [ { "id": 2283605, "author_name": "lyu", "author_url": "", "post_date": "2023-06-01T11:37:51.667000", "content": "

thanks for your reply. We have the same situation😂

", "votes": 4, "replies": [ { "id": 2321784, "author_name": "Bartley", "author_url": "", "post_date": "2023-06-28T21:44:15.937000", "content": "

@maksimovka How are you scaling up the size of the images by x2?

\n

-

\n

Just found the answer. In the preprint, the authors \"perform bilinear resampling to enlarge the input image\". Looks likes this is the default interpolation method in torchvision.

\n

ex. torchvision.transforms.Resize((size, size))

", "votes": 1, "replies": [] }, { "id": 2330205, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2023-07-04T19:39:30.577000", "content": "

Yes, I have used the same

", "votes": 1, "replies": [] } ] } ] }, { "id": 2283623, "author_name": "Kenni", "author_url": "", "post_date": "2023-06-01T11:58:13.747000", "content": "

Hi everyone,

\n

My initial approach was U-Net(Vgg-16 then Efficientnet-b7) with 12 channels as output used in combination with CrossEntropy loss. This pushed my score to public lb 5 in the beginning stages of the competition. I then changed the channels to 1 and applied BinaryCrossEntropy loss and it performed worse. I shifted towards segmentation models (Pytorch) with efficientnet b7. Small gains in dice_loss was directly co-related with the public LB.

\n

But I found something interesting and made a post about it, deleted it and now I am posting again (Apologies for this, I should have given learning a higher priority over lb).

\n

dice_loss - 0.4447, iou_score - 0.4002 ( VS ) dice_loss - 0.4475, iou_score - 0.4044

\n

The former brought me to public lb position 13 with score of 0.629 and the latter to public lb position 12 with score of 0.638. Seems like IOU is more of an important metric in conjunction with dice_loss.

\n

So far i have been playing with a single model and trying to squeeze as much as I can before trying something diverse for future ensembles later on in the competition.
\nI have only covered things that are different from my approach to your approach. Things like augmentation, small model etc didnt do the trick for me either.

\n

Thanks for the post and happy kaggling guys.

\n

If any1 is interested in teaming up with me, I am all open to a transparent relation of learning and winning together.

", "votes": 5, "replies": [ { "id": 2319640, "author_name": "豆柴金鯱", "author_url": "", "post_date": "2023-06-27T08:38:00.237000", "content": "

I'm a little bit surprised IoU can work on a segmentation task👀

", "votes": 0, "replies": [ { "id": 2329963, "author_name": "Kenni", "author_url": "", "post_date": "2023-07-04T16:17:31.273000", "content": "

@fuckvenkatraman, I was just as surprised as you are. I was following the enclosed notebook. I have seen some good results by taking IOU into consideration apart from DICE Co-efficient alone. Let me know your thoughts.

\n
https:thub.comsegmentation_models.pytorchastercars%segmentation%(camvid).ipynb\n
", "votes": 0, "replies": [] } ] } ] }, { "id": 2283993, "author_name": "Jan H", "author_url": "", "post_date": "2023-06-01T16:46:25.240000", "content": "

Hey Lyu,

\n

thanks for your very valuable insights! Currently I am also pursuing Unet+effnetb0 from Segmentation Models Pytorch.
\nI also experimented with different augmentations (also with my earlier DeepLabV3+ Model) and none of them worked for me. Flipping or Rotating always decreased the LB and CV scores…
\nI have two further questions regarding your training loop / hyperparameters:
\n1.) Do you guys also use the train and validation folders for train/val split or do you modify them in any other way?
\n2.) What callbacks do you use? I experimented with ReduceLROnPlateau as well as LRScheduler with LR decay after each epoch. Also I used a LinearWarmup, but it seems like none of these made any difference regarding my CV and LB Scores.

\n

Good Luck!

", "votes": 1, "replies": [ { "id": 2284382, "author_name": "lyu", "author_url": "", "post_date": "2023-06-02T01:53:26.113000", "content": "

reply :
\n1.yes, using the train and validation folders for train/val use the train and validation folders for train/val
\n2.I have not try different lr-scheduler ,just use CosineAnnealingLR

", "votes": 3, "replies": [] } ] }, { "id": 2286917, "author_name": "dmitrykonovalov", "author_url": "", "post_date": "2023-06-04T00:54:06.890000", "content": "

Unet+B0,…B6 gets LB0.60-0.62. Albu augments rot90, hv-flips. Could not find other augmentations which improved LB. Interestingly, averaging multiple folds makes only a tiny LB improvement.

", "votes": 2, "replies": [ { "id": 2287597, "author_name": "Shashwat Raman", "author_url": "", "post_date": "2023-06-04T17:35:45.977000", "content": "

I think you should train on full data for a single type of model and then try ensembling different backbones. It should bring a big boost.

", "votes": 5, "replies": [] } ] }, { "id": 2283912, "author_name": "MD Mushfirat Mohaimin", "author_url": "", "post_date": "2023-06-01T15:24:42.597000", "content": "

As I said in this thread, the public LB is based on just 264-292 samples. That is why CV is not consistent with LB.

", "votes": 1, "replies": [] }, { "id": 2283636, "author_name": "Arunodhayan", "author_url": "", "post_date": "2023-06-01T12:11:22.070000", "content": "

is segformer allowed here?

", "votes": -3, "replies": [] }, { "id": 2364678, "author_name": "DaydaycodingDaydayup", "author_url": "", "post_date": "2023-07-29T14:18:05.213000", "content": "

thank you all for sharing! useful for me

", "votes": 0, "replies": [] }, { "id": 2331806, "author_name": "Kostiantyn Maksymov", "author_url": "", "post_date": "2023-07-05T19:32:05.337000", "content": "

@zhuwanglju I have a question about point 6 - you have used pseudo labels to relabel the training data?

", "votes": 0, "replies": [ { "id": 2332939, "author_name": "lyu", "author_url": "", "post_date": "2023-07-06T14:35:13.197000", "content": "

just predict the unlabeled train data.

", "votes": 0, "replies": [ { "id": 2332965, "author_name": "Patchef", "author_url": "", "post_date": "2023-07-06T15:01:45.323000", "content": "

Where did you get unlabeled train data from?

", "votes": 0, "replies": [] } ] } ] }, { "id": 2286919, "author_name": "dmitrykonovalov", "author_url": "", "post_date": "2023-06-04T01:00:27.617000", "content": "

Has anyone tried to train only on images with non-zero labels? On my todo list.

", "votes": 0, "replies": [ { "id": 2287593, "author_name": "Shashwat Raman", "author_url": "", "post_date": "2023-06-04T17:33:55.787000", "content": "

Ya I tried this, it has a worse score than training on all images.

", "votes": 6, "replies": [] } ] }, { "id": 2283870, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-06-01T14:51:18.327000", "content": "

Very interesting, thank you all for sharing

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2283509": "Hello everyone, I have been participating in this competition for almost a week now. This week, I have conducted a lot of experiments, from the initial confusion about how to calculate CV to the subsequent parameter tuning, training models, and so on.\n\nHere, I would like to share some of my experimental records. I find some experimental results very strange, and I hope everyone can communicate together!\n\nMy first baseline model is unet+efficientnet-b0, with a score of around 610 for both cv and lb, and all the following experiments were conducted based on this.\n\n1. ALL data augmentation do not works for me. It's vert strange. I have tried simple flipping, rotating, and other operations, as well as advanced methods such as mixup, which can cause a decrease in my CV and LB\n\n2. I tried to fuse images from multiple times to create a 2.5D image, but cv droped...\n\n3. I replaced efficientnet with mit, but the cv was even worse. eficientnet seems to performe very well in this competition\n\n4. I simply tried models such as Segformer and Deeplabv3, the results were even worse. but, I did not perform detailed parameter tuning on them. This competition is like those medical image segmentation tasks in Kaggle, where simple unet can achieve good results...\n\n5. A larger backbone can bring great improvements, similarly, a larger resolution can also bring great improvements. With a larger backbone and a larger resolution, my baseline can be improved to 650+\n\n6.The improvement that pseudo labels is also very significant. I simply tried some methods of pseudo labels and the trained model has a score of nearly 670\n\n7. Post processing is very important, and an appropriate threshold generally increases 2% -3%. And I think maybe we can consider implementing this paragraph in the post processing , but currently I don't have a very suitable idea\n>Some key labeling guidance:\nContrails must contain at least 10 pixels\nAt some time in their life, Contrails must be at least 3x longer than they are wide\nContrails must either appear suddenly or enter from the sides of the image\nContrails should be visible in at least two image\n\n8. I have tried many losses, bce and dice achieve the best results, but the combination of the two has not improved😟\n\n9. Adding auxiliary classification header, failed\n\n10. As the experiment progresses, my CV seems to be somewhat inconsistent with LB...\n\nIs anyone willing to share their model and scores？\n\n| cv | lb |\n| --- | --- |\n| 584 | 589 |\n| 607| 604 |\n| 639 | 643 |\n| 643 | 665 |\n| 648 | 673 |\n| 653 | 675 |\n| 653 | 670 |\n| 656 | 682 |\n| 657 | 678 |\n\n__________________________________________________________________\nupdate:\nso upset...I already get 0.682 lb. these days, I get new 0.681cv, but lb only 0.680...\nAlthough I know it's because there are too few images in lb, the fact that the score hasn't improved is still very disappointing.", "2283543": "First of all thank you @zhuwanglju so much for writing so much details about your solution when you are already Number 1 in the current public LB . It takes lot of courage to do so . Therefore let me also write my details and ask few questions \n\nbaseline model is unet+effb0 , 256 size - LB .643\n\n1. I have always used hard spatial and color augmentations . I have not tried with minimal augmentation . Mixup gives low CV and LB . I dont have a reason why . \n\n2. I tried to fuse frames from the images with 2.5D , my CV dropped and hence I didnt pursue it for now. \n\n3. MIT , Convnext , Densenet etc , drops CV and also LB .\n\n4.DeeplabV3 + and Deeplabv3plus gives lower CV , I have not tested on LB . Unet works well in my experiment \n\n5.Larger backbones are better to certain extent . B1 is better than B0 , but B2 was worse for me than B0 and B1 .So for me not all bigger backbone works . \n\n6.Have not tried Pseudolabel . Are you trying pseudolabel by prediction on train and valid data ? (You need not answer if this gives away very special tricks )\n\n7.Have not tried post processing \n\n8.Adding auxiliary classifier reduces the CV and LB . \n\n9.CV and LB is not consistent for me always . I have a question about your dice score . I use the public notebook one and the CV is always quite high . \nI see you are using the below . I am not sure if you are already passing mask after applying sigmoid and threshold ? or passing only the raw logits ? Are you only passing the positive images into this ? Any guidance would be helpful . \ndef dice_score(y_p, y_t, smooth=1e-6):\n i = torch.sum(y_p * y_t, dim=(2, 3))\n u = torch.sum(y_p, dim=(2, 3)) + torch.sum(y_t, dim=(2, 3))\n score = (2 * i + smooth)/(u + smooth)\n return torch.mean(score)\n\n10. I am using combination of losses . I have not tried only BCE . \n\n11.I am using 256 size images , I saw couple of experiment that 512 images are better . But since it has a longer training time , I got impatient with it for now and only testing with 256 size . Single model score is 0.661 . ", "2283521": "I face same problems, augmentations did not help at all and scaling x2 helps and low threshold also helps", "2283623": "Hi everyone,\n\nMy initial approach was U-Net(Vgg-16 then Efficientnet-b7) with 12 channels as output used in combination with CrossEntropy loss. This pushed my score to public lb 5 in the beginning stages of the competition. I then changed the channels to 1 and applied BinaryCrossEntropy loss and it performed worse. I shifted towards segmentation models (Pytorch) with efficientnet b7. Small gains in dice_loss was directly co-related with the public LB.\n\nBut I found something interesting and made a post about it, deleted it and now I am posting again (Apologies for this, I should have given learning a higher priority over lb).\n\ndice_loss - 0.4447, iou_score - 0.4002 ( VS ) dice_loss - 0.4475, iou_score - 0.4044\n\nThe former brought me to public lb position 13 with score of 0.629 and the latter to public lb position 12 with score of 0.638. Seems like IOU is more of an important metric in conjunction with dice_loss. \n\nSo far i have been playing with a single model and trying to squeeze as much as I can before trying something diverse for future ensembles later on in the competition.\nI have only covered things that are different from my approach to your approach. Things like augmentation, small model etc didnt do the trick for me either. \n\nThanks for the post and happy kaggling guys.\n\nIf any1 is interested in teaming up with me, I am all open to a transparent relation of learning and winning together.", "2283993": "Hey Lyu,\n\nthanks for your very valuable insights! Currently I am also pursuing Unet+effnetb0 from Segmentation Models Pytorch.\nI also experimented with different augmentations (also with my earlier DeepLabV3+ Model) and none of them worked for me. Flipping or Rotating always decreased the LB and CV scores... \nI have two further questions regarding your training loop / hyperparameters:\n1.) Do you guys also use the train and validation folders for train/val split or do you modify them in any other way?\n2.) What callbacks do you use? I experimented with ReduceLROnPlateau as well as LRScheduler with LR decay after each epoch. Also I used a LinearWarmup, but it seems like none of these made any difference regarding my CV and LB Scores.\n\nGood Luck!", "2286917": "Unet+B0,...B6 gets LB0.60-0.62. Albu augments rot90, hv-flips. Could not find other augmentations which improved LB. Interestingly, averaging multiple folds makes only a tiny LB improvement.", "2283912": "As I said [in this thread](https://www.kaggle.com/competitions/google-research-identify-contrails-reduce-global-warming/discussion/413068#2276374), the public LB is based on just 264-292 samples. That is why CV is not consistent with LB.", "2283636": "is segformer allowed here?", "2364678": "thank you all for sharing! useful for me", "2331806": "@zhuwanglju I have a question about point 6 - you have used pseudo labels to relabel the training data?", "2286919": "Has anyone tried to train only on images with non-zero labels? On my todo list.", "2283870": "Very interesting, thank you all for sharing" } }