![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F27a6c63daf65b4887c4d085bfed4502b%2Ftu-tf_efficientnetv2_s.in21k_ft_in1k.png?generation=1691027373549300&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2Fcad797b6d9af207cffd6daaa0d7a5cd3%2Ftimm-resnest50d%201h.png?generation=1691027397604247&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F16cfe3c7530d56fd110b7316c70e9954%2Fdensenet201.png?generation=1691027414948922&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F721d19e04c8a6fccc784b3a0c4a36c10%2Fefficientnet-b4.png?generation=1691027516902642&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F7725c93f18783b0a12f08cc572e8d811%2Fresnet34.png?generation=1691027526480277&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F4f89c53325858a965c690c9aa8d2614f%2Fresnet50.png?generation=1691027537306212&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F71a6521794d095ce099ba0380da51cf8%2Fresnext50_32x4d.png?generation=1691027552120372&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F70f7fc6b52c05815b35a43f76b921d3e%2Fse_resnet50.png?generation=1691027562441275&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2Fddc4b9c755a81092887c03bd64f76b50%2Ftimm-gernet_m.png?generation=1691027574114775&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F88b01199ffe65373ab612110dd5cf42a%2Ftimm-regnetx_064.png?generation=1691027585319764&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F82709af39ef988d2fd0b3fb0e83dadaa%2Ftimm-res2net50_26w_4s.png?generation=1691027598172579&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2Ffba64b20a7cf47afaaf65c362a9fe490%2Ftimm-resnest26d.png?generation=1691027623756614&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2Fb1a57bb320b1c1d19a146b5cfaca5604%2Ftimm-resnest50d_4s2x40d.png?generation=1691027642640303&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2Fa202f6b0921142fb948f48dcbf79652e%2Ftu-timm-repvgg_a2.rvgg_in1k.png.png?generation=1691027671144519&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F41831%2F679e3c2a13d676cf302ac68736d1d2cc%2Fxception.png?generation=1691027680854697&alt=media\")
I wanted to get an idea of how different encoders would perform so I set up an experiment training an smp Unet with the following:
\nI let each one train on the Contrails Ash dataset for about an hour. I let it finish the epoch it was on, but it would not start another epoch if the hour was past. I optimized DiceLoss with AdamW, lr=1e-4, and ReduceLROnPlateau scheduler.
\nIn this experiment, tu-tf_efficientnetv2_s is the clear winner and timm-resnest50d in second. I'm late to this competition and don't have much time, but my plan is to run with tu-tf_efficientnetv2_s, try to optimize its hyperparams and if I have time consider ensembling with timm-resnest50d.
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
I wanted to get an idea of how different encoders would perform so I set up an experiment training an smp Unet with the following:
\nI let each one train on the Contrails Ash dataset for about an hour. I let it finish the epoch it was on, but it would not start another epoch if the hour was past. I optimized DiceLoss with AdamW, lr=1e-4, and ReduceLROnPlateau scheduler.
\nIn this experiment, tu-tf_efficientnetv2_s is the clear winner and timm-resnest50d in second. I'm late to this competition and don't have much time, but my plan is to run with tu-tf_efficientnetv2_s, try to optimize its hyperparams and if I have time consider ensembling with timm-resnest50d.
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
Nice work man, I agree, effv2 are good encoders, but resnest also, try a 101e, with minor augmentations, you can reach 0.64 dice in 20 epochs on the valid set. Transformers work too, but the training time it's much higher.
", "rawMarkdown": "Nice work man, I agree, effv2 are good encoders, but resnest also, try a 101e, with minor augmentations, you can reach 0.64 dice in 20 epochs on the valid set. Transformers work too, but the training time it's much higher.", "votes": 4 }, { "id": 2371625, "postDate": "2023-08-03T08:01:39.183Z", "content": "Thanks, that's very helpful!
\nI slightly disagree with your conclusion though.
\nI think ResNeSt looks stronger here, it seems like dice is still improving a lot if you continued training,
\nwhile tu-tf_efficientnetv2_s is already showing diminishing returns.
\nA reason could be that much less epochs (4 vs 7) fit into your 1 hour run, which could mean that ResNeSt is further away from overfitting, having seen each datapoint less often.
@zacstewart Is this only for Frame 5th(the one on which the masks are provided)
\nEven a simple Unet(4 en-de) trained from scratch doesn't go below 0.4 dice loss on validation data, trained only on 5th frame.
", "rawMarkdown": "@zacstewart Is this only for Frame 5th(the one on which the masks are provided)\n\n\nEven a simple Unet(4 en-de) trained from scratch doesn't go below 0.4 dice loss on validation data, trained only on 5th frame.", "replies": [ { "id": 2377879, "postDate": "2023-08-07T11:15:26.977Z", "content": "Yes, I used this dataset, which is only the labeled frame, and Ash false-color images, not all IR bands.
", "rawMarkdown": "Yes, I used [this dataset](https://www.kaggle.com/datasets/shashwatraman/contrails-images-ash-color), which is only the labeled frame, and Ash false-color images, not all IR bands.", "votes": 1 } ] }, { "id": 2374655, "postDate": "2023-08-05T06:20:53.943Z", "content": "thanks you
", "rawMarkdown": "thanks you" }, { "id": 2371309, "postDate": "2023-08-03T04:18:00.237Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ], "comments": [ { "id": 2371918, "author_name": "Maximiliano Diaz Battan", "author_url": "", "post_date": "2023-08-03T10:52:01.677000", "content": "Nice work man, I agree, effv2 are good encoders, but resnest also, try a 101e, with minor augmentations, you can reach 0.64 dice in 20 epochs on the valid set. Transformers work too, but the training time it's much higher.
", "votes": 4, "replies": [] }, { "id": 2371625, "author_name": "Raki", "author_url": "", "post_date": "2023-08-03T08:01:39.183000", "content": "Thanks, that's very helpful!
\nI slightly disagree with your conclusion though.
\nI think ResNeSt looks stronger here, it seems like dice is still improving a lot if you continued training,
\nwhile tu-tf_efficientnetv2_s is already showing diminishing returns.
\nA reason could be that much less epochs (4 vs 7) fit into your 1 hour run, which could mean that ResNeSt is further away from overfitting, having seen each datapoint less often.
@zacstewart Is this only for Frame 5th(the one on which the masks are provided)
\nEven a simple Unet(4 en-de) trained from scratch doesn't go below 0.4 dice loss on validation data, trained only on 5th frame.
", "votes": 0, "replies": [ { "id": 2377879, "author_name": "zacstewart", "author_url": "", "post_date": "2023-08-07T11:15:26.977000", "content": "Yes, I used this dataset, which is only the labeled frame, and Ash false-color images, not all IR bands.
", "votes": 1, "replies": [] } ] }, { "id": 2374655, "author_name": "ahao1759828826", "author_url": "", "post_date": "2023-08-05T06:20:53.943000", "content": "thanks you
", "votes": 0, "replies": [] }, { "id": 2371309, "author_name": "Darshan Patel", "author_url": "", "post_date": "2023-08-03T04:18:00.237000", "content": "Thank you!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2371203": "I wanted to get an idea of how different encoders would perform so I set up an experiment training an smp Unet with the following:\n\n- tu-tf_efficientnetv2_s\n- timm-resnest50d\n- efficientnet-b4\n- timm-gernet_m\n- timm-resnest50d_4s2x40d\n- timm-resnest26d\n- timm-regnetx_064\n- tu-timm-repvgg_a2\n- densenet201\n- resnet50\n- resnet34\n- resnext50_32x4d\n- se_resnet50\n- timm-res2net50_26w_4s\n- xception\n\nI let each one train on the [Contrails Ash](https://www.kaggle.com/datasets/shashwatraman/contrails-images-ash-color) dataset for about an hour. I let it finish the epoch it was on, but it would not start another epoch if the hour was past. I optimized DiceLoss with AdamW, lr=1e-4, and ReduceLROnPlateau scheduler.\n\nIn this experiment, tu-tf_efficientnetv2_s is the clear winner and timm-resnest50d in second. I'm late to this competition and don't have much time, but my plan is to run with tu-tf_efficientnetv2_s, try to optimize its hyperparams and if I have time consider ensembling with timm-resnest50d.\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n", "2371918": "Nice work man, I agree, effv2 are good encoders, but resnest also, try a 101e, with minor augmentations, you can reach 0.64 dice in 20 epochs on the valid set. Transformers work too, but the training time it's much higher.", "2371625": "Thanks, that's very helpful!\n\nI slightly disagree with your conclusion though.\nI think ResNeSt looks stronger here, it seems like dice is still improving a lot if you continued training, \nwhile tu-tf_efficientnetv2_s is already showing diminishing returns. \nA reason could be that much less epochs (4 vs 7) fit into your 1 hour run, which could mean that ResNeSt is further away from overfitting, having seen each datapoint less often.", "2376583": "@zacstewart Is this only for Frame 5th(the one on which the masks are provided)\n\n\nEven a simple Unet(4 en-de) trained from scratch doesn't go below 0.4 dice loss on validation data, trained only on 5th frame.", "2374655": "thanks you", "2371309": "Thank you!" } }