![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3747152%2F03dac235a6cf386bd242942fbec14d4d%2FSoft%20Labels.png?generation=1691232543701634&alt=media\")
I iterated through a range of learning rates for BCE loss and Dice Loss and compared both with the best lrs.
\nThe difference for me was enormous, I have the feeling that I made some mistake in the implementation, as I don't see how the difference between these losses could be so large.
weight = torch.FloatTensor([10.0]).cuda() # Move to GPU if available
\ncriterion = smp.losses.SoftBCEWithLogitsLoss(pos_weight=weight)
\nvs
\ncriterion = smp.losses.DiceLoss(mode=\"binary\", smooth=1.0)
https://wandb.ai/ralf-c-kinkel/Contrail/reports/Dice-vs-WBCE--Vmlldzo1MDQ0MDQ2
\nEDIT: I found my error and now WBCE looks better (by around +0.01), my mistake was that I did not adjust the threshold. When optimizing with WBCE the model assigns overly high confidence to contrails. When training a simple model with Dice Loss (ResNeSt 26d) I get around 0.62 val dice. When training with WBCE, weight=10 and iterating over the thresholds my best threshold at 0.8 gets 0.632. Some of that comes from overfitting the threshold, but even at 0.75 and 0.85 threshold Val Dice still almost at 0.63 (still +0.01 vs Dice Loss trained).
\nEDIT: I integrated soft-labels, meaning that I define groundtruth as mean of individual masks, I attached an example. It turns out that Dice Loss benefits from that much more than WBCE, though only with a high threshold (0.99+ was ideal for me), I will make a post later today where I look at interaction of Dice Loss and these averaged labels in more detail. My Val Dice is now at 0.65 with this setup for ResNest26d.
During the initials stages of experimenting I have run some hyper-parameter optimization and found that for bce loss pos_weight selection is very important because there is a narrow optimum around ~10 value. And it seems that it yields worse performance vs only dice or sum of bce and dice. However, for sum of bce and dice with 0.5 weight each the optimum is lower (~1e-2) and seems to be much more flatten.
\nSee the figure below: 4 different models, bce / bce + dice loss and pos_weight in [1e-3, 1e3] interval selected randomly. I highlighted the bce by red color and bce + dice by green color, different models seem to behave similarly.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2190976%2Fc7d9a37e2087cbdb41fda4e3da915a87%2Fbest_vdice_vs_pos_weight.png?generation=1691145428101753&alt=media\")
So my experiment results also suggest that bce is worse than dice.
", "rawMarkdown": "During the initials stages of experimenting I have run some hyper-parameter optimization and found that for bce loss pos_weight selection is very important because there is a narrow optimum around ~10 value. And it seems that it yields worse performance vs only dice or sum of bce and dice. However, for sum of bce and dice with 0.5 weight each the optimum is lower (~1e-2) and seems to be much more flatten.\n\nSee the figure below: 4 different models, bce / bce + dice loss and pos_weight in [1e-3, 1e3] interval selected randomly. I highlighted the bce by red color and bce + dice by green color, different models seem to behave similarly.\n\n\n\nSo my experiment results also suggest that bce is worse than dice.", "votes": 3, "replies": [ { "id": 2373611, "postDate": "2023-08-04T10:41:29.423Z", "content": "Note: there is only few points for bce loss option, only single fold (out of 5) and low number of epochs (20) was used, so decide it for yourself whether to trust it
", "rawMarkdown": "Note: there is only few points for bce loss option, only single fold (out of 5) and low number of epochs (20) was used, so decide it for yourself whether to trust it" } ] }, { "id": 2371888, "postDate": "2023-08-03T10:19:35.123Z", "content": "I iterated through a range of learning rates for BCE loss and Dice Loss and compared both with the best lrs.
\nThe difference for me was enormous, I have the feeling that I made some mistake in the implementation, as I don't see how the difference between these losses could be so large.
weight = torch.FloatTensor([10.0]).cuda() # Move to GPU if available
\ncriterion = smp.losses.SoftBCEWithLogitsLoss(pos_weight=weight)
\nvs
\ncriterion = smp.losses.DiceLoss(mode=\"binary\", smooth=1.0)
https://wandb.ai/ralf-c-kinkel/Contrail/reports/Dice-vs-WBCE--Vmlldzo1MDQ0MDQ2
\nEDIT: I found my error and now WBCE looks better (by around +0.01), my mistake was that I did not adjust the threshold. When optimizing with WBCE the model assigns overly high confidence to contrails. When training a simple model with Dice Loss (ResNeSt 26d) I get around 0.62 val dice. When training with WBCE, weight=10 and iterating over the thresholds my best threshold at 0.8 gets 0.632. Some of that comes from overfitting the threshold, but even at 0.75 and 0.85 threshold Val Dice still almost at 0.63 (still +0.01 vs Dice Loss trained).
\nEDIT: I integrated soft-labels, meaning that I define groundtruth as mean of individual masks, I attached an example. It turns out that Dice Loss benefits from that much more than WBCE, though only with a high threshold (0.99+ was ideal for me), I will make a post later today where I look at interaction of Dice Loss and these averaged labels in more detail. My Val Dice is now at 0.65 with this setup for ResNest26d.
Here is the Dice Loss trained vs WBCE trained model global val dice comparison for different thresholds, while WBCE peaks a bit higher, the Dice Loss trained model is much more stable with non-optimal thresholds.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3747152%2F18734ca960d92fd96cde750431538402%2FDice%20vs%20WBCE%20trained%20threshold.png?generation=1691167916813679&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13974146%2F612e6fed31e39f1d44558714843fbb38%2F2023-08-04%2015.24.43.png?generation=1691133924754866&alt=media\")
During the initials stages of experimenting I have run some hyper-parameter optimization and found that for bce loss pos_weight selection is very important because there is a narrow optimum around ~10 value. And it seems that it yields worse performance vs only dice or sum of bce and dice. However, for sum of bce and dice with 0.5 weight each the optimum is lower (~1e-2) and seems to be much more flatten.
\nSee the figure below: 4 different models, bce / bce + dice loss and pos_weight in [1e-3, 1e3] interval selected randomly. I highlighted the bce by red color and bce + dice by green color, different models seem to behave similarly.
\n
So my experiment results also suggest that bce is worse than dice.
", "votes": 3, "replies": [ { "id": 2373611, "author_name": "Mikhail Kotyushev", "author_url": "", "post_date": "2023-08-04T10:41:29.423000", "content": "Note: there is only few points for bce loss option, only single fold (out of 5) and low number of epochs (20) was used, so decide it for yourself whether to trust it
", "votes": 0, "replies": [] } ] }, { "id": 2374085, "author_name": "Raki", "author_url": "", "post_date": "2023-08-04T16:52:04.727000", "content": "Here is the Dice Loss trained vs WBCE trained model global val dice comparison for different thresholds, while WBCE peaks a bit higher, the Dice Loss trained model is much more stable with non-optimal thresholds.