{ "id": 357375, "title": "Training Loss flutuationg", "url": "/competitions/mayo-clinic-strip-ai/discussion/357375", "author_name": "Altair Farooque", "post_date": "2022-10-04T05:40:25.149000", "votes": 0, "comment_count": 6, "views": 0, "content": "

Hi, folks
\nWell, I'm late in the competition,However I'm training a model everything is fine.
\nBut my training loss is fluctuating even if I use different models like resnet* or effici*.
\nI have used higher batch size like 128, also tried lowering or increasing LR
\nWell, I have used image size of 340 x 340 .
\nGlad to know if there's a bug causing this or that's just the way it is.

", "messages": [ { "id": 1976685, "postDate": "2022-10-07T13:56:48.713Z", "content": "

Try to reduce the number of parameters that you're trying to learn. For example, freeze your backbone and learn only head FC layer. That's exactly what I did to solve this problem

", "rawMarkdown": "Try to reduce the number of parameters that you're trying to learn. For example, freeze your backbone and learn only head FC layer. That's exactly what I did to solve this problem", "votes": 1, "replies": [ { "id": 1977720, "postDate": "2022-10-08T07:45:19.333Z", "content": "

Congratulations 🎉 on runner-up. I would like to see your solution in the near future. Yes, I will update soon, Well I used
\nEfficient net v2_s with 1024 Linear + 256 Linear + final output head.

", "rawMarkdown": "Congratulations 🎉 on runner-up. I would like to see your solution in the near future. Yes, I will update soon, Well I used \nEfficient net v2_s with 1024 Linear + 256 Linear + final output head.", "votes": 1 } ] }, { "id": 1974666, "postDate": "2022-10-06T11:38:58.263Z", "content": "

It would be helpful if you provide more info like: the number of epochs you trained for, the values of LR you used, the loss function

", "rawMarkdown": "It would be helpful if you provide more info like: the number of epochs you trained for, the values of LR you used, the loss function", "replies": [ { "id": 1976660, "postDate": "2022-10-07T13:40:57.543Z", "content": "

Crossentropy , epochs - 10 , lr - 3e-4

", "rawMarkdown": "Crossentropy , epochs - 10 , lr - 3e-4" }, { "id": 1977009, "postDate": "2022-10-07T17:41:41.847Z", "content": "

If you have pretrained weights, reduce the LR more (like to 8e-5), and freeze the pretrained weights, like the other comment said, at least for the first 5 epochs.

", "rawMarkdown": "If you have pretrained weights, reduce the LR more (like to 8e-5), and freeze the pretrained weights, like the other comment said, at least for the first 5 epochs.", "votes": 1 }, { "id": 1977721, "postDate": "2022-10-08T07:45:39.890Z", "content": "

Yeah , i will update soon .

", "rawMarkdown": "Yeah , i will update soon ." } ] }, { "id": 1970483, "postDate": "2022-10-04T05:40:25.150Z", "content": "

Hi, folks
\nWell, I'm late in the competition,However I'm training a model everything is fine.
\nBut my training loss is fluctuating even if I use different models like resnet* or effici*.
\nI have used higher batch size like 128, also tried lowering or increasing LR
\nWell, I have used image size of 340 x 340 .
\nGlad to know if there's a bug causing this or that's just the way it is.

", "rawMarkdown": "Hi, folks\nWell, I'm late in the competition,However I'm training a model everything is fine.\nBut my training loss is fluctuating even if I use different models like resnet* or effici*.\nI have used higher batch size like 128, also tried lowering or increasing LR\nWell, I have used image size of 340 x 340 .\nGlad to know if there's a bug causing this or that's just the way it is." } ], "comments": [ { "id": 1976685, "author_name": "Ilia los", "author_url": "", "post_date": "2022-10-07T13:56:48.713000", "content": "

Try to reduce the number of parameters that you're trying to learn. For example, freeze your backbone and learn only head FC layer. That's exactly what I did to solve this problem

", "votes": 1, "replies": [ { "id": 1977720, "author_name": "Altair Farooque", "author_url": "", "post_date": "2022-10-08T07:45:19.333000", "content": "

Congratulations 🎉 on runner-up. I would like to see your solution in the near future. Yes, I will update soon, Well I used
\nEfficient net v2_s with 1024 Linear + 256 Linear + final output head.

", "votes": 1, "replies": [] } ] }, { "id": 1974666, "author_name": "Fanatic Lizard", "author_url": "", "post_date": "2022-10-06T11:38:58.263000", "content": "

It would be helpful if you provide more info like: the number of epochs you trained for, the values of LR you used, the loss function

", "votes": 0, "replies": [ { "id": 1976660, "author_name": "Altair Farooque", "author_url": "", "post_date": "2022-10-07T13:40:57.543000", "content": "

Crossentropy , epochs - 10 , lr - 3e-4

", "votes": 0, "replies": [] }, { "id": 1977009, "author_name": "Fanatic Lizard", "author_url": "", "post_date": "2022-10-07T17:41:41.847000", "content": "

If you have pretrained weights, reduce the LR more (like to 8e-5), and freeze the pretrained weights, like the other comment said, at least for the first 5 epochs.

", "votes": 1, "replies": [] }, { "id": 1977721, "author_name": "Altair Farooque", "author_url": "", "post_date": "2022-10-08T07:45:39.890000", "content": "

Yeah , i will update soon .

", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1976685": "Try to reduce the number of parameters that you're trying to learn. For example, freeze your backbone and learn only head FC layer. That's exactly what I did to solve this problem", "1974666": "It would be helpful if you provide more info like: the number of epochs you trained for, the values of LR you used, the loss function", "1970483": "Hi, folks\nWell, I'm late in the competition,However I'm training a model everything is fine.\nBut my training loss is fluctuating even if I use different models like resnet* or effici*.\nI have used higher batch size like 128, also tried lowering or increasing LR\nWell, I have used image size of 340 x 340 .\nGlad to know if there's a bug causing this or that's just the way it is." } }