I am using EfficientNetB3 with AdamW optimizer and OneCycleLR scheduler. Training on ROI images of size 1024x512. Using BCEWithLogitsLoss with pos_weight=20 (also tried focal loss).
\nBut I am still stuck on pF1 score of 0.03-0.04. Is this normal or am I missing something? I would be highly grateful if someone can help me out.
\nThanks in advance!
I am using EfficientNetB3 with AdamW optimizer and OneCycleLR scheduler. Training on ROI images of size 1024x512. Using BCEWithLogitsLoss with pos_weight=20 (also tried focal loss).
\nBut I am still stuck on pF1 score of 0.03-0.04. Is this normal or am I missing something? I would be highly grateful if someone can help me out.
\nThanks in advance!
assume there is no bug in your code or pippline, your problem is \"not able to learn\"
\npF1 score of 0.03-0.04 means the model is predicting all zeros (the majority negative class), i.e. it is not learning anything.
\nin machine learning terms, it cannot perform gradient or perform gradient goes to nowhere.
a few possibility:
\n1) poor initialisation : did you forget to load your pretrain model?
\n2) the input to the pretrain model is not the same: did you forget to to do std and mean normalisation? are the std, mean values correct?
\n3) wrong sampling : you did not feed all train data, you only feed one class, etc
\n4) wrong loss, wrong model configuration: e.g. you forget to do relu before last global pooling, you feed prob in bce loss function which actually accept logit
\n5) wrong hyper paramters : your learning rate is wrong etc ….
\n6) wrong use of software framework : forget to set to train mode? your model tensor graph is not connected, etc …
Hello,
\nYou should resample the data's pos_neg_rate to a normal number or delete some negative samples, which is helpful to model and debug. You can easily check the accuracy and training loss. But that is a shortcoming that is easier to overfit.
Thanks, I will try WeightedRandomSampler to get more positive samples. Will update you once its done :)
", "rawMarkdown": "Thanks, I will try WeightedRandomSampler to get more positive samples. Will update you once its done :)", "votes": 1 }, { "id": 2121104, "postDate": "2023-01-30T04:18:38.280Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2121795, "postDate": "2023-01-30T14:56:04.837Z", "content": "Using a sampler worked well. pF1 straightaway increased to 0.11 on validation set. And upon using a threshold, it gave 0.24. Thanks a lot!
", "rawMarkdown": "Using a sampler worked well. pF1 straightaway increased to 0.11 on validation set. And upon using a threshold, it gave 0.24. Thanks a lot!", "votes": 1 } ] }, { "id": 2121012, "postDate": "2023-01-30T01:16:24.007Z", "content": "I am having the same problem. I will try the tips provided by @hengck23 . Thank both of you! :)
\nI recently published a notebook with the code I am using just in case someone wants to point out what is more likely to be wrong there.
", "rawMarkdown": "I am having the same problem. I will try the tips provided by @hengck23 . Thank both of you! :)\n\nI recently published a notebook with the code I am using just in case someone wants to point out what is more likely to be wrong there. " }, { "id": 2120742, "postDate": "2023-01-29T19:35:24.683Z", "content": "my guess, big image size -> small batch, combined with the imbalanced dataset -> the model doesn't learn. I have a similar problem with big resolution, but I am able to train … something, on a smaller resolution
", "rawMarkdown": "my guess, big image size -> small batch, combined with the imbalanced dataset -> the model doesn't learn. I have a similar problem with big resolution, but I am able to train ... something, on a smaller resolution\n" } ], "comments": [ { "id": 2120611, "author_name": "hengck23", "author_url": "", "post_date": "2023-01-29T17:57:32.483000", "content": "assume there is no bug in your code or pippline, your problem is \"not able to learn\"
\npF1 score of 0.03-0.04 means the model is predicting all zeros (the majority negative class), i.e. it is not learning anything.
\nin machine learning terms, it cannot perform gradient or perform gradient goes to nowhere.
a few possibility:
\n1) poor initialisation : did you forget to load your pretrain model?
\n2) the input to the pretrain model is not the same: did you forget to to do std and mean normalisation? are the std, mean values correct?
\n3) wrong sampling : you did not feed all train data, you only feed one class, etc
\n4) wrong loss, wrong model configuration: e.g. you forget to do relu before last global pooling, you feed prob in bce loss function which actually accept logit
\n5) wrong hyper paramters : your learning rate is wrong etc ….
\n6) wrong use of software framework : forget to set to train mode? your model tensor graph is not connected, etc …
Hello,
\nYou should resample the data's pos_neg_rate to a normal number or delete some negative samples, which is helpful to model and debug. You can easily check the accuracy and training loss. But that is a shortcoming that is easier to overfit.
Thanks, I will try WeightedRandomSampler to get more positive samples. Will update you once its done :)
", "votes": 1, "replies": [] }, { "id": 2121104, "author_name": "", "author_url": "", "post_date": "2023-01-30T04:18:38.280000", "content": "", "votes": 0, "replies": [] }, { "id": 2121795, "author_name": "Aryamaan Thakur", "author_url": "", "post_date": "2023-01-30T14:56:04.837000", "content": "Using a sampler worked well. pF1 straightaway increased to 0.11 on validation set. And upon using a threshold, it gave 0.24. Thanks a lot!
", "votes": 1, "replies": [] } ] }, { "id": 2121012, "author_name": "Enrique Alanís", "author_url": "", "post_date": "2023-01-30T01:16:24.007000", "content": "I am having the same problem. I will try the tips provided by @hengck23 . Thank both of you! :)
\nI recently published a notebook with the code I am using just in case someone wants to point out what is more likely to be wrong there.
", "votes": 0, "replies": [] }, { "id": 2120742, "author_name": "FlaviuPaul", "author_url": "", "post_date": "2023-01-29T19:35:24.683000", "content": "my guess, big image size -> small batch, combined with the imbalanced dataset -> the model doesn't learn. I have a similar problem with big resolution, but I am able to train … something, on a smaller resolution
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2120171": "I am using EfficientNetB3 with AdamW optimizer and OneCycleLR scheduler. Training on ROI images of size 1024x512. Using BCEWithLogitsLoss with pos_weight=20 (also tried focal loss).\nBut I am still stuck on pF1 score of 0.03-0.04. Is this normal or am I missing something? I would be highly grateful if someone can help me out.\nThanks in advance!", "2120611": "assume there is no bug in your code or pippline, your problem is \"not able to learn\"\n\npF1 score of 0.03-0.04 means the model is predicting all zeros (the majority negative class), i.e. it is not learning anything.\nin machine learning terms, it cannot perform gradient or perform gradient goes to nowhere.\n\na few possibility:\n1) poor initialisation : did you forget to load your pretrain model?\n2) the input to the pretrain model is not the same: did you forget to to do std and mean normalisation? are the std, mean values correct?\n3) wrong sampling : you did not feed all train data, you only feed one class, etc \n4) wrong loss, wrong model configuration: e.g. you forget to do relu before last global pooling, you feed prob in bce loss function which actually accept logit\n5) wrong hyper paramters : your learning rate is wrong etc ....\n6) wrong use of software framework : forget to set to train mode? your model tensor graph is not connected, etc ...", "2120359": "Hello,\nYou should resample the data's pos_neg_rate to a normal number or delete some negative samples, which is helpful to model and debug. You can easily check the accuracy and training loss. But that is a shortcoming that is easier to overfit. ", "2121012": "I am having the same problem. I will try the tips provided by @hengck23 . Thank both of you! :)\n\nI recently published a notebook with the code I am using just in case someone wants to point out what is more likely to be wrong there. ", "2120742": "my guess, big image size -> small batch, combined with the imbalanced dataset -> the model doesn't learn. I have a similar problem with big resolution, but I am able to train ... something, on a smaller resolution\n" } }