![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5286406%2F5e0ced3d5f1da65a979d86035515c60e%2F2.PNG?generation=1675925073648187&alt=media\")
Hello,
\nI trained the model, but the best threshold is too low.
\nI saw other people's result, but their threshold is almost 0.3~0.5.
\nSo, I think something is wrong in my code.
\nDo you have any idea what and when causes the low threshold?
\nAt best threshold, my model yields the following metric
\n\n\nval_pF1: 0.3170 - val_f1_score: 0.3370 - val_precision: 0.5897 - val_recall: 0.2359 - val_auc: 0.6855 - val_binary_accuracy: 0.9774
\n
Hello,
\nI trained the model, but the best threshold is too low.
\nI saw other people's result, but their threshold is almost 0.3~0.5.
\nSo, I think something is wrong in my code.
\nDo you have any idea what and when causes the low threshold?
\nAt best threshold, my model yields the following metric
\n\n\nval_pF1: 0.3170 - val_f1_score: 0.3370 - val_precision: 0.5897 - val_recall: 0.2359 - val_auc: 0.6855 - val_binary_accuracy: 0.9774
\n
Keep in mind the validation set consists of just 20% of the training data with ~200 positive samples. The best threshold is likely to be overfitted on this small validation set with limited number of positive samples. Additionally, the F1 score by threshold line is almost flat with no clear peak, further indicating the best threshold is likely overfitting.
\nMoreover, the validation predictions are overconfident, since all predictions are near 0/1. The threshold does not influence the binary predictions much, since the threshold will change the predicted value of just a few samples when changed (there are almost no samples with a predicted value between 0.20-0.80).
\nThe threshold would only influence the F1 score when many predictions are not near 0/1.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4433335%2F0711dee7b2441c64c999f802ea92932e%2Ftemp.png?generation=1675932274603709&alt=media\")
You have a few options:
\n1) stick to a threshold of 0.50
\n2) Performs a full 5-Fold run and determine the best out-of-fold best threshold on the full dataset
\n3) Use a different training strategy (i.e. loss function) to have less confident predictions
Thank you for answer! Your answer is very helpful.
", "rawMarkdown": "Thank you for answer! Your answer is very helpful." }, { "id": 2142384, "postDate": "2023-02-13T13:42:55.890Z", "content": "I am new to this competition. Can you please help me? No matter what I do, my public score is always 0.04. I trained resnet model from scratch, with pre-trained weight, with / without validation set. No matter what I do the score is always 0.04. I trained those models in my PC and uploaded the dataset to kaggle and loaded those weight to models. What am I doing wrong, I cannot figure out.
", "rawMarkdown": "I am new to this competition. Can you please help me? No matter what I do, my public score is always 0.04. I trained resnet model from scratch, with pre-trained weight, with / without validation set. No matter what I do the score is always 0.04. I trained those models in my PC and uploaded the dataset to kaggle and loaded those weight to models. What am I doing wrong, I cannot figure out.", "replies": [ { "id": 2142486, "postDate": "2023-02-13T15:05:16.737Z", "content": "One tip I would have is to debug your inference notebook by generating a submission for your training data and comparing the results with your training notebook to validate your inference pipeline.
", "rawMarkdown": "One tip I would have is to debug your inference notebook by generating a submission for your training data and comparing the results with your training notebook to validate your inference pipeline." } ] } ] }, { "id": 2136327, "postDate": "2023-02-09T08:47:55.910Z", "content": "Did you transform the logits into probabilities using sigmoid/softmax?
", "rawMarkdown": "Did you transform the logits into probabilities using sigmoid/softmax?", "replies": [ { "id": 2141877, "postDate": "2023-02-13T06:25:58.003Z", "content": "Yes, I use sigmoid at the last layer of the output.
", "rawMarkdown": "Yes, I use sigmoid at the last layer of the output." } ] } ], "comments": [ { "id": 2136332, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2023-02-09T08:52:39.607000", "content": "Keep in mind the validation set consists of just 20% of the training data with ~200 positive samples. The best threshold is likely to be overfitted on this small validation set with limited number of positive samples. Additionally, the F1 score by threshold line is almost flat with no clear peak, further indicating the best threshold is likely overfitting.
\nMoreover, the validation predictions are overconfident, since all predictions are near 0/1. The threshold does not influence the binary predictions much, since the threshold will change the predicted value of just a few samples when changed (there are almost no samples with a predicted value between 0.20-0.80).
\nThe threshold would only influence the F1 score when many predictions are not near 0/1.
\n
You have a few options:
\n1) stick to a threshold of 0.50
\n2) Performs a full 5-Fold run and determine the best out-of-fold best threshold on the full dataset
\n3) Use a different training strategy (i.e. loss function) to have less confident predictions
Thank you for answer! Your answer is very helpful.
", "votes": 0, "replies": [] }, { "id": 2142384, "author_name": "Rukesh Prajapati", "author_url": "", "post_date": "2023-02-13T13:42:55.890000", "content": "I am new to this competition. Can you please help me? No matter what I do, my public score is always 0.04. I trained resnet model from scratch, with pre-trained weight, with / without validation set. No matter what I do the score is always 0.04. I trained those models in my PC and uploaded the dataset to kaggle and loaded those weight to models. What am I doing wrong, I cannot figure out.
", "votes": 0, "replies": [ { "id": 2142486, "author_name": "Mark Wijkhuizen", "author_url": "", "post_date": "2023-02-13T15:05:16.737000", "content": "One tip I would have is to debug your inference notebook by generating a submission for your training data and comparing the results with your training notebook to validate your inference pipeline.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2136327, "author_name": "Feng Qilong", "author_url": "", "post_date": "2023-02-09T08:47:55.910000", "content": "Did you transform the logits into probabilities using sigmoid/softmax?
", "votes": 0, "replies": [ { "id": 2141877, "author_name": "Ju7on9", "author_url": "", "post_date": "2023-02-13T06:25:58.003000", "content": "Yes, I use sigmoid at the last layer of the output.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "2136198": "Hello, \n\nI trained the model, but the best threshold is too low.\n\nI saw other people's result, but their threshold is almost 0.3~0.5.\n\nSo, I think something is wrong in my code.\n\nDo you have any idea what and when causes the low threshold?\n\nAt best threshold, my model yields the following metric\n\n>val_pF1: 0.3170 - val_f1_score: 0.3370 - val_precision: 0.5897 - val_recall: 0.2359 - val_auc: 0.6855 - val_binary_accuracy: 0.9774\n\n\n", "2136332": "Keep in mind the validation set consists of just 20% of the training data with ~200 positive samples. The best threshold is likely to be overfitted on this small validation set with limited number of positive samples. Additionally, the F1 score by threshold line is almost flat with no clear peak, further indicating the best threshold is likely overfitting.\n\nMoreover, the validation predictions are overconfident, since all predictions are near 0/1. The threshold does not influence the binary predictions much, since the threshold will change the predicted value of just a few samples when changed (there are almost no samples with a predicted value between 0.20-0.80).\n\nThe threshold would only influence the F1 score when many predictions are not near 0/1.\n\n\n\nYou have a few options:\n\n1) stick to a threshold of 0.50\n2) Performs a full 5-Fold run and determine the best out-of-fold best threshold on the full dataset\n3) Use a different training strategy (i.e. loss function) to have less confident predictions", "2136327": "Did you transform the logits into probabilities using sigmoid/softmax?" } }