Hi everyone, until now I got a loss around 0.56 but the accuracy is very low (0.73).
\nThanks in advance
", "messages": [ { "id": 1946046, "postDate": "2022-09-19T15:21:23.137Z", "content": ">very low
\n>0.73
\nAre you even sure that you aren't overfitted yet
", "rawMarkdown": "*>very low*\n\n*>0.73*\n\nAre you even sure that you aren't overfitted yet", "votes": 5 }, { "id": 1945923, "postDate": "2022-09-19T13:48:17.560Z", "content": "Hi everyone, until now I got a loss around 0.56 but the accuracy is very low (0.73).
\nThanks in advance
", "rawMarkdown": "Hi everyone, until now I got a loss around 0.56 but the accuracy is very low (0.73).\n\nThanks in advance", "votes": 2 }, { "id": 1954335, "postDate": "2022-09-25T06:48:06.270Z", "content": "I use 5 fold cross-validation (stratify target and non-overlapping patient_ids). This is my best model. Not sure this is good or not…
\n{\n \"fold_scores\": {\n \"fold1\": {\n \"accuracy\": 0.7450980392156863,\n \"roc_auc\": 0.6655420602789024,\n \"log_loss\": 0.5360595129819867\n },\n \"fold2\": {\n \"accuracy\": 0.6948051948051948,\n \"roc_auc\": 0.5798369457148538,\n \"log_loss\": 0.6151186826747733\n },\n \"fold3\": {\n \"accuracy\": 0.7635135135135135,\n \"roc_auc\": 0.6552579365079365,\n \"log_loss\": 0.5286322120778464\n },\n \"fold4\": {\n \"accuracy\": 0.74,\n \"roc_auc\": 0.6325799955247259,\n \"log_loss\": 0.5594933565209309\n },\n \"fold5\": {\n \"accuracy\": 0.7181208053691275,\n \"roc_auc\": 0.7004329004329004,\n \"log_loss\": 0.5591903770679996\n }\n },\n \"oof_scores\": {\n \"accuracy\": 0.7320954907161804,\n \"roc_auc\": 0.6483851310176723,\n \"log_loss\": 0.5599818563222174\n }\n}\nAccuracy doesn't mean much in this competition. I have an accuracy of about 0.9. The validation model confidently determines the image class (adjusted for accuracy). But here is another metric, good accuracy only harms it. How it is calculated, I still do not understand.
", "rawMarkdown": "Accuracy doesn't mean much in this competition. I have an accuracy of about 0.9. The validation model confidently determines the image class (adjusted for accuracy). But here is another metric, good accuracy only harms it. How it is calculated, I still do not understand.", "replies": [ { "id": 1958878, "postDate": "2022-09-27T17:22:54.553Z", "content": "You mean the scoring metric? It's a variant on the weighted log loss (weighted binary crossentropy). The formula is shown on the dataset page
", "rawMarkdown": "You mean the scoring metric? It's a variant on the weighted log loss (weighted binary crossentropy). The formula is shown on the dataset page ", "votes": 1 }, { "id": 1958914, "postDate": "2022-09-27T17:55:01.823Z", "content": "Yes, good accuracy worsens the metric of this competition.
", "rawMarkdown": "Yes, good accuracy worsens the metric of this competition.", "votes": 1 }, { "id": 1959813, "postDate": "2022-09-28T10:39:31.477Z", "content": "Keep in mind that the test set for the public score is really small and that the current scores on the LB can be totally wrong. You should try to optimize your model with cross-validation for a better idea of how it'll fare in the end. :)
", "rawMarkdown": "Keep in mind that the test set for the public score is really small and that the current scores on the LB can be totally wrong. You should try to optimize your model with cross-validation for a better idea of how it'll fare in the end. :)", "votes": 1 }, { "id": 1960644, "postDate": "2022-09-28T17:33:07.280Z", "content": "Thanks for the advice, next time I'll definitely do it that way))
", "rawMarkdown": "Thanks for the advice, next time I'll definitely do it that way))" } ] }, { "id": 1952138, "postDate": "2022-09-23T13:17:58.687Z", "content": "I have got 0.68 with a small homemade cnn with 21K parameters, but without taking into account the images of the same person.
", "rawMarkdown": "I have got 0.68 with a small homemade cnn with 21K parameters, but without taking into account the images of the same person." }, { "id": 1947252, "postDate": "2022-09-20T11:32:05.610Z", "content": "Public test took 9 minutes
", "rawMarkdown": "Public test took 9 minutes", "replies": [ { "id": 1947572, "postDate": "2022-09-20T14:34:25.940Z", "content": "There's ~70x the images in the hidden set compared to the public set. We don't know the sizes of them, so that affects the public execution time too. You'll likely have to get the inference time below 6min to get it passing, I think.
", "rawMarkdown": "There's ~70x the images in the hidden set compared to the public set. We don't know the sizes of them, so that affects the public execution time too. You'll likely have to get the inference time below 6min to get it passing, I think. " } ] }, { "id": 1947227, "postDate": "2022-09-20T11:05:49.853Z", "content": "Anyway my submission failed by timeout ☹️
", "rawMarkdown": "Anyway my submission failed by timeout ☹️", "replies": [ { "id": 1947251, "postDate": "2022-09-20T11:28:58.940Z", "content": "How long did it take to run on the public test set?
", "rawMarkdown": "How long did it take to run on the public test set?" } ] }, { "id": 1946597, "postDate": "2022-09-19T23:52:18.783Z", "content": "Hi, I want to clarify that the value 0.73 is a mean of 5 folders, I have accuracy of 0.77, 0.84, 0.69, 0.63 , 0.72
", "rawMarkdown": "Hi, I want to clarify that the value 0.73 is a mean of 5 folders, I have accuracy of 0.77, 0.84, 0.69, 0.63 , 0.72", "replies": [ { "id": 1947009, "postDate": "2022-09-20T07:56:40.350Z", "content": "\n\n0.84
\n
oh boy
", "rawMarkdown": ">0.84\n\noh boy", "votes": 1 }, { "id": 1947110, "postDate": "2022-09-20T09:08:53.320Z", "content": "Weighted or unweighted accuracies and how have you addressed class imbalance? Just a single metric like this doesn't tell us a lot about your workflow, and pretty much every step in the process can affect it. 0.84 is not bad at all :)
\nSince there's a fair bit of wiggle room between your fold accuracies, a couple of things that come to mind are:
\nAgain, there's a million things that can cause this, but these are some common reasons that might be affecting your workflow. If you share your notebook, we can look into it :)
", "rawMarkdown": "Weighted or unweighted accuracies and how have you addressed class imbalance? Just a single metric like this doesn't tell us a lot about your workflow, and pretty much every step in the process can affect it. 0.84 is not bad at all :)\n\nSince there's a fair bit of wiggle room between your fold accuracies, a couple of things that come to mind are:\n\n- Increase your fold sizes (and hence the support for the metrics)\n- Lower your learning rates\n- Check for overfitting\n\nAgain, there's a million things that can cause this, but these are some common reasons that might be affecting your workflow. If you share your notebook, we can look into it :)", "votes": 1 }, { "id": 1953172, "postDate": "2022-09-24T09:20:07.140Z", "content": "What is your AUC-ROC score?
", "rawMarkdown": "What is your AUC-ROC score?" }, { "id": 1953753, "postDate": "2022-09-24T17:56:48.530Z", "content": "0.57 is my AUC ROC
", "rawMarkdown": "0.57 is my AUC ROC", "votes": 1 }, { "id": 1953812, "postDate": "2022-09-24T18:45:11.810Z", "content": "I was able to get up to a 0.49(avg) AUC score leading to 0.6 lb. 0.57 is a questionable score.
\nAlso, I would like to argue whether accuracy is a good way to judge the performance of our models for this comp? or even AUC-ROC for that matter?
An accuracy of 0.73 isn't very low, assuming it's weighted, and that you've accounted for the imbalance in the set. Apparently, many people in the forum here struggle to get it past 0.5.
\nThat being said, it's unknown how easily discernable these images are (hence, there's a competition to chart the territory), so nobody will be able to tell what a good accuracy for this dataset is. Just keep doing your best and good luck :)
>very low
\n>0.73
\nAre you even sure that you aren't overfitted yet
", "votes": 5, "replies": [] }, { "id": 1954335, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-25T06:48:06.270000", "content": "I use 5 fold cross-validation (stratify target and non-overlapping patient_ids). This is my best model. Not sure this is good or not…
\n{\n \"fold_scores\": {\n \"fold1\": {\n \"accuracy\": 0.7450980392156863,\n \"roc_auc\": 0.6655420602789024,\n \"log_loss\": 0.5360595129819867\n },\n \"fold2\": {\n \"accuracy\": 0.6948051948051948,\n \"roc_auc\": 0.5798369457148538,\n \"log_loss\": 0.6151186826747733\n },\n \"fold3\": {\n \"accuracy\": 0.7635135135135135,\n \"roc_auc\": 0.6552579365079365,\n \"log_loss\": 0.5286322120778464\n },\n \"fold4\": {\n \"accuracy\": 0.74,\n \"roc_auc\": 0.6325799955247259,\n \"log_loss\": 0.5594933565209309\n },\n \"fold5\": {\n \"accuracy\": 0.7181208053691275,\n \"roc_auc\": 0.7004329004329004,\n \"log_loss\": 0.5591903770679996\n }\n },\n \"oof_scores\": {\n \"accuracy\": 0.7320954907161804,\n \"roc_auc\": 0.6483851310176723,\n \"log_loss\": 0.5599818563222174\n }\n}\nAccuracy doesn't mean much in this competition. I have an accuracy of about 0.9. The validation model confidently determines the image class (adjusted for accuracy). But here is another metric, good accuracy only harms it. How it is calculated, I still do not understand.
", "votes": 0, "replies": [ { "id": 1958878, "author_name": "David Landup", "author_url": "", "post_date": "2022-09-27T17:22:54.553000", "content": "You mean the scoring metric? It's a variant on the weighted log loss (weighted binary crossentropy). The formula is shown on the dataset page
", "votes": 1, "replies": [] }, { "id": 1958914, "author_name": "Zaakcii Ru", "author_url": "", "post_date": "2022-09-27T17:55:01.823000", "content": "Yes, good accuracy worsens the metric of this competition.
", "votes": 1, "replies": [] }, { "id": 1959813, "author_name": "David Landup", "author_url": "", "post_date": "2022-09-28T10:39:31.477000", "content": "Keep in mind that the test set for the public score is really small and that the current scores on the LB can be totally wrong. You should try to optimize your model with cross-validation for a better idea of how it'll fare in the end. :)
", "votes": 1, "replies": [] }, { "id": 1960644, "author_name": "Zaakcii Ru", "author_url": "", "post_date": "2022-09-28T17:33:07.280000", "content": "Thanks for the advice, next time I'll definitely do it that way))
", "votes": 0, "replies": [] } ] }, { "id": 1952138, "author_name": "Pierre Tisseur", "author_url": "", "post_date": "2022-09-23T13:17:58.687000", "content": "I have got 0.68 with a small homemade cnn with 21K parameters, but without taking into account the images of the same person.
", "votes": 0, "replies": [] }, { "id": 1947252, "author_name": "Pablo Larrosa", "author_url": "", "post_date": "2022-09-20T11:32:05.610000", "content": "Public test took 9 minutes
", "votes": 0, "replies": [ { "id": 1947572, "author_name": "David Landup", "author_url": "", "post_date": "2022-09-20T14:34:25.940000", "content": "There's ~70x the images in the hidden set compared to the public set. We don't know the sizes of them, so that affects the public execution time too. You'll likely have to get the inference time below 6min to get it passing, I think.
", "votes": 0, "replies": [] } ] }, { "id": 1947227, "author_name": "Pablo Larrosa", "author_url": "", "post_date": "2022-09-20T11:05:49.853000", "content": "Anyway my submission failed by timeout ☹️
", "votes": 0, "replies": [ { "id": 1947251, "author_name": "David Landup", "author_url": "", "post_date": "2022-09-20T11:28:58.940000", "content": "How long did it take to run on the public test set?
", "votes": 0, "replies": [] } ] }, { "id": 1946597, "author_name": "Pablo Larrosa", "author_url": "", "post_date": "2022-09-19T23:52:18.783000", "content": "Hi, I want to clarify that the value 0.73 is a mean of 5 folders, I have accuracy of 0.77, 0.84, 0.69, 0.63 , 0.72
", "votes": 0, "replies": [ { "id": 1947009, "author_name": "majoraregalia", "author_url": "", "post_date": "2022-09-20T07:56:40.350000", "content": "\n\n0.84
\n
oh boy
", "votes": 1, "replies": [] }, { "id": 1947110, "author_name": "David Landup", "author_url": "", "post_date": "2022-09-20T09:08:53.320000", "content": "Weighted or unweighted accuracies and how have you addressed class imbalance? Just a single metric like this doesn't tell us a lot about your workflow, and pretty much every step in the process can affect it. 0.84 is not bad at all :)
\nSince there's a fair bit of wiggle room between your fold accuracies, a couple of things that come to mind are:
\nAgain, there's a million things that can cause this, but these are some common reasons that might be affecting your workflow. If you share your notebook, we can look into it :)
", "votes": 1, "replies": [] }, { "id": 1953172, "author_name": "ANANDIYA SHEEL DIWAN", "author_url": "", "post_date": "2022-09-24T09:20:07.140000", "content": "What is your AUC-ROC score?
", "votes": 0, "replies": [] }, { "id": 1953753, "author_name": "Pablo Larrosa", "author_url": "", "post_date": "2022-09-24T17:56:48.530000", "content": "0.57 is my AUC ROC
", "votes": 1, "replies": [] }, { "id": 1953812, "author_name": "ANANDIYA SHEEL DIWAN", "author_url": "", "post_date": "2022-09-24T18:45:11.810000", "content": "I was able to get up to a 0.49(avg) AUC score leading to 0.6 lb. 0.57 is a questionable score.
\nAlso, I would like to argue whether accuracy is a good way to judge the performance of our models for this comp? or even AUC-ROC for that matter?
An accuracy of 0.73 isn't very low, assuming it's weighted, and that you've accounted for the imbalance in the set. Apparently, many people in the forum here struggle to get it past 0.5.
\nThat being said, it's unknown how easily discernable these images are (hence, there's a competition to chart the territory), so nobody will be able to tell what a good accuracy for this dataset is. Just keep doing your best and good luck :)