Why do my models have lower performances than sample submission? it's a weird competition
\nin local environment, anyway it learns the pattern.
\nbut when i submit it i just score 0.6 or 0.7
Thanks for the all the discussions! Good Luck For Everyone Here!
", "messages": [ { "id": 1953430, "postDate": "2022-09-24T13:33:33.223Z", "content": "Why do my models have lower performances than sample submission? it's a weird competition
\nin local environment, anyway it learns the pattern.
\nbut when i submit it i just score 0.6 or 0.7
Thanks for the all the discussions! Good Luck For Everyone Here!
", "rawMarkdown": "Why do my models have lower performances than sample submission? it's a weird competition \nin local environment, anyway it learns the pattern. \nbut when i submit it i just score 0.6 or 0.7 \nit's weird!! \n-----------------------------------------------------------------------------\n Thanks for the all the discussions! Good Luck For Everyone Here! \n\n", "votes": 8 }, { "id": 1970621, "postDate": "2022-10-04T07:02:38.270Z", "content": "I am really not sure what is going on in this competition, I just started three days ago ! I did a basic training and got 0.4 at first shot. When I trained with \"focus\" and deeply I got a score of 1.0 and another try score of 0.6! I didn't really do much and will most likely go down sliding in the private board ! but I don't get my results ! … my 0.4 is a lucky shot maybe !
", "rawMarkdown": "I am really not sure what is going on in this competition, I just started three days ago ! I did a basic training and got 0.4 at first shot. When I trained with \"focus\" and deeply I got a score of 1.0 and another try score of 0.6! I didn't really do much and will most likely go down sliding in the private board ! but I don't get my results ! ... my 0.4 is a lucky shot maybe ! ", "votes": 4, "replies": [ { "id": 1970627, "postDate": "2022-10-04T07:13:03.680Z", "content": "Can you share some details like what are your oof scores, what model are you using, tiles or whole image etc.?
", "rawMarkdown": "Can you share some details like what are your oof scores, what model are you using, tiles or whole image etc.? " }, { "id": 1970747, "postDate": "2022-10-04T09:08:25.013Z", "content": "There isn't much signal, as I originally thought. The 0.4 is likely a lucky shot, unfortunately :(
", "rawMarkdown": "There isn't much signal, as I originally thought. The 0.4 is likely a lucky shot, unfortunately :(", "votes": 2 }, { "id": 1971406, "postDate": "2022-10-04T15:28:52.507Z", "content": "@asalhi thanks for sharing this. All the best.
", "rawMarkdown": "@asalhi thanks for sharing this. All the best." }, { "id": 1971450, "postDate": "2022-10-04T16:09:08.207Z", "content": "@asalhi please tell us more details about lucky shot)
", "rawMarkdown": "@asalhi please tell us more details about lucky shot)" }, { "id": 1971471, "postDate": "2022-10-04T16:23:17.820Z", "content": "I prefer not to go into details now, till the competition ends. But I am pretty sure it's a really super lucky shot that I didn't see coming, it will surely not replace hard efforts done by you and others who surely spent hours on this competition, I just passed by a few days ago and had some free time and was struggling in another competition so I said let's try this!
\nI'm pretty sure my lucky shot will shake badly! since my new tests are far from lucky. I would say. However I will choose it as one of the final submissions, I will share how to regenerate this lucky shot once the competition ends, untill then best of luck to you all! I am just “a passing by guy” in this competition …
", "rawMarkdown": "I prefer not to go into details now, till the competition ends. But I am pretty sure it's a really super lucky shot that I didn't see coming, it will surely not replace hard efforts done by you and others who surely spent hours on this competition, I just passed by a few days ago and had some free time and was struggling in another competition so I said let's try this! \n\nI'm pretty sure my lucky shot will shake badly! since my new tests are far from lucky. I would say. However I will choose it as one of the final submissions, I will share how to regenerate this lucky shot once the competition ends, untill then best of luck to you all! I am just “a passing by guy” in this competition ... \n", "votes": 4 }, { "id": 1971487, "postDate": "2022-10-04T16:34:04.507Z", "content": "@asalhi I'm sure we'll all learn a lot by the time we are done analyzing solutions for this competition. Looking forward to understand what worked, and also how can we go wrong in such cases. Hope you do not slide in the rankings.
", "rawMarkdown": "@asalhi I'm sure we'll all learn a lot by the time we are done analyzing solutions for this competition. Looking forward to understand what worked, and also how can we go wrong in such cases. Hope you do not slide in the rankings.", "votes": 1 }, { "id": 1971498, "postDate": "2022-10-04T16:40:24.823Z", "content": "I hope not to go sliding, However I have a feeling that it will be a rollercoaster hhhh
", "rawMarkdown": "I hope not to go sliding, However I have a feeling that it will be a rollercoaster hhhh", "votes": 1 } ] }, { "id": 1972166, "postDate": "2022-10-05T03:05:15.953Z", "content": "Well, I still think that anything that produces score less than 0.6-0.5 is a product of a mere luck .
", "rawMarkdown": "Well, I still think that anything that produces score less than 0.6-0.5 is a product of a mere luck ~~or overfitting that happens in such specific conditions as a lack of signal in the data~~.", "votes": 1 }, { "id": 1971445, "postDate": "2022-10-04T16:04:57.503Z", "content": "Any last minute guesses what the medal range will be for private LB?
\nI think low but possible chance sample submission may make it into bronze lol
\nMaybe 0.5 Gold to 0.7 Bronze?
", "rawMarkdown": "Any last minute guesses what the medal range will be for private LB?\n\nI think low but possible chance sample submission may make it into bronze lol\n\nMaybe 0.5 Gold to 0.7 Bronze?", "votes": 1, "replies": [ { "id": 1971628, "postDate": "2022-10-04T17:44:11.163Z", "content": "we can not know about it as discussed above
", "rawMarkdown": "we can not know about it as discussed above" }, { "id": 1971713, "postDate": "2022-10-04T18:21:29.543Z", "content": "I think private leaderboard will be between 0.63 and 0.75. 0.63 is the gold zone, 0.64 is silver.
", "rawMarkdown": "I think private leaderboard will be between 0.63 and 0.75. 0.63 is the gold zone, 0.64 is silver.", "votes": 4 } ] }, { "id": 1965228, "postDate": "2022-10-01T07:44:58.250Z", "content": "def binary_weighted_log_loss_forcewithme(y_true, y_pred):\n\n log_loss_positive = log_loss(y_true, y_pred)\n log_loss_negative = log_loss(y_true, 1 - y_pred)\n log_loss_weighted = 0.5 * log_loss_positive + 0.5 * log_loss_negative\n\n return log_loss_positive, log_loss_negative, log_loss_weighted\n\n\ndef binary_weighted_log_loss_theo(y_true, y_pred):\n\n y_true_positive = y_true == 1\n y_true_negative = y_true == 0\n\n log_loss_positive = log_loss(y_true[y_true_positive], y_pred[y_true_positive], labels=[0, 1])\n log_loss_negative = log_loss(y_true[y_true_negative], y_pred[y_true_negative], labels=[0, 1])\n log_loss_weighted = 0.5 * log_loss_positive + 0.5 * log_loss_negative\n\n return log_loss_positive, log_loss_negative, log_loss_weighted\nThese two functions yield different results. This is my final blend score. I have no idea which one is the correct implementation so I use both of them for now.
\nBlend - {'accuracy': 0.6286472148541115, 'roc_auc': 0.66280723136299, 'log_loss_positive_forcewithme': 0.6593462375631115, 'log_loss_negative_forcewithme': 0.7662855417748091, 'log_loss_weighted_forcewithme': 0.7128158896689603, 'log_loss_positive_theo': 0.6395257262674887, 'log_loss_negative_theo': 0.6668468698084385, 'log_loss_weighted_theo': 0.6531862980379637} - Predictions Mean: 0.4952 Std: 0.0957 Min: 0.2987 Max: 0.8416
My final blend scores AUC 0.68 / log_loss_weighted_theo 0.64 but I'm afraid I am overfitting my validation scores and this wont translate to private :/
", "rawMarkdown": "My final blend scores AUC 0.68 / log_loss_weighted_theo 0.64 but I'm afraid I am overfitting my validation scores and this wont translate to private :/", "votes": 1 }, { "id": 1965800, "postDate": "2022-10-01T14:37:01.587Z", "content": "0.68 AUC is really good. My AUC becomes 0.6725 when I aggregate predictions on patient_id groups. I guess my submissions were kinda consistent with my oof scores. I have no idea how people are reaching below 0.5.
", "rawMarkdown": "0.68 AUC is really good. My AUC becomes 0.6725 when I aggregate predictions on patient_id groups. I guess my submissions were kinda consistent with my oof scores. I have no idea how people are reaching below 0.5.", "votes": 1 }, { "id": 1965875, "postDate": "2022-10-01T15:29:23.657Z", "content": "Scores below 0.5 might very well be models overfit for the test set, created by tuning the model on the test set score until it went down. No clue if that's what really happened, but so few people got a score below 0.5 that it makes it seem lowkey overfit on that 7% of data points…
", "rawMarkdown": "Scores below 0.5 might very well be models overfit for the test set, created by tuning the model on the test set score until it went down. No clue if that's what really happened, but so few people got a score below 0.5 that it makes it seem lowkey overfit on that 7% of data points...", "votes": 1 }, { "id": 1966991, "postDate": "2022-10-02T09:21:26.690Z", "content": "If I compute my logloss on 20 randomly sampled images from my validation set 1000 times, I get 10% of \"lucky\" runs with a score of 0.4 or lower.
\nIf you take into account that people also use LB feedback to tweak their models then I'm not really surprised. I will be however if private LBs of 0.5 (or lower) are reached, it's much harder to get a lucky draw with 260 samples.
", "rawMarkdown": "If I compute my logloss on 20 randomly sampled images from my validation set 1000 times, I get 10% of \"lucky\" runs with a score of 0.4 or lower. \n\nIf you take into account that people also use LB feedback to tweak their models then I'm not really surprised. I will be however if private LBs of 0.5 (or lower) are reached, it's much harder to get a lucky draw with 260 samples.", "votes": 3 }, { "id": 1967022, "postDate": "2022-10-02T09:40:26.783Z", "content": "Exactly. Not discounting the work from the current top of the LB, but I think it's more likely random chance than actually capturing signal. As a last attempt, I tried doing weakly supervised multiple instance learning with attention to which image in the bag made the bag positive (LAA in my case) - and there was basically nothing by the end.
\nI wonder how pathologists figure this out…
", "rawMarkdown": "Exactly. Not discounting the work from the current top of the LB, but I think it's more likely random chance than actually capturing signal. As a last attempt, I tried doing weakly supervised multiple instance learning with attention to which image in the bag made the bag positive (LAA in my case) - and there was basically nothing by the end.\n\nI wonder how pathologists figure this out..." }, { "id": 1967686, "postDate": "2022-10-02T16:30:03.797Z", "content": "@theoviel I have observed similar spread of log loss over prediction scores, it could very much be the case here.
", "rawMarkdown": "@theoviel I have observed similar spread of log loss over prediction scores, it could very much be the case here." }, { "id": 1970384, "postDate": "2022-10-04T04:53:47.863Z", "content": "if my process doesn't deviate from the right, weighted CV scores lower than 0.5 are a very rare case
", "rawMarkdown": "if my process doesn't deviate from the right, weighted CV scores lower than 0.5 are a very rare case", "votes": 2 } ] }, { "id": 1953465, "postDate": "2022-09-24T13:56:28.853Z", "content": "Same here. It's because the public test set is quite small. My cv score is 0.564 but my lb score is 0.8.
", "rawMarkdown": "Same here. It's because the public test set is quite small. My cv score is 0.564 but my lb score is 0.8.", "votes": 1, "replies": [ { "id": 1953501, "postDate": "2022-09-24T14:24:13.330Z", "content": "Nice to see you here again after the hubmap finishes!
\nMy 5fold cv scores are around 0.65 anyway it's lower than the sample submission. Thanks for replying!
Are you sure you are using the right metrics? A 0.564 cv seems very high in this competition.
", "rawMarkdown": "Are you sure you are using the right metrics? A 0.564 cv seems very high in this competition.", "votes": 1 }, { "id": 1955541, "postDate": "2022-09-26T04:00:06.670Z", "content": "@forcewithme may i ask you what your cv scores are around?
", "rawMarkdown": "@forcewithme may i ask you what your cv scores are around?" }, { "id": 1955556, "postDate": "2022-09-26T04:05:53.180Z", "content": "I use sklearn.metrics.log_loss. I tried it with both binary and multiclass predictions and the result was same. My current cv score is 0.5598 right now. Am I missing something and overfitting like crazy?
@gunesevitan don't you consider applying weighted log loss?
", "rawMarkdown": "@gunesevitan don't you consider applying weighted log loss?" }, { "id": 1955654, "postDate": "2022-09-26T04:54:14.097Z", "content": "I didn't use weighted log loss since organizers said that each class is roughly equally important. I expect 0.02 deviation from the regular log loss.
", "rawMarkdown": "I didn't use weighted log loss since organizers said that each class is roughly equally important. I expect 0.02 deviation from the regular log loss.", "votes": 2 }, { "id": 1955765, "postDate": "2022-09-26T06:13:05.793Z", "content": "\nsklearn.metrics.log_loss to get the score of each class, and average them. For example, with sklearn.metrics.log_loss, you get 0.5 for CE, get 0.8 for LAA, then your score should be 0.65.===================================================
\nAbout the weight of each class. I am pretty sure that the weight of each class should be 0.5:0.5. Because the score should be 17.2 when all the predicted ce/laa probability are 0/1(1/0), and 0.6+(actually 0.69) when all the predicted ce/laa probability are 0.5 /0.5. Both scores(17.2 0.6) are verified.
So is it like macro log loss? I'm currently using log loss on binary sigmoided logits. I guess I have to create another dimension with 1 - y and do mean(log_loss(y_true, y_pred), log_loss(1 - y_true, 1 - y_pred)). It doesn't make much sense though. I think it would have the same score.
Yeah, I always created 1-y for class 0
@gunesevitan In my opinion,
\nmean(log_loss(y_true, y_pred), log_loss(1 - y_true, 1 - y_pred)) is wrong.
\nmean(log_loss(y_true, y_pred), log_loss(y_true, 1 - y_pred)) is right.
\nPlease correct me if you find something wrong.
I created two vectors and their names are self-explanatory…
\ny_true = np.random.randint(0, 2, 100)\ny_pred = np.random.rand(100)\nAs I thought, these two yield same scores
\n>>> log_loss(y_true, y_pred)\n0.9869673142342817\n>>> log_loss(1 - y_true, 1 - y_pred)\n0.9869673142342817\nbut this one yields a slightly different score
\n>>> log_loss(y_true, 1 - y_pred)\n0.9635681791900268\nand the average of them is
\n>>> np.mean([log_loss(y_true, y_pred), log_loss(y_true, 1 - y_pred)])\n0.9752677467121542\nso the deviation is even less than 0.02. However that is totally dependent to predictions. I don't think macro and micro log loss scores wouldn't be too much different from each other since both class predictions are dependent to each other.
", "rawMarkdown": "I created two vectors and their names are self-explanatory...\n\n```\ny_true = np.random.randint(0, 2, 100)\ny_pred = np.random.rand(100)\n```\n\nAs I thought, these two yield same scores\n\n```\n>>> log_loss(y_true, y_pred)\n0.9869673142342817\n>>> log_loss(1 - y_true, 1 - y_pred)\n0.9869673142342817\n```\n\nbut this one yields a slightly different score\n\n```\n>>> log_loss(y_true, 1 - y_pred)\n0.9635681791900268\n```\n\nand the average of them is\n```\n\n>>> np.mean([log_loss(y_true, y_pred), log_loss(y_true, 1 - y_pred)])\n0.9752677467121542\n```\n\nso the deviation is even less than 0.02. However that is totally dependent to predictions. I don't think macro and micro log loss scores wouldn't be too much different from each other since both class predictions are dependent to each other.", "votes": 2 }, { "id": 1956090, "postDate": "2022-09-26T09:20:40.017Z", "content": "Here's the code I use
\n# Group by patient\ndfg = df[[\"patient_id\", \"pred\", \"target\"]].groupby(\"patient_id\").mean()\n\n# Separate the two classes\ndf1 = dfg[dfg['target'] == 1]\ndf0 = dfg[dfg['target'] == 0]\n\n# Compute the losses and average\nl0 = log_loss(df0[\"target\"], df0[\"pred\"], labels=[0, 1])\nl1 = log_loss(df1[\"target\"], df1[\"pred\"], labels=[0, 1])\nloss = 0.5 * (l0 + l1)\n\nprint(f\"\\nCV loss : {loss:.3f} ({l0:.3f}, {l1:.3f})\")\n\n# For comparison, this is the binary logloss\nloss = log_loss(dfg[\"target\"], dfg[\"pred\"])\nprint(f\"\\nlogloss : {loss:.3f}\")\nI can get 0.55 logloss but my weighted logloss scores are around 0.65.
\nThe binary logloss benefits from the class imbalance hence if I have a model that predicts more CE the loss will be lower.
@gunesevitan What is your CV AUC though ? That's what I'm optimizing on my side =)
", "rawMarkdown": "Here's the code I use\n``` \n# Group by patient\ndfg = df[[\"patient_id\", \"pred\", \"target\"]].groupby(\"patient_id\").mean()\n\n# Separate the two classes\ndf1 = dfg[dfg['target'] == 1]\ndf0 = dfg[dfg['target'] == 0]\n\n# Compute the losses and average\nl0 = log_loss(df0[\"target\"], df0[\"pred\"], labels=[0, 1])\nl1 = log_loss(df1[\"target\"], df1[\"pred\"], labels=[0, 1])\nloss = 0.5 * (l0 + l1)\n\nprint(f\"\\nCV loss : {loss:.3f} ({l0:.3f}, {l1:.3f})\")\n\n# For comparison, this is the binary logloss\nloss = log_loss(dfg[\"target\"], dfg[\"pred\"])\nprint(f\"\\nlogloss : {loss:.3f}\")\n``` \n\nI can get 0.55 logloss but my weighted logloss scores are around 0.65.\nThe binary logloss benefits from the class imbalance hence if I have a model that predicts more CE the loss will be lower.\n\n@gunesevitan What is your CV AUC though ? That's what I'm optimizing on my side =)", "votes": 2 }, { "id": 1956117, "postDate": "2022-09-26T09:39:06.200Z", "content": "That makes sense… My results were close to each other because dummy vectors had normal distribution. AUC seems to be a good auxiliary metric. These are my fold scores. I use stratified group kfold, splits are stratified on label and groups are patient_id.
\n{\n \"fold_scores\": {\n \"fold1\": {\n \"accuracy\": 0.7450980392156863,\n \"roc_auc\": 0.6655420602789024,\n \"log_loss\": 0.5360595129819867\n },\n \"fold2\": {\n \"accuracy\": 0.6948051948051948,\n \"roc_auc\": 0.5798369457148538,\n \"log_loss\": 0.6151186826747733\n },\n \"fold3\": {\n \"accuracy\": 0.7635135135135135,\n \"roc_auc\": 0.6552579365079365,\n \"log_loss\": 0.5286322120778464\n },\n \"fold4\": {\n \"accuracy\": 0.74,\n \"roc_auc\": 0.6325799955247259,\n \"log_loss\": 0.5594933565209309\n },\n \"fold5\": {\n \"accuracy\": 0.7181208053691275,\n \"roc_auc\": 0.7004329004329004,\n \"log_loss\": 0.5591903770679996\n }\n },\n \"oof_scores\": {\n \"accuracy\": 0.7320954907161804,\n \"roc_auc\": 0.6483851310176723,\n \"log_loss\": 0.5599818563222174\n }\n}\nThanks @gunesevitan, I can confirm I have similary AUCs (~0.65) using the same splitting strategy
", "rawMarkdown": "Thanks @gunesevitan, I can confirm I have similary AUCs (~0.65) using the same splitting strategy", "votes": 2 }, { "id": 1956204, "postDate": "2022-09-26T10:43:04.443Z", "content": "my weighted log loss converges to 0.65 but in terms of accuracy, lower than @gunesevitan
", "rawMarkdown": "my weighted log loss converges to 0.65 but in terms of accuracy, lower than @gunesevitan " }, { "id": 1956211, "postDate": "2022-09-26T10:50:29.147Z", "content": "After implementing @forcewithme's metric as a loss function, my model didn't learn. Weighted log loss and regular log loss starts at 0.69 and finishes like that. Accuracy and ROC AUC randomly oscillates. This is what I'm using right now.
\nclass MacroBCEWithLogitsLoss(_WeightedLoss):\n\n def __init__(self, weight=None, reduction='mean'):\n\n super(MacroBCEWithLogitsLoss, self).__init__(weight=weight, reduction=reduction)\n\n self.weight = weight\n self.reduction = reduction\n\n def forward(self, inputs, targets):\n\n inputs = torch.sigmoid(inputs)\n loss_positive = F.binary_cross_entropy(inputs, targets, self.weight)\n loss_negative = F.binary_cross_entropy(1 - inputs, targets, self.weight)\n loss = (0.5 * loss_positive) + (0.5 * loss_negative)\n\n if self.reduction == 'mean':\n loss = loss.mean()\n elif self.reduction == 'sum':\n loss = loss.sum()\n\n return loss\nEdit: Optimizing this loss function didn't work for me. After one epoch, model learns to make safe predictions between 0.48-0.52, and loss doesn't improve after 0.69. This behavior continues when I tweak the weights.
", "rawMarkdown": "After implementing @forcewithme's metric as a loss function, my model didn't learn. Weighted log loss and regular log loss starts at 0.69 and finishes like that. Accuracy and ROC AUC randomly oscillates. This is what I'm using right now.\n\n```\nclass MacroBCEWithLogitsLoss(_WeightedLoss):\n\n def __init__(self, weight=None, reduction='mean'):\n\n super(MacroBCEWithLogitsLoss, self).__init__(weight=weight, reduction=reduction)\n\n self.weight = weight\n self.reduction = reduction\n\n def forward(self, inputs, targets):\n\n inputs = torch.sigmoid(inputs)\n loss_positive = F.binary_cross_entropy(inputs, targets, self.weight)\n loss_negative = F.binary_cross_entropy(1 - inputs, targets, self.weight)\n loss = (0.5 * loss_positive) + (0.5 * loss_negative)\n\n if self.reduction == 'mean':\n loss = loss.mean()\n elif self.reduction == 'sum':\n loss = loss.sum()\n\n return loss\n\n```\n\nEdit: Optimizing this loss function didn't work for me. After one epoch, model learns to make safe predictions between 0.48-0.52, and loss doesn't improve after 0.69. This behavior continues when I tweak the weights.", "votes": 2 }, { "id": 1956287, "postDate": "2022-09-26T11:24:12.887Z", "content": "I only used the competition metric for validation and save weights. And I can't get a significantly better score than sample_submission.csv 🤕
I believe public LB score is only on 20 images out of 280, there is good chance that private LB will look very different. Plus do also check the spread of log loss over the predictions, mean values only tell part of the story. Even if I get a mean log loss of under 0.6, the spread is from 0.2 to 0.8, which hints at some overfitting for sure - I guess this is where this dataset is hard. Also waiting to see top private LB solutions :)
", "rawMarkdown": "I believe public LB score is only on 20 images out of 280, there is good chance that private LB will look very different. Plus do also check the spread of log loss over the predictions, mean values only tell part of the story. Even if I get a mean log loss of under 0.6, the spread is from 0.2 to 0.8, which hints at some overfitting for sure - I guess this is where this dataset is hard. Also waiting to see top private LB solutions :)", "votes": 1 }, { "id": 1956964, "postDate": "2022-09-26T17:36:43.627Z", "content": "@icemantd checking the spread of log loss over the predictions seems really good strategy!!
", "rawMarkdown": "@icemantd checking the spread of log loss over the predictions seems really good strategy!!" } ] }, { "id": 1970760, "postDate": "2022-10-04T09:19:08.317Z", "content": "Turns out submitting 1 - preds scores better on LB for me.
This should happen in ~6% of the cases if my LB predictions follow the same distribution as my oof ones, but still I am a bit worried.
", "rawMarkdown": "Turns out submitting `1 - preds` scores better on LB for me.\n\nThis should happen in ~6% of the cases if my LB predictions follow the same distribution as my oof ones, but still I am a bit worried.\n\n", "votes": 2, "replies": [ { "id": 1971341, "postDate": "2022-10-04T14:39:37.203Z", "content": "Same here. I tried also reversing CE and LAA, it gave me better score in LB (0.7 -> 0.6)
\ndoes it imply there is no signal?
We'll find out soon enough. Hold on to your two submissions with some popcorn :D
", "rawMarkdown": "We'll find out soon enough. Hold on to your two submissions with some popcorn :D", "votes": 2 }, { "id": 1972627, "postDate": "2022-10-05T08:53:51.617Z", "content": "My best submission's AUC is 0.6842 and weighted log loss is 0.6387. It's weird that we are so close to each other in public leaderboard @theoviel, but I haven't inverted my predictions.
", "rawMarkdown": "My best submission's AUC is 0.6842 and weighted log loss is 0.6387. It's weird that we are so close to each other in public leaderboard @theoviel, but I haven't inverted my predictions.", "votes": 1 } ] }, { "id": 1955382, "postDate": "2022-09-26T01:15:27.510Z", "content": "I'm looking forward to the \"1st place solution\" post by sample_submission.csv.
", "rawMarkdown": "I'm looking forward to the \"1st place solution\" post by sample_submission.csv.", "votes": 2 } ], "comments": [ { "id": 1970621, "author_name": "Ali", "author_url": "", "post_date": "2022-10-04T07:02:38.270000", "content": "I am really not sure what is going on in this competition, I just started three days ago ! I did a basic training and got 0.4 at first shot. When I trained with \"focus\" and deeply I got a score of 1.0 and another try score of 0.6! I didn't really do much and will most likely go down sliding in the private board ! but I don't get my results ! … my 0.4 is a lucky shot maybe !
", "votes": 4, "replies": [ { "id": 1970627, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-10-04T07:13:03.680000", "content": "Can you share some details like what are your oof scores, what model are you using, tiles or whole image etc.?
", "votes": 0, "replies": [] }, { "id": 1970747, "author_name": "David Landup", "author_url": "", "post_date": "2022-10-04T09:08:25.013000", "content": "There isn't much signal, as I originally thought. The 0.4 is likely a lucky shot, unfortunately :(
", "votes": 2, "replies": [] }, { "id": 1971406, "author_name": "tdiceman", "author_url": "", "post_date": "2022-10-04T15:28:52.507000", "content": "@asalhi thanks for sharing this. All the best.
", "votes": 0, "replies": [] }, { "id": 1971450, "author_name": "anthony", "author_url": "", "post_date": "2022-10-04T16:09:08.207000", "content": "@asalhi please tell us more details about lucky shot)
", "votes": 0, "replies": [] }, { "id": 1971471, "author_name": "Ali", "author_url": "", "post_date": "2022-10-04T16:23:17.820000", "content": "I prefer not to go into details now, till the competition ends. But I am pretty sure it's a really super lucky shot that I didn't see coming, it will surely not replace hard efforts done by you and others who surely spent hours on this competition, I just passed by a few days ago and had some free time and was struggling in another competition so I said let's try this!
\nI'm pretty sure my lucky shot will shake badly! since my new tests are far from lucky. I would say. However I will choose it as one of the final submissions, I will share how to regenerate this lucky shot once the competition ends, untill then best of luck to you all! I am just “a passing by guy” in this competition …
", "votes": 4, "replies": [] }, { "id": 1971487, "author_name": "tdiceman", "author_url": "", "post_date": "2022-10-04T16:34:04.507000", "content": "@asalhi I'm sure we'll all learn a lot by the time we are done analyzing solutions for this competition. Looking forward to understand what worked, and also how can we go wrong in such cases. Hope you do not slide in the rankings.
", "votes": 1, "replies": [] }, { "id": 1971498, "author_name": "Ali", "author_url": "", "post_date": "2022-10-04T16:40:24.823000", "content": "I hope not to go sliding, However I have a feeling that it will be a rollercoaster hhhh
", "votes": 1, "replies": [] } ] }, { "id": 1972166, "author_name": "majoraregalia", "author_url": "", "post_date": "2022-10-05T03:05:15.953000", "content": "Well, I still think that anything that produces score less than 0.6-0.5 is a product of a mere luck .
", "votes": 1, "replies": [] }, { "id": 1971445, "author_name": "JM", "author_url": "", "post_date": "2022-10-04T16:04:57.503000", "content": "Any last minute guesses what the medal range will be for private LB?
\nI think low but possible chance sample submission may make it into bronze lol
\nMaybe 0.5 Gold to 0.7 Bronze?
", "votes": 1, "replies": [ { "id": 1971628, "author_name": "kaggler", "author_url": "", "post_date": "2022-10-04T17:44:11.163000", "content": "we can not know about it as discussed above
", "votes": 0, "replies": [] }, { "id": 1971713, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-10-04T18:21:29.543000", "content": "I think private leaderboard will be between 0.63 and 0.75. 0.63 is the gold zone, 0.64 is silver.
", "votes": 4, "replies": [] } ] }, { "id": 1965228, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-10-01T07:44:58.250000", "content": "def binary_weighted_log_loss_forcewithme(y_true, y_pred):\n\n log_loss_positive = log_loss(y_true, y_pred)\n log_loss_negative = log_loss(y_true, 1 - y_pred)\n log_loss_weighted = 0.5 * log_loss_positive + 0.5 * log_loss_negative\n\n return log_loss_positive, log_loss_negative, log_loss_weighted\n\n\ndef binary_weighted_log_loss_theo(y_true, y_pred):\n\n y_true_positive = y_true == 1\n y_true_negative = y_true == 0\n\n log_loss_positive = log_loss(y_true[y_true_positive], y_pred[y_true_positive], labels=[0, 1])\n log_loss_negative = log_loss(y_true[y_true_negative], y_pred[y_true_negative], labels=[0, 1])\n log_loss_weighted = 0.5 * log_loss_positive + 0.5 * log_loss_negative\n\n return log_loss_positive, log_loss_negative, log_loss_weighted\nThese two functions yield different results. This is my final blend score. I have no idea which one is the correct implementation so I use both of them for now.
\nBlend - {'accuracy': 0.6286472148541115, 'roc_auc': 0.66280723136299, 'log_loss_positive_forcewithme': 0.6593462375631115, 'log_loss_negative_forcewithme': 0.7662855417748091, 'log_loss_weighted_forcewithme': 0.7128158896689603, 'log_loss_positive_theo': 0.6395257262674887, 'log_loss_negative_theo': 0.6668468698084385, 'log_loss_weighted_theo': 0.6531862980379637} - Predictions Mean: 0.4952 Std: 0.0957 Min: 0.2987 Max: 0.8416
My final blend scores AUC 0.68 / log_loss_weighted_theo 0.64 but I'm afraid I am overfitting my validation scores and this wont translate to private :/
", "votes": 1, "replies": [] }, { "id": 1965800, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-10-01T14:37:01.587000", "content": "0.68 AUC is really good. My AUC becomes 0.6725 when I aggregate predictions on patient_id groups. I guess my submissions were kinda consistent with my oof scores. I have no idea how people are reaching below 0.5.
", "votes": 1, "replies": [] }, { "id": 1965875, "author_name": "David Landup", "author_url": "", "post_date": "2022-10-01T15:29:23.657000", "content": "Scores below 0.5 might very well be models overfit for the test set, created by tuning the model on the test set score until it went down. No clue if that's what really happened, but so few people got a score below 0.5 that it makes it seem lowkey overfit on that 7% of data points…
", "votes": 1, "replies": [] }, { "id": 1966991, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-10-02T09:21:26.690000", "content": "If I compute my logloss on 20 randomly sampled images from my validation set 1000 times, I get 10% of \"lucky\" runs with a score of 0.4 or lower.
\nIf you take into account that people also use LB feedback to tweak their models then I'm not really surprised. I will be however if private LBs of 0.5 (or lower) are reached, it's much harder to get a lucky draw with 260 samples.
", "votes": 3, "replies": [] }, { "id": 1967022, "author_name": "David Landup", "author_url": "", "post_date": "2022-10-02T09:40:26.783000", "content": "Exactly. Not discounting the work from the current top of the LB, but I think it's more likely random chance than actually capturing signal. As a last attempt, I tried doing weakly supervised multiple instance learning with attention to which image in the bag made the bag positive (LAA in my case) - and there was basically nothing by the end.
\nI wonder how pathologists figure this out…
", "votes": 0, "replies": [] }, { "id": 1967686, "author_name": "tdiceman", "author_url": "", "post_date": "2022-10-02T16:30:03.797000", "content": "@theoviel I have observed similar spread of log loss over prediction scores, it could very much be the case here.
", "votes": 0, "replies": [] }, { "id": 1970384, "author_name": "kaggler", "author_url": "", "post_date": "2022-10-04T04:53:47.863000", "content": "if my process doesn't deviate from the right, weighted CV scores lower than 0.5 are a very rare case
", "votes": 2, "replies": [] } ] }, { "id": 1953465, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-24T13:56:28.853000", "content": "Same here. It's because the public test set is quite small. My cv score is 0.564 but my lb score is 0.8.
", "votes": 1, "replies": [ { "id": 1953501, "author_name": "kaggler", "author_url": "", "post_date": "2022-09-24T14:24:13.330000", "content": "Nice to see you here again after the hubmap finishes!
\nMy 5fold cv scores are around 0.65 anyway it's lower than the sample submission. Thanks for replying!
Are you sure you are using the right metrics? A 0.564 cv seems very high in this competition.
", "votes": 1, "replies": [] }, { "id": 1955541, "author_name": "kaggler", "author_url": "", "post_date": "2022-09-26T04:00:06.670000", "content": "@forcewithme may i ask you what your cv scores are around?
", "votes": 0, "replies": [] }, { "id": 1955556, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-26T04:05:53.180000", "content": "I use sklearn.metrics.log_loss. I tried it with both binary and multiclass predictions and the result was same. My current cv score is 0.5598 right now. Am I missing something and overfitting like crazy?
@gunesevitan don't you consider applying weighted log loss?
", "votes": 0, "replies": [] }, { "id": 1955654, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-26T04:54:14.097000", "content": "I didn't use weighted log loss since organizers said that each class is roughly equally important. I expect 0.02 deviation from the regular log loss.
", "votes": 2, "replies": [] }, { "id": 1955765, "author_name": "ForcewithMe", "author_url": "", "post_date": "2022-09-26T06:13:05.793000", "content": "\nsklearn.metrics.log_loss to get the score of each class, and average them. For example, with sklearn.metrics.log_loss, you get 0.5 for CE, get 0.8 for LAA, then your score should be 0.65.===================================================
\nAbout the weight of each class. I am pretty sure that the weight of each class should be 0.5:0.5. Because the score should be 17.2 when all the predicted ce/laa probability are 0/1(1/0), and 0.6+(actually 0.69) when all the predicted ce/laa probability are 0.5 /0.5. Both scores(17.2 0.6) are verified.
So is it like macro log loss? I'm currently using log loss on binary sigmoided logits. I guess I have to create another dimension with 1 - y and do mean(log_loss(y_true, y_pred), log_loss(1 - y_true, 1 - y_pred)). It doesn't make much sense though. I think it would have the same score.
Yeah, I always created 1-y for class 0
@gunesevitan In my opinion,
\nmean(log_loss(y_true, y_pred), log_loss(1 - y_true, 1 - y_pred)) is wrong.
\nmean(log_loss(y_true, y_pred), log_loss(y_true, 1 - y_pred)) is right.
\nPlease correct me if you find something wrong.
I created two vectors and their names are self-explanatory…
\ny_true = np.random.randint(0, 2, 100)\ny_pred = np.random.rand(100)\nAs I thought, these two yield same scores
\n>>> log_loss(y_true, y_pred)\n0.9869673142342817\n>>> log_loss(1 - y_true, 1 - y_pred)\n0.9869673142342817\nbut this one yields a slightly different score
\n>>> log_loss(y_true, 1 - y_pred)\n0.9635681791900268\nand the average of them is
\n>>> np.mean([log_loss(y_true, y_pred), log_loss(y_true, 1 - y_pred)])\n0.9752677467121542\nso the deviation is even less than 0.02. However that is totally dependent to predictions. I don't think macro and micro log loss scores wouldn't be too much different from each other since both class predictions are dependent to each other.
", "votes": 2, "replies": [] }, { "id": 1956090, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-09-26T09:20:40.017000", "content": "Here's the code I use
\n# Group by patient\ndfg = df[[\"patient_id\", \"pred\", \"target\"]].groupby(\"patient_id\").mean()\n\n# Separate the two classes\ndf1 = dfg[dfg['target'] == 1]\ndf0 = dfg[dfg['target'] == 0]\n\n# Compute the losses and average\nl0 = log_loss(df0[\"target\"], df0[\"pred\"], labels=[0, 1])\nl1 = log_loss(df1[\"target\"], df1[\"pred\"], labels=[0, 1])\nloss = 0.5 * (l0 + l1)\n\nprint(f\"\\nCV loss : {loss:.3f} ({l0:.3f}, {l1:.3f})\")\n\n# For comparison, this is the binary logloss\nloss = log_loss(dfg[\"target\"], dfg[\"pred\"])\nprint(f\"\\nlogloss : {loss:.3f}\")\nI can get 0.55 logloss but my weighted logloss scores are around 0.65.
\nThe binary logloss benefits from the class imbalance hence if I have a model that predicts more CE the loss will be lower.
@gunesevitan What is your CV AUC though ? That's what I'm optimizing on my side =)
", "votes": 2, "replies": [] }, { "id": 1956117, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-26T09:39:06.200000", "content": "That makes sense… My results were close to each other because dummy vectors had normal distribution. AUC seems to be a good auxiliary metric. These are my fold scores. I use stratified group kfold, splits are stratified on label and groups are patient_id.
\n{\n \"fold_scores\": {\n \"fold1\": {\n \"accuracy\": 0.7450980392156863,\n \"roc_auc\": 0.6655420602789024,\n \"log_loss\": 0.5360595129819867\n },\n \"fold2\": {\n \"accuracy\": 0.6948051948051948,\n \"roc_auc\": 0.5798369457148538,\n \"log_loss\": 0.6151186826747733\n },\n \"fold3\": {\n \"accuracy\": 0.7635135135135135,\n \"roc_auc\": 0.6552579365079365,\n \"log_loss\": 0.5286322120778464\n },\n \"fold4\": {\n \"accuracy\": 0.74,\n \"roc_auc\": 0.6325799955247259,\n \"log_loss\": 0.5594933565209309\n },\n \"fold5\": {\n \"accuracy\": 0.7181208053691275,\n \"roc_auc\": 0.7004329004329004,\n \"log_loss\": 0.5591903770679996\n }\n },\n \"oof_scores\": {\n \"accuracy\": 0.7320954907161804,\n \"roc_auc\": 0.6483851310176723,\n \"log_loss\": 0.5599818563222174\n }\n}\nThanks @gunesevitan, I can confirm I have similary AUCs (~0.65) using the same splitting strategy
", "votes": 2, "replies": [] }, { "id": 1956204, "author_name": "kaggler", "author_url": "", "post_date": "2022-09-26T10:43:04.443000", "content": "my weighted log loss converges to 0.65 but in terms of accuracy, lower than @gunesevitan
", "votes": 0, "replies": [] }, { "id": 1956211, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-09-26T10:50:29.147000", "content": "After implementing @forcewithme's metric as a loss function, my model didn't learn. Weighted log loss and regular log loss starts at 0.69 and finishes like that. Accuracy and ROC AUC randomly oscillates. This is what I'm using right now.
\nclass MacroBCEWithLogitsLoss(_WeightedLoss):\n\n def __init__(self, weight=None, reduction='mean'):\n\n super(MacroBCEWithLogitsLoss, self).__init__(weight=weight, reduction=reduction)\n\n self.weight = weight\n self.reduction = reduction\n\n def forward(self, inputs, targets):\n\n inputs = torch.sigmoid(inputs)\n loss_positive = F.binary_cross_entropy(inputs, targets, self.weight)\n loss_negative = F.binary_cross_entropy(1 - inputs, targets, self.weight)\n loss = (0.5 * loss_positive) + (0.5 * loss_negative)\n\n if self.reduction == 'mean':\n loss = loss.mean()\n elif self.reduction == 'sum':\n loss = loss.sum()\n\n return loss\nEdit: Optimizing this loss function didn't work for me. After one epoch, model learns to make safe predictions between 0.48-0.52, and loss doesn't improve after 0.69. This behavior continues when I tweak the weights.
", "votes": 2, "replies": [] }, { "id": 1956287, "author_name": "ForcewithMe", "author_url": "", "post_date": "2022-09-26T11:24:12.887000", "content": "I only used the competition metric for validation and save weights. And I can't get a significantly better score than sample_submission.csv 🤕
I believe public LB score is only on 20 images out of 280, there is good chance that private LB will look very different. Plus do also check the spread of log loss over the predictions, mean values only tell part of the story. Even if I get a mean log loss of under 0.6, the spread is from 0.2 to 0.8, which hints at some overfitting for sure - I guess this is where this dataset is hard. Also waiting to see top private LB solutions :)
", "votes": 1, "replies": [] }, { "id": 1956964, "author_name": "kaggler", "author_url": "", "post_date": "2022-09-26T17:36:43.627000", "content": "@icemantd checking the spread of log loss over the predictions seems really good strategy!!
", "votes": 0, "replies": [] } ] }, { "id": 1970760, "author_name": "Theo Viel", "author_url": "", "post_date": "2022-10-04T09:19:08.317000", "content": "Turns out submitting 1 - preds scores better on LB for me.
This should happen in ~6% of the cases if my LB predictions follow the same distribution as my oof ones, but still I am a bit worried.
", "votes": 2, "replies": [ { "id": 1971341, "author_name": "kaggler", "author_url": "", "post_date": "2022-10-04T14:39:37.203000", "content": "Same here. I tried also reversing CE and LAA, it gave me better score in LB (0.7 -> 0.6)
\ndoes it imply there is no signal?
We'll find out soon enough. Hold on to your two submissions with some popcorn :D
", "votes": 2, "replies": [] }, { "id": 1972627, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2022-10-05T08:53:51.617000", "content": "My best submission's AUC is 0.6842 and weighted log loss is 0.6387. It's weird that we are so close to each other in public leaderboard @theoviel, but I haven't inverted my predictions.
", "votes": 1, "replies": [] } ] }, { "id": 1955382, "author_name": "Joe Marturano", "author_url": "", "post_date": "2022-09-26T01:15:27.510000", "content": "I'm looking forward to the \"1st place solution\" post by sample_submission.csv.
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1953430": "Why do my models have lower performances than sample submission? it's a weird competition \nin local environment, anyway it learns the pattern. \nbut when i submit it i just score 0.6 or 0.7 \nit's weird!! \n-----------------------------------------------------------------------------\n Thanks for the all the discussions! Good Luck For Everyone Here! \n\n", "1970621": "I am really not sure what is going on in this competition, I just started three days ago ! I did a basic training and got 0.4 at first shot. When I trained with \"focus\" and deeply I got a score of 1.0 and another try score of 0.6! I didn't really do much and will most likely go down sliding in the private board ! but I don't get my results ! ... my 0.4 is a lucky shot maybe ! ", "1972166": "Well, I still think that anything that produces score less than 0.6-0.5 is a product of a mere luck ~~or overfitting that happens in such specific conditions as a lack of signal in the data~~.", "1971445": "Any last minute guesses what the medal range will be for private LB?\n\nI think low but possible chance sample submission may make it into bronze lol\n\nMaybe 0.5 Gold to 0.7 Bronze?", "1965228": "```\ndef binary_weighted_log_loss_forcewithme(y_true, y_pred):\n\n log_loss_positive = log_loss(y_true, y_pred)\n log_loss_negative = log_loss(y_true, 1 - y_pred)\n log_loss_weighted = 0.5 * log_loss_positive + 0.5 * log_loss_negative\n\n return log_loss_positive, log_loss_negative, log_loss_weighted\n\n\ndef binary_weighted_log_loss_theo(y_true, y_pred):\n\n y_true_positive = y_true == 1\n y_true_negative = y_true == 0\n\n log_loss_positive = log_loss(y_true[y_true_positive], y_pred[y_true_positive], labels=[0, 1])\n log_loss_negative = log_loss(y_true[y_true_negative], y_pred[y_true_negative], labels=[0, 1])\n log_loss_weighted = 0.5 * log_loss_positive + 0.5 * log_loss_negative\n\n return log_loss_positive, log_loss_negative, log_loss_weighted\n```\n\nThese two functions yield different results. This is my final blend score. I have no idea which one is the correct implementation so I use both of them for now.\n\n` Blend - {'accuracy': 0.6286472148541115, 'roc_auc': 0.66280723136299, 'log_loss_positive_forcewithme': 0.6593462375631115, 'log_loss_negative_forcewithme': 0.7662855417748091, 'log_loss_weighted_forcewithme': 0.7128158896689603, 'log_loss_positive_theo': 0.6395257262674887, 'log_loss_negative_theo': 0.6668468698084385, 'log_loss_weighted_theo': 0.6531862980379637} - Predictions Mean: 0.4952 Std: 0.0957 Min: 0.2987 Max: 0.8416`", "1953465": "Same here. It's because the public test set is quite small. My cv score is 0.564 but my lb score is 0.8.", "1970760": "Turns out submitting `1 - preds` scores better on LB for me.\n\nThis should happen in ~6% of the cases if my LB predictions follow the same distribution as my oof ones, but still I am a bit worried.\n\n", "1955382": "I'm looking forward to the \"1st place solution\" post by sample_submission.csv." } }