I think the Competition metric needs to be normalised by weights. Is it correct to first divide each row by each weight and then average the loss?
\n\nAbove is my mistake…. I am really embarassed….
\nThank you, @harshitsheoran!
\nNow I can calculate the right competition loss!
def competiton_loss_row_norm(y_hat, y):\n loss = loss_fn(y_hat, y)\n weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n loss = (loss * weights).sum(axis=1)\n w_sum = weights.sum(axis=1)\n loss = torch.div(loss, w_sum)\n return loss.mean()\nnotebook here. I updated ver 4.
\nhttps://www.kaggle.com/code/yosukeyama/comp-metric-findings-about-test-data
I think the Competition metric needs to be normalised by weights. Is it correct to first divide each row by each weight and then average the loss?
\n\nAbove is my mistake…. I am really embarassed….
\nThank you, @harshitsheoran!
\nNow I can calculate the right competition loss!
def competiton_loss_row_norm(y_hat, y):\n loss = loss_fn(y_hat, y)\n weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n loss = (loss * weights).sum(axis=1)\n w_sum = weights.sum(axis=1)\n loss = torch.div(loss, w_sum)\n return loss.mean()\nnotebook here. I updated ver 4.
\nhttps://www.kaggle.com/code/yosukeyama/comp-metric-findings-about-test-data
Just 1 small/tiny/petite problem which is causing all your loss calculations to go haywire is you are using BCEWithLogitsLoss instead of BCELoss, rest you are wise enough.
", "rawMarkdown": "Just 1 small/tiny/petite problem which is causing all your loss calculations to go haywire is you are using BCEWithLogitsLoss instead of BCELoss, rest you are wise enough.", "votes": 3, "replies": [ { "id": 1900045, "postDate": "2022-08-15T17:03:57.857Z", "content": "It's true! I'm so embarrassed!
\nI thought it was quirky and weird, but now the value fit perfectly, thank you!
", "rawMarkdown": "It's true! I'm so embarrassed!\n\nI thought it was quirky and weird, but now the value fit perfectly, thank you!", "votes": 2 } ] }, { "id": 1900361, "postDate": "2022-08-16T00:18:49.600Z", "content": "You need to be careful to understand the difference between a loss per batch and a loss for an epoch. Most frameworks assume that the losses for each batch are averaged to get the epoch loss, but that would be incorrect for this competition. That is because the samples are weighted by positive/negative and there may be different numbers of positive and negative samples in each batch
", "rawMarkdown": "You need to be careful to understand the difference between a loss per batch and a loss for an epoch. Most frameworks assume that the losses for each batch are averaged to get the epoch loss, but that would be incorrect for this competition. That is because the samples are weighted by positive/negative and there may be different numbers of positive and negative samples in each batch\n", "votes": 1 }, { "id": 1900241, "postDate": "2022-08-15T20:28:00.123Z", "content": "I see that in the notebook, your version of competition loss from a discussion is too small, I personally wrote that normalization line, and I am using that in my pipeline, the intended way to use that line is 1 patient at a time, so doing something like this
\nprint(np.mean([competiton_loss(mean_values[i:i+1], labels[i:i+1].double()) for i in range(len(mean_values))])) would give a perfect score.
Thanks a lot for full code and discussion, I was also struggling to replicate metric😅
", "rawMarkdown": "Thanks a lot for full code and discussion, I was also struggling to replicate metric😅", "votes": 1 } ], "comments": [ { "id": 1900026, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-08-15T16:49:41.753000", "content": "Just 1 small/tiny/petite problem which is causing all your loss calculations to go haywire is you are using BCEWithLogitsLoss instead of BCELoss, rest you are wise enough.
", "votes": 3, "replies": [ { "id": 1900045, "author_name": "YYama", "author_url": "", "post_date": "2022-08-15T17:03:57.857000", "content": "It's true! I'm so embarrassed!
\nI thought it was quirky and weird, but now the value fit perfectly, thank you!
", "votes": 2, "replies": [] } ] }, { "id": 1900361, "author_name": "SolverWorld", "author_url": "", "post_date": "2022-08-16T00:18:49.600000", "content": "You need to be careful to understand the difference between a loss per batch and a loss for an epoch. Most frameworks assume that the losses for each batch are averaged to get the epoch loss, but that would be incorrect for this competition. That is because the samples are weighted by positive/negative and there may be different numbers of positive and negative samples in each batch
", "votes": 1, "replies": [] }, { "id": 1900241, "author_name": "Harshit Sheoran", "author_url": "", "post_date": "2022-08-15T20:28:00.123000", "content": "I see that in the notebook, your version of competition loss from a discussion is too small, I personally wrote that normalization line, and I am using that in my pipeline, the intended way to use that line is 1 patient at a time, so doing something like this
\nprint(np.mean([competiton_loss(mean_values[i:i+1], labels[i:i+1].double()) for i in range(len(mean_values))])) would give a perfect score.
Thanks a lot for full code and discussion, I was also struggling to replicate metric😅
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1900018": "I think the Competition metric needs to be normalised by weights. Is it correct to first divide each row by each weight and then average the loss?\n\n~~If this is the case, then using mean prediction, the loss in train data would be around 0.7. If only data with a patient_overall of 1 were used, the loss would be approximately 0.57. This is consistent with the public LB score of the mean preds. ~~\n\nAbove is my mistake.... I am really embarassed....\nThank you, @harshitsheoran!\nNow I can calculate the right competition loss!\n\n```\ndef competiton_loss_row_norm(y_hat, y):\n loss = loss_fn(y_hat, y)\n weights = y * competition_weights['+'] + (1 - y) * competition_weights['-']\n loss = (loss * weights).sum(axis=1)\n w_sum = weights.sum(axis=1)\n loss = torch.div(loss, w_sum)\n return loss.mean()\n```\n\nnotebook here. I updated ver 4.\nhttps://www.kaggle.com/code/yosukeyama/comp-metric-findings-about-test-data", "1900026": "Just 1 small/tiny/petite problem which is causing all your loss calculations to go haywire is you are using BCEWithLogitsLoss instead of BCELoss, rest you are wise enough.", "1900361": "You need to be careful to understand the difference between a loss per batch and a loss for an epoch. Most frameworks assume that the losses for each batch are averaged to get the epoch loss, but that would be incorrect for this competition. That is because the samples are weighted by positive/negative and there may be different numbers of positive and negative samples in each batch\n", "1900241": "I see that in the notebook, your version of competition loss from a discussion is too small, I personally wrote that normalization line, and I am using that in my pipeline, the intended way to use that line is 1 patient at a time, so doing something like this\n`print(np.mean([competiton_loss(mean_values[i:i+1], labels[i:i+1].double()) for i in range(len(mean_values))]))` would give a perfect score.", "1900205": "Thanks a lot for full code and discussion, I was also struggling to replicate metric😅" } }