{ "id": 349669, "title": "Question Regarding the Formula & Log", "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/349669", "author_name": "Andrada", "post_date": "2022-09-02T08:29:04.099000", "votes": 20, "comment_count": 4, "views": 0, "content": "

Hi!

\n

This might be a very dumb question and if it is so please excuse me, but I really am trying to understand what am I missing.

\n

In this competition the formula is as follows:

\n

\n

where, I believe:

\n\n

We need to predict probabilities (between 0 and 1) for each of the 8 labels (like in the example provided):

\n

\n

So I created the following formula to compute the Loss:

\n
def get_custom_loss(logits, targets):\n\n    # Compute the weights\n    weights = targets * competition_weights['+'] + (1 - targets) * competition_weights['-']\n\n    # Losses on label and exam level\n    L = torch.zeros(targets.shape, device=DEVICE)\n\n    w = weights\n    y = targets\n    p = logits\n\n    for i in range(L.shape[0]):\n        for j in range(L.shape[1]):\n            L[i, j] = -w[i, j] * (\n                y[i, j] * math.log(p[i, j]) +\n                (1 - y[i, j]) * math.log(1 - p[i, j]))\n\n    # Average Loss on Exam (or patient)\n    Exams_Loss = torch.div(torch.sum(L, dim=1), torch.sum(w, dim=1))\n\n    return Exams_Loss\n
\n

However I get an error from the math.log() function because:

\n\n

What am I missing? Any advice would be highly appreciated 🙏

", "messages": [ { "id": 1923444, "postDate": "2022-09-02T08:29:04.100Z", "content": "

Hi!

\n

This might be a very dumb question and if it is so please excuse me, but I really am trying to understand what am I missing.

\n

In this competition the formula is as follows:

\n

\n

where, I believe:

\n\n

We need to predict probabilities (between 0 and 1) for each of the 8 labels (like in the example provided):

\n

\n

So I created the following formula to compute the Loss:

\n
def get_custom_loss(logits, targets):\n\n    # Compute the weights\n    weights = targets * competition_weights['+'] + (1 - targets) * competition_weights['-']\n\n    # Losses on label and exam level\n    L = torch.zeros(targets.shape, device=DEVICE)\n\n    w = weights\n    y = targets\n    p = logits\n\n    for i in range(L.shape[0]):\n        for j in range(L.shape[1]):\n            L[i, j] = -w[i, j] * (\n                y[i, j] * math.log(p[i, j]) +\n                (1 - y[i, j]) * math.log(1 - p[i, j]))\n\n    # Average Loss on Exam (or patient)\n    Exams_Loss = torch.div(torch.sum(L, dim=1), torch.sum(w, dim=1))\n\n    return Exams_Loss\n
\n

However I get an error from the math.log() function because:

\n\n

What am I missing? Any advice would be highly appreciated 🙏

", "rawMarkdown": "Hi!\n\nThis might be a very dumb question and if it is so please excuse me, but I really am trying to understand what am I missing.\n\nIn this competition the formula is as follows:\n\n\n\nwhere, I believe:\n* L - total Loss\n* w - the weight\n* y - the true value\n* p - the predicted value\n* i -> the exam and j -> the label\n\nWe need to predict probabilities (between 0 and 1) for each of the 8 labels (like in the example provided):\n\n\n\nSo I created the following formula to compute the Loss:\n\n```\ndef get_custom_loss(logits, targets):\n \n # Compute the weights\n weights = targets * competition_weights['+'] + (1 - targets) * competition_weights['-']\n \n # Losses on label and exam level\n L = torch.zeros(targets.shape, device=DEVICE)\n\n w = weights\n y = targets\n p = logits\n\n for i in range(L.shape[0]):\n for j in range(L.shape[1]):\n L[i, j] = -w[i, j] * (\n y[i, j] * math.log(p[i, j]) +\n (1 - y[i, j]) * math.log(1 - p[i, j]))\n \n # Average Loss on Exam (or patient)\n Exams_Loss = torch.div(torch.sum(L, dim=1), torch.sum(w, dim=1))\n \n return Exams_Loss\n```\n\nHowever I get an error from the `math.log()` function because:\n* if the model predicts p=1 (and y=0), then => math.log(1 - p[i, j]) = math.log(1-1) = log(0) (undefined)\n* if the model predicts p=0 (and y=1) then => math.log(p[i, j]) = log(0) (undefined)\n\nWhat am I missing? Any advice would be highly appreciated 🙏", "votes": 20 }, { "id": 1923531, "postDate": "2022-09-02T09:50:29.157Z", "content": "

Take a look at how this competition is addressing this: https://www.kaggle.com/competitions/mayo-clinic-strip-ai/overview/evaluation

\n
\n

max(min(p,1−epsilon),epsilon) with epsilon=10^-15

\n
", "rawMarkdown": "Take a look at how this competition is addressing this: https://www.kaggle.com/competitions/mayo-clinic-strip-ai/overview/evaluation\n\n> max(min(p,1−epsilon),epsilon) with epsilon=10^-15", "votes": 3 }, { "id": 1923455, "postDate": "2022-09-02T08:41:18.563Z", "content": "

Usually when computing log loss there is an epsilon parameter which avoids the undefined log(0) error.

\n

If you take a look at pytorch losses for example https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#NLLLoss
\nyou'll see that eps=1e-8 is used to prevent the error.

\n

So when you compute the loss you can never predict less than eps or more than 1-eps.

\n

I would also recommend you to use a vectorized version of the competition metric because for loops are going to be very slow.

", "rawMarkdown": "Usually when computing log loss there is an epsilon parameter which avoids the undefined log(0) error.\n\nIf you take a look at pytorch losses for example https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#NLLLoss\nyou'll see that `eps=1e-8` is used to prevent the error.\n\nSo when you compute the loss you can never predict less than `eps` or more than `1-eps`.\n\nI would also recommend you to use a vectorized version of the competition metric because for loops are going to be very slow.", "votes": 3, "replies": [ { "id": 1923478, "postDate": "2022-09-02T08:58:11.457Z", "content": "

So it's kinda like \"bending\" the rule a little bit. I thought of making the prediction be in the interval (0, 1) rather than [0, 1], but it felt like cheating so I concluded that I was doing something wrong.

\n

Thank you much much for the advice 🙏

", "rawMarkdown": "So it's kinda like \"bending\" the rule a little bit. I thought of making the prediction be in the interval (0, 1) rather than [0, 1], but it felt like cheating so I concluded that I was doing something wrong.\n\nThank you much much for the advice 🙏", "votes": 2 }, { "id": 1923495, "postDate": "2022-09-02T09:15:43.130Z", "content": "

The output of your neural network should never answer exactly 0 or exactly 1 anyway.
\nAlso I'm pretty sure that the hosts also apply an epsilon value when they compute the competition metric so you can probably decide to over-rule your network and answer 0 or 1 and metric computation will still be fine.

", "rawMarkdown": "The output of your neural network should never answer exactly 0 or exactly 1 anyway.\nAlso I'm pretty sure that the hosts also apply an epsilon value when they compute the competition metric so you can probably decide to over-rule your network and answer 0 or 1 and metric computation will still be fine.", "votes": 1 } ] } ], "comments": [ { "id": 1923531, "author_name": "Samuel Cortinhas", "author_url": "", "post_date": "2022-09-02T09:50:29.157000", "content": "

Take a look at how this competition is addressing this: https://www.kaggle.com/competitions/mayo-clinic-strip-ai/overview/evaluation

\n
\n

max(min(p,1−epsilon),epsilon) with epsilon=10^-15

\n
", "votes": 3, "replies": [] }, { "id": 1923455, "author_name": "Optimo", "author_url": "", "post_date": "2022-09-02T08:41:18.563000", "content": "

Usually when computing log loss there is an epsilon parameter which avoids the undefined log(0) error.

\n

If you take a look at pytorch losses for example https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#NLLLoss
\nyou'll see that eps=1e-8 is used to prevent the error.

\n

So when you compute the loss you can never predict less than eps or more than 1-eps.

\n

I would also recommend you to use a vectorized version of the competition metric because for loops are going to be very slow.

", "votes": 3, "replies": [ { "id": 1923478, "author_name": "Andrada", "author_url": "", "post_date": "2022-09-02T08:58:11.457000", "content": "

So it's kinda like \"bending\" the rule a little bit. I thought of making the prediction be in the interval (0, 1) rather than [0, 1], but it felt like cheating so I concluded that I was doing something wrong.

\n

Thank you much much for the advice 🙏

", "votes": 2, "replies": [] }, { "id": 1923495, "author_name": "Optimo", "author_url": "", "post_date": "2022-09-02T09:15:43.130000", "content": "

The output of your neural network should never answer exactly 0 or exactly 1 anyway.
\nAlso I'm pretty sure that the hosts also apply an epsilon value when they compute the competition metric so you can probably decide to over-rule your network and answer 0 or 1 and metric computation will still be fine.

", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1923444": "Hi!\n\nThis might be a very dumb question and if it is so please excuse me, but I really am trying to understand what am I missing.\n\nIn this competition the formula is as follows:\n\n\n\nwhere, I believe:\n* L - total Loss\n* w - the weight\n* y - the true value\n* p - the predicted value\n* i -> the exam and j -> the label\n\nWe need to predict probabilities (between 0 and 1) for each of the 8 labels (like in the example provided):\n\n\n\nSo I created the following formula to compute the Loss:\n\n```\ndef get_custom_loss(logits, targets):\n \n # Compute the weights\n weights = targets * competition_weights['+'] + (1 - targets) * competition_weights['-']\n \n # Losses on label and exam level\n L = torch.zeros(targets.shape, device=DEVICE)\n\n w = weights\n y = targets\n p = logits\n\n for i in range(L.shape[0]):\n for j in range(L.shape[1]):\n L[i, j] = -w[i, j] * (\n y[i, j] * math.log(p[i, j]) +\n (1 - y[i, j]) * math.log(1 - p[i, j]))\n \n # Average Loss on Exam (or patient)\n Exams_Loss = torch.div(torch.sum(L, dim=1), torch.sum(w, dim=1))\n \n return Exams_Loss\n```\n\nHowever I get an error from the `math.log()` function because:\n* if the model predicts p=1 (and y=0), then => math.log(1 - p[i, j]) = math.log(1-1) = log(0) (undefined)\n* if the model predicts p=0 (and y=1) then => math.log(p[i, j]) = log(0) (undefined)\n\nWhat am I missing? Any advice would be highly appreciated 🙏", "1923531": "Take a look at how this competition is addressing this: https://www.kaggle.com/competitions/mayo-clinic-strip-ai/overview/evaluation\n\n> max(min(p,1−epsilon),epsilon) with epsilon=10^-15", "1923455": "Usually when computing log loss there is an epsilon parameter which avoids the undefined log(0) error.\n\nIf you take a look at pytorch losses for example https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#NLLLoss\nyou'll see that `eps=1e-8` is used to prevent the error.\n\nSo when you compute the loss you can never predict less than `eps` or more than `1-eps`.\n\nI would also recommend you to use a vectorized version of the competition metric because for loops are going to be very slow." } }