Looking at the weighted log loss defined for this competition, it is different than the pytorch definition of weighted log loss, since it takes weighted average from the average loss of each class, rather than taking a weighted average of the loss for each input.
\nThis ignores the fact that there's a major typo in the metric definition which confuses i and j that was never fixed.
\nTherefore, if you use the pytorch implementation of weighted log loss, you'll get something different than what this competition defines.
\nIn code:
\nimport numpy as np\n\n\ndef weighted_log_loss_kaggle(probs, weights, target):\n probs = np.array(probs)\n log_probs = np.log(probs)\n weights = np.array(weights)\n\n target = np.array(target)\n res = 0\n for c in np.unique(target):\n class_log_probs = log_probs[target == c][:, c]\n class_weight = weights[c]\n res += class_weight * class_log_probs.mean()\n return - (res / weights.sum())\n\n\ndef weighted_log_loss_pytorch(probs, weights, target):\n probs = np.array(probs)\n log_probs = np.log(probs)\n weights = np.array(weights)\n\n target = np.array(target)\n res = 0\n weight_sum = 0\n for c in np.unique(target):\n class_log_probs = log_probs[target == c][:, c]\n class_weight = weights[c]\n res += class_weight * class_log_probs.sum()\n weight_sum += len(class_log_probs) * class_weight\n return - (res / weight_sum)\n\n\n>>> weighted_log_loss_pytorch([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]], [0.3, 0.7], [0, 0, 1])\n>>> 0.7904898498886928\n\n# Checking against official pytorch\n>>> nll_loss = nn.NLLLoss(weight=torch.tensor([0.3, 0.7]))\n>>> nll_loss(torch.log(torch.tensor([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]])), torch.tensor([0, 0, 1]))\n>>> tensor(0.7905)\n\n>>> weighted_log_loss_kaggle([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]], [0.3, 0.7], [0, 0, 1])\n>>> 0.9352088839417277\nLooking at the weighted log loss defined for this competition, it is different than the pytorch definition of weighted log loss, since it takes weighted average from the average loss of each class, rather than taking a weighted average of the loss for each input.
\nThis ignores the fact that there's a major typo in the metric definition which confuses i and j that was never fixed.
\nTherefore, if you use the pytorch implementation of weighted log loss, you'll get something different than what this competition defines.
\nIn code:
\nimport numpy as np\n\n\ndef weighted_log_loss_kaggle(probs, weights, target):\n probs = np.array(probs)\n log_probs = np.log(probs)\n weights = np.array(weights)\n\n target = np.array(target)\n res = 0\n for c in np.unique(target):\n class_log_probs = log_probs[target == c][:, c]\n class_weight = weights[c]\n res += class_weight * class_log_probs.mean()\n return - (res / weights.sum())\n\n\ndef weighted_log_loss_pytorch(probs, weights, target):\n probs = np.array(probs)\n log_probs = np.log(probs)\n weights = np.array(weights)\n\n target = np.array(target)\n res = 0\n weight_sum = 0\n for c in np.unique(target):\n class_log_probs = log_probs[target == c][:, c]\n class_weight = weights[c]\n res += class_weight * class_log_probs.sum()\n weight_sum += len(class_log_probs) * class_weight\n return - (res / weight_sum)\n\n\n>>> weighted_log_loss_pytorch([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]], [0.3, 0.7], [0, 0, 1])\n>>> 0.7904898498886928\n\n# Checking against official pytorch\n>>> nll_loss = nn.NLLLoss(weight=torch.tensor([0.3, 0.7]))\n>>> nll_loss(torch.log(torch.tensor([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]])), torch.tensor([0, 0, 1]))\n>>> tensor(0.7905)\n\n>>> weighted_log_loss_kaggle([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]], [0.3, 0.7], [0, 0, 1])\n>>> 0.9352088839417277\nHow did you come up with these weights [0.3, 0.7]?
", "rawMarkdown": "How did you come up with these weights [0.3, 0.7]?", "votes": 1, "replies": [ { "id": 1957413, "postDate": "2022-09-27T00:47:00.163Z", "content": "@salmanahmedtamu they are just arbitrary weights used for the example
", "rawMarkdown": "@salmanahmedtamu they are just arbitrary weights used for the example" } ] }, { "id": 1956553, "postDate": "2022-09-26T14:04:46.773Z", "content": "@aamster I have created this for TensorFlow. Can you point out in case there is any mistake in the computation considering each class has weight of 0.5.
\ndef test(y_true, y_pred, weights = tf.convert_to_tensor([0.5, 0.5])):\n y_true = tf.cast(y_true, dtype = tf.float32)\n y_pred = tf.cast(y_pred, dtype = tf.float32)\n indices_first = tf.where(tf.equal(y_true, tf.Variable(0.0)))\n y_true_gather = tf.reshape(tf.gather_nd(y_true, indices_first), (-1, 1))\n y_pred_gather = tf.reshape(tf.gather_nd(tf.math.log(y_pred), indices_first), (-1, 1))\n first = tf.matmul(tf.transpose(y_true_gather), y_pred_gather) * weights[0]\n indices_second = tf.where(tf.equal(y_true, tf.Variable(1.0)))\n y_true_gather = tf.reshape(tf.gather_nd(y_true, indices_second), (-1, 1))\n y_pred_gather = tf.reshape(tf.gather_nd(tf.math.log(y_pred), indices_second), (-1, 1))\n second = tf.matmul(tf.transpose(y_true_gather), y_pred_gather) * weights[1]\n result = tf.reduce_mean(first) + tf.reduce_mean(second)\n return -result\nI wrote a function with loops :
\ndef fc_log_loss(W,table):
\n D = W.sum()
\n sigma=0
\n N = [0,0]
\n N[0] = table['CE_true'].sum()
\n N[1]+=table['LAA_true'].sum()
\n for ind_class in [0,1]:
\n for img_ind in range(0,len(table)):
\n if ind_class == 0:
\n y = table['CE_true'].iloc[img_ind]
\n class_pred = table['CE']
\n else:
\n y = table['LAA_true'].iloc[img_ind]
\n class_pred = table['LAA']
\n sigma+= y/N[ind_class]*math.log(class_pred.iloc[img_ind])
sigma = sigma*W[ind_class]\nreturn -sigma/D \nThis function does not give log loss = 0.6 something for the base level[0.5,0.5] but gives 0.63 for [0.3,0.7]
", "rawMarkdown": "I wrote a function with loops : \ndef fc_log_loss(W,table):\n D = W.sum()\n sigma=0\n N = [0,0]\n N[0] = table['CE_true'].sum()\n N[1]+=table['LAA_true'].sum()\n for ind_class in [0,1]:\n for img_ind in range(0,len(table)):\n if ind_class == 0:\n y = table['CE_true'].iloc[img_ind]\n class_pred = table['CE']\n else:\n y = table['LAA_true'].iloc[img_ind]\n class_pred = table['LAA']\n sigma+= y/N[ind_class]*math.log(class_pred.iloc[img_ind])\n \n sigma = sigma*W[ind_class]\n return -sigma/D \n\nThis function does not give log loss = 0.6 something for the base level[0.5,0.5] but gives 0.63 for [0.3,0.7]" } ] }, { "id": 1944463, "postDate": "2022-09-18T10:25:28.727Z", "content": "I agree that there are typos, but I think the Kaggle definition is reasonable and very good. Let's use it.😄
", "rawMarkdown": "I agree that there are typos, but I think the Kaggle definition is reasonable and very good. Let's use it.😄", "replies": [ { "id": 1957414, "postDate": "2022-09-27T00:49:35.550Z", "content": "I feel like this is a wrong approach. As far as I know, class weights are used while training a CNN and not during evaluation. These weights can help in fixing the bias of the model towards a specific class in class imbalance issue and make the weights generalize better to class imbalance. Employing these weights would remove that generalization of weights and make it more biased towards a specific class. Do correct me if I am wrong.
", "rawMarkdown": "I feel like this is a wrong approach. As far as I know, class weights are used while training a CNN and not during evaluation. These weights can help in fixing the bias of the model towards a specific class in class imbalance issue and make the weights generalize better to class imbalance. Employing these weights would remove that generalization of weights and make it more biased towards a specific class. Do correct me if I am wrong. " }, { "id": 1957420, "postDate": "2022-09-27T01:01:19.947Z", "content": "Exactly @tr1gg3rtrash but I think, if in evaluation they have imbalance distribution of classes, then it might make sense to add weights in evaluation.
", "rawMarkdown": "Exactly @tr1gg3rtrash but I think, if in evaluation they have imbalance distribution of classes, then it might make sense to add weights in evaluation." }, { "id": 1957474, "postDate": "2022-09-27T02:23:55.283Z", "content": "Nope. In case of imbalance, the model has to be generalized. Hence, due to that, you add class_weights param in the fit method while training using TensorFlow. This will make the model adjusted to the class imbalance present in the training set. We cant use these weights during evaluation as after training model already knows which class is in majority and which is in minority. Hence if we do a weighted evaluation it will ultimately push the minority class with more class weights and in that case major class with suffering by a factor of 2. First by the generalized model and second by the class weights during evaluation.
", "rawMarkdown": "Nope. In case of imbalance, the model has to be generalized. Hence, due to that, you add class_weights param in the fit method while training using TensorFlow. This will make the model adjusted to the class imbalance present in the training set. We cant use these weights during evaluation as after training model already knows which class is in majority and which is in minority. Hence if we do a weighted evaluation it will ultimately push the minority class with more class weights and in that case major class with suffering by a factor of 2. First by the generalized model and second by the class weights during evaluation. " }, { "id": 1962727, "postDate": "2022-09-29T23:11:44.397Z", "content": "It's a definition in evaluation (not an approach). Don't you think the equation beautiful?😄
", "rawMarkdown": "It's a definition in evaluation (not an approach). Don't you think the equation beautiful?😄" } ] } ], "comments": [ { "id": 1957412, "author_name": "Salman Ahmed", "author_url": "", "post_date": "2022-09-27T00:45:59.893000", "content": "How did you come up with these weights [0.3, 0.7]?
", "votes": 1, "replies": [ { "id": 1957413, "author_name": "cosmosaa", "author_url": "", "post_date": "2022-09-27T00:47:00.163000", "content": "@salmanahmedtamu they are just arbitrary weights used for the example
", "votes": 0, "replies": [] } ] }, { "id": 1956553, "author_name": "Mrinal Tyagi", "author_url": "", "post_date": "2022-09-26T14:04:46.773000", "content": "@aamster I have created this for TensorFlow. Can you point out in case there is any mistake in the computation considering each class has weight of 0.5.
\ndef test(y_true, y_pred, weights = tf.convert_to_tensor([0.5, 0.5])):\n y_true = tf.cast(y_true, dtype = tf.float32)\n y_pred = tf.cast(y_pred, dtype = tf.float32)\n indices_first = tf.where(tf.equal(y_true, tf.Variable(0.0)))\n y_true_gather = tf.reshape(tf.gather_nd(y_true, indices_first), (-1, 1))\n y_pred_gather = tf.reshape(tf.gather_nd(tf.math.log(y_pred), indices_first), (-1, 1))\n first = tf.matmul(tf.transpose(y_true_gather), y_pred_gather) * weights[0]\n indices_second = tf.where(tf.equal(y_true, tf.Variable(1.0)))\n y_true_gather = tf.reshape(tf.gather_nd(y_true, indices_second), (-1, 1))\n y_pred_gather = tf.reshape(tf.gather_nd(tf.math.log(y_pred), indices_second), (-1, 1))\n second = tf.matmul(tf.transpose(y_true_gather), y_pred_gather) * weights[1]\n result = tf.reduce_mean(first) + tf.reduce_mean(second)\n return -result\nI wrote a function with loops :
\ndef fc_log_loss(W,table):
\n D = W.sum()
\n sigma=0
\n N = [0,0]
\n N[0] = table['CE_true'].sum()
\n N[1]+=table['LAA_true'].sum()
\n for ind_class in [0,1]:
\n for img_ind in range(0,len(table)):
\n if ind_class == 0:
\n y = table['CE_true'].iloc[img_ind]
\n class_pred = table['CE']
\n else:
\n y = table['LAA_true'].iloc[img_ind]
\n class_pred = table['LAA']
\n sigma+= y/N[ind_class]*math.log(class_pred.iloc[img_ind])
sigma = sigma*W[ind_class]\nreturn -sigma/D \nThis function does not give log loss = 0.6 something for the base level[0.5,0.5] but gives 0.63 for [0.3,0.7]
", "votes": 0, "replies": [] } ] }, { "id": 1944463, "author_name": "Hiroshi Sakiyama", "author_url": "", "post_date": "2022-09-18T10:25:28.727000", "content": "I agree that there are typos, but I think the Kaggle definition is reasonable and very good. Let's use it.😄
", "votes": 0, "replies": [ { "id": 1957414, "author_name": "Mrinal Tyagi", "author_url": "", "post_date": "2022-09-27T00:49:35.550000", "content": "I feel like this is a wrong approach. As far as I know, class weights are used while training a CNN and not during evaluation. These weights can help in fixing the bias of the model towards a specific class in class imbalance issue and make the weights generalize better to class imbalance. Employing these weights would remove that generalization of weights and make it more biased towards a specific class. Do correct me if I am wrong.
", "votes": 0, "replies": [] }, { "id": 1957420, "author_name": "Salman Ahmed", "author_url": "", "post_date": "2022-09-27T01:01:19.947000", "content": "Exactly @tr1gg3rtrash but I think, if in evaluation they have imbalance distribution of classes, then it might make sense to add weights in evaluation.
", "votes": 0, "replies": [] }, { "id": 1957474, "author_name": "Mrinal Tyagi", "author_url": "", "post_date": "2022-09-27T02:23:55.283000", "content": "Nope. In case of imbalance, the model has to be generalized. Hence, due to that, you add class_weights param in the fit method while training using TensorFlow. This will make the model adjusted to the class imbalance present in the training set. We cant use these weights during evaluation as after training model already knows which class is in majority and which is in minority. Hence if we do a weighted evaluation it will ultimately push the minority class with more class weights and in that case major class with suffering by a factor of 2. First by the generalized model and second by the class weights during evaluation.
", "votes": 0, "replies": [] }, { "id": 1962727, "author_name": "Hiroshi Sakiyama", "author_url": "", "post_date": "2022-09-29T23:11:44.397000", "content": "It's a definition in evaluation (not an approach). Don't you think the equation beautiful?😄
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1943757": "Looking at the weighted log loss defined for this competition, it is different than the pytorch definition of weighted log loss, since it takes weighted average from the average loss of each class, rather than taking a weighted average of the loss for each input.\n\nThis ignores the fact that there's a major typo in the metric definition which confuses i and j that was never fixed. \n\nTherefore, if you use the pytorch implementation of weighted log loss, you'll get something different than what this competition defines. \n\nIn code:\n```\nimport numpy as np\n\n\ndef weighted_log_loss_kaggle(probs, weights, target):\n probs = np.array(probs)\n log_probs = np.log(probs)\n weights = np.array(weights)\n\n target = np.array(target)\n res = 0\n for c in np.unique(target):\n class_log_probs = log_probs[target == c][:, c]\n class_weight = weights[c]\n res += class_weight * class_log_probs.mean()\n return - (res / weights.sum())\n\n\ndef weighted_log_loss_pytorch(probs, weights, target):\n probs = np.array(probs)\n log_probs = np.log(probs)\n weights = np.array(weights)\n\n target = np.array(target)\n res = 0\n weight_sum = 0\n for c in np.unique(target):\n class_log_probs = log_probs[target == c][:, c]\n class_weight = weights[c]\n res += class_weight * class_log_probs.sum()\n weight_sum += len(class_log_probs) * class_weight\n return - (res / weight_sum)\n\n\n>>> weighted_log_loss_pytorch([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]], [0.3, 0.7], [0, 0, 1])\n>>> 0.7904898498886928\n\n# Checking against official pytorch\n>>> nll_loss = nn.NLLLoss(weight=torch.tensor([0.3, 0.7]))\n>>> nll_loss(torch.log(torch.tensor([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]])), torch.tensor([0, 0, 1]))\n>>> tensor(0.7905)\n\n>>> weighted_log_loss_kaggle([[0.9, 0.1], [0.6, 0.4], [0.7, 0.3]], [0.3, 0.7], [0, 0, 1])\n>>> 0.9352088839417277\n```", "1957412": "How did you come up with these weights [0.3, 0.7]?", "1956553": "@aamster I have created this for TensorFlow. Can you point out in case there is any mistake in the computation considering each class has weight of 0.5. \n\n```\ndef test(y_true, y_pred, weights = tf.convert_to_tensor([0.5, 0.5])):\n y_true = tf.cast(y_true, dtype = tf.float32)\n y_pred = tf.cast(y_pred, dtype = tf.float32)\n indices_first = tf.where(tf.equal(y_true, tf.Variable(0.0)))\n y_true_gather = tf.reshape(tf.gather_nd(y_true, indices_first), (-1, 1))\n y_pred_gather = tf.reshape(tf.gather_nd(tf.math.log(y_pred), indices_first), (-1, 1))\n first = tf.matmul(tf.transpose(y_true_gather), y_pred_gather) * weights[0]\n indices_second = tf.where(tf.equal(y_true, tf.Variable(1.0)))\n y_true_gather = tf.reshape(tf.gather_nd(y_true, indices_second), (-1, 1))\n y_pred_gather = tf.reshape(tf.gather_nd(tf.math.log(y_pred), indices_second), (-1, 1))\n second = tf.matmul(tf.transpose(y_true_gather), y_pred_gather) * weights[1]\n result = tf.reduce_mean(first) + tf.reduce_mean(second)\n return -result\n\n```", "1944463": "I agree that there are typos, but I think the Kaggle definition is reasonable and very good. Let's use it.😄" } }