![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12313384%2Fcb07be6420caa6786c339d9fa71e88e4%2F2023-08-19%2021.37.42.png?generation=1692452862606079&alt=media\")
I'm confused about the metric. Are the losses weighted across injury types or patients? In the implementation notebook, the API log_loss in sklearn is used. And the sample_weight argument should be the number of data samples (patients) instead of nubmer of classes. So, if the losses are weighted across injury types, I don't how this API can achieve it. If they are weighted across patients, then how should we assign weight values to each patient? What if a patient has multiple injuries?
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12313384%2Fbf9b21b87f8a7628280a4166215da87d%2F2023-08-19%2021.38.09.png?generation=1692452875758048&alt=media\")
I'm confused about the metric. Are the losses weighted across injury types or patients? In the implementation notebook, the API log_loss in sklearn is used. And the sample_weight argument should be the number of data samples (patients) instead of nubmer of classes. So, if the losses are weighted across injury types, I don't how this API can achieve it. If they are weighted across patients, then how should we assign weight values to each patient? What if a patient has multiple injuries?
As far as I understand from competition evaluation section and competition metric notebook, it seems that the weights are fixed and each log loss calculated separately and then average is taken. To correctly calculate the score using sklearn, you need to create a sample weight array consisting of fixed weights for each patient. For example, let's assume we create a sample weight array for predictions of three patients.
\n\n\n1 for all healthy labels.
\n
\n 2 for low grade solid organ injuries (liver, spleen, kidney).
\n 4 for high grade solid organ injuries.
\n 2 for bowel injuries.
\n 6 for extravasation.
\n 6 for the auto-generated any_injury label.
Sample weights arrays should be as follows
\nAfter calculating these scores separately for each prediction type, taking the average of them should give the competition metric.
", "rawMarkdown": "As far as I understand from competition evaluation section and competition metric notebook, it seems that the weights are fixed and each log loss calculated separately and then average is taken. To correctly calculate the score using sklearn, you need to create a sample weight array consisting of fixed weights for each patient. For example, let's assume we create a sample weight array for predictions of three patients. \n\n> 1 for all healthy labels.\n2 for low grade solid organ injuries (liver, spleen, kidney).\n4 for high grade solid organ injuries.\n2 for bowel injuries.\n6 for extravasation.\n6 for the auto-generated any_injury label.\n\n- Patient 1 : bowel_injury = 1 / kidney_low = 1 / spleen_healthy = 1\n- Patient 2 : bowel_healthy = 1 / kidney_high = 1 / spleen_high = 1\n- Patient 3 : bowel_injury = 1 / kidney_healthy = 1 / spleen_low = 1\n\nSample weights arrays should be as follows\n- Bowel_injury_log_loss_weights = [ 2, 1, 2 ]\n- Kidney_injury_log_loss_weights = [ 2, 4, 1 ]\n- Spleen_healty_log_loss_weights = [ 1, 4, 2 ] \n\nAfter calculating these scores separately for each prediction type, taking the average of them should give the competition metric. \n", "votes": 4, "replies": [ { "id": 2404404, "postDate": "2023-08-23T08:25:56.297Z", "content": "That makes sense. Thanks a lot.
", "rawMarkdown": "That makes sense. Thanks a lot." } ] }, { "id": 2401964, "postDate": "2023-08-21T23:39:06.820Z", "content": "I think the weights in the solution dataframe are just stored as ones Series multiplied by the organ-condition weight value. So it is constant across patients but each weight has it's own column in the solution df.
", "rawMarkdown": "I think the weights in the solution dataframe are just stored as ones Series multiplied by the organ-condition weight value. So it is constant across patients but each weight has it's own column in the solution df.", "replies": [ { "id": 2404409, "postDate": "2023-08-23T08:34:20.080Z", "content": "Thanks for reply! But I'm afraid this is not correct. Because the loss will be normalized. Please try the following code:
\nl1 = log_loss(\n y_true=[,],\n y_pred=[,],\n normalize=,\n sample_weight=[,]\n)\nl2 = log_loss(\n y_true=[,],\n y_pred=[,],\n normalize=,\n sample_weight=[,]\n)\nl1 and l2 are the same.
", "rawMarkdown": "Thanks for reply! But I'm afraid this is not correct. Because the loss will be normalized. Please try the following code:\n```python\nl1 = log_loss(\n y_true=[1,0],\n y_pred=[0.8,0.2],\n normalize=True,\n sample_weight=[1,1]\n)\nl2 = log_loss(\n y_true=[1,0],\n y_pred=[0.8,0.2],\n normalize=True,\n sample_weight=[2,2]\n)\n```\nl1 and l2 are the same.", "votes": 2, "replies": [ { "id": 2404851, "postDate": "2023-08-23T14:43:28.653Z", "content": "Oh that's right.. I just read the comment above me ans that makes perfect sense! Thankd!
", "rawMarkdown": "Oh that's right.. I just read the comment above me ans that makes perfect sense! Thankd!" } ] } ] } ], "comments": [ { "id": 2401076, "author_name": "turkenm", "author_url": "", "post_date": "2023-08-21T12:16:58.610000", "content": "As far as I understand from competition evaluation section and competition metric notebook, it seems that the weights are fixed and each log loss calculated separately and then average is taken. To correctly calculate the score using sklearn, you need to create a sample weight array consisting of fixed weights for each patient. For example, let's assume we create a sample weight array for predictions of three patients.
\n\n\n1 for all healthy labels.
\n
\n 2 for low grade solid organ injuries (liver, spleen, kidney).
\n 4 for high grade solid organ injuries.
\n 2 for bowel injuries.
\n 6 for extravasation.
\n 6 for the auto-generated any_injury label.
Sample weights arrays should be as follows
\nAfter calculating these scores separately for each prediction type, taking the average of them should give the competition metric.
", "votes": 4, "replies": [ { "id": 2404404, "author_name": "PT0X0E", "author_url": "", "post_date": "2023-08-23T08:25:56.297000", "content": "That makes sense. Thanks a lot.
", "votes": 0, "replies": [] } ] }, { "id": 2401964, "author_name": "Parham Mostame", "author_url": "", "post_date": "2023-08-21T23:39:06.820000", "content": "I think the weights in the solution dataframe are just stored as ones Series multiplied by the organ-condition weight value. So it is constant across patients but each weight has it's own column in the solution df.
", "votes": 0, "replies": [ { "id": 2404409, "author_name": "PT0X0E", "author_url": "", "post_date": "2023-08-23T08:34:20.080000", "content": "Thanks for reply! But I'm afraid this is not correct. Because the loss will be normalized. Please try the following code:
\nl1 = log_loss(\n y_true=[,],\n y_pred=[,],\n normalize=,\n sample_weight=[,]\n)\nl2 = log_loss(\n y_true=[,],\n y_pred=[,],\n normalize=,\n sample_weight=[,]\n)\nl1 and l2 are the same.
", "votes": 2, "replies": [ { "id": 2404851, "author_name": "Parham Mostame", "author_url": "", "post_date": "2023-08-23T14:43:28.653000", "content": "Oh that's right.. I just read the comment above me ans that makes perfect sense! Thankd!
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2398150": "I'm confused about the metric. Are the losses weighted across injury types or patients? In the implementation notebook, the API `log_loss` in sklearn is used. And the `sample_weight` argument should be the number of data samples (patients) instead of nubmer of classes. So, if the losses are weighted across injury types, I don't how this API can achieve it. If they are weighted across patients, then how should we assign weight values to each patient? What if a patient has multiple injuries?\n\n\n\n", "2401076": "As far as I understand from competition evaluation section and competition metric notebook, it seems that the weights are fixed and each log loss calculated separately and then average is taken. To correctly calculate the score using sklearn, you need to create a sample weight array consisting of fixed weights for each patient. For example, let's assume we create a sample weight array for predictions of three patients. \n\n> 1 for all healthy labels.\n2 for low grade solid organ injuries (liver, spleen, kidney).\n4 for high grade solid organ injuries.\n2 for bowel injuries.\n6 for extravasation.\n6 for the auto-generated any_injury label.\n\n- Patient 1 : bowel_injury = 1 / kidney_low = 1 / spleen_healthy = 1\n- Patient 2 : bowel_healthy = 1 / kidney_high = 1 / spleen_high = 1\n- Patient 3 : bowel_injury = 1 / kidney_healthy = 1 / spleen_low = 1\n\nSample weights arrays should be as follows\n- Bowel_injury_log_loss_weights = [ 2, 1, 2 ]\n- Kidney_injury_log_loss_weights = [ 2, 4, 1 ]\n- Spleen_healty_log_loss_weights = [ 1, 4, 2 ] \n\nAfter calculating these scores separately for each prediction type, taking the average of them should give the competition metric. \n", "2401964": "I think the weights in the solution dataframe are just stored as ones Series multiplied by the organ-condition weight value. So it is constant across patients but each weight has it's own column in the solution df." } }