Authors: I-Min Chiu, Wun-Huei Zeng, Chi-Yung Cheng, Shih-Hsuan Chen, and Chun-Hung Richard Lin
\nDiagnostics (Basel). 2021 Jan; 11(1): 80. - Published online 2021 Jan 6. doi: 10.3390/diagnostics11010080
\n\"Prediction of functional outcome in ischemic stroke patients is useful for clinical decisions. Previous studies mostly elaborate on the prediction of favorable outcomes. Miserable outcomes, which are usually defined as modified Rankin Scale (mRS) 5–6, should be considered as well before further invasive intervention. By using a machine learning algorithm, the authors aimed to develop a multiclass classification model for outcome prediction in acute ischemic stroke patients requiring reperfusion therapy.\"
\n\" Patients with acute ischemic stroke who visited between January 2016 and December 2019 and who were candidates for reperfusion therapy were included. Clinical outcomes were classified as favorable outcome, intermediate outcome, and miserable outcome.\"
\n\"The authors developed four different multiclass machine learning models (Logistic Regression, Supportive Vector Machine, Random Forest, and Extreme Gradient Boosting) to predict clinical outcomes and compared their performance to the DRAGON score.\"
\n\"All selected machine learning models outperformed the DRAGON score on accuracy of outcome prediction (Logistic Regression: 0.70, Supportive Vector Machine: 0.67, Random Forest: 0.69, and Extreme Gradient Boosting: 0.67, vs. DRAGON: 0.51, p < 0.001).\"
\n\" Among all selected models, Logistic Regression also had a better performance than the DRAGON score on positive predictive value, sensitivity, and specificity. Compared with the DRAGON score, the multiclass machine learning approach showed better performance on the prediction of the 3-month functional outcome of acute ischemic stroke patients requiring reperfusion therapy.\"
\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7825282/
\n\"The DRAGON score is a scoring system composed by six different variables evaluated from stroke patients at admission. It was created in 2012 to predict both favorable and miserable functional outcomes 3 months after the stroke , and it has been validated by several studies with good performance.\"
\n\"On the other hand, to the best of the autors knowledge, there is no ML approach for the prediction of the whole spectrum functional outcomes in stroke patients. To assist with the generation of accurate treatment decisions, comprehensive outcome prediction for acute stroke patients is needed. Thus, this study aims to develop a multiclass machine learning prediction model on the 3-month outcome of acute ischemic stroke patients who were candidates for reperfusion therapy at the time of admission.\"
\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7825282/
\nMagnetic Resonance Imaging-DRAGON Score
\n\"The DRAGON score, which includes clinical and computed tomographic scan parameters, showed a high specificity to predict 3-month outcome in patients with acute ischemic stroke treated by intravenous tissue plasminogen activator. We adapted the score for patients undergoing MRI as the first-line diagnostic tool.\"
\n\nNotebook: Maybe *_any weight is x2 of others By Kambarakun
\nhttps://www.kaggle.com/code/kambarakun/maybe-any-weight-is-x2-of-others/script
\nDiscussion Topic: \"Tricks for Kaggle log loss\" By the amazing GM hengck23
\n\"sampling/weighing is important in training
\nwhen the metric is accuracy, we usually use : predict_label = predict_prob > threshold
\nwe do not care about predict_prob, but rather thresholded prediction
\nwe sometimes use loss weighing (e.g focal loss) or class balancing to speed up training or improve accuracy. but improving accuracy is \"not the same\" as reducing loss. the loss is modified to improve accuracy by artificially changing the train distribution and affecting the decision boundary.
\"for this challenge, they care about the actual log loss itself. so it is important to \"calibrate your probability\" to the actual one you expect in the test during training. you may have to use random sampling (instead of balanced sampling) at the final fine tuning. likewise you may need to disable any sample weighing.\"
\nprobability shaping
\n\"this is is very difficult (and very risky in scoring). e.g. a test sample is predicted to be negative with probability 0.0001. you may round it to zero. you will gain a little if the truth is really negative, but you will lose big if it is truth is instead positive.\"
\n\"if you can train with \"large margin\" and use some probability calibration trick such that you are almost 100% sure that there cannot be positive truth below a certain low values, you can zero out the values.\"
\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/114627
\n\"Relationship between accuracy and loss?\" By Ryan Epp
\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/114087
\n\"Code of weighted log loss is here\" By Rajnish Chauhan
\nTake care of BATCH_SIZE, it is same as taken in ImageGenerator.
\nUse \"custom_log_loss\" in model.compile( loss ='customers_log_loss')
\ndef custom_log_loss(y_true, y_pred):
\nweights_cons = tf.constant([2.0, 1.0, 1.0, 1.0, 1.0, 1.0])
\nloss_val = tf.losses.sigmoid_cross_entropy(y_true, y_pred, weights = tf.broadcast_to(weights_cons, [BATCH_SIZE, 6]))
loss_val = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)
\nloss_val = tf.losses.sigmoid_cross_entropy(y_true, y_pred)
\nprint(y_true.shape)
\nprint(ck_val)
\nreturn loss_val
https://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/111719
\n\"Why were the weight values in the evaluation metric not given to us? \"
\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/109447
\nDemographic characteristics of included Stroke patients:
\nMale, Mean Age (67.9 ± 12.4), Sugar (mg/dL: 141 ± 57), Pre-Stroke, Old Stroke, Atrial fibrillation (227 (38.5)), Diabetes mellitus,
\nHypertension, Dyslipidemia, Coronary artery disease, Heart failure, Smoking (Source NIHSS: National Institutes of Health Stroke Scale)
Authors: I-Min Chiu, Wun-Huei Zeng, Chi-Yung Cheng, Shih-Hsuan Chen, and Chun-Hung Richard Lin
\nDiagnostics (Basel). 2021 Jan; 11(1): 80. - Published online 2021 Jan 6. doi: 10.3390/diagnostics11010080
\n\"Prediction of functional outcome in ischemic stroke patients is useful for clinical decisions. Previous studies mostly elaborate on the prediction of favorable outcomes. Miserable outcomes, which are usually defined as modified Rankin Scale (mRS) 5–6, should be considered as well before further invasive intervention. By using a machine learning algorithm, the authors aimed to develop a multiclass classification model for outcome prediction in acute ischemic stroke patients requiring reperfusion therapy.\"
\n\" Patients with acute ischemic stroke who visited between January 2016 and December 2019 and who were candidates for reperfusion therapy were included. Clinical outcomes were classified as favorable outcome, intermediate outcome, and miserable outcome.\"
\n\"The authors developed four different multiclass machine learning models (Logistic Regression, Supportive Vector Machine, Random Forest, and Extreme Gradient Boosting) to predict clinical outcomes and compared their performance to the DRAGON score.\"
\n\"All selected machine learning models outperformed the DRAGON score on accuracy of outcome prediction (Logistic Regression: 0.70, Supportive Vector Machine: 0.67, Random Forest: 0.69, and Extreme Gradient Boosting: 0.67, vs. DRAGON: 0.51, p < 0.001).\"
\n\" Among all selected models, Logistic Regression also had a better performance than the DRAGON score on positive predictive value, sensitivity, and specificity. Compared with the DRAGON score, the multiclass machine learning approach showed better performance on the prediction of the 3-month functional outcome of acute ischemic stroke patients requiring reperfusion therapy.\"
\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7825282/
\n\"The DRAGON score is a scoring system composed by six different variables evaluated from stroke patients at admission. It was created in 2012 to predict both favorable and miserable functional outcomes 3 months after the stroke , and it has been validated by several studies with good performance.\"
\n\"On the other hand, to the best of the autors knowledge, there is no ML approach for the prediction of the whole spectrum functional outcomes in stroke patients. To assist with the generation of accurate treatment decisions, comprehensive outcome prediction for acute stroke patients is needed. Thus, this study aims to develop a multiclass machine learning prediction model on the 3-month outcome of acute ischemic stroke patients who were candidates for reperfusion therapy at the time of admission.\"
\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7825282/
\nMagnetic Resonance Imaging-DRAGON Score
\n\"The DRAGON score, which includes clinical and computed tomographic scan parameters, showed a high specificity to predict 3-month outcome in patients with acute ischemic stroke treated by intravenous tissue plasminogen activator. We adapted the score for patients undergoing MRI as the first-line diagnostic tool.\"
\n\nNotebook: Maybe *_any weight is x2 of others By Kambarakun
\nhttps://www.kaggle.com/code/kambarakun/maybe-any-weight-is-x2-of-others/script
\nDiscussion Topic: \"Tricks for Kaggle log loss\" By the amazing GM hengck23
\n\"sampling/weighing is important in training
\nwhen the metric is accuracy, we usually use : predict_label = predict_prob > threshold
\nwe do not care about predict_prob, but rather thresholded prediction
\nwe sometimes use loss weighing (e.g focal loss) or class balancing to speed up training or improve accuracy. but improving accuracy is \"not the same\" as reducing loss. the loss is modified to improve accuracy by artificially changing the train distribution and affecting the decision boundary.
\"for this challenge, they care about the actual log loss itself. so it is important to \"calibrate your probability\" to the actual one you expect in the test during training. you may have to use random sampling (instead of balanced sampling) at the final fine tuning. likewise you may need to disable any sample weighing.\"
\nprobability shaping
\n\"this is is very difficult (and very risky in scoring). e.g. a test sample is predicted to be negative with probability 0.0001. you may round it to zero. you will gain a little if the truth is really negative, but you will lose big if it is truth is instead positive.\"
\n\"if you can train with \"large margin\" and use some probability calibration trick such that you are almost 100% sure that there cannot be positive truth below a certain low values, you can zero out the values.\"
\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/114627
\n\"Relationship between accuracy and loss?\" By Ryan Epp
\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/114087
\n\"Code of weighted log loss is here\" By Rajnish Chauhan
\nTake care of BATCH_SIZE, it is same as taken in ImageGenerator.
\nUse \"custom_log_loss\" in model.compile( loss ='customers_log_loss')
\ndef custom_log_loss(y_true, y_pred):
\nweights_cons = tf.constant([2.0, 1.0, 1.0, 1.0, 1.0, 1.0])
\nloss_val = tf.losses.sigmoid_cross_entropy(y_true, y_pred, weights = tf.broadcast_to(weights_cons, [BATCH_SIZE, 6]))
loss_val = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)
\nloss_val = tf.losses.sigmoid_cross_entropy(y_true, y_pred)
\nprint(y_true.shape)
\nprint(ck_val)
\nreturn loss_val
https://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/111719
\n\"Why were the weight values in the evaluation metric not given to us? \"
\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/109447
\nDemographic characteristics of included Stroke patients:
\nMale, Mean Age (67.9 ± 12.4), Sugar (mg/dL: 141 ± 57), Pre-Stroke, Old Stroke, Atrial fibrillation (227 (38.5)), Diabetes mellitus,
\nHypertension, Dyslipidemia, Coronary artery disease, Heart failure, Smoking (Source NIHSS: National Institutes of Health Stroke Scale)
\n\nBy the way, what are the weights?
\n
The evaluation page says that 'each class is roughly equally important for the final score'. So I'm guessing that both have equal weights.
\nThen by simple lb probing (committing 0 and 1), the results of my calculation support my guess.
Thank you zzy990106.
\nI decide to include that topic from the last Competition RSNA Intracranial Hemorrhage Detection.
\"Why were the weight values in the evaluation metric not given to us? \"
\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/109447
Since here is Ischemic, I found interesting to add that too ;)
\nCongrats for you 18th place (till now) @zzy990106
", "rawMarkdown": "Thank you zzy990106.\nI decide to include that topic from the last Competition RSNA Intracranial Hemorrhage Detection.\n\n\"Why were the weight values in the evaluation metric not given to us? \"\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/109447\n\nSince here is Ischemic, I found interesting to add that too ;)\n\nCongrats for you 18th place (till now) @zzy990106 \n", "votes": 2 }, { "id": 1920852, "postDate": "2022-08-31T12:53:59.927Z", "content": "Each class being roughly equal to me means that the less common class has a higher weight so as to make both classes as important when calculating the loss
", "rawMarkdown": "Each class being roughly equal to me means that the less common class has a higher weight so as to make both classes as important when calculating the loss" }, { "id": 1938546, "postDate": "2022-09-14T08:06:02.427Z", "content": "The number of samples are already devided before mutiplying by the weights, so each class would be equal important iff weights are equal
", "rawMarkdown": "The number of samples are already devided before mutiplying by the weights, so each class would be equal important iff weights are equal" }, { "id": 1944007, "postDate": "2022-09-18T01:35:52.687Z", "content": "Thank you @azirzhang , you are right. It is different from the usual definition of weighted log loss
", "rawMarkdown": "Thank you @azirzhang , you are right. It is different from the usual definition of weighted log loss" } ] }, { "id": 1851831, "postDate": "2022-07-11T15:08:04.440Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1848292, "author_name": "Leon", "author_url": "", "post_date": "2022-07-08T13:55:10.763000", "content": "\n\nBy the way, what are the weights?
\n
The evaluation page says that 'each class is roughly equally important for the final score'. So I'm guessing that both have equal weights.
\nThen by simple lb probing (committing 0 and 1), the results of my calculation support my guess.
Thank you zzy990106.
\nI decide to include that topic from the last Competition RSNA Intracranial Hemorrhage Detection.
\"Why were the weight values in the evaluation metric not given to us? \"
\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/109447
Since here is Ischemic, I found interesting to add that too ;)
\nCongrats for you 18th place (till now) @zzy990106
", "votes": 2, "replies": [] }, { "id": 1920852, "author_name": "cosmosaa", "author_url": "", "post_date": "2022-08-31T12:53:59.927000", "content": "Each class being roughly equal to me means that the less common class has a higher weight so as to make both classes as important when calculating the loss
", "votes": 0, "replies": [] }, { "id": 1938546, "author_name": "Chen Zhang", "author_url": "", "post_date": "2022-09-14T08:06:02.427000", "content": "The number of samples are already devided before mutiplying by the weights, so each class would be equal important iff weights are equal
", "votes": 0, "replies": [] }, { "id": 1944007, "author_name": "cosmosaa", "author_url": "", "post_date": "2022-09-18T01:35:52.687000", "content": "Thank you @azirzhang , you are right. It is different from the usual definition of weighted log loss
", "votes": 0, "replies": [] } ] }, { "id": 1851831, "author_name": "", "author_url": "", "post_date": "2022-07-11T15:08:04.440000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1847390": "#Using a Multiclass Machine Learning Model to Predict the Outcome of Acute Ischemic Stroke Requiring Reperfusion Therapy\n\nAuthors: I-Min Chiu, Wun-Huei Zeng, Chi-Yung Cheng, Shih-Hsuan Chen, and Chun-Hung Richard Lin\n\nDiagnostics (Basel). 2021 Jan; 11(1): 80. - Published online 2021 Jan 6. doi: 10.3390/diagnostics11010080\n\n\"Prediction of functional outcome in ischemic stroke patients is useful for clinical decisions. Previous studies mostly elaborate on the prediction of favorable outcomes. Miserable outcomes, which are usually defined as modified Rankin Scale (mRS) 5–6, should be considered as well before further invasive intervention. By using a machine learning algorithm, the authors aimed to develop a multiclass classification model for outcome prediction in acute ischemic stroke patients requiring reperfusion therapy.\"\n\n\" Patients with acute ischemic stroke who visited between January 2016 and December 2019 and who were candidates for reperfusion therapy were included. Clinical outcomes were classified as favorable outcome, intermediate outcome, and miserable outcome.\"\n\n\"The authors developed four different multiclass machine learning models (Logistic Regression, Supportive Vector Machine, Random Forest, and Extreme Gradient Boosting) to predict clinical outcomes and compared their performance to the DRAGON score.\"\n\n\"All selected machine learning models outperformed the DRAGON score on accuracy of outcome prediction (Logistic Regression: 0.70, Supportive Vector Machine: 0.67, Random Forest: 0.69, and Extreme Gradient Boosting: 0.67, vs. DRAGON: 0.51, p < 0.001).\"\n\n\" Among all selected models, Logistic Regression also had a better performance than the DRAGON score on positive predictive value, sensitivity, and specificity. Compared with the DRAGON score, the multiclass machine learning approach showed better performance on the prediction of the 3-month functional outcome of acute ischemic stroke patients requiring reperfusion therapy.\"\n\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7825282/\n\n#The DRAGON score\n\n\"The DRAGON score is a scoring system composed by six different variables evaluated from stroke patients at admission. It was created in 2012 to predict both favorable and miserable functional outcomes 3 months after the stroke , and it has been validated by several studies with good performance.\"\n\n\"On the other hand, to the best of the autors knowledge, there is no ML approach for the prediction of the whole spectrum functional outcomes in stroke patients. To assist with the generation of accurate treatment decisions, comprehensive outcome prediction for acute stroke patients is needed. Thus, this study aims to develop a multiclass machine learning prediction model on the 3-month outcome of acute ischemic stroke patients who were candidates for reperfusion therapy at the time of admission.\"\n\nhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7825282/\n\nMagnetic Resonance Imaging-DRAGON Score\n\n\"The DRAGON score, which includes clinical and computed tomographic scan parameters, showed a high specificity to predict 3-month outcome in patients with acute ischemic stroke treated by intravenous tissue plasminogen activator. We adapted the score for patients undergoing MRI as the first-line diagnostic tool.\"\n\nhttps://www.ahajournals.org/doi/10.1161/strokeaha.111.000127#:~:text=The%20DRAGON%20score%2C%20which%20includes,the%20first%2Dline%20diagnostic%20tool.\n\n#On Kaggle (Loss) : RSNA Intracranial Hemorrhage Detection (evaluated by weighted multi-label logarithmic loss)\n\nNotebook: Maybe *_any weight is x2 of others By Kambarakun\n\nhttps://www.kaggle.com/code/kambarakun/maybe-any-weight-is-x2-of-others/script\n\nDiscussion Topic: \"Tricks for Kaggle log loss\" By the amazing GM hengck23\n\n\"sampling/weighing is important in training\n\nwhen the metric is accuracy, we usually use : predict_label = predict_prob > threshold\nwe do not care about predict_prob, but rather thresholded prediction\nwe sometimes use loss weighing (e.g focal loss) or class balancing to speed up training or improve accuracy. but improving accuracy is \"not the same\" as reducing loss. the loss is modified to improve accuracy by artificially changing the train distribution and affecting the decision boundary.\n\n\n\"for this challenge, they care about the actual log loss itself. so it is important to \"calibrate your probability\" to the actual one you expect in the test during training. you may have to use random sampling (instead of balanced sampling) at the final fine tuning. likewise you may need to disable any sample weighing.\"\n\nprobability shaping\n\n\"this is is very difficult (and very risky in scoring). e.g. a test sample is predicted to be negative with probability 0.0001. you may round it to zero. you will gain a little if the truth is really negative, but you will lose big if it is truth is instead positive.\"\n\n\"if you can train with \"large margin\" and use some probability calibration trick such that you are almost 100% sure that there cannot be positive truth below a certain low values, you can zero out the values.\"\n\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/114627\n\n\"Relationship between accuracy and loss?\" By Ryan Epp\n\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/114087\n\n\"Code of weighted log loss is here\" By Rajnish Chauhan\n\nTake care of BATCH_SIZE, it is same as taken in ImageGenerator.\n\nUse \"custom_log_loss\" in model.compile( loss ='customers_log_loss')\n\ndef custom_log_loss(y_true, y_pred):\nweights_cons = tf.constant([2.0, 1.0, 1.0, 1.0, 1.0, 1.0])\nloss_val = tf.losses.sigmoid_cross_entropy(y_true, y_pred, weights = tf.broadcast_to(weights_cons, [BATCH_SIZE, 6]))\n\nloss_val = tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=y_pred)\nloss_val = tf.losses.sigmoid_cross_entropy(y_true, y_pred)\nprint(y_true.shape)\nprint(ck_val)\nreturn loss_val\n\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/111719\n\n#By the way, what are the weights? Don't bother to answer that, it's just a rhetorical question.\n\n\"Why were the weight values in the evaluation metric not given to us? \"\n\nhttps://www.kaggle.com/competitions/rsna-intracranial-hemorrhage-detection/discussion/109447\n\n#I hope that can help our community. And I don't have any stroke, since there is a lot to learn in this Competition.\n\nDemographic characteristics of included Stroke patients:\n\nMale, Mean Age (67.9 ± 12.4), Sugar (mg/dL: 141 ± 57), Pre-Stroke, Old Stroke, Atrial fibrillation (227 (38.5)), Diabetes mellitus, \nHypertension, Dyslipidemia, Coronary artery disease, Heart failure, Smoking (Source NIHSS: National Institutes of Health Stroke Scale)\n\n#Therefore dear kaggler, stand up from your Coding chair and avoid Sedentarism. The cause of many diseases. And Happy kaggling!\n\n", "1848292": "> By the way, what are the weights?\n\nThe evaluation page says that 'each class is roughly equally important for the final score'. So I'm guessing that both have equal weights. \nThen by simple lb probing (committing 0 and 1), the results of my calculation **support my guess**.", "1851831": "" } }