Radiologist here, so obviously interested in the clinical application. \nReached LB score ~ 0.090, and attempted to find out what the area under ROC curve is - to see what the clinical performance of such a model would be. \nSo, I quarantined ~ 10% of the training dataset (for ground truth) and trained the model on the remaining 90% of the dataset (includes both train and val). \nTo my surprise, the AU ROC on the \"quarantined data\" is ~ 0.51 which means, the trained model has the clinical utility of a coin-toss. \nSure, the model is training to maximize the weighted log loss, but would still expect it to be learning something.\nAm I missing something here? Would love to hear your experience.
\n\nQuick code to get AUROC: \nfrom sklearn.metrics import roc-auc-score\nytrue = np.array(df['any']) \nyscores = np.array(df['predicted']) \nauroc = roc-auc-score(ytrue, yscores)\nNote that \"roc-auc-score\" above is actually \"roc underscore auc underscore score\"
", "messages": [ { "id": 652913, "postDate": "2019-10-19T16:02:21.500Z", "content": "In my latest run I get AUC ROC in range 0.9868-0.9882, LB 0.063, single model. Please check your code!
Radiologist here, so obviously interested in the clinical application. \nReached LB score ~ 0.090, and attempted to find out what the area under ROC curve is - to see what the clinical performance of such a model would be. \nSo, I quarantined ~ 10% of the training dataset (for ground truth) and trained the model on the remaining 90% of the dataset (includes both train and val). \nTo my surprise, the AU ROC on the \"quarantined data\" is ~ 0.51 which means, the trained model has the clinical utility of a coin-toss. \nSure, the model is training to maximize the weighted log loss, but would still expect it to be learning something.\nAm I missing something here? Would love to hear your experience.
\n\nQuick code to get AUROC: \nfrom sklearn.metrics import roc-auc-score\nytrue = np.array(df['any']) \nyscores = np.array(df['predicted']) \nauroc = roc-auc-score(ytrue, yscores)\nNote that \"roc-auc-score\" above is actually \"roc underscore auc underscore score\"
", "rawMarkdown": "Radiologist here, so obviously interested in the clinical application. \nReached LB score ~ 0.090, and attempted to find out what the area under ROC curve is - to see what the clinical performance of such a model would be. \nSo, I quarantined ~ 10% of the training dataset (for ground truth) and trained the model on the remaining 90% of the dataset (includes both train and val). \nTo my surprise, the AU ROC on the \"quarantined data\" is ~ 0.51 which means, the trained model has the clinical utility of a coin-toss. \nSure, the model is training to maximize the weighted log loss, but would still expect it to be learning something.\nAm I missing something here? Would love to hear your experience. \n\nQuick code to get AUROC: \nfrom sklearn.metrics import roc-auc-score\nytrue = np.array(df['any']) \nyscores = np.array(df['predicted']) \nauroc = roc-auc-score(ytrue, yscores)\nNote that \"roc-auc-score\" above is actually \"roc underscore auc underscore score\"", "votes": 3 }, { "id": 653282, "postDate": "2019-10-20T06:59:11.593Z", "content": "Are you sure your ytrue and yscore arrays are in the same order?
", "rawMarkdown": "Are you sure your ytrue and yscore arrays are in the same order?", "votes": 1 }, { "id": 654307, "postDate": "2019-10-21T17:27:51.583Z", "content": "I liked the expression clinical utility.
", "rawMarkdown": "I liked the expression clinical utility." } ], "comments": [ { "id": 652913, "author_name": "nosound", "author_url": "", "post_date": "2019-10-19T16:02:21.500000", "content": "In my latest run I get AUC ROC in range 0.9868-0.9882, LB 0.063, single model. Please check your code!
Are you sure your ytrue and yscore arrays are in the same order?
", "votes": 1, "replies": [] }, { "id": 654307, "author_name": "Marília Prata", "author_url": "", "post_date": "2019-10-21T17:27:51.583000", "content": "I liked the expression clinical utility.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "652913": "In my latest run I get AUC ROC in range `0.9868-0.9882`, LB 0.063, single model. Please check your code!", "652875": "Radiologist here, so obviously interested in the clinical application. \nReached LB score ~ 0.090, and attempted to find out what the area under ROC curve is - to see what the clinical performance of such a model would be. \nSo, I quarantined ~ 10% of the training dataset (for ground truth) and trained the model on the remaining 90% of the dataset (includes both train and val). \nTo my surprise, the AU ROC on the \"quarantined data\" is ~ 0.51 which means, the trained model has the clinical utility of a coin-toss. \nSure, the model is training to maximize the weighted log loss, but would still expect it to be learning something.\nAm I missing something here? Would love to hear your experience. \n\nQuick code to get AUROC: \nfrom sklearn.metrics import roc-auc-score\nytrue = np.array(df['any']) \nyscores = np.array(df['predicted']) \nauroc = roc-auc-score(ytrue, yscores)\nNote that \"roc-auc-score\" above is actually \"roc underscore auc underscore score\"", "653282": "Are you sure your ytrue and yscore arrays are in the same order?", "654307": "I liked the expression clinical utility." } }