Now that the competition has ended, is there any chance the hosts at Mayo could review the private leaderboard results with a physician who treats stroke patients and post their impressions? I'm guessing that the ROCAUC on the private leaderboard is probably not more than 0.6 (but is worth calculating). Would a physician feel confident enough in a model like that for acute ischemic stroke and act on the result? I'd be curious to hear the risk-benefit tradeoff for tailoring a treatment for CE vs. LAA outside of standard of care if the model performs with an AUC score around 0.6.
\nIn a previous life I worked for a medical diagnostics company and there physicians were still apprehensive with assays with ROCAUC scores of 0.97, but I would say most were comfortable with 0.99 or sometimes 0.995. That was a different context with high risks, so the threshold may be different in acute ischemic stroke management.
\nBecause if the risk of mis-management is too high, then I wonder if perhaps the MSB histology stain alone is not enough. The basic hypothesis of the competition was that the histology alone - no demographics, patient history, or other biomarkers - would discriminate the two stroke etiologies. But I have not seen a clear biological reason why that would be true. I understand that histology is cost effective and available at most hospitals, but perhaps we need other technology to dive deeper into the protein content, such as mass spectrometry.
\nWould the competition hosts be open to amending the IRB, collecting data with more assays (e.g., mass spectrometry), and coming back to kaggle in a few years for another chance? Thanks for your consideration.
", "messages": [ { "id": 1974621, "postDate": "2022-10-06T11:09:48.177Z", "content": "Now that the competition has ended, is there any chance the hosts at Mayo could review the private leaderboard results with a physician who treats stroke patients and post their impressions? I'm guessing that the ROCAUC on the private leaderboard is probably not more than 0.6 (but is worth calculating). Would a physician feel confident enough in a model like that for acute ischemic stroke and act on the result? I'd be curious to hear the risk-benefit tradeoff for tailoring a treatment for CE vs. LAA outside of standard of care if the model performs with an AUC score around 0.6.
\nIn a previous life I worked for a medical diagnostics company and there physicians were still apprehensive with assays with ROCAUC scores of 0.97, but I would say most were comfortable with 0.99 or sometimes 0.995. That was a different context with high risks, so the threshold may be different in acute ischemic stroke management.
\nBecause if the risk of mis-management is too high, then I wonder if perhaps the MSB histology stain alone is not enough. The basic hypothesis of the competition was that the histology alone - no demographics, patient history, or other biomarkers - would discriminate the two stroke etiologies. But I have not seen a clear biological reason why that would be true. I understand that histology is cost effective and available at most hospitals, but perhaps we need other technology to dive deeper into the protein content, such as mass spectrometry.
\nWould the competition hosts be open to amending the IRB, collecting data with more assays (e.g., mass spectrometry), and coming back to kaggle in a few years for another chance? Thanks for your consideration.
", "rawMarkdown": "Now that the competition has ended, is there any chance the hosts at Mayo could review the private leaderboard results with a physician who treats stroke patients and post their impressions? I'm guessing that the ROCAUC on the private leaderboard is probably not more than 0.6 (but is worth calculating). Would a physician feel confident enough in a model like that for acute ischemic stroke and act on the result? I'd be curious to hear the risk-benefit tradeoff for tailoring a treatment for CE vs. LAA outside of standard of care if the model performs with an AUC score around 0.6.\n\nIn a previous life I worked for a medical diagnostics company and there physicians were still apprehensive with assays with ROCAUC scores of 0.97, but I would say most were comfortable with 0.99 or sometimes 0.995. That was a different context with high risks, so the threshold may be different in acute ischemic stroke management.\n\nBecause if the risk of mis-management is too high, then I wonder if perhaps the MSB histology stain alone is not enough. The basic hypothesis of the competition was that the histology alone - no demographics, patient history, or other biomarkers - would discriminate the two stroke etiologies. But I have not seen a clear biological reason why that would be true. I understand that histology is cost effective and available at most hospitals, but perhaps we need other technology to dive deeper into the protein content, such as mass spectrometry.\n\nWould the competition hosts be open to amending the IRB, collecting data with more assays (e.g., mass spectrometry), and coming back to kaggle in a few years for another chance? Thanks for your consideration.", "votes": 10 }, { "id": 1975729, "postDate": "2022-10-07T01:59:52.940Z", "content": "I believe Mayo Clinic would take these results either as techniques are unreliable up to this point, or that data needs to be expanded and this exercise tried again. It would be good to know from some domain expert on the difficulty in obtaining and ready availability of such data. As I understood from some discussions, these scans are a relatively new things. So some advances on both ends (data and modelling) could make this viable in the future.
\nIt would surely be interesting to know what physicians make of the scans. I read somewhere it is not simple even for them!
", "rawMarkdown": "I believe Mayo Clinic would take these results either as techniques are unreliable up to this point, or that data needs to be expanded and this exercise tried again. It would be good to know from some domain expert on the difficulty in obtaining and ready availability of such data. As I understood from some discussions, these scans are a relatively new things. So some advances on both ends (data and modelling) could make this viable in the future. \n\nIt would surely be interesting to know what physicians make of the scans. I read somewhere it is not simple even for them!", "votes": 1 } ], "comments": [ { "id": 1975729, "author_name": "tdiceman", "author_url": "", "post_date": "2022-10-07T01:59:52.940000", "content": "I believe Mayo Clinic would take these results either as techniques are unreliable up to this point, or that data needs to be expanded and this exercise tried again. It would be good to know from some domain expert on the difficulty in obtaining and ready availability of such data. As I understood from some discussions, these scans are a relatively new things. So some advances on both ends (data and modelling) could make this viable in the future.
\nIt would surely be interesting to know what physicians make of the scans. I read somewhere it is not simple even for them!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1974621": "Now that the competition has ended, is there any chance the hosts at Mayo could review the private leaderboard results with a physician who treats stroke patients and post their impressions? I'm guessing that the ROCAUC on the private leaderboard is probably not more than 0.6 (but is worth calculating). Would a physician feel confident enough in a model like that for acute ischemic stroke and act on the result? I'd be curious to hear the risk-benefit tradeoff for tailoring a treatment for CE vs. LAA outside of standard of care if the model performs with an AUC score around 0.6.\n\nIn a previous life I worked for a medical diagnostics company and there physicians were still apprehensive with assays with ROCAUC scores of 0.97, but I would say most were comfortable with 0.99 or sometimes 0.995. That was a different context with high risks, so the threshold may be different in acute ischemic stroke management.\n\nBecause if the risk of mis-management is too high, then I wonder if perhaps the MSB histology stain alone is not enough. The basic hypothesis of the competition was that the histology alone - no demographics, patient history, or other biomarkers - would discriminate the two stroke etiologies. But I have not seen a clear biological reason why that would be true. I understand that histology is cost effective and available at most hospitals, but perhaps we need other technology to dive deeper into the protein content, such as mass spectrometry.\n\nWould the competition hosts be open to amending the IRB, collecting data with more assays (e.g., mass spectrometry), and coming back to kaggle in a few years for another chance? Thanks for your consideration.", "1975729": "I believe Mayo Clinic would take these results either as techniques are unreliable up to this point, or that data needs to be expanded and this exercise tried again. It would be good to know from some domain expert on the difficulty in obtaining and ready availability of such data. As I understood from some discussions, these scans are a relatively new things. So some advances on both ends (data and modelling) could make this viable in the future. \n\nIt would surely be interesting to know what physicians make of the scans. I read somewhere it is not simple even for them!" } }