文献速递神经技术与安全CTMR

大语言模型预测神经影像检查的能力评估

Use of Large Language Models to Predict Neuroimaging.
PubMed 文献 · J Am Coll Radiol · 2023-07-08

本研究评估ChatGPT与Glass AI在给定临床场景下推荐最佳神经影像检查的能力,并以ACR Appropriateness Criteria为参考标准,与一名经验丰富的神经放射科医生比较。结果显示两种LLM得分分别为1.75和1.83,无统计学显著差异,而神经放射科医生得分2.20,显著优于两者。ChatGPT输出一致性较差,但LLM整体在推荐神经影像检查方面表现良好。

要点

  • LLM可辅助推荐神经影像检查,但准确性不及经验丰富的神经放射科医生。
  • ChatGPT与Glass AI表现相当,提示医学文本训练可能提升ChatGPT性能。
  • ChatGPT输出一致性较差,临床应用中需注意结果稳定性。

局限性

  • 仅纳入一名神经放射科医生作为对照,医生间差异性未评估。
  • 参考标准为ACR Appropriateness Criteria,未涉及其他指南。
  • 样本为147个临床场景,未报告场景选择方法及代表性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

PURPOSE: Large language models (LLMs) have demonstrated a level of competency within the medical field. The aim of this study was to explore the ability of LLMs to predict the best neuroradiologic imaging modality given specific clinical presentations. In addition, the authors seek to determine if LLMs can outperform an experienced neuroradiologist in this regard.

METHODS: ChatGPT and Glass AI, a health care-based LLM by Glass Health, were used. ChatGPT was prompted to rank the three best neuroimaging modalities while taking the best responses from Glass AI and the neuroradiologist. The responses were compared with the ACR Appropriateness Criteria for 147 conditions. Clinical scenarios were passed into each LLM twice to account for stochasticity. Each output was scored out of 3 on the basis of the criteria. Partial scores were given for nonspecific answers.

RESULTS: ChatGPT and Glass AI scored 1.75 and 1.83, respectively, with no statistically significant difference. The neuroradiologist scored 2.20, significantly outperforming both LLMs. ChatGPT was also found to be the more inconsistent of the two LLMs, with the score difference between both outputs being statistically significant. Additionally, scores between different ranks output by ChatGPT were statistically significant.

CONCLUSIONS: LLMs perform well in selecting appropriate neuroradiologic imaging procedures when prompted with specific clinical scenarios. ChatGPT performed the same as Glass AI, suggesting that with medical text training, ChatGPT could significantly improve its function in this application. LLMs did not outperform an experienced neuroradiologist, indicating the need for continued improvement in the medical context.

来源:J Am Coll Radiol,查看原文。版权归原作者所有。

引用

  1. Nazario-Johnson L, Zaki HA, Tung GA, et al. Use of Large Language Models to Predict Neuroimaging. J Am Coll Radiol. 2023. PMID: 37423349. doi:10.1016/j.jacr.2023.06.008

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。