文献速递儿科影像技术与安全

多模态大语言模型在儿科放射学中检测需立即报告病例的准确性:基于公开临床案例的可行性研究

Accuracy of Large Language Models in Detecting Cases Requiring Immediate Reporting in Pediatric Radiology: A Feasibility Study Using Publicly Available Clinical Vignettes.
PubMed 文献 · Korean J Radiol · 2025-01-01

该可行性研究使用71个公开的儿科临床案例(含图像),评估七种具备视觉能力的大语言模型(温度0和1)与四位人类读者检测需立即报告病例的准确性。Gemini-Flash在温度1时准确率最高(83.1%),与最准确的人类读者(儿科专家84.5%)相当,但模型对危急值报告的敏感性低(3/11,27.3%),作者认为模型可能用于工作列表优先级排序,但敏感性不可靠。

要点

  • Gemini-Flash t1检测需立即报告病例的准确率为83.1%,与儿科专家(84.5%)相当。
  • 该模型对危急值报告(CVR)病例的敏感性仅27.3%(3/11),敏感性不可靠。
  • 模型对KTAS 1-2级病例的准确率最高达81.7%,对人类读者为73.2%-84.5%。

核心结论

  • 多模态大语言模型在识别需立即放射学报告的病例时,总体准确率可能达到与人类读者相当或更高的水平,支持其用于儿科放射学工作列表优先级排序的潜在用途,但模型检测此类病例的敏感性不可靠。

研究设计与样本

  • 可行性研究,使用71个公开的、经过改写的儿科临床案例(含图像),来源包括《新英格兰医学杂志》《柳叶刀》《儿科学与青少年医学档案》和《放射学》;由七种具备视觉能力的大语言模型(温度水平0和1)和四位人类读者(儿科放射学专家、放射科受训者、儿科专家、儿科受训者)进行评估;需立即报告的病例定义为符合韩国分诊与 acuity 量表(KTAS)1-2级(n=24)或符合危急值报告(CVR)标准(n=11),共33例。

诊断标准与关键数值

  • 需立即报告的病例判定标准为对应KTAS 1-2级或符合CVR标准;LLM检测需立即报告病例的准确率为60.6%-83.1%,KTAS 1-2级为57.7%-81.7%,CVR病例为53.5%-87.3%;Gemini-Flash t1准确率最高:需立即报告83.1%(95%CI 74.6%-91.5%),KTAS 1-2级81.7%(95%CI 71.8%-90.1%),CVR病例87.3%(95%CI 78.9%-94.4%),但其CVR敏感性为3/11(27.3%);人类读者准确率:需立即报告62.0%-84.5%,KTAS 1-2级73.2%-84.5%,CVR病例39.4%-94.4%。

临床应用与阅片要点

  • 作者认为多模态大语言模型可能用于儿科放射学工作列表优先级排序,但模型检测需立即报告病例的敏感性不可靠。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVE: To evaluate the accuracy of multimodal large language models (LLMs) in detecting cases requiring immediate radiology reporting in pediatric radiology.

MATERIALS AND METHODS: Seventy-one publicly available, paraphrased pediatric clinical vignettes with images-sourced from the New England Journal of Medicine, The Lancet, Archives of Pediatrics & Adolescent Medicine, and Radiology-were assessed by seven vision-capable LLMs (temperature levels 0 and 1; t0 and t1) and four human readers (an expert pediatric radiologist, a trainee radiologist, an expert pediatrician, and a trainee pediatrician). Cases were classified as requiring immediate reporting (n = 33) if they corresponded to Korean Triage and Acuity Scale (KTAS) levels 1-2 (n = 24) or met the criteria for a critical value report (CVR) (n = 11). The most accurate LLM was compared with each human reader, with significance set at P < 0.013.

RESULTS: LLMs demonstrated 60.6%-83.1% accuracy in detecting cases requiring immediate radiology reporting: 57.7%-81.7% and 53.5%-87.3% for KTAS levels 1-2 and CVR cases, respectively. Gemini-Flash with t1 showed the highest accuracy among the LLMs: 83.1% (95% confidence interval [CI]: 74.6%-91.5%), 81.7% (95% CI: 71.8%-90.1%), and 87.3% (95% CI: 78.9%-94.4%) for identifying cases requiring immediate reporting, KTAS level 1-2 cases, and CVR cases, respectively, despite its low sensitivity for CVR detection (3/11, 27.3%). Human readers demonstrated 62.0%-84.5% accuracy for immediate radiology reporting, 73.2%-84.5% for KTAS levels 1-2, and 39.4%-94.4% for CVR cases. The accuracy of Gemini-Flash t1 in identifying cases requiring immediate radiology reporting was comparable to that of the most accurate human reader (vs. expert pediatrician: 84.5% [95% CI: 76.1%-93.0%]; P < 0.99).

CONCLUSION: Multimodal LLMs may achieve overall accuracy comparable to or exceeding that of human readers in identifying cases requiring immediate radiology reporting, supporting their potential use for pediatric radiology worklist prioritization. However, the models' sensitivity in detecting such cases was not reliable.

来源:Korean J Radiol,查看原文。版权归原作者所有。

引用

  1. Park JS, Hwang J, Kim PH, et al. Accuracy of Large Language Models in Detecting Cases Requiring Immediate Reporting in Pediatric Radiology: A Feasibility Study Using Publicly Available Clinical Vignettes. Korean J Radiol. 2025. PMID: 40873376. doi:10.3348/kjr.2025.0240

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。