多LLM共识法构建影像报告研究队列
该回顾性队列研究纳入6718份脑MRI报告,比较多个大语言模型共识流程与单个模型。专家参考集显示:两种模型组合自动判定正常与异常报告的精确率均达100%,优于最强单模型GPT-5。全队列中正常组中枢神经系统相关ICD-10诊断患病率仅1.6%。
要点
- 多模型一致才自动判定,意见分歧转人工复核。
- 正常报告残留CNS相关诊断率仅1.6%,阴性报告可信度高。
- GPT-5单模型有17例专家标签不匹配,不宜单独使用。
核心结论
- 多模型一致性共识可获得高精确率自动判定,并将模糊病例转人工复核。
- 该策略可减少人工审核负担,支持从非结构化文本构建高纯度研究队列。
研究设计与样本
- 回顾性队列研究,纳入6718份脑MRI报告,来自5856例患者。
- 专家判定参考集含680份报告,来自675例患者。
- 参考集中372份正常,90份仅白质高信号但可能模糊,218份异常。
- 评估专有模型组合Qwen3/Kimi-Linear/GPT-5与开源组合Qwen3/Kimi-Linear/DeepSeek-R1。
关键影像表现/征象
- 脑MRI报告示白质高信号仅表现为孤立非特异性WMH,对二分类可能属模糊类别。
诊断标准与关键数值
- 专家参考集内,两组模型自动判定正常与异常报告的精确率均为100%。
- GPT-5单模型正常精确率98.9%,异常精确率94.1%,专家标签不匹配17例。
- 专有模型组合正常组CNS相关ICD-10诊断率为1.6%,不确定组29.7%,异常组51.3%。
- 开源模型组合对应三组ICD-10诊断率分别为1.6%、31.1%和51.6%。
临床应用与阅片要点
- 所有模型一致为正常或异常才自动判定,其余报告设不确定并转人工复核。
- 正常组残留活动性CNS相关ICD-10诊断患病率仅1.6%,阴性报告可用于高纯度队列筛选。
- GPT-5单模型出现17例专家标签不匹配,日常不宜依赖单一模型判定。
局限性
- 研究局限于单一中心脑MRI报告,未涉及其他部位或检查类型。
- 以ICD-10诊断作为全队列验证指标,可能存在编码遗漏或滞后。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
BACKGROUND: Large-scale retrospective studies often require researchers to adjudicate outcomes or cohort eligibility from unstructured clinical records. Manual review can provide reliable labels but is difficult to scale. We developed and evaluated a consensus workflow using multiple large language models (LLMs) to automatically assign high-confidence outcome labels while deferring ambiguous cases for manual review.
METHODS: We included 6,718 brain MRI reports from 5,856 patients. The expert-adjudicated reference set included 680 reports from 675 patients: 372 normal (age-appropriate without significant intracranial pathology), 90 white matter hyperintensity-only (WMH-only; isolated nonspecific WMHs potentially ambiguous for binary classification), and 218 abnormal (other intracranial pathology) reports. We evaluated proprietary-inclusive (Qwen3/Kimi-Linear/GPT-5) and open-source (Qwen3/Kimi-Linear/DeepSeek-R1) ensembles against individual LLMs. Ensembles assigned a normal or abnormal label only when all models agreed; all other reports were classified as uncertain and deferred for manual review. Individual LLMs assigned normal or abnormal predictions and classified WMH-only predictions as uncertain. In the full cohort, we compared the prevalence of active central nervous system (CNS)-related International Classification of Diseases, Tenth Revision (ICD-10) diagnoses across the normal, uncertain, and abnormal groups generated by each ensemble.
RESULTS: In the expert-adjudicated reference set, both ensembles achieved 100% precision for automatically assigned normal and abnormal reports. GPT-5 was the strongest individual LLM but had lower precision than either ensemble, with normal and abnormal precision of 98.9% and 94.1%, respectively, and 17 expert-label mismatches versus none for the consensus ensembles. In the full cohort, 1.6% of reports classified as normal were associated with CNS-related ICD-10 diagnoses, compared with 29.7% of uncertain and 51.3% of abnormal reports for the proprietary-inclusive ensemble; corresponding rates were 1.6%, 31.1%, and 51.6% for the open-source ensemble.
CONCLUSIONS: Requiring unanimous agreement across multiple LLMs enabled high-precision automatic adjudication while directing ambiguous cases to manual review. This strategy may reduce the burden of manual adjudication and support scalable construction of high-purity research cohorts from unstructured clinical text.
引用
- Rui, Ma, Stepanov, et al. Large language model consensus for reliable research cohort construction from radiology reports: a retrospective cohort study. medRxiv. 2026. PMID: 42926896. doi:10.64898/2026.10.06.26364312
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。