本地部署的上下文感知聊天机器人优于通用大语言模型,可提供符合指南的儿科影像推荐
本研究为模拟研究,基于50个源自ACR指南变体的儿科临床场景,比较本地部署的检索增强生成聊天机器人ped-Llama与三种通用大语言模型及三位放射科医生在影像推荐上的表现。ped-Llama的「通常合适」推荐率为80%,优于通用模型,与专科儿科放射科医生相当。
要点
- ped-Llama在50个儿科场景中「通常合适」推荐率达80%,与专科儿科放射科医生(76%)相当。
- 通用大语言模型「通常合适」推荐率为46%–54%,均低于ped-Llama。
- ped-Llama三次独立运行一致性为72%,通用大语言模型为44%–50%。
核心结论
- 本地运行、基于开源大语言模型的检索增强生成聊天机器人可提供符合指南的儿科影像推荐,达到专家级准确率。
研究设计与样本
- 模拟研究,使用50个源自ACR指南变体的儿科临床场景,比较ped-Llama(基于本地部署Llama-3.1-8B模型的检索增强生成方法)、三种通用大语言模型及三位放射科医生。
诊断标准与关键数值
- 推荐按ACR指南分类为「通常合适」「可能合适」「通常不合适」;ped-Llama「通常合适」80%(40/50),Llama-3.1-8B 46%,GPT-4o 54%,Claude Opus 46%,专科儿科放射科医生76%;同时接受「通常合适」与「可能合适」时ped-Llama准确率90%。
临床应用与阅片要点
- 作者认为此类系统为放射科人工智能辅助决策支持提供了实用且可扩展的方法。
局限性
- 本研究为模拟研究,使用50个源自ACR指南变体的儿科临床场景。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
BACKGROUND: Accurate modality selection in pediatric imaging is critical, yet adherence to the American College of Radiology (ACR) Appropriateness Criteria remains limited. Large language models (LLMs) offer potential as decision support tools but often lack domain-specific accuracy.
OBJECTIVE: To evaluate the performance of a locally run, context-aware chatbot (ped-Llama) based on an open-source LLM for providing personalized pediatric imaging recommendations grounded in ACR guidelines.
MATERIALS AND METHODS: A simulation-based study was conducted using 50 pediatric clinical scenarios derived from ACR guideline variants. The ped-Llama chatbot, built using a retrieval-augmented generation (RAG) approach with a locally deployed Llama-3.1-8B model, was compared against three generic LLMs (Llama-3.1-8B without RAG, Generative pre-trained transformer-4o [GPT-4o], Claude Opus) and three radiologists (junior resident, senior resident, specialist pediatric radiologist). Each provided imaging recommendations, which were classified as "Usually appropriate," "May be appropriate," or "Usually not appropriate" per ACR guidelines. Modal outputs were analyzed, and consistency across three independent LLM runs was assessed.
RESULTS: ped-Llama achieved 80% (40/50) "Usually appropriate" recommendations, outperforming Llama-3.1-8B (46%), GPT-4o (54%), and Claude Opus (46%), and matching the specialist radiologist (76%). When both "Usually appropriate" and "May be appropriate" were accepted, ped-Llama achieved 90% accuracy. Consistency across runs was 72% for ped-Llama versus 44-50% for generic LLMs.
CONCLUSION: This study shows that a locally run, RAG-enabled chatbot based on an open-source LLM can provide guideline-concordant pediatric imaging recommendations with expert-level accuracy. Such systems offer a practical and scalable approach to artificial intelligence-assisted decision support in radiology.
引用
- Gupta A, Rangarajan K, Krishna Kumar RG, et al. Locally deployed context-aware chatbot outperforms generic large language models for guideline-concordant pediatric imaging recommendations. Pediatr Radiol. 2025. PMID: 41231298. doi:10.1007/s00247-025-06453-6
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。