OsteoCHAT:基于指南的骨质疏松聊天机器人真实世界评估与基准测试
在德国七中心真实世界评估中,基于德国骨质疏松指南的检索增强聊天机器人OsteoCHAT记录1075次问答互动,389条单独评分中97.4%获正面反馈,217份问卷中84.3%认为答案易理解。专家盲评显示OsteoCHAT与三种通用大语言模型回答质量相当(中位总分14/15),但所有系统在安全沟通方面存在不足。
要点
- OsteoCHAT在真实世界多中心评估中用户接受度和可用性高,97.4%的单独评分回答获正面反馈。
- 专家盲评中OsteoCHAT与Gemini、ChatGPT中位总分均为14/15,Meta AI为13/15,回答质量相当。
- 所有评估系统在药物治疗指征及不良事件和罕见安全关键事件沟通方面存在弱点。
核心结论
- OsteoCHAT在真实世界多中心评估中表现出高用户接受度和可用性,被视为现有教育材料的有价值补充。
- OsteoCHAT的专家评分回答质量与通用大语言模型相当,但所有受检系统在安全沟通方面均存在局限性。
研究设计与样本
- 开发基于当前德国骨质疏松指南的检索增强对话系统OsteoCHAT,并在德国七个中心进行多中心评估。
- 参与者与OsteoCHAT互动并提供实时反馈,完成评估可用性、可理解性、有用性、信任、感知质量和相比传统互联网搜索偏好的标准化问卷。
- 使用源自BfO患者指南的十个骨质疏松常见问题作为金标准,将OsteoCHAT与三种通用大语言模型进行基准测试,由五名临床专家独立从五个领域评分(每项0-3分)。
主要发现
- 共记录1075次问答互动;389条单独评分的回答中379条(97.4%)获得正面反馈。
- 217份完整问卷中,84.3%认为答案易于理解,79.8%认为OsteoCHAT易于使用,77.2%感知到节省时间,78.7%认为其是现有教育材料的有用补充,信任度相对较低(64.7%同意)。
- 盲法专家基准测试显示所有大语言模型回答为可接受至高质量(Gemini、ChatGPT和OsteoCHAT中位总分14/15,Meta AI为13/15),专家识别的主要弱点涉及药物治疗指征及不良和罕见安全关键事件的沟通。
临床怎么用
- 基于指南的聊天机器人可作为骨质疏松患者教育的可扩展辅助工具,补充常规护理中信息提供的不足。
- 使用中需注意其在药物治疗指征和安全关键事件沟通方面的局限,对相关回答应谨慎对待。
局限性
- 评估在德国七个中心进行,结果外推至其他医疗环境需谨慎。
- 专家盲评发现所有系统在安全沟通方面存在不足。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
UNLABELLED: Patients with osteoporosis have information needs that routine care inconsistently addresses. In this multicenter real-world evaluation, the LLM-based, guideline-grounded chatbot OsteoCHAT showed high participant acceptability and achieved expert-rated answer quality comparable to general-purpose LLMs. Despite challenges in complex treatment and safety-related topics, guideline-grounded chatbots may represent useful educational adjuncts. PURPOSE: Patients with osteoporosis have information needs that routine care inconsistently addresses. Guideline-grounded chatbots may offer scalable educational support METHODS: We developed OsteoCHAT, a retrieval-augmented conversational system based on the current German osteoporosis guideline, and conducted a multicenter evaluation at seven centers in Germany. Participants interacted with OsteoCHAT, provided real-time feedback on individual responses, and completed a standardized questionnaire assessing usability, comprehensibility, usefulness, trust, perceived quality, and preference over conventional internet search. In parallel, OsteoCHAT was benchmarked against three general-purpose large language models (LLMs) using ten osteoporosis frequently asked questions derived from the BfO patient guideline as the gold standard. Five clinical experts independently rated these responses across five domains, each scored 0-3. RESULTS: Overall, 1,075 question-answer interactions were recorded. Of 389 individually rated responses, 379 (97.4%) received positive feedback. Among 217 complete questionnaire responses, 84.3% rated answers as easily understandable, 79.8% found OsteoCHAT easy to use, 77.2% perceived time savings, and 78.7% considered it a useful addition to existing educational materials. Trust was comparatively lower (64.7% agreement). Blinded expert benchmarking showed acceptable-to-high-quality responses across all LLMs (median total scores 14/15 for Gemini, ChatGPT, and OsteoCHAT and 13/15 for Meta AI), with expert-identified weaknesses mainly concerning pharmacological treatment indication and communication of adverse and rare safety-critical events. CONCLUSION: OsteoCHAT demonstrated high user acceptability and usability in a real-world multicenter evaluation, was rated as a valuable addition to existing educational materials, and achieved expert-rated response quality comparable to general-purpose LLMs, although limitations in safety communication were identified across all examined systems.
引用
- Schulz N, Gehlen M, Mahn K, et al. OsteoCHAT: real-world patient evaluation and benchmarking of a guideline-grounded osteoporosis chatbot. Osteoporos Int. 2026. PMID: 42828563. doi:10.1007/s00198-026-08247-4
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。