全科医生还是ChatGPT?大型语言模型在支持全科医生开具抗生素方面的能力
基于24个涵盖感染类型、性别、年龄组和合并症的小案例,比较四国(爱尔兰、英国、美国、挪威)全科医生与六种大型语言模型(ChatGPT、Gemini、Copilot、Mistral AI、Claude和Llama 3.1)在抗生素处方上的表现。全科医生诊断准确率96%-100%,抗生素处方准确率83%-92%,但剂量/疗程准确率仅50%-75%;大型语言模型诊断准确率92%-100%,抗生素处方准确率88%-100%,抗生素选择准确率59%-100%,但正确引用指南常失败(38%-96%),挪威指南引用率仅0%-13%。大型语言模型或可安全指导全科抗生素处方,但解读案例、应用国家指南及确定正确剂量和疗程仍以全科医生为最佳。
要点
- 全科医生诊断准确率96%-100%,抗生素处方准确率83%-92%,剂量/疗程准确率50%-75%。
- 大型语言模型诊断准确率92%-100%,抗生素处方准确率88%-100%,抗生素选择准确率59%-100%。
- 大型语言模型正确引用指南常失败(38%-96%),挪威指南引用率仅0%-13%,且存在数据泄露问题。
核心结论
- 大型语言模型或可安全指导全科抗生素处方,但解读案例、应用国家指南及确定正确剂量和疗程仍以全科医生为最佳。
研究设计与样本
- 基于24个提供感染类型、性别、年龄组和合并症信息的小案例,纳入爱尔兰、英国、美国和挪威四国,每国一名全科医生与六种大型语言模型(ChatGPT、Gemini、Copilot、Mistral AI、Claude和Llama 3.1)根据案例及所在国家生成治疗建议,并与该国国家处方指南比较,同时评估大型语言模型的幻觉、毒性和数据泄露等局限性。
主要发现
- 全科医生诊断准确率96%-100%,抗生素处方是/否准确率83%-92%,引用指南率100%,按指南处方率58%-92%,剂量/疗程准确率50%-75%,总体平均准确率74%。
- 大型语言模型诊断准确率92%-100%,抗生素处方准确率88%-100%,抗生素选择准确率59%-100%,但正确引用指南常失败(38%-96%),挪威指南引用率仅0%-13%。
- 数据泄露显示为问题,个人信息化在模型对案例的回复中被重复。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
INTRODUCTION: Large language models (LLMs) are becoming ubiquitous and widely implemented. LLMs could also be used for diagnosis and treatment. National antibiotic prescribing guidelines are customized and informed by local laboratory data on antimicrobial resistance. METHODS: Based on 24 vignettes with information on type of infection, gender, age group and comorbidities, GPs and LLMs were prompted to provide a treatment. Four countries (Ireland, UK, USA and Norway) were included and a GP from each country and six LLMs (ChatGPT, Gemini, Copilot, Mistral AI, Claude and Llama 3.1) were provided with the vignettes, including their location (country). Responses were compared with the country's national prescribing guidelines. In addition, limitations of LLMs such as hallucination, toxicity and data leakage were assessed. RESULTS: GPs' answers to the vignettes showed high accuracy in relation to diagnosis (96%-100%) and yes/no antibiotic prescribing (83%-92%). GPs referenced (100%) and prescribed (58%-92%) according to national guidelines, but dose/duration of treatment was less accurate (50%-75%). Overall, the GPs' accuracy had a mean of 74%. LLMs scored high in relation to diagnosis (92%-100%), antibiotic prescribing (88%-100%) and the choice of antibiotic (59%-100%) but correct referencing often failed (38%-96%), in particular for the Norwegian guidelines (0%-13%). Data leakage was shown to be an issue as personal information was repeated in the models' responses to the vignettes. CONCLUSIONS: LLMs may be safe to guide antibiotic prescribing in general practice. However, to interpret vignettes, apply national guidelines and prescribe the right dose and duration, GPs remain best placed.
引用
- Ngoc Nguyen O, Amin D, Bennett J, et al. GP or ChatGPT? Ability of large language models (LLMs) to support general practitioners when prescribing antibiotics. J Antimicrob Chemother. 2025. PMID: 40079276. doi:10.1093/jac/dkaf077
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。