大语言模型生成鼻科常见病内容与参考文献的准确性、完整性与幻觉评估
这项前瞻性比较两部分研究由国际鼻科学会成员评估ChatGPT-3.5生成的四种鼻科常见病(鼻出血、变应性鼻炎、牙源性鼻窦炎、慢性鼻窦炎)100词摘要的准确性与完整性,并评估当代大语言模型版本为相同提示生成参考文献的准确性。耳鼻咽喉科医生使用标准化量表对每条陈述的准确性、完整性和参考文献总体质量进行评分。
要点
- 评估ChatGPT-3.5生成的四种鼻科常见病摘要的准确性与完整性。
- 评估ChatGPT-5 thinking、ChatGPT-5 non-thinking、Claude Sonnet 4.5 non-thinking和Gemini 2.5生成参考文献的准确性。
- 由国际鼻科学会成员及耳鼻咽喉科医生使用标准化量表进行评分。
核心结论
- 研究评估了大语言模型在鼻科常见病内容生成与参考文献方面的准确性、完整性与幻觉问题。
研究设计与样本
- 前瞻性比较两部分研究,由国际鼻科学会成员评估ChatGPT-3.5生成的100词摘要。
- 评估对象为四种鼻科常见病:鼻出血、变应性鼻炎、牙源性鼻窦炎和慢性鼻窦炎。
- 摘要内容包括背景、检查和治疗,由耳鼻咽喉科医生用标准化量表评分。
- 第二部分评估ChatGPT-5 thinking、ChatGPT-5 non-thinking、Claude Sonnet 4.5 non-thinking和Gemini 2.5为相同提示生成参考文献的准确性。
关键数值与推荐
- 使用标准化量表评估每条陈述的准确性、完整性和参考文献总体质量。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Large language models (LLMs) are increasingly adopted for medical documentation, clinical decision-making, and scientific writing. Although these applications demonstrated impressive capabilities, concerns persist regarding their accuracy, fabricated references, and ethical implications (1-3). The Massive Multitask Language Understanding (MMLU) test was introduced as a comprehensive benchmark to evaluate how well LLMs perform across a diverse range of tasks (4). This prospective comparative two part study evaluated the accuracy and completeness of ChatGPT (CGPT)-3.5-generated 100 word summaries of four common rhinologic disorders (epistaxis, allergic rhinitis, odontogenic sinusitis, and chronic rhinosinusitis) including background, investigations, and magement as assessed by international rhinology society members. Furthermore, we evaluated the accuracy of references generated by contemporary LLM versions for the same prompts (CGPT-5 thinking (5), non-thinking (6), Claude Sonnet 4.5 non-thinking (6) and Gemini 2.5 (7)). Otolaryngologists evaluated each statement using standardized scales for accuracy, completeness, and overall reference quality.
引用
- Margulis I, Slim MAM, Sommer EC, et al. Accuracy, completeness and hallucinations in large language model generated statements and references for common rhinologic disorders. Rhinology. 2026. PMID: 42873484. doi:10.4193/Rhin26.460
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。