文献速递小儿耳鼻咽喉

儿童耳鼻咽喉科信息质量比较分析:临床医生、住院医师与大语言模型

Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models.
PubMed 文献 · Otolaryngol Head Neck Surg · 2025-03-19

本研究为盲法比较研究,向ChatGPT-3.5、-4.0、-4o、Claude-3、Gemini、Perplexity、Copilot、一名二年级住院医师及一名儿童耳鼻咽喉科专家提出54项难度递增的问题(18项理论问题、18项临床情景、18项患者问题),由意大利耳鼻咽喉科学会青年耳鼻咽喉科医师组与国际耳鼻咽喉科学会联合会专家小组使用QAMAI工具进行盲法评估。结果显示,大语言模型在理论和标准化临床情景中表现与专家相当,Bing Copilot获得最高QAMAI评分,但AI在引用可靠来源方面缺乏透明度,在回答以患者为中心的问题上效果较差。

要点

  • 大语言模型在理论学习和标准化临床情景中表现与专家相当,Bing Copilot获最高QAMAI评分。
  • AI回答在引用可靠来源方面缺乏透明度,在患者中心问题上的表现较差。
  • 评审者间一致性差,难以区分人工与AI生成的回答;鼻科主题得分最高,喉科和患者中心问题一致性与表现较低。

核心结论

  • 大语言模型在儿童耳鼻咽喉科中显示出作为支持性资源的潜力,尤其在理论学习和标准化病例方面。
  • 仍存在显著局限性,包括来源透明度和患者互动中的情境沟通问题,人类监督对降低风险至关重要。
  • 未来应致力于改进AI能力,以实现循证且富有同理心的沟通,从而支持临床医生和家庭。

研究设计与样本

  • 比较盲法研究,在受控研究环境中使用大语言模型。
  • 共54项难度递增的问题:18项理论问题、18项临床情景和18项患者问题。
  • 问题分别提给ChatGPT-3.5、-4.0、-4o、Claude-3、Gemini、Perplexity、Copilot、一名二年级住院医师和一名儿童耳鼻咽喉科专家。
  • 由意大利耳鼻咽喉科学会青年耳鼻咽喉科医师组与国际耳鼻咽喉科学会联合会的专家成员组成小组,使用QAMAI工具进行盲法评估。

主要发现

  • 大语言模型在理论和标准化临床情景中表现与专家相当,Bing Copilot获得最高QAMAI评分。
  • AI回答在引用可靠来源方面缺乏透明度,且在回答以患者为中心的问题上效果较差。
  • 评审者间一致性差,凸显了区分人工生成与AI生成回答的挑战。
  • 鼻科主题得分最高,而喉科和患者中心问题的一致性与表现较低。

临床怎么用

  • 原文结论指出,大语言模型在儿童耳鼻咽喉科中显示出作为支持性资源的潜力,尤其在理论学习和标准化病例方面。
  • 原文结论指出,仍存在显著局限性,包括来源透明度和患者互动中的情境沟通问题,人类监督对降低风险至关重要。
  • 原文结论指出,未来应致力于改进AI能力,以实现循证且富有同理心的沟通,从而支持临床医生和家庭。

局限性

  • AI回答在引用可靠来源方面缺乏透明度。
  • AI在回答以患者为中心的问题上效果较差。
  • 评审者间一致性差,凸显了区分人工生成与AI生成回答的挑战。
  • 喉科和患者中心问题的一致性与表现较低。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: Pediatric otorhinolaryngology (ORL) addresses complex conditions in children, requiring a tailored approach for patients and families. With artificial intelligence (AI) gaining traction in medical applications, this study evaluates the quality of information provided by large language models (LLMs) in comparison to clinicians, identifying strengths and limitations in the field of pediatric ORL. STUDY DESIGN: Comparative blinded study. SETTING: Controlled research environment using LLMs. METHODS: Fifty-four items of increasing difficulty, namely 18 theoretical questions, 18 clinical scenarios, and 18 patient questions, were posed to ChatGPT-3.5, -4.0, -4o, Claude-3, Gemini, Perplexity, Copilot, a second-year resident, and an expert in the field of pediatric ORL. The Quality Analysis of Medical Artificial Intelligence (QAMAI) tool was used for blinded evaluation of the quality of medical information by a panel of expert members from the Young Otolaryngologists Group of the Italian Society of ORL and the International Federation of ORL Societies. RESULTS: LLMs performed comparably to specialist in theoretical and standardized clinical scenarios, with Bing Copilot achieving the highest QAMAI scores. However, AI responses lacked transparency in citing reliable sources and were less effective in addressing patient-centered questions. Poor interrater agreement among reviewers highlighted challenges in distinguishing human-generated from AI-generated responses. Rhinology topics received the highest scores, whereas laryngology and patient-centered questions showed lower agreement and performance. CONCLUSION: LLMs show promise as supportive resources in pediatric ORL, particularly in theoretical learning and standardized cases. However, significant limitations remain, including source transparency and contextual communication in patient interactions. Human oversight is essential to mitigate risks. Future developments should focus on refining AI capabilities for evidence-based and empathetic communication to support both clinicians and families.

来源:Otolaryngol Head Neck Surg,查看原文。版权归原作者所有。

引用

  1. Trecca EMC, Caponio VCA, Turri-Zanoni M, et al. Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models. Otolaryngol Head Neck Surg. 2025. PMID: 40105497. doi:10.1002/ohn.1225

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。