文献速递小儿泌尿与泌尿修复

AI聊天机器人对小儿泌尿外科常见问题的回答评估

Evaluation of AI Chatbot Responses to Pediatric Urology Frequently Asked Questions.
PubMed 文献 · Urology · 2026-08-17

本研究评估4种公开可用的大语言模型(ChatGPT-4o、Claude 3.7、Gemini 2.5和Copilot)对小儿泌尿外科常见问题的回答质量。两位认证小儿泌尿外科医生从准确性、完整性、安全性、清晰度、可操作性、简洁性和整体质量7个维度进行5分Likert评分。结果显示所有模型在准确性(平均4.27)、安全性(4.36)和清晰度(4.42)上评分较高,Claude整体质量最高(4.35),但阅读难度达11-15年级水平,超过推荐的6-8年级患者教育标准。

要点

  • LLMs在准确性、安全性和清晰度上获专家高评分,但完整性不足且阅读难度偏高。
  • Claude整体质量最高(4.35),ChatGPT安全性高但完整性较低。
  • 阅读水平达11-15年级,远超推荐的6-8年级患者教育标准。

核心结论

  • LLMs可提供准确、安全且支持性的信息,但受限于完整性不足和高阅读难度。

研究设计与样本

  • 评估4种公开LLM(ChatGPT-4o、Claude 3.7、Gemini 2.5、Copilot)对小儿泌尿外科FAQ的回答。
  • 两位认证小儿泌尿外科医生独立从7个维度以5分Likert量表评分。

主要发现

  • 所有LLM在准确性(平均4.27)、安全性(4.36)和清晰度(4.42)上获高评分。
  • 整体质量评分:Claude 4.35,Gemini 4.03,ChatGPT 4.00,Copilot 3.80。
  • 情感基调主要为积极和支持性。
  • 阅读难度对应11-15年级水平。

关键数值与推荐

  • 准确性平均4.27,安全性4.36,清晰度4.42(5分Likert量表)。
  • 整体质量:Claude 4.35,Gemini 4.03,ChatGPT 4.00,Copilot 3.80。
  • 推荐患者教育阅读标准为6-8年级。

临床怎么用

  • LLMs可用于为家长提供准确、安全且支持性的信息,但需注意完整性不足和阅读难度高的问题。

局限性

  • 未报告具体样本量、P值、置信区间等统计细节。
  • 仅评估了4种LLM,未涉及其他模型。
  • 未报告评分者间一致性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: To evaluate the quality of responses from 4 publicly available LLMs (ChatGPT-4o, Claude 3.7, Gemini 2.5, and Copilot) to frequently asked questions (FAQs) in pediatric urology. METHODS: FAQs were generated using standardized prompts and submitted to each LLM using parent-centered instructions. Two board-certified pediatric urologists independently rated responses across 7 domains as follows: accuracy, completeness, safety, clarity, actionability, conciseness, and Global Quality Score using a 5-point Likert scale. Emotional tone was analyzed using the NRC Word-Emotion Association Lexicon, and readability was assessed using 7 established metrics. RESULTS: All LLMs produced generally high expert ratings for accuracy (mean 4.27), safety (4.36), and clarity (4.42). Claude achieved the highest overall quality score (4.35), followed by Gemini (4.03), ChatGPT (4.00), and Copilot (3.80). Emotional tone was predominantly positive and supportive across models. Reading corresponded to an 11th-15th grade reading level, exceeding the recommended 6th-8th grade patient-education standards. CONCLUSION: LLMs can provide accurate, safe, and supportive information for parents, but their usefulness is limited by gaps in completeness and high readability levels. Claude achieved the highest overall quality, whereas ChatGPT demonstrated high safety with lower completeness. Future development would prioritize plain-language optimization, context-aware emotional framing, and parent co-design to improve comprehension.

来源:Urology,查看原文。版权归原作者所有。

引用

  1. Mazhari N, Freedman A, Friedrich N, et al. Evaluation of AI Chatbot Responses to Pediatric Urology Frequently Asked Questions. Urology. 2026. PMID: 42607894. doi:10.1016/j.urology.2026.08.006

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。