指南与共识妇科

评估大型语言模型在欧洲宫颈癌指南上的准确性:一项计算机模拟基准研究

Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study.
指南 / 共识 · BJOG · 2025-11-24

本研究基于ESGO/ESTRO/ESP宫颈癌指南设计50个问题,前瞻性比较ChatGPT 4.0、DeepSeek R1和Gemini 2.0的回答准确性、一致性和可靠性。ChatGPT 4.0的GQS 5比例最高(42%),中位GQS为4.00,Gemini 2.0和DeepSeek R1均为3.50。各模型可靠性无显著差异(p=0.602),但准确性均不理想,作者认为专家监督对安全临床应用仍必不可少。

要点

  • ChatGPT 4.0的GQS 5回答比例为42%,高于Gemini 2.0的30%和DeepSeek R1的28%。
  • ChatGPT 4.0中位GQS为4.00,Gemini 2.0和DeepSeek R1均为3.50。
  • 各模型间可靠性无显著差异(p=0.602),但准确性均不理想,专家监督仍必不可少。

核心结论

  • 所有模型与临床指南对齐的准确性均不理想;ChatGPT 4.0最准确且一致性最好,DeepSeek R1表现较差;尽管各模型可靠性相似,专家监督对确保安全临床应用和防止错误信息仍必不可少。

研究设计与样本

  • 前瞻性、比较性计算机模拟基准研究,在意大利罗马Fondazione Policlinico Universitario A. Gemelli开展。
  • 样本为源自ESGO/ESTRO/ESP宫颈癌指南的50个问题。
  • 每个问题同时提交给ChatGPT 4.0、DeepSeek R1和Gemini 2.0,并重复输入两次以评估回答可重复性。

主要发现

  • ChatGPT 4.0的GQS 5回答比例为42%,Gemini 2.0为30%,DeepSeek R1为28%。
  • ChatGPT 4.0中位GQS为4.00,Gemini 2.0和DeepSeek R1的中位GQS均为3.50。
  • 回答一致性存在显著差异:ChatGPT 4.0与Gemini 2.0比较p=0.034,DeepSeek R1与Gemini 2.0比较p=0.044。
  • 可靠性无显著差异(p=0.602)。

关键数值与推荐

  • 准确性采用全局质量评分(GQS)评估,范围1分(差)至5分(完全准确)。
  • 一致性(模型内回答稳定性)和可靠性(与指南的一致性)采用二分类评估。

临床怎么用

  • 原文结论指出,尽管各模型可靠性相似,专家监督对确保安全临床应用和防止错误信息仍必不可少。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: Although large language models are increasingly used in clinical and research settings, the validity of the information they provide remains uncertain. This study aimed to evaluate the accuracy, consistency, and reliability of three large language models-ChatGPT 4.0, DeepSeek R1, and Gemini 2.0-in answering cervical cancer-related questions based on the ESGO/ESTRO/ESP guidelines. DESIGN: Prospective, comparative in silico benchmarking study. SETTING: Fondazione Policlinico Universitario A. Gemelli, Rome, Italy. POPULATION OR SAMPLE: Fifty questions derived from the ESGO/ESTRO/ESP (European Society of Gynaecologic Oncology/European Society for Radiotherapy and Oncology/European Society of Pathology) Guidelines for Cervical Cancer. METHODS: Each question was submitted simultaneously to ChatGPT 4.0, DeepSeek R1, and Gemini 2.0, and re-entered twice to assess response repeatability. Answers were evaluated for accuracy using a Global Quality Score (GQS) from 1 (poor) to 5 (completely accurate). Consistency (intra-model response stability) and reliability (alignment with guidelines) were assessed using binary classification. MAIN OUTCOME MEASURES: Median GQS, percentage of GQS 5 responses, consistency between repeated answers, and reliability. RESULTS: ChatGPT 4.0 achieved the highest performance, with 42% of responses rated GQS 5, followed by Gemini 2.0 (30%) and DeepSeek R1 (28%). DeepSeek R1 and Gemini 2.0 scored lower in median GQS (3.50) compared to ChatGPT 4.0 (4.00). Response consistency varied significantly, with ChatGPT 4.0 and DeepSeek R1 showing differences from Gemini 2.0 (p = 0.034 and p = 0.044, respectively). No significant difference was observed in reliability (p = 0.602). CONCLUSION: All models demonstrated suboptimal accuracy in aligning with clinical guidelines. ChatGPT 4.0 was the most accurate and consistent whereas DeepSeek R1 underperformed. Despite similar reliability across models, expert oversight remains essential to ensure safe clinical application and prevent misinformation.

来源:BJOG,查看原文。版权归原作者所有。

引用

  1. Pavone M, Innocenzi C, Macellari N, et al. Assessing the Accuracy of Large Language Models on European Guidelines for Cervical Cancer: An In Silico Benchmarking Study. BJOG. 2025. PMID: 41287196. doi:10.1111/1471-0528.70095

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。