文献速递胸部技术与安全

重新审视大语言模型与肺癌问题:两年后人工智能如何回答常见肺癌问题

Revisiting LLMs and Lung Cancer Questions: How AI Responds to Common Lung Cancer Questions Two Years Later.
PubMed 文献 · Acad Radiol · 2025-09-16

本研究由经验丰富的放射科医生基于Lung-RADS和Fleischner学会指南制定40个肺癌相关问题,于2025年2月16日测试五个公开可用的大语言模型(GPT-o3-mini、Gemini、DeepSeek R1、Claude、Perplexity),由三位放射学专家独立评分。GPT-o3-mini完全正确率最高(75.83%),Gemini次之(74.17%);GPT-o3-mini对Claude的完全正确比值比有统计学显著优势(p=0.02)。结论认为GPT-o3-mini总体准确率最佳但评分者间变异明显,尽管较旧版本有改善,不一致表现、错误回答和拒答等问题仍持续存在,专家监督仍很重要。

要点

  • GPT-o3-mini完全正确率最高(75.83%),Gemini次之(74.17%),Claude部分正确率最高(17.5%)。
  • GPT-o3-mini对Claude的完全正确比值比有统计学显著优势(p=0.02)。
  • GPT-o3-mini识别故意错误Lung-RADS问题的可靠性高于DeepSeek或Gemini。

核心结论

  • GPT-o3-mini在受测模型中总体准确率最佳(75.83%),但评分者间变异明显。
  • GPT-o3-mini和Gemini较旧版本准确率有改善(75.83%和74.17% vs 旧版GPT-3.5 70.8%、Bard 51.7%),但不一致表现、错误回答和拒答等问题仍持续存在。
  • 作者指出近两年后专家监督仍很重要。

研究设计与样本

  • 研究类型为横断面标准化问题测试,由经验丰富的放射科医生基于Lung-RADS和Fleischner学会指南制定40个肺癌相关问题。
  • 测试于2025年2月16日进行,五个公开可用大语言模型(GPT-o3-mini、Gemini、DeepSeek R1、Claude、Perplexity)各回答40个问题。
  • 由三位放射学专家以多数投票系统独立评分,答案分为正确、部分正确、错误或未回答(拒答),采用Cohen's Kappa评估评分者间一致性。

诊断标准与关键数值

  • GPT-o3-mini完全正确率75.83%,Gemini完全正确率74.17%,Claude部分正确率17.5%。
  • GPT-o3-mini较Claude完全正确比值比有统计学显著优势(p=0.02)。
  • DeepSeek R1评分者间一致性最高;GPT-o3-mini在加权Kappa上评分者间存在明显分歧。
  • GPT-o3-mini识别故意错误Lung-RADS问题的可靠性高于DeepSeek或Gemini。

临床应用与阅片要点

  • 作者指出尽管模型准确率较旧版有改善,不一致表现、错误回答和拒答等问题仍持续存在。
  • 作者认为近两年后专家监督仍很重要。

局限性

  • 原文结论指出不一致表现、错误回答和拒答等问题仍持续存在。
  • 原文结论指出GPT-o3-mini评分者间变异明显。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: This study aimed to evaluate and compare the accuracy, consistency, and reliability of five state-of-the-art LLMs available to the public (chatbots) - GPT-o3-mini, Gemini, DeepSeek R1, Claude, and Perplexity-in answering standardized lung cancer-related questions, building upon prior assessments of earlier models in 2023.

METHODS: Forty lung cancer-related questions were developed by experienced radiologists, based on Lung-RADS and Fleischner Society guidelines. On February 16, 2025, each of the five models answered these standardized questions. Responses were independently graded by three radiology experts using a majority-voting system to classify answers as correct, partially correct, incorrect, or no answer (refusal). Statistical analyses included accuracy calculations, logistic regression models to compare response correctness across chatbots, and inter-rater agreement assessments (Cohen's Kappa).

RESULTS: GPT-o3-mini achieved the highest accuracy (75.83% correct responses), closely followed by Gemini (74.17%). Claude demonstrated the highest rate of partially correct answers (17.5%). Statistical analysis showed GPT-o3-mini had significantly higher odds of fully correct answers compared to Claude (p=0.02). DeepSeek R1 showed the highest agreement, while GPT-o3-mini had substantial disagreement among raters on weighted Kappa. Notably, GPT-o3-mini identified intentionally incorrect Lung-RADS questions more reliably than DeepSeek or Gemini.

CONCLUSION: GPT-o3-mini showed the best overall accuracy among the models tested but had notable inter-rater variability. While GPT-o3-mini and Gemini showed improved accuracy (75.83% and 74.17%) over their older iterations in prior assessments (GPT-3.5 70.8%, Bard 51.7%), ongoing issues persist. Nearly two years later, problems such as inconsistent performance, incorrect responses, and refusals emphasize the continued importance of expert oversight.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Nguyen D, Rahsepar AA, Chalian H, et al. Revisiting LLMs and Lung Cancer Questions: How AI Responds to Common Lung Cancer Questions Two Years Later. Acad Radiol. 2025. PMID: 40962633. doi:10.1016/j.acra.2025.08.056

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。