重新审视大语言模型与肺癌问题:两年后人工智能如何回答常见肺癌问题
本研究由经验丰富的放射科医生基于Lung-RADS和Fleischner学会指南制定40个肺癌相关问题,于2025年2月16日测试五个公开可用的大语言模型(GPT-o3-mini、Gemini、DeepSeek R1、Claude、Perplexity),由三位放射学专家独立评分。GPT-o3-mini完全正确率最高(75.83%),Gemini次之(74.17%);GPT-o3-mini对Claude的完全正确比值比有统计学显著优势(p=0.02)。结论认为GPT-o3-mini总体准确率最佳但评分者间变异明显,尽管较旧版本有改善,不一致表现、错误回答和拒答等问题仍持续存在,专家监督仍很重要。
要点
- GPT-o3-mini完全正确率最高(75.83%),Gemini次之(74.17%),Claude部分正确率最高(17.5%)。
- GPT-o3-mini对Claude的完全正确比值比有统计学显著优势(p=0.02)。
- GPT-o3-mini识别故意错误Lung-RADS问题的可靠性高于DeepSeek或Gemini。
核心结论
- GPT-o3-mini在受测模型中总体准确率最佳(75.83%),但评分者间变异明显。
- GPT-o3-mini和Gemini较旧版本准确率有改善(75.83%和74.17% vs 旧版GPT-3.5 70.8%、Bard 51.7%),但不一致表现、错误回答和拒答等问题仍持续存在。
- 作者指出近两年后专家监督仍很重要。
研究设计与样本
- 研究类型为横断面标准化问题测试,由经验丰富的放射科医生基于Lung-RADS和Fleischner学会指南制定40个肺癌相关问题。
- 测试于2025年2月16日进行,五个公开可用大语言模型(GPT-o3-mini、Gemini、DeepSeek R1、Claude、Perplexity)各回答40个问题。
- 由三位放射学专家以多数投票系统独立评分,答案分为正确、部分正确、错误或未回答(拒答),采用Cohen's Kappa评估评分者间一致性。
诊断标准与关键数值
- GPT-o3-mini完全正确率75.83%,Gemini完全正确率74.17%,Claude部分正确率17.5%。
- GPT-o3-mini较Claude完全正确比值比有统计学显著优势(p=0.02)。
- DeepSeek R1评分者间一致性最高;GPT-o3-mini在加权Kappa上评分者间存在明显分歧。
- GPT-o3-mini识别故意错误Lung-RADS问题的可靠性高于DeepSeek或Gemini。
临床应用与阅片要点
- 作者指出尽管模型准确率较旧版有改善,不一致表现、错误回答和拒答等问题仍持续存在。
- 作者认为近两年后专家监督仍很重要。
局限性
- 原文结论指出不一致表现、错误回答和拒答等问题仍持续存在。
- 原文结论指出GPT-o3-mini评分者间变异明显。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: This study aimed to evaluate and compare the accuracy, consistency, and reliability of five state-of-the-art LLMs available to the public (chatbots) - GPT-o3-mini, Gemini, DeepSeek R1, Claude, and Perplexity-in answering standardized lung cancer-related questions, building upon prior assessments of earlier models in 2023.
METHODS: Forty lung cancer-related questions were developed by experienced radiologists, based on Lung-RADS and Fleischner Society guidelines. On February 16, 2025, each of the five models answered these standardized questions. Responses were independently graded by three radiology experts using a majority-voting system to classify answers as correct, partially correct, incorrect, or no answer (refusal). Statistical analyses included accuracy calculations, logistic regression models to compare response correctness across chatbots, and inter-rater agreement assessments (Cohen's Kappa).
RESULTS: GPT-o3-mini achieved the highest accuracy (75.83% correct responses), closely followed by Gemini (74.17%). Claude demonstrated the highest rate of partially correct answers (17.5%). Statistical analysis showed GPT-o3-mini had significantly higher odds of fully correct answers compared to Claude (p=0.02). DeepSeek R1 showed the highest agreement, while GPT-o3-mini had substantial disagreement among raters on weighted Kappa. Notably, GPT-o3-mini identified intentionally incorrect Lung-RADS questions more reliably than DeepSeek or Gemini.
CONCLUSION: GPT-o3-mini showed the best overall accuracy among the models tested but had notable inter-rater variability. While GPT-o3-mini and Gemini showed improved accuracy (75.83% and 74.17%) over their older iterations in prior assessments (GPT-3.5 70.8%, Bard 51.7%), ongoing issues persist. Nearly two years later, problems such as inconsistent performance, incorrect responses, and refusals emphasize the continued importance of expert oversight.
引用
- Nguyen D, Rahsepar AA, Chalian H, et al. Revisiting LLMs and Lung Cancer Questions: How AI Responds to Common Lung Cancer Questions Two Years Later. Acad Radiol. 2025. PMID: 40962633. doi:10.1016/j.acra.2025.08.056
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。