大语言模型提供的甲状腺结节癌症风险评估与管理建议的适当性
研究基于ATA和NCCN指南设计24个临床问题,由322名美国放射科医生评估ChatGPT、Gemini和Claude回答的适当性。配对样本t检验显示三模型总体表现无统计学显著差异,Claude平均分最高(21.84),ChatGPT准确率最高(92.5%)。作者认为大语言模型有潜力辅助甲状腺结节风险评估,但需要临床监督以确保与指南一致。
要点
- ChatGPT、Gemini、Claude三模型总体表现无统计学显著差异,ChatGPT准确率最高(92.5%)。
- Claude平均分最高(21.84),ChatGPT次之(21.83),Gemini最低(21.47)。
- 作者认为大语言模型需临床监督以确保与指南一致,尚需进一步开发提高可靠性。
核心结论
- ChatGPT、Gemini和Claude在支持甲状腺结节癌症风险评估方面显示潜力,但需要临床监督以确保与指南一致;Claude与ChatGPT总体表现几乎相同,Claude平均分最高但差异 marginal;需进一步开发以提高临床使用的可靠性。
研究设计与样本
- 研究由一名医学影像信息学专家和两名放射科医生基于ATA和NCCN指南开发24个临床相关问题,通过Amazon Mechanical Turk招募322名美国放射科培训或执业医生评估AI回答,使用Readability Scoring System评估可读性,SPSS定量分析,Dedoose定性分析。
诊断标准与关键数值
- Claude平均分最高(21.84),ChatGPT次之(21.83),Gemini最低(21.47);ChatGPT准确率最高(92.5%),Claude次之(92.1%),Gemini最低(90.4%);不适当回答率无显著差异,Gemini呈较高趋势。
临床应用与阅片要点
- 作者认为大语言模型如ChatGPT、Gemini和Claude在支持甲状腺结节癌症风险评估方面显示潜力,但需要临床监督以确保与指南一致。
局限性
- 原文报告需进一步开发以提高大语言模型临床使用的可靠性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
The study evaluates the appropriateness and reliability of thyroid nodule cancer risk assessment recommendations provided by large language models (LLMs) ChatGPT, Gemini, and Claude in alignment with clinical guidelines from the American Thyroid Association (ATA) and the National Comprehensive Cancer Network (NCCN). A team comprising a medical imaging informatics specialist and two radiologists developed 24 clinically relevant questions based on ATA and NCCN guidelines. The readability of AI-generated responses was evaluated using the Readability Scoring System. A total of 322 radiologists in training or practice from the United States, recruited via Amazon Mechanical Turk, assessed the AI responses. Quantitative analysis using SPSS measured the appropriateness of recommendations, while qualitative feedback was analyzed through Dedoose. The study compared the performance of three AI models ChatGPT, Gemini, and Claude in providing appropriate recommendations. Paired samples t-tests showed no statistically significant differences in overall performance among the models. Claude achieved the highest mean score (21.84), followed closely by ChatGPT (21.83) and Gemini (21.47). Inappropriate response rates did not differ significantly, though Gemini showed a trend toward higher rates. However, ChatGPT achieved the highest accuracy (92.5%) in providing appropriate responses, followed by Claude (92.1%) and Gemini (90.4%). Qualitative feedback highlighted ChatGPT's clarity and structure, Gemini's accessibility but shallowness, and Claude's organization with occasional divergence from focus. LLMs like ChatGPT, Gemini, and Claude show potential in supporting thyroid nodule cancer risk assessment but require clinical oversight to ensure alignment with guidelines. Claude and ChatGPT performed nearly identically overall, with Claude having the highest mean score, though the difference was marginal. Further development is necessary to enhance their reliability for clinical use.
引用
- Alarifi M. Appropriateness of Thyroid Nodule Cancer Risk Assessment and Management Recommendations Provided by Large Language Models. J Imaging Inform Med. 2025. PMID: 40032759. doi:10.1007/s10278-025-01454-1
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。