文献速递胸部技术与安全CTPET

使用胸部CT和FDG PET/CT自由文本报告进行肺癌分期:三种ChatGPT大语言模型与六名不同经验人类读者的比较

Lung Cancer Staging Using Chest CT and FDG PET/CT Free-Text Reports: Comparison Among Three ChatGPT Large Language Models and Six Human Readers of Varying Experience.
PubMed 文献 · AJR Am J Roentgenol · 2024-09-04

这项回顾性研究纳入韩国四家机构700例接受胸部CT或FDG PET/CT初始分期的非小细胞肺癌患者,比较三种ChatGPT模型与六名不同经验人类读者基于自由文本报告确定总体分期分组的准确性。GPT-4o总体分期准确率为74.1%,显著优于GPT-4(70.1%)和GPT-3.5(57.4%),与住院医师1(72.3%)无显著差异,但显著低于两名专科培训放射科医师(82.3%和85.4%)。结论不支持用大语言模型替代专家进行肺癌分期。

要点

  • GPT-4o基于自由文本报告进行肺癌分期的准确率为74.1%,与专科培训医师相比无显著差异。
  • GPT-4o准确率显著低于两名专科培训放射科医师(82.3%和85.4%),不支持替代专家。
  • GPT-4o准确率显著高于GPT-4(70.1%)和GPT-3.5(57.4%)。

核心结论

  • 表现最佳的GPT-4o模型分期准确率与专科培训医师无显著差异,但显著低于专科培训放射科医师;研究结果不支持使用大语言模型替代专家医疗人员进行肺癌分期。

研究设计与样本

  • 回顾性研究,纳入韩国四家机构700例患者(平均年龄73.8±29.5岁;男509例,女191例),于2020年1月至2023年12月因非小细胞肺癌初始分期接受胸部CT或FDG PET/CT,报告为自由文本格式,仅英文或英文与韩文混合书写;由两名胸部放射科医师根据第8版AJCC癌症分期手册共识确定参考标准总体分期分组(IA、IB、IIA、IIB、IIIA、IIIB、IIIC、IVA或IVB);三种ChatGPT模型通过基于脚本的应用程序编程接口、零样本学习和包含分期系统摘要的提示确定总体分期分组;六名人类读者(两名经验少于参考标准医师的专科培训放射科医师、两名专科培训医师和两名住院医师)独立确定总体分期分组;使用McNemar检验比较GPT-4o的总体准确率与其他大语言模型和人类读者。

诊断标准与关键数值

  • GPT-4o在九个分期分组中确定正确分期的总体准确率为74.1%;GPT-4准确率为70.1%(与GPT-4o比较p=0.02);GPT-3.5准确率为57.4%(与GPT-4o比较p<0.001);专科培训放射科医师1准确率为82.3%(与GPT-4o比较p<0.001);专科培训放射科医师2准确率为85.4%(与GPT-4o比较p<0.001);专科培训医师1准确率为77.7%(与GPT-4o比较p=0.09);专科培训医师2准确率为75.6%(与GPT-4o比较p=0.53);住院医师1准确率为72.3%(与GPT-4o比较p=0.42);住院医师2准确率为65.7%(与GPT-4o比较p<0.001)。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

BACKGROUND. Although radiology reports are commonly used for lung cancer staging, this task can be challenging given radiologists' variable reporting styles as well as reports' potentially ambiguous and/or incomplete staging-related information. OBJECTIVE. The purpose of this study was to compare the performance of ChatGPT large language models (LLMs) and human readers of varying experience in lung cancer staging using chest CT and FDG PET/CT free-text reports. METHODS. This retrospective study included 700 patients (mean age, 73.8 ± 29.5 [SD] years; 509 men, 191 women) from four institutions in Korea who underwent chest CT or FDG PET/CT for non-small cell lung cancer initial staging from January 2020 to December 2023. Examinations' reports used a free-text format, written exclusively in English or in mixed English and Korean. Two thoracic radiologists in consensus determined the overall stage group (IA, IB, IIA, IIB, IIIA, IIIB, IIIC, IVA, or IVB) for each report using the 8th-edition AJCC Cancer Staging Manual to establish the reference standard. Three ChatGPT models (GPT-4o, GPT-4, GPT-3.5) determined an overall stage group for each report using a script-based application programming interface, zero-shot learning, and a prompt incorporating a staging system summary. The code for this web application was made publicly available through a GitHub repository (https://github.com/elmidion/GPT_Information_Extractor). Six human readers (two fellowship-trained radiologists with less experience than the radiologists who determined the reference standard, two fellows, and two residents) also independently determined overall stage groups. GPT-4o's overall accuracy for determining the correct stage among the nine groups was compared with that of the other LLMs and human readers using McNemar tests. RESULTS. GPT-4o had an overall staging accuracy of 74.1%, significantly better than the accuracy of GPT-4 (70.1%, p = .02), GPT-3.5 (57.4%, p < .001), and resident 2 (65.7%, p < .001); significantly worse than the accuracy of fellowship-trained radiologist 1 (82.3%, p < .001) and fellowship-trained radiologist 2 (85.4%, p < .001); and not significantly different from the accuracy of fellow 1 (77.7%, p = .09), fellow 2 (75.6%, p = .53), and resident 1 (72.3%, p = .42). CONCLUSION. The best-performing model, GPT-4o, showed no significant difference in staging accuracy versus fellows but showed significantly worse performance versus fellowship-trained radiologists. The findings do not support use of LLMs for lung cancer staging in place of expert health care professionals. CLINICAL IMPACT. The findings indicate the importance of domain expertise for performing complex specialized tasks such as cancer staging.

来源:AJR Am J Roentgenol,查看原文。版权归原作者所有。

引用

  1. Lee JE, Park KS, Kim YH, et al. Lung Cancer Staging Using Chest CT and FDG PET/CT Free-Text Reports: Comparison Among Three ChatGPT Large Language Models and Six Human Readers of Varying Experience. AJR Am J Roentgenol. 2024. PMID: 39230409. doi:10.2214/AJR.24.31696

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。