大语言模型用于胰腺癌自动化结构化报告及可切除性分类
本回顾性研究纳入180份胰腺导管腺癌分期CT报告,评估GPT-3.5与GPT-4自动生成结构化报告及分类可切除性的性能;GPT-4生成结构化报告的F1为0.997,优于GPT-3.5的0.967;GPT-4采用思维链提示分类可切除性准确率最高达92%;外科医师使用AI生成报告分类可切除性更准确且用时更少。
要点
- GPT-4从原始报告生成胰腺癌结构化报告的F1为0.997,优于GPT-3.5的0.967。
- GPT-4采用思维链提示分类可切除性准确率92%,高于上下文知识提示的83%和默认知识的67%。
- 外科医师使用AI生成报告分类可切除性准确率83%,高于原始报告的76%,每份报告用时减少58%。
核心结论
- GPT-4可从原始报告生成接近完美的胰腺导管腺癌结构化报告,采用思维链提示分类可切除性准确率高,外科医师使用AI生成报告更准确且更高效。
研究设计与样本
- 经机构审查委员会批准的回顾性研究,纳入2018年1月至12月转诊至作者所在欧洲肿瘤内科学会指定癌症中心的连续180份胰腺导管腺癌分期CT报告。
- 由两名放射科医师审阅报告,建立14项关键发现及美国国家综合癌症网络可切除性分类的参考标准。
- 使用GPT-3.5与GPT-4(2023年9月18日至29日访问)从原始报告生成包含相同14项特征的结构化报告,评估召回率、精确率与F1分数。
- 可切除性分类对两种模型采用三种提示策略(默认知识、上下文知识、思维链),由肝胆胰外科医师审阅原始报告与AI生成报告确定可切除性并比较准确率与审阅时间。
- 统计方法包括McNemar检验、t检验、Wilcoxon符号秩检验及混合效应逻辑回归模型。
诊断标准与关键数值
- GPT-4生成结构化报告的F1为0.997,GPT-3.5为0.967。
- GPT-4对全部14项提取特征的F1分数等于或高于GPT-3.5。
- GPT-4提取肠系膜上动脉受累的精确率为100%,GPT-3.5为88.8%。
- GPT-4各提示策略的可切除性分类均优于GPT-3.5;GPT-4思维链提示准确率92%,优于上下文知识提示的83%(P=0.002),后者优于默认知识策略的67%(P<0.001)。
- 外科医师使用AI生成报告分类可切除性准确率83%,使用原始报告为76%(P=0.03)。
临床应用与阅片要点
- 外科医师使用AI生成报告分类可切除性准确率83%,高于原始报告的76%,每份报告用时减少58%(95%CI:0.53,0.62)。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background Structured radiology reports for pancreatic ductal adenocarcinoma (PDAC) improve surgical decision-making over free-text reports, but radiologist adoption is variable. Resectability criteria are applied inconsistently. Purpose To evaluate the performance of large language models (LLMs) in automatically creating PDAC synoptic reports from original reports and to explore performance in categorizing tumor resectability. Materials and Methods In this institutional review board-approved retrospective study, 180 consecutive PDAC staging CT reports on patients referred to the authors' European Society for Medical Oncology-designated cancer center from January to December 2018 were included. Reports were reviewed by two radiologists to establish the reference standard for 14 key findings and National Comprehensive Cancer Network (NCCN) resectability category. GPT-3.5 and GPT-4 (accessed September 18-29, 2023) were prompted to create synoptic reports from original reports with the same 14 features, and their performance was evaluated (recall, precision, F1 score). To categorize resectability, three prompting strategies (default knowledge, in-context knowledge, chain-of-thought) were used for both LLMs. Hepatopancreaticobiliary surgeons reviewed original and artificial intelligence (AI)-generated reports to determine resectability, with accuracy and review time compared. The McNemar test, t test, Wilcoxon signed-rank test, and mixed effects logistic regression models were used where appropriate. Results GPT-4 outperformed GPT-3.5 in the creation of synoptic reports (F1 score: 0.997 vs 0.967, respectively). Compared with GPT-3.5, GPT-4 achieved equal or higher F1 scores for all 14 extracted features. GPT-4 had higher precision than GPT-3.5 for extracting superior mesenteric artery involvement (100% vs 88.8%, respectively). For categorizing resectability, GPT-4 outperformed GPT-3.5 for each prompting strategy. For GPT-4, chain-of-thought prompting was most accurate, outperforming in-context knowledge prompting (92% vs 83%, respectively; P = .002), which outperformed the default knowledge strategy (83% vs 67%, P < .001). Surgeons were more accurate in categorizing resectability using AI-generated reports than original reports (83% vs 76%, respectively; P = .03), while spending less time on each report (58%; 95% CI: 0.53, 0.62). Conclusion GPT-4 created near-perfect PDAC synoptic reports from original reports. GPT-4 with chain-of-thought achieved high accuracy in categorizing resectability. Surgeons were more accurate and efficient using AI-generated reports. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Chang in this issue.
引用
- Bhayana R, Nanda B, Dehkharghanian T, et al. Large Language Models for Automated Synoptic Reports and Resectability Categorization in Pancreatic Cancer. Radiology. 2024. PMID: 38888478. doi:10.1148/radiol.233117
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。