ChatGPT(GPT-4o)自动Lung-RADS分级:提示策略评估
本研究评价GPT-4o依据327份低剂量肺癌筛查CT报告自动分配Lung-RADS分数的能力,比较结构化与非结构化零样本提示策略。结果显示结构化提示总体准确率68.8%、加权Cohen's kappa 0.758,均高于非结构化提示,提示LLM有潜力整合入报告流程提高效率。
要点
- 结构化提示下GPT-4o与参考标准一致性更高,优先用于辅助Lung-RADS分级。
- 结构化提示过判更少、漏判略多,阅片时需注意假阴性风险。
- GPT-4o输出不能替代人工判读,仍应作为报告效率工具使用。
核心结论
- LLM在仅提供影像报告所见时能自动分配Lung-RADS评分,具备整合入报告流程的潜力。
- 结构化提示的总体准确率68.8%、加权Cohen's kappa 0.758,高于非结构化提示的58.1%与0.695。
研究设计与样本
- 研究对象为327份去标识的低剂量肺癌筛查CT报告。
- 使用GPT-4o依据报告内容分配Lung-RADS评分,比较结构化与非结构化零样本提示策略。
- 以评估者给出的分级作为参考标准,McNemar's test用于比较两种提示策略的准确率差异。
关键影像表现/征象
- 结构化提示产生的过判较非结构化提示更少,漏判更多。
诊断标准与关键数值
- 参考标准为评估者的Lung-RADS分级,评价指标包括准确率、过判、漏判及加权Cohen's kappa。
- 结构化提示:总体准确率68.8%,加权Cohen's kappa 0.758。
- 非结构化提示:总体准确率58.1%,加权Cohen's kappa 0.695。
临床应用与阅片要点
- 在将LLM辅助Lung-RADS分级纳入工作流时,建议采用结构化提示以提高输出与人工分级的符合度。
- 使用自动分级结果时需重点核查结构化提示可能多发的漏判情况。
局限性
- 研究报告来自单中心,属于回顾性设计,所评价对象为GPT-4o单一模型。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
AIM: The aim of this study is to explore the capabilities of large language models (LLMs), such as ChatGPT, in assigning Lung-Reporting and Data Systems (RADS) scores when given radiology report findings, and to explore prompt engineering practices for efficient and effective integration of artificial intelligence (AI) into radiology workflows.
MATERIAL AND METHODS: A total of 327 de-identified low-dose lung screening computed tomography (CT) reports were evaluated using GPT- 4o, with instructions to assign Lung-RADS scores based on the reports provided. Structured and unstructured zero-shot prompting strategies were tested. Accuracy, overcalls, undercalls, and weighted Cohen's kappa (linear weights) were computed for each prompt type compared with the rater. McNemar's test was used to compare accuracy between structured and unstructured prompts.
RESULTS: Agreement between GPT outputs and the rater's reference standard was higher for the structured prompt compared with the unstructured prompt. For the structured prompt, overall accuracy was 68.8% with a weighted Cohen's kappa of 0.758, while the unstructured prompt achieved an accuracy of 58.1% and a weighted Cohen's kappa of 0.695. Structured prompting produced fewer overcalls and more undercalls compared with unstructured prompting.
CONCLUSION: LLMs demonstrate promise in automatically assigning scores to screening imaging studies when provided with findings and can potentially be incorporated into reporting workflows to improve reporting efficiency.
引用
- Gulati, Parizadeh, Virador, et al. Performance of ChatGPT (GPT-4o) in automated Lung-Reporting and Data Systems (RADS) classification: a category-stratified evaluation of prompting strategies. Clin Radiol. 2026. PMID: 42784931. doi:10.1016/j.crad.2026.107466
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。