ChatGPT与GPT-4在肺癌自由文本CT报告数据挖掘中的潜力
本回顾性研究纳入2021年9月至2023年3月接受肺癌随访CT的患者424例,比较ChatGPT与GPT-4在用户自定义提示下从自由文本CT报告中提取病变直径、标注转移性疾病及评估肿瘤学进展的性能。GPT-4在提取病变参数、识别转移性疾病、生成肿瘤学进展正确标签及肿瘤学推理方面均优于ChatGPT,且虚构内容发生率更低。
要点
- GPT-4提取病变参数的准确率为98.6%,高于ChatGPT的84.0%(P < .001)。
- GPT-4识别转移性疾病的准确率为98.1%(95% CI: 97.7, 98.5),高于ChatGPT的90.3%(95% CI: 89.4, 91.0)。
- GPT-4肿瘤学推理的事实正确性Likert评分为4.3,准确性为4.4,均高于ChatGPT的3.9和3.3。
核心结论
- 在使用用户自定义提示时,GPT-4在从肺癌自由文本CT报告中提取肿瘤学表型方面优于ChatGPT,并表现出更好的肿瘤学推理能力且虚构内容更少。
研究设计与样本
- 回顾性研究纳入2021年9月至2023年3月接受肺癌随访CT的患者424例(平均年龄65岁±11[SD],男性265例),保留25份报告用于提示工程,输出输入基于规则的自然语言处理流程,与四位放射科医师的真实标注匹配。
诊断标准与关键数值
- GPT-4提取病变参数的准确率为98.6%,ChatGPT为84.0%(P < .001)。
- GPT-4正确挖掘报告的比例为96%,ChatGPT为67%(P < .001)。
- GPT-4识别转移性疾病的准确率为98.1%(95% CI: 97.7, 98.5),ChatGPT为90.3%(95% CI: 89.4, 91.0)。
- GPT-4生成肿瘤学进展正确标签的F1分数为0.96(95% CI: 0.94, 0.98),ChatGPT为0.91(95% CI: 0.89, 0.94)(两者P < .001)。
- GPT-4肿瘤学推理的事实正确性Likert评分为4.3,准确性为4.4;ChatGPT分别为3.9和3.3(均P < .001)。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background The latest large language models (LLMs) solve unseen problems via user-defined text prompts without the need for retraining, offering potentially more efficient information extraction from free-text medical records than manual annotation. Purpose To compare the performance of the LLMs ChatGPT and GPT-4 in data mining and labeling oncologic phenotypes from free-text CT reports on lung cancer by using user-defined prompts. Materials and Methods This retrospective study included patients who underwent lung cancer follow-up CT between September 2021 and March 2023. A subset of 25 reports was reserved for prompt engineering to instruct the LLMs in extracting lesion diameters, labeling metastatic disease, and assessing oncologic progression. This output was fed into a rule-based natural language processing pipeline to match ground truth annotations from four radiologists and derive performance metrics. The oncologic reasoning of LLMs was rated on a five-point Likert scale for factual correctness and accuracy. The occurrence of confabulations was recorded. Statistical analyses included Wilcoxon signed rank and McNemar tests. Results On 424 CT reports from 424 patients (mean age, 65 years ± 11 [SD]; 265 male), GPT-4 outperformed ChatGPT in extracting lesion parameters (98.6% vs 84.0%, P < .001), resulting in 96% correctly mined reports (vs 67% for ChatGPT, P < .001). GPT-4 achieved higher accuracy in identification of metastatic disease (98.1% [95% CI: 97.7, 98.5] vs 90.3% [95% CI: 89.4, 91.0]) and higher performance in generating correct labels for oncologic progression (F1 score, 0.96 [95% CI: 0.94, 0.98] vs 0.91 [95% CI: 0.89, 0.94]) (both P < .001). In oncologic reasoning, GPT-4 had higher Likert scale scores for factual correctness (4.3 vs 3.9) and accuracy (4.4 vs 3.3), with a lower rate of confabulation (1.7% vs 13.7%) than ChatGPT (all P < .001). Conclusion When using user-defined prompts, GPT-4 outperformed ChatGPT in extracting oncologic phenotypes from free-text CT reports on lung cancer and demonstrated better oncologic reasoning with fewer confabulations. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Hafezi-Nejad and Trivedi in this issue.
引用
- Fink MA, Bischoff A, Fink CA, et al. Potential of ChatGPT and GPT-4 for Data Mining of Free-Text CT Reports on Lung Cancer. Radiology. 2023. PMID: 37724963. doi:10.1148/radiol.231362
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。