文献速递腹部技术与安全CT

利用大语言模型辅助分析增强肿瘤学监测:GPT-4与Gemini在评估系列腹部CT扫描报告肿瘤学问题中的比较研究

Enhancing Oncological Surveillance Through Large Language Model-Assisted Analysis: A Comparative Study of GPT-4 and Gemini in Evaluating Oncological Issues From Serial Abdominal CT Scan Reports.
PubMed 文献 · Acad Radiol · 2024-12-09

本研究纳入205例各有两份连续影像报告的患者,设计三步任务提示词,以两名放射科医生共识的六级分类为参考标准,比较GPT-4与Gemini匹配连续报告所见并归类肿瘤状态的能力。GPT-4匹配正确比例96.2%高于Gemini的91.7%(P<0.01),肿瘤相关所见精确率0.68对0.63(P=0.006)、召回率0.91对0.80(P<0.001)、F1值0.78对0.70,GPT-4判断肿瘤状态更准确(P<0.001)。

要点

  • GPT-4匹配连续报告所见正确比例96.2%,高于Gemini的91.7%(P<0.01)。
  • 肿瘤相关所见精确率GPT-4 0.68 vs Gemini 0.63(P=0.006),召回率0.91 vs 0.80(P<0.001)。
  • GPT-4判断肿瘤状态正确性优于Gemini(P<0.001),F1值分别为0.78和0.70。

核心结论

  • 使用精心设计的提示词,大语言模型辅助分析连续影像报告可能增强肿瘤学监测;GPT-4在匹配相应所见、识别肿瘤相关所见及准确判断肿瘤状态方面表现优于Gemini。

研究设计与样本

  • 比较研究,纳入205例患者,每例各两份连续影像报告,以两名放射科医生共识的六级分类(肿瘤所见分为改善、稳定、加重,以及良性、无肿瘤描述、其他恶性肿瘤)为参考标准,比较GPT-4与Gemini匹配报告所见并归类的能力。

诊断标准与关键数值

  • 匹配连续报告所见正确比例GPT-4为96.2%、Gemini为91.7%(P<0.01);肿瘤相关所见判断精确率GPT-4 0.68 vs Gemini 0.63(P=0.006),召回率GPT-4 0.91 vs Gemini 0.80(P<0.001),F1值GPT-4 0.78 vs Gemini 0.70;GPT-4判断肿瘤状态正确性优于Gemini(P<0.001)。

临床应用与阅片要点

  • 作者认为大语言模型辅助分析连续影像报告可能增强肿瘤学监测,需使用精心设计的提示词。
  • 作者结论显示GPT-4在匹配相应所见、识别肿瘤相关所见及判断肿瘤状态方面表现优于Gemini。

局限性

  • 原文未报告局限性

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: We aimed to compare the capabilities of two leading large language models (LLMs), GPT-4 and Gemini, in analyzing serial radiology reports, to highlight oncological issues that require further clinical attention.

MATERIALS AND METHODS: This study included 205 patients, each with two consecutive radiological reports. We designed a prompt comprising a three-step task to analyze report findings using LLMs. To establish a ground truth, two radiologists reached a consensus on a six-level categorization, comprising tumor findings (categorized as improved, stable, or aggravated), "benign", "no tumor description," and "other malignancy." The performance of GPT-4 and Gemini was then compared based on their ability to match corresponding findings between two radiological reports and accurately reflect these categories.

RESULTS: In terms of accuracy in matching findings between serial reports, the proportion of correctly matched findings was significantly higher for GPT-4 (96.2%) than for Gemini (91.7%) (P < 0.01). For oncological issue identification, the precision for tumor-related finding determinations, recall, and F1-scores were 0.68 and 0.63 (P = 0.006), 0.91 and 0.80 (P < 0.001), and 0.78 and 0.70 for GPT-4 and Gemini, respectively. GPT-4 was more accurate than Gemini in determining the correct tumor status for tumor-related findings (P < 0.001).

CONCLUSION: This study demonstrated the potential of LLM-assisted analysis of serial radiology reports in enhancing oncological surveillance, using a carefully engineered prompt. GPT-4 showed superior performance compared to Gemini in matching corresponding findings, identifying tumor-related findings, and accurately determining tumor status.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Han NY, Shin K, Kim MJ, et al. Enhancing Oncological Surveillance Through Large Language Model-Assisted Analysis: A Comparative Study of GPT-4 and Gemini in Evaluating Oncological Issues From Serial Abdominal CT Scan Reports. Acad Radiol. 2024. PMID: 39658474. doi:10.1016/j.acra.2024.10.050

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。