指南与共识腹部技术与安全CTMR

多模态大语言模型结合流程图嵌入知识检索为胰腺囊性病变形成随访建议

Multimodal Large Language Model With Knowledge Retrieval Using Flowchart Embedding for Forming Follow-Up Recommendations for Pancreatic Cystic Lesions.
指南 / 共识 · AJR Am J Roentgenol · 2025-07-16

本回顾性研究纳入2023年9月1日至2024年9月1日腹部CT或MRI报告中提及胰腺囊性病变的患者,评估多模态大语言模型(GPT-4o)结合流程图嵌入知识检索形成随访建议的准确性。结果显示,流程图嵌入方法对病变特征评估准确率为98.0-99.0%,对随访建议的准确率在三位放射科医生独立评估下分别为89.9%、91.9%、91.9%,显著高于默认知识和纯文本检索增强生成方法。

要点

  • 流程图嵌入方法使大语言模型随访建议准确率达89.9%-91.9%,显著优于纯文本检索增强生成(23.7%-25.3%)。
  • 大语言模型对胰腺囊性病变特征(存在、大小、位置、主胰管沟通、担忧特征或高风险标志)评估准确率为98.0%-99.0%。
  • 输入既往相关报告后,模型对间期病变变化的评估准确率为96.5%,调整后随访计划准确率为81.2%。

核心结论

  • 多模态流程图嵌入有助于大语言模型自动提供符合临床指导文件的随访建议。

研究设计与样本

  • 回顾性研究,任务1纳入198例患者(平均年龄69.0±13.0岁;女110例,男88例),任务2纳入85例患者(初始平均年龄69.2±10.8岁;女48例,男37例);使用腹部CT或MRI报告,以多模态大语言模型(GPT-4o)为工具,参考标准为美国放射学会偶发发现委员会算法,由三位放射科医生共识评估任务1的病变特征、独立评估随访建议,一位放射科医生评估任务2。

诊断标准与关键数值

  • 任务1:流程图嵌入模型对胰腺囊性病变特征评估准确率为98.0%-99.0%。
  • 任务1:随访建议准确率(放射科医生1)默认知识42.4%、纯文本检索增强生成23.7%、流程图嵌入89.9%(p < .001)。
  • 任务1:随访建议准确率(放射科医生2)默认知识39.9%、纯文本检索增强生成24.2%、流程图嵌入91.9%(p < .001)。
  • 任务1:随访建议准确率(放射科医生3)默认知识40.9%、纯文本检索增强生成25.3%、流程图嵌入91.9%(p < .001)。
  • 任务2:流程图嵌入模型对间期胰腺囊性病变变化评估准确率为96.5%,调整后随访计划准确率为81.2%。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

BACKGROUND. The American College of Radiology (ACR) Incidental Findings Committee (IFC) algorithm provides guidance for pancreatic cystic lesion (PCL) management. Its implementation using plain-text large language model (LLM) solutions is challenging given that key components include multimodal data (e.g., figures and tables). OBJECTIVE. The purpose of the study is to evaluate a multimodal LLM approach incorporating knowledge retrieval using flowchart embedding for forming follow-up recommendations for PCL management. METHODS. This retrospective study included patients who underwent abdominal CT or MRI from September 1, 2023, to September 1, 2024, and whose report mentioned a PCL. The reports' Findings sections were inputted to a multimodal LLM (GPT-4o). For task 1 (198 patients: mean age, 69.0 ± 13.0 [SD] years; 110 women, 88 men), the LLM assessed PCL features (presence of PCL, PCL size and location, presence of main pancreatic duct communication, presence of worrisome features or high-risk stigmata) and formed a follow-up recommendation using three knowledge retrieval methods (default knowledge, plain-text retrieval-augmented generation [RAG] from the ACR IFC algorithm PDF document, and flowchart embedding using the LLM's image-to-text conversion for in-context integration of the document's flowcharts and tables). For task 2 (85 patients: mean initial age, 69.2 ± 10.8 years; 48 women, 37 men), an additional relevant prior report was inputted; the LLM assessed for interval PCL change and provided an adjusted follow-up schedule accounting for prior imaging using flowchart embedding. Three radiologists assessed LLM accuracy in task 1 for PCL findings in consensus and follow-up recommendations independently; one radiologist assessed accuracy in task 2. RESULTS. For task 1, the LLM with flowchart embedding had accuracy for PCL features of 98.0-99.0%. The accuracy of the LLM follow-up recommendations based on default knowledge, plain-text RAG, and flowchart embedding for radiologist 1 was 42.4%, 23.7%, and 89.9% (p < .001), respectively; radiologist 2 was 39.9%, 24.2%, and 91.9% (p < .001); and radiologist 3 was 40.9%, 25.3%, and 91.9% (p < .001). For task 2, the LLM using flowchart embedding showed an accuracy for interval PCL change of 96.5% and for adjusted follow-up schedules of 81.2%. CONCLUSION. Multimodal flowchart embedding aided the LLM's automated provision of follow-up recommendations adherent to a clinical guidance document. CLINICAL IMPACT. The framework could be extended to other incidental findings through the use of other clinical guidance documents as the model input.

来源:AJR Am J Roentgenol,查看原文。版权归原作者所有。

引用

  1. Zhu Z, Liu J, Hong CW, et al. Multimodal Large Language Model With Knowledge Retrieval Using Flowchart Embedding for Forming Follow-Up Recommendations for Pancreatic Cystic Lesions. AJR Am J Roentgenol. 2025. PMID: 40237430. doi:10.2214/AJR.25.32729

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。