大型语言模型可利用叙述性手术记录准确填充血管质量倡议手术数据库
单中心回顾性研究分析2021至2023年颈动脉内膜切除术、血管内动脉瘤修复术和腹股沟下下肢旁路术各50份手术记录,使用ChatGPT机构版(gpt-35-turbo和gpt-4)自动填充血管质量倡议(VQI)数据库。gpt-35-turbo填充各术式字段的中位准确率为84.0%–92.2%,排除手术记录中不常记录的字段后为93.2%–95.5%;gpt-4未较gpt-35显著提升性能,150份记录处理成本分别为0.12和3.39美元。
要点
- gpt-35-turbo填充VQI字段的中位准确率:CEA 84.0%、EVAR 92.2%、LEB 84.3%。
- 排除手术记录中不常记录的字段后,中位准确率升至93.2%–95.5%。
- gpt-4较gpt-35未显著提升性能,150份记录成本分别为3.39与0.12美元。
核心结论
- 大型语言模型可利用叙述性手术记录准确填充VQI手术数据库的结构化和非结构化数据,且处理成本较低。
- 工作流程效率的提升可能改善中心成功参与VQI的能力。
- 作者认为需要进一步研究其他VQI数据库及提高准确性的方法。
研究设计与样本
- 单中心回顾性研究,使用2021至2023年机构VQI数据。
- 分析颈动脉内膜切除术(CEA)、血管内动脉瘤修复术(EVAR)和腹股沟下下肢旁路术(LEB)各50份手术记录,共150份。
- 使用符合HIPAA的机构版ChatGPT(Versa),采用gpt-35-turbo和gpt-4两个模型,通过云端编程接口自动分析手术记录并生成可共享的VQI文件。
- 将模型输出与VQI数据比较以评估准确性;若某指标在少于20%的手术记录中被外科医生讨论,则定义为对模型「不可用」。
主要发现
- gpt-35-turbo在所有字段的中位准确率:CEA为84.0%(四分位距80.0%–88.0%),EVAR为92.2%(四分位距87.2%–94.0%),LEB为84.3%(四分位距80.2%–88.1%)。
- CEA、EVAR和LEB的VQI数据库分别包含25、179和51个指标;其中分别有3/25、6/179和7/51个变量在手术记录中不可用。
- 排除手术记录中常规不可用的指标信息后,中位准确率:CEA为95.5%(四分位距90.9%–100.0%),EVAR为94.8%(四分位距92.2%–98.5%),LEB为93.2%(四分位距90.2%–96.4%)。
- gpt-4较gpt-35未显著改善整体性能(CEA P=0.97,EVAR P=0.85,LEB P=0.95)。
- 150份手术记录的分析成本:gpt-35-turbo为0.12美元,gpt-4为3.39美元。
临床怎么用
- 原文未据此给出临床推荐。
局限性
- 原文结论指出需要进一步研究其他VQI数据库及提高准确性的方法。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: Participation in the Vascular Quality Initiative (VQI) provides important resources to surgeons, but the ability to do so is often limited by time and data entry personnel. Large language models (LLMs) such as ChatGPT (OpenAI) are examples of generative artificial intelligence products that may help bridge this gap. Trained on large volumes of data, the models are used for natural language processing and text generation. We evaluated the ability of LLMs to accurately populate VQI procedural databases using operative reports. METHODS: A single-center, retrospective study was performed using institutional VQI data from 2021 to 2023. The most recent procedures for carotid endarterectomy (CEA), endovascular aneurysm repair (EVAR), and infrainguinal lower extremity bypass (LEB) were analyzed using Versa, a HIPAA (Health Insurance Portability and Accountability Act)-compliant institutional version of ChatGPT. We created an automated function to analyze operative reports and generate a shareable VQI file using two models: gpt-35-turbo and gpt-4. Application of the LLMs was accomplished with a cloud-based programming interface. The outputs of this model were compared with VQI data for accuracy. We defined a metric as "unavailable" to the LLM if it was discussed by surgeons in <20% of operative reports. RESULTS: A total of 150 operative notes were analyzed, including 50 CEA, 50 EVAR, and 50 LEB. These procedural VQI databases included 25, 179, and 51 metrics, respectively. For all fields, gpt-35-turbo had a median accuracy of 84.0% for CEA (interquartile range [IQR]: 80.0%-88.0%), 92.2% for EVAR (IQR: 87.2%-94.0%), and 84.3% for LEB (IQR: 80.2%-88.1%). A total of 3 of 25, 6 of 179, and 7 of 51 VQI variables were unavailable in the operative reports, respectively. Excluding metric information routinely unavailable in operative reports, the median accuracy rate was 95.5% for each CEA procedure (IQR: 90.9%-100.0%), 94.8% for EVAR (IQR: 92.2%-98.5%), and 93.2% for LEB (IQR: 90.2%-96.4%). Across procedures, gpt-4 did not meaningfully improve performance compared with gpt-35 (P = .97, .85, and .95 for CEA, EVAR, and LEB overall performance, respectively). The cost for 150 operative reports analyzed with gpt-35-turbo and gpt-4 was $0.12 and $3.39, respectively. CONCLUSIONS: LLMs can accurately populate VQI procedural databases with both structured and unstructured data, while incurring only minor processing costs. Increased workflow efficiency may improve center ability to successfully participate in the VQI. Further work examining other VQI databases and methods to increase accuracy is needed.
引用
- Flanagan CP, Trang K, Nacario J, et al. Large language models can accurately populate Vascular Quality Initiative procedural databases using narrative operative reports. J Vasc Surg. 2024. PMID: 39694151. doi:10.1016/j.jvs.2024.12.002
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。