基准评估基础大语言模型按美国医学物理学家协会第263号任务组报告重新标记结构名称的能力
研究将GPT-4部署于DICOM服务器,按AAPM TG-263报告对前列腺、头颈和胸部三个病种的结构名称进行重标记,每病种随机选150例调优提示词、50例评估。每例准确率分别为97.2%、98.3%和97.1%,临床靶区重标记正确率分别为100%、95.3%和92.9%。作者认为大语言模型有望成为放射肿瘤学结构名称标准化的重要方法。
要点
- GPT-4按AAPM TG-263重标记结构名称,每例准确率前列腺97.2%、头颈98.3%、胸部97.1%。
- 临床靶区重标记正确率前列腺100%、头颈95.3%、胸部92.9%。
- 每病种150例用于分批调优提示词,50例用于评估。
核心结论
- 鉴于GPT-4重标记结构名称的准确率,大语言模型有望成为放射肿瘤学结构名称标准化的重要方法,尤其考虑到大语言模型能力的快速进步可能持续。
研究设计与样本
- 将生成式预训练变换器(GPT)-4部署在医学数字成像和通信(DICOM)服务器中,服务器接收结构集DICOM文件后提示GPT-4按美国医学物理学家协会第263号任务组(AAPM TG-263)报告重标记结构名称。
- 评估前列腺、头颈和胸部3个病种,每个病种随机选150例患者分批(每批50例)手动调优指令提示词,另随机选50例患者用于评估。
- 考虑的结构名称为最可能对使用多名患者结构轮廓的研究相关的名称。
主要发现
- 每例准确率前列腺为97.2%、头颈为98.3%、胸部为97.1%。
- 按结构计算,临床靶区重标记正确率前列腺为100%、头颈为95.3%、胸部为92.9%。
关键数值与推荐
- 每例准确率前列腺97.2%、头颈98.3%、胸部97.1%。
- 临床靶区重标记正确率前列腺100%、头颈95.3%、胸部92.9%。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
PURPOSE: To introduce the concept of using large language models (LLMs) to relabel structure names in accordance with the American Association of Physicists in Medicine Task Group-263 standard and to establish a benchmark for future studies to reference. METHODS AND MATERIALS: Generative Pretrained Transformer (GPT)-4 was implemented within a Digital Imaging and Communications in Medicine server. Upon receiving a structure-set Digital Imaging and Communications in Medicine file, the server prompts GPT-4 to relabel the structure names according to the American Association of Physicists in Medicine Task Group-263 report. The results were evaluated for 3 disease sites: prostate, head and neck, and thorax. For each disease site, 150 patients were randomly selected for manually tuning the instructions prompt (in batches of 50), and 50 patients were randomly selected for evaluation. Structure names considered were those that were most likely to be relevant for studies using structure contours for many patients. RESULTS: The per-patient accuracy was 97.2%, 98.3%, and 97.1% for prostate, head and neck, and thorax disease sites, respectively. On a per-structure basis, the clinical target volume was relabeled correctly in 100%, 95.3%, and 92.9% of cases, respectively. CONCLUSIONS: Given the accuracy of GPT-4 in relabeling structure names as presented in this work, LLMs are poised to become an important method for standardizing structure names in radiation oncology, especially considering the rapid advancements in LLM capabilities that are likely to continue.
引用
- Holmes J, Zhang L, Ding Y, et al. Benchmarking a Foundation Large Language Model on its Ability to Relabel Structure Names in Accordance With the American Association of Physicists in Medicine Task Group-263 Report. Pract Radiat Oncol. 2024. PMID: 39243241. doi:10.1016/j.prro.2024.04.017
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。