大语言模型在前列腺癌放疗试验毒性提取中的真实世界测评
本研究以前列腺癌放疗前瞻性试验(NCT02874014,55例患者、8968条毒性记录)为数据源,用规则增强的少样本提示策略测评5款现成大语言模型从临床记录中提取CTCAE分级胃肠与泌尿生殖毒性的表现与成本。各模型二分类准确率84.6–87.4%、分级准确率79.1–82.3%,GPT-4o二分类与分级准确率最高,Gemini 2.5 Pro敏感度最高(74.0%),GPT-4o特异度最高(96.0%);全量提取成本从0.77美元到超过21美元不等。结论为现成LLM可低成本的接近人类评分者间信度水平提取临床相关毒性,但分级准确率仍有限。
要点
- GPT-4o二分类准确率87.4%、分级准确率83.5%,为所测模型中最高。
- 各模型Cohen kappa提示一致性仅为中等(二分类0.552–0.560,分级0.401–0.465)。
- 成本差异显著:Gemini 2.0 Flash全量提取0.77美元,Gemini 2.5 Pro与GPT-5超过21美元。
核心结论
- 现成LLM可提取临床相关毒性且性能接近人类评分者间信度,成本可变但常可忽略,分级准确率仍有限。
研究设计与样本
- 基于前列腺癌放疗前瞻性试验(NCT02874014;55例患者、8968条毒性记录)的真实世界基准测评。
- 采用规则增强的少样本提示策略,测评Gemini 2.0 Flash、Gemini 2.5 Flash、Gemini 2.5 Pro、GPT-4o、GPT-5共5款现成LLM。
- 评估指标包括二分类与分级准确率、精确率、召回率、特异度、F1、Cohen kappa及计算成本。
主要发现
- 全部模型二分类准确率84.6–87.4%,分级准确率79.1–82.3%。
- GPT-4o二分类准确率87.4%、分级准确率83.5%,均为最高。
- Gemini 2.5 Pro敏感度最高(74.0%),GPT-4o特异度最高(96.0%)。
- Cohen kappa为二分类0.552–0.560、分级0.401–0.465,提示中等一致性。
- 全量提取成本Gemini 2.0 Flash为0.77美元,Gemini 2.5 Pro与GPT-5超过21美元。
关键数值与推荐
- 二分类准确率84.6–87.4%,分级准确率79.1–82.3%(5款现成LLM,规则增强少样本提示)。
- GPT-4o:二分类准确率87.4%、分级准确率83.5%、特异度96.0%。
- Gemini 2.5 Pro敏感度74.0%。
- Cohen kappa:二分类0.552–0.560,分级0.401–0.465。
- 成本:Gemini 2.0 Flash全量提取0.77美元;Gemini 2.5 Pro与GPT-5均超过21美元。
临床怎么用
- 可将LLM作为毒性监测与临床研究数据抽象的可扩展、低成本辅助工具嵌入肿瘤学工作流。
- 关注模型间成本差异,按准确率与预算选择模型,如兼顾成本可选Gemini 2.0 Flash。
- 分级准确率有限,LLM输出仍需人工核对,不宜单独替代CTCAE人工评级。
局限性
- 摘要未报告具体研究局限性内容。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Accurate toxicity assessment is critical in oncology trials, yet current reporting frameworks such as the Common Terminology Criteria for Adverse Events (CTCAE) remain labor-intensive and subject to inter-observer variability. Large language models (LLMs) offer potential to automate extraction and grading of adverse events from clinical notes and patient-reported outcomes (PROs), but their comparative performance and cost-effectiveness remain underexplored. METHODS: We evaluated five off-the-shelf LLMs (Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-4o, and GPT-5) using a rule-augmented few-shot prompting strategy to extract CTCAE-graded gastrointestinal and genitourinary toxicities from a prospective prostate radiotherapy trial (NCT02874014; n = 55 patients, 8968 toxicity records). Binary and grade-level accuracy, precision, recall, specificity, F1 score, Cohen's kappa, and computational costs were assessed. RESULTS: All models achieved high binary accuracy (84.6-87.4 %) and moderate grade accuracy (79.1-82.3 %). GPT-4o reached the best binary (87.4 %) and grade (83.5 %) accuracy, while Gemini 2.5 Pro demonstrated highest sensitivity (74.0 %). Specificity peaked with GPT-4o (96.0 %). Cohen's kappa values indicated moderate agreement (0.552-0.560 for binary; 0.401-0.465 for grades). Costs for the entire extraction varied substantially: Gemini 2.0 Flash delivered competitive accuracy at $0.77 total, whereas Gemini 2.5 Pro and GPT-5 exceeded $21. CONCLUSIONS: Off-the-shelf LLMs can extract clinically relevant toxicities with performance approaching human inter-rater reliability, at variable but often negligible costs. While grade-level accuracy remains limited, LLM integration into oncology workflows is feasible, offering scalable, low-cost support for toxicity monitoring and data abstraction in clinical research.
引用
- Mastroleo F, Borras-Osorio M, Patel SP, et al. Large language models for toxicity extraction in oncology trials: A real-world benchmark in prostate radiotherapy. Radiother Oncol. 2025. PMID: 41419026. doi:10.1016/j.radonc.2025.111348
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。