文献速递泌尿生殖与妇科肿瘤放疗放疗物理与技术

大语言模型在前列腺癌放疗试验毒性提取中的真实世界测评

Large language models for toxicity extraction in oncology trials: A real-world benchmark in prostate radiotherapy.
PubMed 文献 · Radiother Oncol · 2025-12-17

本研究以前列腺癌放疗前瞻性试验(NCT02874014,55例患者、8968条毒性记录)为数据源,用规则增强的少样本提示策略测评5款现成大语言模型从临床记录中提取CTCAE分级胃肠与泌尿生殖毒性的表现与成本。各模型二分类准确率84.6–87.4%、分级准确率79.1–82.3%,GPT-4o二分类与分级准确率最高,Gemini 2.5 Pro敏感度最高(74.0%),GPT-4o特异度最高(96.0%);全量提取成本从0.77美元到超过21美元不等。结论为现成LLM可低成本的接近人类评分者间信度水平提取临床相关毒性,但分级准确率仍有限。

要点

  • GPT-4o二分类准确率87.4%、分级准确率83.5%,为所测模型中最高。
  • 各模型Cohen kappa提示一致性仅为中等(二分类0.552–0.560,分级0.401–0.465)。
  • 成本差异显著:Gemini 2.0 Flash全量提取0.77美元,Gemini 2.5 Pro与GPT-5超过21美元。

核心结论

  • 现成LLM可提取临床相关毒性且性能接近人类评分者间信度,成本可变但常可忽略,分级准确率仍有限。

研究设计与样本

  • 基于前列腺癌放疗前瞻性试验(NCT02874014;55例患者、8968条毒性记录)的真实世界基准测评。
  • 采用规则增强的少样本提示策略,测评Gemini 2.0 Flash、Gemini 2.5 Flash、Gemini 2.5 Pro、GPT-4o、GPT-5共5款现成LLM。
  • 评估指标包括二分类与分级准确率、精确率、召回率、特异度、F1、Cohen kappa及计算成本。

主要发现

  • 全部模型二分类准确率84.6–87.4%,分级准确率79.1–82.3%。
  • GPT-4o二分类准确率87.4%、分级准确率83.5%,均为最高。
  • Gemini 2.5 Pro敏感度最高(74.0%),GPT-4o特异度最高(96.0%)。
  • Cohen kappa为二分类0.552–0.560、分级0.401–0.465,提示中等一致性。
  • 全量提取成本Gemini 2.0 Flash为0.77美元,Gemini 2.5 Pro与GPT-5超过21美元。

关键数值与推荐

  • 二分类准确率84.6–87.4%,分级准确率79.1–82.3%(5款现成LLM,规则增强少样本提示)。
  • GPT-4o:二分类准确率87.4%、分级准确率83.5%、特异度96.0%。
  • Gemini 2.5 Pro敏感度74.0%。
  • Cohen kappa:二分类0.552–0.560,分级0.401–0.465。
  • 成本:Gemini 2.0 Flash全量提取0.77美元;Gemini 2.5 Pro与GPT-5均超过21美元。

临床怎么用

  • 可将LLM作为毒性监测与临床研究数据抽象的可扩展、低成本辅助工具嵌入肿瘤学工作流。
  • 关注模型间成本差异,按准确率与预算选择模型,如兼顾成本可选Gemini 2.0 Flash。
  • 分级准确率有限,LLM输出仍需人工核对,不宜单独替代CTCAE人工评级。

局限性

  • 摘要未报告具体研究局限性内容。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Accurate toxicity assessment is critical in oncology trials, yet current reporting frameworks such as the Common Terminology Criteria for Adverse Events (CTCAE) remain labor-intensive and subject to inter-observer variability. Large language models (LLMs) offer potential to automate extraction and grading of adverse events from clinical notes and patient-reported outcomes (PROs), but their comparative performance and cost-effectiveness remain underexplored. METHODS: We evaluated five off-the-shelf LLMs (Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-4o, and GPT-5) using a rule-augmented few-shot prompting strategy to extract CTCAE-graded gastrointestinal and genitourinary toxicities from a prospective prostate radiotherapy trial (NCT02874014; n = 55 patients, 8968 toxicity records). Binary and grade-level accuracy, precision, recall, specificity, F1 score, Cohen's kappa, and computational costs were assessed. RESULTS: All models achieved high binary accuracy (84.6-87.4 %) and moderate grade accuracy (79.1-82.3 %). GPT-4o reached the best binary (87.4 %) and grade (83.5 %) accuracy, while Gemini 2.5 Pro demonstrated highest sensitivity (74.0 %). Specificity peaked with GPT-4o (96.0 %). Cohen's kappa values indicated moderate agreement (0.552-0.560 for binary; 0.401-0.465 for grades). Costs for the entire extraction varied substantially: Gemini 2.0 Flash delivered competitive accuracy at $0.77 total, whereas Gemini 2.5 Pro and GPT-5 exceeded $21. CONCLUSIONS: Off-the-shelf LLMs can extract clinically relevant toxicities with performance approaching human inter-rater reliability, at variable but often negligible costs. While grade-level accuracy remains limited, LLM integration into oncology workflows is feasible, offering scalable, low-cost support for toxicity monitoring and data abstraction in clinical research.

来源:Radiother Oncol,查看原文。版权归原作者所有。

引用

  1. Mastroleo F, Borras-Osorio M, Patel SP, et al. Large language models for toxicity extraction in oncology trials: A real-world benchmark in prostate radiotherapy. Radiother Oncol. 2025. PMID: 41419026. doi:10.1016/j.radonc.2025.111348

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。