大语言模型从膝关节MRI报告提取软骨分级的多阅片者基准研究
本研究提出可复现的批量大语言模型框架,从100例膝关节MRI报告中提取软骨损伤结构化数据,按改良Outerbridge分级评估。5名放射科医生与7种LLM配置在20个关节面解剖模板上独立分级,以四分加权κ与人类参照组比较。三个旗舰Azure模型一致性不劣于人类阅片者间变异,成本优化版和开源模型显著低于人类参照。
要点
- GPT-5.4、GPT-5.3、GPT-5.1-mini表面级κ与人类间一致性无统计学差异,可用于辅助结构化报告
- 软骨分级应从报告文本提取,不直接分析MRI图像,避免过度依赖影像征象解读
- 成本优化型与开源模型GPT-4.1、GPT-5.4-mini、Qwen2.5-32B、GPT-5.4-nano一致性显著低于人类,不宜单独用于临床
核心结论
- 三个旗舰OpenAI部署GPT-5.4、GPT-5.3、GPT-5.1-mini在表面级软骨分级上达到与人类阅片者间变异无差异的一致性
研究设计与样本
- IRB批准的回顾性研究,随机抽取100份非增强膝关节MRI报告,时期为2019年1月至2025年1月
- 共有66479例合格检查,入组100例,由5名阅片者独立分级
- 5名阅片者包括4名经专科培训的肌骨放射科医生(培训后经验6至21年)和1名四年级住院医师
- 7种LLM配置包括6个Azure OpenAI部署(GPT-4.1、GPT-5.1-mini、GPT-5.3、GPT-5.4、GPT-5.4-mini、GPT-5.4-nano)和1个本地部署开源模型Qwen2.5-32B-Instruct
- 所有分级在固定的20个关节面解剖模板上进行,采用成对Cohen四分加权κ
诊断标准与关键数值
- 表面级人类间平均κ为0.80(95%CI 0.78至0.83),腔室级为0.94(95%CI 0.93至0.95)
- GPT-5.4表面级κ=0.83,GPT-5.3 κ=0.80,GPT-5.1-mini κ=0.77,三者与人类参照组差值95%CI包含零
- 一致性使用改良Outerbridge分级作为诊断标准,在20个关节面解剖模板上评估
临床应用与阅片要点
- 三个旗舰Azure模型可用于从膝关节MRI报告文本中批量提取结构化软骨损伤分级数据
- 表面级评估比腔室级更有区分度,适用于精细化软骨损伤分级比较
- 不建议使用成本优化型或开源模型进行临床软骨分级提取
- 该框架可用于批量化、可复现的影像报告结构化后处理
局限性
- 研究为单中心回顾性设计
- 仅纳入非增强膝关节MRI报告,未涉及其他关节或增强扫描
- 样本量为100例,基于成对κ比较的统计学效能有限
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVE: To describe a reproducible framework for bulk large language model (LLM)-based extraction of structured cartilage-lesion data from knee MRI reports and to benchmark seven LLM configurations against multiple radiologists using the modified Outerbridge classification.
MATERIALS AND METHODS: In this IRB-approved retrospective study, 100 non-contrast knee MRI reports (January 2019 to January 2025) were randomly selected from 66,479 eligible examinations and independently graded by five readers (four fellowship-trained musculoskeletal radiologists with 6-21 years of post-fellowship experience and one fourth-year resident) and seven LLM configurations, comprising six Azure OpenAI deployments (GPT-4.1, GPT-5.1-mini, GPT-5.3, GPT-5.4, GPT-5.4-mini, GPT-5.4-nano) and one locally hosted open-weight model (Qwen2.5-32B-Instruct), across a fixed 20-surface anatomic taxonomy. Pairwise Cohen quadratic-weighted κ was computed at compartment and surface levels. Bootstrap 95% confidence intervals (CIs) and p-values on the paired difference between each rater's mean κ and the human-human reference were derived by case-level cluster resampling (B = 1000).
RESULTS: Mean human-human κ was 0.94 (95% CI 0.93, 0.95) at the compartment level and 0.80 (95% CI 0.78, 0.83) at the surface level. Three Azure deployments-GPT-5.4 (surface κ = 0.83), GPT-5.3 (κ = 0.80), and GPT-5.1-mini (κ = 0.77)-had 95% CIs on the difference from the human reference that included zero. Four models had CIs entirely below zero, indicating agreement significantly below the human reference: GPT-4.1, GPT-5.4-mini, Qwen2.5-32B, and GPT-5.4-nano (all p ≤ .02).
CONCLUSION: Three flagship OpenAI deployments achieved cartilage grading agreement indistinguishable from human inter-rater variability, while cost-optimized and open-weight variants performed measurably below.
引用
- Zhao, Roux, Wieschhoff, et al. Multi-reader, multi-model benchmark of large language models for modified Outerbridge cartilage grading from knee MRI reports. Skeletal Radiol. 2026. PMID: 42611194. doi:10.1007/s00256-026-05342-9
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。