Auto-METRICS:大语言模型辅助的影像组学研究科学质量控制
本研究评估大语言模型(LLM)能否用METRICS工具可靠评估影像组学研究的方法学质量,将商业云模型Gemini Flash 2.0对46篇文章的评分与放射科医生比较,结果显示人机一致性(平均κ=0.48)与人际一致性(平均κ=0.48)相当,开源隐私保护模型Phi4-Reasoning表现与Gemini Flash 2.0相当。
要点
- LLM与放射科医生之间的METRICS评分一致性(平均κ=0.48)与放射科医生彼此之间的一致性(平均κ=0.48)相当。
- 开源隐私保护模型Phi4-Reasoning与商业云模型Gemini Flash 2.0的评分表现相当(中位排名1 vs 3)。
- LLM可辅助标准化影像组学质量评估,开源隐私保护模型或可支持人类评分者评估研究完整性。
核心结论
- LLM可协助进行标准化的影像组学质量评估,开源隐私保护模型能提供与商业云LLM相当的性能,提示其可用于支持人类评分者评估影像组学研究完整性。
研究设计与样本
- 方法学质量评估研究,比较商业云LLM(Gemini Flash 2.0)与放射科医生对46篇文章的METRICS评估,使用ADA2025(6组放射科医生)和K2025(3名放射科医生)两项可重复性研究,并比较24个隐私保护开源LLM与Gemini Flash 2.0。
诊断标准与关键数值
- ADA2025中商业LLM与人评分者间平均κ=0.48,人-人间平均κ=0.48,Wilcoxon秩和检验p=0.41;平均METRICS Pearson相关PC=0.62 vs 0.56,p=0.11。
- K2025中平均人-LLM κ=0.58,人-人κ=0.57,Wilcoxon秩和检验p=0.28;PC=0.68 vs 0.51,p=0.55。
- Phi4-Reasoning与Gemini Flash 2.0在所有评分者中的中位排名分别为1 vs 3。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
PURPOSE: The quality of radiomics research is critical for reliable clinical translation, yet methodological flaws remain prevalent. This study evaluates whether large language models (LLMs) can reliably assess radiomics methodological quality using the METhodological RadiomICs Score (METRICS).
METHODS: We compared a commercial cloud-based LLM (Gemini Flash 2.0) METRICS assessments for 46 articles with those of radiologists using two reproducibility studies (ADA2025 and K2025, with 6 radiologist groups and 3 radiologists, respectively, with varying degrees of experience). Cohen's kappa (κ) and METRICS Pearson's correlation (PC), and error rates between LLMs and human raters were evaluated. Prompt clarifications to METRICS were suggested to improve human-LLM agreement. Twenty four privacy-preserving open LLMs were compared with Gemini Flash 2.0.
RESULTS: In ADA2025, the commercial LLM achieved inter-rater agreements with human raters comparable to those between human raters (average κ = 0.48 vs. average κ = 0.48, respectively, Wilcoxon rank-sum test p = 0.41), leading to similar correlation values in METRICS scoring (average PC = 0.62 vs. average PC = 0.56, Wilcoxon rank-sum test p = 0.11). This was confirmed with K2025 (mean human-LLM κ = 0.58 vs. human-human κ = 0.57, Wilcoxon rank-sum test p = 0.28), with no evidence for correlation differences (PC = 0.68 vs. PC = 0.51, respectively, Wilcoxon rank-sum test p = 0.55). Phi4-Reasoning, an open model which can be run locally, performed comparably to Gemini Flash 2.0 (median ranking = 1 vs. median ranking = 3, respectively, across all raters).
CONCLUSION: LLMs can assist in standardized radiomics quality assessment. Open privacy-preserving models can offer comparable performance to commercial cloud-based LLMs, suggesting their utility in supporting human raters for evaluating radiomics research integrity.
引用
- de Almeida JG, Papanikolaou N. Auto-METRICS: LLM-assisted scientific quality control for radiomics research. Eur J Radiol. 2025. PMID: 40812087. doi:10.1016/j.ejrad.2025.112358
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。