使用大语言模型评估放射组学研究的方法学质量:METRICS-E3框架的附加价值
这项元研究用48篇开放获取放射组学文章比较GPT-5在不同提示流程下按METRICS评估方法学质量的表现,并与GPT-4o及人类共识对照。提供METRICS-E3后GPT-5与人类评分的一致性提高,Kendall τ由0.474升至0.626,组内相关系数由0.539升至0.793。作者认为该辅助评估可在专家监督下作为可扩展的预筛或审计辅助工具,但尚需进一步验证。
要点
- GPT-5 E3与人类共识的Kendall τ为0.626,高于GPT-5的0.474。
- GPT-5 E3的组内相关系数为0.793,高于GPT-5的0.539。
- GPT-5 E3在25/30个条目和4/5个条件上改善一致性。
核心结论
- 向GPT-5提供METRICS-E3可提高其使用METRICS框架评估放射组学研究方法学质量时与人类共识的一致性及可靠性。
- 大语言模型辅助、经METRICS和METRICS-E3引导的评估可在专家监督下作为可扩展的预筛或审计辅助工具。
- 作者指出尚需跨数据集、读者和模型进一步验证。
研究设计与样本
- 研究类型为元研究,纳入48篇开放获取放射组学文章,来源于既往研究。
- 采用两种GPT-5流程:仅使用原始METRICS工具的基线单提示评估(GPT-5),以及在基线METRICS提示基础上补充METRICS-E3的两步评估(GPT-5 E3)。
- 每篇文章在每个流程下运行三次并采用多数投票。
- 对照来源为既往报告的GPT-4o结果和已发表队列的参考人类共识。
诊断标准与关键数值
- METRICS评分中位数(IQR):GPT-4o为79.50%(13.25%),人类共识为68.95%(22.58%),GPT-5为66.20%(17.85%),GPT-5 E3为64.55%(16.80%)。
- 与人类评分的一致性:Kendall τ由GPT-5的0.474提高至GPT-5 E3的0.626。
- 组内相关系数由GPT-5的0.539提高至GPT-5 E3的0.793。
- 条目层面,GPT-5 E3在25/30(83.3%)个条目和4/5(80%)个条件上改善一致性。
- 多数两两比较差异显著(如人类 vs GPT-4o、人类 vs GPT-5 E3,均p < 0.001),但人类 vs GPT-5无显著差异(p = 1.000)。
临床应用与阅片要点
- 作者认为,大语言模型辅助、经METRICS和METRICS-E3引导的评估可在专家监督下作为可扩展的预筛或审计辅助工具。
局限性
- 作者指出尚需跨数据集、读者和模型进一步验证。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVES: To determine whether providing the METRICS-E3 (Explanation and Elaboration with Examples) resource improves the ability of a large language model (LLM) to appraise methodological quality in radiomics research using the METhodological RadiomICs Score (METRICS).
METHODS: In this meta-research study, 48 open-access radiomics articles from previous research were assessed with METRICS under two GPT-5 pipelines: (i) a baseline single-prompt evaluation using only the original METRICS tool (GPT-5) and (ii) a two-step evaluation with the baseline METRICS prompt supplemented by METRICS-E3 (GPT-5 E3). Each article was run three times per pipeline with majority voting. For comparison, we sourced previously reported GPT-4o results and the reference human consensus from the published cohort.
RESULTS: Median METRICS scores (IQR) were 79.50 % (13.25 %) for GPT-4o, 68.95 % (22.58 %) for human consensus, 66.20 % (17.85 %) for GPT-5, and 64.55 % (16.80 %) for GPT-5 E3. Pairwise differences were significant for most comparisons (e.g., human vs GPT-4o and human vs GPT-5 E3, both p < 0.001) but not for human vs GPT-5 (p = 1.000). Concordance with human ratings improved with METRICS-E3: Kendall's τ increased from 0.474 (GPT-5) to 0.626 (GPT-5 E3). Agreement measured by intra-class correlation coefficient similarly increased from 0.539 to 0.793. Item-wise, GPT-5 E3 improved agreement on 25/30 (83.3 %) items and 4/5 (80 %) conditions.
CONCLUSION: Supplying METRICS-E3 to GPT-5 increased alignment with human consensus and improved reliability in the task of assessing methodological quality of radiomics studies using the METRICS framework. LLM-assisted, METRICS and METRICS-E3-guided appraisal may serve as a scalable pre-screening or auditing aid under expert supervision. Further validation across datasets, readers, and models is warranted.
引用
- Mese I, Kocak B. Evaluating methodological quality in radiomics research using large language models: Added value of METRICS-E3 framework. Eur J Radiol. 2025. PMID: 41273921. doi:10.1016/j.ejrad.2025.112519
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。