文献速递技术与安全

影像医学大语言模型研究的报告质量:高分期刊横断面审计

Reporting Quality of Large Language Model Studies: A Cross-Sectional Audit of High-Ranking Radiology and Medical Imaging Journals.
PubMed 文献 · Korean J Radiol · 2026-08-04

该研究对2025年发表于放射学与医学影像领域Q1期刊的102项大语言模型(LLM)原创研究进行横断面审计,按MI-CLEAR-LLM 2025版评分。总体依从性中等(均值51.2%±14.7%,范围22.2%-84.2%),提示词执行设置与随机性管理报告最差,各期刊间差异显著(P=0.011),KJR均值最高(72.8%±2.7%)。

要点

  • LLM研究报告总体依从MI-CLEAR-LLM仅约一半,复现关键项缺失明显。
  • 训练数据截止日期(9.8%)与提示词措辞依据(15.6%)报告率最低。
  • 提示词执行设置(29.4%)与随机性管理(33.1%)是报告薄弱环节。

核心结论

  • 2025年放射与医学影像LLM研究对MI-CLEAR-LLM的依从性在条目和期刊间不一致,部分复现关键要素存在明显缺陷,需更广泛采用报告规范。

研究设计与样本

  • 共识别201项符合条件研究,按期刊发文量配额抽样后最终分析102项,并额外纳入KJR全部4篇符合条件文章作为基准。
  • 依从性按MI-CLEAR-LLM 2025更新版评分,采用两轮共识流程:一名评审初评,二次评审复核,必要时由另一名评审共识裁决。
  • 期刊间差异用Kruskal-Wallis检验,随后Dunn事后两两比较并以Holm校正P值。

诊断标准与关键数值

  • 总体依从MI-CLEAR-LLM均值51.2%±14.7%,范围22.2%-84.2%。
  • 依从性最高条目为输入数据类型(100%)、测试数据独立性(80.2%)与适配策略(78.1%)。
  • 依从性最低条目为提示词执行设置(29.4%)与随机性管理(33.1%);报告最少的是训练数据截止日期(9.8%)与提示词措辞依据(15.6%)。
  • 期刊间依从性差异显著(P=0.011),KJR均值最高(72.8%±2.7%)。

临床应用与阅片要点

  • 阅读或撰写LLM影像研究时,重点核查提示词执行设置、随机性管理与训练数据截止日期是否报告。
  • 评价LLM诊断准确性研究可复现性时,可参照MI-CLEAR-LLM条目清单逐项核对。
  • 注意到不同期刊报告质量差异显著,跨期刊比较LLM研究结果时需谨慎。

局限性

  • 采用配额抽样选取约100项研究,非全部201项纳入,可能存在选择偏倚。
  • 依从性评分依赖评审主观判断,虽经两轮共识仍可能存在评分差异。
  • 仅纳入Q1期刊,结论外推至其他期刊层级受限。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVE: To evaluate adherence to the Minimum Reporting Items for Clear Evaluation of Accuracy Reports of Large Language Models in Healthcare (MI-CLEAR-LLM) in radiology and medical imaging studies involving large language models (LLMs).

MATERIALS AND METHODS: We conducted a cross-sectional audit of original LLM research studies published between January 1 and December 26, 2025, in Q1 journals within the Web of Science "Radiology, Nuclear Medicine, and Medical Imaging" category. PubMed and Scopus were searched to identify eligible studies. A quota-based subsampling strategy, based on journal publication volume, was used to select approximately 100 studies. All four eligible articles from the Korean Journal of Radiology (KJR) were additionally included as a benchmark. Adherence to the 2025 update of MI-CLEAR-LLM was scored through a two-round, consensus-based process: an initial assessment by one reviewer followed by a critical re-evaluation by secondary reviewers, with consensus adjudication by an additional reviewer when needed. Between-journal differences were analyzed with the Kruskal-Wallis test, followed by Dunn post hoc pairwise comparisons with Holm-adjusted P-values.

RESULTS: Of 201 eligible studies identified, 102 were finally analyzed after applying the subsampling strategy. Overall adherence to MI-CLEAR-LLM was moderate (mean, 51.2% ± 14.7%; range, 22.2%-84.2%). Adherence was highest for input data type (100%), test-data independence (80.2%), and adaptation strategy (78.1%), and lowest for prompt execution setup (29.4%) and stochasticity management (33.1%). The least frequently reported items were training-data cutoff date (9.8%) and rationale for prompt wording (15.6%). Adherence varied significantly across journals (P = 0.011), with KJR showing the highest mean adherence (72.8% ± 2.7%).

CONCLUSION: Reporting transparency in radiology and medical imaging LLM studies published in 2025 was inconsistent across reporting items and journals, with substantial deficiencies in some reproducibility-critical elements. Broader adoption of reporting standards is essential to improve the reproducibility and interpretability of future accuracy evaluations.

来源:Korean J Radiol,查看原文。版权归原作者所有。

引用

  1. Mese I, Gunes ST, Coskun O, et al. Reporting Quality of Large Language Model Studies: A Cross-Sectional Audit of High-Ranking Radiology and Medical Imaging Journals. Korean J Radiol. 2026. PMID: 42551868. doi:10.3348/kjr.2026.0505

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。