指南与共识运动医学

人工智能无法替代同行评审但可辅助编辑初筛:运动医学杂志大语言模型与人类评审建议的比较分析

Artificial Intelligence Cannot Replace Peer Reviewers but May Help Editors Triage: A Comparative Analysis of a Large Language Model and Human Reviewer Recommendations at the American Journal of Sports Medicine.
指南 / 共识 · Am J Sports Med · 2026-07-12

该横断面一致性研究随机选取《美国运动医学杂志》2024年9月至10月投稿的54篇稿件,比较本地部署的大语言模型与最终编辑决定的一致性。人类评审汇总与最终决定呈fair一致(κ=0.181),AI呈slight且不显著一致(κ=0.126)。AI推荐拒稿或降级时90.5%获相同最终决定,提示其或可作为探索性初筛工具。

要点

  • 人类评审汇总与最终编辑决定一致性fair(κ=0.181),AI一致性slight且不显著(κ=0.126)。
  • AI推荐拒稿或降级(n=21)时90.5%获相同最终决定,PPV 90.5%(95%CI 71.1%-97.3%)。
  • AI对61.1%稿件推荐修回,其中72.7%最终被拒或降级,存在系统性的「修回偏倚」。

核心结论

  • AI无法复现高影响力运动医学期刊人类同行评审的细致判断。
  • AI推荐拒稿或降级时90.5%稿件获相同最终决定(描述性PPV 90.5%,95%CI 71.1%-97.3%),提示或可作为探索性初筛工具,但需更大样本验证。
  • AI无法可靠区分应直接拒稿与应降级至姊妹刊的稿件,这是编辑初筛应用的重要局限。

研究设计与样本

  • 横断面一致性研究,随机选取2024年9月至10月投稿至《美国运动医学杂志》的54篇稿件。
  • 使用本地部署的大语言模型(Ministral 3 14B;Mistral AI)以标准化提示词评审,产出分类建议(拒稿、降级、修回、接收)和0-100分数。
  • 以Cohen kappa(4分类模型)评估与最终编辑决定的一致性,汇总人类评审n=139条,AI n=54。
  • 以阳性预测值(PPV)、敏感度和特异度评估筛选性能。

主要发现

  • AI对61.1%稿件推荐修回,其中72.7%最终被拒或降级,显示系统性「修回偏倚」。
  • AI推荐拒稿时,54.5%最终拒稿、27.3%降级;AI推荐降级时,50%拒稿、50%降级。
  • AI推荐拒稿或降级(n=21)时,90.5%获最终拒稿或降级(PPV 90.5%;特异度81.8%)。

临床怎么用

  • AI不宜替代人类同行评审,但可在编辑初筛中作为辅助工具,优先关注AI推荐拒稿或降级及评分<70的稿件。
  • AI无法区分直接拒稿与降级至姊妹刊,编辑初筛决策仍需人工判断。

局限性

  • 样本量较小(54篇稿件),AI推荐拒稿或降级的PPV置信区间较宽(95%CI 71.1%-97.3%)。
  • AI无法可靠区分直接拒稿与降级至姊妹刊,限制其在编辑初筛中的应用。
  • 结论需在更大队列中进一步验证。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: The peer review system faces increasing strain from rising manuscript volumes, reviewer fatigue, and well-documented interreviewer disagreement. Large language models (LLMs) have shown potential to support the peer review process, but their ability to replicate editorial decisions at high-impact medical journals and their utility as manuscript screening tools remain unknown. PURPOSE: To compare the agreement between an LLM and the final editorial decision on manuscripts submitted to the American Journal of Sports Medicine and to evaluate the potential of LLMs as a manuscript screening tool. STUDY DESIGN: Cross-sectional agreement study. METHODS: Fifty-four manuscripts randomly selected from submissions to the American Journal of Sports Medicine (September 2024-October 2024) were reviewed by a locally deployed LLM (Ministral 3 14B; Mistral AI) using a standardized prompt. The artificial intelligence (AI) produced a categorical recommendation (reject, cascade, revision, or accept) and a numerical score (0-100) for each manuscript. Agreement with the final editorial decision was assessed by Cohen kappa (4-category model) for pooled human reviewers (n = 139 reviews) and the AI (n = 54). Screening performance was evaluated by positive predictive value (PPV), sensitivity, and specificity. RESULTS: Pooled human reviewers demonstrated fair agreement with the final decision (κ = 0.181 [P < .001]; 42.4% agreement), while the AI demonstrated slight, nonsignificant agreement (κ = 0.126 [P = .099]; 37.0% agreement). The AI recommended revision for 61.1% of manuscripts, of which 72.7% were ultimately rejected or cascaded, demonstrating systematic "revision bias." When the AI recommended rejection, 54.5% of those manuscripts were ultimately rejected and 27.3% were cascaded; when the AI recommended cascade, 50% were rejected and 50% were cascaded. However, when the AI recommended rejection or cascade (n = 21), 90.5% received a final decision of rejection or cascade (PPV, 90.5%; specificity, 81.8%). Manuscripts with an AI score <70 were rejected or cascaded 88.0% of the time (PPV, 88.0%). CONCLUSION: AI cannot replicate the nuanced judgment of human peer reviewers at a high-impact sports medicine journal. When AI recommended rejection or cascade, 90.5% of manuscripts received that final decision (descriptive PPV, 90.5%; 95% CI, 71.1%-97.3%), suggesting potential utility as an exploratory first-pass screening tool warranting further validation in larger cohorts. However, AI could not reliably distinguish manuscripts destined for outright rejection from those that would be cascaded to a sister journal-an important limitation for editorial triage applications.

来源:Am J Sports Med,查看原文。版权归原作者所有。

引用

  1. Patel R, Shultz C, Ollivier M, et al. Artificial Intelligence Cannot Replace Peer Reviewers but May Help Editors Triage: A Comparative Analysis of a Large Language Model and Human Reviewer Recommendations at the American Journal of Sports Medicine. Am J Sports Med. 2026. PMID: 42438216. doi:10.1177/03635465261463006

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。