文献速递创伤骨科脊柱外科关节外科

评估已发表骨科研究中基于人工智能的写作辅助:检测及未来解读趋势

Evaluating Artificial Intelligence-Based Writing Assistance Among Published Orthopaedic Studies: Detection and Trends for Future Interpretation.
PubMed 文献 · J Bone Joint Surg Am · 2025-05-30

该研究以2000年发表的300篇骨科文稿建立基线,用ZeroGPT分析其人工智能生成内容,并分析ChatGPT发布后连续发表的3374篇骨科文稿;基线平均AI检测百分比为10.84%±11.02%,以32.875%为显著AI参与阈值,ChatGPT后文稿中16.7%超过该阈值。

要点

  • 以2000年300篇文稿建立基线,平均AI检测百分比为10.84%±11.02%。
  • ChatGPT后3374篇文稿中16.7%超过32.875%的显著AI参与阈值。
  • 各期刊显著AI参与率从5.6%到38.3%不等,差异有统计学意义(p<0.001)。

核心结论

  • 该研究首次提供了解读AI检测百分比的循证阈值,发现近期发表的骨科文献中16.7%存在显著AI参与。
  • 作者认为该发现凸显了明确指南、伦理标准、负责任使用AI及改进检测工具对维护骨科研究质量、真实性和诚信的重要性。

研究设计与样本

  • 以2000年发表的300篇文稿建立AI检测基线,并用ZeroGPT分析其AI生成内容。
  • 随后分析ChatGPT发布后连续发表的3374篇骨科文稿,计算95%置信区间以设定显著AI参与阈值。

主要发现

  • 2000年300篇前AI时代文稿的平均AI检测百分比(及标准差)为10.84%±11.02%。
  • ChatGPT后3374篇文稿中,16.7%超过32.875%的AI检测阈值(前AI时代基线的2个标准差以上),提示显著AI参与。
  • 原创文稿与综述研究中显著AI参与百分比分别为16.4%和18.2%,两者无显著差异(p=0.40)。
  • 显著AI参与率在各期刊间差异显著,从《The American Journal of Sports Medicine》的5.6%到《The Journal of Bone & Joint Surgery》的38.3%(p<0.001)。

关键数值与推荐

  • AI检测百分比阈值设为32.875%,即前AI时代基线的2个标准差以上,超过该阈值视为内容生成中有显著AI参与。
  • 前AI时代基线平均AI检测百分比为10.84%±11.02%(300篇2000年发表的文稿)。
  • ChatGPT后文稿中16.7%超过32.875%阈值(3374篇中)。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: The integration of artificial intelligence (AI), particularly large language models (LLMs), into scientific writing has led to questions about its ethics, prevalence, and impact in orthopaedic literature. While tools have been developed to detect AI-generated content, the interpretation of AI detection percentages and their clinical relevance remain unclear. The aim of this study was to quantify AI involvement in published orthopaedic manuscripts and to establish a statistical threshold for interpreting AI detection percentages. METHODS: To establish a baseline, 300 manuscripts published in the year 2000 were analyzed for AI-generated content with use of ZeroGPT. This was followed by an analysis of 3,374 consecutive orthopaedic manuscripts published after the release of ChatGPT. A 95% confidence interval was calculated in order to set a threshold for significant AI involvement. Manuscripts with AI detection percentages above this threshold (32.875%) were considered to have significant AI involvement in their content generation. RESULTS: Empirical analysis of the 300 pre-AI-era manuscripts revealed a mean AI detection percentage (and standard deviation [SD]) of 10.84% ± 11.02%. Among the 3,374 post-AI-era manuscripts analyzed, 16.7% exceeded the AI detection threshold of 32.875% (2 SDs above the baseline for the pre-AI era), indicating significant AI involvement. No significant difference was found between primary manuscripts and review studies (percentage with significant AI involvement, 16.4% and 18.2%, respectively; p = 0.40). Significant AI involvement varied significantly across journals, with rates ranging from 5.6% in The American Journal of Sports Medicine to 38.3% in The Journal of Bone & Joint Surgery (p < 0.001). CONCLUSIONS: This study examined AI assistance in the writing of published orthopaedic manuscripts and provides the first evidence-based threshold for interpreting AI detection percentages. Our results revealed significant AI involvement in 16.7% of recently published orthopaedic literature. This finding highlights the importance of clear guidelines, ethical standards, responsible AI use, and improved detection tools to maintain the quality, authenticity, and integrity of orthopaedic research.

来源:J Bone Joint Surg Am,查看原文。版权归原作者所有。

引用

  1. Callanan T, Marquez J, Pisani C, et al. Evaluating Artificial Intelligence-Based Writing Assistance Among Published Orthopaedic Studies: Detection and Trends for Future Interpretation. J Bone Joint Surg Am. 2025. PMID: 40446076. doi:10.2106/JBJS.24.01462

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。