文献速递技术与安全CTMRUSDR

人工智能研究的透明度:放射学与核医学开放科学可用性项目的系统综述

Transparency in Artificial Intelligence Research: a Systematic Review of Availability Items Related to Open Science in Radiology and Nuclear Medicine.
PubMed 文献 · Acad Radiol · 2022-12-14

该研究系统综述了放射学与核医学领域AI研究的透明度与开放科学实践,随机抽样纳入194篇Q1/Q2期刊原创论文,评估6项基线特征和5项可用性项目。结果显示原始数据仅约五分之一论文公开(34/194;18%),私有数据仅1篇公开(1/161;1%),预建模、建模和后建模文件公开率分别为13%、14%和8%,97%论文未尝试创建可直接使用的系统,总体可用性差。

要点

  • 仅18%论文公开原始数据,私有数据公开率仅1%,建模文件公开率14%
  • 97%的AI研究未提供可直接用于真实世界的系统,转化落地严重不足
  • 使用私有数据与至少一项可用性项目缺失显著负相关(p<0.001)

核心结论

  • 放射学与核医学AI论文的开放科学可用性项目总体公开率很低,有较大提升空间。

研究设计与样本

  • 系统综述,在PubMed检索AI原创研究,限Q1/Q2且Web of Science收录期刊,随机抽样纳入194篇论文。
  • 由两组共8名独立读者评估6项基线特征和5项可用性项目,分析评分者间一致性,分歧经共识解决。

诊断标准与关键数值

  • 原始数据可用率18%(34/194),私有数据可用率1%(1/161)。
  • 预建模文件可用率13%(25/194),建模文件可用率14%(28/194),后建模文件可用率8%(15/194)。
  • 97%(189/194)的论文未尝试创建可直接用于真实世界的系统。
  • 单独使用私有数据与至少一项可用性项目缺失负相关(p<0.001)。

临床应用与阅片要点

  • 阅读AI研究时需关注数据、代码和模型是否公开,以评估可重复性。
  • 报告AI研究时应尽量公开数据、代码和模型文件,提升透明度和可重复性。

局限性

  • 仅纳入Q1/Q2且Web of Science收录期刊的论文,可能低估整体可用性水平。
  • 采用随机抽样而非全样本,结果存在抽样误差。
  • 仅评估5项可用性项目,未覆盖所有开放科学维度。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: Reproducibility of artificial intelligence (AI) research has become a growing concern. One of the fundamental reasons is the lack of transparency in data, code, and model. In this work, we aimed to systematically review the radiology and nuclear medicine papers on AI in terms of transparency and open science.

MATERIALS AND METHODS: A systematic literature search was performed in PubMed to identify original research studies on AI. The search was restricted to studies published in Q1 and Q2 journals that are also indexed on the Web of Science. A random sampling of the literature was performed. Besides six baseline study characteristics, a total of five availability items were evaluated. Two groups of independent readers including eight readers participated in the study. Inter-rater agreement was analyzed. Disagreements were resolved with consensus.

RESULTS: Following eligibility criteria, we included a final set of 194 papers. The raw data was available in about one-fifth of the papers (34/194; 18%). However, the authors made their private data available only in one paper (1/161; 1%). About one-tenth of the papers made their pre-modeling (25/194; 13%), modeling (28/194; 14%), or post-modeling files (15/194; 8%) available. Most of the papers (189/194; 97%) did not attempt to create a ready-to-use system for real-world usage. Data origin, use of deep learning, and external validation had statistically significantly different distributions. The use of private data alone was negatively associated with the availability of at least one item (p<0.001).

CONCLUSION: Overall rates of availability for items were poor, leaving room for substantial improvement.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Kocak, Yardimci, Yuzkan, et al. Transparency in Artificial Intelligence Research: a Systematic Review of Availability Items Related to Open Science in Radiology and Nuclear Medicine. Acad Radiol. 2022. PMID: 36526532. doi:10.1016/j.acra.2022.11.030

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。