FDA监管的人工智能算法:验证研究的趋势、优势与不足
该研究审计了2008年至2021年4月FDA监管的影像人工智能/机器学习算法公开信息,共审阅127款监管软件(其中118款为AI/ML),分类记录了公司、亚专科、解剖部位、成像方式、适应证、目标病理、技术类型及验证数据;结果显示118款中有17款未公布验证声明或数据,仅9款验证数据集超过1000例患者;作者认为公开验证信息不足使临床应用难以论证。
要点
- 118款FDA监管AI/ML算法中17款未公布验证声明或数据,仅9款验证数据集超过1000例患者。
- 最常见技术类型为图像处理/量化(59/118)和分诊(27/118),脑、乳腺和肺为最主要目标区域。
- 公开验证信息不足使算法泛化性和偏倚存在与否无法推断。
核心结论
- 多项FDA监管AI/ML算法公开验证数据集信息不足,使临床应用难以论证,因其泛化性和偏倚存在与否无法推断。
研究设计与样本
- 研究为对2008年至2021年4月FDA监管影像AI/ML算法公开信息的审计,共审阅127款监管软件(其中118款为AI/ML)。
- 记录了母公司、亚专科、身体区域及具体解剖类型、成像方式、FDA许可日期、适应证、目标病理与发现、技术类型(CAD分诊、CAD检测和/或表征、CAD采集或改进、图像处理/量化)、产品性能及临床验证数据的呈现、类型、强度和可及性。
- 对可获得的验证数据记录了纳入患者或研究数、敏感度、特异度、准确度和/或受试者工作特征曲线下面积,以及用例参考标准信息;数据以数据透视表和图表进行描述性统计与趋势分析。
诊断标准与关键数值
- 118款受监管AI/ML算法中17款未公布验证声明或数据。
- 仅9/118款受监管AI/ML算法验证数据集规模超过1000例患者。
- 最常见技术类型为图像处理/量化(n=59/118)和分诊(n=27/118)。
- 脑、乳腺和肺为最主要的目标感兴趣身体区域。
临床应用与阅片要点
- 原文未据此给出推荐。
局限性
- 多项FDA监管AI/ML算法公开验证数据集信息不足,使临床应用难以论证,因其泛化性和偏倚存在与否无法推断。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: To assess key trends, strengths, and gaps in validation studies of the Food and Drug Administration (FDA)-regulated imaging-based artificial intelligence/machine learning (AI/ML) algorithms.
MATERIALS AND METHODS: We audited publicly available details of regulated AI/ML algorithms in imaging from 2008 until April 2021. We reviewed 127 regulated software (118 AI/ML) to classify information related to their parent company, subspecialty, body area and specific anatomy type, imaging modality, date of FDA clearance, indications for use, target pathology (such as trauma) and findings (such as fracture), technique (CAD triage, CAD detection and/or characterization, CAD acquisition or improvement, and image processing/quantification), product performance, presence, type, strength and availability of clinical validation data. Pertaining to validation data, where available, we recorded the number of patients or studies included, sensitivity, specificity, accuracy, and/or receiver operating characteristic area under the curve, along with information on ground-truthing of use-cases. Data were analyzed with pivot tables and charts for descriptive statistics and trends.
RESULTS: We noted an increasing number of FDA-regulated AI/ML from 2008 to 2021. Seventeen (17/118) regulated AI/ML algorithms posted no validation claims or data. Just 9/118 reviewed AI/ML algorithms had a validation dataset sizes of over 1000 patients. The most common type of AI/ML included image processing/quantification (IPQ; n = 59/118), and triage (CADt; n = 27/118). Brain, breast, and lungs dominated the targeted body regions of interest.
CONCLUSION: Insufficient public information on validation datasets in several FDA-regulated AI/ML algorithms makes it difficult to justify clinical applications since their generalizability and presence of bias cannot be inferred.
引用
- Ebrahimian, Kalra, Agarwal, et al. FDA-regulated AI Algorithms: Trends, Strengths, and Gaps of Validation Studies. Acad Radiol. 2021. PMID: 34969610. doi:10.1016/j.acra.2021.09.002
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。