文献速递胸部技术与安全DR

人工智能系统在症状性患者胸部X线片上检测COVID-19的多中心验证

Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients.
PubMed 文献 · Eur Radiol · 2022-07-02

研究开发并验证了基于胸部X线片(CXR)检测COVID-19的深度学习模型RadGenX,在9个研究站点(美国、意大利、香港特别行政区)及2个公共数据集共5,894例症状性患者中独立测试AUC为0.79(95%CI 0.78-0.80)。模型在地区、疾病严重程度、性别和年龄间性能存在统计学差异,相比放射科医生敏感度更高但特异度更低。

要点

  • RadGenX在5,894例独立测试队列中AUC为0.79(95%CI 0.78-0.80),4折交叉验证AUC为0.89。
  • 模型性能在不同地区(p < 0.01)、疾病严重程度(p < 0.001)、性别(p < 0.001)和年龄(p = 0.03)间存在统计学差异。
  • 患病率模拟显示阴性预测值从33.3%患病率时的86.1%升至任何低于4.5%患病率时的98.5%以上。

核心结论

  • RadGenX模型可在症状性患者CXR上预测COVID-19感染,并在大型国际队列中验证,为AI系统在CXR上进行COVID-19预测的测试和性能预期提供了有价值的信息。

研究设计与样本

  • 深度学习模型(RadGenX)在168,850张CXR上训练,在来自9个研究站点(美国、意大利、香港特别行政区)及2个公共数据集(美国和欧洲)的大型国际测试集上验证;性能以受试者工作特征曲线下面积(AUC)衡量,并进行自举模拟评估3.33%至33.3%的COVID-19患病率范围下的性能;在852例独立测试集上与放射科医生比较。

诊断标准与关键数值

  • RadGenX在4折交叉验证中AUC为0.89。
  • RadGenX在5,894例患者独立测试队列中AUC为0.79(95%CI 0.78-0.80)。
  • 患病率模拟显示阴性预测值从33.3%患病率时的86.1%升至任何低于4.5%患病率时的98.5%以上。
  • 与放射科医生比较,McNemar检验显示模型敏感度更高(p < 0.001)但特异度更低(p < 0.001)。

局限性

  • 模型性能在不同地区(p < 0.01)、疾病严重程度(p < 0.001)、性别(p < 0.001)和年龄(p = 0.03)间存在统计学差异。
  • 与放射科医生相比,模型敏感度更高(p < 0.001)但特异度更低(p < 0.001)。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVES: While chest radiograph (CXR) is the first-line imaging investigation in patients with respiratory symptoms, differentiating COVID-19 from other respiratory infections on CXR remains challenging. We developed and validated an AI system for COVID-19 detection on presenting CXR.

METHODS: A deep learning model (RadGenX), trained on 168,850 CXRs, was validated on a large international test set of presenting CXRs of symptomatic patients from 9 study sites (US, Italy, and Hong Kong SAR) and 2 public datasets from the US and Europe. Performance was measured by area under the receiver operator characteristic curve (AUC). Bootstrapped simulations were performed to assess performance across a range of potential COVID-19 disease prevalence values (3.33 to 33.3%). Comparison against international radiologists was performed on an independent test set of 852 cases.

RESULTS: RadGenX achieved an AUC of 0.89 on 4-fold cross-validation and an AUC of 0.79 (95%CI 0.78-0.80) on an independent test cohort of 5,894 patients. Delong's test showed statistical differences in model performance across patients from different regions (p < 0.01), disease severity (p < 0.001), gender (p < 0.001), and age (p = 0.03). Prevalence simulations showed the negative predictive value increases from 86.1% at 33.3% prevalence, to greater than 98.5% at any prevalence below 4.5%. Compared with radiologists, McNemar's test showed the model has higher sensitivity (p < 0.001) but lower specificity (p < 0.001).

CONCLUSION: An AI model that predicts COVID-19 infection on CXR in symptomatic patients was validated on a large international cohort providing valuable context on testing and performance expectations for AI systems that perform COVID-19 prediction on CXR.

KEY POINTS: • An AI model developed using CXRs to detect COVID-19 was validated in a large multi-center cohort of 5,894 patients from 9 prospectively recruited sites and 2 public datasets. • Differences in AI model performance were seen across region, disease severity, gender, and age. • Prevalence simulations on the international test set demonstrate the model's NPV is greater than 98.5% at any prevalence below 4.5%.

来源:Eur Radiol,查看原文。版权归原作者所有。

引用

  1. Kuo, Chiu, Wang, et al. Multi-center validation of an artificial intelligence system for detection of COVID-19 on chest radiographs in symptomatic patients. Eur Radiol. 2022. PMID: 35779089. doi:10.1007/s00330-022-08969-z

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。