COVID-19患者肺超声评分的人与人及人与人工智能评分者间一致性研究
该研究纳入59例COVID-19患者的1035个肺超声视频,由6名人工操作者评分并用人工智能算法分析,评估评分者间一致性。结果显示评估至少500个视频时一致性稳定,视频层面人工配对Fleiss kappa为0.464,人工智能与人工比较为0.404;预后层面分别为0.505和0.506,人工智能算法结果与人工操作者相当。
要点
- 评估至少500个肺超声视频时评分者间一致性稳定。
- 视频层面人工配对Fleiss kappa为0.464,人工智能与人工比较为0.404。
- 预后层面人工配对Fleiss kappa为0.505,人工智能与人工比较为0.506。
核心结论
- 为评估评分者间一致性并获得可靠评价,建议至少分析500个视频;所用人工智能算法取得与人工操作者相当的结果。
研究设计与样本
- 从1807个视频数据集中随机选取59例COVID-19患者的1035个肺超声视频,由6名人工操作者评分,并用人工智能算法分析,计算Fleiss kappa系数,在视频层面和预后层面评估。
关键影像表现/征象
- 肺超声评估基于垂直伪影和实变的检测与分析。
诊断标准与关键数值
- 视频层面:人工操作者两两之间Fleiss kappa为0.464(95%CI 0.455-0.473),人工智能与人工操作者之间为0.404(95%CI 0.396-0.412)。
- 预后层面:人工操作者两两之间Fleiss kappa为0.505(95%CI 0.448-0.562),人工智能与人工操作者之间为0.506(95%CI 0.458-0.555)。
- 评估至少500个视频时获得稳定的一致性。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVES: Lung ultrasound (LUS) has sparked significant interest during COVID-19. LUS is based on the detection and analysis of imaging patterns. Vertical artifacts and consolidations are some of the recognized patterns in COVID-19. However, the interrater reliability (IRR) of these findings has not been yet thoroughly investigated. The goal of this study is to assess IRR in LUS COVID-19 data and determine how many LUS videos and operators are required to obtain a reliable result.
METHODS: A total of 1035 LUS videos from 59 COVID-19 patients were included. Videos were randomly selected from a dataset of 1807 videos and scored by six human operators (HOs). The videos were also analyzed by artificial intelligence (AI) algorithms. Fleiss' kappa coefficient results are presented, evaluated at both the video and prognostic levels.
RESULTS: Findings show a stable agreement when evaluating a minimum of 500 videos. The statistical analysis illustrates that, at a video level, a Fleiss' kappa coefficient of 0.464 (95% confidence interval [CI] = 0.455-0.473) and 0.404 (95% CI = 0.396-0.412) is obtained for pairs of HOs and for AI versus HOs, respectively. At prognostic level, a Fleiss' kappa coefficient of 0.505 (95% CI = 0.448-0.562) and 0.506 (95% CI = 0.458-0.555) is obtained for pairs of HOs and for AI versus HOs, respectively.
CONCLUSIONS: To examine IRR and obtain a reliable evaluation, a minimum of 500 videos are recommended. Moreover, the employed AI algorithms achieve results that are comparable with HOs. This research further provides a methodology that can be useful to benchmark future LUS studies.
引用
- Fatima, Mento, Zanforlin, et al. Human-to-AI Interrater Agreement for Lung Ultrasound Scoring in COVID-19 Patients. J Ultrasound Med. 2022. PMID: 35796343. doi:10.1002/jum.16052
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。