糖尿病视网膜病变机器学习模型评估中的分级者变异性与参考标准的重要性
该回顾性分析利用裁决共识作为参考标准,量化了基于个体分级者和多数决策的糖尿病视网膜病变分级误差,并训练了改进的自动分级算法。算法在中等或更严重DR的AUC达0.986,性能与美国委员会认证眼科医生和视网膜专科医生个体相当。
要点
- 视网膜专科医生裁决与眼科医生多数决策间193处差异中,最常见为漏诊微动脉瘤(36%)、伪影(20%)和出血误分类(16%)。
- 算法在中等或更严重DR的敏感度为0.971、特异度为0.923、AUC为0.986。
- 使用少量裁决共识分级作为调优数据集和高分辨率图像输入,算法在中等或更严重DR的AUC从0.934提升至0.986。
核心结论
- 裁决可减少糖尿病视网膜病变分级中的误差。
- 少量裁决的DR分级可显著改善算法性能。
- 所得算法性能与美国委员会认证眼科医生和视网膜专科医生个体相当。
研究设计与样本
- 回顾性分析,纳入来自DR筛查项目的视网膜眼底图像。
- 图像分别由算法、美国委员会认证眼科医生和视网膜专科医生分级,以视网膜专科医生裁决共识为参考标准。
主要发现
- 视网膜专科医生裁决与眼科医生多数决策间193处差异中,漏诊微动脉瘤占36%、伪影占20%、出血误分类占16%。
- 个体视网膜专科医生、眼科医生和算法的kappa范围分别为0.82至0.91、0.80至0.84和0.84。
- 对于中等或更严重DR,眼科医生多数决策的敏感度为0.838、特异度为0.981。
- 算法在中等或更严重DR的敏感度为0.971、特异度为0.923、AUC为0.986。
- 算法在轻度或更严重DR的敏感度为0.970、特异度为0.917、AUC为0.986。
关键数值与推荐
- 使用二次加权kappa评分评估分级者间及分级者与算法间的一致性。
- 使用AUC、敏感度和特异度比较不同形式人工分级和算法在不同DR严重程度截断值下的性能。
- 以少量裁决共识分级作为调优数据集和高分辨率图像作为输入,算法在中等或更严重DR的AUC从0.934提升至0.986。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
PURPOSE: Use adjudication to quantify errors in diabetic retinopathy (DR) grading based on individual graders and majority decision, and to train an improved automated algorithm for DR grading. DESIGN: Retrospective analysis. PARTICIPANTS: Retinal fundus images from DR screening programs. METHODS: Images were each graded by the algorithm, U.S. board-certified ophthalmologists, and retinal specialists. The adjudicated consensus of the retinal specialists served as the reference standard. MAIN OUTCOME MEASURES: For agreement between different graders as well as between the graders and the algorithm, we measured the (quadratic-weighted) kappa score. To compare the performance of different forms of manual grading and the algorithm for various DR severity cutoffs (e.g., mild or worse DR, moderate or worse DR), we measured area under the curve (AUC), sensitivity, and specificity. RESULTS: Of the 193 discrepancies between adjudication by retinal specialists and majority decision of ophthalmologists, the most common were missing microaneurysm (MAs) (36%), artifacts (20%), and misclassified hemorrhages (16%). Relative to the reference standard, the kappa for individual retinal specialists, ophthalmologists, and algorithm ranged from 0.82 to 0.91, 0.80 to 0.84, and 0.84, respectively. For moderate or worse DR, the majority decision of ophthalmologists had a sensitivity of 0.838 and specificity of 0.981. The algorithm had a sensitivity of 0.971, specificity of 0.923, and AUC of 0.986. For mild or worse DR, the algorithm had a sensitivity of 0.970, specificity of 0.917, and AUC of 0.986. By using a small number of adjudicated consensus grades as a tuning dataset and higher-resolution images as input, the algorithm improved in AUC from 0.934 to 0.986 for moderate or worse DR. CONCLUSIONS: Adjudication reduces the errors in DR grading. A small set of adjudicated DR grades allows substantial improvements in algorithm performance. The resulting algorithm's performance was on par with that of individual U.S. Board-Certified ophthalmologists and retinal specialists.
引用
- Krause J, Gulshan V, Rahimy E, et al. Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology. 2018. PMID: 29548646. doi:10.1016/j.ophtha.2018.01.034
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。