文献速递胸部DR

专家判定的参考标准对深度学习模型诊断性能评估的影响:一项胸部X线片恶性肺结节检测任务

Effects of Expert-Determined Reference Standards in Evaluating the Diagnostic Performance of a Deep Learning Model: A Malignant Lung Nodule Detection Task on Chest Radiographs.
PubMed 文献 · Korean J Radiol · 2023-01-01

本研究纳入50例经病理证实肺癌患者和50例对照的胸部X线片,构建5种专家判定标准并与临床金标准(病理确认)比较。潜在类别分析模型与临床金标准最相似(敏感度72.6%,特异度100%);使用专家判定标准会高估放射科医生和DLAD模型的敏感度、低估特异度,DLAD辅助可减轻敏感度高估但加重特异度低估。

要点

  • 潜在类别分析模型与病理金标准最相似,敏感度72.6%、特异度100%。
  • 采用专家判定标准时,放射科医生和DLAD模型敏感度被高估、特异度被低估。
  • DLAD辅助可减轻敏感度高估,但加重特异度低估。

核心结论

  • 潜在类别分析模型与病理金标准在胸部X线片恶性肺结节检测中最相似;专家判定标准会偏倚人工智能模型诊断性能的测量。

研究设计与样本

  • 研究纳入50例经病理证实肺癌患者和50例对照的胸部X线片,由10位专家构建5种专家判定标准,另10位放射科医生独立判读并在DLAD模型辅助下判读,以临床金标准和各专家判定标准评估诊断性能。

诊断标准与关键数值

  • 潜在类别分析模型:敏感度72.6%,特异度100%,与临床金标准最相似。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVE: Little is known about the effects of using different expert-determined reference standards when evaluating the performance of deep learning-based automatic detection (DLAD) models and their added value to radiologists. We assessed the concordance of expert-determined standards with a clinical gold standard (herein, pathological confirmation) and the effects of different expert-determined reference standards on the estimates of radiologists' diagnostic performance to detect malignant pulmonary nodules on chest radiographs with and without the assistance of a DLAD model.

MATERIALS AND METHODS: This study included chest radiographs from 50 patients with pathologically proven lung cancer and 50 controls. Five expert-determined standards were constructed using the interpretations of 10 experts: individual judgment by the most experienced expert, majority vote, consensus judgments of two and three experts, and a latent class analysis (LCA) model. In separate reader tests, additional 10 radiologists independently interpreted the radiographs and then assisted with the DLAD model. Their diagnostic performance was estimated using the clinical gold standard and various expert-determined standards as the reference standard, and the results were compared using the t test with Bonferroni correction.

RESULTS: The LCA model (sensitivity, 72.6%; specificity, 100%) was most similar to the clinical gold standard. When expert-determined standards were used, the sensitivities of radiologists and DLAD model alone were overestimated, and their specificities were underestimated (all p-values < 0.05). DLAD assistance diminished the overestimation of sensitivity but exaggerated the underestimation of specificity (all p-values < 0.001). The DLAD model improved sensitivity and specificity to a greater extent when using the clinical gold standard than when using the expert-determined standards (all p-values < 0.001), except for sensitivity with the LCA model (p = 0.094).

CONCLUSION: The LCA model was most similar to the clinical gold standard for malignant pulmonary nodule detection on chest radiographs. Expert-determined standards caused bias in measuring the diagnostic performance of the artificial intelligence model.

来源:Korean J Radiol,查看原文。版权归原作者所有。

引用

  1. Huh JE, Lee JH, Hwang EJ, et al. Effects of Expert-Determined Reference Standards in Evaluating the Diagnostic Performance of a Deep Learning Model: A Malignant Lung Nodule Detection Task on Chest Radiographs. Korean J Radiol. 2023. PMID: 36725356. doi:10.3348/kjr.2022.0548

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。