文献速递儿科影像骨肌DR

深度学习模型在临床骨龄判定中的偏差与误差

The uncovered biases and errors in clinical determination of bone age by using deep learning models.
PubMed 文献 · Eur Radiol · 2022-12-20

AI整体平均绝对误差为0.42年,但与医生临床判定(正常、提前、延迟)的一致性Kappa仅0.714,存在显著差异。热图特征与人类视觉并非完全一致,数据偏差(性别、年龄、机构、阅片者)可能导致模型性能变化及判定分歧。

要点

  • AI骨龄模型整体平均绝对误差0.42年,但个体临床判定与医生一致性仅Kappa 0.714。
  • AI热图关注区域与放射科医生在X线片上的视觉判断不完全一致,需警惕模型依据。
  • 性别、年龄、机构及阅片者等数据偏差可导致AI骨龄模型性能变化和判定分歧。

关键影像表现/征象

  • CHNm-CHNt上平均绝对误差(MAD)为0.42年。
  • CHNm-CHNt的Grad-CAM热图特征与人类在X线片上的视觉判断不完全一致。
  • 不同AI模型骨龄估计性能存在变化,AI与放射科医生临床判定存在分歧。

诊断标准与关键数值

  • 采用Brush数据将骨龄临床判定分为正常、提前和延迟三组。
  • AI与放射科医生临床判定的一致性Kappa值为0.714,卡方检验显示差异有统计学意义。
  • CHNm-CHNt上MAD值为0.42年。

临床应用与阅片要点

  • AI骨龄整体误差虽可接受,但个体临床判定(正常/提前/延迟)与医生一致性有限,报告时不宜直接采信AI分级。
  • 需警惕性别、年龄、机构及阅片者等偏差对AI骨龄模型性能的影响,尤其跨机构应用时。
  • AI热图关注区域与人类视觉不完全一致,阅片时应结合医生自身对X线征象的判读,不可仅依赖热图。

局限性

  • 测试数据CHNt仅来自单一数据集,未在外部多中心数据上验证临床判定一致性。
  • 原文未报告AI模型的具体架构、训练样本量及设备/序列信息。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

OBJECTIVES: To evaluate AI biases and errors in estimating bone age (BA) by comparing AI and radiologists' clinical determinations of BA.

METHODS: We established three deep learning models from a Chinese private dataset (CHNm), an American public dataset (USAm), and a joint dataset combining the above two datasets (JOIm). The test data CHNt (n = 1246) were labeled by ten senior pediatric radiologists. The effects of data site differences, interpretation bias, and interobserver variability on BA assessment were evaluated. The differences between the AI models' and radiologists' clinical determinations of BA (normal, advanced, and delayed BA groups by using the Brush data) were evaluated by the chi-square test and Kappa values. The heatmaps of CHNm-CHNt were generated by using Grad-CAM.

RESULTS: We obtained an MAD value of 0.42 years on CHNm-CHNt; this result indicated an appropriate accuracy for the whole group but did not indicate an accurate estimation of individual BA because with a kappa value of 0.714, the agreement between AI and human clinical determinations of BA was significantly different. The features of the heatmaps were not fully consistent with the human vision on the X-ray films. Variable performance in BA estimation by different AI models and the disagreement between AI and radiologists' clinical determinations of BA may be caused by data biases, including patients' sex and age, institutions, and radiologists.

CONCLUSIONS: The deep learning models outperform external validation in predicting BA on both internal and joint datasets. However, the biases and errors in the models' clinical determinations of child development should be carefully considered.

KEY POINTS: • With a kappa value of 0.714, clinical determinations of bone age by using AI did not accord well with clinical determinations by radiologists. • Several biases, including patients' sex and age, institutions, and radiologists, may cause variable performance by AI bone age models and disagreement between AI and radiologists' clinical determinations of bone age. • AI heatmaps of bone age were not fully consistent with human vision on X-ray films.

来源:Eur Radiol,查看原文。版权归原作者所有。

引用

  1. Bai, Gao, Ji, et al. The uncovered biases and errors in clinical determination of bone age by using deep learning models. Eur Radiol. 2022. PMID: 36538072. doi:10.1007/s00330-022-09330-0

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。