探索大规模公共医学影像数据集
该研究对两个大型公共影像数据集进行视觉审查:ChestXray14 含 112,120 张正位胸片,MURA 含 40,561 张上肢 X 线片,各抽取约 700 张由委员会认证的放射科医生复核原始标签质量。ChestXray14 标签阳性预测值较原始文档低约 10%–30%,存在隐藏分层和标签消歧失败;MURA 标签总体较准确,但在退行性关节病亚组中正常/异常标签的敏感度为 60%、特异度为 82%。
要点
- ChestXray14 标签阳性预测值较原始文档低约 10%–30%,标签不能准确反映图像视觉内容。
- ChestXray14 存在隐藏分层和标签消歧失败等显著问题。
- MURA 正常/异常标签在退行性关节病亚组中敏感度 60%、特异度 82%。
核心结论
- 视觉审查图像是理解大型影像数据集的必要环节,数据集生产团队应执行该质量控制程序并在文档中详述发现、数据生成流程与标注规则。
研究设计与样本
- 研究对两个大型公共数据集进行视觉探索,以确定所提供标签的准确程度及是否存在其他细微问题。
- ChestXray14 数据集包含 112,120 张正位胸片,MURA 数据集包含 40,561 张上肢 X 线片。
- 从两个数据集中各抽取约 700 张图像,由一名委员会认证的放射科医生复核,判定原始标签质量。
诊断标准与关键数值
- ChestXray14 标签的阳性预测值大多比原始文档中给出的值低 10%–30%。
- MURA 正常/异常标签在退行性关节病病例亚组中敏感度为 60%、特异度为 82%。
临床应用与阅片要点
- 作者建议生产公共数据集的团队执行视觉审查这一质量控制程序,并在数据集文档中纳入发现的详细描述以及数据生成流程和标注规则的说明。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: Medical artificial intelligence systems are dependent on well characterized large-scale datasets. Recently released public datasets have been of great interest to the field, but pose specific challenges due to the disconnect they cause between data generation and data usage, potentially limiting the utility of these datasets.
MATERIALS AND METHODS: We visually explore two large public datasets, to determine how accurate the provided labels are and whether other subtle problems exist. The ChestXray14 dataset contains 112,120 frontal chest films, and the Musculoskeletal Radiology (MURA) dataset contains 40,561 upper limb radiographs. A subset of around 700 images from both datasets was reviewed by a board-certified radiologist, and the quality of the original labels was determined.
RESULTS: The ChestXray14 labels did not accurately reflect the visual content of the images, with positive predictive values mostly between 10% and 30% lower than the values presented in the original documentation. There were other significant problems, with examples of hidden stratification and label disambiguation failure. The MURA labels were more accurate, but the original normal/abnormal labels were inaccurate for the subset of cases with degenerative joint disease, with a sensitivity of 60% and a specificity of 82%.
CONCLUSION: Visual inspection of images is a necessary component of understanding large image datasets. We recommend that teams producing public datasets should perform this important quality control procedure and include a thorough description of their findings, along with an explanation of the data generating procedures and labeling rules, in the documentation for their datasets.
引用
- Oakden-Rayner L. Exploring Large-scale Public Medical Image Datasets. Acad Radiol. 2019. PMID: 31706792. doi:10.1016/j.acra.2019.10.006
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。