通过基于深度学习的自然语言处理评估放射学报告中的确定性
从学术医学中心随机抽取594份头部MR报告,3名放射科医师对2352句印象部分句子按4类确定性标注,开发并验证基于BERT的自然语言处理系统,BioBERT在验证集宏观平均AUC为0.931,在留出测试集敏感度77.29%、特异度92.89%、AUC 0.93。
要点
- 3名放射科医师对不确定性解读的标注者间一致性κ=0.74。
- 验证集上BioBERT宏观平均AUC 0.931,BERT-base 0.928,ClinicalBERT 0.925。
- 留出测试集BioBERT宏观平均敏感度77.29%、特异度92.89%、AUC 0.93。
核心结论
- 基于深度迁移学习的模型可可靠评估放射学报告中传达的不确定性水平。
研究设计与样本
- 从学术医学中心随机抽取594份头部MR报告,3名委员会认证放射科医师对印象部分2352句句子按4类确定性标注,开发并验证基于BERT的自然语言处理系统,评估3种BERT变体模型。
诊断标准与关键数值
- 确定性分为4类:非确定性、确定性-轻度、确定性-强烈、其他;3名放射科医师间κ=0.74;验证集BioBERT宏观平均AUC 0.931、BERT-base 0.928、ClinicalBERT 0.925;3种模型宏观平均特异度93.13%-93.65%;BERT-base宏观平均敏感度79.46%、BioBERT 79.08%、ClinicalBERT 78.52%;留出测试集BioBERT宏观平均敏感度77.29%、特异度92.89%、AUC 0.93。
临床应用与阅片要点
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
BACKGROUND AND PURPOSE: Communication gaps exist between radiologists and referring physicians in conveying diagnostic certainty. We aimed to explore deep learning-based bidirectional contextual language models for automatically assessing diagnostic certainty expressed in the radiology reports to facilitate the precision of communication.
MATERIALS AND METHODS: We randomly sampled 594 head MR imaging reports from an academic medical center. We asked 3 board-certified radiologists to read sentences from the Impression section and assign each sentence 1 of the 4 certainty categories: "Non-Definitive," "Definitive-Mild," "Definitive-Strong," "Other." Using the annotated 2352 sentences, we developed and validated a natural language-processing system based on the start-of-the-art bidirectional encoder representations from transformers (BERT), which can capture contextual uncertainty semantics beyond the lexicon level. Finally, we evaluated 3 BERT variant models and reported standard metrics including sensitivity, specificity, and area under the curve.
RESULTS: A κ score of 0.74 was achieved for interannotator agreement on uncertainty interpretations among 3 radiologists. For the 3 BERT variant models, the biomedical variant (BioBERT) achieved the best macro-average area under the curve of 0.931 (compared with 0.928 for the BERT-base and 0.925 for the clinical variant [ClinicalBERT]) on the validation data. All 3 models yielded high macro-average specificity (93.13%-93.65%), while the BERT-base obtained the highest macro-average sensitivity of 79.46% (compared with 79.08% for BioBERT and 78.52% for ClinicalBERT). The BioBERT model showed great generalizability on the heldout test data with a macro-average sensitivity of 77.29%, specificity of 92.89%, and area under the curve of 0.93.
CONCLUSIONS: A deep transfer learning model can be developed to reliably assess the level of uncertainty communicated in a radiology report.
引用
- Liu F, Zhou P, Baccei SJ, et al. Qualifying Certainty in Radiology Reports through Deep Learning-Based Natural Language Processing. AJNR Am J Neuroradiol. 2021. PMID: 34413062. doi:10.3174/ajnr.A7241
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。