评估放射学中的侧别错误:生成式人工智能与自然语言处理的比较
本研究比较了生成式人工智能(ATARI)与自然语言处理(NLP)工具在识别放射学报告和图像中侧别错误的表现。从NLP标记的898份报告中,64%为NLP误报,36%为报告错误;ATARI文本查询识别NLP误报的准确率为97.4%,结合视觉与文本查询准确率为98.3%。结论认为ATARI原型优于所评估的NLP工具,但复杂报告中的文本查询错误仍需改进。
要点
- NLP标记的侧别错误中64%为误报,36%为真实报告错误。
- ATARI文本查询识别NLP误报的准确率为97.4%。
- ATARI结合视觉与文本查询准确率98.3%,可进行基于图像的侧别判定。
核心结论
- 生成式AI赋能的ATARI原型在判断放射学报告中真实与虚假侧别错误方面优于所评估的NLP工具,并能实现基于图像的侧别判定;ATARI文本查询在复杂放射学报告中的潜在错误强调该技术需进一步改进。
研究设计与样本
- 比较研究;初始NLP标记的1124份放射学报告,选取898份涵盖X线摄影、CT、MRI和超声的报告由放射科医生复核;对237份报告和图像(连续118份NLP真阳性、119份NLP假阳性)应用ATARI;参考标准为放射科医生复核。
诊断标准与关键数值
- NLP标记的898份报告中,64%(574/898)为NLP误报,36%(324/898)为报告错误;ATARI文本查询识别NLP误报的准确率为97.4%(115/118,95%CI 96.5%-98.3%);结合视觉与文本查询准确率为98.3%(116/118,95%CI 97.6%-99.0%);单独查询准确率为98.3%(116/118,95%CI 97.6%-99.0%)。
局限性
- ATARI文本查询在复杂放射学报告中的潜在错误强调该技术需进一步改进。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
PURPOSE: We compared the performance of generative artificial intelligence (AI) (Augmented Transformer Assisted Radiology Intelligence [ATARI, Microsoft Nuance, Microsoft Corporation, Redmond, Washington]) and natural language processing (NLP) tools for identifying laterality errors in radiology reports and images.
METHODS: We used an NLP-based (mPower, Microsoft Nuance) tool to identify radiology reports flagged for laterality errors in its Quality Assurance Dashboard. The NLP model detects and highlights laterality mismatches in radiology reports. From an initial pool of 1,124 radiology reports flagged by the NLP for laterality errors, we selected and evaluated 898 reports that encompassed radiography, CT, MRI, and ultrasound modalities to ensure comprehensive coverage. A radiologist reviewed each radiology report to assess if the flagged laterality errors were present (reporting error-true-positive) or absent (NLP error-false-positive). Next, we applied ATARI to 237 radiology reports and images with consecutive NLP true-positive (118 reports) and false-positive (119 reports) laterality errors. We estimated accuracy of NLP and generative AI tools to identify overall and modality-wise laterality errors.
RESULTS: Among the 898 NLP-flagged laterality errors, 64% (574 of 898) had NLP errors and 36% (324 of 898) were reporting errors. The text query ATARI feature correctly identified the absence of laterality mismatch (NLP false-positives) with a 97.4% accuracy (115 of 118 reports; 95% confidence interval [CI] = 96.5%-98.3%). Combined vision and text query resulted in 98.3% accuracy (116 of 118 reports or images; 95% CI = 97.6%-99.0%), and query alone had a 98.3% accuracy (116 of 118 images; 95% CI = 97.6%-99.0%).
CONCLUSION: The generative AI-empowered ATARI prototype outperformed the assessed NLP tool for determining true and false laterality errors in radiology reports while enabling an image-based laterality determination. Underlying errors in ATARI text query in complex radiology reports emphasize the need for further improvement in the technology.
引用
- Kathait AS, Garza-Frias E, Sikka T, et al. Assessing Laterality Errors in Radiology: Comparing Generative Artificial Intelligence and Natural Language Processing. J Am Coll Radiol. 2024. PMID: 38960083. doi:10.1016/j.jacr.2024.06.014
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。