评估自然语言处理工具在胸片报告标注中的性能与偏倚
本回顾性研究于2022年4月至2024年4月开展,在按年龄、性别、种族和民族均衡抽样的MIMIC胸片数据集(n=692)和印第安纳大学(IU)胸片数据集(n=3665)上,评估四种自然语言处理(NLP)工具对14个胸部疾病标签的报告标注性能与人口学偏倚。四种工具总体准确率较高(如GPT-4在IU为94.3%、在MIMIC为91.6%),但人口学亚组分析显示存在显著偏倚,老年患者中表现较差。
要点
- 四种NLP工具总体标注准确率较高,但亚组分析显示老年患者中错误率更高。
- CheXpert、RRA和cTAKES在80岁以上患者错误率最高(均值15.8%±5.0)。
- GPT-4错误率最高出现在60-80岁患者(8.3%)。
核心结论
- 常用胸片报告标注NLP工具在总体评估时准确率较高,但人口学亚组分析显示显著偏倚,老年患者中表现较差。
研究设计与样本
- 回顾性研究,于2022年4月至2024年4月开展,评估四种NLP工具(CheXpert[基于规则]、RadReportAnnotator[RRA;基于深度学习]、OpenAI的GPT-4[基于深度学习]、cTAKES[混合方法])在MIMIC胸片数据集按年龄、性别、种族和民族均衡抽样的子集(n=692)及整个印第安纳大学(IU)胸片数据集(n=3665)上的报告标注性能;由三名委员会认证的放射科医师对14个胸部疾病标签进行标注。
诊断标准与关键数值
- IU数据集3665例(平均年龄49.7岁±17[SD];1963例女性),MIMIC数据集692例(平均年龄54.1岁±23.1;357例女性)。
- IU与MIMIC数据集各工具准确率:CheXpert 92.6%(47 516/51 310)与90.2%(8742/9688),RRA 82.9%(19 746/23 829)与92.2%(2870/3114),GPT-4 94.3%(45 586/48 342)与91.6%(6721/7336),cTAKES 84.7%(43 436/51 310)与88.7%(8597/9688)。
- RRA和cTAKES在MIMIC数据集上准确率更高(P < .001),CheXpert和GPT-4在IU数据集上准确率更高。
- 除RRA在MIMIC数据集外,所有NLP工具的错误率在各年龄组间差异有统计学意义(P < .001);CheXpert、RRA和cTAKES在80岁以上患者错误率最高(均值15.8%±5.0),GPT-4在60-80岁患者错误率最高(8.3%)。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background Natural language processing (NLP) is commonly used to annotate radiology datasets for training deep learning (DL) models. However, the accuracy and potential biases of these NLP methods have not been thoroughly investigated, particularly across different demographic groups. Purpose To evaluate the accuracy and demographic bias of four NLP radiology report labeling tools on two chest radiograph datasets. Materials and Methods This retrospective study, performed between April 2022 and April 2024, evaluated chest radiograph report labeling using four NLP tools (CheXpert [rule-based], RadReportAnnotator [RRA; DL-based], OpenAI's GPT-4 [DL-based], cTAKES [hybrid]) on a subset of the Medical Information Mart for Intensive Care (MIMIC) chest radiograph dataset balanced for representation of age, sex, and race and ethnicity (n = 692) and the entire Indiana University (IU) chest radiograph dataset (n = 3665). Three board-certified radiologists annotated the chest radiograph reports for 14 thoracic disease labels. NLP tool performance was evaluated using several metrics, including accuracy and error rate. Bias was evaluated by comparing performance between demographic subgroups using the Pearson χ2 test. Results The IU dataset included 3665 patients (mean age, 49.7 years ± 17 [SD]; 1963 female), while the MIMIC dataset included 692 patients (mean age, 54.1 years ± 23.1; 357 female). All four NLP tools demonstrated high accuracy across findings in the IU and MIMIC datasets, as follows: CheXpert (92.6% [47 516 of 51 310], 90.2% [8742 of 9688]), RRA (82.9% [19 746 of 23 829], 92.2% [2870 of 3114]), GPT-4 (94.3% [45 586 of 48 342], 91.6% [6721 of 7336]), and cTAKES (84.7% [43 436 of 51 310], 88.7% [8597 of 9688]). RRA and cTAKES had higher accuracy (P < .001) on the MIMIC dataset, while CheXpert and GPT-4 had higher accuracy on the IU dataset. Differences (P < .001) in error rates were observed across age groups for all NLP tools except RRA on the MIMIC dataset, with the highest error rates for CheXpert, RRA, and cTAKES in patients older than 80 years (mean, 15.8% ± 5.0) and the highest error rate for GPT-4 in patients 60-80 years of age (8.3%). Conclusion Although commonly used NLP tools for chest radiograph report annotation are accurate when evaluating reports in aggregate, demographic subanalyses showed significant bias, with poorer performance in older patients. © RSNA, 2024 Supplemental material is available for this article. See also the editorial by Cai in this issue.
引用
- Santomartino SM, Zech JR, Hall K, et al. Evaluating the Performance and Bias of Natural Language Processing Tools in Labeling Chest Radiograph Reports. Radiology. 2024. PMID: 39436298. doi:10.1148/radiol.232746
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。