衡量人工智能在医院住院患者诊断中的影响:一项随机临床情景调查研究
该随机临床情景调查研究于2022年4月至2023年1月在美国13个州纳入457名临床医生(医院医师、执业护士和医师助理),通过9个急性呼吸衰竭住院患者的情景评估其对肺炎、心力衰竭或慢性阻塞性肺疾病作为病因的诊断准确率。基线准确率为73.0%(95%CI 68.3%至77.8%);标准AI模型无解释时准确率提高2.9个百分点(95%CI 0.5至5.2),有解释时提高4.4个百分点(95%CI 2.0至6.9);系统性偏倚AI模型使准确率较基线下降11.3个百分点(95%CI 7.2至15.5),提供解释时下降9.1个百分点(95%CI 4.9至13.2),与偏倚AI模型相比解释带来的改善为2.3个百分点(95%CI -2.7至7.2),无统计学意义。结论提示标准AI可改善诊断准确率,但系统性偏倚AI模型会降低准确率,且基于图像的AI模型解释未能减轻这一有害影响。
要点
- 标准AI模型无解释时临床医生诊断准确率较基线73.0%提高2.9个百分点(95%CI 0.5至5.2),有解释时提高4.4个百分点(95%CI 2.0至6.9)。
- 系统性偏倚AI模型预测使诊断准确率较基线下降11.3个百分点(95%CI 7.2至15.5),提供解释时下降9.1个百分点(95%CI 4.9至13.2)。
- 系统性偏倚AI模型加解释与不加解释相比,准确率改善2.3个百分点(95%CI -2.7至7.2),未达统计学显著。
- 研究纳入457名临床医生,随机分至AI预测无解释组(231人)和AI预测有解释组(226人),评估急性呼吸衰竭住院患者情景。
适用人群与场景
- 美国13个州的医院医师、执业护士和医师助理,参与于2022年4月至2023年1月进行的随机临床情景调查研究。
研究设计与关键结果
- 随机临床情景调查研究,纳入457名临床医生,随机分为AI模型预测无解释组(231人)和AI模型预测有解释组(226人),完成至少1个情景。
- 临床医生观看9个急性呼吸衰竭住院患者情景,其中2个无AI模型输入用于建立基线诊断准确率,6个随机分配有或无AI模型解释,这6个中3个为标准模型预测、3个为系统性偏倚模型预测。
- 基线诊断准确率为73.0%(95%CI 68.3%至77.8%)。
- 标准AI模型无解释时准确率较基线提高2.9个百分点(95%CI 0.5至5.2),有解释时提高4.4个百分点(95%CI 2.0至6.9)。
- 系统性偏倚AI模型预测使准确率较基线下降11.3个百分点(95%CI 7.2至15.5),提供解释时下降9.1个百分点(95%CI 4.9至13.2),与偏倚AI模型相比解释带来的改善为2.3个百分点(95%CI -2.7至7.2),无统计学意义。
- 参与者中位年龄34岁(IQR 31-39),241人(57.7%)为女性。
证据类型、等级与局限
- 证据类型:随机临床情景调查研究。
- 局限:原文未报告局限性。
对床旁实践的改变
- 标准AI模型可能提高临床医生对住院急性呼吸衰竭患者肺炎、心力衰竭或慢性阻塞性肺疾病的诊断准确率。
- 系统性偏倚的AI模型可能降低诊断准确率,常用的基于图像的AI模型解释未能减轻这一有害影响。
提炼说明
- 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开
IMPORTANCE: Artificial intelligence (AI) could support clinicians when diagnosing hospitalized patients; however, systematic bias in AI models could worsen clinician diagnostic accuracy. Recent regulatory guidance has called for AI models to include explanations to mitigate errors made by models, but the effectiveness of this strategy has not been established.
OBJECTIVES: To evaluate the impact of systematically biased AI on clinician diagnostic accuracy and to determine if image-based AI model explanations can mitigate model errors.
DESIGN, SETTING, AND PARTICIPANTS: Randomized clinical vignette survey study administered between April 2022 and January 2023 across 13 US states involving hospitalist physicians, nurse practitioners, and physician assistants.
INTERVENTIONS: Clinicians were shown 9 clinical vignettes of patients hospitalized with acute respiratory failure, including their presenting symptoms, physical examination, laboratory results, and chest radiographs. Clinicians were then asked to determine the likelihood of pneumonia, heart failure, or chronic obstructive pulmonary disease as the underlying cause(s) of each patient's acute respiratory failure. To establish baseline diagnostic accuracy, clinicians were shown 2 vignettes without AI model input. Clinicians were then randomized to see 6 vignettes with AI model input with or without AI model explanations. Among these 6 vignettes, 3 vignettes included standard-model predictions, and 3 vignettes included systematically biased model predictions.
MAIN OUTCOMES AND MEASURES: Clinician diagnostic accuracy for pneumonia, heart failure, and chronic obstructive pulmonary disease.
RESULTS: Median participant age was 34 years (IQR, 31-39) and 241 (57.7%) were female. Four hundred fifty-seven clinicians were randomized and completed at least 1 vignette, with 231 randomized to AI model predictions without explanations, and 226 randomized to AI model predictions with explanations. Clinicians' baseline diagnostic accuracy was 73.0% (95% CI, 68.3% to 77.8%) for the 3 diagnoses. When shown a standard AI model without explanations, clinician accuracy increased over baseline by 2.9 percentage points (95% CI, 0.5 to 5.2) and by 4.4 percentage points (95% CI, 2.0 to 6.9) when clinicians were also shown AI model explanations. Systematically biased AI model predictions decreased clinician accuracy by 11.3 percentage points (95% CI, 7.2 to 15.5) compared with baseline and providing biased AI model predictions with explanations decreased clinician accuracy by 9.1 percentage points (95% CI, 4.9 to 13.2) compared with baseline, representing a nonsignificant improvement of 2.3 percentage points (95% CI, -2.7 to 7.2) compared with the systematically biased AI model.
CONCLUSIONS AND RELEVANCE: Although standard AI models improve diagnostic accuracy, systematically biased AI models reduced diagnostic accuracy, and commonly used image-based AI model explanations did not mitigate this harmful effect.
TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT06098950.
参考来源
- Jabbour S, Fouhey D, Shepard S, et al. Measuring the Impact of AI in the Diagnosis of Hospitalized Patients: A Randomized Clinical Vignette Survey Study. JAMA. 2023. PMID: 38112814. doi:10.1001/jama.2023.22295
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。