OpenEvidence在结构化分诊推荐测试中偏向安全侧
该研究用60个临床情景(含客观数据与否×人口学/情境修饰的2×2×2×2析因设计,共960条提示)评估医生端检索增强平台OpenEvidence的分诊安全性,并与既往ChatGPT Health基准比较。在449条返回推荐的明确病例中,准确率71.3%;OpenEvidence对急诊表现漏诊12.5%(ChatGPT Health为51.6%),对非紧急家庭表现过度分诊68.0%(ChatGPT Health为64.8%)。锚定语句未改变推荐(OR=1.08,95%CI 0.62-1.88;Holm校正P=1.0)。提供客观临床数据可消除急诊漏诊(25%降至0%;P=0.005)并减少非紧急过度分诊(78.7%降至57.8%;P=0.014)。960条回复中65条(6.8%)拒绝分诊,仅见于仅有症状的家庭或常规提示。
要点
- 在449条返回推荐的明确病例中,OpenEvidence分诊准确率为71.3%。
- OpenEvidence对急诊表现漏诊12.5%,低于既往ChatGPT Health基准的51.6%。
- OpenEvidence对非紧急家庭表现过度分诊68.0%(ChatGPT Health为64.8%)。
- 提供客观临床数据使急诊漏诊从25%降至0%(P=0.005),非紧急过度分诊从78.7%降至57.8%(P=0.014)。
- 锚定语句未改变推荐(OR=1.08,95%CI 0.62-1.88;Holm校正P=1.0)。
适用人群与场景
- 评估对象为医生端检索增强平台OpenEvidence,使用60个临床医生撰写的临床情景,覆盖21个领域、30个临床场景,每个场景含客观临床数据与否,并与人口学和情境修饰因素交叉。
- 提示分为480条明确病例和480条边缘病例,回复按临床医生金标准分类为正确分诊、漏诊、过度分诊或寻求证据的拒绝。
研究设计与关键结果
- 研究类型为结构化基准测试,使用60个临床情景生成960条提示(480条明确病例、480条边缘病例),采用聚类自助重采样、混合效应逻辑回归和Holm-Bonferroni校正分析。
- 在449条返回推荐的明确病例中,OpenEvidence分诊准确率为71.3%。
- OpenEvidence对急诊表现漏诊12.5%,既往ChatGPT Health基准为51.6%;对非紧急家庭表现过度分诊68.0%,ChatGPT Health为64.8%。
- 锚定语句未改变推荐(OR=1.08,95%CI 0.62-1.88;Holm校正P=1.0)。
- 提供客观临床数据使急诊漏诊从25%降至0%(P=0.005),非紧急过度分诊从78.7%降至57.8%(P=0.014)。
- 960条回复中65条(6.8%)拒绝分诊,仅见于仅有症状的家庭或常规提示。
证据类型、等级与局限
- 证据类型为结构化基准测试,比较OpenEvidence与既往报告的ChatGPT Health基准。
- 局限:原文未报告局限性。
对床旁实践的改变
- 作者认为,这些发现支持在部署情境中评估健康人工智能,并将拒绝分诊视为一种独立的输出类别,其临床意义需要单独评估。
提炼说明
- 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开
BACKGROUND: Large language model (LLM) chatbots are increasingly consulted for triage decisions. A structured benchmark reported that ChatGPT Health, a consumer-facing health assistant, under-triaged 51.6 % of true emergencies and was susceptible to social anchoring. Whether a physician-facing clinical decision support platform fails similarly is unknown.
OBJECTIVE: To characterize the triage safety profile of OpenEvidence, a retrieval-augmented, physician-facing platform, using the identical benchmark previously applied to ChatGPT Health.
METHODS: We evaluated 60 clinician-authored vignettes from 30 clinical scenarios across 21 domains. Each scenario was written with and without objective clinical data and crossed with demographic and contextual modifiers in a 2 × 2 × 2 × 2 factorial design, yielding 960 prompts (480 clear-case, 480 edge-case). Responses were classified against a clinician gold standard as correct triage, under-triage, over-triage, or evidence-seeking refusal. Analyses used cluster bootstrap resampling, mixed-effects logistic regression, and Holm-Bonferroni correction.
RESULTS: Among 449 clear-case responses that returned a recommendation, accuracy was 71.3 %. OpenEvidence under-triaged 12.5 % of emergency presentations versus 51.6 % in the previously reported ChatGPT Health benchmark, and over-triaged 68.0 % of nonurgent Home presentations (ChatGPT Health, 64.8 %). Anchoring statements did not alter recommendations (OR = 1.08, 95 % CI 0.62-1.88; Holm-adjusted p = 1.0). Objective clinical data eliminated emergency under-triage (25 % to 0 %; p = 0.005) and reduced nonurgent over-triage (78.7 % to 57.8 %; p = 0.014). In 65 of 960 responses (6.8 %), the platform declined to assign a triage level, exclusively in symptom-only Home or Routine prompts.
CONCLUSIONS: Under this benchmark, OpenEvidence produced fewer missed emergencies than the historical ChatGPT Health comparison, while errors concentrated in over-triage and evidence-seeking refusal. These findings support evaluating health AI within its deployment context and treating refusal as a distinct output category whose clinical implications require separate assessment.
参考来源
- Jia E, Omar M, Barash Y, et al. OpenEvidence errs on the safe side in a structured test of triage recommendations. Int J Med Inform. 2026. PMID: 42673790. doi:10.1016/j.ijmedinf.2026.106687
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。