文献速递重症综合管理与镇静镇痛

评估o1推理大语言模型认知偏倚的病例情景研究

Evaluating the o1 reasoning large language model for cognitive bias: a vignette study.
PubMed 文献 · Crit Care · 2025-08-21
一句话结论o1推理模型在10个临床情景中有7个未显示可测量的认知偏倚,2个有显著偏倚但幅度低于GPT-4和人类临床医生,1个(奥卡姆剃刀)持续偏倚。

该研究用10对临床情景(每对含两个仅细微修改的版本,如呈现死亡率与生存率)测试o1模型(o1-2024-12-17)的认知偏倚,每个版本生成90条独立临床建议,共1800条回复,并与既往发表的GPT-4和人类临床医生研究结果比较。o1在7个情景中未显示可测量的认知偏倚;2个情景显示显著偏倚但绝对幅度低于GPT-4和人类临床医生;1个情景(奥卡姆剃刀)显示持续偏倚。模型在包含「弥合差距线索」的情景中更易出现偏倚,8个情景版本的组内一致性超过94%,决策变异性低于既往GPT-4和人类临床医生。作者认为推理模型可能减少认知偏倚和判断随机变异(即「噪声」),但仍非完全免疫,作为决策支持工具可能带来一些益处,也需进一步探索其可能失败的情形。

要点

  • o1模型在10个临床情景中的7个未显示可测量的认知偏倚,2个有显著偏倚但绝对幅度低于GPT-4和人类临床医生。
  • 在奥卡姆剃刀情景中,o1模型表现出持续偏倚,是唯一一个偏倚持续存在的情景。
  • 模型在包含「弥合差距线索」(似乎解决临床不确定性)的情景中更易出现偏倚。
  • 8个情景版本的组内一致性超过94%,决策变异性低于既往GPT-4和人类临床医生。
  • 作者认为推理模型可能减少认知偏倚和判断随机变异(「噪声」),但仍非完全免疫,需进一步探索其失败情形。

适用人群与场景

  • 测试对象为o1模型(o1-2024-12-17),非人类患者或临床人群;情景模拟临床决策场景,涉及高决策负荷、时间有限和诊断不确定性的临床环境(如重症监护)。

诊断标准与关键阈值

  • 采用Wang和Redelmeier建立的方法学,使用10对临床情景,每对情景的两个版本通过细微修改触发特定认知偏倚(如呈现死亡率与生存率)。
  • 认知偏倚定义为配对情景之间建议率的系统性差异,无偏倚推理下不应出现。
  • 每个情景版本生成90条独立临床建议,共1800条回复。

研究设计与关键结果

  • 研究类型为病例情景研究,测试o1模型(o1-2024-12-17)对认知偏倚的易感性,每个情景版本生成90条独立临床建议,共1800条回复。
  • o1模型在10个情景中的7个未显示可测量的认知偏倚。
  • 在2个情景中,o1模型显示显著偏倚,但绝对幅度低于既往报告的GPT-4和人类临床医生数值。
  • 在1个情景(奥卡姆剃刀)中,o1模型表现出持续偏倚。
  • 模型在包含「弥合差距线索」的情景中更易出现偏倚。
  • 8个情景版本的组内一致性超过94%,决策变异性低于既往描述的GPT-4和人类临床医生。

证据类型、等级与局限

  • 证据类型:病例情景研究(vignette study),比较o1模型与既往发表的GPT-4和人类临床医生研究结果。
  • 局限:原文未报告局限性。

对床旁实践的改变

  • 作者认为推理模型可能减少认知偏倚和判断随机变异(即「噪声」),作为决策支持工具在医学中可能带来一些益处。
  • 作者提示推理模型仍非完全免疫于认知偏倚,需进一步探索这些工具可能失败的情形。
  • 模型在包含「弥合差距线索」的情景中更易出现偏倚,提示在此类临床决策场景中使用时需注意。

提炼说明

  • 本条依据论文摘要整理;摘要未涉及「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开

BACKGROUND: Cognitive biases, systematic deviations from logical judgment, are well documented in clinical decision-making, particularly in clinical settings characterized by high decision load, limited time, and diagnostic uncertainty-such as critical care. Prior work demonstrated that large language models, particularly GPT-4, reproduce many of these biases, sometimes to a greater extent than human clinicians.

METHODS: We tested whether the o1 model (o1-2024-12-17), a newly released AI system with enhanced reasoning capabilities, is susceptible to cognitive biases that commonly affect medical decision-making. Following the methodology established by Wang and Redelmeier [15], we used ten pairs of clinical scenarios, each designed to test a specific cognitive bias known to influence clinicians. Each scenario had two versions, differed by subtle modifications designed to trigger the bias (such as presenting mortality rates versus survival rates). The o1 model generated 90 independent clinical recommendations for each scenario version, totalling 1,800 responses. We measured cognitive bias as systematic differences in recommendation rates between the paired scenarios, which should not occur with unbiased reasoning. The o1 model's performance was compared against previously published results from both the GPT-4 model and historical human clinician studies.

RESULTS: The o1 model showed no measurable cognitive bias in seven of the ten vignettes. In two vignettes, the o1 model showed significant bias, but its absolute magnitude was lower than values previously reported for GPT-4 and human clinicians. In a single vignette, Occam's razor, the o1 model exhibited consistent bias. Therefore, although overall bias appears less frequent overall with the reasoning model than with GPT-4, it was worse in one vignette. The model was more prone to bias in vignettes that included a gap-closing cue, seemingly resolving the clinical uncertainty. Across eight vignette versions, intra‑scenario agreement exceeded 94%, indicating lower decision variability than previously described with GPT‑4 and human clinicians.

CONCLUSION: Reasoning models may reduce cognitive bias and random variation in judgment (i.e., "noise"). However, our findings caution that reasoning models are still not entirely immune to cognitive bias. These findings suggest that reasoning models may impart some benefits as decision-support tools in medicine, but they also imply a need to explore further the circumstances in which these tools may fail.

来源:Crit Care,查看原文。版权归原作者所有。

参考来源

  1. Degany O, Laros S, Idan D, et al. Evaluating the o1 reasoning large language model for cognitive bias: a vignette study. Crit Care. 2025. PMID: 40841953. doi:10.1186/s13054-025-05591-5

供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。