诊断放射学住院医师选拔中人工智能自动排序的局限性
单中心前瞻性试点研究纳入2025-2026周期148名面试申请者,比较七种大语言模型配置在仅用申请材料和加入面试评分两种条件下的排序结果与人类委员会最终排序的一致性。仅用申请材料时模型一致性低,加入面试评分后一致性显著提高,提示人工智能不应替代人类判断作为主要排序工具。
要点
- 仅用申请材料自动生成的排序可能系统性压低女性、国际申请者及非MD背景申请者
- AI排序工具不宜作为住院医师选拔的主要排序手段,仅可辅助数据检索与回顾性审计
- 面试评分对最终排序起决定作用,单独面试评分排序与最终名单高度一致
核心结论
- 仅使用面试前申请材料时,所有大语言模型配置与人类委员会最终排序的一致性均较低,不能可靠复制最终排序名单。
- 加入人类面试评分后模型一致性升至优秀水平,说明初始差异主要源于仅提供申请材料的局限性而非模型排序能力不足。
- 研究结果不能证明模型与人类委员会哪一方排序更优,但仅基于申请材料生成的AI排序可能对特定亚组产生系统性偏倚。
研究设计与样本
- 单中心试点研究,纳入2025-2026周期148名面试申请者,竞争7个住院医师岗位。
- 使用三个专有大语言模型各设中等与高推理水平及一个开放权重模型,共七种模型配置。
- 每种配置在仅用申请材料和加入面试评分两种条件下各生成10个独立排序,共140个模型排序。
- 以人类委员会最终排序为参照标准,采用加权Kendall τ及precision@k(k=10-40)评估一致性。
诊断标准与关键数值
- 仅用申请材料时,所有模型配置与最终排序的一致性均低,Kendall τ中位数为0.15-0.36。
- 加入面试评分后一致性升至优秀,Kendall τ中位数为0.84-0.93。
- 仅按面试官评分总和排序与最终名单的τ-b为0.83,仅按USMLE Step 2 CK评分排序τ-b为0.12。
- 推理水平从「中等」升至「高等」未一致改善与人类排序的一致性。
- 仅用申请材料时,模型将女性、国际及非MD申请者排在人类委员会之后(q<.05),项目信号者排位更高。
临床应用与阅片要点
- 人工智能不应作为住院医师选拔的主要排序生成工具,也不应替代人类判断。
- AI可承担非核心支持性任务,如数据检索与回顾性审计。
- 人类委员会最终排序约需390人时,提示排序整合人力成本高但仍是必要的决策环节。
- 使用自动排序工具时需警惕对特定申请者亚组的系统性排序偏倚。
局限性
- 研究为单中心设计,结果外推至其他机构需谨慎。
- 无法确定人类委员会与模型排序之间的一致性差异究竟代表哪一方更优。
- 模型排序矛盾可能源于仅提供申请材料的限制,而非模型算法本身能力不足。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
OBJECTIVE: To evaluate whether large language models (LLMs) can reliably reproduce a residency program's rank order list (ROL) from pre-interview application data alone versus with human interviews included and to determine whether automated ranking tools safely mirror human consensus or inadvertently introduce systematic displacements against specific applicant subgroups.
METHODS: In this single-institution pilot study of 148 applicants interviewed during the 2025-2026 cycle, seven LLM configurations (three proprietary models [each at "medium" and "high" reasoning levels] and one open-weight model) each generated 10 independent ROLs from de-identified application data under two conditions: pre-interview "application-only" data alone, and then with interview scores added (140 lists total). Nonmodel baselines (sorting solely by interview score or United States Medical Licensing Examination Step 2 Clinical Knowledge score) were also established. LLM lists were compared with the human committee's final ROL using a truth-anchored weighted Kendall's τ and precision@k (k = 10-40); rank displacement by applicant subgroup was also assessed.
RESULTS: The human committee's final ROL required approximately 390 faculty-hours to order 148 applicants for 7 positions. For LLMs in the application-only condition, all LLM configurations showed low agreement with the final ROL (median τ 0.15-0.36). In the positive control condition with human interviews included, agreement rose to excellent (median τ 0.84-0.93). Sorting applicants by summed interviewer score alone reproduced the final list at τ-b = 0.83, whereas sorting by United States Medical Licensing Examination Step 2 Clinical Knowledge score alone produced τ-b = 0.12. Increasing reasoning level from "medium" to "high" did not consistently improve agreement. In the application-only condition, LLM rankings systematically placed female, international, and non-MD applicants below the human committee's list (q < .05), and program signalers were placed higher.
DISCUSSION: LLMs failed to replicate the human program's final ROL from pre-interview application-only data alone. The positive control including the human interview confirms this human-LLM lack of agreement may stem from the limited application-only data provided, rather than the models' inability to perform ranking tasks once provided the additional human judgment information. Importantly, these findings do not establish whether the LLM model or the human committee produced the "better" ROL; these data suggest that LLM rankings generated solely from pre-interview data systematically differed from the human committee's consensus, and may disproportionately penalize certain subgroups. Our findings suggest that artificial intelligence should not be used as a primary ROL generator or a replacement for human judgment but may serve supportive roles in tasks such as data retrieval and retrospective auditing.
引用
- Savage, Hossain, Tonascia, et al. Human Judgment and the Limits of Artificial Intelligence for Automated Rank Order Lists in Diagnostic Radiology Residency Selection. J Am Coll Radiol. 2026. PMID: 42727622. doi:10.1016/j.jacr.2026.09.006
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。