文献速递呼吸衰竭与机械通气重症综合管理与镇静镇痛

机械通气领域人专家与大型语言模型生成的多选题质量比较

Quality of Human Expert vs Large Language Model-Generated Multiple-Choice Questions in the Field of Mechanical Ventilation.
PubMed 文献 · Chest · 2025-07-18
一句话结论ChatGPT-o1生成的机械通气多选题在质量上非劣于人类专家编写题,且评审专家无法可靠区分两者来源。

本研究比较了ChatGPT-o1生成与人类专家编写的机械通气多选题质量。选取三个机械通气主题和15个学习目标,生成15道AI题,31名定期教授机械通气的faculty专家在盲态下从目标一致性、答案准确性、题干清晰度、干扰项合理性和难度五个维度评分。AI题在总分上非劣于人类专家题(95%置信区间上限[-1.15, ∞]),且评审者无法可靠区分题目来源(P = .32)。结果表明AI辅助出题可能成为医学教育评估的高效可扩展替代方案,即使在机械通气这类高度专业化领域。

要点

  • 31名机械通气教学专家盲态评审ChatGPT-o1生成的15道多选题与人类专家题,AI题总分非劣于人类专家题(95%置信区间上限[-1.15, ∞])。
  • 评审者无法可靠区分AI生成题与人类专家编写题(P = .32)。
  • 评分维度包括目标一致性、答案准确性、题干清晰度、干扰项合理性和难度水平。
  • 非劣效界值预设为总可能得分的15%(-3.45)。
  • 作者认为AI辅助出题可能成为医学教育评估的高效可扩展替代方案。

适用人群与场景

  • 接受机械通气教育的学员(原文以机械通气教育为场景)
  • 定期教授机械通气的faculty专家作为评审者(31名)

诊断标准与关键阈值

  • 非劣效界值预设为总可能得分的15%(-3.45)
  • 评分维度:与学习目标的一致性、所选答案的准确性、题干的清晰度、干扰项选项的合理性、难度水平

研究设计与关键结果

  • 研究设计:比较ChatGPT-o1生成与人类专家编写的机械通气多选题质量,31名faculty专家盲态评审。
  • 选取三个机械通气主题:运动方程与欧姆定律、Tau与自动PEEP、氧合。
  • 使用15个学习目标,通过标准化提示词经ChatGPT-o1(预览模型,2024年9月12日发布)生成15道AI多选题。
  • AI生成多选题在总分上非劣于人类专家编写题(95%置信区间上限[-1.15, ∞])。
  • 评审者无法可靠区分AI生成题与人类专家编写题(P = .32)。

证据类型、等级与局限

  • 证据类型:比较研究(专家盲态评审)
  • 局限:原文未报告局限性

对床旁实践的改变

  • 作者认为,鉴于人类出题耗时且资源密集,AI辅助出题可能作为医学教育评估的高效可扩展替代方案,即使在机械通气这类高度专业化领域。

提炼说明

  • 本条依据论文摘要整理;摘要未涉及「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开

BACKGROUND: Although mechanical ventilation (MV) is a critical competency in critical care training, standardized methods for assessing MV-related knowledge are lacking. Traditional multiple-choice question (MCQ) development is resource intensive, and prior studies have suggested that generative AI tools could streamline question creation. However, the quality of AI-generated MCQs remains unclear.

RESEARCH QUESTION: Are MCQs generated by ChatGPT noninferior to human expert (HE)-created questions in terms of quality and relevance for MV education?

STUDY DESIGN AND METHODS: Three key MV topics were selected: Equation of Motion and Ohm's Law, Tau and Auto-PEEP, and Oxygenation. Fifteen learning objectives were used to generate 15 AI-written MCQs via a standardized prompt with ChatGPT-o1 (preview model; made available September 12, 2024). A group of 31 faculty experts, all of whom regularly teach MV, evaluated both AI- and HE-generated MCQs. Each MCQ was assessed based on its alignment with learning objectives, accuracy of chosen answer, clarity of the question stem, plausibility of distractor options, and difficulty level. The faculty members were blinded to the provenance of the MCQ questions. The noninferiority margin was predefined as 15% of the total possible score (-3.45).

RESULTS: AI-generated MCQs were statistically noninferior to the HE-written MCQs (95% upper CI, [-1.15, ∞]). In additions, respondents were unable to reliably differentiate AI-generated MCQs from HE-written MCQs (P = .32).

INTERPRETATION: Our results suggest that AI-generated MCQs using ChatGPT-o1 are comparable in quality to those written by HEs. Given the time and resource-intensive nature of human MCQ development, AI-assisted question generation may serve as an efficient and scalable alternative for medical education assessment, even in highly specialized domains such as MV.

来源:Chest,查看原文。版权归原作者所有。

参考来源

  1. Safadi S, Amirahmadi R, Tlimat A, et al. Quality of Human Expert vs Large Language Model-Generated Multiple-Choice Questions in the Field of Mechanical Ventilation. Chest. 2025. PMID: 40684906. doi:10.1016/j.chest.2025.07.005

供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。