文献速递复苏、休克与脓毒症

GPT-4o作为心肺复苏技能考试评估者的适用性

Suitability of GPT-4o as an evaluator of cardiopulmonary resuscitation skills examinations.
PubMed 文献 · Resuscitation · 2024-09-28
一句话结论GPT-4o评分心肺复苏技能视频的准确性与高年资专家相似,提示其可能用于医学考试场景。

本研究纳入6名具备国家执业医师考试考官资质的专家(3名初级、3名高级),审阅103名考生的心肺复苏技能考试视频,所有视频同时由GPT-4o自动评估。专家与GPT-4o均对患者评估、胸外按压、人工呼吸和重复操作四个部分评分。GPT-4o与初级专家相比,上述四部分平均准确性评分分别为0.65、0.58、0.60、0.31;与高级专家相比分别为0.75、0.65、0.72、0.41。专家对GPT-4o可靠性的中位Likert评分为:初级专家4.00(IQR 3.66-4.33,均值[标准差]3.95[0.55]),高级专家4.33(IQR 4.00-4.67,均值4.29[0.50])。结论认为GPT-4o在检查心肺复苏技能考试视频方面表现出与高级专家相似的准确性,展示了在医学考试环境中部署该大语言模型的潜力。

要点

  • GPT-4o与高级专家评分的一致性在患者评估部分为0.75,胸外按压0.65,人工呼吸0.72,重复操作0.41。
  • GPT-4o与初级专家评分的一致性在患者评估部分为0.65,胸外按压0.58,人工呼吸0.60,重复操作0.31。
  • 高级专家对GPT-4o可靠性的中位Likert评分为4.33(IQR 4.00-4.67),初级专家为4.00(IQR 3.66-4.33)。
  • 重复操作部分的准确性评分最低(与初级专家0.31,与高级专家0.41),提示该部分一致性较差。
  • 作者认为GPT-4o在检查心肺复苏技能考试视频方面表现出与高级专家相似的准确性,展示了在医学考试环境中部署的潜力。

适用人群与场景

  • 纳入103名参加心肺复苏技能考试的考生视频。
  • 由6名认证考官(3名初级、3名高级)审阅视频,所有视频同时由GPT-4o自动评估。

研究设计与关键结果

  • 研究类型为对103名考生心肺复苏技能考试视频的评估者间一致性研究。
  • GPT-4o与初级专家相比,患者评估、胸外按压、人工呼吸、重复操作的平均准确性评分分别为0.65、0.58、0.60、0.31;与高级专家相比分别为0.75、0.65、0.72、0.41。
  • 专家对GPT-4o可靠性的中位Likert评分为:初级专家4.00(IQR 3.66-4.33,均值[标准差]3.95[0.55]),高级专家4.33(IQR 4.00-4.67,均值4.29[0.50])。

证据类型、等级与局限

  • 证据类型:评估者间一致性研究(非随机对照试验)。
  • 局限:原文未报告局限性。

对床旁实践的改变

  • 作者认为GPT-4o在检查心肺复苏技能考试视频方面表现出与高级专家相似的准确性,展示了在医学考试环境中部署该大语言模型的潜力。

提炼说明

  • 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开

AIM: To assess the accuracy and reliability of GPT-4o for scoring examinees' performance on cardiopulmonary resuscitation (CPR) skills tests.

METHODS: This study included six experts certified to supervise the national medical licensing examination (three junior and three senior) who reviewed the CPR skills test videos across 103 examinees. All videos reviewed by the experts were subjected to automated assessment by GPT-4o. Both the experts and GPT-4o scored the videos across four sections: patient assessment, chest compressions, rescue breathing, and repeated operations. The experts subsequently rated GPT-4o's reliability on a 5-point Likert scale (1, completely unreliable; 5, completely reliable). GPT-4o's accuracy was evaluated using the intraclass correlation coefficient (for the first three sections) and Fleiss' Kappa (for the last section) to assess the agreement between its scores vs. those of the experts.

RESULTS: The mean accuracy scores for the patient assessment, chest compressions, rescue breathing, and repeated operation sections were 0.65, 0.58, 0.60, and 0.31, respectively, when comparing the GPT-4o's vs. junior experts' scores and 0.75, 0.65, 0.72, and 0.41, respectively, when comparing the GPT-4o's vs. senior experts' scores. For reliability, the median Likert scale scores were 4.00 (interquartile range [IQR] = 3.66-4.33, mean [standard deviation] = 3.95 [0.55]) and 4.33 (4.00-4.67, 4.29 [0.50]) for the junior and senior experts, respectively.

CONCLUSIONS: GPT-4o demonstrated a level of accuracy that was similar to that of senior experts in examining CPR skills examination videos. The results demonstrate the potential for deploying this large language model in medical examination settings.

来源:Resuscitation,查看原文。版权归原作者所有。

参考来源

  1. Wang L, Mao Y, Wang L, et al. Suitability of GPT-4o as an evaluator of cardiopulmonary resuscitation skills examinations. Resuscitation. 2024. PMID: 39343124. doi:10.1016/j.resuscitation.2024.110404

供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。