开放腹主动脉瘤修复能力评估工具OPERATE的开发与验证
本研究开发了针对开放腹主动脉瘤(AAA)修复的手术特异性评估工具OPERATE,并基于Messick框架收集效度证据、建立及格标准。
要点
- OPERATE工具在模拟环境中能有效区分新手与经验丰富的开放AAA修复医生。
- 及格分设为27.7,仅1名新手通过而6名经验者未通过,提示标准较高。
- 经验者首次尝试得分低,提示评估前需熟悉模拟设备。
核心结论
- OPERATE工具获得了效度证据,能区分新手与经验丰富的开放AAA修复医生,可用于模拟环境下的形成性和终结性评估。
- 较高的及格分强调新手需要在模拟环境中训练达到一定能力水平后,再在监督下进行临床学徒培训。
- 评估前必须确保熟悉模拟设备,因为经验组首次尝试得分较低。
研究设计与样本
- 基于Messick框架的效度研究,由三位开放AAA修复经验丰富的血管外科医生和一位医学教育评估与验证专家共同开发OPERATE工具。
- 三位专家在相互盲评的设置下独立评分,采用Angoff标准设定法建立可信的及格/不及格分数。
主要发现
- 及格分设为27.7,1名新手通过测试,6名经验丰富医生未通过。
关键数值与推荐
- 及格/不及格分数设定为27.7。
临床怎么用
- OPERATE工具可用于模拟环境下开放AAA修复的形成性评估和终结性评估。
- 新手应在模拟环境中训练达到及格水平后,再进入临床环境在监督下进行学徒培训。
- 评估前应确保受试者熟悉模拟设备,以避免因不熟悉设备导致得分偏低。
局限性
- 研究仅在模拟环境中进行,未在真实临床环境中验证。
- 经验丰富医生首次尝试得分较低,可能受模拟设备熟悉度影响。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard. METHODS: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score. RESULTS: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed. CONCLUSION: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.
引用
- Nayahangan LJ, Lawaetz J, Strøm M, et al. Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool. Eur J Vasc Endovasc Surg. 2020. PMID: 32089508. doi:10.1016/j.ejvs.2020.01.021
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。