指南与共识

神经外科住院医师申请中标准化推荐信评分量表的考察

An Examination of Standardized Letters of Recommendation Rating Scales Among Neurosurgical Residency Candidates During the 2020-2021 Application Cycle.
指南 / 共识 · Neurosurgery · 2021-01-01

本研究分析2020至2021申请周期内提交至单一神经外科住院医师培训项目的全部标准化推荐信(SLOR),记录7个胜任力域及总体评级并按推荐人学术类别分层。结果显示约58.3%的神经外科主任信件将申请人评为各项胜任力前5%,所有评级类别的评分者间信度均较差(总体α=0.18),提示首版SLOR受评分膨胀与信度不足限制。

要点

  • SLOR首版存在明显评分膨胀,多数信件将申请人评为前5%。
  • 各胜任力域及总体评级的评分者间信度均较差(总体α=0.18)。
  • 若要使SLOR提供有意义信息,未来申请周期需加以改进。

核心结论

  • 首版神经外科SLOR因显著评分膨胀和评分者间信度差而实用性受限,需改进方能在未来申请周期提供有意义信息。

研究设计与样本

  • 横断面研究,分析2020至2021申请周期提交至单一神经外科住院医师培训项目的全部SLOR,记录7个胜任力域与总体评级并按推荐人学术类别分层,用Krippendorff's alpha评估评分者间信度。

主要发现

  • 393份申请中298份(76%)附有至少一封SLOR。
  • 约58.3%由神经外科主任撰写的信件将申请人评为各项胜任力前5%。
  • 约44.4%的项目主任信件将申请人评为前5%,其他神经外科医师与普外科评估者的信件分别为73.2%和81.4%。
  • 所有评级类别评分者间信度均较差(<0.33),总体α=0.18。

关键数值与推荐

  • 7个胜任力域及总体评级均纳入分析。
  • 评分者间信度以Krippendorff's alpha评估,总体α=0.18。

临床怎么用

  • 解读神经外科SLOR时需警惕评分膨胀,不应仅凭前5%等等级区分申请人。
  • 因评分者间信度差,SLOR目前不宜作为客观比较申请人的主要依据。

局限性

  • 仅分析单一神经外科住院医师培训项目在一个申请周期内收到的SLOR,样本来源单一。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Letters of recommendation (LORs) are historically an important, though subjective, component of the neurosurgery residency application process. Standardized LORs (SLORs) were introduced during the 2020 to 2021 application cycle. The intent of SLORs is to allow objective comparison of applicants and to reduce bias. OBJECTIVE: To examine the utility of SLORs during this application cycle. We hypothesized that "grade inflation" and poor inter-rater reliability, as described by other specialties using SLORs, would limit the utility of SLORs in their current form. METHODS: This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2020 to 2021 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of letter writer. Inter-rater reliability was evaluated using Krippendorff's alpha. RESULTS: One or more SLORs was submitted as part of 298 of 393 applications (76%). Approximately 58.3% of letters written by neurosurgery chairpersons rated a given applicant as being within the top 5% across all competencies. Approximately 44.4% of program director letters similarly rated applicants as amongst the top 5%, while 73.2% and 81.4% of letters by other neurosurgeons and general surgery evaluators, respectively, rated applicants in the top 5%. Inter-rater reliability was poor (<0.33) in all rating categories, including overall (α = 0.18). CONCLUSION: The utility of the first iteration of SLORs in neurosurgery applications is undermined by significant "grade inflation" and poor inter-rater reliability. Improvements are necessary for SLORs if they are to provide meaningful information in future application cycles.

来源:Neurosurgery,查看原文。版权归原作者所有。

引用

  1. Feldman MJ, Ortiz AV, Roth SG, et al. An Examination of Standardized Letters of Recommendation Rating Scales Among Neurosurgical Residency Candidates During the 2020-2021 Application Cycle. Neurosurgery. 2021. PMID: 34624075. doi:10.1093/neuros/nyab346

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。