专家脑电图医师识别癫痫发作及节律性和周期性模式的评分者间信度
该前瞻性分析纳入来自18家机构的30名临床神经生理学专科专家,独立判读2006至2020年马萨诸塞总医院的2711份脑电图中的50697个10秒片段,分类为癫痫发作、偏侧周期性放电、广泛性周期性放电、偏侧节律性δ活动、广泛性节律性δ活动或其他。总体两两评分者间信度中等(符合率52%,κ 42%),与小组共识的信度较高(符合率65%,κ 61%)。噪声-偏倚分析显示专家间差异主要源于决策阈值不同而非专业水平差异。
要点
- 专家识别癫痫发作及节律性和周期性模式的两两信度中等(符合率52%,κ 42%)。
- 专家与小组共识的信度较高(符合率65%,κ 61%)。
- 专家间评分差异主要归因于决策阈值变异而非专业知识差异。
核心结论
- 基于大样本多样本获得专家信度的精确估计,并提出解释专家间分歧来源的简约理论。
- 结果为自动化发作间期-发作期损伤连续体分类器需达到的专家级性能建立标准。
研究设计与样本
- 前瞻性分析,纳入来自18家机构的30名具有临床神经生理学亚专科培训的专家。
- 专家独立判读不同数量的10秒脑电图片段,片段来自2006至2020年马萨诸塞总医院临床指征脑电图。
- 共分析2711份脑电图,其中49%来自女性,中位(IQR)年龄55(41)岁。
- 专家共判读50697个脑电图片段,每位专家判读片段数中位数[范围]为6287.5[1002, 45267]。
主要发现
- 总体两两评分者间信度中等(符合率52%,κ 42%)。
- 总体与小组共识的信度较高(符合率65%,κ 61%)。
- 噪声-偏倚分析显示单一受试者工作特征曲线可解释专家假阳性与真阳性特征间的大部分变异(方差解释中位数[范围]95[93, 98]%)。
- 同一曲线也可解释专家精确度与敏感度特征间的大部分变异(方差解释中位数[范围]75[59, 89]%)。
- 专家间变异主要归因于决策阈值差异而非专业知识差异。
关键数值与推荐
- 专家将每个10秒脑电图片段分类为癫痫发作、偏侧周期性放电、广泛性周期性放电、偏侧节律性δ活动、广泛性节律性δ活动或其他。
- 主要结局指标为各分类的两两评分者间信度(专家两两间平均符合率)和与小组共识的信度(平均符合率)及超越机会的一致性(κ)。
- 次要结局为专家评分与小组共识的校准以及潜在特质分析。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND AND OBJECTIVES: The validity of brain monitoring using electroencephalography (EEG), particularly to guide care in patients with acute or critical illness, requires that experts can reliably identify seizures and other potentially harmful rhythmic and periodic brain activity, collectively referred to as "ictal-interictal-injury continuum" (IIIC). Previous interrater reliability (IRR) studies are limited by small samples and selection bias. This study was conducted to assess the reliability of experts in identifying IIIC. METHODS: This prospective analysis included 30 experts with subspecialty clinical neurophysiology training from 18 institutions. Experts independently scored varying numbers of ten-second EEG segments as "seizure (SZ)," "lateralized periodic discharges (LPDs)," "generalized periodic discharges (GPDs)," "lateralized rhythmic delta activity (LRDA)," "generalized rhythmic delta activity (GRDA)," or "other." EEGs were performed for clinical indications at Massachusetts General Hospital between 2006 and 2020. Primary outcome measures were pairwise IRR (average percent agreement [PA] between pairs of experts) and majority IRR (average PA with group consensus) for each class and beyond chance agreement (κ). Secondary outcomes were calibration of expert scoring to group consensus, and latent trait analysis to investigate contributions of bias and noise to scoring variability. RESULTS: Among 2,711 EEGs, 49% were from women, and the median (IQR) age was 55 (41) years. In total, experts scored 50,697 EEG segments; the median [range] number scored by each expert was 6,287.5 [1,002, 45,267]. Overall pairwise IRR was moderate (PA 52%, κ 42%), and majority IRR was substantial (PA 65%, κ 61%). Noise-bias analysis demonstrated that a single underlying receiver operating curve can account for most variation in experts' false-positive vs true-positive characteristics (median [range] of variance explained ([Formula: see text]): 95 [93, 98]%) and for most variation in experts' precision vs sensitivity characteristics ([Formula: see text]: 75 [59, 89]%). Thus, variation between experts is mostly attributable not to differences in expertise but rather to variation in decision thresholds. DISCUSSION: Our results provide precise estimates of expert reliability from a large and diverse sample and a parsimonious theory to explain the origin of disagreements between experts. The results also establish a standard for how well an automated IIIC classifier must perform to match experts. CLASSIFICATION OF EVIDENCE: This study provides Class II evidence that an independent expert review reliably identifies ictal-interictal injury continuum patterns on EEG compared with expert consensus.
引用
- Jing J, Ge W, Struck AF, et al. Interrater Reliability of Expert Electroencephalographers Identifying Seizures and Rhythmic and Periodic Patterns in EEGs. Neurology. 2022. PMID: 36460472. doi:10.1212/WNL.0000000000201670
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。