文献速递脑血管病神经免疫神经肌肉病

神经科专家群体预测三项随机试验结果的准确性

Wisdom of the expert crowd prediction of response for 3 neurology randomized trials.
PubMed 文献 · Neurology · 2020-06-16

该研究探索神经科专家联合预测试验主要终点的准确性。结果显示专家平均预测的准确性很少优于无信息预测算法,提示依赖简单方法整合专家意见需谨慎。

要点

  • 专家联合预测在卒中试验中准确性低于无信息预测算法。
  • 在MS和ALS试验中,专家联合预测与无信息预测准确性大致相当。
  • 仅18%的专家预测技能超过无信息预测。

核心结论

  • 专家平均判断很少优于无信息预测,但专家至少预见了试验中观察到的效应可能性。
  • 若在不同试验样本中重复, findings 提示不应依赖简单方法整合专家意见进行研究和政策决策。

主要发现

  • 卒中试验:专家联合预测的合并Brier分数=0.340(95% sPI 0.340至0.340),无信息预测为0.185。
  • MS研究:专家联合预测的合并Brier分数=0.107(95% CI 0.081至0.133),无信息预测为0.098。
  • ALS研究:专家联合预测的合并Brier分数=0.090(95% CI 0.081至0.185),无信息预测为0.090。
  • 44%的专家个体预测的95% sPI包含实际试验结果。
  • 仅18%的专家表现出超过无信息预测的预测技能。

关键数值与推荐

  • 主要结局为平均预测的准确性,使用有序Brier分数测量,并与提供非承诺性预测的算法比较。

临床怎么用

  • 临床决策中应谨慎对待简单整合的专家意见预测。

局限性

  • 探索性研究,样本量有限,需在不同试验样本中重复验证。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: To explore the accuracy of combined neurology expert forecasts in predicting primary endpoints for trials. METHODS: We identified one major randomized trial each in stroke, multiple sclerosis (MS), and amyotrophic lateral sclerosis (ALS) that was closing within 6 months. After recruiting a sample of neurology experts for each disease, we elicited forecasts for the primary endpoint outcomes in the trial placebo and treatment arms. Our main outcome was the accuracy of averaged predictions, measured using ordered Brier scores. Scores were compared against an algorithm that offered noncommittal predictions. RESULTS: Seventy-one neurology experts participated. Combined forecasts of experts were less accurate than a noncommittal prediction algorithm for the stroke trial (pooled Brier score = 0.340, 95% subjective probability interval [sPI] 0.340 to 0.340 vs 0.185 for the uninformed prediction), and approximately as accurate for the MS study (pooled Brier score = 0.107, 95% confidence interval [CI] 0.081 to 0.133 vs 0.098 for the noncommittal prediction) and the ALS study (pooled Brier score = 0.090, 95% CI 0.081 to 0.185 vs 0.090). The 95% sPIs of individual predictions contained actual trial outcomes among 44% of experts. Only 18% showed prediction skill exceeding the noncommittal prediction. Independent experts and coinvestigators achieved similar levels of accuracy. CONCLUSION: In this first-of-kind exploratory study, averaged expert judgments rarely outperformed noncommittal forecasts. However, experts at least anticipated the possibility of effects observed in trials. Our findings, if replicated in different trial samples, caution against the reliance on simple approaches for combining expert opinion in making research and policy decisions.

来源:Neurology,查看原文。版权归原作者所有。

引用

  1. Atanasov P, Diamantaras A, MacPherson A, et al. Wisdom of the expert crowd prediction of response for 3 neurology randomized trials. Neurology. 2020. PMID: 32546652. doi:10.1212/WNL.0000000000009819

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。