不良事件证据合成实践中数据提取的有效性:可重复性研究
该可重复性研究检索PubMed 2015年1月1日至2020年1月1日发表的系统评价,纳入201篇系统评价、829对Meta分析、涉及10386项随机对照试验,由4名作者按原始数据源复核数据提取。1762项(17.0%)试验的数据提取无法重现,554项(66.8%)Meta分析至少有一项试验数据提取错误,171项(85.1%)系统评价至少有一项Meta分析存在数据提取错误;这些错误可改变效应方向或P值显著性,作者认为亟需实施指南以提高数据提取有效性。
要点
- 10386项随机对照试验中1762项(17.0%)的数据提取无法重现。
- 829项Meta分析中554项(66.8%)至少有一项试验存在数据提取错误。
- 201项系统评价中171项(85.1%)至少有一项Meta分析存在数据提取错误。
核心结论
- 不良事件系统评价在数据提取可重复性方面可能存在严重问题,这些错误可能误导结论。
- 作者认为亟需实施指南以帮助未来系统评价作者提高数据提取的有效性。
研究设计与样本
- 可重复性研究,检索PubMed 2015年1月1日至2020年1月1日发表的合格系统评价。
- 由4名作者从系统评价中提取随机对照试验的元数据,再参照原始数据源(如全文和ClinicalTrials.gov)重现这些Meta分析所用数据。
- 纳入标准为基于随机对照试验的医疗保健干预系统评价、仅报告安全性结局、至少有一对Meta分析纳入5项或以上随机对照试验、且每项试验干预组和对照组的事件数和样本量2×2表数据可得。
- 共纳入201项系统评价、829对Meta分析,涉及10386项随机对照试验。
主要发现
- 10386项试验中1762项(17.0%)的数据提取无法重现。
- 829项Meta分析中554项(66.8%)至少有一项随机对照试验存在数据提取错误;201项系统评价中171项(85.1%)至少有一项Meta分析存在数据提取错误。
- 最常见的数据提取错误类型为数值错误(49.2%,867/1762)和模糊错误(29.9%,526/1762),主要由结局定义模糊所致,其后依次为零假设错误、错误识别和错配错误。
- 在288项Meta分析中分析错误影响:数据提取错误导致10项(3.5%)Meta分析改变效应方向、19项(6.6%)Meta分析改变P值显著性。
- 存在两种或以上不同类型错误的Meta分析比仅有一种类型错误者更易发生上述改变:中度改变为39项中11项(28.2%)vs 249项中26项(10.4%),P=0.002;大幅改变为39项中5项(12.8%)vs 249项中8项(3.2%),P=0.01。
临床怎么用
- 作者认为亟需实施指南以帮助未来系统评价作者提高数据提取的有效性。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: To investigate the validity of data extraction in systematic reviews of adverse events, the effect of data extraction errors on the results, and to develop a classification framework for data extraction errors to support further methodological research. DESIGN: Reproducibility study. DATA SOURCES: PubMed was searched for eligible systematic reviews published between 1 January 2015 and 1 January 2020. Metadata from the randomised controlled trials were extracted from the systematic reviews by four authors. The original data sources (eg, full text and ClinicalTrials.gov) were then referred to by the same authors to reproduce the data used in these meta-analyses. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: Systematic reviews were included when based on randomised controlled trials for healthcare interventions that reported safety as the exclusive outcome, with at least one pair meta-analysis that included five or more randomised controlled trials and with a 2×2 table of data for event counts and sample sizes in intervention and control arms available for each trial in the meta-analysis. MAIN OUTCOME MEASURES: The primary outcome was data extraction errors summarised at three levels: study level, meta-analysis level, and systematic review level. The potential effect of such errors on the results was further investigated. RESULTS: 201 systematic reviews and 829 pairwise meta-analyses involving 10 386 randomised controlled trials were included. Data extraction could not be reproduced in 1762 (17.0%) of 10 386 trials. In 554 (66.8%) of 829 meta-analyses, at least one randomised controlled trial had data extraction errors; 171 (85.1%) of 201 systematic reviews had at least one meta-analysis with data extraction errors. The most common types of data extraction errors were numerical errors (49.2%, 867/1762) and ambiguous errors (29.9%, 526/1762), mainly caused by ambiguous definitions of the outcomes. These categories were followed by three others: zero assumption errors, misidentification, and mismatching errors. The impact of these errors were analysed on 288 meta-analyses. Data extraction errors led to 10 (3.5%) of 288 meta-analyses changing the direction of the effect and 19 (6.6%) of 288 meta-analyses changing the significance of the P value. Meta-analyses that had two or more different types of errors were more susceptible to these changes than those with only one type of error (for moderate changes, 11 (28.2%) of 39 v 26 (10.4%) 249, P=0.002; for large changes, 5 (12.8%) of 39 v 8 (3.2%) of 249, P=0.01). CONCLUSION: Systematic reviews of adverse events potentially have serious issues in terms of the reproducibility of the data extraction, and these errors can mislead the conclusions. Implementation guidelines are urgently required to help authors of future systematic reviews improve the validity of data extraction.
引用
- Xu C, Yu T, Furuya-Kanamori L, et al. Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study. BMJ. 2022. PMID: 35537752. doi:10.1136/bmj-2021-069155
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。