文本挖掘工具生成标题-摘要筛选工作量节省:与单人工筛选的性能评估对比
本研究以复发/难治性弥漫性大B细胞淋巴瘤治疗的系统评价(n=7,723)为对象,比较Abstrackr辅助筛选(单人筛选者与Abstrackr)与单人工筛选的性能。Abstrackr使第一阶段工作量减少67%(5.4天),灵敏度91%,假阴性率9%但无相关引文在全文筛选中被纳入;假阳性高(n=2,001)致特异度72%、精确率15.5%。结论认为Abstrackr辅助筛选节省工作量且未遗漏相关引文,但高估引文相关性可能增加全文筛选负担。
要点
- Abstrackr辅助筛选使第一阶段工作量减少67%(5.4天),灵敏度91%。
- 第一阶段假阴性率9%,但无相关引文在全文筛选中被纳入。
- 假阳性n=2,001,特异度72%、精确率15.5%,可能增加全文筛选负担。
核心结论
- Abstrackr辅助筛选节省了第一阶段工作量,且未以遗漏相关引文为代价;但Abstrackr高估引文相关性,可能对全文筛选的工作量产生负面影响。
研究设计与样本
- 基于复发/难治性弥漫性大B细胞淋巴瘤治疗的系统评价(n=7,723),将引文上传至Abstrackr,由一名人工筛选者筛选至预设最大预测评分0.39540,并与第二名人工筛选者(在Covidence中筛选全部引文)的判断进行比较。
主要发现
- 与单人工筛选相比,Abstrackr使第一阶段工作量减少67%(5.4天)。
- 灵敏度高(91%)。
- 第一阶段假阴性率为9%,但那些引文在全文筛选后均未被纳入。
- 假阳性比例高(n=2,001),导致特异度低(72%)和精确率低(15.5%)。
关键数值与推荐
- 预设最大预测评分为0.39540。
- 灵敏度91%,特异度72%,精确率15.5%,假阴性率9%。
- 假阳性n=2,001。
- 工作量减少67%(5.4天)。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND AND OBJECTIVES: Text-mining tool, Abstrackr, may potentially reduce the workload burden of title and abstract screening (Stage 1), using screening prioritization and truncation. This study aimed to evaluate the performance of Abstrackr's text-mining functions ('Abstrackr-assisted screening'; screening undertaken by a single-human screener and Abstrackr) vs. Single-human screening. METHODS: A systematic review of treatments for relapsed/refractory diffuse large B cell lymphoma (n = 7,723) was used. Citations, uploaded to Abstrackr, were screened by a human screener until a pre-specified maximum prediction score of 0.39540 was reached. Abstrackr's predictions were compared with the judgments of a second, human screener (who screened all citations in Covidence). The performance metrics were sensitivity, specificity, precision, false negative rate, proportion of relevant citations missed, workload savings, and time savings. RESULTS: Abstrackr reduced Stage 1 workload by 67% (5.4 days), when compared with Single-human screening. Sensitivity was high (91%). The false negative rate at Stage 1 was 9%; however, none of those citations were included following full-text screening. The high proportion of false positives (n = 2,001) resulted in low specificity (72%) and precision (15.5%). CONCLUSION: Abstrackr-assisted screening provided Stage 1 workload savings that did not come at the expense of omitting relevant citations. However, Abstrackr overestimated citation relevance, which may have negative workload implications at full-text screening.
引用
- Carey N, Harte M, Mc Cullagh L, et al. A text-mining tool generated title-abstract screening workload savings: performance evaluation versus single-human screening. J Clin Epidemiol. 2022. PMID: 35654270. doi:10.1016/j.jclinepi.2022.05.017
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。