自然语言处理有效辅助系统综述更新时的快速标题摘要筛选
本研究基于已发表系统综述的检索文献,训练并测试了一种集成四个预训练BERT模型的LightGBM自然语言处理模型,用于系统综述更新时的快速标题和摘要筛选。测试集包含947条引文,模型以96%灵敏度和78%特异度纳入340条、排除607条,若接受分类器评估可减少64.1%工作量,但会丢失180条合格引文中的8条(4%),且这8条经全文评估后均未最终纳入系统综述。结论认为该集成学习方法可有效辅助系统综述更新时的快速文献筛选。
要点
- NLP集成模型在测试集947条引文中达96%灵敏度、78%特异度。
- 接受模型评估可减少64.1%筛选工作量,但丢失4%合格引文。
- 丢失的8条引文经全文评估后均未最终纳入系统综述。
核心结论
- 使用集成学习方法的NLP技术可能有效辅助系统综述更新时的快速文献筛选。
研究设计与样本
- 基于已发表系统综述的检索文献,训练并测试NLP模型用于系统综述更新时的快速标题和摘要筛选。
- 模型为LightGBM,一种集成学习分类器,整合四个预训练BERT模型。
- 检索引文分为训练集和测试集,模型在训练集训练、测试集评估筛选性能。
- 检索引文的合格性由两名独立评审员判定,作为参考标准。
主要发现
- 测试集包含947条引文,模型纳入340条、排除607条。
- 模型达到96%灵敏度和78%特异度。
- 若接受分类器评估,评审员将丢失180条合格引文中的8条(4%),这8条经全文考虑后均未最终纳入系统综述。
- 接受分类器评估可减少64.1%的工作量。
临床怎么用
- 在系统综述更新时,可考虑使用集成学习NLP模型辅助快速标题和摘要筛选。
- 应用时需权衡工作量减少与合格引文丢失的风险。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND AND OBJECTIVE: To examine whether the use of natural language processing (NLP) technology is effective in assisting rapid title and abstract screening when updating a systematic review. STUDY DESIGN: Using the searched literature from a published systematic review, we trained and tested an NLP model that enables rapid title and abstract screening when updating a systematic review. The model was a light gradient boosting machine (LightGBM), an ensemble learning classifier which integrates four pretrained Bidirectional Encoder Representations from Transformers (BERT) models. We divided the searched citations into two sets (ie, training and test sets). The model was trained using the training set and assessed for screening performance using the test set. The searched citations, whose eligibility was determined by two independent reviewers, were treated as the reference standard. RESULTS: The test set included 947 citations; our model included 340 citations, excluded 607 citations, and achieved 96% sensitivity, and 78% specificity. If the classifier assessment in the case study was accepted, reviewers would lose 8 of 180 eligible citations (4%), none of which were ultimately included in the systematic review after full-text consideration, while decreasing the workload by 64.1%. CONCLUSION: NLP technology using the ensemble learning method may effectively assist in rapid literature screening when updating systematic reviews.
引用
- Qin X, Liu J, Wang Y, et al. Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews. J Clin Epidemiol. 2021. PMID: 33485929. doi:10.1016/j.jclinepi.2021.01.010
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。