生物医学BERT集成工具TITAN-SR在系统综述标题摘要筛选中优于主动学习与LLM聊天机器人:开发与外部验证
本研究开发了基于生物医学BERT模型的文献筛选工具TITAN-SR,并使用近2万篇已完成综述的762,934条引文-标签对进行训练。在3434篇测试综述中,TITAN-SR在99%召回率下中位特异度达0.902,中位AUC-ROC为0.974,优于四种商业LLM聊天机器人及GPT-4o-mini。在22篇时间独立的Cochrane综述外部验证中,TITAN-SR保留99.8%的纳入研究(95%CI 99.5%至100.0%),并在18/22篇综述中优于ASReview。
要点
- TITAN-SR在99%召回率下中位特异度0.902,中位AUC-ROC 0.974。
- 外部验证中保留99.8%纳入研究(95%CI 99.5%至100.0%)。
- TITAN-SR在18/22篇综述中优于ASReview,中位特异度差值+0.145。
核心结论
- 基于生物医学BERT模型的专用筛选工具TITAN-SR在系统综述标题摘要筛选中优于未针对任务训练的ASReview和商业LLM聊天机器人。
- TITAN-SR在时间外部验证中表现出对模型开发期之后发表综述的泛化能力。
研究设计与样本
- 从19,787篇已完成综述中组装762,934条引文-标签对,按预设的综述层面划分训练集和测试集。
- TITAN-SR结合PubMedBERT和BioLinkBERT读取综述的纳入排除标准与每条引文的标题和摘要,训练时假阴性惩罚为假阳性的10倍。
- 与四种零样本使用的聊天机器人(Claude Sonnet 4、GPT-4o、DeepSeek-V3、Gemini 2.0 Flash)在分层500篇综述基准上比较,与GPT-4o-mini在全部3434篇可评估测试综述上比较。
- 与ASReview在22篇时间独立的Cochrane综述(142,504条记录)上比较,ASReview获得已知纳入研究20%的预热。
主要发现
- 在3434篇测试综述中,TITAN-SR在99%召回率下中位特异度为0.902,中位AUC-ROC为0.974。
- TITAN-SR优于所有四种聊天机器人(Bonferroni校正配对Wilcoxon P ≤ 5 × 10-31)和GPT-4o-mini。
- 聊天机器人出现校准崩溃:无论准确率如何,82%-99%的置信度分数≥0.9。
- 在22篇外部综述上,中位AUC为0.976,95%召回率下特异度为0.897。
- 在内部验证集固定的阈值下,TITAN-SR满足预注册主要标准,保留99.8%的纳入研究(95%CI 99.5%至100.0%)。
临床怎么用
- TITAN-SR可帮助综述作者优先阅读可能相关的文章,并作为第二筛选者使用,而非替代人工判断。
- 该工具运行于普通计算机,低成本,模型可向研究者申请获取。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: Machine-learning tools for literature screening have existed for a decade, yet adoption remains limited: active-learning needs hundreds of decisions before prioritizing well, and commercial large language model (LLM) chatbots are used without task-specific training. We aimed to develop TITAN-SR (Training Infrastructure for Automated Nomination in Systematic Reviews), a screening tool built on biomedical Bidirectional Encoder Representations from Transformers (BERT) models, and to compare it with the active-learning tool ASReview and commercial LLM chatbots. STUDY DESIGN AND SETTING: We assembled 762,934 citation-label pairs from 19,787 completed reviews and trained TITAN-SR on a prespecified review-level split, testing on held-out reviews. The tool combines biomedical BERT models (PubMedBERT and BioLinkBERT) that read a review's eligibility criteria with each citation's title and abstract; training penalized false negatives 10 times more heavily than false positives. We compared TITAN-SR with four chatbots used zero-shot on a stratified 500 review benchmark (Claude Sonnet 4, GPT-4o, DeepSeek-V3, Gemini 2.0 Flash), with GPT-4o-mini on all 3434 evaluable test reviews, and with ASReview on 22 temporally independent Cochrane reviews (142,504 records); ASReview received a warm-up of 20% of known includes. RESULTS: Across 3434 test reviews, TITAN-SR achieved a median specificity at 99% recall of 0.902 and median area under the receiver operating characteristic curve (AUC-ROC) of 0.974. It outperformed all four chatbots (Bonferroni-corrected paired Wilcoxon P ≤ 5 × 10-31) and GPT-4o-mini. The chatbots showed calibration collapse: 82%-99% of confidence scores were ≥0.9 regardless of accuracy. On the 22 external reviews, performance was virtually identical (median AUC 0.976; specificity at 95% recall 0.897), and at a threshold fixed on the internal validation split, TITAN-SR met the preregistered primary criterion, retaining 99.8% of included studies (95% confidence interval [CI] 99.5% to 100.0%). It outperformed ASReview on 18 of 22 reviews (median paired difference in specificity at 95% recall +0.145, 95% CI +0.101 to +0.201). CONCLUSION: A purpose-built screening tool based on biomedical BERT models, developed from nearly 20,000 completed reviews, outperformed ASReview and commercial LLM chatbots evaluated without task-specific training. Temporal validation on 22 held-out Cochrane reviews confirmed generalization to reviews published after the model development period. PLAIN LANGUAGE SUMMARY: Screening thousands of search results is one of the most time-consuming steps in a systematic review. We built Training Infrastructure for Automated Nomination in Systematic Review (TITAN-SR), a low-cost tool that runs on an ordinary computer and reads each article's title and summary together with the review's eligibility criteria to flag the articles most likely to be relevant. Across thousands of completed reviews, it kept about 99% of the relevant articles, while setting aside about 90% of the irrelevant ones, and in our evaluation, it identified relevant studies more effectively than two widely used alternatives (an active-learning tool and general purpose artificial intelligence chatbots). It is intended to help reviewers prioritize their reading and act as a second screener, not to replace human judgment; the model is available to researchers on request.
引用
- Pitre T, Zeraatkar D, Granton J, et al. A biomedical BERT ensemble (TITAN-SR) outperformed active-learning and large language model chatbots for systematic review title and abstract screening: development and external validation. J Clin Epidemiol. 2026. PMID: 42767575. doi:10.1016/j.jclinepi.2026.112514
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。