文献速递区域麻醉与超声

大语言模型生成麻醉学系统综述检索式的效用评估

Evaluating the utility of large language models in generating search strings for systematic reviews in anesthesiology: a comparative analysis of top-ranked journals.
PubMed 文献 · Reg Anesth Pain Med · 2026-05-05

该研究从Web of Science排名前10的麻醉学期刊中选取85篇系统综述,以其原始检索式检索PubMed的结果为基准,比较ChatGPT 4o与结构化模型Meta-Analysis Librarian生成的检索式。原始检索式中位检索率为65%(IQR 43%-81%),显著优于两个大语言模型组(p=0.001);Meta-Analysis Librarian中位检索率24%(IQR 13%-38%),优于ChatGPT 4o的6%(IQR 0%-14%)。

要点

  • 原始检索式检索率65%,显著优于大语言模型生成检索式(p=0.001)。
  • Meta-Analysis Librarian中位检索率24%,优于ChatGPT 4o的6%。
  • 大语言模型生成的检索式目前不宜替代人工制定的系统综述检索策略。

核心结论

  • 在PubMed检索研究中,原始检索式较大语言模型生成的检索式具有显著优势。
  • Meta-Analysis Librarian的检索表现明显优于ChatGPT 4o。
  • 大语言模型生成检索式的更广泛适用性仍需进一步研究,尤其是跨多个数据库时。

研究设计与样本

  • 从Web of Science排名前10的麻醉学期刊中选取85篇系统综述,提取其参考文献列表作为基准。
  • 以研究标题为输入,每篇系统综述生成4条检索式:3条由ChatGPT 4o使用通用提示生成,1条由Meta-Analysis Librarian按结构化PICO方法生成。
  • 每条检索式用于查询PubMed,并与原系统综述原始检索式在PubMed中检出的研究进行比较以评估检索准确性。
  • 对各个模型的表现进行统计分析比较。

主要发现

  • 原始检索式的检索率为65%(IQR 43%-81%),与两个大语言模型组在PubMed检出研究方面存在统计学差异(p=0.001)。
  • Meta-Analysis Librarian的中位检索率高于ChatGPT 4o:24%(IQR 13%-38%)对6%(IQR 0%-14%)。

关键数值与推荐

  • 原始检索式中位检索率65%(IQR 43%-81%)。
  • Meta-Analysis Librarian中位检索率24%(IQR 13%-38%)。
  • ChatGPT 4o中位检索率6%(IQR 0%-14%)。
  • 原始检索式与大语言模型组比较p=0.001。

临床怎么用

  • 麻醉学系统综述检索策略目前仍应以人工制定的原始检索式为准,不宜直接用大语言模型生成检索式替代。
  • 若尝试使用大语言模型辅助,结构化PICO式模型(Meta-Analysis Librarian)表现优于通用提示的ChatGPT 4o。

局限性

  • 大语言模型生成检索式的更广泛适用性尚需进一步研究评估。
  • 尤其需要在多个数据库环境下验证,而非仅限PubMed。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: This study evaluated the effectiveness of large language models (LLMs), specifically ChatGPT 4o and a custom-designed model, Meta-Analysis Librarian, in generating accurate search strings for systematic reviews (SRs) in the field of anesthesiology. METHODS: We selected 85 SRs from the top 10 anesthesiology journals, according to Web of Science rankings, and extracted reference lists as benchmarks. Using study titles as input, we generated four search strings per SR: three with ChatGPT 4o using general prompts and one with the Meta-Analysis Librarian model, which follows a structured, Population, Intervention, Comparator, Outcome-based approach aligned with Cochrane Handbook standards. Each search string was used to query PubMed, and the retrieved results were compared with the PubMed retrieved studies from the original search string in each SR to assess retrieval accuracy. Statistical analysis compared the performance of each model. RESULTS: Original search strings demonstrated superior performance with a 65% (IQR: 43%-81%) retrieval rate, which was statistically different from both LLM groups in PubMed retrieved studies (p=0.001). The Meta-Analysis Librarian achieved a superior median retrieval rate to ChatGPT 4o (median, (IQR); 24% (13%-38%) vs 6% (0%-14%), respectively). CONCLUSION: The findings of this study highlight the significant advantage of using original search strings over LLM-generated search strings in PubMed retrieval studies. The Meta-Analysis Librarian demonstrated notable superiority in retrieval performance compared with ChatGPT 4o. Further research is needed to assess the broader applicability of LLM-generated search strings, especially across multiple databases.

来源:Reg Anesth Pain Med,查看原文。版权归原作者所有。

引用

  1. De Cassai A, Dost B, Karapinar YE, et al. Evaluating the utility of large language models in generating search strings for systematic reviews in anesthesiology: a comparative analysis of top-ranked journals. Reg Anesth Pain Med. 2026. PMID: 39828514. doi:10.1136/rapm-2024-106231

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。