文献速递美容整形

大语言模型与Web of Science腹壁整形文献计量对比

Benchmarking Large Language Models Against Web of Science: A Comparative Bibliometric Analysis on Panniculectomy.
PubMed 文献 · Ann Plast Surg · 2026-03-30

该研究以Web of Science手工提取的50篇最高被引腹壁整形(panniculectomy)文献为参照,比较ChatGPT-4o、Deepseek与Claude 3.7生成同类文献列表的准确性。结果显示各模型与人工列表重叠有限(ChatGPT-4o 14.00%、Claude 3.7 4.00%、Deepseek 0.00%,P<0.001),且普遍存在虚构与幻觉文献,ChatGPT-4o表现相对最好但仍有明显局限。

要点

  • LLM生成的文献计量数据与WoS人工数据重叠低,不能替代数据库检索。
  • ChatGPT-4o准确率34.00%,Deepseek 100.00%为幻觉文献。
  • 文献计量分析仍应以Web of Science为金标准。

核心结论

  • 当前大语言模型难以准确复现文献计量数据,ChatGPT-4o表现最佳但仍存在重大局限,WoS仍为金标准,LLM输出在文献计量分析中应谨慎对待。

研究设计与样本

  • 研究从Web of Science手工提取50篇最高被引腹壁整形文献作为参照数据集,并分别提示ChatGPT-4o、Deepseek和Claude 3.7 Sonnet生成各自的50篇最高被引文献列表进行比较。

主要发现

  • 人工数据集总被引2494次(密度49.8),ChatGPT-4o、Deepseek和Claude 3.7分别产生2111(42.2)、4736(94.7)和8592(171.8)次引用。
  • 与人工列表重叠率分别为ChatGPT-4o 14.00%、Claude 3.7 4.00%、Deepseek 0.00%(P<0.001)。
  • Plastic and Reconstructive Surgery在所有输出中均为最高被引期刊。
  • 独立作者数:人工241、ChatGPT-4o 114、Deepseek 72、Claude 3.7 129。
  • 文章准确性:ChatGPT-4o 34.00%准确、26.00%虚构、40.00%幻觉;Claude 3.7 4.00%准确、26.00%虚构、70.00%幻觉;Deepseek 100.00%幻觉(P<0.001)。

临床怎么用

  • 整形外科医生进行文献计量或学术写作时应以Web of Science等数据库为准,LLM输出仅作参考并需逐条核验。
  • 对LLM生成的参考文献需警惕虚构与幻觉,避免直接引用未核实条目。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: As large language models (LLMs) grow in sophistication, their potential role in scientific writing is being explored with growing interest and caution. However, LLMs vary in their performance, contextual accuracy, and reliability. This study compares the outputs of 3 leading LLMs (ChatGPT-4o, Deepseek, and Claude 3.7) against a manually curated bibliometric analysis of the most highly cited panniculectomy articles. METHODS: The 50 most highly cited panniculectomy publications were manually extracted from Web of Science (WoS) to serve as a reference data set. ChatGPT-4o, Deepseek, and Claude 3.7 Sonnet were each prompted to generate their own list of the 50 most cited panniculectomy articles. Outputs were compared across citation totals and averages, publication year trends, journal distribution, author co-occurrence, and article authenticity. RESULTS: The manual data set totaled 2494 citations (density: 49.8). ChatGPT-4o, Deepseek, and Claude 3.7 produced 2111 (42.2), 4736 (94.7), and 8592 (171.8) citations, respectively. Overlap with the manual list was limited: ChatGPT-4o (14.00%), Claude 3.7 (4.00%), Deepseek (0.00%) ( P <0.001). "Plastic and Reconstructive Surgery" was the most cited journal across all outputs. Unique authors: manual (241), ChatGPT-4o (114), Deepseek (72), and Claude 3.7 (129). Article accuracy: ChatGPT-4o had 34.00% accurate, 26.00% confabulated, and 40.00% hallucinated articles. Claude 3.7: 4.00% accurate, 26.00% confabulated, and 70.00% hallucinated. Deepseek: 100.00% hallucinated ( P <0.001). Year trends and journal representation varied notably from the manual set. CONCLUSIONS: Current LLMs struggle to replicate accurate bibliometric data. ChatGPT-4o performed best but still showed major limitations. WoS remains the gold standard, and LLM-generated outputs should be treated cautiously in bibliometric analyses.

来源:Ann Plast Surg,查看原文。版权归原作者所有。

引用

  1. Mangal R, Jessup M, Desai A, et al. Benchmarking Large Language Models Against Web of Science: A Comparative Bibliometric Analysis on Panniculectomy. Ann Plast Surg. 2026. PMID: 41910547. doi:10.1097/SAP.0000000000004730

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。