评估检索增强大语言模型在牙颈部外吸收中的应用:Gemini与NotebookLM的比较研究
本研究由三名牙髓病学专家基于既定资料开发了46个关于牙颈部外吸收的二分临床问题,每个问题通过三个独立账户分别提交给Google Gemini(GG)和NotebookLM(NLM),共获得276条回答。三名牙髓病学专家独立评估所有回答的准确性和一致性。结果显示GG准确率89%(41/46;95% CI,76.96-95.27),一致性93%(κ=0.89;P<.001);NLM准确率96%(44/46;95% CI,85.47-98.79),一致性相同(κ=0.90;P<.001)。两种模型在准确性和一致性上无显著差异。结论认为NLM和GG均表现出高水平的准确性和一致性,尽管NLM略优,但检索增强并未显著提升对结构化临床任务的回答。
要点
- NLM在牙颈部外吸收临床问题上准确率96%,GG为89%。
- 两种模型一致性均高(κ分别为0.90和0.89),无显著差异。
- 检索增强未显著提升结构化临床任务的回答表现。
核心结论
- NLM和GG在回答牙颈部外吸收临床问题时均表现出高准确性和一致性,但检索增强未显著增强回答。
研究设计与样本
- 由三名牙髓病学专家基于既定资料开发46个二分临床问题,每个问题通过三个独立账户提交给GG和NLM,共276条回答,由三名牙髓病学专家独立评估。
主要发现
- GG准确率为89%(41/46;95% CI,76.96-95.27),一致性为93%(κ=0.89;P<.001)。
- NLM准确率为96%(44/46;95% CI,85.47-98.79),一致性相同(κ=0.90;P<.001)。
- 两种模型在准确性和一致性上无显著差异。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
INTRODUCTION: This study evaluated the accuracy and consistency of two large language models developed by Alphabet Inc., Google Gemini (GG), a base configuration, and NotebookLM (NLM), a document-grounded configuration, when answering clinical questions regarding external cervical resorption using a retrieval-augmented framework. METHODS: Forty-six dichotomous clinical questions related to external cervical resorption were developed by three academic endodontists based on established sources. Each question was submitted to GG and NLM using three independent user accounts, yielding 276 responses. The retrieval-augmented generation configuration was replicated by NLM, which was programmed to generate responses exclusively from the documents provided. Three endodontic experts independently evaluated all responses against predefined gold standard answers. Accuracy was defined as agreement with the gold standard; consistency referred to identical responses across the three trials. Statistical analyses included 95% confidence intervals (CIs) (Wald and Wilson), Fleiss' kappa, and Fisher's exact test. RESULTS: GG achieved an accuracy of 89% (41/46; 95% CI, 76.96-95.27) and a consistency rate of 93% (κ = 0.89; P < .001). NLM achieved an accuracy of 96% (44/46; 95% CI, 85.47-98.79) and the same consistency (κ = 0.90; P < .001). No significant differences occurred between the large language models for accuracy and consistency. CONCLUSIONS: The NLM and GG models exhibited a high level of accuracy and consistency. Although NLM had a slightly superior performance, retrieval augmentation did not significantly enhance the responses to structured clinical tasks.
引用
- Garcia-Font M, Dufey-Portilla N, Durán-Sindreu F, et al. Evaluating Retrieval-Augmented Large Language Models on External Cervical Resorption: A Comparative Study of Gemini and NotebookLM. J Endod. 2025. PMID: 41207474. doi:10.1016/j.joen.2025.10.016
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。