大型语言模型在口腔修复学与牙髓病学中的准确性、一致性和语境理解能力
本研究从法国牙体牙髓病学教学档案中选取517道多选题和539条定义,评估ChatGPT-3.5、ChatGPT-4、Claude-3和Mistral 7B四种大型语言模型的准确性、一致性和语境理解能力。ChatGPT-4和Claude-3的准确性和可重复性显著高于Mistral 7B,ChatGPT-4稳健性最佳。提供完整课程章节后准确性普遍提升但跨主题不一致。
要点
- ChatGPT-4和Claude-3的准确性和可重复性显著高于Mistral 7B。
- ChatGPT-4在应对改写问题时的稳健性表现最佳。
- 提供课程章节可普遍提升回答准确性,但各主题间改善不一致。
核心结论
- 即使是最先进的大型语言模型如ChatGPT-4和Claude-3也仅达到中等表现,且因稳健性不一致需谨慎使用。
- 未来研究应聚焦于整合经过验证的内容并改进提示工程,以提升大型语言模型的教育和临床效用。
研究设计与样本
- 数据集提取自法国牙体牙髓病学教学者全国学院(CNEOC)的国家教育档案,涵盖牙科住院医师申请者参考手册的全部章节。
- 经三名独立学术专家审核后选取多选题,共纳入517道多选题和539条定义。
- 评估四种大型语言模型:ChatGPT-3.5、ChatGPT-4(OpenAI)、Claude-3(Anthropic)和Mistral 7B(Mistral AI)。
主要发现
- ChatGPT-4和Claude-3的准确性和可重复性显著高于Mistral 7B。
- ChatGPT-4表现出最佳的稳健性。
- 先进大型语言模型在呈现牙科内容方面显示高准确性,但在密切相关的概念上表现有差异。
- 提供课程章节后回答准确性普遍提升,但各主题间改善不一致。
关键数值与推荐
- 准确性通过将模型回答与专家提供的答案进行比较来确定。
- 一致性通过稳健性(对改写问题提供相同回答的能力)和可重复性(对同一问题提供相同回答的能力)来衡量。
- 语境理解通过模型正确分类问题和从定义推断术语的能力来评估。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: This study aimed to evaluate and compare the performance of several large language models (LLMs) in the context of restorative dentistry and endodontics, focusing on their accuracy, consistency, and contextual understanding. METHODS: The dataset was extracted from the national educational archives of the Collège National des Enseignants en Odontologie Conservatrice (CNEOC) and includes all chapters from the reference manual for dental residency applicants. Multiple-choice questions (MCQs) were selected following a review by three independent academic experts. Four LLMs were assessed: ChatGPT-3.5, ChatGPT-4 (OpenAI), Claude-3 (Anthropic), and Mistral 7B (Mistral AI). Model accuracy was determined by comparing responses with expert-provided answers. Consistency was measured through robustness (the ability to provide identical responses to paraphrased questions) and repeatability (the ability to provide identical responses to the same question). Contextual understanding was evaluated based on the model's ability to categorise questions correctly and infer terms from definitions. Additionally, accuracy was reassessed after providing the LLMs with the relevant full course chapter. RESULTS: A total of 517 MCQs and 539 definitions were included. ChatGPT-4 and Claude-3 demonstrated significantly higher accuracy and repeatability than Mistral 7B, with ChatGPT-4 showing the greater robustness. Advanced LLMs displayed high accuracy in presenting dental content, although performance varied on closely related concepts. Supplying course chapters generally improved response accuracy, though inconsistently across topics. CONCLUSION: Even the most advanced LLMs, such as ChatGPT-4 and Claude 3, achieve moderate performance and require cautious use due to inconsistencies in robustness. Future studies should focus on integrating validated content and refining prompt engineering to enhance the educational and clinical utility of LLMs. CLINICAL SIGNIFICANCE: The findings underscore the potential of advanced LLMs and context-based prompting in restorative dentistry and endodontics.
引用
- Lafourcade C, Kérourédan O, Ballester B, et al. Accuracy, consistency, and contextual understanding of large language models in restorative dentistry and endodontics. J Dent. 2025. PMID: 40246058. doi:10.1016/j.jdent.2025.105764
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。