中文AI大语言模型在口腔黏膜病咨询中的评价
该研究调查200例口腔黏膜病患者使用AI咨询的现状,并由10名专科医师用QAMAI工具评价6种主流中文LLM对9个口腔白斑病临床问题的回答质量。结果显示仅37.5%曾使用AI,DeepSeek和腾讯混元质量最佳,但各模型来源引用得分普遍较低。
要点
- 仅37.5%口腔黏膜病患者曾用AI咨询,使用率与年轻及高学历显著相关。
- DeepSeek(25.4分)和腾讯混元(25.3分)回答质量最佳,评为「质量非常好」。
- 各模型在「来源与参考文献」维度普遍得分低,字节豆包可读性最佳。
核心结论
- 口腔黏膜病患者AI咨询使用率与满意度待提升,但未来使用意愿强烈。
- 6种主流中文LLM在口腔白斑病咨询中的专业信息质量与文本可读性存在显著差异,且普遍缺乏可靠的循证支持。
研究设计与样本
- 问卷调查200例于2025年11月在四川大学华西口腔医院口腔黏膜病科就诊的口腔黏膜病患者。
- 将9个口腔白斑病相关临床问题标准化后输入6种主流LLM平台获取回答。
- 由10名口腔黏膜病专科医师用QAMAI工具对回答的准确性、清晰度、相关性、完整性和实用性量化打分,并用ARC工具评估可读性。
主要发现
- 仅37.5%(75/200)曾使用AI进行口腔黏膜病相关咨询,AI使用率与年龄、受教育程度显著相关(P<0.001)。
- 仅40.0%(30/75)使用者对当前AI咨询较为满意,仅21.3%(16/75)会采纳AI治疗或护理建议,96.0%(72/75)对未来继续使用AI咨询表现出积极意愿。
- QAMAI总分显示DeepSeek(25.4分)和腾讯混元(25.3分)表现最佳,评价为「质量非常好」,其余模型为「质量良好」。
- 各模型在「来源与参考文献」维度普遍得分较低;ARC可读性分析显示字节豆包可读性最佳(加权总分0.511),DeepSeek和腾讯混元可读性相对较差(分别为0.358和0.369)。
关键数值与推荐
- QAMAI评价等级中DeepSeek(25.4分)与腾讯混元(25.3分)为「质量非常好」,其余模型为「质量良好」。
- ARC可读性加权总分字节豆包0.511,DeepSeek 0.358,腾讯混元0.369。
临床怎么用
- 口腔黏膜病患者对AI咨询使用率与满意度不高,但继续使用意愿强,提示可引导患者合理使用并加强科普。
- 不同LLM在口腔白斑病咨询中的专业质量与可读性差异明显,临床选用时应关注其来源引用不足的问题。
局限性
- 问卷仅纳入单中心就诊患者,AI使用行为与满意度的影响因素分析可能受样本来源限制。
- 质量评价仅针对口腔白斑病9个临床问题,结论外推至其他口腔黏膜病需谨慎。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Objective: To investigate the current application status and potential of artificial intelligence (AI) large language models (LLM) in oral mucosal disease consultation. Methods: A questionnaire survey was conducted to inform the utilization of AI for oral mucosal disease-related consultations among patients attending the Department of Oral Medicine, West China Hospital of Stomatology, Sichuan University in November 2025, and to compare the factors influencing AI usage behavior and satisfaction. Nine standardized clinical questions concerning the etiology, symptoms, treatment, care, and prognosis of oral leukoplakia (OLK) were put into major LLM platforms. The responses were quantitatively scored by ten oral medicine specialists for accuracy, clarity, relevance, completeness, and practicality using the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool. Concurrently, the readability of the responses was assessed using the Alpha Readability Chinese (ARC) tool. Results: A total of 200 patients with oral mucosal diseases were included. Only 37.5% (75/200) had ever used AI for related consultations. AI usage rate was significantly correlated with younger age and higher education level (P<0.001). Merely 40.0% (30/75) of users were relatively satisfied with current AI consultations, and only 21.3% (16/75) would adopt AI's treatment or care suggestions. However, 96.0% (72/75) expressed positive willingness to continue using AI for future consultations. Based on the QAMAI total scores for the nine typical OLK-related clinical questions, DeepSeek (25.4 points) and Tencent Hunyuan (25.3 points) performed best, rated as "very good quality", while the other models were rated "good quality." All models scored relatively low on the "sources and references" dimension. ARC readability analysis indicated that ByteDance Doubao had the best readability (weighted total score 0.511), while DeepSeek and Tencent Hunyuan had relatively poor readability (0.358 and 0.369, respectively). Conclusions: This study indicates that while current usage rates and satisfaction with AI consultation among patients with oral mucosal diseases need improvement, the future willingness to use it is strong. The systematic evaluation of six mainstream Chinese LLMs reveals significant disparities in their professional information quality and text readability for OLK consultation, alongside a prevalent lack of reliable evidence-based support. This underscores that enhancing the comprehensive quality of AI-generated responses is crucial for realizing its clinical application value. 目的: 探讨人工智能(AI)大语言模型(LLM)在口腔黏膜病疾病咨询中的应用现状与潜力。 方法: 通过问卷调查的形式了解2025年11月于四川大学华西口腔医院口腔黏膜病科就诊患者使用AI进行口腔黏膜病相关咨询的情况,并对AI使用行为和满意度的影响因素进行比较。将收集的口腔白斑病患者关注的病因、症状、治疗、护理、转归等方面的9个临床问题进行标准化后输入6种主流LLM平台(Kimi、字节豆包、通义千问、文心一言、腾讯混元、DeepSeek)获取回答,根据医学AI质量分析工具(QAMAI)的要求,由10名口腔黏膜病专科医师对获得回答的准确性、清晰度、相关性、完整性和实用性进行量化打分;同时,使用汉语文本可读性工具ARC评估回答的可读性。 结果: 本研究共纳入200例口腔黏膜病患者,仅37.5%(75/200)曾使用过AI进行口腔黏膜病相关咨询。AI使用率与年龄、受教育程度显著相关(P<0.001)。仅40.0%(30/75)的患者对当前AI咨询较为满意,仅21.3%(16/75)的患者会采纳AI的治疗或护理建议,而96.0%(72/75)的患者对未来继续使用AI进行咨询表现出积极意愿。与口腔白斑病相关的9个典型临床咨询问题的QAMAI总分显示,DeepSeek(25.4分)和腾讯混元(25.3分)表现最佳,评价等级为“质量非常好”,其余模型则被评为“质量良好”。在“来源与参考文献”维度,各模型普遍得分较低。ARC可读性分析显示,字节豆包的可读性最佳(加权总分0.511分),而DeepSeek和腾讯混元的可读性相对较差(分别为0.358和0.369分)。 结论: 当前口腔黏膜病患者对AI咨询的使用率与满意度均有待提升,但其未来使用意愿强烈;6种主流中文LLM在口腔白斑病咨询中的专业质量与文本可读性存在显著差异,且普遍缺乏可靠的循证支持,提示提升AI回答的综合质量是实现其临床应用价值的关键。.
引用
- Zhang HT, Sun WX, Li XY, et al. [Evaluation of Chinese artificial intelligence large language models in oral mucosal disease consultation]. Zhonghua Kou Qiang Yi Xue Za Zhi. 2026. PMID: 41786521. doi:10.3760/cma.j.cn112144-20251201-00482
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。