语言变体对两种大语言模型牙髓病诊断测试的影响
GPT-5 Plus在基准集准确率80%高于Gemini的66%(P=0.0066),但语言变体尤其扰动使其性能显著下降(P=0.003),Gemini则表现稳定。
要点
- GPT-5 Plus基准集准确率80%,显著高于Gemini 2.5 Flash的66%。
- 扰动使GPT-5 Plus性能显著下降(P=0.003),Gemini对各变体表现稳定。
- 定性分析中Gemini获得更多「好」评,但样本量有限需谨慎解读。
核心结论
- GPT-5 Plus在基准数据集上优于Gemini 2.5 Flash,但对语言变体敏感。
- 扰动显著降低GPT-5 Plus的表现,提示需进一步研究语言现象对模型退化的影响。
研究设计与样本
- 两种模型使用标准化提示呈现病例,计算准确率/召回率与F-1分数。
- 模型间与模型内一致性用Cohen's κ分析,配对差异用McNemar检验,显著性p<0.05。
- 对部分样本进行描述性定性分析,采用3点Likert量表评估推理质量。
主要发现
- GPT-5 Plus在基准数据集上准确率为80%,Gemini 2.5 Flash为66%(McNemar p=0.0066)。
- 引入语言变体后GPT-5 Plus性能下降,扰动影响最显著(McNemar p=0.003)。
- Gemini 2.5 Flash在基准集表现较低,但在所有变换下决策模式均匀,无显著进一步下降。
- 定性分析中Gemini 2.5 Flash获评「好」的比例为原始数据集8/10(80%)、语言变体7/10(70%),均高于GPT-5 Plus。
临床怎么用
- 在牙髓病智能诊断中,语言变体可能影响模型稳定性,需对输入表述进行标准化。
局限性
- 定性分析为描述性且样本量有限,结果需谨慎解读。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
AIM: To assess the diagnostic performance of two language models, GPT-5 Plus and Gemini 2.5 Flash using a curated benchmark dataset of unseen endodontic and restorative dentistry related clinical case scenarios and the linguistic variations introduced around the original dataset. Additionally, a descriptive qualitative analysis was performed on a subset of cases to evaluate the quality of reasoning generated by both models. METHODOLOGY: One hundred single best answer MCQs were generated using standardised resources, constituting a benchmark dataset. Controlled linguistic variations were introduced around the original dataset; paraphrasing (sentence/clause rewording), perturbation (token-level substitutions), and permutation (answer-order shuffle). These case scenarios were presented to both models using a standardised prompt, and the performance metrics (accuracy/recall, F-1 score) were computed. Agreement between and within models was analysed using Cohen's κ, while paired differences were evaluated using McNemar's test with a significant p-value < 0.05. Qualitative analysis was performed on a subset of the total sample, and the responses were evaluated on a 3-point Likert scale. RESULTS: GPT-5 Plus achieved 80% accuracy on benchmark dataset compared to 66% for Gemini 2.5 Flash (McNemar's p-value = 0.0066). When linguistic variations were introduced, the performance of GPT-5 Plus declined, with perturbation having the most significant effect (McNemar's p-value = 0.003). Gemini 2.5 Flash, on the other hand, though inferior initial performance on benchmark dataset, maintained uniform decision patterns across all transformations with no significant drop further. The descriptive qualitative analysis demonstrated an overall higher proportion of responses rated as good (8/10, 80% for original dataset; 7/10, 70% for linguistic variations) for Gemini 2.5 Flash as opposed to GPT-5 Plus. CONCLUSION: GPT-5 Plus outperformed Gemini 2.5 Flash on benchmark dataset; however, it was sensitive to linguistic variations. Perturbation negatively influenced the performance of GPT-5 Plus, emphasising the need to further investigate the linguistic phenomenon that may have affected the model's degradation. Additionally, the descriptive qualitative analysis demonstrated relatively higher performance for Gemini 2.5 Flash compared to GPT-5 Plus on the original dataset and across linguistic variations. However, owing to the descriptive nature of findings and limited sample size, the results should be interpreted with caution.
引用
- Batool I, Naved N, Umer F, et al. Testing LLM Diagnostics in Endodontics: The Impact of Linguistic Variation on Unseen Cases. Int Endod J. 2026. PMID: 41645564. doi:10.1111/iej.70109
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。