DeepSeek-R1在眼科的表现:临床决策与成本效益评估
本横断面研究从StatPearls收集10个眼科亚专科共300例临床病例,每例含一道诊断或管理选择题,比较DeepSeek-R1与OpenAI o1的表现和成本效益。DeepSeek-R1与OpenAI o1总体正确率均为82.0%(n=246/300;95%CI:77.3至85.9),两模型间一致性总体为中等(κ=0.503,p<0.001),DeepSeek-R1每次查询的token相关费用较OpenAI o1降低约15倍。
要点
- DeepSeek-R1与OpenAI o1总体正确率均为82.0%(n=246/300;95%CI:77.3至85.9)。
- 亚专科表现数值有差异,但各比较均未达统计学显著(p>0.05)。
- 两模型正确诊断一致性总体为中等(κ=0.503,p<0.001)。
核心结论
- DeepSeek-R1在眼科各亚专科的诊断和管理表现与OpenAI o1相当,同时显著降低成本。
- 这些结果支持将其作为专有模型的一种成本效益好的开放权重替代方案。
研究设计与样本
- 横断面研究,从StatPearls收集覆盖10个眼科亚专科的300例临床病例,每例含一道诊断或管理选择题。
- DeepSeek-R1通过其公开聊天界面访问,OpenAI o1通过其应用程序编程接口以标准化温度0.3查询。
- 两个模型均使用plan-and-solve+提示,表现以正确答案比例计算,采用McNemar检验比较配对数据,通过Cohen's kappa评估模型间正确诊断一致性。
- 基于token的成本分析用于估算各模型规模化运行(包括输入提示和模型生成输出)的比较支出。
主要发现
- DeepSeek-R1与OpenAI o1总体表现相同,均为82.0%(n=246/300;95%CI:77.3至85.9)。
- 亚专科特异性分析显示表现存在数值差异,但这些比较均未达统计学显著(p>0.05)。
- 两模型表现一致性总体为中等(κ=0.503,p<0.001),屈光管理/干预病例为高度一致(κ=0.698,p<0.001),视网膜/玻璃体病例为中等一致(κ=0.561,p<0.001),眼病理/肿瘤病例为中等一致(κ=0.495,p<0.01)。
- 成本分析表明,相同工作负载下使用DeepSeek-R1较OpenAI o1每次查询的token相关费用降低约15倍。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND/AIMS: To compare the performance and cost-effectiveness of DeepSeek-R1 with OpenAI o1 in diagnosing and managing ophthalmology clinical cases. METHODS: In this cross-sectional study, a total of 300 clinical cases spanning 10 ophthalmology subspecialties were collected from StatPearls, each with a multiple-choice question on diagnosis or management. DeepSeek-R1 was accessed through its public chat interface, while OpenAI o1 was queried via its Application Programming Interface with a standardised temperature of 0.3. Both models were prompted using plan-and-solve+. Performance was calculated as the proportion of correct answers. McNemar's test was employed to compare the two models' performance on paired data. Intermodel agreement for correct diagnoses was evaluated via Cohen's kappa. Token-based cost analyses were performed to estimate the comparative expenditures of running each model at scale, including input prompts and model-generated output. RESULTS: DeepSeek-R1 and OpenAI o1 achieved an identical overall performance of 82.0% (n=246/300; 95% CI: 77.3 to 85.9). Subspecialty-specific analysis revealed numerical variation in performance, though none of these comparisons reached statistical significance (p>0.05). Agreement in performance between the models was moderate overall (κ=0.503, p<0.001), with substantial agreement in refractive management/intervention (κ=0.698, p<0.001) and moderate agreement in retina/vitreous (κ=0.561, p<0.001) and ocular pathology/oncology (κ=0.495, p<0.01) cases. Cost analysis indicated an approximately 15-fold reduction in per-query, token-related expenses when using DeepSeek-R1 vs OpenAI o1 for the same workload. CONCLUSIONS: DeepSeek-R1 shows strong diagnostic and management performance comparable to OpenAI o1 across ophthalmic subspecialties, while significantly reducing costs. These results support its use as a cost-effective, open-weight alternative to proprietary models.
引用
- Mikhail D, Farah A, Milad J, et al. Performance of DeepSeek-R1 in ophthalmology: an evaluation of clinical decision-making and cost-effectiveness. Br J Ophthalmol. 2025. PMID: 40701781. doi:10.1136/bjo-2025-327360
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。