基于AI的烧伤图像评估:多模态大语言模型在重复推理研究中的可靠性与临床错误模式
本研究评估四种多模态大语言模型(GPT-5.4 Pro、Grok 4.1、Gemini 3.1 Pro、Claude Opus 4.6)对50张临床烧伤照片的烧伤深度分类和TBSA估计,采用每模型五次独立重复推理设计。结果显示深度准确率34.0±6.5%至76.4±6.8%,TBSA准确率32.8±9.4%至68.4±3.3%,运行间可靠性从轻微(κ=0.171)到几乎完美(κ=0.916),但无模型同时具备高准确率和高可靠性。
要点
- 四种多模态大语言模型对烧伤深度和TBSA的评估准确率差异大,深度准确率34.0%~76.4%,TBSA准确率32.8%~68.4%。
- 模型重复推理的可靠性从轻微(κ=0.171)到几乎完美(κ=0.916),单次查询无法反映响应不稳定性。
- 所有模型倾向于高估烧伤深度,甚至错误判定数据集中不存在的四度烧伤。
核心结论
- 多模态大语言模型在烧伤图像评估中表现出随机响应不稳定性,单次查询评估无法捕捉这种不一致性。
- 没有模型能同时实现高准确率和高可靠性,表明性能与一致性之间存在分离。
研究设计与样本
- 研究评估了四种多模态大语言模型:GPT-5.4 Pro、Grok 4.1、Gemini 3.1 Pro和Claude Opus 4.6。
- 使用50张临床烧伤照片,采用重复推理设计,每个模型独立运行五次。
主要发现
- 烧伤深度准确率范围为34.0±6.5%至76.4±6.8%,TBSA准确率范围为32.8±9.4%至68.4±3.3%。
- 运行间可靠性(Fleiss' κ)从轻微(κ=0.171)到几乎完美(κ=0.916)不等。
- 所有模型均倾向于高估烧伤深度,包括将图像判定为四度烧伤而数据集中并无四度烧伤。
- 误差方向分析显示模型特异性和任务依赖性偏倚,同一模型内存在相反模式。
- 数值与文本分类的内部一致性接近完美(99.6-100%),表明输出格式不变但存在系统性偏倚。
临床怎么用
- 在需要重复评估一致输出的临床工作流程中,应谨慎对待多模态大语言模型的单次查询结果,因其响应存在随机不稳定性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
Accurate assessment of burn depth and total body surface area (TBSA) is critical for clinical decision-making; however, it remains subjective and prone to interobserver variability. Multimodal large language models (MLLMs) are increasingly encountered in clinical contexts, but whether these systems can reliably assess burn images remains unclear. We evaluated four MLLMs (GPT-5.4 Pro, Grok 4.1, Gemini 3.1 Pro, and Claude Opus 4.6) on 50 clinical burn photographs using a repeated-inference design with five independent runs per model. Burn depth classification was assessed in numeric and text-based formats, alongside ordinal TBSA estimation. Performance varied across the models, with burn depth accuracy ranging from 34.0 ± 6.5% to 76.4 ± 6.8% and TBSA accuracy from 32.8 ± 9.4% to 68.4 ± 3.3%. Inter-run reliability (Fleiss' κ) ranged from slight (κ = 0.171) to almost perfect (κ = 0.916), demonstrating response variability not captured by single-query evaluations. Notably, no model combined high accuracy and high reliability, indicating a dissociation between performance and consistency. All models showed a tendency toward overestimation of burn depth, including assignment of fourth-degree burns despite their absence in the dataset. Error direction analysis revealed model-specific and task-dependent biases, including opposing patterns within the same model. Internal consistency between numeric and text classifications was near-perfect (99.6-100%), indicating format-invariant but systematically biased outputs. These findings demonstrate that MLLM performance is characterized by stochastic response instability invisible to single-query evaluations. Such inconsistency for identical inputs represents a fundamental limitation for workflows requiring consistent outputs across repeated evaluations.
引用
- Güler I, Kraus A, Grieb G, et al. AI-based burn image assessment: Reliability and clinical error patterns of multimodal large language models in a repeated-inference study. J Plast Reconstr Aesthet Surg. 2026. PMID: 42303556. doi:10.1016/j.bjps.2026.05.070
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。