应用基础模型评估注射喉成形术技能的可行性研究
本研究评估商用多模态基础模型(Google Gemini 2.5 Pro)在模拟注射喉成形术中的结构效度。三十段视频按操作者经验分为新手、中间和专家各10段,采用零样本和少样本两种提示策略评估。少样本提示与操作者经验呈强正相关(Spearman's ρ=0.66,p=0.0002),能分层技能水平;零样本则不能(ρ=0.12,p=0.52)。
要点
- 少样本提示使模型评估与操作者经验强相关(ρ=0.66),可区分专家与新手/中间者。
- 零样本提示无法区分技能水平(ρ=0.12,p=0.52)。
- 少样本提示降低平均绝对误差近半(0.74→0.41,p=0.04),但模型在相同条件下存在变异性,需平均重复评估。
核心结论
- 通用多模态模型缺乏内在手术判断力,但少样本提示可校准模型以区分专家与受训者表现。
- 当前模型存在固有变异性,需通过平均重复评估等方法缓解。
研究设计与样本
- 可行性研究,评估商用多模态基础模型(Google Gemini 2.5 Pro)对模拟注射喉成形术视频的评估能力。
- 三十段视频按操作者经验分层:新手10段、中间10段、专家10段。
- 采用零样本(基于评分细则,无示例)和少样本(评分细则加示例)两种提示策略,并与操作者训练水平(真实标准)比较。
- 通过90次重复推理试验评估模型可靠性和稳定性。
主要发现
- 零样本策略下模型无法区分技能水平(Spearman's ρ=0.12,p=0.52)。
- 少样本提示与操作者经验呈强正相关(Spearman's ρ=0.66,p=0.0002),成功分层技能水平(Kruskal-Wallis H=12.4,p=0.002)。
- 少样本模型显著区分专家与新手(p=0.002)及专家与中间者(p=0.026)。
- 少样本提示显著提高精确度,平均绝对误差降低近半(0.74-0.41,p=0.04)。
- 可靠性分析显示100%序数一致性(75.6%精确匹配稳定性),模型在相同条件下有变异但无严重分类错误。
临床怎么用
- 少样本提示可作为可扩展的评估工具,用于区分专家与受训者的注射喉成形术表现。
- 因模型存在固有变异性,临床评估时应平均重复模型评价结果。
局限性
- 模型在相同条件下存在变异性(75.6%精确匹配稳定性),需缓解措施。
- 研究为模拟环境下的可行性研究,证据等级为N/A。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVES: To evaluate the construct validity of a commercially available multimodal foundation model (Google Gemini 2.5 Pro) in assessing simulated injection laryngoplasty. METHODS: Thirty video recordings of simulated injection laryngoplasty procedures were stratified by operator experience (10 novice, 10 intermediate, and 10 expert participants). Videos were evaluated by Gemini 2.5 Pro using two prompt engineering strategies: zero-shot (rubric-based, no examples) and few-shot (rubric plus examples). Performance was compared against operator training level (ground truth). Model reliability and stability were assessed through 90 repeated inference trials. RESULTS: Under a zero-shot strategy, the model failed to discriminate between skill levels (Spearman's ρ = 0.12, p = 0.52). Conversely, few-shot prompting demonstrated a strong, positive correlation with operator experience (Spearman's ρ = 0.66, p = 0.0002) and successfully stratified skill levels (Kruskal-Wallis H = 12.4, p = 0.002). Pairwise analysis confirmed the few-shot model significantly differentiated experts from both novices (p = 0.002) and intermediates (p = 0.026). Additionally, few-shot prompting significantly improved precision, reducing mean absolute error by nearly half (0.74-0.41, p = 0.04). Reliability analysis revealed 100% ordinal consistency (75.6% exact match stability), indicating the model varied under identical conditions, but did not commit any gross classification errors. CONCLUSION: General-purpose multimodal models lack the intrinsic surgical judgment necessary to assess procedural skill. However, resource-efficient few-shot prompting successfully calibrates the model to distinguish expert from trainee performance. While promising as a scalable assessment tool, current models exhibit inherent variability that requires mitigation, such as averaging repeat model evaluations. LEVEL OF EVIDENCE: N/A.
引用
- Cheng AT, Elkhadrawy A, Setzen SA, et al. Evaluating Injection Laryngoplasty Skills Using a Foundation Model: A Feasibility Study. Laryngoscope. 2026. PMID: 42185941. doi:10.1002/lary.70635
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。