指南与共识区域麻醉与超声疼痛医学

基于股神经超声分割的人工智能与人类专家校准共识框架

Calibrating artificial intelligence against human expertise using femoral nerve segmentation on ultrasound: a consensus framework.
指南 / 共识 · Anaesthesia · 2026-09-01

本研究开发并应用共识框架,由九名临床医生独立分割100张超声图像上的股神经,生成900个标注,通过多数投票建立共识标准。评估一个学术深度学习模型与此共识及个体表现,AI达到中位Dice系数0.72,匹配人类中位表现并优于九名标注者中的五名。

要点

  • AI模型中位Dice系数0.72,匹配人类中位表现。
  • AI敏感性0.94,精确性0.60,显示可靠检测但过度分割。
  • 共识框架可转移用于其他医学影像AI校准。

核心结论

  • AI模型匹配人类中位表现并优于五名标注者,但评估存在循环性局限。
  • Dice 0.72代表人类专家表现上限而非中等准确性。

研究设计与样本

  • 九名临床医生独立分割100张超声图像上的股神经,生成900个标注,多数投票建立共识标准。
  • 评估学术深度学习模型与共识及个体人类表现。

主要发现

  • AI对共识中位Dice系数0.72(IQR 0.56-0.84,范围0.00-0.91)。
  • AI敏感性0.94(IQR 0.88-0.97,范围0.33-1.00),精确性0.60(IQR 0.44-0.76,范围0.00-0.89)。
  • 个体人类Dice范围0.32-0.73(中位0.60),AI优于九名标注者中的五名(相对改善31%-125%)。
  • 留一标注者分析确认共识稳定性,AI Dice中位0.749(IQR 0.745-0.752,范围0.742-0.769)。
  • 评分者间信度中等,Fleiss's κ 0.54,p < 0.001。

临床怎么用

  • AI可辅助神经检测,但医生应检查AI输出后再用于患者。
  • 共识框架可用于测试其他医学影像AI工具。

局限性

  • 评估标注者与自身参与定义的共识比较存在循环性,限制AI与人类评分的直接比较。
  • 评分者间信度中等,反映神经 delineation 的主观性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: Meaningful validation of artificial intelligence for medical image interpretation requires comparison against human expert performance, yet multi-rater frameworks establishing such comparisons remain uncommon. METHODS: We developed and applied a consensus framework using nine clinicians who independently segmented the femoral nerve on 100 ultrasound images, yielding 900 annotations and a combined consensus standard established by majority voting. We then evaluated an academic deep learning model against this consensus and individual human performance. RESULTS: The artificial intelligence model achieved a median (IQR [range]) Dice coefficient of 0.72 (0.56-0.84 [0.00-0.91]) against combined consensus. Sensitivity was 0.94 (0.88-0.97 [0.33-1.00]) and precision 0.60 (0.44-0.76 [0.00-0.89]). Individual human Dice scores ranged from 0.32 to 0.73 (median 0.60). The artificial intelligence model matched median human performance and outperformed five of nine annotators (31%-125% relative improvement), with the greatest benefit for the lowest-performing practitioners. Leave-one-annotator-out analysis confirmed consensus stability (median (IQR [range]) artificial intelligence Dice 0.749 (0.745-0.752 [0.742-0.769])). Inter-rater reliability was moderate overall (Fleiss's κ 0.54, p < 0.001). DISCUSSION: The sensitivity and precision profile of the artificial intelligence model indicated reliable nerve detection with over-segmentation that remained clinically interpretable. The moderate inter-rater reliability is consistent with the inherent subjectivity of nerve delineation on ultrasound. The circularity inherent in evaluating annotators against a consensus they helped define limits direct comparison of artificial intelligence and human scores. A Dice score of 0.72 represents the upper range of human expert performance rather than moderate accuracy. The framework methodology is independent of the specific artificial intelligence system evaluated and offers a transferable approach for calibrating artificial intelligence performance in clinical imaging where no single correct interpretation exists. WHAT WE DID: We tested our own created artificial intelligence program that finds a leg nerve on ultrasound scans. Doctors use this nerve, called the femoral nerve, to guide a small injection of numbing medicine. The injection blocks pain after a broken hip or leg surgery. We collected 100 ultrasound images of this nerve. Doctors and experienced specialists then outlined the nerve on every image. The artificial intelligence also outlined the nerve on each image. We compared the outlines created by the artificial intelligence and the doctors. WHY DID WE DO IT: Finding a nerve on ultrasound is difficult. Even experienced doctors disagree about exactly where a nerve begins and ends. For doctors who are in training, this can mean the injection works less reliably. Artificial intelligence tools may help, but until now there is no one way to test them. Most studies compare artificial intelligence to a single outline made by a doctor, which does not allow for the natural differences between doctors. We wanted another method, so we tested the artificial intelligence against the combined judgement of nine doctors, rather than just one. WHAT WE FOUND: The artificial intelligence matched the typical doctor in our group and performed better than five of the nine. The benefit was greatest for the doctors in training. The artificial intelligence nearly always found the nerve, but it often outlined a slightly larger area than needed. Doctors should still check what the artificial intelligence shows before using it on a patient. Our method can also be used to test other medical artificial intelligence tools in the same way.

来源:Anaesthesia,查看原文。版权归原作者所有。

引用

  1. Berggreen J, Johansson A, Möller S, et al. Calibrating artificial intelligence against human expertise using femoral nerve segmentation on ultrasound: a consensus framework. Anaesthesia. 2026. PMID: 42680190. doi:10.1111/anae.70364

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。