放射治疗科全科评估与实施商业化人工智能自动勾画框架
研究开发了一种多维评估方法,在4家诊所的121例患者计划CT扫描(共859个临床批准轮廓)上测试3种商业化AI自动勾画软件(AI1、AI2、AI3),覆盖脑、头颈、胸部、腹部和盆腔部位。AI1、AI2、AI3分别在27.8%、32.8%、34.1%的病例中DSC>0.9,在盆腔和胸部表现较好,在脑和头颈表现较差。定性评估中AI1和AI2分别在70.7%和74.6%的ROI被评为可接受(1-2分)仅需微小修改,用户调查显示对全自动化的强烈偏好。
要点
- AI1、AI2、AI3分别在27.8%、32.8%、34.1%的病例中DSC>0.9,盆腔和胸部表现较好。
- AI1和AI2分别在70.7%和74.6%的ROI定性评为可接受(1-2分)仅需微小修改,头颈区域较低(29.0%/35.6%)。
- 用户调查显示对全自动化的强烈偏好,对精度与生成结构总数的偏好不一。
核心结论
- 该评估方法对商业化预训练AI自动勾画算法提供了定量和定性措施的全面分析。
- 该评估框架作为临床整合的路线图,与用户工作流程偏好一致。
研究设计与样本
- 研究纳入4家诊所的121例患者计划CT扫描,共859个临床批准用于治疗的轮廓。
- 使用3种商业化AI自动勾画软件(AI1、AI2、AI3)生成感兴趣区,覆盖脑、头颈、胸部、腹部和盆腔部位。
- 定量一致性用Dice相似系数(DSC)和Hausdorff距离(HD)测量,定性评估由多位专家使用Likert量表对盲法AI轮廓评分,并进行工作流程和可用性调查。
主要发现
- AI1、AI2、AI3分别在27.8%、32.8%、34.1%的病例中DSC>0.9,盆腔中位DSC分别为0.86、0.88、0.91,胸部中位DSC分别为0.91、0.89、0.91。
- AI1、AI2、AI3分别在7.4%、8.8%、6.1%的病例中DSC<0.5,脑中位DSC分别为0.65、0.78、0.75,头颈中位DSC分别为0.76、0.80、0.81。
- 定性评估中,AI1和AI2轮廓分别在70.7%和74.6%的ROI(共2906次评分)被评为可接受(1-2分)仅需微小修改,腹部(AI1:79.2%)和胸部(AI2:90.2%)较高,头颈(29.0%/35.6%)较低。
- 最终用户调查显示对全自动化的强烈偏好,对精度与生成结构总数的偏好不一。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
PURPOSE: Artificial intelligence (AI)-based autocontouring in radiation oncology has potential benefits such as standardization and time savings. However, commercial AI solutions require careful evaluation before clinical integration. We developed a multidimensional evaluation method to test pretrained AI-based automated contouring solutions across a network of clinics. METHODS AND MATERIALS: Curated data included 121 patient planning computed tomography (CT) scans with a total of 859 clinically approved contours used for treatment from 4 clinics. Regions of interest (ROIs) were generated with 3 commercial AI-based automated contouring software solutions (AI1, AI2, AI3) spanning the following disease sites: brain, head and neck (H&N), thorax, abdomen, and pelvis. Quantitative agreement between AI-generated and clinical contours was measured by Dice similarity coefficient (DSC) and Hausdorff distance (HD). Qualitative assessment was performed by multiple experts scoring blinded AI-contours using a Likert scale. Workflow and usability surveying was also conducted. RESULTS: AI1, AI2, and AI3 contours had high quantitative agreement in 27.8%, 32.8%, and 34.1% of cases (DSC >0.9), performing well in pelvis (median DSC = 0.86/0.88/0.91) and thorax (median DSC = 0.91/0.89/0.91). All 3 solutions had low quantitative agreement in 7.4%, 8.8%, and 6.1% of cases (DSC <0.5), performing worse in brain (median DSC = 0.65/0.78/0.75) and H&N (median DSC = 0.76/0.80/0.81). Qualitatively, AI1 and AI2 contours were acceptable (rated 1-2) with at most minor edits in 70.7% and 74.6% of ROIs (2906 ratings), higher for abdomen (AI1: 79.2%) and thorax (AI2: 90.2%), and lower for H&N (29.0/35.6%). An end-user survey showed strong user preference for full automation and mixed preferences for accuracy versus total number of structures generated. CONCLUSIONS: Our evaluation method provided a comprehensive analysis of both quantitative and qualitative measures of commercially available pretrained AI autocontouring algorithms. The evaluation framework served as a roadmap for clinical integration that aligned with user workflow preference.
引用
- Maes D, Gates EDH, Meyer J, et al. Framework for Radiation Oncology Department-wide Evaluation and Implementation of Commercial Artificial Intelligence Autocontouring. Pract Radiat Oncol. 2023. PMID: 37935308. doi:10.1016/j.prro.2023.10.011
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。