文献速递甲状腺疾病

评估市售人工智能算法在超声甲状腺结节风险分层中的诊断性能

Assessment of the Diagnostic Performance of a Commercially Available Artificial Intelligence Algorithm for Risk Stratification of Thyroid Nodules on Ultrasound.
PubMed 文献 · Thyroid · 2024-10-15

本研究回顾性评估了FDA批准的AI工具Koios DS Thyroid在649个甲状腺结节(32个恶性,617个良性)中的诊断性能。结果显示,启用AI适配器的Koios与放射科医生的AUC均为0.70,但特异性更高(0.63 vs 0.43)而敏感性更低(0.69 vs 0.81)。放射科报告联合AI适配器时AUC最高(0.76),特异性从0.43提升至0.53,FNA推荐减少17%,敏感性不变(0.81)。

要点

  • Koios联合AI适配器与放射科医生的AUC均为0.70,但特异性更高(0.63 vs 0.43),敏感性更低(0.69 vs 0.81)。
  • 放射科报告联合AI适配器时AUC最高(0.76),特异性从0.43提升至0.53(p<0.001)。
  • 联合AI适配器使FNA推荐减少17%,敏感性保持0.81不变(p=1)。

核心结论

  • Koios DS独立性能与放射科医生相似,但敏感性更低、特异性更高。
  • 放射科解读联合AI适配器组件时性能最佳,特异性提高且减少了不必要的FNA推荐。

研究设计与样本

  • 回顾性评估杜克大学医院及其附属社区医院连续一年甲状腺超声(≥1个结节)的649个结节(来自347例患者)。
  • 纳入结节有手术病理、FNA或三年随访超声稳定的地面真实诊断。
  • FDA批准的AI工具(Koios DS Thyroid)分析每个结节,生成ACR TI-RADS描述符、评分和随访建议,以及AI适配器评分以进一步调整风险评估和建议。
  • 比较四组:Koios联合AI适配器、Koios未联合AI适配器、临床放射科报告、放射科报告联合AI适配器。
  • 基于地面真实诊断确定最终建议(FNA或不FNA)的性能,使用敏感性、特异性和ROC分析比较四组。

主要发现

  • 649个结节中32个恶性,617个良性。
  • Koios联合AI适配器的AUC为0.70(CI 0.60-0.81),放射科医生AUC为0.70(CI 0.60-0.79)。
  • Koios联合AI适配器特异性0.63(CI 0.59-0.67)高于放射科医生0.43(CI 0.38-0.48),敏感性0.69(CI 0.50-0.83)低于放射科医生0.81(CI 0.61-0.92)。
  • 放射科解读联合AI适配器AUC最高0.76(CI 0.67-0.85)。
  • 联合AI适配器后放射科医生特异性从0.43(CI 0.38-0.48)提高至0.53(CI 0.49-0.58)(McNemar检验p<0.001),FNA推荐减少17%,敏感性不变(0.81,p=1)。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

Background: Thyroid nodules are challenging to accurately characterize on ultrasound (US), though the emergence of risk stratification systems and more recently artificial intelligence (AI) algorithms has improved nodule classification. The purpose of this study was to evaluate the performance of a recent Food and Drug Administration (FDA)-cleared AI tool for detection of malignancy in thyroid nodules on US. Methods: One year of consecutive thyroid US with ≥1 nodule from Duke University Hospital and its affiliate community hospital (649 nodules from 347 patients) were retrospectively evaluated. Included nodules had ground truth diagnoses by surgical pathology, fine needle aspiration (FNA), or three-year follow-up US showing stability. An FDA-cleared AI tool (Koios DS Thyroid) analyzed each nodule to generate (i) American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) descriptors, scores, and follow-up recommendations and (ii) an AI-adapter score to further adjust risk assessments and recommendations. Four groups were then compared: (i) Koios with AI-adapter, (ii) Koios without AI-adapter, (iii) clinical radiology report, and (iv) radiology report combined with AI-adapter. Performance of the final recommendations (FNA or no FNA) was determined based on ground truth, and comparison between the four groups was made using sensitivity, specificity, and receiver-operating-curve analysis. Results: Of 649 nodules, 32 were malignant and 617 were benign. Performance of Koios with AI-adapter enabled was similar to radiologists (area under the curve [AUC] 0.70 for both, [CI 0.60-0.81] and [0.60-0.79], respectively). Koios with AI-adapter had improved specificity compared to radiologists (0.63 [CI: 0.59-0.67] versus 0.43 [CI: 0.38-0.48]) but decreased sensitivity (0.69 [CI: 0.50-0.83) versus 0.81 [CI: 0.61, 0.92]). Highest performance was seen when the radiology interpretation was combined with the AI-adapter (AUC 0.76, [CI: 0.67-0.85]). Combined with the AI-adapter, radiologist specificity improved from 0.43 ([CI: 0.38-0.48]) to 0.53 ([CI: 0.49-0.58]) (McNemar's test p < 0.001), resulting in 17% fewer FNA recommendations, with unchanged sensitivity (0.81, p = 1). Conclusion: Koios DS demonstrated standalone performance similar to radiologists, though with lower sensitivity and higher specificity. Performance was best when radiologist interpretations were combined with the AI-adapter component, with improved specificity and reduced unnecessary FNA recommendations.

来源:Thyroid,查看原文。版权归原作者所有。

引用

  1. Ashton J, Morrison S, Erkanli A, et al. Assessment of the Diagnostic Performance of a Commercially Available Artificial Intelligence Algorithm for Risk Stratification of Thyroid Nodules on Ultrasound. Thyroid. 2024. PMID: 39405186. doi:10.1089/thy.2024.0410

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。