文献速递神经技术与安全CT

ChatGPT对颅内出血检测人工智能系统部署后性能的监测

Post-deployment Monitoring of AI Performance in Intracranial Hemorrhage Detection by ChatGPT.
PubMed 文献 · Acad Radiol · 2025-08-12

这项回顾性研究评估了Aidoc人工智能系统在37家美国放射执业机构共332,809次头部CT检查中检测颅内出血的部署后性能,并使用ChatGPT-4 Turbo从放射报告中自动提取数据。以200例随机抽样病例的放射科医生复核为参考标准,ChatGPT-4 Turbo的阳性预测值为1、阴性预测值为0.988(AUC 0.996);Aidoc辅助放射科医生的敏感度为0.936、特异度为1。飞利浦扫描仪(OR 6.97)和伪影(OR 3.79)与假阳性增加相关。

要点

  • ChatGPT-4 Turbo从放射报告识别颅内出血的阳性预测值为1、阴性预测值为0.988(AUC 0.996)。
  • Aidoc辅助放射科医生的敏感度为0.936、特异度为1。
  • 飞利浦扫描仪(OR 6.97)和伪影(OR 3.79)与Aidoc假阳性分类增加相关。

核心结论

  • 本研究强调在临床实践中对人工智能系统进行持续性能监测的重要性,并认为大语言模型的整合为评估人工智能性能提供了可扩展的解决方案。

研究设计与样本

  • 回顾性研究,评估美国37家放射执业机构2023年12月至2024年5月的332,809次头部CT检查,其中13,569例被Aidoc人工智能系统标记为颅内出血阳性;使用符合HIPAA的ChatGPT-4 Turbo从放射报告中提取数据,以放射科医生复核200例随机抽样病例为参考标准。

诊断标准与关键数值

  • ChatGPT-4 Turbo从放射报告识别颅内出血的阳性预测值为1、阴性预测值为0.988(AUC 0.996);Aidoc辅助放射科医生的敏感度为0.936、特异度为1。

临床应用与阅片要点

  • 原文未据此给出推荐。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

RATIONALE AND OBJECTIVES: To evaluate the post-deployment performance of an artificial intelligence (AI) system (Aidoc) for intracranial hemorrhage (ICH) detection and assess the utility of ChatGPT-4 Turbo for automated AI monitoring.

MATERIALS AND METHODS: This retrospective study evaluated 332,809 head CT examinations from 37 radiology practices across the United States (December 2023-May 2024). Of these, 13,569 cases were flagged as positive for ICH by the Aidoc AI system. A HIPAA (Health Insurance Portability and Accountability Act) -compliant version of ChatGPT-4 Turbo was used to extract data from radiology reports. Ground truth was established through radiologists' review of 200 randomly selected cases. Performance metrics were calculated for ChatGPT, Aidoc and radiologists.

RESULTS: ChatGPT-4 Turbo demonstrated high diagnostic accuracy in identifying intracranial hemorrhage (ICH) from radiology reports, with a positive predictive value of 1 and a negative predictive value of 0.988 (AUC:0.996). Aidoc's false positive classifications were influenced by scanner manufacturer, midline shift, mass effect, artifacts, and neurologic symptoms. Multivariate analysis identified Philips scanners (OR: 6.97, p=0.003) and artifacts (OR: 3.79, p=0.029) as significant contributors to false positives, while midline shift (OR: 0.08, p=0.021) and mass effect (OR: 0.18, p=0.021) were associated with a reduced false positive rate. Aidoc-assisted radiologists achieved a sensitivity of 0.936 and a specificity of 1.

CONCLUSION: This study underscores the importance of continuous performance monitoring for AI systems in clinical practice. The integration of LLMs offers a scalable solution for evaluating AI performance, ensuring reliable deployment and enhancing diagnostic workflows.

来源:Acad Radiol,查看原文。版权归原作者所有。

引用

  1. Rohren E, Ahmadzade M, Colella S, et al. Post-deployment Monitoring of AI Performance in Intracranial Hemorrhage Detection by ChatGPT. Acad Radiol. 2025. PMID: 40796469. doi:10.1016/j.acra.2025.07.055

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。