大语言模型在临床决策支持中的表现:ChatGPT与乳腺放射科医师在ACR适宜性标准上的对比分析
放射科医师平均偏倚最低(0.2438,p=0.489),所有ChatGPT版本偏倚显著更大(GPT-4o: 2.463; GPT-4: 1.7623; GPT-3.5: 2.4691, 均p<0.001)。结论认为ChatGPT在影像决策中有潜力,但当前版本在准确性、一致性和可重复性上仍不及有经验的放射科医师。
要点
- ChatGPT各版本与ACR适宜性标准的偏倚均显著大于乳腺放射科医师。
- 放射科医师组内变异性与同一ChatGPT版本相当或更小。
- 当前LLM用于影像决策仍需人工监督,不可替代医师判断。
核心结论
- ChatGPT在影像决策支持中有潜力,但当前版本缺乏有经验放射科医师的准确性、一致性和可重复性。
研究设计与样本
- 采用广义估计方程(GEE)和Bland-Altman图分析ChatGPT、放射科医师与ACR标准之间评分的一致性和偏倚。
诊断标准与关键数值
- 放射科医师相对ACR标准的总体平均偏倚最低(0.2438,p=0.489),无显著差异。
- GPT-4o平均偏倚2.463、GPT-4平均偏倚1.7623、GPT-3.5平均偏倚2.4691,均显著大于放射科医师(均p<0.001)。
- 所有评分者均存在斜率偏倚(p<0.001),但放射科医师斜率偏倚最小。
临床应用与阅片要点
- 在乳腺影像决策中,LLM输出不宜直接作为适宜性判断依据,需结合ACR适宜性标准并由放射科医师审核。
- 注意LLM与指南间存在系统性偏倚,且不同版本间偏倚程度不一,报告或临床应用中应警惕其一致性局限。
局限性
- 单中心研究,仅纳入四名乳腺放射科医师,样本量有限。
- ChatGPT版本限于2024年7月可用版本,结果可能随模型更新而变化。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: This study evaluates the performance of ChatGPT, a large language model (LLM), in selecting appropriate imaging modalities for breast imaging scenarios using the American College of Radiology (ACR) Appropriateness Criteria (AC). We aim to compare the agreement of ChatGPT with the ACR AC to that of breast radiologists at a single institution in selecting appropriate imaging modalities.
METHODS/MATERIALS: The study utilized ten randomly selected clinical variants from the ACR AC breast imaging category. Outputs were obtained from ChatGPT-3.5, ChatGPT-4, and ChatGPT-4o using the versions available in July 2024. The ChatGPT versions and four breast radiologists rated the appropriateness of 81 imaging decisions on a scale from 1 to 9. For each imaging option within a clinical scenario, the ratings provided by the radiologists and four independent samplings of ChatGPT's responses were aggregated. Agreement between ratings from ChatGPT, radiologists, and the ACR AC was analyzed using generalized estimating equations (GEEs) and Bland-Altman plots to assess consistency and bias.
RESULTS: Radiologists had the lowest overall mean bias (0.2438) relative to the ACR (p = 0.489). All versions of ChatGPT had larger mean biases that were significant (GPT-4o: 2.463; GPT-4: 1.7623, GPT-3.5: 2.4691, all p<0.001). All had a slope bias (p < 0.001), but radiologists had the smallest slope bias. In summary, radiologists were closer to the ACR AC and were oftentimes as variable or even less variable as a group than the same ChatGPT version at the same time.
CONCLUSION: ChatGPT shows promise as an AI tool for imaging decision-making, but current versions lack the accuracy, consistency, and reproducibility demonstrated by experienced radiologists. The study underscores the importance for human oversight in clinical applications and the need for further development to improve ChatGPT's and other LLMs' reliability and alignment with established guidelines.
引用
- Escobar SA, Rhee J, Zaki HA, et al. Large Language Models in Clinical Decision Support: A Comparative Analysis of Chat-GPT and Breast Radiologists on ACR Appropriateness Criteria. Acad Radiol. 2026. PMID: 41925458. doi:10.1016/j.acra.2026.03.015
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。