评估ChatGPT在放射肿瘤学中的能力:跨临床场景的全面评估
本研究设计覆盖12个病种的临床病例,由皇家学院(Royal College)认证的放射肿瘤科医生按简洁性、完整性和正确性三项标准对ChatGPT的回答进行评分。182个问题总分为2317/2730(84%),流行病学和肿瘤生物学领域表现最佳(93%),治疗计划和临床管理领域较弱(分别为78%和81%)。ChatGPT在多个肿瘤学领域显示出高准确度,但在复杂病例中存在错误和不完整的回答。
要点
- ChatGPT在12个病种182个问题上总得分率84%(2317/2730),简洁性92%最高,完整性79%最低。
- 流行病学和肿瘤生物学得分最高(93%),治疗计划(78%)和临床管理(81%)较弱。
- 统计学差异由完整性(p<0.01)和正确性(p=0.04)驱动,简洁性普遍高分(p=0.91)。
核心结论
- ChatGPT在放射肿瘤学中显示出作为工具的潜力,在多个肿瘤学领域表现出高度准确性,但复杂病例中存在错误和不完整的回答。
研究设计与样本
- 设计覆盖12个病种的临床病例系列,问题分为流行病学、分期与检查、临床管理、治疗计划、肿瘤生物学、物理学和监测等领域。
- 由皇家学院(Royal College)认证的放射肿瘤科医生审阅病例并提供参考答案,按简洁性、完整性和正确性三项标准使用标准化评分表对回答评分,每项0至5分,总分15分。
主要发现
- 12个病例共182个问题,AI总分为2317/2730(84%)。
- 按标准评分:完整性79%(范围70-99%),简洁性92%(范围83-99%),正确性81%(范围72-92%)。
- AI在流行病学(93%)和肿瘤生物学(93%)领域表现最佳,在分期与检查(89%)、物理学(86%)和监测(82%)领域表现合理。
- 较弱领域包括治疗计划(78%)和临床管理(81%)。
- 统计学差异由完整性(p<0.01)和正确性(p=0.04)标准的变异驱动,而简洁性普遍得分高(p=0.91),这些趋势在各病种间一致。
局限性
- 本研究强调了复杂病例中存在错误和不完整答案的局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
PURPOSE: Artificial intelligence (AI) and machine learning present an opportunity to enhance clinical decision-making in radiation oncology. This study aims to evaluate the competency of ChatGPT, an AI language model, in interpreting clinical scenarios and assessing its oncology knowledge. METHODS AND MATERIALS: A series of clinical cases were designed covering 12 disease sites. Questions were grouped into domains: epidemiology, staging and workup, clinical management, treatment planning, cancer biology, physics, and surveillance. Royal College-certified radiation oncologists (ROs) reviewed cases and provided solutions. ROs scored responses on 3 criteria: conciseness (focused answers), completeness (addressing all aspects of the question), and correctness (answer aligns with expert opinion) using a standardized rubric. Scores ranged from 0 to 5 for each criterion for a total possible score of 15. RESULTS: Across 12 cases, 182 questions were answered with a total AI score of 2317/2730 (84 %). Scores by criteria were: completeness (79 %, range: 70-99 %), conciseness (92 %, range: 83-99 %), and correctness (81 %, range: 72-92 %). AI performed best in the domains of epidemiology (93 %) and cancer biology (93 %) and reasonably in staging and workup (89 %), physics (86 %) and surveillance (82 %). Weaker domains included treatment planning (78 %) and clinical management (81 %). Statistical differences were driven by variations in the completeness (p < 0.01) and correctness (p = 0.04) criteria, whereas conciseness scored universally high (p = 0.91). These trends were consistent across disease sites. CONCLUSIONS: ChatGPT showed potential as a tool in radiation oncology, demonstrating a high degree of accuracy in several oncologic domains. However, this study highlights limitations with incorrect and incomplete answers in complex cases.
引用
- Ramadan S, Mutsaers A, Chen PC, et al. Evaluating ChatGPT's competency in radiation oncology: A comprehensive assessment across clinical scenarios. Radiother Oncol. 2024. PMID: 39571686. doi:10.1016/j.radonc.2024.110645
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。