AI生成患者友好放射报告的质评量表
本研究开发了一套用于评价AI生成患者友好放射报告质量与安全性的量表,包含清晰性、内容、确定性、语气和冗长度五个核心属性,每项按3分制评分。由ChatGPT-4.1和Claude-4.0生成不同质量的报告后,由研究员、非专业参与者、放射科医生及ChatGPT-5进行评价。结果显示评分者间一致性为中等至几乎完全,提示该量表可作为发布前的标准化安全把关工具。
要点
- 除冗长度外任一属性评为1分的报告应判定为不安全,不予发放给患者。
- 放射科与非专业研究员对总体分级几乎完全一致,但非专业评价者一致性仅中等。
- AI自动量表评分与参考标准的一致性为中等,需进一步训练和验证后方可推广。
核心结论
- 该量表可作为AI生成患者友好放射报告发布前的标准化安全保障工具。
- 虽然AI自动评价的一致性仅为中等,但规则化的分发决策与参考标准的一致性达88.1%。
- 量表应用需要进一步培训与验证,才可能实现可扩展的质量保证和更安全的临床整合。
研究设计与样本
- 前瞻性研究,于2025年2月至2025年12月进行,采用调查-工作坊循环开发量表。
- 量表开发阶段纳入19名参与者,平均年龄39±7岁,其中女性11名,男性8名。
- 量表评价阶段纳入6名研究团队成员和111名额外参与者,平均年龄47±18岁,女性43名,男性68名。
- 使用ChatGPT-4.1和Claude-4.0基于公开数据集的放射报告结论生成不同质量的患者友好报告。
- 评价者包括研究团队成员、额外非专业参与者与放射科医生,以及ChatGPT-5。
诊断标准与关键数值
- 量表包含五个核心属性:清晰性、内容、确定性、语气、冗长度,每项按3分制评分。
- 除冗长度外任一属性评为1分(不安全或不可接受)的患者友好报告应判定为不安全,不予发放。
- 非专业与放射科研究团队成员各3名评价60份报告,总体分级组间一致性α=0.87。
- 额外19名非专业参与者评价者间总体分级一致性α=0.51,分发决策与规则化决策一致率为91.2%。
- 12名放射科医生评价者间总体分级一致性α=0.65,分发决策与规则化决策一致率为95.8%。
临床应用与阅片要点
- 在将AI生成的患者友好报告发放给患者前,应由经培训的评价者使用本量表进行逐项评分。
- 报告中任何核心属性(除冗长度外)被评为1分时,应暂缓向患者发放该报告。
- AI辅助自动评分可作为初筛工具,但其一致性尚未达到可完全替代人工评价的水平。
- 日常应用中需注意非专业评价者与放射科医生之间的判断差异,必要时以放射科医生意见为准。
局限性
- AI自动化量表评价与参考标准的一致性仅为中等(κ=0.44),提示仍需改进。
- 研究使用了公开数据集的放射报告结论,可能不完全反映真实临床报告的复杂性。
- 非专业评价者在广泛现场测试中与参考标准的一致性较低(κ=0.43),量表可靠性仍需加强。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
Background: Artificial intelligence (AI) tools are being used to translate radiology reports into plain language, but translation errors may compromise comprehension and safety. Objective: To develop and evaluate a rubric for assessing the quality and safety of AI-generated patient-friendly radiology reports. Methods: In this prospective study (conducted from February 2025 to December 2025), survey-workshop cycles, involving lay participants and a multidisciplinary panel, were used to develop a rubric for grading AI-generated patient-friendly report quality across core attributes and determining whether such reports are safe for patient distribution. ChatGPT-4.1 and Claude-4.0 were used to generate patient-friendly reports of varying quality based on radiology report impressions from a public dataset and prespecified quality targets across attributes. Research-team members, additional lay participants and radiologists, and ChatGPT-5 evaluated these patient-friendly reports using the rubric. Results: Development included 19 participants (39±7 years; 11 women, 8 men); evaluation included six research-team members and 111 additional participants (47±18 years; 43 women, 68 men). The final rubric included five core attributes-clarity, content, certainty, tone, verbosity-each graded on a 3-point scale; by the rubric's decision rule, patient-friendly reports assessed as grade 1 (unsafe or unacceptable) in any attribute other than verbosity are unsafe for distribution and warrant withholding. Lay and radiologist research-team members (n=3 participants each; 60 reports) had almost-perfect intergroup agreement (α=0.87) for overall grade assignments. Additional lay (n=19) and radiologist (n=12) participants, each evaluating six reports, had moderate (α=0.51) and substantial (α=0.65) interreader agreement, respectively, for overall grade assignments and 91.2% and 95.8% agreement, respectively, between subjective and rubric rule-based distribution decisions. In wider field testing, 80 lay participants (480 reports) had moderate agreement (κ=0.43) with prespecified reference-standard grades; subjective distribution decisions had 73.5% agreement with rule-based distribution decisions. Across 480 reports, AI had moderate agreement (κ=0.44) with prespecified reference-standard grades; rule-based distribution decisions using AI-assigned grades had 88.1% agreement with rule-based distribution decisions using reference-standard grades. Conclusions: The rubric may provide a standardized safeguard before release of AI-generated patient-friendly reports. Clinical Impact: Although requiring further training and validation, AI rubric application could enable scalable quality assurance and safer clinical integration of AI-generated communications.
引用
- Armstrong, Koirala, Na, et al. A Quality Assessment Rubric for Artificial Intelligence-Generated Patient-Friendly Radiology Reports. AJR Am J Roentgenol. 2026. PMID: 42714439. doi:10.2214/AJR.26.35532
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。