指南与共识牙体牙髓病

人类共识与生成式人工智能在牙髓病学系统评价方法学质量及总体置信度评估中的一致性比较

Human Consensus Versus Generative Artificial Intelligence When Assessing the Methodological Quality and Overall Confidence in the Results of Systematic Reviews With Meta-Analysis in Endodontology.
指南 / 共识 · Int Endod J · 2026-10-28

本研究纳入2021至2024年发表的100篇牙髓病学系统评价与Meta分析,由三位人类专家使用AMSTAR 2评估并形成共识,同时用ChatGPT-5.5(三个新账户)生成评级。结果显示总体置信度直接一致率为52%(加权κ=0.182),ChatGPT-5.5更常判定为极低置信度,不支持其作为独立AMSTAR 2评估者。

要点

  • ChatGPT-5.5与人类共识的总体置信度直接一致率仅52%,加权κ=0.182。
  • ChatGPT-5.5将85篇SR判为极低置信度,人类共识为47篇。
  • 三个ChatGPT-5.5账户间总体置信度一致85篇,但账户间一致性不保证与人类判断一致。

核心结论

  • 总体置信度直接一致率对Partial Yes的处理敏感,但机会校正一致性在两种标准下相似。
  • ChatGPT-5.5比人类共识更频繁地给出极低总体置信度分类,且各AMSTAR 2条目一致性差异较大。
  • 研究结果不支持将ChatGPT-5.5作为独立的AMSTAR 2评估者,探索性应用需由方法学专家逐条核验。

研究设计与样本

  • 纳入2021至2024年发表的英文牙髓病学系统评价与Meta分析共100篇。
  • 三位人类专家使用AMSTAR 2评估方法学质量并生成共识数据集。
  • 通过三个新建账户应用ChatGPT-5.5生成AMSTAR 2条目级评级和总体置信度分类。

主要发现

  • 人类共识和ChatGPT-5.5分别将47篇和85篇SR分类为极低置信度。
  • 主要标准(Partial Yes=Yes)下总体置信度匹配52/100篇(52%;加权κ=0.182)。
  • 替代标准(Partial Yes=No)下直接一致率增至83%,加权κ=0.173。
  • 三个ChatGPT-5.5账户间总体置信度在85篇SR中完全一致,全局条目级3/3一致率为86.12%。
  • 在100篇抽样SR中,15%满足报告原始研究资助的标准,21%满足论证研究设计的标准。

关键数值与推荐

  • 总体置信度主要标准:Partial Yes=Yes,直接一致率52%(加权κ=0.182)。
  • 总体置信度替代标准:Partial Yes=No,直接一致率83%(加权κ=0.173)。
  • 三个ChatGPT-5.5账户间总体置信度完全一致见于85篇SR,全局条目级3/3一致率86.12%。

临床怎么用

  • 不建议将ChatGPT-5.5作为独立的AMSTAR 2评估工具使用。
  • 若探索性使用ChatGPT-5.5辅助评估,需由具备方法学专长的评审者逐条核验。

局限性

  • 总体置信度直接一致率对Partial Yes评级处理方式敏感,可能影响一致性解读。
  • ChatGPT-5.5账户间输出一致性较高,但并不能保证与人类判断一致。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVE: To evaluate the level of agreement between ChatGPT-5.5 and human assessments when appraising the methodological quality and overall confidence of the results of systematic reviews (SRs) with meta-analysis (MA) in endodontology. METHODS: SRs with MA related to endodontology and published in English between 2021 and 2024 were included. The methodological quality of the SRs was appraised by three human experts using AMSTAR 2, and a consensus dataset was generated. ChatGPT-5.5 was applied to generate AMSTAR 2 item-level ratings and overall confidence classifications for each SR through three newly created accounts. Agreement between consolidated ChatGPT-5.5 ratings and human consensus was assessed for both overall confidence and each AMSTAR 2 item. Overall-confidence agreement was evaluated under the primary criterion (Partial Yes = Yes) and an alternative sensitivity criterion (Partial Yes = No). Direct agreement and linearly weighted Cohen's kappa were calculated, and complete 3/3 agreement was used to assess consistency across ChatGPT-5.5 accounts. RESULTS: One hundred SRs were included. Human consensus and ChatGPT-5.5 classified 47 and 85 SRs, respectively, as critically low. Under the primary criterion (Partial Yes = Yes), overall confidence matched in 52/100 SRs (52%; weighted κ = 0.182). Under the alternative criterion (Partial Yes = No), agreement increased to 83%, while weighted κ remained similar (κ = 0.173), indicating that direct agreement was sensitive to the handling of Partial Yes. Item-level agreement varied substantially. Overall confidence was identical across the three ChatGPT-5.5 accounts in 85 SRs, and global item-level 3/3 agreement was 86.12%. In the 100 sampled SRs, 15% met the criterion for reporting primary-study funding and 21% for justifying study designs. CONCLUSION: Overall-confidence direct agreement was sensitive to the handling of Partial Yes ratings, whereas chance-corrected agreement remained similar under both criteria. ChatGPT-5.5 assigned lower overall confidence classifications more frequently than human consensus, and agreement varied across AMSTAR 2 domains. Although outputs were frequently consistent across the three ChatGPT-5.5 accounts, between-account consistency did not ensure concordance with human judgements. The findings do not support the use of ChatGPT-5.5 as an independent AMSTAR 2 assessor; any exploratory application requires item-by-item verification by reviewers with methodological expertise.

来源:Int Endod J,查看原文。版权归原作者所有。

引用

  1. Nagendrababu V, Dufey-Portilla N, Abraham SB, et al. Human Consensus Versus Generative Artificial Intelligence When Assessing the Methodological Quality and Overall Confidence in the Results of Systematic Reviews With Meta-Analysis in Endodontology. Int Endod J. 2026. PMID: 42949178. doi:10.1111/iej.70297

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。