指南与共识妇科生殖医学

AI能讲子宫内膜异位症吗?三大语言模型对ESHRE指南的多平台评估

Can AI speak endo? A multi-platform evaluation of large language models against ESHRE endometriosis guidelines.
指南 / 共识 · Hum Reprod · 2026-01-01

模型A准确性最高,模型C一致性最优,但所有模型在免费层可靠性均不理想;API访问下准确性趋同且可靠性提升。

要点

  • 模型A的GQS中位数最高(5,IQR 4-5),显著优于模型B(P<0.001),但与模型C无显著差异(P=0.123)。
  • 模型C重复查询一致率92.0%,显著高于模型A(72.0%,P=0.028)和模型B(68.0%,P=0.008)。
  • 三模型可靠性无显著差异(χ2=1.029,P=0.598),均未达到无监督临床使用所需水平。

核心结论

  • 模型A准确性最高,模型C一致性最优,但所有模型在免费层下的可靠性均不理想。
  • API访问下各提供商准确性中位数趋同,可靠性提升至至少76%。
  • 没有一个模型达到可无监督用于临床的可靠性水平。

研究设计与样本

  • 每个问题在独立会话中重复输入两次以评估一致性和可靠性。
  • 2026年5月对API版本和免费层版本进行亚组分析。

主要发现

  • 准确性存在显著组间差异(Kruskal-Wallis H=37.10,P<0.001):模型A的GQS中位数5(IQR 4-5),模型C为4(IQR 3-5),模型B为3(IQR 3-4)。
  • 事后分析:模型A显著优于模型B(P<0.001),与模型C无显著差异(P=0.123);模型C显著优于模型B(P<0.001)。
  • 一致性:模型C可重复响应率92.0%,显著高于模型A的72.0%(P=0.028)和模型B的68.0%(P=0.008)。
  • 可靠性无显著组间差异(χ2=1.029,P=0.598),模型C为76.0%,模型A和模型B均为68.0%。
  • 2026年5月亚组分析中,三个提供商的GQS中位数均趋同至4,可靠性均升至至少76%。

临床怎么用

  • 模型A和模型C可检索到与指南一致的信息,但均不能用于无监督临床决策。
  • 临床解读模型输出时仍需专家监督,尤其在子宫内膜异位症诊疗方面。
  • 免费层下准确性与一致性分离,提示模型能力和商业层级共同影响用户端输出。

局限性

  • 模型输出可能随后续更新而变化。
  • GQS尽管经专家裁定仍具有一定主观性。
  • 研究测试的是事实回忆而非复杂临床推理,限制了对真实决策场景的推广。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

STUDY QUESTION: How do the three most widely accessible large language models perform in terms of accuracy, consistency, and reliability when answering clinically relevant questions derived from the 2022 ESHRE guideline on endometriosis? SUMMARY ANSWER: Model A achieved the highest accuracy scores, model C demonstrated significantly superior consistency across repeated queries, and all three models showed comparable but suboptimal reliability under free-tier access, while under official API access median accuracy converged across providers and reliability increased. WHAT IS KNOWN ALREADY: Large language models are increasingly consulted by both clinicians and patients as readily accessible sources of medical information. In reproductive medicine, preliminary evidence suggests that individual platforms may retrieve endometriosis-related content with mixed fidelity. However, no study has simultaneously benchmarked multiple models against a single, internationally recognized endometriosis guideline. The extent to which these tools can be trusted to faithfully reproduce evidence-based recommendations on endometriosis diagnosis and management remains largely unexplored. STUDY DESIGN, SIZE, DURATION: Cross-sectional, multi-platform comparative study. Fifty clinically relevant questions covering the diagnostic and therapeutic domains of the 2022 ESHRE endometriosis guideline were simultaneously submitted to all three models during December 2025. Each question was entered in duplicate using independent sessions to assess response consistency and reliability. Subgroup analyses were carried out by submitting the questions to the API version and to the free-tier version of the platforms available in May 2026. PARTICIPANTS/MATERIALS, SETTING, METHODS: The three models were accessed through their respective free web interfaces using new accounts, without prompt engineering, prior training, or retrieval-augmented generation. A zero-shot prompting approach was adopted. Accuracy was evaluated by two independent experts using the Global Quality Score (GQS); consistency was defined as identical responses across the three iterations; reliability was defined as the alignment of each response with the ESHRE guideline. Discrepancies were settled by a third reviewer. MAIN RESULTS AND THE ROLE OF CHANCE: Significant differences in accuracy were observed across models (Kruskal-Wallis H = 37.10, P < 0.001). Model A achieved the highest median GQS (5, interquartile range [IQR] 4-5), followed by model C (4, IQR 3-5) and model B (3, IQR 3-4). Post-hoc analysis confirmed that model A significantly outperformed model B (P < 0.001) but not model C (P = 0.123), while model C also scored significantly higher than model B (P < 0.001). For consistency, model C demonstrated a significantly higher rate of reproducible responses (92.0%) compared with model A (72.0%, P = 0.028) and model B (68.0%, P = 0.008). No significant between-model differences were found for reliability (χ2 = 1.029, P = 0.598), with rates of 76.0% for model C, 68.0% for model A, and 68.0% for model B. In API and free-tier May 2026 subgroup analyses, median GQS converged to 4 across all three providers, and reliability rose to at least 76% for each provider. LIMITATIONS, REASONS FOR CAUTION: Model outputs may change with subsequent updates. GQS retains a degree of subjectivity despite expert adjudication. The study tested factual recall rather than complex clinical reasoning, limiting generalizability to real-world decision-making scenarios. WIDER IMPLICATIONS OF THE FINDINGS: These findings provide the first multi-platform benchmark of large language models against the latest endometriosis guideline. While models A and C retrieved guideline-concordant information with acceptable fidelity, none of the models achieved a level of reliability required for unsupervised clinical use. The dissociation between accuracy and consistency at the free tier, and its attenuation under API and more recent accesses, indicates that both model capability and commercial tier shape user-facing outputs. The dissociation between accuracy and consistency underscores that a model producing high-quality answers does not necessarily do so in a reproducible manner. Expert oversight remains crucial when interpreting their output regarding endometriosis care. Future research should extend this framework to additional endometriosis guidelines and to longitudinal monitoring of models' performance with their upgrades. FUNDING: No external funding was received for this study. DISCLOSURES: The authors declare no competing interests. TRIAL REGISTRATION NUMBER: N/A.

来源:Hum Reprod,查看原文。版权归原作者所有。

引用

  1. Riemma G, Caniglia FM, Casolari C, et al. Can AI speak endo? A multi-platform evaluation of large language models against ESHRE endometriosis guidelines. Hum Reprod. 2026. PMID: 42478094. doi:10.1093/humrep/deag116

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。