指南与共识技术与安全

用于多语言医学影像指南与共识质量评估的大语言模型的开发与性能

Development and Performance of a Large Language Model for the Quality Evaluation of Multi-Language Medical Imaging Guidelines and Consensus.
指南 / 共识 · J Evid Based Med · 2025-01-01

本研究开发并评估了基于大语言模型的QPC-HASE-GuidelineEval算法,用于评估不同语言医学影像指南与共识的质量。模型在2021至2022年发表的45部指南(中文36部、英文9部)上验证,平均准确率77%,中英文版本准确率分别为76%和74%(p=0.37),混合搜索在段落匹配准确性和信息完整性上表现更优,并显著降低评估时间与成本。

要点

  • 该算法平均准确率77%,简单任务表现较好,复杂评估(解释和视觉辅助)准确率仅29%-40%。
  • 英文与中文指南评估平均准确率分别为74%和76%,两者无显著差异(p=0.37)。
  • 混合搜索段落匹配准确性和信息完整性均为4.42,优于关键词搜索和稀疏-稠密检索。

核心结论

  • 基于大语言模型的QPC-HASE-GuidelineEval算法在提升指南评估效率、可扩展性和多语言能力方面展现出较强潜力,但处理需要更深层解读的复杂任务仍需进一步改进。

研究设计与样本

  • 开发并验证QPC-HASE-GuidelineEval算法,该算法整合四象限问题分类策略与混合搜索增强。
  • 在2021年和2022年发表的45部医学影像指南(中文36部、英文9部)上进行验证。
  • 关键评估指标包括与专家评估的一致性、混合搜索段落匹配准确性、信息完整性、不同段落匹配方法比较以及成本时间效率。

诊断标准与关键数值

  • 算法平均准确率为77%,在较简单任务中表现优异,但在解释和视觉辅助等复杂评估中准确率较低(29%-40%)。
  • 英文版和中文版指南与共识质量评估的平均准确率分别为74%和76%(p=0.37)。
  • 混合搜索的段落匹配准确性为4.42、信息完整性为4.42,显著优于关键词搜索(1.05/1.05)和稀疏-稠密检索(4.26/3.63)。
  • 算法将每部指南的评估时间显著缩短至8分30秒,成本降低至每部指南约0.5美元。

临床应用与阅片要点

  • 原文未据此给出推荐。

局限性

  • 算法在解释和视觉辅助等复杂评估中准确率较低(29%-40%),需进一步改进以处理需要更深层解读的复杂任务。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

AIM: This study aimed to develop and evaluate an automated large language model (LLM)-based system for assessing the quality of medical imaging guidelines and consensus (GACS) in different languages, focusing on enhancing evaluation efficiency, consistency, and reducing manual workload.

METHOD: We developed the QPC-HASE-GuidelineEval algorithm, which integrates a Four-Quadrant Questions Classification Strategy and Hybrid Search Enhancement. The model was validated on 45 medical imaging guidelines (36 in Chinese and 9 in English) published in 2021 and 2022. Key evaluation metrics included consistency with expert assessments, hybrid search paragraph matching accuracy, information completeness, comparisons of different paragraph matching approaches, and cost-time efficiency.

RESULTS: The algorithm demonstrated an average accuracy of 77%, excelling in simpler tasks but showing lower accuracy (29%-40%) in complex evaluations, such as explanations and visual aids. The average accuracy rates of the English and Chinese versions of the GACS were 74% and 76%, respectively (p = 0.37). Hybrid search demonstrated superior performance with paragraph matching accuracy (4.42) and information completeness (4.42), significantly outperforming keyword-based search (1.05/1.05) and sparse-dense retrieval (4.26/3.63). The algorithm significantly reduced evaluation time to 8 min and 30 s per guideline and reduced costs to approximately 0.5 USD per guideline, offering a considerable advantage over traditional manual methods.

CONCLUSION: The QPC-HASE-GuidelineEval algorithm, powered by LLMs, showed strong potential for improving the efficiency, scalability, and multi-language capability of guideline evaluations, though further enhancements are needed to handle more complex tasks that require deeper interpretation.

来源:J Evid Based Med,查看原文。版权归原作者所有。

引用

  1. Wang Z, Sun J, Liu H, et al. Development and Performance of a Large Language Model for the Quality Evaluation of Multi-Language Medical Imaging Guidelines and Consensus. J Evid Based Med. 2025. PMID: 40181523. doi:10.1111/jebm.70020

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。