大型语言模型在医疗保健中的应用测试与评估:一项系统综述
该研究系统检索了2022年1月1日至2024年2月19日发表的519项评估大型语言模型(LLM)在医疗保健中应用的研究,归纳了评估数据、任务、自然语言处理与理解、评估维度及医学专科。结果显示仅5%使用真实患者护理数据,多数聚焦答题准确性(95.4%),公平性、偏倚与毒性(15.8%)及部署考量(4.6%)等维度较少。作者建议未来评估应采用标准化应用与指标、使用临床数据并拓宽任务与专科范围。
要点
- 519项研究中仅5%使用真实患者护理数据评估LLM。
- 95.4%的研究以准确性为主要评估维度。
- 公平性、偏倚与毒性(15.8%)及部署考量(4.6%)评估较少。
核心结论
- 现有LLM评估多聚焦医学考试答题准确性,未考虑真实患者护理数据;公平性、偏倚与毒性及部署考量等维度关注有限;未来评估应采用标准化应用与指标、使用临床数据并拓宽任务与专科范围。
研究设计与样本
- 系统综述,检索PubMed和Web of Science,纳入2022年1月1日至2024年2月19日发表的评估1个或多个LLM在医疗保健中应用的研究,共519项,由3名独立评审员通过关键词搜索归类。
主要发现
- 519项研究中仅5%使用真实患者护理数据评估LLM。
- 最常见医疗保健任务为评估医学知识如回答医师执照考试问题(44.5%)和做出诊断(19.5%)。
- 行政任务如分配计费代码(0.2%)和开具处方(0.2%)研究较少。
- 自然语言处理与理解任务中,多数研究聚焦问答(84.2%),摘要(8.9%)和对话对话(3.3%)较少。
- 医学专科中,通用医疗保健应用(25.6%)、内科(16.4%)、外科(11.4%)和眼科(6.9%)较多,核医学(0.6%)、物理医学(0.4%)和医学遗传学(0.2%)较少。
关键数值与推荐
- 评估数据方面:仅5%的研究使用真实患者护理数据。
- 医疗保健任务方面:回答医师执照考试问题占44.5%,诊断占19.5%,分配计费代码占0.2%,开具处方占0.2%。
- 自然语言处理与理解任务方面:问答占84.2%,摘要占8.9%,对话对话占3.3%。
- 评估维度方面:准确性占95.4%,公平性、偏倚与毒性占15.8%,部署考量占4.6%,校准与不确定性占1.2%。
- 医学专科方面:通用医疗保健应用占25.6%,内科占16.4%,外科占11.4%,眼科占6.9%,核医学占0.6%,物理医学占0.4%,医学遗传学占0.2%。
临床怎么用
- 作者建议未来评估应采用标准化应用与指标、使用临床数据并拓宽任务与专科范围。
局限性
- 原文未报告局限性
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
IMPORTANCE: Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas. OBJECTIVE: To summarize existing evaluations of LLMs in health care in terms of 5 components: (1) evaluation data type, (2) health care task, (3) natural language processing (NLP) and natural language understanding (NLU) tasks, (4) dimension of evaluation, and (5) medical specialty. DATA SOURCES: A systematic search of PubMed and Web of Science was performed for studies published between January 1, 2022, and February 19, 2024. STUDY SELECTION: Studies evaluating 1 or more LLMs in health care. DATA EXTRACTION AND SYNTHESIS: Three independent reviewers categorized studies via keyword searches based on the data used, the health care tasks, the NLP and NLU tasks, the dimensions of evaluation, and the medical specialty. RESULTS: Of 519 studies reviewed, published between January 1, 2022, and February 19, 2024, only 5% used real patient care data for LLM evaluation. The most common health care tasks were assessing medical knowledge such as answering medical licensing examination questions (44.5%) and making diagnoses (19.5%). Administrative tasks such as assigning billing codes (0.2%) and writing prescriptions (0.2%) were less studied. For NLP and NLU tasks, most studies focused on question answering (84.2%), while tasks such as summarization (8.9%) and conversational dialogue (3.3%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias, and toxicity (15.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in generic health care applications (25.6%), internal medicine (16.4%), surgery (11.4%), and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%), and medical genetics (0.2%) being the least represented. CONCLUSIONS AND RELEVANCE: Existing evaluations of LLMs mostly focus on accuracy of question answering for medical examinations, without consideration of real patient care data. Dimensions such as fairness, bias, and toxicity and deployment considerations received limited attention. Future evaluations should adopt standardized applications and metrics, use clinical data, and broaden focus to include a wider range of tasks and specialties.
引用
- Bedi S, Liu Y, Orr-Ewing L, et al. Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review. JAMA. 2025. PMID: 39405325. doi:10.1001/jama.2024.21700
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。