文献速递药物相互作用药物不良反应与药物警戒

大型语言模型在用药管理任务中的性能评估

Performance evaluation of a large language model for medication management tasks.
PubMed 文献 · Am J Health Syst Pharm · 2026-09-10

本研究评估了GPT-4o在3项用药管理任务中的表现:识别仿制药可用剂型、识别药物相互作用(DDI)及生成药品医嘱。结果显示模型在剂型匹配准确率49%、DDI识别准确率54.7%、医嘱无错误率65.8%,整体性能较差,提示需领域特异性训练与评估框架。

要点

  • GPT-4o匹配仿制药所有可用剂型的准确率为49%,平均每种药遗漏1.23个、幻觉1.14个。
  • GPT-4o识别药物相互作用对的准确率为54.7%。
  • GPT-4o生成药品医嘱无药物或缩写错误的案例占65.8%。

核心结论

  • GPT-4o在基本用药管理任务上表现持续较差。
  • 需通过临床医生标注数据集进行领域特异性训练,并建立全面评估框架以基准测试性能。

研究设计与样本

  • 研究评估GPT-4o在3项用药管理任务上的性能:识别仿制药可用剂型、识别药物相互作用、生成药品医嘱。
  • 采用临床医生评估及标准LLM指标(TF-IDF向量、归一化Levenshtein相似度、ROUGE-1/ROUGE-L F1分数)对模型原始文本响应进行评价。

主要发现

  • 剂型匹配任务中,GPT-4o匹配仿制药所有可用剂型的准确率为49%,平均每种药遗漏1.23个、幻觉1.14个。
  • 药物相互作用识别任务中,GPT-4o识别DDI对的准确率为54.7%。
  • 药品医嘱生成任务中,GPT-4o生成的医嘱句子无药物或缩写错误的案例占65.8%。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

PURPOSE: Large language models (LLMs) have proven performance for certain diagnostic tasks; however, limited studies have evaluated their consistency in recommending appropriate medication regimens for a given diagnosis. Medication management is a complex task that requires synthesis of drug formulation and complete order instructions for safe use. Here, the performance of GPT-4o, an LLM available with OpenAI's ChatGPT, was tested on 3 medication management tasks. METHODS: GPT-4o performance was tested on 3 medication tasks: identifying available formulations for a given generic drug name, identifying drug-drug interactions (DDIs) for a given medication regimen, and preparing a medication order for a given generic drug name. For each experiment, the model's raw text response was captured exactly as returned and evaluated using clinician evaluation in addition to standard LLM metrics, including Term Frequency-Inverse Document Frequency (TF-IDF) vectors, normalized Levenshtein similarity, and Recall-Oriented Understudy for Gisting Evaluation (ROUGE-1/ROUGE-L) F1 score between each response and its reference string. RESULTS: For the first task of drug-formulation matching, GPT-4o had 49% accuracy for generic medications being matched to all available formulations, with an average of 1.23 omissions per medication and 1.14 hallucinations per medication. For the second task of drug-drug interaction identification, the accuracy was 54.7% for identifying the DDI pair. For the third task, GPT-4o generated order sentences containing no medication or abbreviation errors in 65.8% of the cases. CONCLUSION: Model performance for basic medication tasks was consistently poor. This evaluation highlights the need for domain-specific training through clinician-annotated datasets and a comprehensive evaluation framework for benchmarking performance.

来源:Am J Health Syst Pharm,查看原文。版权归原作者所有。

引用

  1. Henry K, Xu S, Blotske K, et al. Performance evaluation of a large language model for medication management tasks. Am J Health Syst Pharm. 2026. PMID: 42717660. doi:10.1093/ajhp/zxag282

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。