利用GPT-4评估基于代码识别肝硬化及其并发症的阳性预测值
研究对3788份肝硬化入院出院小结进行有限病历审查,比较基于诊断代码与基于GPT-4的分类方法识别肝硬化及其并发症的阳性预测值。以病历审查为金标准,代码分类识别肝硬化的PPV为82.2%,GPT-4识别准确率达87.8%-98.8%;以LLM为银标准,代码分类识别肝硬化的PPV为79.8%。结论认为基于LLM的分类高度准确,可能增强或替代基于代码的队列分类。
要点
- 以病历审查为金标准,代码分类识别肝硬化的PPV为82.2%。
- GPT-4识别肝硬化及其并发症的准确率为87.8%-98.8%。
- 以LLM为银标准,代码分类识别肝硬化的PPV为79.8%。
核心结论
- 基于LLM的分类在识别肝硬化及其并发症方面与人工病历审查相比高度准确。
- LLM可作为银标准在大规模上评估基于代码的分类性能。
- 结果提示LLM可能增强或替代基于代码的队列分类,并对病历审查的必要性提出疑问。
研究设计与样本
- 提取3788份肝硬化入院出院小结并进行有限病历审查。
- 使用生成式预训练变换器4构建零样本提示,判断肝硬化及其并发症是否为本次住院的活动性问题。
- 以病历审查为金标准,比较基于代码分类与基于LLM分类的阳性预测值;并以LLM为银标准,比较基于代码分类与基于LLM分类的阳性预测值。
主要发现
- 与金标准病历审查相比,代码分类识别肝硬化的PPV为82.2%。
- 与金标准病历审查相比,代码分类识别HE的PPV为41.7%。
- 与金标准病历审查相比,代码分类识别腹水的PPV为72.8%。
- 与金标准病历审查相比,代码分类识别胃肠道出血的PPV为59.8%。
- 与金标准病历审查相比,代码分类识别自发性细菌性腹膜炎的PPV为48.8%。
关键数值与推荐
- 以LLM为银标准,代码分类识别肝硬化的PPV为79.8%。
- 以LLM为银标准,代码分类识别HE的PPV为53.9%。
- 以LLM为银标准,代码分类识别腹水的PPV为55.3%。
- 以LLM为银标准,代码分类识别胃肠道出血的PPV为67.6%。
- 以LLM为银标准,代码分类识别自发性细菌性腹膜炎的PPV为65.5%。
临床怎么用
- 原文未据此给出临床推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND AND AIMS: Diagnosis code classification is a common method for cohort identification in cirrhosis research, but it is often inaccurate and augmented by labor-intensive chart review. Natural language processing using large language models (LLMs) is a potentially more accurate method. To assess LLMs' potential for cirrhosis cohort identification, we compared code-based versus LLM-based classification with chart review as a "gold standard." APPROACH AND RESULTS: We extracted and conducted a limited chart review of 3788 discharge summaries of cirrhosis admissions. We engineered zero-shot prompts using a Generative Pre-trained Transformer 4 to determine whether cirrhosis and its complications were active hospitalization problems. We calculated positive predictive values (PPVs) of LLM-based classification versus limited chart review and PPVs of code-based versus LLM-based classification as a "silver standard" in all 3788 summaries. Compared to gold standard chart review, code-based classification achieved PPVs of 82.2% for identifying cirrhosis, 41.7% for HE, 72.8% for ascites, 59.8% for gastrointestinal bleeding, and 48.8% for spontaneous bacterial peritonitis. Compared to the chart review, Generative Pre-trained Transformer 4 achieved 87.8%-98.8% accuracies for identifying cirrhosis and its complications. Using LLM as a silver standard, code-based classification achieved PPVs of 79.8% for identifying cirrhosis, 53.9% for HE, 55.3% for ascites, 67.6% for gastrointestinal bleeding, and 65.5% for spontaneous bacterial peritonitis. CONCLUSIONS: LLM-based classification was highly accurate versus manual chart review in identifying cirrhosis and its complications. This allowed us to assess the performance of code-based classification at scale using LLMs as a silver standard. These results suggest LLMs could augment or replace code-based cohort classification and raise questions regarding the necessity of chart review.
引用
- Far AT, Bastani A, Lee A, et al. Evaluating the positive predictive value of code-based identification of cirrhosis and its complications utilizing GPT-4. Hepatology. 2024. PMID: 39378414. doi:10.1097/HEP.0000000000001115
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。