文献速递关节外科

大型语言模型能否解读电子健康记录数据以推断全膝关节置换术后膝关节骨关节炎结局评分-关节置换评分的最小临床重要差异达成情况

Can a Large Language Model Interpret Data in the Electronic Health Record to Infer Minimum Clinically Important Difference Achievement of Knee Osteoarthritis Outcome Score-Joint Replacement Score Following Total Knee Arthroplasty?
PubMed 文献 · J Arthroplasty · 2025-03-24

这项回顾性病例对照研究选取40例达成1年KOOS-JR最小临床重要差异的全膝关节置换患者和40例未达成者,由ChatGPT 3.5解读去标识化的电子健康记录笔记判断术后改善,并与一名盲法骨科医生的评估比较;ChatGPT敏感度97%但特异度仅33%,医生敏感度90%、特异度63%,ChatGPT准确率65%,医生76%。

要点

  • ChatGPT判断KOOS-JR最小临床重要差异达成的敏感度为97%,特异度仅33%。
  • 盲法骨科医生评估的敏感度为90%,特异度为63%。
  • ChatGPT准确率为65%,医生为76%,假阳性常因模型未获取或未正确解读病历中的问题征象。

核心结论

  • ChatGPT对KOOS-JR最小临床重要差异达成的评估敏感度为97%,但特异度仅33%。
  • 假阳性常见原因是大型语言模型未获取或未正确解读病历中的问题征象。
  • 这是对通用大型语言模型基于病历笔记评估患者结局当前能力的初步评价。
  • 骨科医生对完整病历的评估提示有机会改善这一基线表现,可能实现大型医疗系统内的质量监测和最佳实践识别。
  • 需要额外工作来优化模型性能并确认该方法的效用。

研究设计与样本

  • 回顾性病例对照研究,选取40例达成1年KOOS-JR最小临床重要差异的全膝关节置换患者和40例未达成者。
  • 纳入术前零至六个月及术后九至十五个月的骨科、急诊和初级保健笔记,并去标识化。
  • ChatGPT 3.5解读笔记以判断患者术后是否改善。
  • 一名盲法骨科医生使用全部病历信息对患者进行分类。
  • 计算ChatGPT和医生回答的敏感度、特异度和准确率。

主要发现

  • ChatGPT对80例中的78例进行了分类,敏感度97%,特异度仅33%。
  • 骨科医生评估的敏感度为90%,特异度为63%。
  • 在达成与未达成最小临床重要差异患者等量分布的情况下,ChatGPT准确率为65%,医生为76%。

临床怎么用

  • 原文未据此给出推荐。

局限性

  • 这是对通用大型语言模型基于病历笔记评估患者结局当前能力的初步评价。
  • 需要额外工作来优化模型性能并确认该方法的效用。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Obtaining total knee arthroplasty patient-reported outcomes for quality assessment is costly and difficult. We asked whether a large language model (LLM) could interpret electronic health record notes to differentiate patients attaining a 1-year minimum clinically important difference (MCID) for the Knee Osteoarthritis Outcome Score-Joint Replacement (KOOS-JR) from those who did not. We also investigated whether sufficient information to infer MCID achievement exists in the chart by having a blinded orthopaedic surgeon make the same determination. METHODS: In this retrospective case-control study, we selected 40 total knee arthroplasty patients who achieved 1-year KOOS-JR MCID and 40 who did not. Orthopaedic, emergency medicine, and primary care notes from zero to six months preoperatively and nine to 15 months postoperatively were deidentified. ChatGPT 3.5 (ChatGPT) interpreted these notes to determine whether the patient improved after surgery. A blinded orthopaedic surgeon classified these patients using all chart information. The sensitivity, specificity, and accuracy of ChatGPT and the surgeon's responses were calculated. RESULTS: ChatGPT classified 78 of 80 cases with 97% sensitivity, but only 33% specificity. The surgeon's assessment had 90% sensitivity and 63% specificity. Given the equal distribution of patients meeting or not meeting MCID, Chat GPT's accuracy was 65%. The surgeon's was 76%. CONCLUSIONS: ChatGPT's assessment of KOOS-JR MCID attainment had 97% sensitivity, but only 33% specificity. False positives were commonly due to the LLM not having access to, or not properly interpreting, signs of problems in the chart. This was an initial evaluation of the current ability of a general-purpose LLM to evaluate patient outcomes based on information in chart notes. An orthopaedic surgeon's assessment of the full chart suggests an opportunity to improve on this baseline performance, possibly enabling quality monitoring and identification of best practices across a large health care system. Additional work is needed to optimize model performance and confirm the utility of this approach.

来源:J Arthroplasty,查看原文。版权归原作者所有。

引用

  1. Zalikha L, Hong TS, Small EA, et al. Can a Large Language Model Interpret Data in the Electronic Health Record to Infer Minimum Clinically Important Difference Achievement of Knee Osteoarthritis Outcome Score-Joint Replacement Score Following Total Knee Arthroplasty? J Arthroplasty. 2025. PMID: 40139476. doi:10.1016/j.arth.2025.03.049

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。