文献速递衰弱、肌少症与跌倒老年综合评估

使用指令调优的大语言模型进行可靠的临床衰弱评分

Use of a large language model with instruction-tuning for reliable clinical frailty scoring.
PubMed 文献 · J Am Geriatr Soc · 2024-08-06

本研究利用七个标准化患者场景评估GPT-3.5-turbo模型进行临床衰弱量表(CFS)评分的一致性和可靠性,比较基础提示与指令调优提示两种方法,并与历史人类评分对比。结果显示指令调优提示的组间可靠性高(Fleiss Kappa为0.887),中位评分与人类评分差异不超过1分,但在日常生活活动信息较少时可能高估衰弱。

要点

  • 指令调优提示使GPT-3.5-turbo的CFS评分组间可靠性达Fleiss Kappa 0.887。
  • LLM中位评分与人类评分差异不超过1分,但分布差异在七个场景中五个显著。
  • 日常生活活动信息不足时LLM可能高估衰弱评分。

核心结论

  • 大语言模型在临床衰弱评分中具有高一致性和可靠性潜力,指令调优是优化医疗应用的有效提示工程方法。

研究设计与样本

  • 研究使用七个标准化患者场景评估OpenAI GPT-3.5-turbo模型进行CFS评分的一致性和可靠性,测试基础提示与指令调优提示两种方法。

主要发现

  • LLM中位评分与人类评分相似,差异不超过1分。
  • 基础提示与指令调优提示在七个场景中的五个出现评分分布显著差异。
  • 指令调优提示的组间可靠性高(Fleiss Kappa为0.887),在所有场景中产生一致响应。
  • 在日常生活活动信息较不明确的场景中评分困难。
  • 当日常生活活动信息较少时,LLM可能高估衰弱评分。

关键数值与推荐

  • 指令调优提示的组间可靠性Fleiss Kappa为0.887。
  • LLM中位评分与人类评分差异不超过1分。
  • 基础与指令调优提示的评分分布在七个场景中的五个有显著差异。

临床怎么用

  • 临床可考虑使用指令调优的大语言模型辅助CFS评分以提高一致性。
  • 在日常生活活动信息不足时需警惕LLM高估衰弱评分。

局限性

  • 研究仅使用七个标准化患者场景,样本量有限。
  • 未报告具体样本量、P值等统计细节。
  • 仅评估了GPT-3.5-turbo模型,结果可能不适用于其他模型。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Frailty is an important predictor of health outcomes, characterized by increased vulnerability due to physiological decline. The Clinical Frailty Scale (CFS) is commonly used for frailty assessment but may be influenced by rater bias. Use of artificial intelligence (AI), particularly Large Language Models (LLMs) offers a promising method for efficient and reliable frailty scoring. METHODS: The study utilized seven standardized patient scenarios to evaluate the consistency and reliability of CFS scoring by OpenAI's GPT-3.5-turbo model. Two methods were tested: a basic prompt and an instruction-tuned prompt incorporating CFS definition, a directive for accurate responses, and temperature control. The outputs were compared using the Mann-Whitney U test and Fleiss' Kappa for inter-rater reliability. The outputs were compared with historic human scores of the same scenarios. RESULTS: The LLM's median scores were similar to human raters, with differences of no more than one point. Significant differences in score distributions were observed between the basic and instruction-tuned prompts in five out of seven scenarios. The instruction-tuned prompt showed high inter-rater reliability (Fleiss' Kappa of 0.887) and produced consistent responses in all scenarios. Difficulty in scoring was noted in scenarios with less explicit information on activities of daily living (ADLs). CONCLUSIONS: This study demonstrates the potential of LLMs in consistently scoring clinical frailty with high reliability. It demonstrates that prompt engineering via instruction-tuning can be a simple but effective approach for optimizing LLMs in healthcare applications. The LLM may overestimate frailty scores when less information about ADLs is provided, possibly as it is less subject to implicit assumptions and extrapolation than humans. Future research could explore the integration of LLMs in clinical research and frailty-related outcome prediction.

来源:J Am Geriatr Soc,查看原文。版权归原作者所有。

引用

  1. Kee XLJ, Sng GGR, Lim DYZ, et al. Use of a large language model with instruction-tuning for reliable clinical frailty scoring. J Am Geriatr Soc. 2024. PMID: 39105505. doi:10.1111/jgs.19114

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。