评估大型语言模型为膝骨关节炎患者提供的全膝关节置换术信息:复合临床可靠性评分的发展与应用
该研究使用90个患者面向的全膝关节置换术问题测试三种大型语言模型,由两名骨科医生从六个维度评分。结果显示Program 2的复合临床可靠性评分最高(中位数84.0),Program 3证据引用有效性指数更好但可读性较差。作者认为大型语言模型输出非确定性,应仅作为专业咨询的辅助。
要点
- Program 2的复合临床可靠性评分最高(中位数84.0,IQR 82.0-85.0)。
- Program 3的证据引用有效性指数优于Program 2和Program 1。
- 大型语言模型输出具有非确定性,不能替代专业医疗咨询。
核心结论
- 在此框架和评估期内,Program 2在复合可靠性和可读性上排名最高,Program 3证据可追溯性更好但文本可读性较差,Program 1居中。
- 大型语言模型输出具有非确定性且持续更新,这些发现是探索性的而非持久排名,大型语言模型应仅作为专业咨询的辅助。
研究设计与样本
- 研究使用来自Google「People Also Ask」的90个患者面向的全膝关节置换术问题测试三种大型语言模型。
- 由两名经过专科培训的骨科医生在兩個时间点独立对270条回复进行评分,涵盖复合临床可靠性评分的六个维度。
主要发现
- Program 2的复合临床可靠性评分最高(中位数84.0,IQR 82.0至85.0),优于Program 3(76.5,65.0至83.8)和Program 1(73.0,69.0至79.0)(P < 0.0001)。
- Program 3的证据引用有效性指数(7.7 ± 1.5)高于Program 2(7.1 ± 0.8)和Program 1(5.6 ± 0.6)。
- 可读性差异显著(P < 0.0001):Program 2最易读(Coleman-Liau指数12.3 ± 1.6),Program 1居中(12.8 ± 1.7),Program 3最难读(13.7 ± 2.5)。
- 评分者间一致性优秀(ICC 0.87,95% CI 0.82至0.91)。
关键数值与推荐
- 复合临床可靠性评分包含六个维度:事实准确性、USMLE风格临床正确性、GRADE指南依从性、幻觉严重性指数(反向)、临床安全评分和证据引用有效性指数。
- 可读性通过Coleman-Liau指数评估,并用Flesch-Kincaid年级水平和SMOG corroborated。
- 评分者间可靠性使用组内相关系数,模型间差异使用Friedman检验。
临床怎么用
- 临床医生应了解大型语言模型提供的全膝关节置换术信息可靠性因模型而异,Program 2在复合可靠性和可读性上表现最佳。
- 大型语言模型应仅作为专业咨询的辅助工具,不应替代骨科医生的专业判断。
- 对于需要证据可追溯性的场景,可参考Program 3,但需注意其文本可读性较差。
局限性
- 大型语言模型输出具有非确定性且持续更新,研究结果是探索性的而非持久排名。
- 研究评估期和框架特定,结论可能不适用于其他时间或场景。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Total knee arthroplasty (TKA) is the definitive surgical treatment for advanced knee osteoarthritis. As patients increasingly consult large language models (LLMs) for medical guidance, reliability, safety, and readability are concerns. This study compared patient-facing TKA information from three LLMs using the Composite Clinical Reliability Score (CCRS). METHODS: We used 90 patient-facing TKA questions from Google's People Also Ask to test three LLMs. We employed two fellowship-trained orthopaedic surgeons to independently score the 270 responses at two timepoints across six CCRS domains: factual accuracy, United States Medical Licensing Examination (USMLE)-style clinical correctness, Grading of Recommendations Assessment, Development, and Evaluation (GRADE) guideline adherence, hallucination severity index (HSI, inverse), clinical safety score (CSS), and evidence citation validity index (ECVI). Readability was assessed with the Coleman-Liau Index (CLI), corroborated by the Flesch-Kincaid Grade Level and Simple Measure of Gobbledygook (SMOG). Inter-rater reliability used the intraclass correlation coefficient (ICC), and inter-model differences the Friedman test. RESULTS: Program 2 achieved the highest CCRS (median 84.0, interquartile range [IQR] 82.0 to 85.0), outperforming Program 3 (76.5, 65.0 to 83.8) and Program 1 (73.0, 69.0 to 79.0) (P < 0.0001), with superior accuracy, guideline adherence, hallucination control, and safety framing. Program 3 showed stronger ECVI (7.7 ± 1.5) than Program 2 (7.1 ± 0.8) and Program 1 (5.6 ± 0.6). Readability differed significantly (P < 0.0001): Program 2 was most accessible (CLI 12.3 ± 1.6), Program 1 intermediate (12.8 ± 1.7), and Program 3 least readable (13.7 ± 2.5). Agreement was excellent (ICC 0.87, 95% confidence interval [CI] 0.82 to 0.91). CONCLUSION: Within this framework and evaluation period, Program 2 ranked highest on composite reliability and readability, Program 3 offered better evidence traceability, but less accessible text, and Program 1 was intermediate. Because LLM outputs are non-deterministic and continuously updated, these findings are exploratory rather than a durable ranking, and LLMs should serve only as adjuncts to professional consultation.
引用
- Muthu S, Kolarpatti Ponnusamy DV, Rathinam MM, et al. Evaluating Large Language Models for Patient-Facing Total Knee Arthroplasty Information in Knee Osteoarthritis: Development and Application of the Composite Clinical Reliability Score. J Arthroplasty. 2026. PMID: 42950900. doi:10.1016/j.arth.2026.10.030
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。