评估ChatGPT回答全膝关节置换术相关问题的能力
本研究将15条美国骨科医师学会临床实践指南转化为问题,由11名成人关节重建专科培训外科医生对ChatGPT的回答按相关性、准确性、清晰度、完整性、循证性和一致性六项特征进行1至5分评分。结果显示所有特征评分均高于平均(>3分),相关性得分最高(4.43±0.77),一致性得分最低(3.54±1.10),总体评分者间信度为0.33(差)。
要点
- ChatGPT对全膝关节置换术指南相关问题回答的各特征评分均高于平均(>3分)。
- 相关性得分最高(4.43±0.77),一致性得分最低(3.54±1.10)。
- 总体评分者间信度为0.33,可靠性差;风险因素组回答最佳,疼痛/功能结局组最差。
核心结论
- ChatGPT能以高于平均的准确性回答关于全膝关节置换术成熟临床指南的问题,但可靠性多变。
研究设计与样本
- 将15条美国骨科医师学会临床实践指南转化为ChatGPT提示问题,创建在线调查,由11名完成成人关节重建专科培训的外科医生对回答进行评分。
主要发现
- 所有回答特征评分均高于平均(即评分>3分)。
- 相关性得分最高(4.43±0.77),一致性得分最低(3.54±1.10)。
- 风险因素组提示的回答最佳,疼痛/功能结局组回答最低。
- 总体评分者间信度为0.33(可靠性差),相关性信度最高(0.43),循证性信度最低(0.28)。
关键数值与推荐
- 外科医生基于相关性、准确性、清晰度、完整性、循证性和一致性六项特征对ChatGPT回答进行1至5分评分。
临床怎么用
- 原文未据此给出推荐。
局限性
- 本研究为理解ChatGPT等大型语言模型人工智能及其在关节置换领域表现的第一步。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Artificial intelligence in the field of orthopaedics has been a topic of increasing interest and opportunity in recent years. Its applications are widespread both for physicians and patients, including use in clinical decision-making, in the operating room, and in research. In this study, we aimed to assess the quality of ChatGPT answers when asked questions related to total knee arthroplasty. METHODS: ChatGPT prompts were created by turning 15 of the American Academy of Orthopaedic Surgeons Clinical Practice Guidelines into questions. An online survey was created, which included screenshots of each prompt and answers to the 15 questions. Surgeons were asked to grade ChatGPT answers from 1 to 5 based on their characteristics: (1) relevance, (2) accuracy, (3) clarity, (4) completeness, (5) evidence-based, and (6) consistency. There were 11 Adult Joint Reconstruction fellowship-trained surgeons who completed the survey. Questions were subclassified based on the subject of the prompt: (1) risk factors, (2) implant/intraoperative, and (3) pain/functional outcomes. The average and standard deviation for all answers, as well as for each subgroup, were calculated. Inter-rater reliability (IRR) was also calculated. RESULTS: All answer characteristics were graded as being above average (ie, a score > 3). Relevance demonstrated the highest scores (4.43 ± 0.77) by surgeons surveyed, and consistency demonstrated the lowest scores (3.54 ± 1.10). ChatGPT prompts in the Risk Factors group demonstrated the best responses, while those in the Pain/Functional Outcome group demonstrated the lowest. The overall IRR was found to be 0.33 (poor reliability), with the highest IRR for relevance (0.43) and the lowest for evidence-based (0.28). CONCLUSIONS: ChatGPT can answer questions regarding well-established clinical guidelines in total knee arthroplasty with above-average accuracy but demonstrates variable reliability. This investigation is the first step in understanding large language model artificial intelligence like ChatGPT and how well they perform in the field of arthroplasty.
引用
- Magruder ML, Rodriguez AN, Wong JCJ, et al. Assessing Ability for ChatGPT to Answer Total Knee Arthroplasty-Related Questions. J Arthroplasty. 2024. PMID: 38364879. doi:10.1016/j.arth.2024.02.023
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。