评估GPT-4在复杂眼科病例中的医学推理能力
本研究在422例《美国医学会杂志·眼科学》临床挑战病例上测试GPT-4,用零样本计划-求解+(PS+)提示策略,诊断平均准确率48.0%(95%CI 43.1%至52.9%),下一步处置准确率63.0%(95%CI 58.2%至67.6%)。诊断正确时下一步处置准确率75.2%,诊断错误时50.2%;诊断正确时下一步处置准确的可能性是诊断错误时的三倍(p<0.001)。与委员会认证眼科医生相比无显著差异,但高年资住院医师在诊断和下一步处置准确率上优于GPT-4。
要点
- PS+提示下GPT-4诊断准确率48.0%,下一步处置准确率63.0%。
- 诊断正确时75.2%的下一步处置正确,诊断错误时仅50.2%正确。
- 高年资住院医师在诊断和下一步处置准确率上均优于GPT-4。
核心结论
- 改进提示可提升GPT-4在复杂临床情境中的表现,但在本研究中未超越眼科受训者。
- 专用大语言模型有望在未来辅助医学决策和诊断。
研究设计与样本
- 在422例《美国医学会杂志·眼科学》临床挑战病例上测试GPT-4。
- 要求模型确定诊断(开放式问题)并识别下一步处置(多项选择题)。
- 使用两种零样本提示策略生成回答,包括零样本计划-求解+(PS+)。
- 将表现最佳的模型与人类评分者进行基准比较。
主要发现
- 使用PS+提示时,GPT-4诊断平均准确率为48.0%(95%CI 43.1%至52.9%),下一步处置平均准确率为63.0%(95%CI 58.2%至67.6%)。
- 下一步处置准确率在各亚专科间无显著差异(p=0.44)。
- 病理和肿瘤的的诊断准确率显著高于葡萄膜炎(p=0.027)。
- 诊断准确时,75.2%(95%CI 68.6%至80.9%)的下一步处置正确;诊断不准确时,50.2%(95%CI 43.8%至56.6%)的下一步处置正确。
- 诊断正确时下一步处置准确的可能性是诊断错误时的三倍(p<0.001)。
关键数值与推荐
- 诊断准确率48.0%(95%CI 43.1%至52.9%),下一步处置准确率63.0%(95%CI 58.2%至67.6%)。
- 诊断正确时下一步处置准确率75.2%(95%CI 68.6%至80.9%),诊断错误时50.2%(95%CI 43.8%至56.6%)。
- 下一步处置准确率在各亚专科间无显著差异(p=0.44)。
- 病理和肿瘤诊断准确率显著高于葡萄膜炎(p=0.027)。
- GPT-4与委员会认证眼科医生在诊断准确率和决策方面无显著差异。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND/AIMS: This study assesses the proficiency of Generative Pre-trained Transformer (GPT)-4 in answering questions about complex clinical ophthalmology cases. METHODS: We tested GPT-4 on 422 Journal of the American Medical Association Ophthalmology Clinical Challenges, and prompted the model to determine the diagnosis (open-ended question) and identify the next-step (multiple-choice question). We generated responses using two zero-shot prompting strategies, including zero-shot plan-and-solve+ (PS+), to improve the reasoning of the model. We compared the best-performing model to human graders in a benchmarking effort. RESULTS: Using PS+ prompting, GPT-4 achieved mean accuracies of 48.0% (95% CI (43.1% to 52.9%)) and 63.0% (95% CI (58.2% to 67.6%)) in diagnosis and next step, respectively. Next-step accuracy did not significantly differ by subspecialty (p=0.44). However, diagnostic accuracy in pathology and tumours was significantly higher than in uveitis (p=0.027). When the diagnosis was accurate, 75.2% (95% CI (68.6% to 80.9%)) of the next steps were correct. Conversely, when the diagnosis was incorrect, 50.2% (95% CI (43.8% to 56.6%)) of the next steps were accurate. The next step was three times more likely to be accurate when the initial diagnosis was correct (p<0.001). No significant differences were observed in diagnostic accuracy and decision-making between board-certified ophthalmologists and GPT-4. Among trainees, senior residents outperformed GPT-4 in diagnostic accuracy (p≤0.001 and 0.049) and in accuracy of next step (p=0.002 and 0.020). CONCLUSION: Improved prompting enhances GPT-4's performance in complex clinical situations, although it does not surpass ophthalmology trainees in our context. Specialised large language models hold promise for future assistance in medical decision-making and diagnosis.
引用
- Milad D, Antaki F, Milad J, et al. Assessing the medical reasoning skills of GPT-4 in complex ophthalmology cases. Br J Ophthalmol. 2024. PMID: 38365427. doi:10.1136/bjo-2023-325053
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。