提升糖尿病视网膜病变查询应答:评估眼科中的大型语言模型
横断面研究于2024年2月1日至3月31日评估六种大型语言模型对42个糖尿病视网膜病变相关问题的应答。三位顾问级眼科医师独立评分显示准确率高,ChatGPT-4.0获97.6%好评,前三款模型无差评;引入自我纠正提示后平均准确分从6.4升至7.5。结论认为模型有潜力,但临床整合前需改善可读性并持续验证。
要点
- 六种大型语言模型对42个糖尿病视网膜病变问题的252条应答显示准确率高。
- ChatGPT-4.0获97.6%好评,前三款模型无差评。
- 自我纠正提示使平均准确分由6.4升至7.5。
核心结论
- 大型语言模型有潜力对糖尿病视网膜病变相关问题提供准确且全面的应答,对眼科应用有优势。
- 临床整合前需进一步改进可读性,并需持续验证评估以确保可靠性。
研究设计与样本
- 横断面研究,于2024年2月1日至2024年3月31日纳入六种大型语言模型,评估其对42个糖尿病视网膜病变相关问题的应答。
- 三位顾问级眼科医师独立评估应答的准确性与全面性,并对自我纠正能力和可读性进行统计分析。
主要发现
- 六种大型语言模型的252条应答平均词数为155.3至304.3,平均字符数为975.3至2043.5。
- 可读性评分显示显著变异,ChatGPT-3.5的可读性水平最低。
- 应答准确性高,ChatGPT-4.0获97.6%好评,前三款模型无差评。
- 引入自我纠正提示后,平均准确分从6.4显著提高至7.5。
局限性
- 临床整合前需进一步改进可读性,并需持续验证评估以确保可靠性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Diabetic retinopathy (DR) is a leading cause of blindness, with an increasing reliance on large language models (LLMs) for health-related information. The specificity of LLM-generated responses to DR queries is yet to be established, prompting an investigation into their suitability for ophthalmological contexts. METHODS: A cross-sectional study involving six LLMs was conducted to ascertain the accuracy and comprehensiveness of responses to 42 DR-related questions from 1 February 2024 to 31 March 2024. Three consultant-level ophthalmologists independently assessed the responses, grading them on accuracy and comprehensiveness. Additionally, the self-correction capability and readability of the responses were analysed statistically. RESULTS: An analysis of 252 responses from six LLMs showed an average word count ranging from 155.3 to 304.3 and an average character count ranging from 975.3 to 2043.5. The readability scores showed significant variability, with ChatGPT-3.5 displaying the lowest readability level. The accuracy of the responses was high, with ChatGPT-4.0 receiving 97.6% good ratings and no 'poor' grades for the top three models. After introducing a self-correction prompt, the average accuracy score demonstrated a significant improvement, increasing from 6.4 to 7.5. CONCLUSION: LLMs have the potential to provide accurate and comprehensive responses to DR-related questions, making them advantageous for ophthalmology applications. However, before clinical integration, further refinement is needed to address readability, and continuous validation assessments are imperative to ensure reliability.
引用
- Wu H, Su Z, Pan X, et al. Enhancing diabetic retinopathy query responses: assessing large language model in ophthalmology. Br J Ophthalmol. 2025. PMID: 40588331. doi:10.1136/bjo-2024-325861
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。