视觉语言模型在光学相干断层扫描上检测黄斑疾病特征的表现
这项横断面诊断准确性研究使用开源光学相干断层扫描图像数据库,纳入50例唯一患者(健康者及黄斑裂孔、糖尿病黄斑水肿、中心性浆液性脉络膜视网膜病变、年龄相关性黄斑变性患者),评估Gemini Pro视觉语言模型在黄斑疾病专家级任务中的表现。模型平均F1评分为10.7%(95%CI,2.4-19.2),正确诊断17/50例(34%;95%CI,22-48),转诊建议正确28/50例(56%;95%CI,42-70)。作者结论认为该通用视觉语言模型在黄斑疾病特征检测和管理方面视觉能力有限,但自我矛盾率低,提示语言能力较强。
要点
- 平均F1评分为10.7%(95%CI,2.4-19.2),黄斑裂孔F1评分最高为36.4%。
- 正确诊断17/50例(34%;95%CI,22-48)。
- 转诊建议正确28/50例(56%),过度谨慎10/50例(20%),谨慎不足12/50例(24%)。
核心结论
- 通用视觉语言模型在黄斑疾病特征检测和管理方面视觉能力有限,但自我矛盾率低,提示语言能力较强。
- 随着视觉语言模型不断改进,在大型基准数据集上验证其性能将有助于确定其在眼科中的潜力。
研究设计与样本
- 这是一项横断面诊断准确性研究,使用开源光学相干断层扫描图像数据库评估通用视觉语言模型在眼科特定任务中的表现。
- 数据集纳入50例唯一患者的OCT B扫描,包括健康个体及黄斑裂孔、糖尿病黄斑水肿、中心性浆液性脉络膜视网膜病变和年龄相关性黄斑变性患者。
- 每张OCT扫描针对10个关键病理特征、转诊建议和治疗进行标注。
- 图像使用Cirrus高清晰度OCT设备(Carl Zeiss Meditec)在印度金奈Sankara Nethralaya眼科医院采集,数据集于2018年12月发布,图像采集日期未说明。
- Gemini Pro于2023年12月15日使用标准提示提取结构化回答。
主要发现
- 平均F1评分为10.7%(95%CI,2.4-19.2)。
- 可获得可测量F1评分的特征包括:黄斑裂孔36.4%(95%CI,0-71.4)、色素上皮脱离26.1%(95%CI,0-46.2)、视网膜下高反射物质24.0%(95%CI,0-45.2)、视网膜下液20.0%(95%CI,0-45.5)。
- 正确诊断17/50例(34%;95%CI,22-48)。
- 转诊建议正确28/50例(56%;95%CI,42-70),过度谨慎10/50例(20%;95%CI,10-32),谨慎不足12/50例(24%;95%CI,12-36)。
- 转诊与治疗一致性分别正确48/50例(96%;95%CI,90-100)和48/49例(98%;95%CI,94-100)。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
IMPORTANCE: Vision-language models (VLMs) are a novel artificial intelligence technology capable of processing image and text inputs. While demonstrating strong generalist capabilities, their performance in ophthalmology has not been extensively studied. OBJECTIVE: To assess the performance of the Gemini Pro VLM in expert-level tasks for macular diseases from optical coherence tomography (OCT) scans. DESIGN, SETTING, AND PARTICIPANTS: This was a cross-sectional diagnostic accuracy study evaluating a generalist VLM on ophthalmology-specific tasks using the open-source Optical Coherence Tomography Image Database. The dataset included OCT B-scans from 50 unique patients: healthy individuals and those with macular hole, diabetic macular edema, central serous chorioretinopathy, and age-related macular degeneration. Each OCT scan was labeled for 10 key pathological features, referral recommendations, and treatments. The images were captured using a Cirrus high definition OCT machine (Carl Zeiss Meditec) at Sankara Nethralaya Eye Hospital, Chennai, India, and the dataset was published in December 2018. Image acquisition dates were not specified. EXPOSURES: Gemini Pro, using a standard prompt to extract structured responses on December 15, 2023. MAIN OUTCOMES AND MEASURES: The primary outcome was model responses compared against expert labels, calculating F1 scores for each pathological feature. Secondary outcomes included accuracy in diagnosis, referral urgency, and treatment recommendation. The model's internal concordance was evaluated by measuring the alignment between referral and treatment recommendations, independent of diagnostic accuracy. RESULTS: The mean F1 score was 10.7% (95% CI, 2.4-19.2). Measurable F1 scores were obtained for macular hole (36.4%; 95% CI, 0-71.4), pigment epithelial detachment (26.1%; 95% CI, 0-46.2), subretinal hyperreflective material (24.0%; 95% CI, 0-45.2), and subretinal fluid (20.0%; 95% CI, 0-45.5). A correct diagnosis was achieved in 17 of 50 cases (34%; 95% CI, 22-48). Referral recommendations varied: 28 of 50 were correct (56%; 95% CI, 42-70), 10 of 50 were overcautious (20%; 95% CI, 10-32), and 12 of 50 were undercautious (24%; 95% CI, 12-36). Referral and treatment concordance were very high, with 48 of 50 (96%; 95 % CI, 90-100) and 48 of 49 (98%; 95% CI, 94-100) correct answers, respectively. CONCLUSIONS AND RELEVANCE: In this study, a generalist VLM demonstrated limited vision capabilities for feature detection and management of macular disease. However, it showed low self-contradiction, suggesting strong language capabilities. As VLMs continue to improve, validating their performance on large benchmarking datasets will help ascertain their potential in ophthalmology.
引用
- Antaki F, Chopra R, Keane PA, et al. Vision-Language Models for Feature Detection of Macular Diseases on Optical Coherence Tomography. JAMA Ophthalmol. 2024. PMID: 38696177. doi:10.1001/jamaophthalmol.2024.1165
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。