文献速递卫生政策与健康促进免疫规划

在线人工智能模型在基层医疗场景中的准确性评估

Accuracy of Online Artificial Intelligence Models in Primary Care Settings.
PubMed 文献 · Am J Prev Med · 2024-02-12

该研究于2023年6月向ChatGPT-4、2023年10月向Google Bard分别提交56个预防医学与基层医疗问题,由两名医师独立将回答评为准确、不准确或准确但信息缺失,分歧由第三名医师裁定。ChatGPT-4回答中28.6%准确、28.6%不准确、42.8%准确但信息缺失;Bard回答中53.6%准确、17.8%不准确、28.6%准确但信息缺失;两者对CDC和免疫相关问题的回答不准确率均达80%。

要点

  • ChatGPT-4回答中准确28.6%、不准确28.6%、准确但信息缺失42.8%。
  • Bard回答中准确53.6%、不准确17.8%、准确但信息缺失28.6%。
  • 两者对CDC和免疫相关问题的回答不准确率均达80%。

核心结论

  • ChatGPT-4和Bard在提供预防保健准确信息方面显示出潜力,同时凸显了定期更新的关键需求,尤其在快速发展的医学领域。
  • 相当比例的人工智能模型回答被判定为「准确但信息缺失」,强调在寻求医疗信息时应将人工智能工具视为补充资源。

研究设计与样本

  • 研究于2023年6月向ChatGPT-4、2023年10月向Google Bard分别提交56个问题,由两名医师独立评审,分歧由第三名医师裁定。

主要发现

  • 评审者初始评分一致性:ChatGPT-4的Cohen Kappa为0.76(95%CI 0.61-0.90),Bard为0.89(95%CI 0.79-0.99)。
  • ChatGPT-4回答中28.6%准确、28.6%不准确、42.8%准确但信息缺失。
  • Bard回答中53.6%准确、17.8%不准确、28.6%准确但信息缺失。
  • 两者对CDC和免疫相关问题的回答不准确率均达80%。

局限性

  • 原文未报告局限性

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

INTRODUCTION: The importance of preventive medicine and primary care in the sphere of public health is expanding, yet a gap exists in the utilization of recommended medical services. As patients increasingly turn to online resources for supplementary advice, the role of artificial intelligence (AI) in providing accurate and reliable information has emerged. The present study aimed to assess ChatGPT-4's and Google Bard's capacity to deliver accurate recommendations in preventive medicine and primary care. METHODS: Fifty-six questions were formulated and presented to ChatGPT-4 in June 2023 and Google Bard in October 2023, and the responses were independently reviewed by two physicians, with each answer being classified as "accurate," "inaccurate," or "accurate with missing information." Disagreements were resolved by a third physician. RESULTS: Initial inter-reviewer agreement on grading was substantial (Cohen's Kappa was 0.76, 95%CI [0.61-0.90] for ChatGPT-4 and 0.89, 95%CI [0.79-0.99] for Bard). After reaching a consensus, 28.6% of ChatGPT-4-generated answers were deemed accurate, 28.6% inaccurate, and 42.8% accurate with missing information. In comparison, 53.6% of Bard-generated answers were deemed accurate, 17.8% inaccurate, and 28.6% accurate with missing information. Responses to CDC and immunization-related questions showed notable inaccuracies (80%) in both models. CONCLUSIONS: ChatGPT-4 and Bard demonstrated potential in offering accurate information in preventive care. It also brought to light the critical need for regular updates, particularly in the rapidly evolving areas of medicine. A significant proportion of the AI models' responses were deemed "accurate with missing information," emphasizing the importance of viewing AI tools as complementary resources when seeking medical information.

来源:Am J Prev Med,查看原文。版权归原作者所有。

引用

  1. Kassab J, Hadi El Hajjar A, Wardrop RM, et al. Accuracy of Online Artificial Intelligence Models in Primary Care Settings. Am J Prev Med. 2024. PMID: 38354991. doi:10.1016/j.amepre.2024.02.006

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。