利用谷歌网络搜索分析评估ChatGPT在全关节置换术中的效用
本研究通过模拟患者网络搜索,比较ChatGPT与谷歌网络搜索在全膝关节置换术和全髋关节置换术常见问题上的表现。结果显示,两种搜索工具仅25%的问题相似,谷歌搜索的常见问题多来自商业网站,而ChatGPT的回答多来自政府网站。对于数值型问题,55%的常见问题在两种工具间回答不同。作者认为ChatGPT作为患者潜在资源需进一步证实。
要点
- 谷歌搜索与ChatGPT在全部搜索词下仅25%的常见问题相似。
- 谷歌搜索的20个常见问题中13个来自商业网站,ChatGPT的20个中15个来自政府网站。
- 数值型常见问题中55%在谷歌搜索与ChatGPT间回答不同。
核心结论
- 谷歌网络搜索与ChatGPT对开放式和离散问题的常见问题及回答呈现异质性。
- ChatGPT应保持作为患者潜在资源的热门使用,直至其提供可信信息的能力被证实并与医患双方目标一致前,需进一步证实。
研究设计与样本
- 研究通过谷歌网络搜索输入「total knee replacement」和「total hip replacement」,提取前10个常见问题及来源网站。
- 将相应指令输入ChatGPT以获取常见问题,并重复谷歌搜索以识别包含数值回答的前10个常见问题,再输入ChatGPT记录问答。
主要发现
- 全部搜索词下,谷歌网络搜索与ChatGPT搜索的20个问题中5个(25%)相似。
- 谷歌网络搜索的20个问题中13个由商业网站提供。
- ChatGPT的20个问题中15个(75%)由政府采购网站回答,最常见为PubMed。
- 数值型问题中,20个常见问题里11个(55%)在谷歌网络搜索与ChatGPT间提供不同回答。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Rapid technological advancements have laid the foundations for the use of artificial intelligence in medicine. The promise of machine learning (ML) lies in its potential ability to improve treatment decision making, predict adverse outcomes, and streamline the management of perioperative healthcare. In an increasing consumer-focused health care model, unprecedented access to information may extend to patients using ChatGPT to gain insight into medical questions. The main objective of our study was to replicate a patient's internet search in order to assess the appropriateness of ChatGPT, a novel machine learning tool released in 2022 that provides dialogue responses to queries, in comparison to Google Web Search, the most widely used search engine in the United States today, as a resource for patients for online health information. For the 2 different search engines, we compared i) the most frequently asked questions (FAQs) associated with total knee arthroplasty (TKA) and total hip arthroplasty (THA) by question type and topic; ii) the answers to the most frequently asked questions; as well as iii) the FAQs yielding a numerical response. METHODS: A Google web search was performed with the following search terms: "total knee replacement" and "total hip replacement." These terms were individually entered and the first 10 FAQs were extracted along with the source of the associated website for each question. The following statements were inputted into ChatGPT: 1) "Perform a google search with the search term 'total knee replacement' and record the 10 most FAQs related to the search term" as well as 2) "Perform a google search with the search term 'total hip replacement' and record the 10 most FAQs related to the search term." A Google web search was repeated with the same search terms to identify the first 10 FAQs that included a numerical response for both "total knee replacement" and "total hip replacement." These questions were then inputted into ChatGPT and the questions and answers were recorded. RESULTS: There were 5 of 20 (25%) questions that were similar when performing a Google web search and a search of ChatGPT for all search terms. Of the 20 questions asked for the Google Web Search, 13 of 20 were provided by commercial websites. For ChatGPT, 15 of 20 (75%) questions were answered by government websites, with the most frequent one being PubMed. In terms of numerical questions, 11 of 20 (55%) of the most FAQs provided different responses between a Google web search and ChatGPT. CONCLUSION: A comparison of the FAQs by a Google web search with attempted replication by ChatGPT revealed heterogenous questions and responses for open and discrete questions. ChatGPT should remain a trending use as a potential resource to patients that needs further corroboration until its ability to provide credible information is verified and concordant with the goals of the physician and the patient alike.
引用
- Dubin JA, Bains SS, Chen Z, et al. Using a Google Web Search Analysis to Assess the Utility of ChatGPT in Total Joint Arthroplasty. J Arthroplasty. 2023. PMID: 37040823. doi:10.1016/j.arth.2023.04.007
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。