文献速递皮肤肿瘤

大语言模型聊天机器人在回答黑色素瘤患者问题方面是否优于现有患者信息资源?一项比较研究

Do large language model chatbots perform better than established patient information resources in answering patient questions? A comparative study on melanoma.
PubMed 文献 · Br J Dermatol · 2025-01-01

本研究比较了ChatGPT-3.5、ChatGPT-4.0、Gemini与三个荷兰黑色素瘤患者信息资源(PIR)回答50个黑色素瘤相关问题的质量。结果显示,表现最佳的大语言模型在完整性和个性化上优于金标准PIR,但准确性及可读性较低,且随时间推移可复现性下降。

要点

  • 大语言模型在完整性和个性化上优于金标准PIR,但准确性及可读性较低。
  • 大语言模型内部,ChatGPT-3.5准确性最高,Gemini在完整性、个性化及可读性上表现最佳。
  • 随时间推移,所有大语言模型的回答可复现性下降,在替代或补充传统PIR前需改善准确性、可复现性与可及性。

核心结论

  • 大语言模型在提供高度个性化和完整的黑色素瘤患者问题回答方面具有潜力,但在替代或补充传统患者信息资源之前,必须改善并保障其准确性、可复现性和可及性。

研究设计与样本

  • 在基线时评估ChatGPT 3.5版和4.0版、Gemini及三个领先的荷兰黑色素瘤患者信息资源对50个黑色素瘤特异性问题的回答;8个月后再次向大语言模型提问。
  • 比较分析在大语言模型内部及患者信息资源内部使用Friedman方差分析,在表现最佳的大语言模型与金标准患者信息资源之间使用Wilcoxon符号秩检验。
  • 结局指标包括(医学)准确性、完整性、个性化、可读性,以及大语言模型的可复现性。

主要发现

  • 大语言模型内部,ChatGPT-3.5准确性最高(P = 0.009)。
  • Gemini在完整性(P < 0.001)、个性化(P = 0.007)和可读性(P < 0.001)方面表现最佳。
  • 患者信息资源在准确性和完整性方面表现一致,全科医生网站在个性化和可读性方面表现突出(个性化P = 0.013,可读性P < 0.001)。
  • 表现最佳的大语言模型在完整性和个性化方面优于金标准患者信息资源,但准确性和可读性较低。
  • 随时间推移,所有大语言模型的回答可复现性下降,各结局指标表现出差异。

关键数值与推荐

  • 大语言模型内部准确性比较:ChatGPT-3.5最高(P = 0.009)。
  • 大语言模型内部完整性、个性化、可读性比较:Gemini最佳(完整性P < 0.001,个性化P = 0.007,可读性P < 0.001)。
  • 患者信息资源内部个性化和可读性比较:全科医生网站最佳(个性化P = 0.013,可读性P < 0.001)。

临床怎么用

  • 大语言模型可作为黑色素瘤患者信息的补充工具,但因其准确性、可读性和可复现性不足,尚不能替代传统患者信息资源。
  • 临床使用时需注意大语言模型回答随时间可复现性下降的问题,对关键信息应加以核实。

局限性

  • 仅评估了荷兰语黑色素瘤相关问题及荷兰患者信息资源,外推至其他语言、疾病领域或地区需谨慎。
  • 仅纳入ChatGPT 3.5、ChatGPT 4.0和Gemini三种大语言模型,未涵盖其他模型。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Large language models (LLMs) have a potential role in providing adequate patient information. OBJECTIVES: To compare the quality of LLM responses with established Dutch patient information resources (PIRs) in answering patient questions regarding melanoma. METHODS: Responses from ChatGPT versions 3.5 and 4.0, Gemini, and three leading Dutch melanoma PIRs to 50 melanoma-specific questions were examined at baseline and for LLMs again after 8 months. Outcomes included (medical) accuracy, completeness, personalization, readability and, additionally, reproducibility for LLMs. Comparative analyses were performed within LLMs and PIRs using Friedman's Anova, and between best-performing LLMs and gold-standard (GS) PIRs using the Wilcoxon signed-rank test. RESULTS: Within LLMs, ChatGPT-3.5 demonstrated the highest accuracy (P = 0.009). Gemini performed best in completeness (P < 0.001), personalization (P = 0.007) and readability (P < 0.001). PIRs were consistent in accuracy and completeness, with the general practitioner's website excelling in personalization (P = 0.013) and readability (P < 0.001). The best-performing LLMs outperformed the GS-PIR on completeness and personalization, yet it was less accurate and less readable. Over time, response reproducibility decreased for all LLMs, showing variability across outcomes. CONCLUSIONS: Although LLMs show potential in providing highly personalized and complete responses to patient questions regarding melanoma, improving and safeguarding accuracy, reproducibility and accessibility is crucial before they can replace or complement conventional PIRs. Large language models (LLMs) are a type of artificial intelligence that can be used to assess large amounts of information, and could play a role in providing patients with accurate information about melanoma. This study, conducted in the Netherlands, aimed to compare the quality of responses from LLMs and established Dutch patient information resources (PIRs) to questions from patients about melanoma. We evaluated the responses of ChatGPT versions 3.5 and 4.0, Gemini, and three leading Dutch melanoma PIRs to 50 specific melanoma-related questions. We looked at these responses initially and asked the questions again to LLMs after 8 months. We assessed medical accuracy, completeness, personalization, readability and reproducibility for LLMs. We used statistical tests to compare results within LLMs, within PIRs, and between the best-performing LLMs and the gold-standard PIR (leaflet from dermatologists). We found that within the LLMs, ChatGPT-3.5 had the highest accuracy. Gemini performed best in terms of completeness, personalization and readability. The PIRs consistently showed high accuracy and completeness, with the general practitioner’s website excelling in personalization and readability. The top-performing LLMs outperformed the best PIR on completeness and personalization, but was less accurate and less readable. Over time, the reproducibility of LLM responses decreased, showing varied outcomes. In conclusion, although LLMs have the potential to provide highly personalized and complete answers to patient questions about melanoma, their accuracy, reproducibility and accessibility need to be improved before they can be a reliable source of information or complement existing PIRs.

来源:Br J Dermatol,查看原文。版权归原作者所有。

引用

  1. Kamminga NCW, Kievits JEC, Plaisier PW, et al. Do large language model chatbots perform better than established patient information resources in answering patient questions? A comparative study on melanoma. Br J Dermatol. 2025. PMID: 39365602. doi:10.1093/bjd/ljae377

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。