文献速递前列腺癌

大型语言模型对公众前列腺癌信息的准确性、可读性与可理解性

Accuracy, readability, and understandability of large language models for prostate cancer information to the public.
PubMed 文献 · Prostate Cancer Prostatic Dis · 2024-05-14

研究通过Google Trends筛选9个前列腺癌相关问题,用ChatGPT 3.5生成回答并二次生成六年级水平简化摘要。36名泌尿科医生和17名住院医师对原始回答评分,2名独立评审评价简化摘要,514名公众评估可理解性。GPT输出被71.7%至94.3%评分者认为正确,简化摘要可读性显著提高,但该技术并非为患者信息传递而设计。

要点

  • GPT生成的9个前列腺癌场景回答被71.7%至94.3%的评分者认为正确。
  • 简化摘要可读性指标均显著优于原始GPT输出(P<0.0001至P=0.0002)。
  • 514名公众对简化摘要正确性评分89.5%至95.7%,内容理解正确率63.0%至87.4%。

核心结论

  • GPT在提供正确的前列腺癌患者教育内容方面显示出前景,但该技术并非为向患者传递信息而设计。
  • 提示模型以准确性、完整性、清晰性和可读性作答,可能提升其作为GPT驱动的医疗聊天机器人的效用。

研究设计与样本

  • 通过Google Trends(全球)识别9个前列腺癌相关问题,分类为诊断、治疗和术后随访。
  • 使用ChatGPT 3.5处理问题并记录回答,再将回答重新输入ChatGPT生成六年级水平的简化摘要。
  • 采用经验证的可读性工具评估原始回答和简化摘要的可读性。
  • 对36名泌尿科医生和17名泌尿科住院医师进行问卷调查,用5点Likert量表对原始回答的准确性、完整性和清晰性评分。
  • 由2名独立评审按正确性三要素(准确性、完整性、决策充分性)评价简化摘要,并通过Amazon Mechanical Turk(MTurk)由公众评估简化摘要的清晰性和可理解性。

主要发现

  • GPT生成的输出在9个场景中被71.7%至94.3%的评分者(36名泌尿科医生和17名泌尿科住院医师)认为正确。
  • GPT生成的简化摘要中,8/9(88.9%)场景被评价为准确,8/9(88.9%)场景被认为足以让患者做出决策。
  • 简化摘要平均可读性高于原始GPT输出:Flesch Reading Ease为36.5(9.1)对70.2(11.2),P<0.0001;Gunning Fog为15.8(1.7)对9.5(2.0),P<0.0001;Flesch Grade Level为12.8(1.2)对7.4(1.7),P<0.0001;Coleman Liau为13.7(2.1)对8.6(2.4),P=0.0002;Smog指数为11.8(1.2)对6.7(1.8),P<0.0001;Automated Readability Index为13.1(1.4)对7.5(2.1),P<0.0001。
  • MTurk工作人员(n=514)将简化摘要评为正确(89.5%至95.7%),并正确理解内容(63.0%至87.4%)。

临床怎么用

  • 原文结论指出GPT在提供正确的前列腺癌患者教育内容方面显示出前景,但该技术并非为向患者传递信息而设计。
  • 作者认为提示模型以准确性、完整性、清晰性和可读性作答,可能提升其作为GPT驱动的医疗聊天机器人的效用。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND: Generative Pretrained Model (GPT) chatbots have gained popularity since the public release of ChatGPT. Studies have evaluated the ability of different GPT models to provide information about medical conditions. To date, no study has assessed the quality of ChatGPT outputs to prostate cancer related questions from both the physician and public perspective while optimizing outputs for patient consumption. METHODS: Nine prostate cancer-related questions, identified through Google Trends (Global), were categorized into diagnosis, treatment, and postoperative follow-up. These questions were processed using ChatGPT 3.5, and the responses were recorded. Subsequently, these responses were re-inputted into ChatGPT to create simplified summaries understandable at a sixth-grade level. Readability of both the original ChatGPT responses and the layperson summaries was evaluated using validated readability tools. A survey was conducted among urology providers (urologists and urologists in training) to rate the original ChatGPT responses for accuracy, completeness, and clarity using a 5-point Likert scale. Furthermore, two independent reviewers evaluated the layperson summaries on correctness trifecta: accuracy, completeness, and decision-making sufficiency. Public assessment of the simplified summaries' clarity and understandability was carried out through Amazon Mechanical Turk (MTurk). Participants rated the clarity and demonstrated their understanding through a multiple-choice question. RESULTS: GPT-generated output was deemed correct by 71.7% to 94.3% of raters (36 urologists, 17 urology residents) across 9 scenarios. GPT-generated simplified layperson summaries of this output was rated as accurate in 8 of 9 (88.9%) scenarios and sufficient for a patient to make a decision in 8 of 9 (88.9%) scenarios. Mean readability of layperson summaries was higher than original GPT outputs ([original ChatGPT v. simplified ChatGPT, mean (SD), p-value] Flesch Reading Ease: 36.5(9.1) v. 70.2(11.2), <0.0001; Gunning Fog: 15.8(1.7) v. 9.5(2.0), p < 0.0001; Flesch Grade Level: 12.8(1.2) v. 7.4(1.7), p < 0.0001; Coleman Liau: 13.7(2.1) v. 8.6(2.4), 0.0002; Smog index: 11.8(1.2) v. 6.7(1.8), <0.0001; Automated Readability Index: 13.1(1.4) v. 7.5(2.1), p < 0.0001). MTurk workers (n = 514) rated the layperson summaries as correct (89.5-95.7%) and correctly understood the content (63.0-87.4%). CONCLUSION: GPT shows promise for correct patient education for prostate cancer-related contents, but the technology is not designed for delivering patients information. Prompting the model to respond with accuracy, completeness, clarity and readability may enhance its utility when used for GPT-powered medical chatbots.

来源:Prostate Cancer Prostatic Dis,查看原文。版权归原作者所有。

引用

  1. Hershenhouse JS, Mokhtar D, Eppler MB, et al. Accuracy, readability, and understandability of large language models for prostate cancer information to the public. Prostate Cancer Prostatic Dis. 2024. PMID: 38744934. doi:10.1038/s41391-024-00826-y

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。