大型语言模型生成结直肠患者教育材料:提示词工程是提高可读性的关键
本研究比较现有结直肠手术教育材料与大型语言模型生成材料的可读性。从大型学术机构获取52份基线材料,使用三种提示词输入Perplexity和ChatGPT 3.5生成材料,计算可读性评分。结果显示ChatGPT在基于指标的提示词下生成的材料可读性优于基线材料,平均达到六年级水平,但多数大型语言模型生成材料仍未达推荐标准,提示词设计至关重要。
要点
- 现有结直肠教育材料未达六年级可读性指南,基线Grade Level 7.0-9.8。
- ChatGPT使用基于指标的提示词时材料可读性最佳,平均Grade Level 5.2。
- Perplexity生成材料除基于指标的提示词外,可读性显著差于基线。
核心结论
- 现有教育材料和多数大型语言模型生成材料未达可读性推荐标准。
- ChatGPT在基于指标的提示词下生成的材料可读性优于基线并达到平均Grade Level推荐。
- 大型语言模型性能差异凸显提示词设计的重要性。
研究设计与样本
- 研究比较现有结直肠手术教育材料与大型语言模型生成材料的可读性。
- 基线材料来自大型学术机构,共52份。
- 使用三种提示词(基础、迭代、基于指标)输入Perplexity和ChatGPT 3.5生成材料。
- 计算Flesch-Kincaid Grade Level、Reading Ease和Modified Grade Level评分,采用非配对t检验比较均值。
主要发现
- 现有材料字数863-956,ChatGPT生成材料170-265字,Perplexity生成材料220-313字,所有P < .01。
- 基线材料Grade Level 7.0-9.8,Modified Grade Level 9.6-11.5,Ease 53.1-65.0,未达六年级标准。
- ChatGPT基于指标提示词生成材料Grade Level 5.2 vs 基线8.1,Modified Grade Level 7.3 vs 10.3,Ease 70.5 vs 60.4,所有P < .01。
- Perplexity生成材料除基于指标提示词外,可读性显著差于基线。
关键数值与推荐
- 六年级阅读水平推荐基于Grade Level评分。
- ChatGPT基于指标提示词生成材料平均Grade Level 5.2,达到推荐。
- 基线材料Grade Level 7.0-9.8,未达推荐。
临床怎么用
- 使用大型语言模型生成患者教育材料时,应采用基于指标的提示词(如指定六年级阅读水平、短句、短词)以提高可读性。
- ChatGPT在基于指标的提示词下生成的材料可读性优于现有材料,可考虑用于辅助患者教育。
- 需注意不同平台和提示词的效果差异,生成后应评估可读性。
局限性
- 研究仅纳入来自单一大型学术机构的52份基线材料。
- 仅测试了Perplexity和ChatGPT 3.5两个平台。
- 未评估材料的内容准确性和临床效果。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Improving patient education has been shown to improve clinical outcomes and reduce disparities, though such efforts can be labor intensive. Large language models may serve as an accessible method to improve patient educational material. The aim of this study was to compare readability between existing educational materials and those generated by large language models. METHODS: Baseline colorectal surgery educational materials were gathered from a large academic institution (n = 52). Three prompts were entered into Perplexity and ChatGPT 3.5 for each topic: a Basic prompt that simply requested patient educational information the topic, an Iterative prompt that repeated instruction asking for the information to be more health literate, and a Metric-based prompt that requested a sixth-grade reading level, short sentences, and short words. Flesch-Kincaid Grade Level or Grade Level, Flesch-Kincaid Reading Ease or Ease, and Modified Grade Level scores were calculated for all materials, and unpaired t tests were used to compare mean scores between baseline and documents generated by artificial intelligence platforms. RESULTS: Overall existing materials were longer than materials generated by the large language models across categories and prompts: 863-956 words vs 170-265 (ChatGPT) and 220-313 (Perplexity), all P < .01. Baseline materials did not meet sixth-grade readability guidelines based on grade level (Grade Level 7.0-9.8 and Modified Grade Level 9.6-11.5) or ease of readability (Ease 53.1-65.0). Readability of materials generated by a large language model varied by prompt and platform. Overall, ChatGPT materials were more readable than baseline materials with the Metric-based prompt: Grade Level 5.2 vs 8.1, Modified Grade Level 7.3 vs 10.3, and Ease 70.5 vs 60.4, all P < .01. In contrast, Perplexity-generated materials were significantly less readable except for those generated with the Metric-based prompt, which did not statistically differ. CONCLUSION: Both existing materials and the majority of educational materials created by large language models did not meet readability recommendations. The exception to this was with ChatGPT materials generated with a Metric-based prompt that consistently improved readability scores from baseline and met recommendations in terms of the average Grade Level score. The variability in performance highlights the importance of the prompt used with large language models.
引用
- Ellison IE, Oslock WM, Abdullah A, et al. De novo generation of colorectal patient educational materials using large language models: Prompt engineering key to improved readability. Surgery. 2025. PMID: 39756334. doi:10.1016/j.surg.2024.109024
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。