机器中的幽灵还是打字机前的猴子:使用人工智能为《BMJ》圣诞研究文章生成标题的观察性研究
该观察性研究由GPT-3与25名人类参与,比较AI生成的《BMJ》圣诞研究文章标题与真实历史标题在可信度、吸引力、趣味性和教育价值上的差异。AI标题在趣味性和吸引力上不劣于真实标题,但可信度和科学教育价值较低;经人工筛选后,科学教育价值差异变为不显著。
要点
- AI生成的标题在趣味性和吸引力上不劣于真实标题,但可信度较低。
- AI标题的科学或教育价值总体低于真实标题,但经人工筛选后差异不显著。
- 最可信的AI标题涉及阴谋论信念与疫苗接种意愿的关联。
核心结论
- AI能为《BMJ》圣诞研究文章生成可信、有趣且具科学意义的标题;与其他医学领域一样,人工干预可提升其表现。
研究设计与样本
- 观察性研究,比较1项AI技术(生成式预训练变换器3,GPT-3)与25名人类生成的标题。
- 研究地点涉及欧洲、澳大利亚和非洲;主要结局为AI生成标题与历史对照标题在可信度、吸引力、趣味性和教育价值方面的评分。
主要发现
- AI生成标题的趣味性评分至少与真实对照标题相当(159/250回应(64%) vs 346/500回应(69%);比值比0.9,95%置信区间0.7至1.2)。
- AI生成标题的吸引力评分与真实对照标题相当(176/250(70%) vs 342/500(68%);1.1,0.8至1.4)。
- 真实对照标题的可信度评分高于AI生成标题(182/250(73%) vs 238/500(48%);3.1,2.3至4.1)。
- AI生成标题总体的科学或教育价值评分低于真实对照(146/250(58%) vs 193/500(39%);2.0,1.5至2.6);但经人工筛选AI输出后,该差异变为不显著(146/250(58%) vs 123/250(49%);1.3,1.0至1.8)。
- 最可信的AI生成标题为「相信阴谋论与接种疫苗意愿之间的关联」,评分最高的标题为「免费精品咖啡对急诊科等待时间的影响:一项观察性研究」。
局限性
- 原文未报告局限性
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
OBJECTIVE: To determine whether artificial intelligence (AI) can generate plausible and engaging titles for potential Christmas research articles in The BMJ. DESIGN: Observational study. SETTING: Europe, Australia, and Africa. PARTICIPANTS: 1 AI technology (Generative Pre-trained Transformer 3, GPT-3) and 25 humans. MAIN OUTCOME MEASURES: Plausibility, attractiveness, enjoyability, and educational value of titles for potential Christmas research articles in The BMJ generated by GPT-3 compared with historical controls. RESULTS: AI generated titles were rated at least as enjoyable (159/250 responses (64%) v 346/500 responses (69%); odds ratio 0.9, 95% confidence interval 0.7 to 1.2) and attractive (176/250 (70%) v 342/500 (68%); 1.1, 0.8 to 1.4) as real control titles, although the real titles were rated as more plausible (182/250 (73%) v 238/500 (48%); 3.1, 2.3 to 4.1). The AI generated titles overall were rated as having less scientific or educational merit than the real controls (146/250 (58%) v 193/500 (39%); 2.0, 1.5 to 2.6); this difference, however, became non-significant when humans curated the AI output (146/250 (58%) v 123/250 (49%); 1.3, 1.0 to 1.8). Of the AI generated titles, the most plausible was "The association between belief in conspiracy theories and the willingness to receive vaccinations," and the highest rated was "The effects of free gourmet coffee on emergency department waiting times: an observational study." CONCLUSIONS: AI can generate plausible, entertaining, and scientifically interesting titles for potential Christmas research articles in The BMJ; as in other areas of medicine, performance was enhanced by human intervention.
引用
- Marlow R, Wood D. Ghost in the machine or monkey with a typewriter-generating titles for Christmas research articles in The BMJ using artificial intelligence: observational study. BMJ. 2021. PMID: 34911737. doi:10.1136/bmj-2021-067732
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。