现代马基雅维利?基于9000条评论分类的ChatGPT生成整形美容患者评论的错觉
本研究比较了真实患者评论与ChatGPT生成的评论,针对美国前五大整形手术。30名Prolific用户评估了150对评论以区分人类与AI生成内容,并用Copyleaks AI检测软件分析。结果显示分类准确率仅59.6%,Copyleaks AI对ChatGPT生成评论的正确分类率为69.3%,表明ChatGPT能逼真模仿真实评论,甚至欺骗商业AI检测软件。
要点
- 在9000次分类任务中,64.3%评论被归类为真实,35.7%为虚假。
- 人类与机器作者的平均正确识别率为59.6%,各手术间表现一致。
- Copyleaks AI对人工撰写和ChatGPT生成评论的正确分类率分别为96.7%和69.3%。
核心结论
- ChatGPT能令人信服地复制真实患者评论,甚至欺骗商业AI检测软件。
- 分析情感基调与评论长度有助于区分真实与虚假评论,强调需教育患者和医生以防止错误信息和不信任。
研究设计与样本
- 从RealSelf收集鼻整形、眼睑成形术、面部提升术、脂肪抽吸术和乳房增大术各30条真实患者评论作为模板,由ChatGPT生成匹配评论。
- 30名Prolific用户评估150对评论以识别人类撰写与人工智能生成评论,并用Copyleaks AI检测软件进一步评估。
主要发现
- 在9000次分类任务中,64.3%评论被归类为真实,35.7%为虚假。
- 平均正确识别作者(人类 vs 机器)的比例为59.6%,且该较低分类表现 across all procedures 一致。
- 有既往美容治疗史的患者分类表现差于无治疗史者(p < 0.05)。
- 人类撰写评论的平均字符数显著高于AI生成评论(p < 0.001),字符数与参与者准确率显著相关(p < 0.001)。
- 评论情感基调差异显著:人类撰写评论中「快乐」更普遍(p < 0.001),AI评论中「失望」更普遍(p = 0.005)。
临床怎么用
- 作者认为分析情感基调和评论长度有助于区分真实与虚假评论。
- 作者强调需教育患者和医生以防止错误信息和不信任。
局限性
- 原文未报告局限性
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Online patient reviews are crucial in guiding individuals who seek plastic surgery, but artificial chatbots pose a threat of disseminating fake reviews. This study aimed to compare real patient feedback with ChatGPT-generated reviews for the top five US plastic surgery procedures. METHODS: Thirty real patient reviews on rhinoplasty, blepharoplasty, facelift, liposuction, and breast augmentation were collected from RealSelf and used as templates for ChatGPT to generate matching patient reviews. Prolific users (n = 30) assessed 150 pairs of reviews to identify human-written and artificial intelligence (AI)-generated reviews. Patient reviews were further assessed using AI content detector software (Copyleaks AI). RESULTS: Among the 9000 classification tasks, 64.3% and 35.7% of reviews were classified as authentic and fake, respectively. On an average, the author (human versus machine) was correctly identified in 59.6% of cases, and this poor classification performance was consistent across all procedures. Patients with prior aesthetic treatment showed poorer classification performance than those without (p < 0.05). The mean character count in human-written reviews was significantly higher (p < 0.001) that that in AI-generated reviews, with a significant correlation between character count and participants' accuracy rate (p < 0.001). Emotional timbre of reviews differed significantly with "happiness" being more prevalent in human-written reviews (p < 0.001), and "disappointment" being more prevalent in AI reviews (p = 0.005). Copyleaks AI correctly classified 96.7% and 69.3% of human-written and ChatGPT-generated reviews, respectively. CONCLUSION: ChatGPT convincingly replicates authentic patient reviews, even deceiving commercial AI detection software. Analyzing emotional tone and review length can help differentiate real from fake reviews, underscoring the need to educate both patients and physicians to prevent misinformation and mistrust.
引用
- Knoedler S, Sofo G, Kern B, et al. Modern Machiavelli? The illusion of ChatGPT-generated patient reviews in plastic and aesthetic surgery based on 9000 review classifications. J Plast Reconstr Aesthet Surg. 2023. PMID: 37972444. doi:10.1016/j.bjps.2023.10.119
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。