文献速递儿童口腔

聊天机器人在儿童口腔医学系统综述中的应用

Chatbots for conducting systematic reviews in pediatric dentistry.
PubMed 文献 · J Dent · 2025-04-05

本研究评估聊天机器人在儿童口腔医学人工智能系统综述各步骤中的表现。两个聊天机器人(ChatGPT4/Gemini)与两名非专家评审员对比两名专家,完成PICO构建、检索、筛选、数据提取和偏倚风险评估五项任务。结论认为聊天机器人可提升筛选和数据提取效率,但人工监督对准确性和完整性仍至关重要。

要点

  • 聊天机器人在筛选任务的灵敏度达90%,但精确度低于25%,F1分数低于40%。
  • 数据提取质量评分ChatGPT为31.6±12.3,Gemini为29.2±12.3,非专家为30.4±11.3(满分45)。
  • 偏倚风险评估与专家一致率ChatGPT为49.4%,Gemini为51.2%,非专家为48.8%,组间p>0.05。

核心结论

  • 聊天机器人可提升系统综述效率,尤其在研究筛选和数据提取步骤,但人工监督对确保准确性和完整性仍至关重要。

研究设计与样本

  • 在儿童口腔医学人工智能系统综述中,将两个聊天机器人(ChatGPT4/Gemini)和两名非专家评审员与两名专家进行对比。
  • 评估五项任务:构建PICO问题、为八个数据库制定检索查询、筛选研究、提取数据和评估偏倚风险。
  • 聊天机器人和非专家接受相同提示,专家提供参考标准;采用准确率、精确度、灵敏度、特异度、F1分数、Cohen's Kappa和改良全球质量评分(1-5分)进行测量。

主要发现

  • PICO构建中ChatGPT略优于Gemini,非专家得分最低。
  • 专家识别1261条记录,ChatGPT识别569条,Gemini识别285条,非专家识别722条。
  • 筛选中聊天机器人灵敏度90%、特异度>60%、精确度<25%、F1分数<40%;非专家灵敏度84%、特异度91%、F1分数39%。
  • 数据提取评分(均值±标准差,满分45):ChatGPT 31.6±12.3,Gemini 29.2±12.3,非专家30.4±11.3。
  • 偏倚风险评估与专家一致率:ChatGPT 49.4%,Gemini 51.2%,非专家48.8%(p>0.05)。

临床怎么用

  • 聊天机器人可简化筛选和数据提取等系统综述任务,可能加速证据合成,但可靠性仍需人工监督。
  • 聊天机器人对系统综述步骤的适用性取决于具体步骤,评审员在使用时需做出知情选择。

局限性

  • 原文未报告局限性。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

OBJECTIVES: The performance of chatbots for discrete steps of a systematic review (SR) on artificial intelligence (AI) in pediatric dentistry was evaluated. METHODS: Two chatbots (ChatGPT4/Gemini) and two non-expert reviewers were compared against two experts in a SR on AI in pediatric dentistry. Five tasks: (1) formulating a PICO question, (2) developing search queries for eight databases, (3) screening studies, (4) extracting data, and (5) assessing the risk of bias (RoB) were assessed. Chatbots and non-experts received identical prompts, with experts providing the reference standard. Performance was measured using accuracy, precision, sensitivity, specificity, and F1-score for search and screening tasks, Cohen's Kappa for risk of bias assessment, and a modified Global Quality Score (1-5) for PICO question formulation and data extraction quality. Statistical comparisons were performed using Kruskal-Wallis and Dunn's post-hoc tests. RESULTS: In PICO formulation, ChatGPT outperformed Gemini slightly, while non-experts scored the lowest. Experts identified 1261 records, compared to 569 (ChatGPT), 285 (Gemini), and 722 (non-experts). Screening showed chatbots having 90 % sensitivity, >60 % specificity, <25 % precision, and F1-scores <40 %, versus non-experts' 84 % sensitivity, 91 % specificity, and 39 % F1-score, respectively. For data extraction, ChatGPT yielded a (mean±standard deviation) score of 31.6 ± 12.3 (max. was 45), Gemini 29.2 ± 12.3, and non-experts 30.4 ± 11.3, respectively. For RoB, the agreement with experts was 49.4 % for ChatGPT, 51.2 % for Gemini 48.8 % for non-experts (p > 0.05). CONCLUSION: Chatbots could enhance SR efficiency, particularly for the study screening and data extraction steps. Human oversight remains critical for ensuring accuracy and completeness. CLINICAL SIGNIFICANCE: Chatbots can streamline SR tasks like screening and data extraction, potentially accelerating evidence synthesis, though human oversight remains needed for reliability. The applicability of chatbots for SR steps was found dependent on the specific step, indicating reviewers need to make informed choices when employing chatbots for this purpose.

来源:J Dent,查看原文。版权归原作者所有。

引用

  1. Rokhshad R, Mohammad FD, Nomani M, et al. Chatbots for conducting systematic reviews in pediatric dentistry. J Dent. 2025. PMID: 40194755. doi:10.1016/j.jdent.2025.105733

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。