文献速递医学人工智能与信息学流行病学与生物统计医学教育与科研方法

使用人工智能半自动化评估随机对照试验可信度:一项案例研究

Using artificial intelligence to semi-automate trustworthiness assessment of randomized controlled trials: a case study.
PubMed 文献 · J Clin Epidemiol · 2025-01-17

该案例研究使用ChatGPT(GPT-4o)对一篇随机对照试验论文进行可信度评估,依据TRACT清单自动提取数据表。ChatGPT与人工评估者在16/19项TRACT条目上一致(84%),四分之三的提取表格准确率达100%,但需用户反复提示。

要点

  • ChatGPT与人工评估者在16/19项TRACT条目上一致(84%)
  • 四分之三的提取表格准确率达100%
  • 需用户反复提示,需进一步测试与改进

核心结论

  • ChatGPT在随机对照试验可信度半自动化评估中展现出潜力,但在我们的经验中这需要用户反复提示。

研究设计与样本

  • 案例研究:使用OpenAI GPT-4o驱动的ChatGPT评估一篇随机对照试验论文,应用TRACT清单并自动化数据表提取。

主要发现

  • ChatGPT与人工评估者在16/19项TRACT条目上一致(84%)。
  • ChatGPT提取的四分之三的表格准确率达100%。

关键数值与推荐

  • TRACT清单共19项,ChatGPT与人工评估者一致16项(84%)。

局限性

  • 在我们的经验中,这需要用户反复提示。
  • 需进一步测试与改进,将ChatGPT应用于随机对照试验论文集合以提高数据捕获准确性并减少用户角色。

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

原文

BACKGROUND AND OBJECTIVE: Randomized controlled trials (RCTs) are the cornerstone of evidence-based medicine. Unfortunately, not all RCTs are based on real data. This serious breach of research integrity compromises the reliability of systematic reviews and meta-analyses, leading to misinformed clinical guidelines and posing a risk to both individual and public health. While methods to detect problematic RCTs have been proposed, they are time-consuming and labor-intensive. The use of artificial intelligence large language models (LLMs) has the potential to accelerate the data collection needed to assess the trustworthiness of published RCTs. METHODS: We present a case study using ChatGPT powered by OpenAI's GPT-4o to assess an RCT paper. The case study focuses on applying the trustworthiness in randomised controlled trials (TRACT checklist) and automating data table extraction to accelerate statistical analysis targeting the trustworthiness of the data. We provide a detailed step-by-step outline of the process, along with considerations for potential improvements. RESULTS: ChatGPT completed all tasks by processing the PDF of the selected publication and responding to specific prompts. ChatGPT addressed items in the TRACT checklist effectively, demonstrating an ability to provide precise "yes" or "no" answers while quickly synthesizing information from both the paper and relevant online resources. A comparison of results generated by ChatGPT and the human assessor showed an 84% level of agreement of (16/19) TRACT items. This substantially accelerated the qualitative assessment process. Additionally, ChatGPT was able to extract efficiently the data tables as Microsoft Excel worksheets and reorganize the data, with three out of four extracted tables achieving an accuracy score of 100%, facilitating subsequent analysis and data verification. CONCLUSION: ChatGPT demonstrates potential in semiautomating the trustworthiness assessment of RCTs, though in our experience this required repeated prompting from the user. Further testing and refinement will involve applying ChatGPT to collections of RCT papers to improve the accuracy of data capture and lessen the role of the user. The ultimate aim is a completely automated process for large volumes of papers that seems plausible given our initial experience.

来源:J Clin Epidemiol,查看原文。版权归原作者所有。

引用

  1. Au LS, Qu L, Nielsen J, et al. Using artificial intelligence to semi-automate trustworthiness assessment of randomized controlled trials: a case study. J Clin Epidemiol. 2025. PMID: 39828224. doi:10.1016/j.jclinepi.2025.111672

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。