评估教育中的人工智能能力:ChatGPT-4在美国放射影像技师注册委员会(ARRT)放射摄影认证考试中的表现
本研究用BoardVitals的200道ARRT认证考试练习题评估ChatGPT-4,每题重复15次共3000次观察。ChatGPT-4总体正确率为80.56%,文本题正确率86.3%高于图像题45.6%,图像题响应时间18.01秒长于文本题13.27秒;各领域表现从安全72.6%到操作53.4%不等,容易题正确率78.5%。作者认为ChatGPT-4在题库上表现有效,但图像处理与解读能力仍需进一步发展。
要点
- ChatGPT-4在ARRT练习题上总体正确率为80.56%。
- 图像题正确率45.6%,明显低于文本题86.3%。
- 图像题响应时间18.01秒,长于文本题13.27秒。
核心结论
- ChatGPT-4在ARRT认证考试的BoardVitals题库上表现出有效性能,未来研究可分析BoardVitals分数与实际考试结果的相关性;人工智能在图像处理与解读方面需进一步发展以提升教育实用性。
研究设计与样本
- 研究使用BoardVitals的200道ARRT认证考试练习选择题,每题输入ChatGPT-4十五次,共3000次观察以考虑响应变异性。
诊断标准与关键数值
- ChatGPT-4总体正确率80.56%,文本题正确率86.3%,图像题正确率45.6%;各领域正确率:安全72.6%、图像生产70.6%、患者护理67.3%、操作53.4%;容易题正确率78.5%。
局限性
- 未来研究可分析BoardVitals分数与实际考试结果的相关性。
- 人工智能在图像处理与解读方面需进一步发展以提升教育实用性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: The American Registry of Radiologic Technologists (ARRT) leads the certification process with an exam comprising 200 multiple-choice questions. This study aims to evaluate ChatGPT-4's performance in responding to practice questions similar to those found in the ARRT board examination.
MATERIALS AND METHODS: We used a dataset of 200 practice multiple-choice questions for the ARRT certification exam from BoardVitals. Each question was fed to ChatGPT-4 fifteen times, resulting in 3000 observations to account for response variability.
RESULTS: ChatGPT's overall performance was 80.56%, with higher accuracy on text-based questions (86.3%) compared to image-based questions (45.6%). Response times were longer for image-based questions (18.01 s) than for text-based questions (13.27 s). Performance varied by domain: 72.6% for Safety, 70.6% for Image Production, 67.3% for Patient Care, and 53.4% for Procedures. As anticipated, performance was best on on easy questions (78.5%).
CONCLUSION: ChatGPT demonstrated effective performance on the BoardVitals question bank for ARRT certification. Future studies could benefit from analyzing the correlation between BoardVitals scores and actual exam outcomes. Further development in AI, particularly in image processing and interpretation, is necessary to enhance its utility in educational settings.
引用
- Al-Naser Y, Halka F, Ng B, et al. Evaluating Artificial Intelligence Competency in Education: Performance of ChatGPT-4 in the American Registry of Radiologic Technologists (ARRT) Radiography Certification Exam. Acad Radiol. 2024. PMID: 39153961. doi:10.1016/j.acra.2024.08.009
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。