ChatGPT能否通过英国放射科医师学会院士考试
研究让ChatGPT回答英国放射科医师学会(FRCR)风格考题:GPT-4在物理第1部分203道五联判断题中答对74.8%,春季2023年及格线75.5%,以微弱差距未通过;GPT-3.5在240道最佳单选题中答对50.8%,GPT-4答对74.2%,冬季2022年2A及格线63.3%,GPT-4明显通过。作者认为AI模型能答对多数FRCR风格考题,进一步发展的AI更可能理解并解答医学尤其是临床放射学问题。
要点
- GPT-4在物理第1部分判断题正确率74.8%,低于春季2023年及格线75.5%。
- GPT-3.5在2A最佳单选题正确率50.8%,GPT-4为74.2%。
- 冬季2022年2A及格线63.3%,GPT-4正确率高于该线。
核心结论
- ChatGPT等AI模型能够答对FRCR风格考试中的大部分题目,GPT-4在2A部分达到及格线以上,在物理第1部分以微弱差距未达及格线。
研究设计与样本
- 研究为考试题目测试,GPT-4回答物理第1部分全部203道五联判断题,GPT-3.5和GPT-4回答240道最佳单选题,题目来自类似FRCR考试的题库。
诊断标准与关键数值
- 物理第1部分:GPT-4正确率74.8%,春季2023年及格线75.5%。
- 2A部分:GPT-3.5正确率50.8%,GPT-4正确率74.2%,冬季2022年及格线63.3%。
临床应用与阅片要点
- 原文结论认为AI模型能够解答多数FRCR风格考题,进一步发展的AI更可能理解并解答医学尤其是临床放射学相关问题,并可能对放射学培训、评估和实践产生影响。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: Chat Generative Pre-trained Transformer (ChatGPT) is an artificial intelligence (AI) tool which utilises machine learning to generate original text resembling human language. AI models have recently demonstrated remarkable ability at analysing and solving problems, including passing professional examinations. We investigate the performance of ChatGPT on some of the UK radiology fellowship equivalent examination questions.
METHODS: ChatGPT was asked to answer questions from question banks resembling the Fellowship of the Royal College of Radiologists (FRCR) examination. The entire physics part 1 question bank (203 5-part true/false questions) was answered by the GPT-4 model and answers recorded. 240 single best answer questions (SBAs) (representing the true length of the FRCR 2A examination) were answered by both GPT-3.5 and GPT-4 models.
RESULTS: ChatGPT 4 answered 74.8% of part 1 true/false statements correctly. The spring 2023 passing mark of the part 1 examination was 75.5% and ChatGPT thus narrowly failed. In the 2A examination, ChatGPT 3.5 answered 50.8% SBAs correctly, while GPT-4 answered 74.2% correctly. The winter 2022 2A pass mark was 63.3% and thus GPT-4 clearly passed.
CONCLUSION: AI models such as ChatGPT are able to answer the majority of questions in an FRCR style examination. It is reasonable to assume that further developments in AI will be more likely to succeed in comprehending and solving questions related to medicine, specifically clinical radiology.
ADVANCES IN KNOWLEDGE: Our findings outline the unprecedented capabilities of AI, adding to the current relatively small body of literature on the subject, which in turn can play a role medical training, evaluation and practice. This can undoubtedly have implications for radiology.
引用
- Ariyaratne S, Jenko N, Mark Davies A, et al. Could ChatGPT Pass the UK Radiology Fellowship Examinations? Acad Radiol. 2023. PMID: 38160089. doi:10.1016/j.acra.2023.11.026
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。