大型语言模型作为生成放射学委员会式多项选择题的工具
本研究评估Llama 2和GPT-4生成104道放射学委员会式多项选择题、答案及解析的能力,由两名放射科医师按五项标准以10分制评分,并与美国放射学会诊断放射学培训考试题目比较;GPT-4各项评分与ACR DXIT无显著差异,答案准确率100%,Llama 2评分较低且准确率69%,作者认为GPT-4可用于开发考题和解析。
要点
- GPT-4生成的放射学考题在清晰度、相关性、难度适宜性、干扰项质量和解析充分性上评分与ACR DXIT考题无显著差异。
- GPT-4生成的答案准确率为100%,Llama 2为69%。
- Llama 2在所有五项评分标准上均显著低于ACR DXIT考题(p < 0.001)。
核心结论
- 最先进的大型语言模型如GPT-4可用于开发放射学委员会式多项选择题和解析,以扩充备考材料和题库,并可能使放射学教育者进一步将多项选择题用作教学工具。
研究设计与样本
- 使用Llama 2和GPT-4基于美国放射学委员会考试蓝图开发104道多项选择题,由两名委员会认证放射科医师按五项标准以10分制Likert量表评分,并与既往ACR DXIT考题比较,评分者对题目来源设盲。
诊断标准与关键数值
- Llama 2各标准平均分(±标准差):清晰度8.7(±1.4)、相关性9.2(±1.3)、适宜性9.0(±1.2)、干扰项质量8.4(±1.9)、解析充分性7.2(±2.2)。
- GPT-4各标准平均分(±标准差):清晰度9.9(±0.4)、相关性9.9(±0.5)、适宜性9.9(±0.4)、干扰项质量9.8(±0.5)、解析充分性9.9(±0.3)。
- ACR DXIT考题各标准平均分(±标准差):清晰度9.9(±0.3)、相关性9.9(±0.2)、适宜性9.9(±0.2)、干扰项质量9.9(±0.4)、解析充分性9.8(±0.6)。
- Llama 2与ACR DXIT在所有标准上p < 0.001;GPT-4与ACR DXIT无统计学显著差异。
- 模型生成答案的准确率:Llama 2为69%,GPT-4为100%。
临床应用与阅片要点
- 作者认为,最先进的大型语言模型如GPT-4可用于开发放射学委员会式多项选择题和解析,以扩充备考材料和题库。
- 作者认为,可能使放射学教育者进一步将多项选择题用作教学工具。
局限性
- 原文未报告局限性。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
英文原文(摘要)· 点击展开
RATIONALE AND OBJECTIVES: To determine the potential of large language models (LLMs) to be used as tools by radiology educators to create radiology board-style multiple choice questions (MCQs), answers, and rationales.
METHODS: Two LLMs (Llama 2 and GPT-4) were used to develop 104 MCQs based on the American Board of Radiology exam blueprint. Two board-certified radiologists assessed each MCQ using a 10-point Likert scale across five criteria-clarity, relevance, suitability for a board exam based on level of difficulty, quality of distractors, and adequacy of rationale. For comparison, MCQs from prior American College of Radiology (ACR) Diagnostic Radiology In-Training (DXIT) exams were also assessed using these criteria, with radiologists blinded to the question source.
RESULTS: Mean scores (±standard deviation) for clarity, relevance, suitability, quality of distractors, and adequacy of rationale were 8.7 (±1.4), 9.2 (±1.3), 9.0 (±1.2), 8.4 (±1.9), and 7.2 (±2.2), respectively, for Llama 2; 9.9 (±0.4), 9.9 (±0.5), 9.9 (±0.4), 9.8 (±0.5), and 9.9 (±0.3), respectively, for GPT-4; and 9.9 (±0.3), 9.9 (±0.2), 9.9 (±0.2), 9.9 (±0.4), and 9.8 (±0.6), respectively, for ACR DXIT items (p < 0.001 for Llama 2 vs. ACR DXIT across all criteria; no statistically significant difference for GPT-4 vs. ACR DXIT). The accuracy of model-generated answers was 69% for Llama 2 and 100% for GPT-4.
CONCLUSION: A state-of-the art LLM such as GPT-4 may be used to develop radiology board-style MCQs and rationales to enhance exam preparation materials and expand exam banks, and may allow radiology educators to further use MCQs as teaching and learning tools.
引用
- Mistry NP, Saeed H, Rafique S, et al. Large Language Models as Tools to Generate Radiology Board-Style Multiple-Choice Questions. Acad Radiol. 2024. PMID: 39013736. doi:10.1016/j.acra.2024.06.046
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。