探索ChatGPT等大语言模型在放射肿瘤学中的能力
本研究通过多项选择题和开放式问题评估ChatGPT在放射肿瘤学临床、物理和生物学知识方面的回答能力。70道多选题中66道(94.3%)给出有效回答,其中60.61%正确;25道开放式问题中12道被6位放射肿瘤学家评为可接受及以上。结论认为大语言模型能提供令人满意的答案,但仍无法一致地提供正确信息,用于获取医学信息存在问题。
要点
- ChatGPT对70道放射肿瘤学多选题中66道(94.3%)给出有效回答,有效回答中60.61%正确。
- 临床问题正确率50.0%,物理问题78.6%,生物学问题58.3%。
- 25道开放式问题中12道被所有6位评估者评为可接受、好或很好。
核心结论
- 现代对话式大语言模型如ChatGPT能为放射治疗中许多相关问题提供令人满意的答案。
- 这些模型仍无法一致地提供正确信息,将其用于获取医学信息存在问题。
- 随着大语言模型未来进一步改进,预计其不仅对一般社会、也对包括放射肿瘤学在内的临床实践产生日益增加的影响。
研究设计与样本
- 探索性研究,创建一组关于放射肿瘤学临床、物理和生物学一般知识的多项选择题以及一组开放式问题,作为提示输入ChatGPT并收集分析答案。
- 多项选择题检查模型答案能否明确归入允许的选项之一,并确定正确答案比例。
- 开放式问题由独立盲法放射肿瘤学家就正确性和有用性采用5点Likert量表评估质量,并请评估者提供改进建议。
- 共6位放射肿瘤学家参与评估。
主要发现
- 70道多项选择题中,ChatGPT在66例(94.3%)给出有效答案。
- 有效答案中60.61%所选答案正确(临床问题50.0%,物理问题78.6%,生物学问题58.3%)。
- 25道开放式问题中,12道答案在正确性和有用性两方面均被全部6位参与评估的放射肿瘤学家评为可接受、好或很好。
- 总体而言,正确性/有用性方面评为很好的分别占29.3%和28%,好的分别占28%和29.3%,可接受的分别占19.3%和19.3%,差的分别占9.3%和9.3%,很差的分别占14%和14%。
临床怎么用
- 原文未据此给出推荐。
局限性
- 原文结论指出,这些模型仍无法一致地提供正确信息,将其用于获取医学信息存在问题。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
PURPOSE: Technological progress of machine learning and natural language processing has led to the development of large language models (LLMs), capable of producing well-formed text responses and providing natural language access to knowledge. Modern conversational LLMs such as ChatGPT have shown remarkable capabilities across a variety of fields, including medicine. These models may assess even highly specialized medical knowledge within specific disciplines, such as radiation therapy. We conducted an exploratory study to examine the capabilities of ChatGPT to answer questions in radiation therapy. METHODS AND MATERIALS: A set of multiple-choice questions about clinical, physics, and biology general knowledge in radiation oncology as well as a set of open-ended questions were created. These were given as prompts to the LLM ChatGPT, and the answers were collected and analyzed. For the multiple-choice questions, it was checked how many of the answers of the model could be clearly assigned to one of the allowed multiple-choice-answers, and the proportion of correct answers was determined. For the open-ended questions, independent blinded radiation oncologists evaluated the quality of the answers regarding correctness and usefulness on a 5-point Likert scale. Furthermore, the evaluators were asked to provide suggestions for improving the quality of the answers. RESULTS: For 70 multiple-choice questions, ChatGPT gave valid answers in 66 cases (94.3%). In 60.61% of the valid answers, the selected answer was correct (50.0% of clinical questions, 78.6% of physics questions, and 58.3% of biology questions). For 25 open-ended questions, 12 answers of ChatGPT were considered as "acceptable," "good," or "very good" regarding both correctness and helpfulness by all 6 participating radiation oncologists. Overall, the answers were considered "very good" in 29.3% and 28%, "good" in 28% and 29.3%, "acceptable" in 19.3% and 19.3%, "bad" in 9.3% and 9.3%, and "very bad" in 14% and 14% regarding correctness/helpfulness. CONCLUSIONS: Modern conversational LLMs such as ChatGPT can provide satisfying answers to many relevant questions in radiation therapy. As they still fall short of consistently providing correct information, it is problematic to use them for obtaining medical information. As LLMs will further improve in the future, they are expected to have an increasing impact not only on general society, but also on clinical practice, including radiation oncology.
引用
- Dennstädt F, Hastings J, Putora PM, et al. Exploring Capabilities of Large Language Models such as ChatGPT in Radiation Oncology. Adv Radiat Oncol. 2023. PMID: 38304112. doi:10.1016/j.adro.2023.101400
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。