利用大语言模型评估医学研究摘要
本研究探索用ChatGPT-3.5-turbo模型对美国疝学会和美国整形外科医师学会年会摘要进行评分与排名,并与人工评审比较。结果两个模型评分与人工评分的Spearman相关系数分别为0.143和0.137,相关性很低,提示当前模型尚不能替代人工摘要评审。
要点
- ChatGPT-3.5-turbo评分与人工评分相关性很低,Spearman系数仅0.143和0.137。
- 疝学会摘要人工均分2.820、ChatGPT均分2.749(5分制),平均差0.072。
- 整形外科摘要人工均分17.131、ChatGPT均分17.032(20分制),平均差0.104。
核心结论
- 尽管ChatGPT模型经过训练用于摘要评分和排名,其评分与人工评分之间相关性很低。
- 该工作为ChatGPT融入摘要评审流程提供了基准示例。
研究设计与样本
- 研究训练两个ChatGPT-3.5-turbo模型评估美国整形外科医师学会和美国疝学会年会投稿摘要,并对两个会议2023年摘要评分和排名。
- 疝学会:用2021年153篇摘要微调,对2023年167篇按5分制评分。
- 整形外科:用2021至2022年前200篇摘要微调,对2023年200篇按20分制评分。
主要发现
- 疝学会:人工评分2.820,ChatGPT评分2.749,平均差0.072,平均距离0.701,Spearman系数0.143。
- 整形外科:人工评分17.131,ChatGPT评分17.032,平均差0.104,平均距离0.718,Spearman系数0.137。
- 基于ChatGPT评分的四分位分组,疝学会平均正确排名1.146±1.042、相对正确排名2.579±0.707;整形外科分别为1.083±1.050和2.493±0.755。
- 桥接算法表现相当,疝学会Spearman系数0.032,整形外科0.023。
临床怎么用
- 目前不宜将ChatGPT-3.5-turbo单独用于会议摘要评分或排名。
- 可将此类模型作为摘要评审的辅助或基准参考,需人工评审把关。
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: The current evaluation of abstracts for national conferences faces challenges in consistency due to subjectivity and implicit biases. This study explores the feasibility of using ChatGPT-3.5-turbo models to grade and rank medical abstracts. METHODS: Two ChatGPT-3.5-turbo models were trained to evaluate abstracts submitted to the American Society of Plastic Surgery and American Hernia Society annual meetings and then asked to grade and rank each meeting's 2023 abstracts. Grades and ranks were compared between human reviewers and ChatGPT. RESULTS: American Hernia Society: ChatGPT was fine-tuned on 153 abstracts from 2021 and scored 167 from 2023 on a 5-point scale, with human scoring 2.820 and ChatGPT scoring 2.749. Average difference between scores was 0.072, average distance was 0.701, and Spearman coefficient was 0.143. American Society of Plastic Surgery: ChatGPT was fine-tuned on the top 200 abstracts from 2021 to 2022 and scored 200 from 2023 on a 20-point scale, with human scoring 17.131 and ChatGPT scoring them 17.032. Average difference was 0.104, average distance was 0.718, and Spearman coefficient was 0.137. For quartiles based on ChatGPT-generated abstract scores, average correct ranking was 1.146 ± 1.042 and relative correct ranking was 2.579 ± 0.707 for American Hernia Society and 1.083 ± 1.050 and 2.493 ± 0.755, respectively, for American Society of Plastic Surgery. A bridging algorithm showed comparable performance, with Spearman coefficient of 0.032 for American Hernia Society and 0.023 for American Society of Plastic Surgery. CONCLUSIONS: Despite ChatGPT models being trained to grade and rank abstracts, there was little correlation with human scores. This work serves as a benchmark example of ChatGPT's potential integration into the abstract review process.
引用
- Holland AM, Lorenz WR, Cavanagh JC, et al. Utilizing large language models to evaluate medical research abstracts. Surgery. 2026. PMID: 41905828. doi:10.1016/j.surg.2026.110166
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。