用机器学习模型识别美国国立卫生研究院资助的心脏骤停研究
该研究比较三种AI分类策略(传统机器学习、大语言模型零样本学习、大语言模型少样本学习)与人工分类的NIH心脏骤停(CA)资助项目摘要,数据来自NIH RePORTER(2007—2021年)。共分类1,505项资助,其中378项(25%)被识别为CA研究,留出测试集302项中有76项为CA研究。少样本学习表现最佳,准确率0.90、F1 0.82;传统机器学习准确率最低(0.87)但精确率最高(0.89),召回率差,假阴性约为两种生成式方法的2.5倍;零样本学习准确率0.88、召回率0.86,但精确率仅0.72。作者认为AI可快速识别CA资助项目,是追踪研究经费的有前景工具。
要点
- 少样本学习(含6个示例资助)表现最佳:准确率0.90,F1 0.82,精确率与召回率最均衡
- 传统机器学习(正则化逻辑回归+text-embedding-3-small嵌入向量)准确率最低(0.87)、精确率最高(0.89),但假阴性约为两种生成式方法的2.5倍
- 零样本学习(GPT-4o-mini)准确率0.88、召回率0.86,但精确率仅0.72,低于传统机器学习
- 测试集为平衡的20%留出集:302项资助中76项为CA研究,评估指标为准确率、精确率、召回率与F1
适用人群与场景
- 数据来源为NIH RePORTER 2007—2021年的心脏骤停资助项目摘要,用于与人工分类结果比较
研究设计与关键结果
- 研究共分类1,505项资助,其中378项(25%)被识别为CA研究;留出测试集302项中有76项为CA研究。
- 少样本学习(含6个示例资助)准确率0.90、F1 0.82,为三种策略中最高,精确率与召回率最均衡。
- 传统机器学习采用正则化逻辑回归模型,训练于OpenAI text-embedding-3-small模型生成的嵌入向量,准确率0.87为最低、精确率0.89为最高,但召回率差,假阴性约为两种生成式方法的2.5倍。
- 零样本学习使用GPT-4o-mini依据任务描述分类、不使用标注示例,准确率0.88、召回率0.86,精确率0.72低于传统机器学习。
- 评估在平衡的20%留出测试集上进行,指标为准确率、精确率(阳性预测值)、召回率(敏感度)与F1分数(精确率与召回率的调和均值)。
证据类型、等级与局限
- 研究类型:方法学比较研究,比较三种AI分类策略与人工分类的NIH CA资助摘要。
- 证据类型:研究论文;原文未给出证据等级或推荐强度。
- 局限:原文未报告局限性。
对床旁实践的改变
- 作者认为AI可快速识别CA资助项目且准确率优良、精确率与召回率很好,是追踪研究经费的有前景工具;该结论不改变急诊/ICU床旁诊疗做法。
提炼说明
- 本条依据论文摘要整理;摘要未涉及「诊断标准与关键阈值」「处置流程与药物剂量」「红旗征与不能漏的情况」「指南或共识出处」,故未列出这些栏目;全文请点「阅读原文」。
英文原文(摘要)· 点击展开
OBJECTIVE: To compare the performance of three artificial intelligence (AI) classification strategies against manually classified National Institutes of Health (NIH) cardiac arrest (CA) grants, with the goal of developing a publicly available tool to track CA research funding in the United States.
METHODS: Three AI strategies-traditional machine learning (ML), large language model (LLM) zero-shot learning, and LLM few-shot learning-were compared to manually categorized CA grant abstracts from NIH RePORTER (2007-2021). Traditional ML used a regularized logistic regression model trained on embedding vectors generated by OpenAI's text-embedding-3-small model. Zero-shot learning, using GPT-4o-mini, classified grants based on task descriptions without labeled examples. Few-shot learning included six example grants. Models were evaluated on a balanced 20% holdout test set using accuracy, precision (positive predictive value), recall (sensitivity), and F1 score (harmonic mean of precision and recall).
RESULTS: Out of 1,505 grants categorized, 378 (25%) were identified as CA research, yielding 302 grants in the holdout test set, 76 of which were CA research. The few-shot approach performed best, achieving the highest accuracy (0.90) and the best balance of precision and recall (F1 score 0.82). In contrast, traditional ML had the lowest accuracy (0.87) and the highest precision (0.89) but suffered from poor recall, with approximately 2.5 times more false negatives than either generative approach. The zero-shot approach outperformed traditional ML in accuracy (0.88) and recall (0.86) but had lower precision (0.72).
CONCLUSION: AI can rapidly identify CA grants with excellent accuracy and very good precision and recall, making it a promising tool for tracking research funding.
参考来源
- Coute RA, Soundararajan K, Kurz MC, et al. Use of machine learning models to identify National Institutes of Health-funded cardiac arrest research. Resuscitation. 2025. PMID: 39961491. doi:10.1016/j.resuscitation.2025.110545
供执业医师抢救与诊疗参考,不替代临床判断;剂量请结合体重、肝肾功能和本院药品规格复核,以指南原文为准。