大语言模型在整形外科各亚专科手术记录模板CPT编码中的准确性评估
模型间总体编码准确率存在显著差异(p=0.02176),Gemini与Copilot准确率最高(各19.2%),GPT-4最低(7.7%)。所有模型均未能正确编码乳房再造或颅颌面创伤手术。
要点
- 模型间总体CPT编码准确率存在显著差异(p=0.02176)。
- Gemini与Copilot准确率最高(各19.2%),GPT-4最低(7.7%)。
- 所有模型均未正确编码乳房再造或颅颌面创伤手术。
核心结论
- 大语言模型在CPT编码自动化方面显示潜力,但目前缺乏可靠准确性所需的上下文理解能力;持续人工监督和模型改进对未来成功至关重要。
主要发现
- 模型间总体编码准确率存在显著差异(p=0.02176)。
- Gemini与Copilot准确率最高(各19.2%),Copilot产生更多部分正确输出(53.8%),GPT-4准确率最低(7.7%)。
- 亚专科分析显示Gemini在美容整形中表现最佳(60%),Copilot在一般重建中准确率最高(42.9%)。
- 所有模型均未正确编码乳房再造或颅颌面创伤手术。
- 常见错误包括手术细节识别错误和CPT编码的不当捆绑。
临床怎么用
- 大语言模型目前尚不能替代人工进行可靠的CPT编码,临床使用中需持续人工监督。
局限性
- 原文未述局限性
以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。
原文
BACKGROUND: Manual CPT coding from operative notes is a time-intensive process that adds to the administrative burden in healthcare. Large Language Models (LLMs) offer a promising solution, but their accuracy in assigning CPT codes based on full operative note templates remains largely untested. Thus, this study evaluates the ability of three LLMs - GPT-4, Gemini, and Copilot - to generate accurate CPT codes from operative note templates across diverse plastic surgery procedures. METHODS: Twenty-six deidentified operative note templates from six plastic surgery subspecialities were entered into each LLM using a standardized prompt requesting appropriate CPT codes. Model outputs were compared to surgeon-verified codes and categorized as correct (all codes accurate), partially correct (some correct codes with errors), or incorrect (no correct codes). Accuracy was analyzed overall and by subspeciality using Extended Fisher's Exact Tests (significance set at p<0.05). RESULTS: There was a significant difference in overall coding accuracy between the models (p = 0.02176). Gemini and Copilot had the highest accuracy rates (19.2% each), though Copilot produced more partially correct outputs (53.8%). GPT-4 had the lowest accuracy (7.7%). Subspeciality analysis showed Gemini performed best in aesthetic surgery (60%), while Copilot was most accurate in general reconstruction (42.9%). None of the models correctly coded breast reconstruction or craniofacial trauma procedures. Frequent errors included misidentification of procedural details and inappropriate bundling of CPT codes. CONCLUSION: LLMs show potential for automating CPT coding but currently lack the contextual understanding required for reliable accuracy. Continued human oversight and model refinement are essential for future success of LLM CPT coding.
引用
- Carrarini MJ, Liu HY, Perez CK, et al. Evaluating Large Language Model's accuracy in current procedural terminology coding given operative note templates across various plastic surgery sub-specialties. J Plast Reconstr Aesthet Surg. 2025. PMID: 40367652. doi:10.1016/j.bjps.2025.04.025
供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。