文献速递胸部技术与安全CT

放射学报告的自动匿名化:公开可用的自然语言处理与大语言模型的比较

Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models.
PubMed 文献 · Eur Radiol · 2024-10-31

该研究比较了两种公开可用的规则型自然语言处理模型(spaCy;NLPac 精度优化版、NLPsp 速度优化版)与一种离线大语言模型方法(LLaMa-2)对自由文本胸部 CT 报告中的个人健康信息进行匿名化的效果。结果显示预训练 NLP 模型能有效匿名化且不删除医学信息,而大语言模型虽精度高但更易删除医学信息且对日期召回率低。

要点

  • NLPac 对日期、病历号、 accession 号及姓名的匿名化表现最稳定,F1 达 1.00。
  • NLPsp 对日期匿名化的精确度 0.86、F1 0.92,低于 NLPac。
  • 大语言模型(LLM)对日期召回率仅 0.52,且在 10% 报告中删除了医学信息。

核心结论

  • 预训练 NLP 模型可有效匿名化自由文本放射学报告,而使用大语言模型进行匿名化更易删除医学信息。

研究设计与样本

  • 研究比较两种公开可用的规则型 NLP 模型(spaCy;NLPac 精度优化、NLPsp 速度优化,在 400 份自由文本 CT 报告上迭代改进作为测试集)与一种离线 LLM 方法(LLaMa-2),在 100 份随机选择的胸部 CT 报告上测试,由两名研究者评估匿名化效果并计算精确度、召回率和 F1 分数。

诊断标准与关键数值

  • NLPac 和 NLPsp 成功移除了所有日期(n=333)、病历号(n=6)和 accession 号(n=92)。
  • LLM 模型移除了所有病历号、96% 的 accession 号和 32% 的日期。
  • NLPac 表现最一致,F1 分数为 1.00;NLPsp 对日期精确度 0.86、F1 分数 0.92。
  • LLM 模型对病历号、accession 号和日期的精确度均为 1.00,但对 accession 号召回率 0.96(F1 0.98)、日期召回率 0.52(F1 0.68)。
  • 姓名完全或主要移除的比例:NLPac 100%、NLPsp 72%、LLM 模型 90%。

临床应用与阅片要点

  • NLPac 和 NLPsp 未删除医学信息,而 LLM 模型在 10%(n=10)的报告中删除了医学信息;快速可靠的自动匿名化有助于符合 HIPAA 的二次使用。

局限性

  • 原文未报告局限性

以上解读依据论文摘要(全文未开放获取或许可不允许翻译转载)。

英文原文(摘要)· 点击展开

PURPOSE: Medical reports, governed by HIPAA regulations, contain personal health information (PHI), restricting secondary data use. Utilizing natural language processing (NLP) and large language models (LLM), we sought to employ publicly available methods to automatically anonymize PHI in free-text radiology reports.

MATERIALS AND METHODS: We compared two publicly available rule-based NLP models (spaCy; NLPac, accuracy-optimized; NLPsp, speed-optimized; iteratively improved on 400 free-text CT-reports (test set)) and one offline LLM approach (LLM-model, LLaMa-2, Meta-AI) for PHI-anonymization. The three models were tested on 100 randomly selected chest CT reports. Two investigators assessed the anonymization of occurring PHI entities and whether clinical information was removed. Subsequently, precision, recall, and F1 scores were calculated.

RESULTS: NLPac and NLPsp successfully removed all instances of dates (n = 333), medical record numbers (MRN) (n = 6), and accession numbers (ACC) (n = 92). The LLM model removed all MRNs, 96% of ACCs, and 32% of dates. NLPac was most consistent with a perfect F1-score of 1.00, followed by NLPsp with lower precision (0.86) and F1-score (0.92) for dates. The LLM model had perfect precision for MRNs, ACCs, and dates but the lowest recall for ACC (0.96) and dates (0.52), corresponding F1 scores of 0.98 and 0.68, respectively. Names were removed completely or majorly (i.e., one first or family name non-anonymized) in 100% (NLPac), 72% (NLPsp), and 90% (LLM-model). Importantly, NLPac and NLPsp did not remove medical information, while the LLM model did in 10% (n = 10).

CONCLUSION: Pre-trained NLP models can effectively anonymize free-text radiology reports, while anonymization with the LLM model is more prone to deleting medical information.

KEY POINTS: Question This study compares NLP and locally hosted LLM techniques to ensure PHI anonymization without losing clinical information. Findings Pre-trained NLP models effectively anonymized radiology reports without removing clinical data, while a locally hosted LLM was less reliable, risking the loss of important information. Clinical relevance Fast, reliable, automated anonymization of PHI from radiology reports enables HIPAA-compliant secondary use, facilitating advanced applications like LLM-driven radiology analysis while ensuring ethical handling of sensitive patient data.

来源:Eur Radiol,查看原文。版权归原作者所有。

引用

  1. Langenbach MC, Foldyna B, Hadzic I, et al. Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models. Eur Radiol. 2024. PMID: 39480533. doi:10.1007/s00330-024-11148-x

供执业医师工作参考,不替代临床判断;数值与分级以指南原文为准。