Algorithmic Identification of Treatment‐Emergent Adverse Events From Clinical Notes Using Large Language Models: A Pilot Study in Inflammatory Bowel Disease

Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers...

Full description

Saved in:
Bibliographic Details
Published in:Clinical pharmacology and therapeutics 2024-06, Vol.115 (6), p.1391-1399
Main Authors: Silverman, Anna L., Sushil, Madhumita, Bhasuran, Balu, Ludwig, Dana, Buchanan, James, Racz, Rebecca, Parakala, Mahalakshmi, El‐Kamary, Samer, Ahima, Ohenewaa, Belov, Artur, Choi, Lauren, Billings, Monisha, Li, Yan, Habal, Nadia, Liu, Qi, Tiwari, Jawahar, Butte, Atul J., Rudrapatna, Vivek A.
Format: Article
Language:English
Subjects:
Adverse Drug Reaction Reporting Systems
Algorithms
Data Mining - methods
Drug-Related Side Effects and Adverse Reactions - diagnosis
Electronic Health Records
Female
Hospitalization - statistics & numerical data
Humans
Immunosuppressive Agents - adverse effects
Inflammatory Bowel Diseases - drug therapy
Male
Natural Language Processing
Pharmacovigilance
Pilot Projects
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Outpatient clinical notes are a rich source of information regarding drug safety. However, data in these notes are currently underutilized for pharmacovigilance due to methodological limitations in text mining. Large language models (LLMs) like Bidirectional Encoder Representations from Transformers (BERT) have shown progress in a range of natural language processing tasks but have not yet been evaluated on adverse event (AE) detection. We adapted a new clinical LLM, University of California – San Francisco (UCSF)‐BERT, to identify serious AEs (SAEs) occurring after treatment with a non‐steroid immunosuppressant for inflammatory bowel disease (IBD). We compared this model to other language models that have previously been applied to AE detection. We annotated 928 outpatient IBD notes corresponding to 928 individual patients with IBD for all SAE‐associated hospitalizations occurring after treatment with a non‐steroid immunosuppressant. These notes contained 703 SAEs in total, the most common of which was failure of intended efficacy. Out of eight candidate models, UCSF‐BERT achieved the highest numerical performance on identifying drug‐SAE pairs from this corpus (accuracy 88–92%, macro F1 61–68%), with 5–10% greater accuracy than previously published models. UCSF‐BERT was significantly superior at identifying hospitalization events emergent to medication use (P 
ISSN:0009-9236
1532-6535
DOI:10.1002/cpt.3226