Ensembles of natural language processing systems for portable phenotyping solutions

Background: Manually curating standardized phenotypic concepts such as Human Phenotype Ontology (HPO) terms from narrative text in electronic health records (EHRs) is time consuming and error prone. Natural language processing (NLP) techniques can facilitate automated phenotype extraction and thus improve the efficiency of curating clinical phenotypes from clinical texts. While individual NLP systems can perform well for a single cohort, an ensemble-based method might shed light on increasing the portability of NLP pipelines across different cohorts. Methods: We compared four NLP systems, MetaMapLite, MedLEE, ClinPhen and cTAKES, and four ensemble techniques, including intersection, union, majority-voting and machine learning, for extracting generic phenotypic concepts. We addressed two important research questions regarding automated phenotype recognition. First, we evaluated the performance of different approaches in identifying generic phenotypic concepts. Second, we compared the performance of different methods to identify patient-specific phenotypic concepts. To better quantify the effects caused by concept granularity differences on performance, we developed a novel evaluation metric that considered concept hierarchies and frequencies. Each of the approaches was evaluated on a gold standard set of clinical documents annotated by clinical experts. One dataset containing 1,609 concepts derived from 50 clinical notes from two different institutions was used in both evaluations, and an additional dataset of 608 concepts derived from 50 case report abstracts obtained from PubMed was used for evaluation of identifying generic phenotypic concepts only. Results: For generic phenotypic concept recognition, the top three performers in the NYP/CUIMC dataset are union ensemble (F₁, 0.634), training-based ensemble (F₁, 0.632), and majority vote-based ensemble (F₁, 0.622). In the Mayo dataset, the top three are majority vote-based ensemble (F₁, 0.642), cTAKES (F₁, 0.615), and MedLEE (F₁, 0.559). In the PubMed dataset, the top three are majority vote-based ensemble (F₁, 0.719), training-based (F₁, 0.696) and MetaMapLite (F₁, 0.694). For identifying patient specific phenotypes, the top three performers in the NYP/CUIMC dataset are majority vote-based ensemble (F₁, 0.610), MedLEE (F₁, 0.609), and training-based ensemble (F₁, 0.585). In the Mayo dataset, the top three are majority vote-based ensemble (F₁, 0.604), cTAKES (F₁, 0.531) and MedLEE (F₁, 0.527). Conclusions: Our study demonstrates that ensembles of natural language processing can improve both generic phenotypic concept recognition and patient specific phenotypic concept identification over individual systems. Among the individual NLP systems, each individual system performed best when they were applied in the dataset that they were primary designed for. However, combining multiple NLP systems to create an ensemble can generally improve the performance. Specifically, the ensemble can increase the results reproducibility across different cohorts and tasks, and thus provide a more portable phenotyping solution compared to individual NLP systems.