Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation

Marika Cusick, Prakash Adekkanattu, Thomas R. Campion, Evan T. Sholle, Annie Myers, Samprit Banerjee, George Alexopoulos, Yanshan Wang, Jyotishman Pathak

Research output: Contribution to journalArticlepeer-review


Mental health concerns, such as suicidal thoughts, are frequently documented by providers in clinical notes, as opposed to structured coded data. In this study, we evaluated weakly supervised methods for detecting “current” suicidal ideation from unstructured clinical notes in electronic health record (EHR) systems. Weakly supervised machine learning methods leverage imperfect labels for training, alleviating the burden of creating a large manually annotated dataset. After identifying a cohort of 600 patients at risk for suicidal ideation, we used a rule-based natural language processing approach (NLP) approach to label the training and validation notes (n = 17,978). Using this large corpus of clinical notes, we trained several statistical machine learning models—logistic classifier, support vector machines (SVM), Naive Bayes classifier—and one deep learning model, namely a text classification convolutional neural network (CNN), to be evaluated on a manually-reviewed test set (n = 837). The CNN model outperformed all other methods, achieving an overall accuracy of 94% and a F1-score of 0.82 on documents with “current” suicidal ideation. This algorithm correctly identified an additional 42 encounters and 9 patients indicative of suicidal ideation but missing a structured diagnosis code. When applied to a random subset of 5,000 clinical notes, the algorithm classified 0.46% (n = 23) for “current” suicidal ideation, of which 87% were truly indicative via manual review. Implementation of this approach for large-scale document screening may play an important role in point-of-care clinical information systems for targeted suicide prevention interventions and improve research on the pathways from ideation to attempt.

Original languageEnglish (US)
Pages (from-to)95-102
Number of pages8
JournalJournal of Psychiatric Research
StatePublished - Apr 2021


  • Deep learning
  • Machine learning
  • Natural language processing
  • Suicidal ideation
  • Weak supervision

ASJC Scopus subject areas

  • Psychiatry and Mental health
  • Biological Psychiatry


Dive into the research topics of 'Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation'. Together they form a unique fingerprint.

Cite this