Malware Detection Using Nonparametric Bayesian Clustering and Classification Techniques

Yimin Kao, Brian Reich, Curtis Storlie, Blake Anderson

Research output: Contribution to journalArticlepeer-review

4 Scopus citations


Computer security requires statistical methods to quickly and accurately flag malicious programs. This article proposes a nonparametric Bayesian approach for classifying programs as benign or malicious and simultaneously clustering malicious programs. The analysis is based on the dynamic trace (DT) of instructions under the first-order Markov assumption. Each row of the traces transition matrix is modeled using the Dirichlet process mixture (DPM) model. The DPM model clusters programs within each class (malicious or benign), and produces the posterior probability of being a malware which is used for classification. The novelty of the model is using this clustering algorithm to improve the classification accuracy. The simulation study shows that the DPM model outperforms the elastic net logistic (ENL) regression and the support vector machine (SVM) in classification performance under most of the scenarios, and also outperforms the spectral clustering method for grouping similar malware. In an analysis of real malicious and benign programs, the DPM model gives significantly better classification performance than the ENL model, and competitive results to the SVM. More importantly, the DPM model identifies clusters of programs during the classification procedure which is useful for reverse engineering.

Original languageEnglish (US)
Pages (from-to)535-546
Number of pages12
Issue number4
StatePublished - Oct 2 2015


  • Classification
  • Clustering
  • Dirichlet process mixture
  • Dynamic trace

ASJC Scopus subject areas

  • Statistics and Probability
  • Modeling and Simulation
  • Applied Mathematics


Dive into the research topics of 'Malware Detection Using Nonparametric Bayesian Clustering and Classification Techniques'. Together they form a unique fingerprint.

Cite this