Stochastic identification of Malware with dynamic traces

Curtis Storlie, Blake Anderson, Scott Vander Wiel, Daniel Quist, Curtis Hash, Nathan Brown

Research output: Contribution to journalArticlepeer-review

13 Scopus citations


A novel approach to malware classification is introduced based on analysis of instruction traces that are collected dynamically from the program in question. The method has been implemented online in a sandbox environment (i.e., a security mechanism for separating running programs) at Los Alamos National Laboratory, and is intended for eventual host-based use, provided the issue of sampling the instructions executed by a given process without disruption to the user can be satisfactorily addressed. The procedure represents an instruction trace with a Markov chain structure in which the transition matrix, P, has rows modeled as Dirichlet vectors. The malware class (malicious or benign) is modeled using a flexible spline logistic regression model with variable selection on the elements of P, which are observed with error. The utility of the method is illustrated on a sample of traces from malware and nonmalware programs, and the results are compared to other leading detection schemes (both signature and classification based). This article also has supplementary materials available online.

Original languageEnglish (US)
Pages (from-to)1-18
Number of pages18
JournalAnnals of Applied Statistics
Issue number1
StatePublished - Mar 2014


  • Adaptive lasso
  • Classification
  • Elastic net
  • Empirical bayes
  • Logistic regression
  • Malware detection
  • Relaxed lasso
  • Splines

ASJC Scopus subject areas

  • Statistics and Probability
  • Modeling and Simulation
  • Statistics, Probability and Uncertainty


Dive into the research topics of 'Stochastic identification of Malware with dynamic traces'. Together they form a unique fingerprint.

Cite this