Power usage of production supercomputers and production workloads

Summary Power is becoming an increasingly important concern for large supercomputer centers. However, to date, there have been a dearth of studies of power usage 'in the wild' - on production supercomputers running production workloads. In this paper, we present the initial results of a project to characterize the power usage of the three Top500 supercomputers at Los Alamos National Laboratory: Cielo, Roadrunner, and Luna (#15, #19, and #47, respectively, on the June 2012 Top500 list). Power measurements taken both at the switchboard level and within the compute racks are presented and discussed. Some noteworthy results of this study are that (1) variability in power consumption differs across architectures, even when running a similar workload and (2) Los Alamos National Laboratory's scientific workload draws, on average, only 70-75% of LINPACK power and only 40-55% of nameplate power, implying that power capping may enable a substantial reduction in power and cooling infrastructure while impacting comparatively few applications.