We present the modular framework SNABSuite (Spiking Neural Architecture Benchmark Suite) for black-box benchmarking of neuromorphic hardware systems and spiking neural network software simulators. The motivation for having a coherent collection of benchmarks is twofold: first, benchmarks evaluated on different platforms provide measures for direct comparison of performance indicators (e.g. resource efficiency, quality of the result, robustness). By using the platforms as they are provided for possible end-users and evaluating selected performance indicators, benchmarks support the decision for or against a system based on use-case requirements. Second, benchmarks may reveal opportunities for effective improvements of a system and can contribute to future development.

Systems like the Heidelberg BrainScaleS project, IBM TrueNorth, the Manchester SpiNNaker project or the Intel Loihi platform drive the evolution of neuromorphic hardware implementations, while comparable benchmarks and corresponding measures are still rare.

We show our methodology for comparing such diverse systems by applying a modular framework, with a user- centric view based on configurable spiking neural network descriptions.