EMPIRE-PIC: A Performance Portable Unstructured Particle-in-Cell Code (Communications in Computational Physics, 2021)
In this paper we introduce EMPIRE-PIC, a finite element method particle-in-cell (FEM-PIC) application developed at Sandia National Laboratories. The codehas been developed in C++ using the Trilinos library and the Kokkos PerformancePortability Framework to enable running on multiple modern compute architectureswhile only requiring maintenance of a single codebase. EMPIRE-PIC is capable of solving both electrostatic and electromagnetic problems in two- and three-dimensions tosecond-order accuracy in space and time. In this paper we validate the code againstthree benchmark problems — a simple electron orbit, an electrostatic Langmuir wave,and a transverse electromagnetic wave propagating through a plasma. We demonstrate the performance of EMPIRE-PIC on four different architectures: Intel HaswellCPUs, Intel's Xeon Phi Knights Landing, ARM Thunder-X2 CPUs, and NVIDIA TeslaV100 GPUs attached to IBM POWER9 processors. This analysis demonstrates scalability of the code up to more than two thousand GPUs, and greater than one hundredthousand CPUs.