Toward a Persistent Event-Streaming System for High-Performance Computing Applications

Matthieu Dorier^1*Amal Gueroudji¹Valerie Hayot-Sasson²Hai Nguyen^1,2Seth Ockerman³Renan Souza⁴Tekin Bicer¹Haochen Pan²Philip Carns¹Kyle Chard^1,2Ryan Chard¹Maxime Gonthier^1,2Eliu A Huerta¹Ben Lenard¹Bogdan Nicolae¹Parth Patel¹Justin M Wozniak¹Ian Foster^1,2Nageswara Rao⁴Robert B Ross¹

• ¹Argonne National Laboratory, Lemont, United States
• ²The University of Chicago Department of Computer Science, Chicago, United States
• ³University of Wisconsin-Madison, Madison, United States
• ⁴Oak Ridge National Laboratory, Oak Ridge, United States

High-performance computing (HPC) applications have traditionally relied on parallel file systems and file transfer services to manage data movement and storage. Alternative approaches have been proposed that use direct communications between application components, trading persistence and fault tolerance for speed. Event-driven architectures, as popularized in enterprise contexts, present a compelling middle ground, avoiding the performance cost and API constraints of parallel file systems while retaining persistence and offering impedance matching between application components. However, adapting streaming frameworks to HPC workloads requires addressing challenges unique to HPC systems. This paper investigates the potential for a streaming framework designed for HPC infrastructures and use cases. We introduce Mofka, a persistent event-streaming framework designed specifically for HPC environments. Mofka combines the capabilities of a traditional streaming service with optimizations tailored to the HPC context, such as support for massively multicore nodes, efficient scaling for large producerconsumer workflows, RDMA-enabled high-performance network communications, specialized network fabrics with multiple links per node, and efficient handling of large scientific data payloads. Built using the Mochi suite of HPC data service components, Mofka provides a lightweight, modular, and high-performance solution for persistent streaming in HPC systems.We present the architecture of Mofka and evaluate its performance against Kafka and Redpanda using benchmarks on diverse platforms, including Argonne's Polaris and Oak Ridge's Frontier supercomputers, showing up to 8× improvement in throughput in some scenarios. We then demonstrate its utility in several real-world applications: a tomographic reconstruction pipeline, a workflow for the discovery of metal-organic frameworks for carbon capture, and the instrumentation of Dask workflows for provenance tracking and performance analysis.