AUTHOR=Nguyen Hai Duc , Bicer Tekin , Nicolae Bogdan , Kettimuthu Rajkumar , Huerta E. A. , Foster Ian T. 

TITLE=Resilient execution of distributed X-ray image analysis workflows

JOURNAL=Frontiers in High Performance Computing

VOLUME=Volume 3 - 2025

YEAR=2025

URL=https://www.frontiersin.org/journals/high-performance-computing/articles/10.3389/fhpcp.2025.1550855

DOI=10.3389/fhpcp.2025.1550855

ISSN=2813-7337

ABSTRACT=Long-running scientific workflows, such as tomographic data analysis pipelines, are prone to a variety of failures, including hardware and network disruptions, as well as software errors. These failures can substantially degrade performance and increase turnaround times, particularly in large-scale, geographically distributed, and time-sensitive environments like synchrotron radiation facilities. In this work, we propose and evaluate resilience strategies aimed at mitigating the impact of failures in tomographic reconstruction workflows. Specifically, we introduce an asynchronous, non-blocking checkpointing mechanism and a dynamic load redistribution technique with lazy recovery, designed to enhance workflow reliability and minimize failure-induced overheads. These approaches facilitate progress preservation, balanced load distribution, and efficient recovery in error-prone environments. To evaluate their effectiveness, we implement a 3D tomographic reconstruction pipeline and deploy it across Argonne's leadership computing infrastructure and synchrotron facilities. Our results demonstrate that the proposed resilience techniques significantly reduce failure impact—by up to 500× —while maintaining negligible overhead (<3%).