AUTHOR=Yildiz Orcun , Morozov Dmitriy , Nigmetov Arnur , Nicolae Bogdan , Peterka Tom 

TITLE=Wilkins: HPC in situ workflows made easy

JOURNAL=Frontiers in High Performance Computing

VOLUME=Volume 2 - 2024

YEAR=2024

URL=https://www.frontiersin.org/journals/high-performance-computing/articles/10.3389/fhpcp.2024.1472719

DOI=10.3389/fhpcp.2024.1472719

ISSN=2813-7337

ABSTRACT=In situ approaches can accelerate the pace of scientific discoveries by allowing scientists to perform data analysis at simulation time. Current in situ workflow systems, however, face challenges in handling the growing complexity and diverse computational requirements of scientific tasks. In this work, we present Wilkins, an in situ workflow system that is designed for ease-of-use while providing scalable and efficient execution of workflow tasks. Wilkins provides a flexible workflow description interface, employs a high-performance data transport layer based on HDF5, and supports tasks with disparate data rates by providing a flow control mechanism. Wilkins seamlessly couples scientific tasks that already use HDF5, without requiring task code modifications. We demonstrate the above features using both synthetic benchmarks and two science use cases in materials science and cosmology.