AUTHOR=Milan Petro Junior , Rong Hongqian , Michaud Craig , Layad Naoufal , Liu Zhengchun , Coffee Ryan 

TITLE=Enabling real-time adaptation of machine learning models at x-ray Free Electron Laser facilities with high-speed training optimized computational hardware

JOURNAL=Frontiers in Physics

VOLUME=Volume 10 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/physics/articles/10.3389/fphy.2022.958120

DOI=10.3389/fphy.2022.958120

ISSN=2296-424X

ABSTRACT=The emergence of novel computational hardware is enabling a new paradigm for rapid machine learning model training.
For the Department of Energy's major research facilities, this developing technology will enable a highly adaptive approach to experimental sciences.
In this manuscript, we present the per-epoch and end-to-end training times for an example of a streaming diagnostic that is planned for the upcoming high-repetition-rate x-ray Free Electron Laser, the Linac Coherent Light Source-II.
We explore the parameter space of batch size and data parallel training across graphics processing units and Reconfigurable DataFlow Units across multiple interconnected host nodes.
We show the landscape of training times with a goal of full model retraining in under 15 minutes.
Although a full from scratch retraining of a model may not be required in all cases, we nevertheless present an example of the application of emerging computational hardware for adapting machine learning models to changing environments in real-time, during streaming data acquisition, at the rates expected for the data fire hoses of accelerator-based user facilities.