Distance-Weighted Graph Neural Networks on FPGAs for Real-Time Particle Reconstruction in High Energy Physics

Graph neural networks have been shown to achieve excellent performance for several crucial tasks in particle physics, such as charged particle tracking, jet tagging, and clustering. An important domain for the application of these networks is the FGPA-based first layer of real-time data filtering at the CERN Large Hadron Collider, which has strict latency and resource constraints. We discuss how to design distance-weighted graph networks that can be executed with a latency of less than one μs on an FPGA. To do so, we consider a representative task associated to particle reconstruction and identification in a next-generation calorimeter operating at a particle collider. We use a graph network architecture developed for such purposes, and apply additional simplifications to match the computing constraints of Level-1 trigger systems, including weight quantization. Using the hls4ml library, we convert the compressed models into firmware to be implemented on an FPGA. Performance of the synthesized models is presented both in terms of inference accuracy and resource usage.


Introduction
At the CERN Large Hadron Collider (LHC), high-energy physics (HEP) experiments collect signals generated by the particles produced in high-energy proton collisions that occur every 25 ns, when two proton beams cross.The readout from the detectors that capture the particles emerging from the collision is filtered by a real-time processing system, known as the trigger, that discards uninteresting collision events, based on a set of predefined algorithms.The trigger system is structured in two stages: a Level-1 trigger (L1T), implemented with custom electronics on-detector and field-programmable gate arrays (FPGAs); and a high-level trigger (HLT), consisting of a computer farm, possibly including co-processor accelerators like graphics processing units (GPUs) and FPGAs.Because of asynchronous event processing at the HLT, the accept/reject decision has to be reached with a typical latency of O(100) ms.However, at the L1T, a decision must be taken within a fixed latency of O(1) µs.The main limitations are the synchronous, "hard-deadline" nature of the processing system and the limited size of the memory buffer for the data from each beam crossing.
While HLT algorithms have a complexity comparable to those used offline to produce the final physics results, a typical L1T algorithm consists of simpler rules based on coarser objects to satisfy the latency constraint.Consequently, the resolution of quantities computed at the L1T is typically poor compared to offline quantities.Recently, the successful deployment of the first machine learning (ML) L1T algorithm, based on a boosted decision tree (BDT), at the LHC [1] has changed this tendency, raising interest in using ML inference as fast-to-execute approximations of complex algorithms with good accuracy.This first example consisted of a large, pre-computed table of input and output values implementing a BDT, which raises the question of how to deploy more complex architectures.This question motivated the creation of hls4ml [2], a library designed to facilitate the deployment of ML algorithms on FPGAs.
A typical hls4ml workflow begins with a neural network model that is implemented and trained using Keras [3], PyTorch [4], or TensorFlow [5].The trained model is passed to hls4ml, directly or through the ONNX [6] interface, and converted to C++ code that can be processed by a high-level synthesis (HLS) compiler to produce an FPGA firmware.By design, hls4ml targets low-latency applications.To this end, its design prioritizes all-on-chip implementations of the most common network components.Its functionality has been demonstrated with dense neural networks (DNNs) [2], extended to also support BDTs [7].Extensions to convolutional and recurrent neural networks are in development.The library comes with handles to compress the model by quantization, up to binary and ternary precision [8].Recently, support for QKeras [9] models has been added, in order to allow for quantization-aware training of models [10].While the hls4ml applications go beyond HEP, its development has been driven by the LHC L1T use case.
Graph neural networks (GNNs) are among the complex architectures whose L1T implementations are in high demand, given the growing list of examples showing how well GNNs can deal with tasks related to HEP [11][12][13][14][15][16][17][18][19][20][21][22].In fact, while the irregular geometry of a typical HEP detector complicates the use of computing vision techniques such as convolutional neural networks, GNNs can naturally deal with the sparse and irregular nature of HEP data.
In this work, we show how a graph model can be efficiently deployed on FPGAs to perform inference within O(1) µs for HEP-related problems.We consider the distance-weighted architecture GarNet, introduced in Ref. [15], which is designed to keep resource consumption under control by reducing as much as possible the number of operations.It has been demonstrated to perform well for a HEP-related task, namely particle reconstruction in a calorimeter.For these reasons, it represents a good candidate for our purpose.The firmware implementation of GarNet presented in this work has been included in hls4ml, representing the first graph-based algorithm available in the library.
We present a case study of a neural network algorithm based on GarNet, applied to a task of identifying the nature of an incoming particle and simultaneously estimating its energy from the energy deposition patterns in a simulated imaging calorimeter.The inference accuracy of the firmware implementation of the algorithm is compared against its offline counterpart running on processors (CPUs and GPUs).Latency and resource utilization of the translated FPGA firmware are reported, along with a discussion on their implications for real-world usage of similar algorithms.This paper is structured as follows.In Section 2, we briefly recount related work.Section 3 defines the main problem by outlining the challenges in designing a graph network compatible with L1T latency and resource constraints.Section 4 describes how GarNet addresses these challenges, and introduces a simplified form of the algorithm with a better affinity to a firmware implementation.The case study using a calorimeter simulation is presented in Section 5, with detailed descriptions of the task setup, model architecture, training results, and the summary of FPGA firmware synthesis.Finally, conclusions are given in Section 6.

Related work
Graph neural networks are gaining interest in HEP applications, mainly due to their intrinsic value when dealing with sparse input datasets, which are very common in HEP.A recent review of applications of GNNs to HEP problems may be found in Ref. [22].In particular, dynamic GNNs [15,[23][24][25] are relevant for particle reconstruction tasks, such as tracking [20] and calorimetry [15].
In order to fully exploit DL-based particle reconstruction, the capability of deploying DL models to the real-time data filtering system of a HEP detector is essential.For the FPGA-based L1T systems of the LHC, automatic tools for network-to-circuit conversation, such as the hls4ml library, are emerging.Using hls4ml, several solutions for HEP-specific tasks (e.g.jet tagging) have been provided [2,7,8,10], exploiting models with simpler architectures than what is shown here.This tool has been applied extensively for tasks in the HL-LHC upgrade of the CMS L1T system, including an autoencoder for anomaly detection, and DNNs for muon energy regression and identification, tau lepton identification, and vector boson fusion event classification [26].

General requirements and challenges
In the framework of Ref. [27], a graph is a triplet (V, E, U), where V is a set of entities (vertices) each possessing some attributes in a fixed format, E is a set of pairwise relations (edges) between the elements in V, potentially possessing some additional attributes, and U are global (graph-level) attributes.While a GNN can be any neural network that acts on such graphs, in this work we specifically consider graph networks (GN) [27], i.e., architectures that consist of repeatable graph-to-graph mapping blocks (GN blocks).Each GN block performs some combination of operations such as edge feature transformation, aggregation of neighbors' features at each vertex, vertex feature transformation, global aggregation of edge and vertex features, and global feature transformation.A GN takes a graph as an input sample, where the cardinality of V may differ sample to sample, and infers its properties, which may be anything from a global scalar, such as a classification label of the sample, to new edge attributes.
To be usable as a part of an LHC L1T system, an algorithm must execute within O(1) µs and have the throughput to accept all inputs from each beam crossing every 25 ns.Time-multiplexing, whereby N copies of the algorithm accept inputs from N different beam crossings, may be used to decrease the throughput requirement by a factor of N .Additionally, there is a practical constraint that the firmware implementation should fit in the FPGA resources of the system, i.e., utilize the resources such as digital signal processing units (DSPs), look-up tables (LUTs), flip-flips (FFs), and block RAM (BRAM) within the limits of chips available on the market.Satisfying these requirements with a GNN can be challenging for multiple reasons listed below.
• Model depth: Within each GN block, vertices exchange information with other directly connected vertices or with global attributes.Therefore, to expand the receptive field of each vertex beyond the nearest neighbors, multiple GN blocks must be repeated in the network.Given that various transformations within each GN block are often themselves multilayer perceptrons (MLPs), GNN models tend to be quite deep.Deep networks go against the latency requirement, as each perceptron layer uses at least one clock cycle on an FPGA under a straightforward implementation, and also against the resource usage requirement, because MLPs utilize multiplications heavily.
• Input size: Typically, for problems where the application of GNNs is interesting, the cardinality of V is at least O(10 2 ).Even with the high degree of parallelism of FPGAs, due to finiteness of the compute resource, such large input will have to be processed serially to a certain extent, increasing the latency and the interval before a new input can be accepted, known as the initiation interval (II).Longer IIs lead to lower throughput values.
• Memory usage: Related to the problem of the input size, if the algorithm requires temporary retention of features for all vertices or edges, memory usage may be prohibitive for an FPGA firmware implementation.
• Memory access pattern: Except for certain cases, algorithms that have both V and E in the input usually requires random memory access, for example when reading or writing features of vertices at the ends of the edges.This poses a challenge in FPGA firmware design not only because it implies that there needs to be a large enough memory bank to store all vertex and/or edge data, but also because random memory access itself is a costly operation [28].The exceptions include when E is trivial (E = ∅ or when the graph is complete) and when all samples have an identical graph topology.
In such cases, the memory access pattern of the algorithm is known at compile time and therefore can be statically scheduled in the FPGA firmware.
The case of E = ∅ is a rather extreme solution to the last challenge, but it is also attractive in terms of memory usage.In fact, even without explicit input edge features, a GNN can infer regional and non-local properties of the graph by globally gathering the vertex features and then scattering the gathered information back to the vertices.This information flow can also be mediated by a learnable attention mechanism [29].
The attention mechanism suppresses information from vertices that are considered unimportant, effectively forming "soft" edges among the unsuppressed vertices.
In the next section, we study a GNN architecture with these exact properties, then discuss the modifications to the architecture to make it suitable for an FPGA firmware implementation.

A simplified GarNet layer in the hls4ml framework
In this work, we consider GarNet [15] as a specific example of GNN.A GarNet layer is a GN block that takes as input a set of V vertices, each possessing F in features, and returns the same set of vertices with F out features.In a GarNet layer, F in features of each vertex are encoded into an internal representation and gathered at S aggregators.A distance parameter between each of the aggregators and vertices is also computed from the vertex attributes.Information gathered at the aggregators are then sent back to individual vertices and decoded into F out features.Communications between the vertices and aggregators are weighted by a decreasing function of the distance parameter, implementing an attention mechanism that allows the network to learn a dynamic, nontrivial graph structure from the vertex input alone.
The original GarNet algorithm, while already using less compute and memory resource than other similar GNN architectures in Ref. [15,23], is still challenging to implement as fast and high-throughput FPGA firmware.The biggest problem arises from the use of the input feature vector as a part of the input to the decoder, which requires retention of the input data until the last steps of the algorithm.An immediate consequence of this requirement is a longer II, because processing of new samples cannot start while the input data for the current sample is still in use.Furthermore, the input feature vector is already used to compute the distance parameter as well as the internal representation of each vertex, and therefore a reuse of the input in the decoder creates a complex data flow, restricting the options for pipelining the algorithm.
We therefore designed a modified GarNet algorithm with a simplified processing flow: • Input transformation (Figs. 1a and 1b): An encoder network converts the features g j v (j = 1, . . ., F in ) of the v th vertex (v = 1, . . ., V ) into an internal learned representation vector f i v (i = 1, . . ., F LR ).In parallel, another network (distance calculator) also acts on g j v and computes the distance parameters d av (a = 1, . . ., S) between the vertices and the S aggregators.Implicitly, this means that a complete bipartite graph with V S edges is built from V and S, where S is the set of aggregators (Fig. 1b).The encoder and distance calculator networks are both single-layer perceptrons with linear activation functions, so one can write them as linear transformations where (w i j , b i ) and (α aj , β a ) are the kernels and biases of the encoder and distance calculator networks, respectively.
• Aggregation (Fig. 1c): The learned representation vectors f i v of the vertices are weighted by a potential function W av = exp(−d 2 av ) and averaged across the vertices.In other words, the ith averaged feature h i a of aggregator a is written as The factor V max in the denominator is the maximum possible value for the vertex multiplicity V (as V may have a different value for each input sample).Through this normalization by a common factor, the information about the size of the sample (cardinality of V) is effectively encoded into h i a .
• Output transformation (Figs. 1d and 1e): The aggregated features are sent back to the vertices using the same weights as ) and then transformed by a single-layer decoder network with linear activation function into the final output representation g k v (k = 1, . . ., F out ).With the kernel u and bias c of the decoder, this is written as This simplified algorithm differs from the original design in the following ways.First, only the mean over vertices is computed at the aggregators, whereas the maximum is also used in the original design.In other words, the aggregators in the original design have as an additional set of features.Secondly, as already noted, the input feature vector is not used as a part of the input to the decoder network.In the original GarNet design, the decoder is expressed as with additional sets of kernel weights u and w .Finally, the original design applies a nonlinear (tanh) activation function to the decoder, while the simplified version uses a linear activation.In the specific case considered in the next section, these simplifications result in negligible degradation of the network performance.
In the remainder of this paper, this simplified version of the algorithm is referred to as GarNet.
It is worth pointing out that while the GarNet layer uses only linear activation functions for all of the internal neural networks, it can still learn nonlinear functions through the non-linearity of the potential function W av .On the other hand, having no nonlinear activation functions allows a compact FPGA firmware implementation of the layer, consisting mostly of multiplications and additions.The only substantial computation comes with the exponential function, whose values can be pre-computed with sufficient granularity and stored.
An FPGA firmware implementation of the GarNet layer using Vivado [30] HLS is integrated into the hls4ml library.The HLS source code is written in C++ and is provided as a template, from which an HLS function for a GarNet layer can be instantiated, specifying the configurable parameters such as S, F LR , and F out .In the following, we provide some noteworthy details of the implementation.
In the HLS source code of GarNet, all quantities appearing in the computation are expressed as either integers or fixed-point numbers with fractional precision of at least eight bits.In particular, the distance parameter d av is represented with three integer bits, eight fractional bits, and one sign bit.During the layer computation, d av is reinterpreted as a 12-bit unsigned integer, which is used to retrieve the corresponding pre-computed value of W av from a table with 4,096 entries.
The processing flow in Eqs.(1) to ( 5) is compactified in the hls4ml implementation by exploiting the linearity of the encoder, average aggregation, and the decoder.Equations ( 1), (3), and ( 5) can be combined into where In particular, the kernel and bias tensors of the encoder and decoder are contracted into w and b at logic synthesis time, resulting in fewer steps to arrive at the output from the input.
With this simplification, the input data from each sample are encoded into W av , G j a , and L a .Therefore, a new sample can be processed as soon as the three quantities from the previous sample are computed.In other words, the II of the overall GarNet layer depends on the number of clock cycles needed to compute the three quantities.Furthermore, G j a and L a can be derived trivially from W av , making the latency of the computation of the latter the critical determinant of the throughput of the algorithm.
The computation of W av is performed independently on each vertex, and is therefore parallelizable across the vertices.In a fully parallelized implementation, there would be V max logic units (one unit per vertex) operated simultaneously.However, with V typically as large as O( 102 ) or greater, this configuration would consume too much of the FPGA resources and would not fit on a single chip.Therefore, the hls4ml implementation of GarNet allows a partial parallelization of the algorithm controlled by a parameter called the reuse factor (R reuse ).For R reuse > 1, the logic unit to compute W av is cloned V max /R reuse times, such that each unit is reused serially up to R reuse times.This serial reuse is fully pipelined with the local II of one clock cycle.The latency T W for computing W av for all vertices is therefore given by where T 0 W ∼ 20 is the number of clock cycles needed to compute W av for one vertex.The value of T 0 W depends on the numerical precision of the fixed-point numbers in the computation.
Finally, the kernel and bias of the encoder and the kernel of the decoder can be quantized, such that each element takes only values −1, 0, or 1 (ternary quantization) [31].In the quantized version of the algorithm, contracted kernel and bias w and b have elements that are O(1) integers.Multiplication of small integers with fixed-point numbers can be performed in FPGAs using LUTs rather than DSPs, which are usually the more scarce resource.Multiplications with LUTs also proceed faster than those with DSPs.

Case study: particle identification and energy regression in an imaging calorimeter
As a case study, the hls4ml implementation of GarNet is applied to a representative task for the LHC L1T, namely reconstructing electrons and pions in a simulated 3D imaging calorimeter.In the following, we first describe the dataset used for the study, then define the task and the architectures of the ML models, and present the inference performance of the models and the resource usage of the synthesized firmware.

Dataset
The calorimeter is a multi-layered full-absorption detector with a geometry similar to the one described in Ref. [15].The detector is made entirely of tungsten, which is considered as both an absorber and a sensitive material, and no noise or threshold effects in the readout electronics are simulated.While this homogeneous calorimeter design is not a faithful representation of a modern sampling calorimeter, this simplification allows us to evaluate the performance of the ML models decoupled from detector effects.
The calorimeter extends 36 cm in x and y and has a total depth in z of 2 m, corresponding to approximately 20 nuclear interaction lengths and 170 radiation lengths.The coordinate origin is placed at the center of the front face of the calorimeter.The calorimeter is segmented into 50 layers along z, with each layer divided into small square cells in the x-y plane, forming a three-dimensional imaging detector.Cells are oriented so their sides are parallel to the x and y axes.Tiling of the cells in each layer is uniform except for in one quadrant, where the cell sides are half as long as those in the other area.The aim of the tiling is to incorporate the irregularity of the geometry of a real-life particle physics calorimeter.The quadrant with smaller cells and the remainder of the layer are respectively called the high granularity (HG) and low granularity (LG) regions.
The first 25 layers in z correspond to the electromagnetic calorimeter, with a layer thickness of 1 cm and cell dimensions of 2.25 cm × 2.25 cm in the HG region (4.5 cm × 4.5 cm in LG).The remaining 25 layers correspond to the hadron calorimeter, with a layer thickness of 7 cm and cell dimensions of 3 cm × 3 cm in the HG region (6 cm × 6 cm in LG).Schematics of the cell tiling in the electromagnetic and hadron parts are shown in Fig. 2. The geometry and the detector response to particles are simulated using Geant4 [32].
Each event used in this study contains a high-energy primary particle and low-energy pileup particles, which represent backgrounds from simultaneous additional proton-proton interactions.The primary particle is either an electron (e − ) or a charged pion (π ± ), shot at the calorimeter with momentum aligned along the z axis, i.e., perpendicular to the front face of the calorimeter.The x and y coordinates of the particle's origin are randomly sampled according to a uniform distribution in a 10 cm × 10 cm region centered at x = y = 0. Following this procedure, we aim to mimic a realistic situation in which the actual calorimeter extends to a much larger surface and the area covered by the geometry used in this study represents a portion of it.The value of the particle momentum is drawn randomly for each event from a uniform distribution between 10 GeV and 100 GeV.The pileup particles consist of photons (γ) and π ± .The number of pileup particles is randomly sampled from a Poisson distribution with a mean of 40, with the π ± multiplicity fixed to twice the γ multiplicity.This setup approximates the flux of pileup particles expected at a pseudorapdity η = 2 in a ∆η × ∆φ = 0.4 × 0.4 patch of the forward region of an LHC detector during the High-Luminosity LHC (HL-LHC) phase [33].The momentum direction and the window of origin of the pileup particles are the same as the primary particle.The momentum value of the pileup particles is sampled from a Landau distribution with µ = 0.6 GeV and c = 0.5 GeV, in a range of 0 to 20 GeV.
The output of the simulation for each event is the array of total energy deposition values by the particles at individual detector cells (hits).Energy depositions by the particles in the homogeneous calorimeter are recorded exactly, i.e., the detector output does not require calibration and is not affected by stochastic noise.
In an L1T system, hits containing energy depositions from a potentially interesting particle would be identified through a low-latency clustering algorithm.The clustering algorithm used in this study mimics the one planned for the L1T system of the HGCAL detector in CMS [34].In this approach, the hit with the largest energy deposition in the event is elected to be the seed, and the cluster consists of all hits contained in a cylinder whose axis passes through the center of the seed cell and extends along the z direction.The radius of the cylinder is set at 6.4 cm so that the resulting cluster contains 95% of the energy of the primary particle for 50% of the pion events.Because electromagnetic showers have a narrower energy spread than hadronic showers in general, all of the electron events have at least 95% of the energy contained in the same cylinder.Typical events with momenta of the primary particles around 50 GeV and the total pileup energy close to the median of the distribution are shown in Fig. 3a.The hits in the figure are colored by the fraction of the hit energy due to the primary particle (primary fraction, f prim ) to help the visualization.
The actual dataset used in this study thus contains one cluster per sample, given as an array of hits in the cluster, and one integer indicating the number of hits in the sample.Only the hits with energy greater than 120 MeV are considered.Each cluster contains at most 128 hits, sorted by hit energy in decreasing order.Note that sorting of the hit has no effect on the neural network, and is only relevant when truncating the list of hits to consider smaller clusters, as explored later.In fact, 0.2% of the events resulted in clusters with more than 128 hits, for which the lowest energy hits were discarded from the dataset.Each hit is represented by four numbers, corresponding to the hit coordinates, given in x, y, and z, and energy.The x and y coordinates are relative to the seed cell.The dataset consists of 500,000 samples, split evenly and randomly into e − and π ± events, stored as NumPy [35] arrays in HDF5 format [36].The dataset together with the ground truth information is available on the Zenodo platform [37].Values in parentheses in the graph titles are the respective energy depositions contained in the cluster around the seed hit.Points represent hits in the detector, with their coordinates at the center of the corresponding detector cells and the size of the markers proportional to the square root of the hit energy.Opaque points are within the cluster, while the translucent ones are not.In (a), the point color scale from blue to red corresponds to the primary fraction (see Section 5.1 for definition).In (b), the color scale from blue to green corresponds to ∆E pred /∆h, which is an indication of the importance the neural network model places to individual hits for energy regression.See Section 5.3 for details.

Task and model architecture
The task in this study is to identify the nature of the primary particle and to simultaneously predict its energy, given the hits in the cluster.The ability to reliably identify the particle type and estimate its energy at the cluster level in a local calorimeter trigger system greatly enhances the efficacy of high-level algorithms, such as particle-flow reconstruction [38][39][40], downstream in the L1T system.However, because of the distortion of the energy deposition pattern in the cluster due to pileup, particle identification based on collective properties of the hits, such as the depth of the energy center of mass, can achieve only modest accuracy.Furthermore, only half of the pion events have 95% of the energy deposition from the pion contained in the cluster, requiring substantial extrapolation in the energy prediction.This task is thus both practically relevant and sufficiently nontrivial as a test bench of a GarNet-based ML model.
The architecture of the model is as follows.First, the input data represented by a two-dimensional array of V max × F in numbers per cluster are processed by a stack of three GarNet layers.The parameters (S, F LR , F out ) for the first two layers are (4,8,8) and for the last layer are (8,16,16).The output of the third GarNet layer is averaged across the vertices for each of the 16 features.The resulting array of 16 numbers is then passed through two fully connected layers with 16 and 8 nodes and ReLU [41] activation.Data flow is split into two branches in the final step.The first branch consists of a fully connected layer with a single node, whose output is activated by a sigmoid function and is interpreted as the classification prediction, i.e., the predicted probability that the primary particle is an electron.The other branch also consists of a single-node fully connected layer, but with a linear activation of the output, which is interpreted as the predicted value of the energy of the particle.This model is built in Keras [3], using the corresponding implementation of GarNet available in Ref. [42].In total, the model has 3,402 trainable parameters (2,976 in the three GarNet layers), whose values are optimized through a supervised training process using the Adam optimizer [43].Input is processed in batches of 64 samples during training.The overall objective function that is minimized in the training is a weighted sum of objective functions for the classification and regression tasks: with β = 0.01.The objective function for classification L class is the binary cross entropy in each batch between the truth labels (electrons are represented by 1 and pions by 0) and the classification output of the model.The objective function for regression L reg is the batch mean of the relative squared error where E pred and E true are the predicted and true energies of the primary particle, respectively.The training is performed on 400,000 training and 100,000 validation samples over a few hundred epochs, with early stopping when the value of the objective function does not improve for ten consecutive epochs.Keeping the full training dataset on RAM and using two NVIDIA GeForce RTX 2080 Ti GPUs in parallel, each epoch takes roughly 30 seconds to process.
Additionally, we prepare a model in which the encoders and decoders of the GarNet layers are quantized as ternary networks using QKeras [9,10], which performs quantization-aware training with the straight-through estimator by quantizing the layers during a forward pass but not a backward pass [44][45][46]10].In the following, this model is referred to as the quantized model, and the original model as the continuous model.The quantized model is trained with the same objective function and training hyperparameters as the continuous model.
To evaluate the inference performance of the trained models, reference algorithms are defined separately for the classification and regression subtasks.The reference algorithm for classification (cut-based classification) computes the energy-weighted mean z and standard deviation σ z of the z coordinates of the hits, where i is the index of hits in the cluster and z i and h i are the z coordinate and energy of the ith hit.The cluster is labeled as an electron if z < zcut and σ z < σ cut z , where zcut and σ cut z are predefined thresholds.Pions, and hadrons in general, tend to penetrate deeper in an absorbing detector and create showers of secondary particles with a larger transverse size than electrons and photons.For regression, the reference algorithm (weight-based regression) predicts the energy of the primary particle through a formula where l(i) is the detector z layer of hit i. Parameters {w l , b l } (l = 1, . . ., 50) are determined by minimizing L reg over the training dataset using E ref pred as the predicted energy.Particle identification based on the energy deposition profile of the cluster and energy estimation based on weighted sum of hit energies are both common strategies in the conventional, non-ML-based event reconstruction approaches.

Training result
Performance of the trained continuous and quantized models, evaluated using the validation sample, are shown in Fig. 4. For each ML model, the inference results based on the original Keras model and the HLS model, converted using hls4ml, are shown.The HLS model provides a realistic emulation of the synthesized FPGA firmware.
The classification performance is given in terms of receiver operating characteristic (ROC) curves that trace the electron identification efficiency (true positive fraction) and pion rejection efficiency (true negative fraction) for different thresholds of the classifiers.The two GarNet-based models perform similarly and better than the cut-based reference in terms of the electron identification efficiency for a given pion rejection efficiency.A detailed comparison of the four sets of results from the GarNet-based models in the inset reveals that the continuous model performs slightly better than the quantized model, and that the difference between the Keras and HLS implementations is smaller for the quantized model.The differences between the Keras and HLS implementations are due to the numerical precision in the computation.While the former represents all fractional numbers in 32-bit floating-point numbers, the latter employs fixed-point numbers with bit widths of at most 18.Consequently, for the quantized model, where the encoder and decoder of the GarNet layers employ integer weights for inference, the difference between the two implementations are smaller.
For both subtasks, the GarNet-based models generally outperform the reference algorithms.The reference algorithm has narrower spread of the response in some energy bins for the regression subtask.However, it is important to note that the weights and biases appearing in Eq. ( 14) are optimized for a specific pileup profile, while in a real particle collider environment, pileup flux changes dynamically even on the timescale of a few hours.In contrast, algorithms based on inference of properties of individual hits, such as the GarNet-based models presented in this study, are expected to be able to identify hits due to pileup even under different pileup environments and thus to have a stable inference performance with respect to change in pileup flux.Since a detailed evaluation of application-specific performance of GarNet is not within the scope of this work, we leave this and other possible improvements to the model architecture and training to future studies.
To verify that GarNet can infer relations between individual vertices without edges E in the input, the following test is performed.Using the two events shown in Fig. 3, the energy of each hit in the clusters is increased one at a time by 10%, and the inference with the continuous model is performed for each perturbed event.If the model has learned to perfectly distinguish the primary particle from pileup at the vertex level, a small change in the energy of a hit from pileup should result in no change in the predicted particle energy.In Fig. 3b, each hit in the cluster is colored by the ratio of the change of predicted particle energy and the amount of perturbation (∆E pred /∆h).While some hits in Fig. 3a with f prim = 0 appear with ∆E pred /∆h > 0, a general correspondence between f prim and ∆E pred /∆h is observed.The occurrence of ∆E pred /∆h > 1 is expected, given the extrapolation required to predict the full particle energy from the energy of the hits included in the cluster.With this test, we are able to probe how the GarNet-based model is learning the structure of the graph.

Model synthesis and performance
The latency, II, and resource usage of the FPGA firmware synthesized from the HLS implementations are summarized in Comparing the continuous and quantized models with V max = 128, the former has a longer latency and II and consumes substantially more DSPs.On the other hand, the quantized model uses more LUTs, mainly for the multiplications in the GarNet encoders and decoders, as discussed in Section 4.However, it is known that the expected LUT usage tend to be overestimated in Vivado HLS, while the expected DSP usage tends to be accurate [8,2].The DSP usage of 3.1 × 10 3 for the continuous model is well within the limit of the target device, but is more than what is available on a single die slice (2.8 × 10 3 ) [48].The quantized model fits in one slice in all metrics.Given the small difference in the inference performance between the two models, it is clear that the quantized model is advantageous for this specific case study.
The latency of the synthesized quantized model at 148 clock periods, corresponding to 740 ns, satisfies the LHC L1T requirement of O(1) µs execution.However, the II of 50 clock periods (250 ns) implies that the logic must be time-multiplexed tenfold to be able to process a single cluster per LHC beam crossing period of 25 ns.With O(100) or more clusters expected per beam crossing in the collision environment of HL-LHC, the throughput of the synthesized firmware is therefore inadequate for a reasonably sized L1T calorimeter system with O(100) FPGAs, and requires down-scoping or implementation improvements.
The simplest down-scoping measure is to reduce the size of the input.This is effective because the most prominent factor driving both the latency and the II of the firmware is R reuse (see Eq. ( 10)), which in turn is determined by V max to be able to fit the logic in a single chip.To test how short the II can be made while retaining a reasonable inference performance, additional models with V max = 64, 32, and 16 are trained and synthesized into FPGA firmware.Clusters with more hits than V max are truncated by discarding the lowest energy hits.The fraction of truncated clusters for the three V max values are 27%, 85%, and 99%, respectively.
The results of synthesis of the additional models are given in the last three rows of Table 1.The values of FPGA resource usage metrics are similar in all quantized models because the ratio V max /R reuse is kept at 4.
The area under the ROC curve (AUC) and the root-mean-square (RMS) of the response are considered as metrics for the inference performance.Only a modest degradation of performance is observed by truncating the clusters to V max = 64, while the II is reduced by 16 clocks as a direct result of the reduction of R reuse by the same amount.This working point might thus represent a reasonable compromise between the inference performance and throughput.Further cluster truncation results in considerable loss of inference accuracy.It is also clear that reduction of R reuse has a diminishing return in terms of shorter II, and improvements to other parts of the algorithm are necessary to further reduce the II.

Conclusions
In this paper, we presented an implementation of a graph neural network algorithm as FPGA firmware with O(1) µs execution time.General considerations and challenges in implementing graph neural networks for real-time trigger systems at particle collider experiments are outlined, along with how algorithms such as GarNet address these issues.We then described the simplified version of GarNet, which is now available as a general-purpose graph network layer in the hls4ml library.An example use case of a machine learning based on the simplified version of GarNet, applied to data from a simulation of a small imaging calorimeter, is presented.The model is able to learn to predict the identity and the energy of the particles detected at the calorimeter with high accuracy, while its firmware implementation executes in 740 ns and fits easily in a commercially available FPGA.Although the throughput of the firmware is not sufficient to make the model readily deployable in a submicrosecond, real-time collider trigger system, its variants with reduced input size are shown to have higher throughput with reasonable inference performance.These results demonstrate that fast inference of graph neural networks in FPGAs is possible, and with hls4ml, various graph-based machine learning architectures can be automatically translated into firmware.

Figure 1 :
Figure 1: Processing flow of the modified GarNet algorithm: (a) The input features (g j v ) of each vertex are processed by a linear network, that returns a new set of features (f i v ) and its distance from the S aggregators (d av ).(b) A graph is built in the learned space, using the d av distances.(c) A message is gathered by each aggregator, as a weighted sum across the vertices of f i v , with W av = exp(−d 2 av ) as weights.(d) A message is from each aggregator ( f i av ) is passed back to each vertex, with the same W av weight.(e) The aggregated outputs of each vertex are given as input to a neural network, which returns the learned representation.

Figure 2 :
Figure 2: Schematics of the high-granularity and low-granularity regions of the electromagnetic (left) and hadron (right) layers.

Electron 49. 2 (Figure 3 :
Figure 3: Examples of electron (left) and pion (right) events.Values in parentheses in the graph titles are the respective energy depositions contained in the cluster around the seed hit.Points represent hits in the detector, with their coordinates at the center of the corresponding detector cells and the size of the markers proportional to the square root of the hit energy.Opaque points are within the cluster, while the translucent ones are not.In (a), the point color scale from blue to red corresponds to the primary fraction (see Section 5.1 for definition).In (b), the color scale from blue to green corresponds to ∆E pred /∆h, which is an indication of the importance the neural network model places to individual hits for energy regression.See Section 5.3 for details.

Figure
Figure Classification (left) and regression (right) inference performance of the continuous and quantized GarNet-based models and the reference algorithms.Results from the Keras and HLS implementations are shown for the GarNet-based models.The classification performance is quantified with a ROC curve of electron identification efficiency versus pion rejection efficiency.The inset in the left graph shows a close-up view of the efficiency range 0.90-0.96for both axes.The regression performance is quantified as the response (E pred /E true ) in 10 GeV bins of E true .The horizontal line in the box corresponds to the median of the distribution, the top and bottom of the box to the upper and lower quartiles, and the upper and lower ends of the whiskers to the 95th and 5th percentiles.

Table 1 :
[48]e. 1. Vitis Core Development Kit 2019.2[47]isused for synthesis, with a Xilinx Kintex UltraScale FPGA (part number xcku115-flvb2104-2-i) as the target device and a clock frequency of 200 MHz.The reported resource usage numbers reflect the synthesis estimates from Vivado HLS.The latency and II reported here are the maximum values for samples with full V max vertices; the actual HLS implementation allows early termination of the serial reuse of the vertex-processing logic unit for samples with fewer vertices.The area under the ROC curve (AUC) and overall response root mean square (RMS) are used to summarize the performance.Summary of the latency, II, FPGA resource usage metrics, and inference accuracy metrics of the synthesized firmware.The reported resource usage numbers reflect the synthesis estimates from Vivado HLS.The target FPGA is a Xilinx Kintex UltraScale FPGA (part number xcku115-flvb2104-2-i), which has 5,520 DSPs, 663,360 LUTs, 1,326,720 FFs, and 77.8 Mb of BRAM[48].The utilized percentage of the targeted FPGA resources are denoted in the square brackets.