Integration of neuromorphic AI in event-driven distributed digitized systems: Concepts and research directions

Increasing complexity and data-generation rates in cyber-physical systems and the industrial Internet of things are calling for a corresponding increase in AI capabilities at the resource-constrained edges of the Internet. Meanwhile, the resource requirements of digital computing and deep learning are growing exponentially, in an unsustainable manner. One possible way to bridge this gap is the adoption of resource-efficient brain-inspired “neuromorphic” processing and sensing devices, which use event-driven, asynchronous, dynamic neurosynaptic elements with colocated memory for distributed processing and machine learning. However, since neuromorphic systems are fundamentally different from conventional von Neumann computers and clock-driven sensor systems, several challenges are posed to large-scale adoption and integration of neuromorphic devices into the existing distributed digital–computational infrastructure. Here, we describe the current landscape of neuromorphic computing, focusing on characteristics that pose integration challenges. Based on this analysis, we propose a microservice-based conceptual framework for neuromorphic systems integration, consisting of a neuromorphic-system proxy, which would provide virtualization and communication capabilities required in distributed systems of systems, in combination with a declarative programming approach offering engineering-process abstraction. We also present concepts that could serve as a basis for the realization of this framework, and identify directions for further research required to enable large-scale system integration of neuromorphic devices.


Introduction
The accelerating developments of digital computing technology and deep learningbased AI are leading towards technological, environmental, and economic impasses (Thompson et al., 2021;Mehonic and Kenyon, 2022).With the end of Dennard transistor-scaling (Davari et al., 1995) and the anticipated end of Moore's law (Waldrop, 2016;Shalf, 2020;Leiserson et al., 2020), conventional digital von Neumann computers and clock-driven sensor systems face considerable hurdles regarding bandwidth and computational efficiency.For example, the gap between the computational requirements for training state-ofthe-art deep learning models and the capacity of the underlying hardware has grown exponentially during the last decade (Mehonic and Kenyon, 2022).Meanwhile, in stark contrast, distributed digitized systems-ever-growing in size and complexity-require increasing computational efficiency for AI applications at the resource-constrained edge of the internet (Zhou et al., 2019;Ye et al., 2021), where sensors are generating increasingly unmanageable amounts of data.
One approach to addressing this lack of computational capacity and efficiency is offered by neuromorphic engineering (Mead, 1990(Mead, , 2020)).There, inspiration is drawn from the most efficient information processing systems known to humanity-brains-for the design of hardware systems for sensing (Tayarani-Najaran and Schmuker, 2021) and processing (Zhang et al., 2020a;Basu et al., 2022) that have the potential to drive the next wave of computational technology and artificial intelligence (Christensen et al., 2022;Frenkel et al., 2021;Shrestha et al., 2022).Neuromorphic-that is, brain-like-computing systems imitate the brain at the level of organizational principles (Indiveri and Liu, 2015), and often also at the level of device physics by leveraging nonlinear phenomena in semiconductors (Chicca et al., 2014;Rubino et al., 2021) and other nanoscale devices (Zidan et al., 2018;Marković et al., 2020) for non-digital computation.The idea of using nonlinear physical phenomena for non-digital computing has been explored for decades.Different choices of underlying mathematical models lead to different definitions of what the concept of "computation" entails (Jaeger, 2021), and likely also influences the set of possible emergent innovations.
Here, we define neuromorphic computing (NC) systems as non-von Neumann information-processing systems, the structure and function of which either emulate or simulate the neuronal dynamics of brains-especially of somas, but sometimes also synapses, dendrites, and axons-typically in the form of spiking neural networks (SNNs) (Maass, 1997;Nunes et al., 2022;Wang et al., 2022).NC systems open up new algorithmic spaces-through asynchronous massive parallelism, sparse, event-driven activity, and co-location of memory and processing (Indiveri and Liu, 2015)-and, in terms of energy-usage and latency, offer superior solutions to a range of brain-like computational problems (Davies et al., 2021;Yin et al., 2021;Stöckl and Maass, 2021;Göltz et al., 2021;Rao et al., 2022).Furthermore, beyond cognitive applications, SNNs and NC systems have also demonstrated potential for applications such as graph algorithms, constrained optimization, random walks, partial-differential-equation solving, signal processing, and algorithm composition (Aimone et al., 2022).Consequently, there is a growing interest for NC technology within application domains such as automotive technology, digitized industrial production and monitoring, mobile devices, robotics, biosensing (such as brain-machine interfaces and wearables), prosthetics, telecommunications-network (5G/6G) optimization, and space technology.
One challenge facing neuromorphic technology is that of integrating emerging diverse hardware systems, such as neuromorphic processors and quantum computers, into a common computational environment (Vetter et al., 2018).Such hardware systems are-due to performance constraints of existing computational hardware in, for instance, energy usage or processing speed-likely to be increasingly included in computational ecosystems to facilitate or accelerate particular types of computation (Shalf, 2020;Leiserson et al., 2020;Hamilton et al., 2020).Fundamental trends in computer-architecture development indicate that nearly all aspects of future high-performance computing architectures will have substantially higher numbers of diverse and unconventional components than past architectures (Becker et al., 2022), leading toward a period of "extreme heterogeneity".Consequently, neuromorphic processors are, in many future use-cases, likely to be part of a broader, heterogeneous computational environment, rather than to be operated in isolation.Thus, there is a need for programming models and abstractions, as well as interparadigmatic communication principles and data models, that enable effective integration of neuromorphic hardware into large-scale systems of systems.
Here, we address the problem of large-scale adoption and integration of NC systems into the present digital-computational infrastructure.We frame this problem in terms of the following main challenges: 1. Communication: How to transcode information between neuromorphic and digital systems?
2. Virtualization: How to interface neuromorphic devices and services in distributed digitized systems?
3. Programming: How to efficiently program hybrid neuromorphic-digital systems?
4. Testing and validation: How to reliably train and test the functionality of such hybrid systems?
We outline the current landscape of NC technology from the perspective of system integration-describing the most significant qualities of NC systems as compared to the fundamentally different paradigm of conventional digital computing (DC)1 .Based on this description, we propose a microservice-based framework for integration of NC systems.The framework consists of a neuromorphicsystem proxy, which provides virtualization and communication capabilities required in a distributed setting, in combination with a declarative programming approach offering engineering-process abstraction.We present established concepts for programming, representation, and communication in distributed systems that could serve as a basis for the realization of this framework, and identify directions for further research required to enable such integration.

Neuromorphic Systems
The field of neuromorphic engineering dates back to the late 1980s (Mead, 1990(Mead, , 2020)), and originally dealt with the creation and use of sensing and processing systems that imitate the brain at the level of structure and device physics.Today, the term "neuromorphic" has broadened, and "neuromorphic processors" typically refer to hardware systems of different architectures that are specialized for running SNNs.Neuromorphic hardware architectures thus range from electronic emulation with analog circuitry or novel electronic devices to digital systems specialized for massively parallel differential-equation solving for spiking neuron models.However, as spiking neural networks (Maass, 1997;Nunes et al., 2022;Wang et al., 2022) are inherently event-driven, asynchronous, timedependent, and highly parallel, all neuromorphic processors, by consequence, differ significantly from von Neumann computers, as summarized in Table 1.In general, analog-based NC systems are more power-efficient than fully digital ones (Basu et al., 2022), by leveraging device physics for real-time neurosynaptic emulation, while digital systems come with the versatility of being fully configurable by logical programming.Due to the need for power-efficient sustainable technologies for AI workloads, neuromorphic solutions can come to constitute up to 20 % of AI computing and sensing by 2035 according to some estimates.

States in Neuromorphic Systems
The state of a neuromorphic processor at any given moment is defined by the properties of the SNN that that processor has been configured to implement and the event-based neurosynaptic activity of that SNN, which is largely reactive in response to input signals.These properties can roughly be arranged into the following categories: • Structural properties: E.g., network topology, synaptic weights, time constants, axonal delays, and neuronal thresholds • Transient properties: E.g., neuronal potentials, synaptic currents, and spiking activity Out of these properties, it is, in general, the structural ones that are subject to direct manipulation by external configuration, optimization, and learning algorithms-while the transient state rather arises in reaction to presented input signals in concert with the structural state.However, a clear line cannot simply be drawn between structural and transient properties, as the biological timescales of synaptic plasticity phenomena-that is, the strengthening and weakening of neuronal connections-range from single milliseconds, in the case of short-term plasticity, to the whole lifetime of an organism, in the case of structural plasticity (Jaeger et al., 2021).Many neuromorphic systems do, however, exclude on-chip implementation of synaptic plasticity due to the complexity and resource cost (Frenkel et al., 2021), in which case, structural parameters are more clearly distinguished as subject to configuration by an external system.There are many learning rules in use due to the knowledge gap associated with long-term plasticity and task-dependent requirements.Therefore, some chips implement flexible coprocessors for learning.

Information in Neuromorphic Systems
Von Neumann computers represent information in clock-driven discrete states, the resolutions of which are determined by the number of bits used for representing binarily encoded variables.NC systems, on the other hand, represent information using unary (one-or-nothing), uniform interneuronal spike-events.These carry explicit information about both space and time in their source of origin and time of arrival-potentially carrying arbitrary temporal precision in the interspike intervals (Thorpe et al., 2001).This form of representation arises already in neuromorphic sensors, as they rely on level-crossing Lebesgue sampling (Astrom and Bernhardsson, 2002) for event-driven generation of sense data, or, alternatively, in delta-modulated spike-data conversion of conventionally sampled signals (Corradi and Indiveri, 2015).Spike-based representations thus allow sparse, event-driven processing with consequently low power-usageespecially for real-time applications in which both sensing and processing are spike-based and event-driven (Liu et al., 2019).

Neural Code
There are several ways in which spatiotemporal combinations of uniform spikes could theoretically be used to encode information.This list is not exhaustive, and the manner in which information is actually represented in the brain is still largely an open question (Brette, 2015;Zenke et al., 2021).It is, for instance, possible that the asynchronous dynamics of SNNs give rise to emergent representations that consist of combinations of coding schemes such as those listed above.Nevertheless, the listed coding schemes-along with their estimated capacity for information transmission, see Table 2-provide an overview of the qualitatively distinct ways in which information can be represented in SNNs, and the quantitative relations between these.The estimates presented in Table 2 were made for a population of N = 10 neurons, during a temporal interval of t = 10 ms, and with a temporal resolution of 1 ms.As the estimations were made for scenarios of rapid processing, they were also limited to a maximum of one spike per neuron.During these conditions, rate code is theoretically equivalent to count code, as rate code would require more than a single spike per neuron-and thus a longer durationto represent more information.While these estimates were made for limited conditions, it illustrates how the information transmission per spike would be maximal if spikes carried information in their precise timings.However, a spiketiming-based coding scheme may demand a high level of complexity and temporal precision in the decoding mechanisms.It is important to note here that the data generated by neuromorphic, event-driven sensors is, at least in part, intrinsically spike-timing coded due to the event-driven, sparse activations that underlie the low-power, low-latency operation of such sensors (Liu et al., 2019).Therefore, in order to gain analogous benefits in the subsequent processing, it is likely necessary to incorporate some degree of spike-timing code in neuromorphic processing systems.However, the choice of coding scheme is likely to be task-specific and subject to optimization (Guo et al., 2021a;Schuman et al., 2022a).

Representation Space
Figure 1 illustrates a conceptualization of the space of possible information representations in NC systems.As discussed previously, the NC hardware substrate may, to varying degrees, rely on digital or analog circuitry, and the temporal encoding may, again to varying degrees, asynchronously rely on the precise timings of spikes in qualitatively different coding schemes, see Table 2.The spatial dimension of a neural network is generally, by default, used for distributed representations, in which the representations of different concepts are distributed over several of the same neurons and synapses of the network.Conversely, localist representations, which are studied in conventional, logical neurosymbolic computation (Garcez and Lamb, 2020;Dold et al., 2022), represent different concepts with single, discrete identifiers, such as single neurons or bits.An example of a completely localist representation could, for instance, be a single "cat neuron", which, when activated, signifies the inferred presence of a cat in the sense data.As in the case of the two other dimensions of Figure 1, it also depicts a possible spectrum of spatial encoding, in the hypothetical extremes of which, representations are either distributed across a whole neural network or localized to single neurons, respectively.

Interfaces and Semantics
For the activities of a neural network to ultimately have discernible implications for action in an external system, symbolic representations are needed-such as, for example: Pattern A implies Action X and Pattern B implies Action Y. "Action" is here meant in a broad sense, ranging from motor-output actuation to read-out signaling.In the kind of hybrid digital-neuromorphic systems discussed here, such symbolic representations would, depending on the application, not necessarily have to be generated within the confines of the NC system itself.However, at every system interface, there needs to be an operational seman- tic-a system of interpretation-for transmitted data (Nilsson, 2022;Nilsson and Sandin, 2018).Such interfaces could, for instance, be implemented in the form of microservices, which is further discussed in Section 3.
An interesting concept for interface semantics for NC systems is that of embeddings-a widely used form of representation in neural networks and machine learning that are essential to many state-of-the-art models.Embeddings are succinct, intermediate representations of associations within usually largescale datasets.For example, embeddings are used in deep learning architectures where concept representations are quickly learned from combinations of sensor data, descriptive sentences, and higher-level knowledge representations (Mei et al., 2022).The learned concepts can be used in downstream applications, such as answering questions by reasoning about unseen sensor inputs.Typically, embeddings emerge as nested function values in a deep learning model, The parameters θ k are optimized to provide the outputs y i that are expected for the corresponding inputs x i with minimum error according to some metric.The embeddings, f k (. ..), of a model that is optimized on a sufficiently large and varied dataset are often useful for other similar datasets, as these embeddings can be used as input to optimize a new output function with less data and computational resources compared to full model retraining.Embedding are used in various models-autoencoders, transcoders, multimodal predictive models, etc.-and are natural components to be used and supported by the envisioned hybrid neuromorphic-digital computing systems and programming models.This entails interesting opportunities and challenges related to the optimization and interoperability of embeddings realized in different hardware and all the related symbols appearing in software and data across an orchestrated system (Nilsson et al., 2020(Nilsson et al., , 2019)).

Programming of Neuromorphic Systems
In contrast to von Neumann computers, the actions of NC systems are not primarily dictated by sequences of explicit logical instructions.Rather, the analog to DC programs in NC systems can be considered as being implicitly defined by the structural properties-as defined in Section 2.1-of the implemented SNNs, as these determine the input-output signal transformations that are performed by an NC system.Furthermore, while von Neumann computers encode information in observable, static, discrete states, NC systems may inhabit unobservable, dynamic, continuous states, and they receive inputs in the form of uniform spike-events, in which information is encoded in the physical time of arrival and source of origin.Thus, the use of NC systems is fundamentally different from that of von Neumann computers, and-in order to fulfil the potential for efficiency-may require a significant change of perspective in the view of programming (Schuman et al., 2022b) and computation (Jaeger et al., 2021), informed by neuroscience and dynamical-systems theory.
Some attempts to develop programming abstractions for NC systems include the Neural Engineering Framework (NEF) (Stewart, 2012) and Dynamic Neural Fields (DNFs) (Sandamirskaya, 2014).However, these are often fairly limited to specific use-cases-biologically plausible neural models for the NEF, and models of embodied cognition for DNFs.Thus, there is a gap in defining more generally useful programming abstractions for neuromorphic computing systems (Schuman et al., 2022b), including virtualization concepts required for seamless edge-to-cloud integration.

Software
As the landscape of neuromorphic computing is made up of different hardware and software architectures that are developed by different groups, it is characterized by a fragmented and noncomposable array of programming models and frameworks.Programming frameworks for SNNs and neuromorphic hardware generally fall into one of the following categories: • Optimization tools: SNN-parameter optimization tools, usually based on supervised deep learning, such as SNN Conversion Toolbox (Rueckauer et al., 2017), SLAYER (Shrestha and Orchard, 2018), Whetstone (Severa et al., 2019), EONS (Schuman et al., 2020), and EXODUS (Bauer et al., 2022) • Simulators: SNN simulators with low-level APIs for conventional computers, such as NEST (Gewaltig and Diesmann, 2007), Brian 2 (Stimberg et al., 2019), Nengo (Bekolay et al., 2014) GeNN (Yavuz et al., 2016), BindsNET (Hazan et al., 2018), SynSense Rockpool and SINABS, and Norse • Hardware interfaces: Low-level interfaces and runtime frameworks for configuration of neuromorphic hardware, such as PyNN (Davison et al., 2009), Fugu (Aimone et al., 2019), SynSense Samna, BrainScaleS OS (Müller et al., 2022), and Intel Lava While several frameworks exist, none of them have so far provided programming abstractions that are composable and span the diverse range of algorithms and methods within NC (Davies et al., 2021;Jaeger, 2021).Furthermore, NC hardware typically have limitations in terms of connectivity, plasticity, and neurosynaptic configurations.Thus, transforming a well-defined SNN specification or general program into a corresponding hardware configuration is challenging and further complicated by imperfections of mixed-signal circuits and limited resources, such as bandwidth, which generate differences from the specified target.Even between generations of the same hardware architecture, such as Spikey and BrainScaleS-1, SpiNNaker-1 and -2, or Loihi-1 and -2, it is often difficult to build upon existing software (Müller et al., 2022).As of today, two candidate models for general-purpose and platform-agnostic NC configuration are PyNN and Lava.
PyNN PyNN (Davison et al., 2009) is a simulator-agnostic language for describing SNN models at the level of network topology, neurosynaptic parameters, plasticity rules, input stimuli, and recording of states, while still allowing access to the details of individual neurons and synapses.PyNN also provides a set of commonly used connectivity algorithms (e.g., all-to-all, random, distancedependent, small-world) but makes it easy to provide custom connectivity in a simulator-independent way.PyNN provides a library of standard models of neurons, synapses, and synaptic plasticity, which have been verified to work the same on the different supported simulators.As of today, common SNN simulators and some hardware emulators support PyNN, which is also the entrypoint to the BrainScaleS and SpiNNaker systems that implement PyNN as an experiment-description language.
Lava Lava2 by Intel is an upcoming open-source software framework for neuroinspired applications and their deployment on neuromorphic hardware, and constitutes an attempt to move towards convergence in the domain of NC software.Lava is designed to be hardware-agnostic, modular, composable, and extensible-allowing developers to construct abstraction layers to meet their needs, and to broaden the accessibility of programming NC systems.The fundamental building-block in Lava, for algorithms and applications alike, are socalled processes-stateful objects with internal variables and input and output ports for message-based communication via channels.This architecture is inspired by the communicating sequential processes (CSP) formal language for asynchronous, parallel systems, which belongs to the family of formal models for concurrent systems known as process calculus, further described in Section 3.2.Every entity in Lava-including neurons, neural networks, conventional computer programs, interfaces to sensors and actuators, and bridges to other software frameworks-is a process with its own memory and message-based communication with its environment.Thus, Lava processes are recursive programming abstractions, from which, modular, large-scale parallel applications can be built.

Challenges to Adoption and Integration
The following are some of the major challenges posed to the adoption of NC technology and its integration into the present computational environment.

Programming Abstractions and Frameworks
As discussed in Section 2.3, there is a lack of common programming abstractions, models, and frameworks for different NC designs (Davies et al., 2021;Schuman et al., 2022b).Intel's launch of the Lava software framework is an attempt at closing this gap, but, being so recent, the degree to which Lava will aid in achieving NC software convergence remains to be proven.Furthermore, there may be a need for further developments of generalized system hierarchies, concepts of completeness (Zhang et al., 2020b), and analytical frameworks (Guo et al., 2021b) for NC systems and other unconventional computing concepts (Jaeger, 2021), to facilitate hardware-software compatibility, programming flexibility, and development productivity.

Interdevice Communication
Most neuromorphic systems-sensors and processors alike-implement an addressevent representation (AER) spike-event communication protocol (Mortara and Vittoz, 1994;Boahen, 2000), in which events, signified by the address of their source of origin, such as a neuron or pixel, are asynchronously generated and transmitted in real-time along the connections of neural networks.However, although it is standard practice to implement some AER protocol, there are slight differences in the implementations between the different neuromorphic sensory and processing devices that currently exist (Basu et al., 2022), as these are developed by different groups.This discrepancy between different neuromorphic devices impairs their interoperability, as well as standardization of NC-DC communication, and thus poses a challenge to the integration of neuromorphic systems into the broader computational environment.

Reliance on Host-Computers
Currently, the use of NC systems relies heavily on conventional host-computers for software deployment and, often, for communication with the environment via sensors and actuators.This reliance on a host-machine-which performs preparation and deployment of the NC model and pre-and post-processing of spike-data-can impact the resource requirements for running the neuromorphic system to such an extent that the performance benefits of using such specialized hardware are lost (Diamond et al., 2016).Thus, there is a need to optimize the host-device communication architecture with regard to scalability, throughput, and latency, as well as to design and implement SNNs and NC systems in a way that minimizes the need for host-device communication in the first place.

Integration Framework
In this section, we outline a framework for the integration of NC systems into the existing environment of distributed DC systems.The framework, see Figure 2, consists of NC-DC abstraction layers and communication channels that we envision to be necessary for the NC-system integration.The framework is aligned with the paradigm of microservices (Larrucea et al., 2018)-small, single-responsibility applications, inspired by service-oriented computing, that can be deployed, scaled, and tested independently-and adapted to the neuromorphic context so that there is a separation between the digital and neuromorphic parts.Specifically, the digital abstraction is provided by a microservice instance, called a neuromorphic-system proxy (NSP), illustrated in the middle box of Figure 2. It represents an NC system, see the bottom box in Figure 2.
The role of the NSP is to provide virtualization and data mapping between the neuromorphic and digital domains, and thereby providing effective and efficient interfacing between the digital and neuromorphic domains, including for example availability and security.The use of microservice architecture provides characteristics such as late binding, loose coupling, and discovery to the framework.In addition to these characteristics, a major benefit of the use of microservices in the NSP is service longevity.Service longevity implies that services are available over time.In contrast to the NC systems that provide transient data in real-time reaction to events, the NSP stores the last data communicated from the NC system and exposes it via the available microservices.
The NSP is software-defined and typically built on a library of base-software images (e.g., docker images) stored in a repository to provide easy instantiation and replication at the edge or cloud as desired.Application-specific programs are added as delta images.Although the figure shows a one-to-one mapping between an NC system and an NSP, this model can be scaled out to arbitrary numbers of NC systems and NC system proxies.
The NSP is here described by first covering the essential interfaces, i.e., the services provided or consumed and then, in Section 3.1, some challenges and approaches for data modeling and mapping are described.The interfaces to an NSP are represented by arrows in Figure 2 as follows: Bottom part (related to NC-system interfaces): • SNN config.: Drives the SNN configuration as needed for specific use cases.This is slowly changing data.Initially, we envision the configuration to be primarily static, while it may be somewhat dynamic to allow some degree of digital reconfiguration as part of normal runtime operation.
• Input spike-data: The input to the SNN, in which most spikes typically come from other neuromorphic systems such as sensors.In addition, stimuli from other systems can also enter through the digital interfaces of the NSP.All incoming spikes are processed by the NC system with the objective to produce an output that aligns with the requirements expressed by the declarative definitions and the validation protocol.
• Output spike-data: Involves information from selected spike-sources, such as read-out neurons (as defined by the SNN config.).Typically, abstractions are made such that only spikes carrying high-level information are communicated to the NSP.This serves to reduce the load of events coming to the NSP, although the transmitted data is still highly dynamic.
Top part (related to digital interfaces): • Objectives: Interface for adding declarative statements that will be translated to an SNN config.This is complementary to code that is part of the NSP image.The need depends on the dynamics of reconfiguration required, and using it is therefore optional.
• System state: Interface for retrieving the SNN configuration in terms of structural SNN properties, see Section 2.1.
• Event notification: Interface where events are sent from the NSP.In alignment with the microservice paradigm, these events are typically sent to some PubSub event-handlers using different channels or topic trees to notify subscribers effectively and efficiently.
• Output data: Interface where other systems can request additional data related to the events seen in the event notification or related to any history or state stored in the NSP.This is basically a restful interface, but aiming for a graph query language where there is a well-defined typed data schema and where under-or over-fetching is avoided.
Within the NSP, a virtualized representation of the neuromorphic system (a digital twin proxy) provides key functionality to allow for representations of the NC system to exist on one or more digital systems, maintaining the state of the system.The virtualization component also provides the essential functionality of data selection and storage that makes it possible to aggregate and select which data from the NC system should be exposed to the digital part of the system.This component also provides storage, allowing access to data a posteriori to its creation-either through short-term caching, which allows efficient access for multiple clients, or longer-term storage for historical access to data.
In the subsequent parts of this section, we discuss established computerscientific concepts that could be used to realize different parts of this framework or, if insufficient, that could inform the extension or development of new such concepts.

Communication and Data Models
In order to integrate NC systems with other entities in distributed DC systems, the standardized protocols and paradigms used for information representation and communication in the DC domain need to be interfaced within the NSPs.This is a challenging problem considering the different information representations used in the DC and NC domains, including for example state-based and time-based representations, differences in semantics, and a variety of information encoding and transcoding methods used.Here, we review some well-established DC concepts that share characteristics with the NC domain, and may thus be helpful to address the challenges of data modeling and information translation and representation.

Soft-State Models
The event-based nature of NC systems maps well to the best-effort messagepassing based nature of many current computer networks, such as the Internet.
A key feature of best-effort computer networks to provide a distributed system state across the network, in a scalable manner that is robust to network disruptions and do not require too much communication overhead, is the concept of soft state (Ji et al., 2007).This is used in Internet routing protocols and other networked distributed systems and becomes even more important in more Figure 2: System-integration framework for neuromorphic devices in digital-computational infrastructure.Arrows symbolize information channels.The coloring indicates information being represented and transmitted within the following domains: orange for spike-based neuromorphic sensing and processing, green for a virtual neuromorphic-system proxy, and blue for conventional digital computing.
In a soft-state system, a distributed state is maintained in individual nodes by periodic update messages (or events) that are sent by other nodes in the network to refresh the current state.Such state can for example be used to maintain routes through the network for forwarding data traffic.If no refresh messages have been received within a certain amount of time, it is assumed that the state of the system has changed (e.g., a neighboring node has moved away and can no longer be reached), and thus the local state is removed.Since the underlying network is best-effort and refresh messages can be lost due to contention or interference, the system might wait until more than one expected refresh message has not been received before determining that the local state should be removed.The threshold used to determine a state change depends on the expected characteristics of the network and is a tradeoff between how rapidly the system reacts to state changes and the desired robustness to temporary disruptions.The benefit of utilizing soft state is that there is no risk of stale state information becoming stored indefinitely in parts of the network that cannot be reached by state-removal messages, and the need for such control traffic is removed completely.
The soft-state model is similar to an NC system, where messages (spikeevents) trigger and maintain state in nodes (neurons).Thus, we believe that there are potential benefits from mapping these models in the NC domain to digital integration in terms of data models.

Event Notification
The Internet has enabled the creation of very large distributed systems, commonly with the purpose of sharing information between geographically distributed data producers and consumers.In such systems, the publish-subscribe (PubSub) messaging paradigm, based on event notification and messaging, has received much attention for the loosely coupled form of interaction it provides in large-scale settings (Liu et al., 2003).In this paradigm, subscribers register their interests in a topic or a pattern of events and then asynchronously receive events matching their interest.PubSub provides decoupling in space (interacting subscribing and publishing parties do not need to know each other), time (parties do not need to interact at the same time), and asynchronous communication (subscribers can get notifications immediately when published) (Eugster et al., 2003).
PubSub systems can be categorized into subject-based, aka topic-based, and content-based (Liu et al., 2003).Subject-based PubSub means that a subscription targets a group, channel, or topic, and the user receives all events that are associated with that group.With content-based PubSub systems, subscriptions are instead based on queries or predicates, based on which, the decision of to whom a message is directed is made on a message-by-message basis.The advantage of a content-based system is that the subscriber can be provided with the needed information only and does not need to learn a set of topic names and their content before subscribing.
The performance of a content-based PubSub network is typically challenged by the expensive matching cost of content messages.Hybrid schemes between these types of PubSub systems exist, where subscribers register content-based subscriptions to one or more topics.For example, HYPER minimizes both the number of matches inside the PubSub network and the delay to receive subscribed content (Zhang and Hu, 2005).The hybrid schemes, aka type-based, thus represent the middle-ground between coarse-grained, topic-based systems and fine-grained, content-based systems.Type-based systems gives a coarsegrained structure on events (like in topic-based) on which fine-grained constraints can be expressed over attributes (like in content-based) (Shen, 2010).
PubSub systems based on event notification and messaging have been adopted by the Service-Oriented Architecture (SOA) model (Perrey and Lycett, 2003) (Levina and Stantchev, 2009) and its later incarnation known as the Application Program Interface (API) and microservice paradigm (Di Francesco, 2017).In contrast to traditional SOA, the API and microservice paradigm build on the requirement of that such service must be independently deployable (Xiao et al., 2016).Modern messaging systems are typically based on message-oriented mid-dleware like Apache Kafka, Rabbit MQ, NATS Streaming, Google Pub/Sub, Microsoft Event Hubs and Amazon Kinesis (Eugster et al., 2003).
We believe that the properties provided by PubSub and event notification as used in SOA and microservice-based systems, i.e., space, time and synchronization decoupling, together with the ability to efficiently monitor states and outcome of NC systems, can prove useful to make NC systems interact efficiently with DC systems.For example, event notification can be used to learn about that something has happened in an NC system that motivates further investigation based on more output data requested using a graph query language like GraphQL, and tools for its use like Apollo GraphQL and Graphene Python.

Graph Data and Querying
Graph theory is one of the corner stones of computer science.A simple graph G (Wilson, 2010) consists of a non-empty finite set V (G) of elements called vertices (or nodes), and a finite set E(G) of distinct unordered pairs of distinct elements of V (G) called edges.A generalization of this is the multigraph, which can have multiple edges between the same pairs of vertices.Often, the edges are labeled to represent different kinds of relationships between vertices.Other generalizations of graphs include weighted graphs where the edges have a numeric weight, and directed graphs where each edge has a direction.In graph databases, it is common to represent directed multigraphs, where both vertices and edges are labeled and annotated with additional data such as weights and properties.This representation is general and can be used to model all abovementioned kinds of graphs.We believe that the graph is a strong candidate to model both neuromorphic structures and the structure of abstract data that neuromorphic sensors can produce-at least to a large extent.
Real life graphs can be complex and have large sizes.In general, they can include cycles, unless we have defined the special case of a directed acyclic graph (DAG).To understand properties of graphs, it is common to traverse them to search for paths or data of interest, where traversals must terminate and not loop cycles indefinitely.It should be noted that the general problem of finding an optimal path through a graph is NP hard.There are many alternative traversal algorithms available, where one of the most well known for finding the shortest path between two points in a graph is Dijkstra's algorithm.
To make sense of the graph data, there must be a way to express queries to extract some subset of the graph meeting given properties, such as the special case of finding a shortest path (a totally ordered sub-graph).A response could be more general than that such as a DAG (where a number of directional trees and paths are special cases).It should be noted that the underlying graph in many application cases can be structured as a DAG or special case thereof, meaning that the traversal would be uncomplicated.As mentioned, one candidate for graph queries is GraphQL, where tree structures can effectively be extracted from the underlying general graph by providing query expressions for matching.

Synthesizing a Data Model
One of the key success factors in neuromorphic-digital integration is to find a data model in the NSP that is suitable for both the digital domain and the neuromorphic domain.The objective is to have a data model in the NSP where data can be filled out by the neuromorphic system according to declarative instructions, and then effectively be provided through the digital interfaces.
The definition of the precise data model is highly application-dependent.The objective in our integration framework is that there is a generic foundation providing graph-oriented data modeling, so there is a well-defined syntax to define the exact data model for each application case.This application-specific model is set by the configuration interface of the NSP, and/or by software defining the application-specific NSP.The same configuration information is also used in concert for SNN config.This is to facilitate that the output spike-data from the NC system can be used by the NSP to fill out the data according to the model.At certain moments, the NSP can trigger a digital event (e.g., caused directly by some specific NC output spike or that the data model is filled with data that should result in an event at the digital level).This resembles an abstract thresholding for digital event triggering.Subscribers of such events can react, and when receiving interesting events, they can access more information by placing a graph query to the NSP to fetch desired parts of the graph data.
Many of the challenges in this come from that state models of the two domains are fundamentally different, but they are also complementary: 1.The neuromorphic state is continuously building up and decaying based on stimuli, which gives rise to spikes from some neurons that in turn stimulate other neurons, resulting in some transient state of the whole system.The output spike data from the final layer of the neuromorphic system should be used to populate the data model of the NSP.How this is to be accomplished is a subject for future research.
2. In the digital domain, state is typically not decaying continuously.On the contrary, in the NSP, it is possible to remember and store state over time (although at an abstract level compared to the neuromorphic system).This is a complementary property to the neuromorphic state.However, the NSPs are microservices typically operating close to their NC systems, in an edge cloud where storage space will be limited.To mitigate this, snapshotting can be done only at interesting events, and soft state can optionally be used to forget and clean out state not triggered to be maintained (typically binary, not continuous), and interesting state can be offloaded asynchronously to central clouds.

Declarative Programming
This section discusses declarative programming as a goal-oriented approach to configuring NC systems.Declarative programming is a programming paradigm in which the logic of computations is described without describing their control flow.In a declarative language, what is described is the goal that a given program is intended to achieve (i.e., the desired outcome) rather than an explicit description of the sequence of computational primitives that the program is to carry out.This is a suitable approach for NC systems, since there is no predefined control flow, and no desire to engineer traditional control flows.The promise and rationale of declarative approaches is that the focus is on describing "what", not "how".Advantages include clarity and unambiguity on the objectives, and within those objectives, opens for automation by AI of the dynamic properties and placements, and that correctness can be checked against the declarative objectives using a validation procedure.The declarations are free from imperative details (the "how"), and side effects.
In deep learning, declarative frameworks (Molino and Ré, 2021), such as Ludwig (Molino et al., 2019), aim to facilitate a new generation of systems that are more user-friendly by focusing on defining the data-schema and tasks, rather than low-level neural network information.However, while ANNs and deep learning techniques are partly brain-inspired and offer a valuable starting point for SNN algorithm discovery, they represent only a subsection of the space of spike-based neural computation (Roy et al., 2019) that is available to neuromorphic hardware (Indiveri and Sandamirskaya, 2019) and hybrid SNN-ANN models (Zhao et al., 2022), see Figure 1 and Table 2. NC abstractions need to be co-designed (Schuman et al., 2022b) with the digitization requirements and NC modules to provide a seamless edge-to-cloud integration and overall efficient hybrid AI solutions (Zhao et al., 2022) to the general computational problems defined in the use cases.Furthermore, constraints associated with the interfaces between the declarative programming level and SNN/ANN descriptions based on existing software frameworks (Qu et al., 2022) need to be considered.
Machine learning models, AI objectives, and NC-system configurations are naturally defined in declarative languages (Gould et al., 2022;Schmitt et al., 2017), while the details of "how" are subject to optimization, search, and plasticity.Over the last years, there has generally been a shift in modeling preferences, from a focus on ANN-and SNN-centric modeling towards high-level interfaces supporting modern machine learning methods and tasks.By declarative programming, the desired outcomes can be used to partly autogenerate and configure the NC modules (PyNN, process model, etc.), and to verify the compliance of system results to the desired outcomes.The outcomes would depend both on the correctness of the declarative statements and on the data that is fed to the system.
The compliance of such declarative systems would not be totally deterministic as is typically the case in traditional declarative programs, in which the outcome can be verified logically and deterministically.This poses some new challenges in conjunction with NC systems.The outcomes must be validated statistically, given the datasets.This leads to a need for verification and validation that resembles traditional software testing However, here, in contrast, declarative statements have been used to both autogenerate settings and to verify whether the desired outcome-according to the declarative statements-have been met, with some statistical reliability (e.g., standard deviations) modulo the input data that has been used in the testing.We propose to denote this as a validation procedure, rather than a testing procedure, to emphasize the difference.By that, we also avoid confusing the validation procedure with testing neuromorphic circuits, e.g., to detect manufacturing defects and faulty circuits, and runtime failures (Gebregiorgis and Tahoori, 2019;Hsieh et al., 2021).

Test and Validation Methodology
As mentioned above, we denote the verification, validation, and testing based on declarative statements and data fed to the system as a validation procedure.This procedure aims to assess the statistical reliability of the system in meeting the declarative statements in the context of the structural and transient properties of the said system.As discussed in Section 2.1, the structural properties are subject to direct manipulation by external configuration, optimization, and learning algorithms, which, we assume, are defined using declarative statements.The transient properties, on the other hand, depend on the data presented to the system and its structural properties.Note that the validation procedure is likely to mainly address a higher layer of abstraction than that of the transient SNN properties.In addition, given the dependence on input data, the transient state of a system that can be observed through a validation procedure may exist only in direct connection to when the data is presented to the system, while structural states may be observable at any time.
The validation procedure should focus on the system's capacity to perform the task learned, and as such it can also be useful in training protocols.Given that the inherent states of an NC system are updated at different time-scales, we believe that a validation procedure should aim to examine the system focusing on relevant use-case-specific time-scales.In this context, structural states should be examined to determine whether the NC system demonstrate the desired capacity to converge to a solution or learn a given type of task according to declarative statements, provided that input data contain the needed information.

Process Calculus
Process calculus is a category of computer scientific approaches for the formal modeling of concurrent systems, which may serve as a base for declarative definitions of desired outcomes for networked NC systems.A process calculus provides a framework for high-level description of interaction, communication, and synchronization between independent processes or agents.Communication between processes is accomplished through channels, which can forward events that include some data.Many processes can send and/or receive from channels, so that complex distributed connectivity can be modeled, and there can be many processes on both the sending and receiving end of each channel.Channels are first class objects, i.e., they can be passed around as data on other channels to achieve dynamic connectivity between processes.
In the context of NC systems, there are new challenges to using process calculus related to the non-deterministic stochastic behavior of such systems, and to the fact that system validation depends on both the declarations and the data.Nevertheless, there exists work tying process calculus to spiking neural systems (Ciobanu and Todoran, 2022), and Intel's NC software-framework Lava is inspired by the process calculus communicating sequential processes (CSP), see Section 2.3.1.

Conclusion
We have described the current landscape of NC technology, focusing on aspects expected to become increasingly relevant for large-scale adoption and integration into the present digital-computational infrastructure.Based on this analysis, we have proposed a microservice-based framework for NC-system integration, consisting of a neuromorphic-system proxy providing virtualization and communication capabilities, in combination with a declarative programming approach offering engineering-process abstraction.We have also presented well-known concepts in computer science that could be combined as a basis for the realization of the proposed framework, comprising the key components listed in the text below with some concluding remarks.Table 3 summarizes the alignment of the identified research directions and concepts with the integration challenges presented in Section 1.The three research directions address the following identified knowledge gaps: 1. Neuromorphic-system proxy: A central question for the NSP is how to virtually represent the capabilities and states of physical NC systems, so that reliable microservices can be provided.Such representations need to balance level of detail with computational expense.For instance, a full simulation of the neuronal dynamics and event communication within an NC system may be motivated at times, while not feasible to be constantly maintained.
2. Communication and data models: In the proposed framework, NC-DC communication requires the development of an interparadigmatic data model that allows fetching relevant information from physical NC systems beyond the sparse event notifications and latent dynamic states.This is closely tied to the challenge of developing transcoding principles and operational semantics for interfacing with the spike-based representations of NC systems.Case studies focusing on optimization of spike encoders indicate that the best performing solutions are application-specific, which suggests that generic solutions require a combination of semantic technologies and machine learning-based optimization.

Declarative programming:
A generic NC-programming framework requires appropriate programming abstractions, which enable engineers with commonly available competence to implement machine learning and AI solutions.Declarative programming, focusing on "what" rather than "how", is a natural approach to describe learning objectives and constraints.There may also be a need for further developments of generalized system hierarchies, concepts of completeness, and analytical frameworks to facilitate hardware-software compatibility, programming flexibility, and development productivity.Furthermore, a declarative framework presupposes methods for testing and validation.For NC systems, this testing and validation would not be fully deterministic and logical, but would rather be closely interconnected with the core machine learning methodology and statistical assessment.
Table 3: Overview of research directions and concepts with relations to neuromorphic systems integration challenges.The symbol "⊗" indicates that a concept has been connected to neuromorphic systems in the literature discussed in this article, while "×" indicates that the connection is introduced here.All connections indicated in this table are subjects for further research.The relations between these concepts and conventional digital computing are described in Section 3, but are not illustrated here.

Communication Virtualization Programming Testing and validation
Neuromorphic In conclusion, there is a need for further research on interparadigmatic NC-DC communication models and virtualization to establish the transparency, reliability, and security that is typically required by large-scale distributed computing applications in cyber-physical systems and the industrial Internet.Furthermore, research on NC programming abstractions and related protocols for training, validation, and testing are required to efficiently develop, integrate, and maintain hybrid neuromorphic-digital AI solutions in such large-scale distributed computing systems of systems.

Figure 1 :
Figure1: Space of possible representations in neuromorphic systems.The illustration depicts a continuum, in which gradual, independent changes along each dimension are possible, as there exists many variations and combinations of the mentioned concepts.

Table 1 :
Qualitative differences between von Neumann and neuromorphic computational architectures.

Table 2 :
Thorpe et al. (2001)sion capacity of different neural coding schemes.Adapted fromNilsson (2021).Arranged in descending order of capacity, as theoretically estimated byThorpe et al. (2001).N is the number of neurons, each spiking maximally once.The number of equivalent bits were calculated for the example scenario of N = 10 and a temporal interval of t = 10 ms.