Abstract
Understanding how neural networks process information is a fundamental challenge in neuroscience and artificial intelligence. A pivotal question in this context is how external stimuli, particularly noise, influence the dynamics and information flow within these networks. Traditionally, noise is perceived as a hindrance to information processing, introducing randomness and diminishing the fidelity of neural signals. However, distinguishing noise from structured input uncovers a paradoxical insight: under specific conditions, noise can actually enhance information processing. This intriguing possibility prompts a deeper investigation into the nuanced role of noise within neural networks. In specific motifs of three recurrently connected neurons with probabilistic response, the spontaneous information flux, defined as the mutual information between subsequent states, has been shown to increase by adding ongoing white noise of some optimal strength to each of the neurons. However, the precise conditions for and mechanisms of this phenomenon called ‘recurrence resonance’ (RR) remain largely unexplored. Using Boltzmann machines of different sizes and with various types of weight matrices, we show that RR can generally occur when a system has multiple dynamical attractors, but is trapped in one or a few of them. In probabilistic networks, the phenomenon is bound to a suitable observation time scale, as the system could autonomously access its entire attractor landscape even without the help of external noise, given enough time. Yet, even in large systems, where time scales for observing RR in the full network become too long, the resonance can still be detected in small subsets of neurons. Finally, we show that short noise pulses can be used to transfer recurrent neural networks, both probabilistic and deterministic, between their dynamical attractors. Our results are relevant to the fields of reservoir computing and neuroscience, where controlled noise may turn out a key factor for efficient information processing leading to more robust and adaptable systems.
1 Introduction
Artificial neural networks are a cornerstone of many contemporary machine learning methods, especially in deep learning (). Over the past decades, these systems have found extensive applications in both industrial and scientific domains (). Typically, neural networks in machine learning are organized in layered structures, where information flows unidirectionally from the input layer to the output layer. In contrast, Recurrent Neural Networks (RNNs) incorporate feedback loops within their neuronal connections, allowing information to continuously circulate within the system (). Consequently, RNNs function as autonomous dynamical systems with ongoing neural activity even in the absence of external input, and they are recognized as ‘universal approximators’ (). These unique characteristics have spurred a significant increase in research on artificial RNNs, leading to both advancements and intriguing unresolved issues: Thanks to their recurrent connectivity, RNNs are particularly well-suited for processing time series data () and for storing sequential inputs over time (Schuecker et al., 2018; ; ; Wallace et al., 2013; ). For example, RNNs have been shown to learn robust representations by dynamically balancing compression and expansion (). Specifically, a dynamic state known as the ‘edge of chaos’, situated at the transition between periodic and chaotic behavior (), has been extensively investigated and identified as crucial for computation (Wang et al., 2011; ; ; Natschläger et al., 2005; ; ; Schrauwen et al., 2009; Toyoizumi and Abbott, 2011; ; Solé and Miramontes, 1995) and short-term memory (; ). Moreover, several studies focus on controlling the dynamics of RNNs (Rajan et al., 2010; ; ), particularly through the influence of external or internal noise (Molgedey et al., 1992; ; ; ; ). RNNs are also proposed as versatile tools in neuroscience research (). Notably, very sparse RNNs, similar to those found in the human brain (Song et al., 2005), exhibit remarkable properties such as superior information storage capacities () (Narang et al., 2017; ; ).
In our previous research, we systematically analyzed the relation between network structure and dynamical properties in recurrent three-neuron motifs (). We also demonstrated how statistical parameters of the weight matrix can be used to control the dynamics in large RNNs (; ). Another focus of our research are noise-induced resonance phenomena (; Schilling et al., 2022; ; Schilling et al., 2021; Schilling et al., 2023). In particular, we discovered that in specific recurrent motifs of three probabilistic neurons, connected with ternary weights, the mutual information between subsequent system states can go through a resonance-like maximum when normal-distributed white noise of increasing standard deviation is added independently to all neurons. The phenomenon was called ‘Recurrence Resonance’ (RR) (), because can be interpreted as the spontaneous recurrent information flux in the network. It grows with the number of visited system states and with the degree of predictability of each successor state from its predecessor.
Since is a key factor for the information processing faculties of RNNs, it became important to understand which types of weight matrices enable a large spontaneous information flux in probabilistic RNNs, such as Symmetric Boltzmann Machines (SBMs, see Methods for details). By reverse-engineering evolutionary optimized networks (), we found indeed a universal design principle for maximizing (). It was called ‘NRooks’, because in optimal N-neuron networks each row and column of the weight matrix only contains a single non-zero entry, resembling the N-rooks-problem in chess (). While these non-zero elements should ideally have the same magnitude , their signs can be arbitrary. In the limit of large magnitudes , the SBMs become quasi-deterministic and the information flux approaches the theoretical maximum . In this extreme case, all possible system states are periodically visited in a fixed order.
The present work aims to understand, on a deeper level than before, the pre-conditions of the RR phenomenon, as well as its mechanism. As model systems, we will mainly use probabilistic SBMs, but we will also briefly consider deterministic networks with ‘’ activation functions. Different types of weight matrices will be investigated, but NRooks system will play a particularly important role, as their information theoretic properties are very well understood ().
2 Methods
2.1 Neural network model
We consider a recurrent network of model neurons. The total sum of inputs entering neuron in the discrete time step is given byHere, the first bracket contains a possible bias , as well as a weighted sum of the momentary output signals from all neurons in the network. The weighting describes the coupling strength from source neuron to target neuron . The second bracket accounts for the momentary external input signal entering neuron , scaled by a global input coupling parameter . Finally, the third bracket accounts for a random signal entering neuron , scaled by a global noise strength parameter . The are statistically independent random numbers, drawn from a standard Gaussian distribution with mean 0 and standard deviation 1. Further on, we will denote the complete weight matrix by , the bias vector by , the momentary state vector by , and the external input vector by . The total sum of inputs, as defined in Equation 1, is used to compute the next output state of neuron . This update, which is performed synchronously for all neurons, differs for the two models considered in this paper:
In the deterministic model, neural output signals are continuous in the range and are computed directly as the hyperbolic tangent of the total sum:
To initialize the deterministic network, the elements of the zero-time state vector are drawn independently from a uniform distribution in the range .
In the probabilistic model, neural output signals are discrete with the two possible values . The probability for the value , also called the ‘on’-probability, is computed as a logistic function of the total sum:
To initialize the probabilistic network, the elements of the zero-time state vector are drawn independently from a Bernoulli distribution, in which the possible outcomes and occur with equal probability.
We also refer to our probabilistic model as a Symmetrical Boltzmann Machine (SBM), which is called ‘symmetric’ because the binary outputs are set to , rather than the values in conventional Boltzmann machines. Our choice makes the SBMs directly comparable to deterministic tanh-networks with the same weight matrix.
Note that here we do not apply any input to the recurrent neural networks, and thus . Also, we do not use any biases, so that as well. For some types of sparse weight matrices considered in this work, the elements have ternary values . In this case, is called the weight magnitude parameter. The same name is also used for a multiplicative parameter that scales the standard deviation of a weight matrix with originally random normal elements .
After defining the weight matrix and randomly initializing a network, the time series of global system states is computed numerically for time steps, a parameter also called the observation time scale.
2.2 Information theoretic quantities
Numerical evaluation of information theoretic quantities requires data with discrete values. The binary output of the SBM is perfectly suited for this purpose, but in the case of the tanh-network we first needed to binarize the continuous outputs , defining if , and if . Thus, for both types of networks, the output time series can be reduced to a binary matrix , or for the time-shifted series. The matrix has rows, each with binary values 0 or 1. The rows correspond to momentary global states of the network, and (due to the binarization) the total number of possible global states is .
The starting point for all our information theoretic quantities is the joint probability that a global system state is followed by a subsequent state in the time series. Since the size of the state space increases exponentially with the network size , the estimation of becomes quickly unfeasible for large systems: Not only does it take too long before the system has ergodically spread over its entire state space and tried out all possible transitions , but in large systems it even becomes difficult to hold the huge matrix in the computer memory. To alleviate the memory problem, we hold only the matrix elements between the subset of global system states that have actually been visited by the system within the time scale , which can be much smaller if the system is trapped in a dynamical attractor. From the joint probability we directly obtain the two marginal probabilities and , which in our case are practically identical, because each final state becomes an initial state in the next time step.
The first information theoretical quantity of interest in the state entropy of global system states, defined bywhere all terms with count as zero.
The next relevant quantity is the mutual information between subsequent system states, defined bywhere all terms with , or , or count as zero.
The final important quantity is the conditional entropy , which in our case can also be written as . This conditional entropy describes the random divergence from a specific initial state to several possible final states and is therefore called the state divergence in the following. A value of would indicate perfectly deterministic behavior, a value of perfectly random behavior.
3 Results
3.1 Conditions and mechanisms of recurrence resonance
Our first goal is to identify the preconditions of RR, in particular regarding the network weight matrices. For this purpose, we consider a Symmetric Boltzmann Machine (SBM, see Methods for details) (Equations 1–3) with binary neurons. Since such a system has only global states , the entropy , the mutual information between subsequent system states and , as well as the divergence can be accurately estimated from the network’s output time series on a manageable time scale of time steps (See Methods for details) (Equations 4, 5).
As a basis for the network’s weight matrix , the 25 elements are first drawn independently from a standard normal distribution (Figure 1A). This ‘frozen random matrix’ is then multiplied (scaled) with a weight magnitude parameter , thus tuning the standard deviation of the matrix, while keeping its fundamental structure invariant. We can then explore how the three information theoretic quantities , and depend on the weight magnitude and on the strength of added white noise (See Methods for details).
FIGURE 1
Before that, it is useful to imagine the dynamical structure of the SBM as a state transition graph, in which the 32 nodes represent the possible global statesand the weighted directional edges represent the possible transitions between pairs of states (Figures 1G, H). The probability of each transition, which is given by the conditional probability , may be indicated by the thickness of the edges. This state transition graph, which is determined only by the weight matrix , describes the possible dynamical behavior of the network completely, irrespective of which path the system is actually taking through the graph.
Below, we will visualize the aggregated activity in the network by the joint probability . While this quantity also respects the fundamental transition possibilities (described by ), it additionally accounts for which states and transitions the system has actually used during the observation time (described by ).
3.1.1 Effect of increasing neural coupling without noise
We first consider the system without applying external noise . For a weight magnitude of , the five neurons are completely isolated from each other and also have no self-connections (autapses). Consequently, each of the probabilistic SBM neurons produces a non-biased, temporally uncorrelated, binary random walk, where the two possible values and occur with equal probabilities, and independently from each other. This means that any momentary global system state can transition to any other global successor state with equal probability. We thus have a ‘structureless’, fully connected state transition graph, with equally ‘thick’ edges.
Since the time scale of is long enough for the system to explore its entire space of global states, we expect that for the entropy reaches its maximum possible value . As the system is purely random with a structureless state transition graph, the mutual information is expected to be . Consequently, the divergence is also maximal at . This is indeed found in the numerical simulation.
As we tune the weight magnitude from zero to increasingly positive values, the state transition graph, being determined by the weight matrix, is gradually developing a structure, that is, some of the transitions become more probable than others. Consequently, certain dominating paths are forming within the graph along the ‘thick’ edges, leading eventually to the emergence of dynamical attractors, such as fixed points (Figure 1I), higher period n-cycles, or transient states. These attractors are quite unstable at low weight magnitudes , so that the system can still transition between them. Nevertheless, within the limited time horizon of time steps, it now becomes impossible to visit all 32 global states with equal probability. Without additional noise , this leads to a gradual decrease of the entropy (Figure 1B). The growth of structure in the graph with increasing makes the system dynamics more deterministic and is thus also connected with a decrease of the random divergence (not shown in the figure).
However, since the entropy and the divergence decrease at different rates with the weight magnitude , the mutual information shows a more complicated behavior (Figure 1C). As is increased from 0.2 to 2, the mutual information without noise is first increasing, reflecting the higher degree of predictability of the next state. But for , we find that is decreasing again, reaching a value of zero for (Figure 1C, red and magenta curve).
This extreme situation of , which is most often found at large weight magnitudes but without noise, means that the system is trapped in a single global state, in other words, a quasi-stable fixed point . Correspondingly, the joint probability of subsequent system states has only a single non-zero entry at (Figure 1 (d,matrix plot)), and also the marginal state probability has only a single entry at the fixed point state (Figure 1 (d,histogram on top)).
3.1.2 System behavior with noise
We now go back to the case of relatively weak weight magnitudes and gradually increase the strength of added noise. Here, the entropy is already quite large without noise, and adding noise increases it even further (Figure 1 (b, green curves on the very top)). The noise thus helps the system to visit all possible states with equal probability, which can be seen as a beneficial effect. However, the noise increases also the divergence at a fast rate (not shown in the figures), so that the mutual information is only decreasing as more noise is added (Figure 1 (c, green, light blue and dark blue curves)). We therefore do not observe RR, that is, a peak of the mutual information as a function of noise, in the regime of weak weight magnitudes.
In contrast, a different behavior is found for stronger weight magnitudes . Here, the entropy is also increased by adding more noise (Figure 1 (b, red and magenta curves)), but the mutual information is now initially increasing with noise - with a small exception at small noise levels (Figure 1 (c, red and magenta curves)). In the case of (red curve), it reaches a maximum at a noise level of around and then decreases again. For (magenta curve), the maximum of is around . Thus, RR is only observed in the regime of sufficiently strong weight magnitudes.
Generally, whenever the mutual information as a function of noise shows a clear maximum in a given network, the joint and marginal probability distributions are characteristically different at the points without noise (Figure 1D, ), close to the RR maximum (Figure 1E, ), and far beyond the RR maximum (Figure 1E, ): Without noise, the system visits relatively few states, spending its time in only one or a few attractors. At the RR peak, the number of visited states is larger, and those states mainly belong to quasi-stable attractors (The matrix plot then shows a distinct set of dominating entries). Beyond the RR peak, the system visits even more states, but now only transiently, without staying in any particular attractor for a longer period of time (The matrix plot then appears more uniform and unstructured than at the RR maximum).
3.2 RR in selected networks with multiple attractors
If RR is a process where noise helps neural networks to reach more attractors in a given time horizon, the phenomenon should be particularly pronounced in systems with multiple (as well as sufficiently stable) attractors. We therefore select the following three specific types of weight matrices, while keeping the network size of the SBM at and the time scale at : an autapses-only network, a Hopfield network, and a NRooks network.
3.2.1 Autapses-only network
We first test a ‘autapses-only’ network, in which all non-diagonal elements of the weight matrix (corresponding to inter-neuron connections) are zero, whereas the diagonal elements (corresponding to neuron self-connections, or ‘autapses’) have the same positive value (Figure 2 (row (a), inset of left plot)). In this case, since the SBM neurons are isolated from each other, they produce mutually independent, binary random walks. The probabilities of and are still equal, because we do not use biases (, See Methods). However, due to the excitatory autapses, the random walks are now temporally correlated (‘persistent’), which means that an output of is more likely followed by another , and analogously for . Each neuron thus tends to produce longer chains of outputs with identical sign, only switching to the opposite sign after a certain correlation time. For the total system, this means that any momentary global state is conserved, with high probability, for a finite number of time steps. Hence, each of the 32 global system states is a quasi-stable fixed point here, and the degree of stability can be increased by the self-connection strength . For this reason, this type of network could be useful as short-term memories in practical applications.
FIGURE 2
In our simulation, the autapse-only network without external noise is spending all of the time steps in only two of its 32 fixed point attractors, consequently leading to a mutual information of about (Figure 2 (row (a), column ‘no noise’)). Adding an optimal amount of noise drives the network to visit all available attractors, yet not with the same frequency. Nevertheless, the peak mutual information is with close to the upper limit of 5 (Figure 2 (row (a), column ‘optim. noise’)). Applying an excessive noise of lets the system undergo almost all possible state-to-state transitions, but these frequent unpredictable jumps between attractors lead to a mutual information of only (Figure 2 (row (a), column ‘strong noise’))
A remarkable feature of the autapse-only network’s RR-curve (Figure 2 (row (a)) is the part between zero and optimal noise. In this regime, the mutual information follows extremely closely the rising curve of the entropy , meaning that the divergence is extremely small. Hence, the noise is on the one hand able to occasionally transfer the system from one (fixed point) attractor to a different one, but on the other hand allows the system to stay for a sufficiently long time interval within each attractor, so that the next state remains predictable to a high degree which constitutes a precondition for a high mutual information. We will see below that other systems also show this initial noise regime where entropy and mutual information rise together, while the divergence remains close to zero. At some level of noise, of course, the divergence must increase as well.
3.2.2 Hopfield network
Another type of recurrent neural network that is famous for its ability to have multiple (designable) fixed point attractors is the Hopfield network (). Weight matrices of Hopfield networks are symmetric and have no self-connections . Its neurons are traditionally updated one by one in an asynchronous manner, but we continue to use a synchronous update in our SBM model.
We have designed the weight matrix to ‘store’ the two patterns and . The magnitude of the weight matrix elements was made large enough to ensure a good stability of the two fixed points corresponding to the stored patterns (Figure 2 (row (b), inset of left plot)).
In a broad initial regime of noise strengths , the RR-curve of the Hopfield network (Figure 2 (row (b), left plot)) shows an entropy and mutual information of zero, which is characteristic for a system that is trapped in a single fixed point. The matrix plot of the joint probability reveals that this fixed point is the global state , the first of the two stored patterns (Figure 2 (row (b), column ‘no noise’)). The mutual information starts to rise sharply at around and reaches a peak at , while the entropy continues to increase toward the upper limit. At the optimal noise level, the system is now visiting both fixed points (state 7 as well as state 24) with similar frequency, resulting in a peak mutual information of about (Figure 2 (row (b), column ‘optim. noise’)). Since the two fixed points are very stable (due to the large magnitude of matrix elements), the system is spending a comparatively large fraction of time in these two states, even at a large noise level of (Figure 2 (row (b), column ‘strong noise’)).
3.2.3 NRooks network
Finally, we test the RR phenomenon in so-called ‘NRooks’ networks, which under ideal conditions (large magnitude of non-zero weights and long observation time scale ) are known to approach the upper theoretical limit of mutual information and entropy, corresponding to (), and a vanishing divergence . The NRooks weight matrix has only one non-zero matrix element in each row and column (hence the name), and these non-zero matrix elements have the same magnitude , but arbitrary signs (Figure 2 (row (c), upper inset of left plot)). It has been shown that all global states of an NRooks system are parts of n-cyles of various sizes, that is, there are no transient states that would merely lead into these attractors ().
Our specific NRooks system turns out to have four different 8-cycles as attractors, and without noise it is trapped in one of them (Figure 2 (row (c), column ‘no noise’)). Since running for thousands of time steps within this attractor involves eight distinct states (creating entropy) in a perfectly predictable order (without divergence ), the entropy and mutual information have already a relatively large value of , even without external noise (Figure 2 (row (c), left plot)). Since attractors are quite stable at , we observe a plateau with in the RR-curve, holding up to a noise level of , where and start to increase rapidly. At the peak of the mutual information, occurring for a noise level of , it reaches the value of , which is close to the theoretical maximum of 5. Indeed, at that point the system is visiting all four 8-cycle attractors with about the same frequency (Figure 2 (row (c), column ‘optim. noise’)). It stays for a very long time in each of them, behaving almost perfectly deterministic. Only occasionally, the noise of optimal strength ‘kicks’ the system randomly to one of the other three attractors. As usual, for a very strong amount of noise, the system looses its predictability, and the mutual information drops accordingly, while the entropy remains at the upper limit.
In the above numerical experiments with multi-attractor SBMs, we have used relatively large weight magnitudes . In the given context, this served the purpose to make the attractors of the autonomous networks more stable. More generally, large weight magnitudes drive the SBM neurons into the saturation regime of the logistic activation function, so that the on-probabilities become either or for all neurons and all time steps . Hence, the probabilistic SBM then behaves quasi deterministic. For this reason, we could use an SBM to implement a Hopfield network, which is usually based on deterministic binary threshold neurons.
In order to further demonstrate the saturation regime of the SBM, we have used the same weight matrix that was used in the NRooks example also in a network with deterministic tanh-Neurons (See Methods for details). The resulting RR-curve is indeed extremely similar to that of the probabilistic SBM (Figure 2 (row (c), lower inset of left plot)).
3.3 Time-scale dependence of RR
As already mentioned above, the observation time scale is another critical factor that determines whether a RR peak will be observable in a given system.
For a demonstration, we now use a NRooks system of only three neurons (Figure 3A). Because of its extremely small state space , it is possible to make the attractors stable by a relatively large weight magnitude of , but nevertheless to approach the ergodic limit of very large time scales , where the system is able to visit all its attractors autonomously, without the injection of external noise.
FIGURE 3
We find that for too strong levels of noise (here ), the mutual information is just declining, irrespective from the time scale . In this regime, the system is operating already at maximum entropy , but the noise is causing an increasing loss of predictability.
For a time scale of , which is appropriate for the given system, a clear RR peak is observed in the mutual information curve at around . As the time scale is now prolonged, up to around , the mutual information is generally rising for moderate noise levels in the range , because the systems gets more opportunities to escape and switch from one attractor state to another. As a consequence, the RR peak is moving to smaller noise levels .
For the moderate time scales considered so far, the system, without noise, is trapped in a 4-cycle, and therefore the mutual information is . However for larger time scales , we find a value close to the theoretical maximum of 3, which means that now even the zero-noise system can visit all its attractor states and run through each of them in an almost perfectly predictable way. Thus, the phenomenon of RR is not observable on extremely long time scales, where systems already operate close to their ergodic regime.
3.4 “Local” mutual information in sub-networks
In actual applications of RNNs, such as reservoir computing, networks are typically so large and consequently the state spaces so huge that the ergodic regime cannot be reached on any practical time scale . Moreover, in such practically non-ergodic systems, it is also impossible to accurately evaluate the full mutual information of subsequent system states, because the joint probability matrices are too large and because the empirical distributions have not enough time to converge toward a stable result. The question then arises how to compute a useful approximation of in large systems, even when they are observed on ‘too short’ (but practically relevant) time scales.
Although a detailed investigation of this question is beyond the scope of the present paper, we provide a first insight using a 15-neuron NRooks system, observed on the non-ergodic time scale of (Figure 3B). To alleviate the matrix size problem, we only consider global states that have actually been visited during the observation time (See Methods for details). Moreover, as a proxy for the full , we compute the ‘local’ mutual information within smaller sub-networks, i.e., subgroups of only neurons.
For large sub-networks , instead of a RR peak, we find an increase of the local mutual information with noise, and finally a saturation. This plateau is also observed for larger noise levels up to (data not shown).
In contrast, for very small sub-networks , we find already a smaller starting value of at zero noise, and eventually a decline of with increasing noise level .
However, for a certain intermediate range of sub-network sizes between and , the curve shows a clear maximum that decays very slowly after the peak. Thus, in large networks, when observed on short quasi non-ergodic time scales, a phenomenon similar to RR can occur within smaller sub-networks, whereas the mutual information of the total system then shows a saturation-type dependence on the noise level.
3.5 Effect of noise pulses in probabilistic, binary-valued RNNs
At the peak of the RR curve, the continuous white noise input of optimal strength is leaving a RNN in its present attractor for long times, but occasionally causes a random transit to one of the other available attractors. It is this combination of high predictability and high entropy that leads to the optimal value of the mutual information.
A natural extension (and putative application) of this concept are short noise pulses - applied only at times when a change of attractor state is required - instead of a continuous feed-in of noise. To test this concept, we have again used the 5-neuron NRooks system of Figure 2C, with its four different 8-cycles as attractors. The system is initially in one of its 8-cycle attractors (Figure 4A), and remains in this attractor for an arbitrarily long period that only depends on the weight magnitude . By applying short (10 time steps) yet strong Gaussian white noise pulses, we could indeed transfer the system randomly to one of the other attractors. It also happens that the system ends up in the same attractor (yet at a different ‘phase’ of the periodic cycle), but eventually we could reach all four 8-cycles by this way (Figures 4B–D).
FIGURE 4
In actual applications, the noise strength could be made a continuously or abruptly changing function of time, designed to accelerate the ‘equilibration’ of the RNN over its state space. This clearly resembles the well-known technique of simulated annealing (; ; ), yet with the important difference that quasi-deterministic RNNs follow a prescribed order of states, once they have entered a cyclic attractor. In biological neural networks, the optimal momentary noise strength might even be provided by a feedback control system that continuously aims to optimize network performance.
3.6 Effect of noise pulses in continuous-valued RNNs
So far, we have only briefly explored the effect of noise on networks of deterministic tanh-neurons (inset of Figure 2C). As a further glimpse into this alternative field of research, we apply a short (5 time steps) and weak noise pulse to a very small (3 neurons) tanh-network, in which the nine matrix elements have been drawn randomly from a standard normal distribution .
Before the noise pulse, the system is allowed to run freely for 100 time steps. The resulting system states at each time step are here continuous points within the three-dimensional cube and can thus be visualized directly as a 3-dimensional trajectory (Figures 4E–G). We find the system initially within a ‘strange’, loop-like attractor (e). During the short noise pulse, the trajectory is erratic and reaches the borders of the state space cube (f). After the pulse, the system has settled in a new strange attractor, which resembles a 2-cycle, but only with an approximate return to the end points in each oscillation period. Thus, it is also possible to achieve a switch of attractor states in deterministic RNNs with continous output values by the injection of noise pulses.
4 Conclusion
In this work, we have re-considered the phenomenon of Recurrence Resonance (RR), i.e., the peak-like dependence of an RNN’s internal information flux on the level of white noise added to each of the neurons (). The information flux is measured by the mutual information between subsequent system states, a quantity that grows as more states become available (larger entropy ), and/or when each successor state can be better predicted from its predecessor (smaller divergence ).
We have shown that a resonance-like peak of can only be observed in networks that fundamentally have a whole set of relatively stable dynamical attractors available, but which - without external intervention - would remain trapped in one of them during the entire observation time scale . In this situation, adding a small level of noise helps such networks to occasionally jump out of the present attractor and switch into another one, without significantly reducing the predictability of the state sequence within each of these quasi-stable attractors (strong increase of , but weak increase of ). If the noise level is however increased beyond the optimal point , predictability is lost and consequently the information flux is declining again, while the entropy is still increasing toward its upper limit. By contrast, networks that already have a high internal information flux from the beginning will not show a RR peak, but only a decline of as a function of .
We have demonstrated the RR phenomenon using Symmetric Boltzmann Machines (SBMs) with different types of weight matrices, including random Gaussian matrices, diagonal matrices (autapse-only networks), Hopfield networks trained on specific patterns, and in NRooks systems that are known to reach the upper limit of information flux. In each case, we demonstrated that the network without noise is trapped in a single or few attractors, based on the joint probability of subsequent system states. An optimal level of noise makes more (or even all) attractors available without too many unpredictable transitions. However, excessive levels of noise cause more or less random jumps between all possible pairs of states. In systems with a very high stability of attractors (induced by a large weight magnitude ), we have found that remains constant at the initial value for a certain range of noise levels, before it abruptly rises in the way of a phase transition.
We have also demonstrated that RR can only be observed in appropriate time scales , relative to the total number of possible system states . For arbitrarily long observation times (which are of theoretical interest but not so much of practical relevance), a neural network operating in the quasi-deterministic regime, but with at least a small probabilistic component (like SBMs with a large but finite weight magnitude ) can eventually visit all its attractors in an ergodic manner, but still stay in each of them for extended time intervals. Then is already close to the optimum value and application of noise only degrades the information flux.
An interesting problem arises therefore in networks with many neurons and thus an exponentially large state space, such as reservoir computers, or brains. Such systems will necessarily spend all their lives within a negligible fraction of the fundamentally available state space, possibly consisting of only a tiny subset of attractors. One way to cope with this ‘practical non-ergodicity’ would be a repeated active switching between attractor subsets (perhaps using noise pulses), until a useful one is found, and then to stay there. Alternatively, the networks may be designed (or optimized) such that the useful attractors have a very large basin of attraction. A similar problem has been discussed in the context of protein folding with the ‘Levinthal paradox’, where naturally existing proteins fold into the desired conformation much faster than expected by a random thermal search in conformation space (Zwanzig et al., 1992; ; ), probably due to funnel-type energy landscapes (; ; Wolynes, 2015; Röder et al., 2019).
In our context of the RR phenomenon, practical non-ergodicity makes it impossible to compute the stationary information flux in a large network, because the system never reaches a stationary state (marked by constant probability distributions) within any practical time scale . When the information flux in a network is evaluated naively, using the ‘transient’ (not yet converged) joint probability distributions, we have found that shows a saturating behavior, rather than a maximum. Nevertheless, the ‘local’ mutual information, evaluated for a suitably sized sub-network, can then still show a RR peak.
Finally, we have explored the repeated application of short noise pulses, rather than feeding continuous noise into the neurons. We could demonstrate that each pulse offers the network a chance to switch to a new random attractor (such as an n-cycle), while the intermediate free running phases allow the system to deterministically and thus predictably follow the fixed order of states within each of the attractors.
If the random noise signals are delivered independently to each individual neuron, and if the levels of these external control signals are strong enough to override the recurrent internal signals from the other neurons, then the network could theoretically end up, after the noise application, in any of its global system states. Thus all the system’s attractors could theoretically be accessed by this way. Nevertheless, a complete exploration of the entire attractor landscape will take an extremely long time in large networks.
Based on this noise-induced random switching mechanism, an evolutionary optimization algorithm could be implemented in a recurrent neural network, in which various attractors are tried until one turns out useful for a given task. We speculate that this principle might be used in central pattern generators of biological brains (), for example, in order to find temporal activation patterns for certain motor tasks ().
5 Discussion
In free running SBMs, a neuron’s probability of being ‘on’ in the next time step is computed by a logistic activation function , where is the weighted sum of inputs from other neurons. Adding white, normally distributed noise to corresponds to a convolution of the logistic function with a Gaussian kernel, resulting in a ‘broadening’ of the activation function. Increasing the noise level thus has an effect similar to turning up the ‘temperature’ parameter in a re-scaled activation function . This opens up a new interpretation of the RR phenomenon in terms of statistical physics, in particular if an energy can be assigned to each global system state .
In this case, the system - after sufficiently long time - would come to thermal equilibrium, and the probability of finding it at any state would be proportional to the Boltzmann distribution . At low temperature, the system would therefore spend most of its time in the deepest valleys of the energy landscape, which may correspond to fixed point attractors. Increasing the temperature would lead to a more uniform distribution of states over the energy landscape, and there might be a sweet spot for the temperature that perfectly balances stability of the attractors with occasional barrier crossings, just as in the RR phenomenon. Moreover, during the equilibration process, it might be advantageous to start with a high temperature (noise level) and then to slowly decrease it, as in Simulated Annealing (Van Laarhoven et al., 1987; ).
Note, however, that in our SBM model the system’s dynamics cannot be visualized as a simple probabilistic downhill relaxation within the energy landscape , because we are using a synchronous update of all neurons and non-symmetric weights. For example, in NRooks systems, each n-cycle attractor can have another energy, but the energy is the same for all states that belong to the same attractor. Despite of this ‘energetical degeneration’ of states within a given n-cycle, the system is not randomly jumping between those states, as it would be expected from a thermal system, but it is running through the state sequence in a perfectly deterministic way.
In this work, we have mainly focused on probabilistic SBMs as model systems of recurrent neural networks. However, we have shown that for sufficiently large weight magnitudes the neurons operate in the saturation regime of the logistic activation function and therefore the SBMs behave quasi-deterministic. In this regime, the RR curves of SBMs turned out to be extremely similar to those of networks with the same weight matrix, but with deterministic tanh-neurons. Also, we have demonstrated that attractor switching by noise pulses works equally well for tanh-networks with continuous outputs. Nevertheless, regarding the details of the RR phenomenon, we expect future work to reveal some fundamental differences between probabilistic and deterministic RNNs. In particular, while the SBM neurons turn into independent random generators when the general weight magnitude of their connections is turned down, deterministic networks can - even for small - produce complex dynamical attractor states. It is not clear at present how sensitive those attractors react on externally injected noise.
Neural networks, both artificial and biological, have a tendency to become trapped in low-entropy dynamical attractors, which correspond to repetitive, predictable patterns of activity (). These attractors are often associated with stable cognitive states or established perceptual interpretations (). In particular, it has been shown that during spontaneous activity the brain does not randomly change between all theoretically possible states, but rather samples from the realm of possible sensory responses (; Schilling et al., 2024). This indicates that the brain’s spontaneous activity encompasses a spectrum of potential responses to stimuli, effectively preconfiguring the neural landscape for incoming sensory information. While stability and predictability are crucial for efficient functioning and reliable behavior, they can also limit the flexibility and adaptability of the system (; ). This is particularly problematic in contexts requiring learning, creativity, and the generation of novel ideas (Sandamirskaya, 2013).
The introduction of noise into neural networks has been suggested as a mechanism to overcome the limitations imposed by these low-entropy attractors (). Noise, in this context, refers to stochastic fluctuations that perturb the network’s activity, pushing it out of stable attractor states and into new regions of the state space (). This process can enhance the network’s ability to explore a wider range of potential states (Sietsma and Dow, 1991), thereby increasing its entropy and promoting the discovery of novel solutions or interpretations.
Biological neural systems, such as the human brain, provide compelling evidence for the utility of noise in cognitive processes. The brain is inherently noisy, with intrinsic fluctuations occurring at multiple levels, from ion channel gating to synaptic transmission and neural firing (). This noise is not merely a byproduct of biological imperfection; rather, it plays a functional role in various cognitive tasks (). For example, noise-induced variability in neural firing can enhance sensory perception by enabling the brain to detect weak signals that would otherwise be drowned out by deterministic activity ().
Another noise-based phenomenon of great importance in physical and biological systems is Stochastic Resonance (; Moss et al., 1993). It is typically occurring in signal detection systems, where the incoming signal needs to exceed a minimal threshold in amplitude to be detected. Adding an appropriate level of noise to the input can then stochastically lift even weak signal above the threshold and thereby improve the detection performance, as it has indeed been observed in various sensory systems (Moss et al., 2004; Stein et al., 2005; Ward, 2013). The phenomenon of Recurrence Resonance discussed in this paper is different from Stochastic Resonance, as it improves the spontaneous, internal information flux in a neural network, rather than the signal transmission from the outside to the inside of the system. However, using mutual information or correlation-based measures, it is also possible to quantify the information flux from the input nodes of a RNN at time to the internal network state at a later time ().
Noise also facilitates learning and plasticity. During development, random fluctuations in neural activity contribute to the refinement of neural circuits, allowing for the fine-tuning of synaptic connections based on experience (; Zhang et al., 2021; ). In adulthood, noise can help the brain escape from local minima during learning processes, thereby preventing overfitting to specific patterns and promoting generalization (Zhang et al., 2021; ). This is particularly relevant in the context of reinforcement learning, where exploration of the state space is crucial for finding optimal strategies (Weng, 2020; ).
Moreover, noise-induced transitions between attractor states can support cognitive flexibility and creativity. For instance, the ability to switch between different interpretations of ambiguous stimuli (Panagiotaropoulos et al., 2013), or to generate novel ideas, relies on the brain’s capacity to break free from dominant attractor states and explore alternative possibilities (Wu and Koutstaal, 2020; ). This is consistent with the observation that certain cognitive disorders, characterized by rigidity and a lack of flexibility (e.g., autism, obsessive-compulsive disorder), are associated with reduced neural noise and hyper-stable attractor dynamics (; Watanabe et al., 2019).
In conclusion, the investigations detailed in our study firmly establish Recurrence Resonance (RR) as a genuine emergent phenomenon within neural dynamics: the mutual information of the system is increased by the addition of noise that itself has zero mutual information. Hence, the application of optimal noise levels can transform neural systems from states of minimal information processing capabilities to significantly enhanced states where information flow is not only possible but also maximized. This effect, whereby noise beneficially modifies system dynamics, underscores the complex and non-intuitive nature of neural information processing, presenting noise not merely as a disruptor but as a critical facilitator of dynamic neural activity. This finding opens up new avenues for exploiting noise in the design and enhancement of neural network models, particularly in areas demanding robust and adaptive information processing.
The introduction of noise into neural networks can be seen as a fundamental mechanism by which the brain enhances its cognitive capabilities. By destabilizing low-entropy attractors and promoting the exploration of new states, noise enables learning, perception, and creativity. This perspective not only aligns with empirical findings from neuroscience but also offers a theoretical framework for understanding how complex cognitive functions can emerge from the interplay between deterministic and stochastic processes in neural systems.
Furthermore, the insights gained from our study provide a valuable foundation for advancing artificial intelligence (AI) technologies, particularly in the realms of reservoir computing and machine learning. Reservoir computing, which leverages the dynamic behavior of recurrent neural networks, can benefit from the strategic introduction of noise to enhance its computational power and adaptability. Similarly, machine learning models can incorporate noise to avoid overfitting, explore diverse solution spaces, and improve generalization. By integrating these principles, AI systems can emulate the brain’s ability to learn and adapt in complex, unpredictable environments, leading to more robust and innovative technological solutions. This convergence of neuroscience and AI not only deepens our understanding of cognitive processes but may also drive the development of next-generation intelligent systems capable of solving real-world problems with unprecedented efficiency and creativity.
Statements
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Author contributions
CM: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Software, Validation, Visualization, Writing–original draft, Writing–review and editing. AS: Funding acquisition, Validation, Writing–original draft. AM: Resources, Supervision, Validation, Writing–original draft. PK: Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Validation, Writing–original draft, Writing–review and editing.
Funding
The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation): KR 5148/3-1 (project number 510395418), KR 5148/5-1 (project number 542747151), and GRK 2839 (project number 468527017) to PK, and grant SCHI 1482/3-1 (project number 451810794) to AS.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
AartsE. H. (1987). Simulated annealing: theory and applications. Reidel.
2
AartsE. H. L.Van LaarhovenP. J. M. (1989). Simulated annealing: an introduction. Stat. Neerl.43 (1), 31–52. 10.1111/j.1467-9574.1989.tb01245.x
3
AlzubaidiL.ZhangJ.HumaidiA. J.Al-DujailiA.DuanYeAl-ShammaO.et al (2021). Review of deep learning: concepts, cnn architectures, challenges, applications, future directions. J. big Data8 (1), 53–74. 10.1186/s40537-021-00444-8
4
BaiC.LiuP.LiuK.WangL.ZhaoY.HanL.et al (2023). Exploration in deep reinforcement learning: from single-agent to multiagent domain. IEEE Trans. Neural Netw. Learn. Syst.34, 4776–4790. 10.1109/tnnls.2021.3129160
5
BarakO. (2017). Recurrent neural networks as versatile tools of neuroscience research. Curr. Opin. Neurobiol.46, 1–6. 10.1016/j.conb.2017.06.003
6
BeerC.BarakO. (2024). Revealing and reshaping attractor dynamics in large networks of cortical neurons. PLOS Comput. Biol.20 (1), e1011784. 10.1371/journal.pcbi.1011784
7
BertschingerN.NatschlägerT. (2004). Real-time computation at the edge of chaos in recurrent neural networks. Neural Comput.16 (7), 1413–1436. 10.1162/089976604323057443
8
BertsimasD.TsitsiklisJ. (1993). Simulated annealing. Stat. Sci.8 (1), 10–15. 10.1214/ss/1177011077
9
BishopC. M. (1995). Training with noise is equivalent to tikhonov regularization. Neural Comput.7 (1), 108–116. 10.1162/neco.1995.7.1.108
10
BoedeckerJ.ObstO.LizierJ. T.MayerN. M.AsadaM. (2012). Information processing in echo state networks at the edge of chaos. Theory Biosci.131 (3), 205–213. 10.1007/s12064-011-0146-8
11
BönselF.KraussP.MetznerC.YamakouM. E. (2022). Control of noise-induced coherent oscillations in three-neuron motifs. Cogn. Neurodynamics16 (4), 941–960.
12
BrunelN. (2016). Is cortical connectivity optimized for storing information?Nat. Neurosci.19 (5), 749–755. 10.1038/nn.4286
13
BryngelsonJ. D.OnuchicJ. N.SocciN. D.WolynesP. G. (1995). Funnels, pathways, and the energy landscape of protein folding: a synthesis. Proteins Struct. Funct. Bioinforma.21 (3), 167–195. 10.1002/prot.340210302
14
BüsingL.SchrauwenB.LegensteinR. (2010). Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons. Neural Comput.22 (5), 1272–1311. 10.1162/neco.2009.01-09-947
15
DambreJ.VerstraetenD.SchrauwenB.MassarS. (2012). Information processing capacity of dynamical systems. Sci. Rep.2 (1), 514–517. 10.1038/srep00514
16
DecoG.RollsE. T.RomoR. (2009). Stochastic dynamics as a principle of brain function. Prog. Neurobiol.88 (1), 1–16. 10.1016/j.pneurobio.2009.01.006
17
DwyerP.VukusicS.WilliamsZ. J.SaronC. D.RiveraS. M. (2024). “neural noise” in auditory responses in young autistic and neurotypical children. J. autism Dev. Disord.54 (2), 642–661. 10.1007/s10803-022-05797-4
18
FaisalA. A.SelenL. P. J.WolpertD. M. (2008). Noise in the nervous system. Nat. Rev. Neurosci.9 (4), 292–303. 10.1038/nrn2258
19
FangY.YuZ.ChenF. (2020). Noise helps optimization escape from saddle points in the synaptic plasticity. Front. Neurosci.14, 343. 10.3389/fnins.2020.00343
20
FarrellM.RecanatesiS.MooreT.LajoieG.Shea-BrownE. (2022). Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion. Nat. Mach. Intell.4 (6), 564–573. 10.1038/s42256-022-00498-0
21
FolliV.GostiG.LeonettiM.RuoccoG. (2018). Effect of dilution in asymmetric recurrent neural networks. Neural Netw.104, 50–59. 10.1016/j.neunet.2018.04.003
22
GammaitoniL.HänggiP.JungP.MarchesoniF. (1998). Stochastic resonance. Rev. Mod. Phys.70 (1), 223–287. 10.1103/revmodphys.70.223
23
GerumR. C.ErpenbeckA.KraussP.SchillingA. (2020). Sparsity through evolutionary pruning prevents neuronal networks from overfitting. Neural Netw.128, 305–312. 10.1016/j.neunet.2020.05.007
24
GononL.OrtegaJ.-P. (2021). Fading memory echo state networks are universal. Neural Netw.138, 10–13. 10.1016/j.neunet.2021.01.025
25
HarunaT.NakajimaK. (2019). Optimal short-term memory before the edge of chaos in driven random recurrent networks. Phys. Rev. E100 (6), 062312. 10.1103/physreve.100.062312
26
HavivD.RivkindA.BarakO. (2019). “Understanding and controlling memory in recurrent neural networks,” in International conference on machine learning (Long Beach, CA, United States: PMLR), 2663–2671.
27
HintonG. E.Van CampD. (1993). “Keeping the neural networks simple by minimizing the description length of the weights,” in Proceedings of the sixth annual conference on Computational learning theory, 5–13.
28
HonigB. (1999). Protein folding: from the levinthal paradox to structure prediction. J. Mol. Biol.293 (2), 283–293. 10.1006/jmbi.1999.3006
29
HooperS. L. (2000). Central pattern generators. Curr. Biol.10 (5), R176–R179. 10.1016/s0960-9822(00)00367-5
30
HopfieldJ. J. (1982). Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci.79 (8), 2554–2558. 10.1073/pnas.79.8.2554
31
IchikawaK.KanekoK. (2021). Short-term memory by transient oscillatory dynamics in recurrent neural networks. Phys. Rev. Res.3 (3), 033193. 10.1103/physrevresearch.3.033193
32
IkemotoS.DallaLiberaF.HosodaK. (2018). Noise-modulated neural networks as an application of stochastic resonance. Neurocomputing277, 29–37. 10.1016/j.neucom.2016.12.111
33
JaegerH. (2001). The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148.
34
JaegerH. (2014). Controlling recurrent neural networks by conceptors. arXiv Prepr. arXiv:1403.3369.
35
Jaimes-ReáteguiR.Huerta-CuellarG.García-LópezJ. H.PisarchikA. N. (2022). Multistability and noise-induced transitions in the model of bidirectionally coupled neurons with electrical synaptic plasticity. Eur. Phys. J. Special Top.231, 255–265. 10.1140/epjs/s11734-021-00349-w
36
KadmonJ.SompolinskyH. (2015). Transition to chaos in random neuronal networks. Phys. Rev. X5 (4), 041030. 10.1103/physrevx.5.041030
37
KanekoK.SuzukiJ. (1994). “Evolution to the edge of chaos in an imitation game,” in Artificial life III (Citeseer).
38
KarplusM. (1997). The levinthal paradox: yesterday and today. Fold. Des.2, S69–S75. 10.1016/s1359-0278(97)00067-9
39
KatzL.SobelM. (1972). “Coverage of generalized chess boards by randomly placed rooks,” in Proceedings of the sixth berkeley symposium on mathematical statistics and probability, probability theory (University of California Press), 6.3, 555–564. 10.1525/9780520375918-031Contributions to Probability Theory
40
KhonaM.FieteI. R. (2022). Attractor and integrator networks in the brain. Nat. Rev. Neurosci.23 (12), 744–766. 10.1038/s41583-022-00642-0
41
KirkpatrickS.GelattC. D.JrVecchiM. P. (1983). Optimization by simulated annealing. science220 (4598), 671–680. 10.1126/science.220.4598.671
42
KraussP.PrebeckK.SchillingA.MetznerC. (2019a). Recurrence resonance in three-neuron motifs. Front. Comput. Neurosci.13, 64. 10.3389/fncom.2019.00064
43
KraussP.SchusterM.DietrichV.SchillingA.SchulzeH.MetznerC. (2019c). Weight statistics controls dynamics in recurrent neural networks. PloS one14 (4), e0214541. 10.1371/journal.pone.0214541
44
KraussP.TziridisK.MetznerC.SchillingA.HoppeU.SchulzeH. (2016). Stochastic resonance controlled upregulation of internal noise after hearing loss as a putative cause of tinnitus-related neuronal hyperactivity. Front. Neurosci.10, 597. 10.3389/fnins.2016.00597
45
KraussP.ZanklA.SchillingA.SchulzeH.MetznerC. (2019b). Analysis of structure and dynamics in three-neuron motifs. Front. Comput. Neurosci.13 (5), 5. 10.3389/fncom.2019.00005
46
LangtonC. G. (1990). Computation at the edge of chaos: phase transitions and emergent computation. Phys. D. Nonlinear Phenom.42 (1-3), 12–37. 10.1016/0167-2789(90)90064-v
47
LeCunY.BengioY.HintonG. (2015). Deep learning. nature521 (7553), 436–444. 10.1038/nature14539
48
LegensteinR.MaassW. (2007). Edge of chaos and prediction of computational performance for neural circuit models. Neural Netw.20 (3), 323–334. 10.1016/j.neunet.2007.04.017
49
LuczakA.BarthóP.HarrisK. D. (2009). Spontaneous events outline the realm of possible sensory responses in neocortical populations. Neuron62 (3), 413–425. 10.1016/j.neuron.2009.03.014
50
MaheswaranathanN.WilliamsA. H.GolubM. D.GanguliS.SussilloD. (2019). Universality and individuality in neural dynamics across large populations of recurrent networks. Adv. neural Inf. Process. Syst.2019, 15629–15641.
51
MarderE.BucherD. (2001). Central pattern generators and the control of rhythmic movements. Curr. Biol.11 (23), R986–R996. 10.1016/s0960-9822(01)00581-4
52
MartínezL. (2014). Introducing the levinthal’s protein folding paradox and its solution. J. Chem. Educ.91 (11), 1918–1923. 10.1021/ed300302h
53
MarzolaP.MelzerT.PavesiE.Gil-MohapelJ.BrocardoP. S. (2023). Exploring the role of neuroplasticity in development, aging, and neurodegeneration. Brain Sci.13 (12), 1610. 10.3390/brainsci13121610
54
Maximilian SchäferA.ZimmermannH. G. (2006). “Recurrent neural networks are universal approximators,” in International conference on artificial neural networks (Springer), 632–640.
55
McDonnellM. D.WardL. M. (2011). The benefits of noise in neural systems: bridging theory and experiment. Nat. Rev. Neurosci.12 (7), 415–425. 10.1038/nrn3061
56
MetznerC.KraussP. (2022). Dynamics and information import in recurrent neural networks. Front. Comput. Neurosci.16, 876315. 10.3389/fncom.2022.876315
57
MetznerC.YamakouM. E.VoelklD.SchillingA.KraussP. (2024). Quantifying and maximizing the information flux in recurrent neural networks. Neural Comput.36 (3), 351–384. 10.1162/neco_a_01651
58
MolgedeyL.SchuchhardtJ.SchusterH. G. (1992). Suppressing chaos in neural networks by noise. Phys. Rev. Lett.69 (26), 3717–3719. 10.1103/physrevlett.69.3717
59
MossF.BulsaraA.ShlesingerM. F. (1993). Stochastic resonance in physics and biology. In Proceedings of the NATO Advanced Research Workshop. Karlsruhe, Germany. J. Stat. Phys., 70.
60
MossF.WardL. M.SannitaW. G. (2004). Stochastic resonance and sensory information processing: a tutorial and review of application. Clin. Neurophysiol.115 (2), 267–281. 10.1016/j.clinph.2003.09.014
61
NarangS.ElsenE.DiamosG.SenguptaS. (2017). Exploring sparsity in recurrent neural networks. arXiv Prepr. arXiv:1704.05119.
62
NatschlägerT.BertschingerN.LegensteinR. (2005). At the edge of chaos: real-time computations and self-organized criticality in recurrent neural networks. Adv. neural Inf. Process. Syst.17, 145–152.
63
PanagiotaropoulosT. I.KapoorV.LogothetisN. K.DecoG. (2013). A common neurodynamical mechanism could mediate externally induced and intrinsically generated transitions in visual awareness. PLoS One8 (1), e53833. 10.1371/journal.pone.0053833
64
RajanK.AbbottL. F.SompolinskyH. (2010). Stimulus-dependent suppression of chaos in recurrent neural networks. Phys. Rev. E82 (1), 011903. 10.1103/physreve.82.011903
65
RöderK.JosephJ. A.HusicB. E.WalesD. J. (2019). Energy landscapes for proteins: from single funnels to multifunctional systems. Adv. Theory Simulations2 (4), 1800175. 10.1002/adts.201800175
66
SandamirskayaY. (2013). Dynamic neural fields as a step toward cognitive neuromorphic architectures. Front. Neurosci.7, 276. 10.3389/fnins.2013.00276
67
SchillingA.GerumR.BoehmC.RasheedJ.MetznerC.MaierA.et al (2024). Deep learning based decoding of single local field potential events. NeuroImage120696. 10.1016/j.neuroimage.2024.120696
68
SchillingA.GerumR.MetznerC.MaierA.KraussP. (2022). Intrinsic noise improves speech recognition in a computational model of the auditory pathway. Front. Neurosci.16, 908330. 10.3389/fnins.2022.908330
69
SchillingA.SedleyW.GerumR.MetznerC.TziridisK.MaierA.et al (2023). Predictive coding and stochastic resonance as fundamental principles of auditory phantom perception. Brain146 (12), 4809–4825. 10.1093/brain/awad255
70
SchillingA.TziridisK.SchulzeH.KraussP. (2021). The stochastic resonance model of auditory perception: a unified explanation of tinnitus development, zwicker tone illusion, and residual inhibition. Prog. Brain Res.262, 139–157. 10.1016/bs.pbr.2021.01.025
71
SchrauwenB.BuesingL.LegensteinR. (2009). “On computational power and the order-chaos phase transition in reservoir computing,” in 22nd annual conference on neural information processing systems (NIPS 2008) (Vancouver, BC, Canada: NIPS Foundation), Vol. 21, 1425–1432.
72
SchueckerJ.GoedekeS.HeliasM. (2018). Optimal sequence memory in driven random networks. Phys. Rev. X8 (4), 041029. 10.1103/physrevx.8.041029
73
SietsmaJ.DowR. J. F. (1991). Creating artificial neural networks that generalize. Neural Netw.4 (1), 67–79. 10.1016/0893-6080(91)90033-2
74
SoléR. V.MiramontesO. (1995). Information at the edge of chaos in fluid neural networks. Phys. D. Nonlinear Phenom.80 (1-2), 171–180. 10.1016/0167-2789(94)00158-m
75
SongS.SjöströmP. J.ReiglM.NelsonS.ChklovskiiD. B. (2005). Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS Biol.3 (3), e68. 10.1371/journal.pbio.0030068
76
SteinR. B.GossenE. R.JonesK. E. (2005). Neuronal variability: noise or part of the signal?Nat. Rev. Neurosci.6 (5), 389–397. 10.1038/nrn1668
77
ToyoizumiT.AbbottL. F. (2011). Beyond the edge of chaos: amplification and temporal integration by recurrent networks in the chaotic regime. Phys. Rev. E84 (5), 051908. 10.1103/physreve.84.051908
78
Van LaarhovenP. J. M.AartsE. H. L.van LaarhovenP. J. M.AartsE. H. L. (1987). Simulated annealing. Springer.
79
WallaceE.MaeiH. R.LathamP. E. (2013). Randomly connected networks have short temporal memory. Neural Comput.25 (6), 1408–1439. 10.1162/neco_a_00449
80
WangX. R.LizierJ. T.ProkopenkoM. (2011). Fisher information at the edge of chaos in random boolean networks. Artif. life17 (4), 315–329. 10.1162/artl_a_00041
81
WardL. M. (2013). The thalamus: gateway to the mind. WIREs Cognitive Sci.4 (6), 609–622. 10.1002/wcs.1256
82
WatanabeT.LawsonR. P.WalldénY. S. E.ReesG. (2019). A neuroanatomical substrate linking perceptual stability to cognitive rigidity in autism. J. Neurosci.39 (33), 6540–6554. 10.1523/jneurosci.2831-18.2019
83
WengL. (2020). Exploration strategies in deep reinforcement learning. Lilianweng. Github. io/lil-log.
84
WolynesP. G. (2015). Evolution, energy landscapes and the paradoxes of protein folding. Biochimie119, 218–230. 10.1016/j.biochi.2014.12.007
85
WuY.KoutstaalW. (2020). Charting the contributions of cognitive flexibility to creativity: self-guided transitions as a process-based index of creativity-related adaptivity. PloS one15 (6), e0234473. 10.1371/journal.pone.0234473
86
ZhangC.ZhangD.StepanyantsA. (2021). Noise in neurons and synapses enables reliable associative memory storage in local cortical circuits. Eneuro8 (1), ENEURO.0302–20.2020. 10.1523/eneuro.0302-20.2020
87
ZwanzigR.SzaboA.BagchiB. (1992). Levinthal’s paradox. Proc. Natl. Acad. Sci.89 (1), 20–22. 10.1073/pnas.89.1.20
Summary
Keywords
recurrent neural networks (RNN), resonance, information processing, dynamics, noise
Citation
Metzner C, Schilling A, Maier A and Krauss P (2024) Recurrence resonance - noise-enhanced dynamics in recurrent neural networks. Front. Complex Syst. 2:1479417. doi: 10.3389/fcpxs.2024.1479417
Received
12 August 2024
Accepted
20 September 2024
Published
08 October 2024
Volume
2 - 2024
Edited by
Byungjoon Min, Chungbuk National University, Republic of Korea
Reviewed by
Yukio Pegio Gunji, Waseda University, Japan
Valentina Lanza, University of Le Havre, France
Updates
Copyright
© 2024 Metzner, Schilling, Maier and Krauss.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Patrick Krauss, patrick.krauss@fau.de
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.