Skip to main content

Hierarchy and dynamics in neural networks

136kviews
Hierarchy and Dynamics of Neural Networks
Editorial23 August 2010
Hierarchy and Dynamics of Neural Networks
Marcus Kaiser, Claus C Hilgetag and Rolf Kötter

Hierarchy is a central feature in the organisation of complex biological systems and particularly the structure and function of neural networks. While other aspects of brain connectivity, such as regional specialization, modularity, or motif composition, have already been discussed extensively (Bullmore and Sporns, 2009; Sporns et al., 2004), no comprehensive analysis has been presented so far on the role of hierarchy and its connection to brain dynamics. Nonetheless, recent discussions among many of our colleagues have shown an increasing interest in the subject of hierarchy. This topic is an emerging key question in neuroscience, as well as generally in the field of network science, due to its links with concepts of control, efficiency, and development across scales (Barthélemy et al., 2004; Breakspear and Stam, 2005; Dehmer et al., 2008; Ravasz and Barabasi, 2003; Zhou et al., 2006).

“Hierarchy” may be understood in several different ways, and can apply to topological, spatial, temporal as well as functional properties of neural networks. The papers of this Special Topic on hierarchy and dynamics reflect this conceptual diversity.

One interpretation of hierarchy is that of a processing sequence. For example, the popular organizational scheme of the primate visual system (Felleman and van Essen, 1991) implies a sequential ordering of visual cortical areas from the visual sensory periphery to “higher-level” areas involved in abstract aspects of vision. This hierarchical concept is formalized by Krumnack, Reid, Wanke, Bezgin and Kötter (“Criteria for optimizing cortical hierarchies with continuous ranges”) and expanded from a recent paper by Reid et al. (2009). The authors re-analyse the anatomical constraints for the hierarchical sorting of visual areas, using linear optimization and mixed integer programming, and demonstrate that there are multiple optimal solutions for visual hierarchies, as well as several alternative definitions of optimality. For instance, optimal hierarchies can be based on minimizing the number of violated constraints, or minimizing the maximal size of a constraint violation (cf. Hilgetag et al., 2000), broadening the perspective for the interpretation of the anatomical data.

Another widely used definition of hierarchy is that of a repeated encapsulation of smaller elements in larger ones (Kaiser et al., 2007a; Robinson et al., 2009), an organization which may also be characterized as recursive or fractal (Sporns, 2006). In that sense, neural networks show a self-similar hierarchical organization across a wide range of metric or non-metric scales. These scales may be spatial, ranging from the lobes of the brain to cortical mini-columns; temporal, stretching from plasticity and learning processes taking days and longer to neuronal firing at the millisecond scale; or topological, containing small functional elements such as “canonical circuits” (Douglas and Martin, 2004) in larger modules such as the “visual cortex.”

The Special Topic contains several examples of such encapsulated hierarchies.
For instance, Meunier, Lambiotte, Fornito, Ersche and Bullmore (“Hierarchical modularity in human brain functional networks”) present techniques for the rapid detection of a hierarchy of encapsulated modules in resting-state fMRI data. They analyzed networks composed of 1,800 regional nodes, extracted from neuroimaging data for 18 human subjects, and found a good degree of similarity between the network hierarchies for different brains. Moreover, out of five modules at the highest level, the occipital modules demonstrated less sub-modular organization than modules comprising regions of multimodal association cortex. Connector nodes and hubs, with a key role in inter-modular connectivity, were also concentrated in association cortical areas. The study demonstrates the feasibility of extracting large-scale hierarchical networks from experimental imaging data, and prepares the ground for characterizing brain function by advanced network analyses.

Modules in hierarchical networks may be overlapping, rather than be cleanly delineated. Moreover, the individual nodes may differ by the topological “reach” that they have across the network. While most nodes have relatively few connections, some regions (such as amygdala and hippocampus in the cat, or the lateral intraparietal area (LIP) and area 7 in the macaque brain) are connected to many nodes of the network and thus form hubs (Kaiser et al., 2007b). Such hubs can be further distinguished into provincial (intra-modular) hubs or connector (inter-modular) hubs (Sporns et al., 2007). Zamora-López, Zhou and Kurths (“Cortical hubs form a module for multisensory integration on top of the hierarchy of cortical networks”) expand such previous approaches, and identify a new element in the cat cortical connectivity network, a hub module, which consists of network nodes that possess many connections with the rest of the network as well as each other. This set of nodes forms a topologically central module of the cortex that appears to be essential for integrating multisensory information.

Several articles in the Special Topic explore the dynamic implications of hierarchical modular network architectures. Kaiser and Hilgetag (“Optimal hierarchical modular topologies for producing limited sustained activation of neural networks”) investigate the influence of the number of hierarchical levels (scales), as well as sub-modules at each level (granularity), on the spreading of activity in hierarchical modular networks of different sizes, using a minimal dynamic node model. In particular, they characterize the conditions leading to the biologically relevant case of limited sustained activity in which activity persists between the extremes of dying out or activating the whole network (Kaiser et al., 2007a). For different network sizes, limited sustained activity is best supported when the average number of connections per node remains similar, and the number of hierarchical levels or modules per level increases. This observation indicates that dynamic constraints may contribute to the evolution of network complexity in brain architecture.

In their work, Jarvis, Rotter and Egert (“Extending stability through hierarchical clusters in Echo State Networks”) also find that increased intricacy of network structure aids network dynamics. They demonstrate that the stability of Echo State Networks (Jaeger and Haas, 2004) is potentially enhanced, as indicated by the range of spectral radius values, when the networks are structured in a multi-modular, hierarchical way. The more clearly the ESNs are structured, the larger the range of spectral radius values, while increasing inter-cluster connectivity decreases the maximal spectral radius. The finding suggests that insights into the organization of biological networks also have the potential to improve the functioning of networks for technical applications.

An important feature of neural systems is sudden changes in their dynamics (a drastic example of such phase transitions is the shift from normal brain activity to epileptic seizures). In a previous paper, Müller-Linow et al. (2008) observed that two different dynamic behaviors may emerge from hierarchical networks: waves propagating from central nodes and module-based synchronization. In the present issue, Hütt and Lesne (“Interplay between topology and dynamics in excitation patterns on hierarchical graphs”) analyze more formally how excitable systems can switch from one of such states to the other. In addition to a mean-field model simulation, a formalism is introduced in which excitation waves are described as avalanches.

Shifting the perspective from network dynamics to network function, the flow of information in hierarchical networks is used in the brain for the processing of external or internal signals. Rodrigues and Costa (“Signal propagation in cortical networks: a digital signal processing approach”) show that signal flow among network nodes can be characterized by a finite impulse response (FIR) filter. With such an approach, filters underlying the cat and macaque cortical organization are found to be low-pass, allowing signal processing to be summarized through respective cut-off frequencies. Furthermore, filtering intensity varies between network modules, and regions involved in object recognition tend to present the highest cut-off frequencies for both the cat and macaque networks.

Hierarchies can also be seen in temporal aspects of brain activity. Natural stimuli, such as speech, possess features at different temporal scales. Therefore, models of speech recognition should be able to represent slowly changing neuronal states that encode trajectories of faster signals. Kiebel, Daunizeau, and Friston (“Perception and hierarchical dynamics”) present a mathematical approach that assumes that sensory input is generated by a hierarchy of attractors in a dynamic system. Future applications of this approach might emerge from modeling perception as non-autonomous recognition dynamics enslaved by autonomous hierarchical dynamics in the sensorium.

The papers presented here offer an exciting glimpse into future directions of the field of hierarchical neural networks. But they also demonstrate that we still need a better understanding of the different kinds of network hierarchies, paralleled by the development of suitable analysis techniques. Most importantly, an improved understanding is required of how the different aspects of topological, spatial, temporal and functional hierarchy in the brain are related to each other.

19,072 views

109 citations

An essential requirement for the representation of functional patterns in complex neural networks, such as the mammalian cerebral cortex, is the existence of stable regimes of network activation, typically arising from a limited parameter range. In this range of limited sustained activity (LSA), the activity of neural populations in the network persists between the extremes of either quickly dying out or activating the whole network. Hierarchical modular networks were previously found to show a wider parameter range for LSA than random or small-world networks not possessing hierarchical organization or multiple modules. Here we explored how variation in the number of hierarchical levels and modules per level influenced network dynamics and occurrence of LSA. We tested hierarchical configurations of different network sizes, approximating the large-scale networks linking cortical columns in one hemisphere of the rat, cat, or macaque monkey brain. Scaling of the network size affected the number of hierarchical levels and modules in the optimal networks, also depending on whether global edge density or the numbers of connections per node were kept constant. For constant edge density, only few network configurations, possessing an intermediate number of levels and a large number of modules, led to a large range of LSA independent of brain size. For a constant number of node connections, there was a trend for optimal configurations in larger-size networks to possess a larger number of hierarchical levels or more modules. These results may help to explain the trend to greater network complexity apparent in larger brains and may indicate that this complexity is required for maintaining stable levels of neural activation.

11,290 views

132 citations

ORIGINAL RESEARCH
31 March 2010
Criteria for optimizing cortical hierarchies with continuous ranges
Antje Krumnack, 3 more and Rolf Kötter
In a recent paper (Reid et al.; 2009, NeuroImage) we introduced a method to calculate optimal hierarchies in the visual network that utilizes continuous, rather than discrete, hierarchical levels, and permits a range of acceptable values rather than attempting to fit fixed hierarchical distances. There, to obtain a hierarchy, the sum of deviations from the constraints that define the hierarchy was minimized using linear optimization. In the short time since publication of that paper we noticed that many colleagues misinterpreted the meaning of the term "optimal hierarchy". In particular, a majority of them were under the impression that there was perhaps only one optimal hierarchy, but a substantial difficulty in finding that one. However, there is not only more than one optimal hierarchy but also more than one option for defining optimality. Continuing the line of this work we look at additional options for optimizing the visual hierarchy: minimizing the number of violated constraints and minimizing the maximal size of a constraint violation using linear optimization and mixed integer programming. The implementation of both optimization criteria is explained in detail. In addition, using constraint sets based on the data from Felleman and Van Essen, optimal hierarchies for the visual network are calculated for both optimization methods.

5,783 views

17 citations

The idea that complex systems have a hierarchical modular organization originates in the early 1960s and has recently attracted fresh support from quantitative studies of large scale, real-life networks. Here we investigate the hierarchical modular (or “modules-within-modules”) decomposition of human brain functional networks, measured using functional magnetic resonance imaging (fMRI) in 18 healthy volunteers under no-task or resting conditions. We used a customized template to extract networks with more than 1800 regional nodes, and we applied a fast algorithm to identify nested modular structure at several hierarchical levels. We used mutual information, 0 < I < 1, to estimate the similarity of community structure of networks in different subjects, and to identify the individual network that is most representative of the group. Results show that human brain functional networks have a hierarchical modular organization with a fair degree of similarity between subjects, I=0.63. The largest 5 modules at the highest level of the hierarchy were medial occipital, lateral occipital, central, parieto-frontal and fronto-temporal systems; occipital modules demonstrated less sub-modular organization than modules comprising regions of multimodal association cortex. Connector nodes and hubs, with a key role in inter-modular connectivity, were also concentrated in association cortical areas. We conclude that methods are available for hierarchical modular decomposition of large numbers of high resolution brain functional networks using computationally expedient algorithms. This could enable future investigations of Simon's original hypothesis that hierarchy or near-decomposability of physical symbol systems is a critical design feature for their fast adaptivity to changing environmental conditions.

40,271 views

697 citations

Sensory stimuli entering the nervous system follow particular paths of processing, typically separated (segregated) from the paths of other modal information. However, sensory perception, awareness and cognition emerge from the combination of information (integration). The corticocortical networks of cats and macaque monkeys display three prominent characteristics: (i) modular organisation (facilitating the segregation), (ii) abundant alternative processing paths and (iii) the presence of highly connected hubs. Here, we study in detail the organisation and potential function of the cortical hubs by graph analysis and information theoretical methods. We find that the cortical hubs form a spatially delocalised, but topologically central module with the capacity to integrate multisensory information in a collaborative manner. With this, we resolve the underlying anatomical substrate that supports the simultaneous capacity of the cortex to segregate and to integrate multisensory information.

15,504 views

354 citations

In a recent publication [Müller-Linow et al. (2008) PLoS Computational Biology 4, 1000190] two types of correlations between network topology and dynamics have been observed: waves propagating from central nodes and module-based synchronization. Remarkably, the dynamic behavior of hierarchical modular networks can switch from one of these modes to the other as the level of spontaneous network activation changes. Here we attempt to capture the origin of this switching behavior in a mean-field model as well in a formalism, where excitation waves are regarded as avalanches on the graph.

7,247 views

26 citations

In this paper, we suggest that perception could be modeled by assuming that sensory input is generated by a hierarchy of attractors in a dynamic system. We describe a mathematical model which exploits the temporal structure of rapid sensory dynamics to track the slower trajectories of their underlying causes. This model establishes a proof of concept that slowly changing neuronal states can encode the trajectories of faster sensory signals. We link this hierarchical account to recent developments in the perception of human action; in particular artificial speech recognition. We argue that these hierarchical models of dynamical systems are a plausible starting point to develop robust recognition schemes, because they capture critical temporal dependencies induced by deep hierarchical structure. We conclude by suggesting that a fruitful computational neuroscience approach may emerge from modeling perception as non-autonomous recognition dynamics enslaved by autonomous hierarchical dynamics in the sensorium.

12,945 views

106 citations

This work reports a digital signal processing approach to representing and modeling transmission and combination of signals in cortical networks. The signal dynamics is modeled in terms of diffusion, which allows the information processing undergone between any pair of nodes to be fully characterized in terms of a finite impulse response (FIR) filter. Diffusion without and with time decay are investigated. All filters underlying the cat and macaque cortical organization are found to be of low-pass nature, allowing the cortical signal processing to be summarized in terms of the respective cutoff frequencies (a high cutoff frequency meaning little alteration of signals through their intermixing). Several findings are reported and discussed, including the fact that the incorporation of temporal activity decay tends to provide more diversified cutoff frequencies. Different filtering intensity is observed for each community in those networks. In addition, the brain regions involved in object recognition tend to present the highest cutoff frequencies for both the cat and macaque networks.

7,662 views

5 citations