Analysis of Structure and Dynamics in Three-Neuron Motifs

Recurrent neural networks can produce ongoing state-to-state transitions without any driving inputs, and the dynamical properties of these transitions are determined by the neuronal connection strengths. Due to non-linearity, it is not clear how strongly the system dynamics is affected by discrete local changes in the connection structure, such as the removal, addition, or sign-switching of individual connections. Moreover, there are no suitable metrics to quantify structural and dynamical differences between two given networks with arbitrarily indexed neurons. In this work, we present such permutation-invariant metrics and apply them to motifs of three binary neurons with discrete ternary connection strengths, an important class of building blocks in biological networks. Using multidimensional scaling, we then study the similarity relations between all 3,411 topologically distinct motifs with regard to structure and dynamics, revealing a strong clustering and various symmetries. As expected, the structural and dynamical distance between pairs of motifs show a significant positive correlation. Strikingly, however, the key parameter controlling motif dynamics turns out to be the ratio of excitatory to inhibitory connections.


Introduction
Recently, a number of projects seek to map the human connectome, aiming to connect its structure to function and behavior [1][2][3].However, even if the connectome would be known completely, it remains an unresolved problem how to translate this detailed structural data into meaningful information processing functions and algorithms [4].For instance, the connectome of Caenorhabditis elegans is known for decades and is with only 302 neurons orders of magnitudes smaller than the human brain.Nevertheless, even this relatively small system is not yet understood in terms of its dynamics, let alone at a functional level [5,6].
Moreover, the problem is complicated by the fact that very similar dynamics of a neural network at a macroscopic level might be realized by very different structures at the microscopic level [7].Therefore, an important step towards extracting function from structure is a tool to quantitatively compare different structures and dynamics.
In a neural network, all relevant structural information is encoded in a weight matrix, containing the mutual connection strength of all neurons [8][9][10][11].Quantifying the similarity of two weight matrices by standard measures, such as the sum of squared differences between corresponding matrix elements, is however not sufficient because of possible permutations of the neuron indices.Similarly, the dynamical properties of a neural network are encoded in a matrix of transition probabilities between all possible network states.As mentioned before, comparing the sum of squared differences between corresponding matrix elements fails in case of neuron permutations.
To solve this problem, we develop permutation-invariant metrics to quantify, both, the structural and dynamical distance between two given networks.We apply these distance metrics to so-called motifs, a class of small recurrent networks which have been shown to be fundamental building blocks of various complex networks [12], such as gene regulatory networks [13,14], the world wide web [12], and the human brain [15].
We exhaustively compute the structural and dynamical distances between all possible pairs of the 3411 different classes of three-neuron motifs with ternary connection strengths, resulting in two distance matrices with 3411 × 3411 entries each.Based on these matrices, we use multidimensional scaling [16][17][18][19][20] to visualize the structural and dynamical similarity relations between different motifs on a two-dimensional plane.
Remarkably, it turns out that the distribution of motifs, both in structural and dynamical 'space', is not uniform, but strongly clustered.Moreover, the position of a motif within structural and dynamical space correlates with the ratio of excitatory and inhibitory connections (balance) in the motif's connection matrix.By contrast, no such correlation is found regarding the density (ratio of non-zero connections and all possible connections) of the motif.

Three-neuron motifs
Our study is based on Boltzmann neurons [21] without bias.The total input z i (t) of neuron i at time t is calculated as: where y j (t − 1) is the binary state of neuron j at time t − 1 and w ij is the corresponding weight from neuron j to neuron i.The probability p i (t) of neuron i to be in state y i (t) = 1 is given by: where σ(x) is the logistic function We investigate the set of all possible network motifs that can be built from 3 Boltzmann neurons with ternary connections w ij ∈ {−1, 0, +1}, where self connections w ii are permitted (Figure 1a).In principle there are 3 9 = 19683 possible ternary 3 × 3 weight matrices.However, due to permutation of the neuron indices, not every matrix corresponds to a unique motif class.By exhaustive listing and classification, we find that under these conditions there are exactly 3411 distinct motif classes.For later convenience we label all motif classes with unique indices, which are derived from the corresponding weight matrices.

State transition matrices of motifs
Since every neuron can be in one of two binary states, a 3-node motif can be in 2 3 = 8 possible motif states.Given the momentary motif state and the weight matrix, the probabilities for all 8 successive motif states can be computed, thus defining the 8 × 8 state transition matrix of a Markov process (Figure 1b).All information theoretical properties of 3-neuron motifs, such as entropy or mutual information of succesive states, are determined by the state transition matrix.We therfore calculate the transition matrices for each of the 3411 motif classes.

Motif classes
A motif class A is defined as the set {A (m) : m = 1 . . .6} of weight matrices, which are all related to each other by index permutations, such as a i,j → a (m) i,j = a πm(i),πm(j) , where π m is the m-th permutation (Figure 2).

Unique labels of motif classes
The nine entries of the weight matrix W =   a b c d e f g h i   of one motif class are treated as a vector w = (abcdef ghi).the components of this vector are then treated as the digits of a number in the ternary system: It can be simplified to Here, w[0] equals the first entry of the vector w and the value of the sum is the name of the motif.Due to the possible entries w[i] ∈ {−1, 0, 1} the motif names range between "-9841" and "9841", starting with the motif with just "-1" as entries and finishing in the motif with just "1" as entries.Of course not every number in this range is assigned a motif class as there are in total only 3411 motif classes.This version of the formula is used because the motif class with just zeros as entries gets the name "0" and the names are approximately symmetrical around that motif class.Furthermore, in order to make the system more balanced, each motif class is represented by the weight matrix with the smallest absolute value of name among all of its permutations (Figure 2)

Structural distance between motif classes
The structural distance is calculated as follows (Figure 3): Given are two motif classes A, B. For each class we derive all 6 permutated weight matrices A (m) and B (n) .For each of the 36 pairs of weight matrices A (m) and B (n) , we compute a generalized Hamming distance ĥ, defined as the number of different ternary matrix elements: where δ x,y is the Kronecker symbol.The structural distance d str between motif classes matrices A, B is defined as the smallest of the above 36 Hamming distannces

Dynamical distance between motif classes
The structural distance is calculated as follows (Figure 4): For each motif classes A, we compute general features F (A), which can be scalars, vectors or matrices.In the case of matrix-like features F and G, the Euclidean distance is defined as To compute the dynamical distance d dyn (A, B) between two motif classes A and B, we derive all 36 pairs of features (e.g. the state transition matrix) from permuted weight matrices (F (A (m) ), F (B (n) )) and calculate the Euclidean distance d F (A (m) , F (B (n) ) of each pair.The dynamical distance d F between motif classes A, B is defined as the smallest of the 36 Euclidean distances

Multidimensional scaling
Multidimensional scaling [16][17][18][19][20] is a statistical method that maps proximity data on pairs of objects (i.e., data expressing the similarity or the dissimilarity of pairs of objects) into distances between points in a multidimensional space so that the pairwise distances follow the given N × N proximity matrix.We apply multidimensional scaling to visualize the relative positions of motifs in structural and dynamical space, according to their mutual distances d str and d dyn on a two dimensional plane.

Results
We have shown that there exist 3411 structurally distinct three-neuron motif classes with ternary connection strengths.Furthermore, we have computed the structural and dynamical distances between all possible pairs of these motif classes, resulting in two 3411 × 3411 distance matrices.A scatter plot of all pairwise dynamical and structural distances reveals that there is only a very weak positive correlation between dynamics and structure on a microscopic level (Figure 5).In a next step, we have used multidimensional scaling [16][17][18][19][20] to identify and visualize the structural and dynamical similarity relations between motif classes on a two-dimensional plane.Strikingly, we find that the distribution of motif classes in both, structural and dynamical 'space' is not uniform, but instead reveals strong clustering (Figure 6).Furthermore, the structural distribution (Figure 6a, c) reveals a 6-fold rotation symmetry, which might be due to the 6 possible permutations of 3-neuron motifs.
Although there is no strong correlation between dynamics and structure on a microscopic level, the position of a motif within structural and dynamical space strongly correlates with a macroscopic parameter, namely the ratio of excitatory and inhibitory connections (balance) in the motif's connection matrix (Figure 6a, b).By contrast, no such correlation is found regarding another macroscopic parameter, the density (ratio of non-zero connections and all possible connections) of the motif (Figure 6c, d).

Discussion
The relation of structure and function is a long-standing topic in biology [22][23][24][25][26][27].On the one hand, the micro-structure of a biological system determines the set of possible functions that this system can serve.On the other hand, human observers may not be able to deduce the function of a system from its structure alone: even if we know all neural connection strengths in some sub-network of the animal brain, as well as all its input and output signals, the specific purpose of this sub-network within the whole of the organism may remain elusive [4][5][6].Indeed, 'function' is not a property of the isolated subsystem alone, but can only be defined in the context of its embedding global system.For this reason, we focus in this work not on the function of neural systems, but on their dynamics -a property that is completely determined by the network structure and, if present, the system's input signals.
An additional advantage of this approach is that dynamics, just as structure, can be conveniently expressed in the form of matrices.Based on these matrices, we have developed suitable metrics that measure the distance of two neural networks in structural or dynamical space respectively.Using this tool, we can investigate how sensitive network dynamics reacts to small changes in network structure.Robustness with respect to structural changes is crucial in biological brains, as the synaptic weights cannot be adjusted with extremely high accuracy [28][29][30].
For the case of isolated three-neuron networks, we have found that the question of robustness has no definitive answer on the microscopic level of individual neuron connection strength: a small change in the connection weights can have, both, small and large dynamical consequences.By contrast, a much clearer correlation is found between certain statistical (macroscopic) properties of a network's weight matrix and its dynamics.In particular, the ratio of excitatory to inhibitory connections affects network dynamics very strongly, while the ratio of non-zero connections is much less important.
This result suggests that biological brains might control the overall statistics of the connection weights in order to drive the neural networks into a desired dynamical regime.Learning would then be just a fine-tuning of the weights with respect to the required information processing task.
Future work will need to extend our methods to larger neural networks, and to networks with continuous connection strengths between the neurons.In this case, it will no longer be feasible to enumerate all possible neuron permutations in order to compute the minimum structural or dynamical distance.However, we expect that a Monte-Carlo based minimization of the distance will yield sufficiently accurate results.Possible entries in the 3 × 3 weight matrix of a motif are −1 (blue), 0 (white) and +1 (red).Shown are all possible permutations of topologically equivalent motifs for two arbitrary chosen cases (a, b).Each motif class is assigned a unique label (green numbers), as described in the method section.For each of the 36 combinations, the generalized Hamming distance ĥ (green numbers) is computed.Black and white matrices indicate the Hamming distances between corresponding matrix elements.As described in the method section, the structural distance is defined as the minimum of all 36 generalized Hamming distances (green numbers with yellow background).5, rows and columns contain the 6 possible permutations of two given motif classes.For each permutation the corresponding state transition matrix is calculated (grey shaded matrices).Subsequently, for each of the 36 combinations, the Euclidean distance between each pair of state transition matrices is calculated (green numbers).As described in the method section, the dynamical distance is defined as the minimum of all 36 Euclidean distances (green number with yellow background).

Figure 1 :
Figure 1: Motifs of three couples Boltzmann neurons.Each motif is characterized by a 3 × 3 weight matrix W (a), defining the connection strength between the neurons.There are 2 3 = 8 possible states Y = 0 . . .7 for each motif.The transition probabilities between these states are summarized in a 8 × 8 state transition matrix (b).

Figure 2 :
Figure 2: Unique labeling of motif classes.Possible entries in the 3 × 3 weight matrix of a motif are −1 (blue), 0 (white) and +1 (red).Shown are all possible permutations of topologically equivalent motifs for two arbitrary chosen cases (a, b).Each motif class is assigned a unique label (green numbers), as described in the method section.

Figure 3 :
Figure 3: Structural distance between motif classes.Rows and columns show the 6 possible permutations of two given motif classes.For each of the 36 combinations, the generalized Hamming distance ĥ (green numbers) is computed.Black and white matrices indicate the Hamming distances between corresponding matrix elements.As described in the method section, the structural distance is defined as the minimum of all 36 generalized Hamming distances (green numbers with yellow background).

Figure 4 :
Figure 4: Dynamical distance between motif classes.As in Figure5, rows and columns contain the 6 possible permutations of two given motif classes.For each permutation the corresponding state transition matrix is calculated (grey shaded matrices).Subsequently, for each of the 36 combinations, the Euclidean distance between each pair of state transition matrices is calculated (green numbers).As described in the method section, the dynamical distance is defined as the minimum of all 36 Euclidean distances (green number with yellow background).

Figure 5 :
Figure 5: Scatterplot of all pairwise dynamical and structural distances.Each point (d str , d dyn ) represents the relation between structural distance d str and dynamical distance d dyn of the same two motifs.There is only a very weak positive correlation between structure and dynamics.

Figure 6 :
Figure 6: Motif distribution in structural (a, c) and dynamical (b, d) space.Plots are color coded according to balance (a, b) and density (c, d) parameters.The structural distribution reveals a 6-fold rotation symmetry due to the 6 possible permutations of 3-neuron motifs.In addition, motifs are ordered linearly according to the balance parameter, in both structural (a) and dynamical (b) space.By contrast, motifs are not ordered with respect to the density parameter (c, d).