Skip to main content

REVIEW article

Front. Psychiatry, 28 October 2021
Sec. Mood Disorders
https://doi.org/10.3389/fpsyt.2021.719490

A Narrative Review of Network Studies in Depression: What Different Methodological Approaches Tell Us About Depression

  • 1University of Groningen, University Medical Center Groningen, Department of Psychiatry, Interdisciplinary Center Psychopathology and Emotion regulation, Groningen, Netherlands
  • 2University of Groningen, University Medical Center Groningen, Department of Psychiatry, Rob Giel Research Center, Groningen, Netherlands

The network theory of psychopathology proposes that mental disorders arise from direct interactions between symptoms. This theory provides a promising framework to understand the development and maintenance of mental disorders such as depression. In this narrative review, we summarize the literature on network studies in the field of depression. Four methodological network approaches are distinguished: (i) studies focusing on symptoms at the macro-level vs. (ii) on momentary states at the micro-level, and (iii) studies based on cross-sectional vs. (iv) time-series (dynamic) data. Fifty-six studies were identified. We found that different methodological approaches to network theory yielded largely inconsistent findings on depression. Centrality is a notable exception: the majority of studies identified either positive affect or anhedonia as central nodes. To aid future research in this field, we outline a novel complementary network theory, the momentary affect dynamics (MAD) network theory, to understand the development of depression. Furthermore, we provide directions for future research and discuss if and how networks might be used in clinical practice. We conclude that more empirical network studies are needed to determine whether the network theory of psychopathology can indeed enhance our understanding of the underlying structure of depression and advance clinical treatment.

Highlights

- This narrative review evaluates empirical network studies on depression.

- We summarize findings regarding comorbidity, centrality, and network connectivity.

- Different network methodological approaches yield inconsistent findings.

- Important challenges for future network research are outlined.

- Empirical evidence is not yet suggestive of the use of networks as a clinical tool.

Introduction

The network theory of psychopathology has gained increasing popularity in recent years (1, 2). This theory postulates that a psychiatric syndrome, such as depression, arises not because of the presence of a latent cause, but rather due to a process in which psychological states or symptoms trigger each other. The theory assumes that this process will eventually result in a cluster of co-occurring symptoms, which we call mental disorders. Empirical research regarding the application of the network theory and its techniques is exponentially growing (3). The popularity of the network theory is also expressed in the eagerness of healthcare professionals to apply these ideas within clinical practice (47). Thus, the field is in urgent need of a comprehensive evaluation of the findings of empirical network studies.

Such an evaluation should take into account that different methodological approaches have been used within each network study, each yielding their own interpretations and implications. These methodological approaches are different, but complementary, operationalizations of how network dynamics may explain the development and maintenance of psychopathology. Networks consists of nodes and edges (the connections between nodes). In networks on psychopathology, nodes have so far signified either clinical symptoms that supposedly operate at the macro-level (1), or momentary affective states operating at a much smaller timescale, termed the micro-level (8). Typically, the macro-level network approach is used to investigate between-person relationships among symptoms cross-sectionally at a given time point in a large group of individuals. The micro-level network approach, on the other hand, is often used to examine the processes underlying the development of clinical symptoms by studying the dynamics between everyday life momentary affective states.

In addition to differences in macro- vs. micro-level approaches, network studies also differ in the type of data on which they base their network analyses, namely: cross-sectional (data with one assessment per participant) vs. time-series data (data with multiple assessments per participant). Often, cross-sectional network studies examine symptoms on the macro-level, whereas dynamic network studies use time-series data to examine momentary affect at the micro-level. Although it might be expected that the distinction between micro- and macro-level experiences is rather a continuum than distinct categories, for the purposes of this narrative review, we have placed each network study in one of four quadrants: cross-sectional1 vs. dynamic associations and micro- vs. macro-level approaches (see Figure 1).

FIGURE 1
www.frontiersin.org

Figure 1. Overview of characteristics of the four methodological approaches to network theory: micro-level affective states vs. macro-level symptoms, based on cross-sectional vs. dynamic (time-series) data. The number of studies are indicated in the corner of each quadrant (total N = 56). Note that one study constructed both a cross-sectional and a dynamic network, and is therefore referenced twice in this Figure.

Distinguishing between these four methodological approaches is relevant as results of these varying network approaches should be interpreted differently (911). Most cross-sectional network studies utilize data from symptom questionnaires to estimate between-person correlations of symptoms at one point in time for a given group of individuals. The associations between variables in such a network demonstrate, at the group level, the probability that these variables tend to occur together (controlling for all other associations). If, for example, those who worry more than others also suffer from sad moods more than others, then the connection between these symptoms will be stronger in such a network. In contrast, networks utilizing time-series data are typically based on the within-person dynamic associations among momentary affective states. Whereas most dynamic networks are based on temporal associations, some studies have also examined the contemporaneous (concurrent) associations. Connections in dynamic networks based on temporal associations show how changes relative to a person's average in one variable follow changes in the other variables within that person. If, for example, individuals start to worry more than usual every time they feel sadder than they usually do, these symptoms will be more strongly connected in the network (see Figure 2). It is important to note here, however, that temporality does not imply causality; temporal associations could also be the result of an unknown third variable. In order to construct dynamic networks, time-series analyses techniques, such as (multilevel) vector auto regression (VAR) analyses (12, 13), are often used. Thus, whereas cross-sectional models mainly express something about the coexistence of different variables at one moment in time at the group-level, dynamic network models say something about how these variables relate to each other over time, within individuals.

FIGURE 2
www.frontiersin.org

Figure 2. Examples of a cross-sectional and a temporal (dynamic) network. In the cross-sectional network, feeling down is the most central node (0.1 + 0.4 + 0.5 = 1.0); it has the strongest connections with the other nodes. In the temporal network, worry has the highest outgoing centrality; this node most strongly predicts other nodes (0.25 + 0.2 = 0.45). The node with the greatest incoming centrality is feeling down (0.1 + 0.2 = 0.3), because this node is most strongly predicted by other nodes.

Unfortunately, when evaluating the meaning of findings brought forward by network studies, network methodologies are often not clearly separated. This complicates our understanding of the meaning of these findings for the network theory. Furthermore, overviews of findings of empirical network studies in individuals suffering from depression are lacking. Two recent reviews have provided a first overview of the network literature on psychopathology in general (3, 14). Such systematic reviews are important because they advance our understanding of how the network theory has, thus far, been operationalized. However, both reviews have not explicitly summarized and compared findings between the different methodological approaches. Furthermore, as these reviews did not solely focus on network studies in depression, a comprehensive evaluation and discussion thereof was beyond their scope. Therefore, we aimed to address this gap in the literature by providing a clear and in-depth overview of the findings of network studies in depression while distinguishing between the aforementioned methodological approaches. Subsequently, we discuss the current status of network research and challenges for both future research and application in clinical practice.

Methods

We conducted a systematic search in PUBMED and PsycINFO to identify empirical studies that have applied network analysis to investigate (risk of) depression. We searched for papers published before October 2020, in which abstracts included the terms; (i) “depression or depressive or MDD or major depression” or “psychopathology,” combined with one of the following terms “network” or “impulse response” or “vector autoregression” or “VAR” or “qgraph,” or “network approach,” or “network intervention” and (ii) “affect” or “mental states” or “emotion” or “experience sampling” or “momentary assessment” or “depressive symptoms” or “network analysis.” Papers with the terms “fMRI” or “connectomics” or “functional connectivity” or “mouse” or “rat” or “social network” were excluded.

So far, network studies have primarily focused on investigating the comorbidity of depression with other mental disorders, node centrality, and network connectivity. Therefore, papers had to meet the following inclusion criteria: (i) the paper reported empirical results on a network of symptoms or affect states, with at least three nodes pertaining to depressive constructs; (ii) the sample population was selected based on their experience of depressive symptoms in the past, present, or future, and (iii) the paper included results on comorbidity, node centrality and/or network connectivity. As a consequence of this focus on depression, some well-cited network papers fell out of the final selection (2, 15, 16). Furthermore, given that our narrative review focuses on the relationships between symptoms and/or affect states, and the inclusion of other nodes might confound estimates of comorbidity, centrality or connectivity, we excluded papers that included contextual factors as nodes such as treatment, genes, coping strategies, or activities (1719).

Results

Our search resulted in 56 network papers spanning the years 2014–2020. Table 1 provides an overview of studies and the methodological approach used to construct networks. Most of these studies used network nodes representing macro-level depressive symptoms (1), assessed with retrospective measures (assessing symptoms of the past week or weeks) via diagnostic interviews or questionnaires (n = 45 see Figure 1). Of these studies at the macro-level, most (N = 40) constructed cross-sectional networks, although there were also five dynamic studies at the macro-level. Eleven studies used momentary affective states as nodes in the network. These states were assessed with frequent questions, often multiple times a day, using the experience sampling methodology (ESM). With ESM, participants indicate their affect states in that particular moment, such as sadness, irritation, or cheerfulness. We refer to this as network research at the “micro-level” because the examined processes occur on a much smaller time scale (8). Of the eleven micro-level studies, the majority examined dynamic temporal associations (N = 8). One study compared three networks: one based on contemporaneous associations, one based on temporal connections, and one based on cross-sectional micro-level data. Another study also constructed both a temporal as well as a contemporaneous network. The final micro-level study solely examined contemporaneous associations in two separately estimated networks. In the following paragraphs, we will synthesize the results of the network studies regarding comorbidity, centrality, and connectivity, while distinguishing between the different methodological approaches.

TABLE 1
www.frontiersin.org

Table 1. Overview of network studies on depression (N = 56).

Comorbidity

Cramer and colleagues (2) were the first to construct a cross-sectional network with clinical symptoms as nodes (macro-level) to map comorbidity of psychiatric symptoms of depression and generalized anxiety disorder (GAD). Although conducted in a general population sample, the study convincingly demonstrated that symptoms attributed to one diagnostic label (e.g., depression) often co-occur with symptoms associated with differing diagnoses (e.g., GAD). Subsequent studies have since then investigated the comorbidity of depression with other forms of psychopathology. With two exceptions, all studies (N = 21) were cross-sectional network studies at the macro-level. Depressive symptoms were most often investigated alongside symptoms of posttraumatic stress disorder (PTSD) and anxiety disorders such as GAD. Other forms of psychopathology that have also been explored include obsessive-compulsive disorder (OCD), prolonged grief disorder, autism, alcohol use disorder (AUD), and somatic symptomatology (see the heading “Comorbidity” in Table 1 for an overview of these studies). Together, these studies confirm that symptoms of depression often co-occur with symptoms of other disorders, which strengthens the conceptualization of psychopathology as transcending boundaries of diagnostic categories (74).

Relatively recently, studies have started to identify so-called “bridge symptoms.” Originally, a bridge symptom was described as a symptom shared by two syndromic clusters (e.g., sleep disturbance in MDD and GAD) (2). However, this definition was later adapted to signify a symptom of one syndromic cluster that has strong connections to symptoms of another cluster (75). Symptoms with high bridge centrality have the strongest connections to nodes of symptoms belonging to another cluster (75, 76). Based on the network theory of psychopathology, it can be hypothesized that comorbidity arises because a bridge symptom of one disorder (e.g., PTSD) also activates symptoms of the other disorder (e.g., depression). Thereafter, activation of symptoms can further expand within the cluster of depression symptoms. Although difficult to directly test this hypothesis, several studies have attempted to identify such bridge symptoms. Notably, the comparison of these studies is complicated because they have operationalized bridge symptoms in different ways: some followed the original operationalization as proposed by Jones and colleagues (2019 preprint, publication in 2021), whereas others relied on visual inspection or other more general centrality measures. For clarity, we will only discuss studies here that have followed the original operationalization, because this stays closest to the original bridge symptoms hypothesis based on network theory (75, 76).

In macro-level studies (N = 8), depressive symptoms that were most often identified as a bridge between depression and other disorders were sadness, followed by loss of interest and/or pleasure (anhedonia), energy loss or fatigue, and guilt. It is notable that only two dynamic network studies have been conducted on identifying bridge symptoms between symptom clusters. Such dynamic network studies could further illuminate whether within-person connections between symptoms of different diagnoses might help to explain comorbidity. A first dynamic network study at the macro-level examined the contemporaneous associations among daily reported symptoms for 10 patients separately and reported large individual differences in identified bridge symptoms (41). The second dynamic network study examined temporal associations at the micro-level in patients diagnosed with depression and anxiety disorder. No evidence for bridge symptoms in overlapping momentary states (irritated and worrying) was found (42). Thus, methodological approaches regarding bridge symptoms have so far yielded inconsistent findings. Further studies investigating bridge symptoms are needed to further validate the bridge symptom hypothesis within persons.

Centrality

Research has further focused on the centrality of nodes in networks. Centrality measures have been suggested to indicate how influential nodes are in transmitting information to other nodes within the network (77). Most studies have focused on three centrality measures: strength, betweenness, and closeness. However, the latter two have been argued to be unsuitable as measures of node importance in psychological networks (78). Strength centrality, on the other hand, is easy to interpret and is therefore used most often. Strength centrality reflects the weight of connections between a certain node and all other nodes in the network. In cross-sectional networks, this is calculated by summing the absolute weights of all connections of that node. In temporal networks, a distinction can also be made between outgoing and incoming connections of a node (see Figure 2 for further explanation).

Again, most network studies investigated centrality in cross-sectional networks at the macro-level (N = 17, see the heading “Centrality” in Table 1), with depressive symptoms as nodes in the network. Four symptoms were most often identified as highly central symptoms in the network, namely: energy loss (or fatigue), anhedonia (loss of pleasure or interest), depressed mood (or sadness), and concentration problems. One of three studies at the micro-level studying contemporaneous associations also indicated sadness as one of the most central symptoms in the networks. As such, it seems that sadness is often associated with other symptoms of depression, both at weekly and momentary levels. There are multiple interpretations of this result. Sadness and anhedonia may be central because they trigger other symptoms of depression. However, another explanation is that sadness and anhedonia are considered the core symptoms of depression and therefore necessary to be able to receive the diagnosis. They may therefore have the greatest chance of frequently coexisting with other symptoms. This speculation is plausible given that many cross-sectional studies included individuals who met the clinical diagnosis of depression, for whom by definition sadness or anhedonia should be present. A similar explanation may be applicable for the high centrality of energy loss and concentration problems in depressive symptom networks.

In dynamic networks based on temporal associations we can distinguish between incoming and outgoing centrality. That is, whether a symptom predicts other symptoms at a later time (outgoing) or is itself predicted by other symptoms at an earlier moment in time (incoming). In particular, symptoms with high outgoing centrality might be interesting in the context of possible interventions within a network structure. This would be the case when one would assume that a node with many outgoing connections strongly influences the rest of the network. Comparing dynamic network studies to cross-sectional ones, we see both similarities and differences. Four studies (see Table 1) looked at the centrality in temporal networks of macro-level depressive symptoms assessed both daily and weekly. In these studies, suicidality often ranked among the symptoms with the highest outgoing centrality, followed by energy loss, similar to cross-sectional networks. In terms of incoming connections, these studies found that suicidality and anhedonia were the most central. Here, again, two of the four symptoms correspond to the most central symptoms seen in the cross-sectional studies. The idea that sadness would be central, in the sense that this symptom may have a major role in triggering other symptoms, is not supported in these dynamic macro-level network studies based on temporal associations.

The dynamic studies examining temporal associations at the micro-level consisted of three group-based studies and three single-subject network analyses. The findings of the three group-based studies were all consistent concerning outgoing centrality of momentary affect: all indicated cheerfulness, or positive affect in general, as the most central node. This means that short-term changes in cheerfulness or positive affect most often preceded short-term changes in other affective states within persons. One of the single-subject studies even showed that as relapse into depressive symptoms drew closer, positive affect became increasingly central in this individual's network (66). The finding that cheerfulness and positive affect were consistently identified as the most influential nodes is interesting. This could indicate that, for many patients, intervening on this node by increasing positive affect could have the greatest impact on the rest of their affective states. Whether this will also be the case in clinical practice still needs to be tested. This finding is in line with psychological theories, such as the “Broaden-and-Build” hypothesis (79, 80), which postulates that positive emotions play an important role in regulating negative emotions, and protect against the negative effects of stress. This finding also partly corresponds to findings in macro-level studies, which often identified anhedonia was often identified as a central symptom. Anhedonia, as defined by a lack of interest or pleasure, could be viewed as a macro-level expression of a lack of momentary positive affect in daily life (81). Interestingly, in the dynamic macro-level networks, anhedonia was mostly identified as being the symptom most often influenced by other symptoms. For now, we can conclude that anhedonia and positive affect play an important role in the syndrome depression.

As such, converging evidence from different symptom network approaches is found on the relative importance of anhedonia and positive affect in networks of depressive individuals. Even still, a critical note is in order. Most studies have not tested the ordering of centrality estimates for statistical significance. This limits our ability to assess whether nodes that are identified as central symptoms actually are significantly more central than others. The findings of our review should be interpreted in light of this limitation. We therefore urge researchers to test for ordering which can be done in cross-sectional statistical designs by bootstrapping (82).

Connectivity

The third topic that has been investigated in network studies is connectivity (also known as density). Connectivity is calculated by summing the weights of all edges within the network. Network theory postulates that depression develops because symptoms or momentary affective states trigger each other over and over again. Therefore, this theory predicts that greater vulnerability to depression is directly related to stronger connectivity within the network (83).

However, studies have reported mixed evidence for this hypothesis. At the macro-level, eight cross-sectional and one dynamic study were conducted. Of these, three cross-sectional studies reported an association between larger connectivity and more current or future depressive symptoms (51, 54, 58). Another study provided indirect support for the hypothesis by relating higher connectivity to an earlier age of onset of depression (68). However, the results of the other five studies were inconsistent with the connectivity hypothesis. Two studies reported increased connectivity after antidepressant treatment (43, 44) and one study did not find stronger connectivity for depressed patients when compared to healthy controls (49). The final cross-sectional macro study attempting to replicate the results of van Borkulo (58) found no indications of higher connectivity predicting worse future development of depression (69). And, finally, the only dynamic (temporal) study at the macro-level also could not confirm that increased connectivity distinguished a worse course from a more favorable course of depression (60).

Therefore, at the macro-level, support for the hypothesis that stronger network connectivity is associated with increased vulnerability to depression is inconsistent. However, these inconsistent findings may simply be due to study design. Studies that failed to find support for this hypothesis compared networks of depressed individuals to networks of remitted patients or healthy controls. It may be expected that samples of healthy or remitted individuals have larger variability in their extent of depressive symptoms than depressed individuals, who may demonstrate mostly high levels of depressive symptoms. Such so-called floor effects are presumed to reduce the strength of associations in a network, and may therefore result in an opposite pattern than would be expected based on network theory (84). In line with this, studies supporting the connectivity hypothesis compared individuals who could be expected to show similar variability in symptoms, by comparing baseline network characteristics between individuals who would later develop depression or stay healthy. More studies are needed to confirm this notion. Another complicating factor when comparing connectivity levels between groups is Berkson's bias (also termed collider bias). This occurs when one estimates symptom associations in multiple groups (e.g., healthy vs. depressed individuals) that were selected based on sum scores of the same (or a similar) questionnaire (85). Because of this bias, spurious negative associations may be uncovered, thereby biasing estimates of connectivity. It is currently unclear how to deal with Berkson's bias, but it can provide another explanation for the mixed results on the connectivity hypothesis thus far.

Eight studies at the micro-level looked at the association between network connectivity and depression, based on the temporal or contemporaneous associations between momentary affective states. Six of them supported the assumption that higher network connectivity indicates a higher vulnerability for depression. Three studies found that depressed patients had a higher network connectivity than healthy controls (64, 67, 70). Two single-subject studies showed that the network connectivity between momentary affective states increased precisely in the weeks preceding a sharp rise in depressive symptoms (66, 73). This finding strongly supports the idea that connectivity between momentary affective states causally affects the development of depressive symptoms. This is especially due to the close temporal association between the increase in connectivity and the moment of sudden change in symptoms. Lastly, one dynamic network study examining contemporaneous associations at the micro-level showed that higher connectivity predicted future depressive symptoms (72).

Two micro-level studies did not support the connectivity hypothesis of the network theory. The first study examined temporal associations and found that treatment with mindfulness or antidepressants did not decrease network connectivity in depressed patients (71). An alternative explanation, also for some of the negative findings regarding treatment effects at the macro-level, could be that treatment does not address underlying vulnerability reflected by the network structure, but rather the symptoms themselves. Moreover, treatment outcomes may differ for each individual. Such heterogeneity may explain why effects are not visible at the group level (71, 86). The second micro-level study that did not support the connectivity assumption examined contemporaneous associations between momentary states. Higher network connectivity at baseline was found in depressed patients who showed higher declines in symptoms in the following year (72). Although inconsistent with the connectivity hypothesis of the network theory, this finding corresponds to another hypothesis based on complex systems theory. From this complex systems theory it is derived that higher connectivity indicates higher instability of the system. An instable system has a higher likelihood to suddenly shift to an alternative stable state, which can be either better or a worse in nature (72, 87, 88). This means that higher levels of connectivity between symptoms or affective states are hypothesized to occur before sudden transitions to alternative states in general. Such a transition to an alternative state could entail a sudden increase in symptom levels, similar as in the network theory, but could also entail a decrease in symptom levels. Thus, it may be that system stability is more relevant to network connectivity levels, rather than vulnerability per se. This might provide an alternative explanation for the mixed findings regarding the connectivity hypotheses both at the macro- and micro-level.

Altogether, although most macro-level studies did not find support for the connectivity hypothesis, the results from most micro-level studies did support this hypothesis. Results largely support the idea that the continuous dynamics between momentary affective states may play an important role in the development of clinical depressive symptoms. Whether this conclusion holds up awaits findings from future research.

Discussion

This review focused on three hypotheses based on network theory: comorbidity, centrality, and connectivity. First, regarding comorbidity, macro-level cross-sectional studies supported the hypothesis that depressive symptoms tend to co-occur with symptoms of other disorders, and that bridge symptoms that connect depression with other psychiatric disorders can be identified. Such bridge symptoms were often found in sadness, anhedonia, energy loss or fatigue, and guilt. However, given the lack of micro-level temporal studies on this topic, we do not yet know whether such bridges can also be detected within individuals; and whether momentary states associated with different syndromes indeed follow upon each other over time.

Second, in research on the centrality of network nodes, we see different outcomes for macro- vs. micro-level research on the most influential symptoms or affective states. In macro-level studies, most often identified central symptoms are energy loss, anhedonia, sadness, and concentration problems. At least partly corresponding to these findings, micro-level networks have consistently identified positive affect as having high outgoing centrality, suggesting that changes in positive affect strongly influence the rest of the network. However, the number of micro-level network studies is still small. Moreover, it is unknown whether highly central momentary states at the micro-level actually trigger the future development of symptoms assessable at the macro-level.

Third, regarding network connectivity, we found that symptom networks at the macro-level do not consistently support the network theory's connectivity hypothesis. This means that stronger network connectivity was not associated with higher symptom levels or future depressive symptoms in more than half of the studies. In contrast, most results from micro-level studies, examining associations between momentary affective states, did support the connectivity hypothesis. This mixed support for the connectivity hypothesis for depression and depressive symptoms is in conformity with findings for psychopathology in general (3).

The above review of the existing literature shows that the network theory of psychopathology has yielded several interesting areas for further research. As our review has demonstrated, network studies have used different methodological approaches to network theory and, although findings at least partly overlap, each has yielded different conclusions regarding comorbidity, centrality, and connectivity. In future evaluations of the network literature, findings derived from different methodological approaches should be clearly distinguished from one another. We propose that researchers should distinguish between the network theory of psychopathology and the complementary momentary affective dynamics (MAD) network theory, which we will introduce in the next section. After elaborating on the MAD theory, we will discuss four important points to be considered regarding methodological approaches to both network theories. Finally, we will elaborate on the application of network theory in clinical practice.

Proposing the Momentary Affect Dynamics (MAD) Network Theory

An important question is at which level network dynamics operate. Or more specifically, whether network dynamics operate at the level of clinical symptoms, momentary affective states, or both, to result in the depressive syndrome. In this review, we have reviewed empirical research conducted at both levels. However, most systemic evaluations of the network literature have not taken this distinction into account (3, 14). One reason for this may be that both methodological approaches are described under the theoretical concept of “the network theory of psychopathology,” even though their focus and assumptions are slightly different. For the purpose of transparency, we will therefore name the theoretical approach on the relevance of network dynamics at the micro-level, as described by Wichers (8), the “momentary affect dynamics (MAD)” network theory of psychopathology. We will shortly summarize the main similarities and differences of this approach to the traditional macro-level network approach as first described by Cramer and colleagues (2) and expanded upon by Borsboom (1).

Both network theories share the assumption that psychological states causally influence one another and that these dynamics play an important role in the further development of psychopathology. An important difference, however, is the level at which network dynamics are assumed to exert their influence, as referenced before. The two theories focus on a different part of the developmental process of psychopathology. The macro-level network theory focuses on the relationships between depressive symptoms. The MAD theory proposes that dynamics between micro-level momentary affective states are actually the building blocks for the development or maintenance of these macro-level symptoms.

Furthermore, the traditional macro-level network theory describes the process of causal influence between symptoms as a serial process with feedback loops. In other words, symptoms develop, which cause other symptoms to develop, and this process continues until it eventually leads to a mental disorder (1). The MAD network theory does not assume such a serial process per se. Rather, it assumes that macro-level network connections result from continuously repeating minor impacts of one momentary affective state onto another (8). Since affect states, such as feeling down, cheerful, or irritated, and their fluctuations are frequent everyday experiences, this assumption makes sense at this level of investigation. For example, within a person, affect state A (feeling down) may often impact affect state B (worrying), which often impacts affect state C (feeling less energetic). The higher the connectivity between these negative momentary affective states, the more these dynamics reinforce one another and draw individuals into cycles of persistent and negative psychological states. It is hypothesized that these persistent psychological states are then eventually experienced as symptoms that can be rated on a traditional psychopathology questionnaire. Note that here, dynamic effects are also likely to occur in parallel and that the whole cycle of dynamic effects does not necessarily need to be precisely timed one after another in order to develop psychopathology. Instead, the MAD network theory assumes that when multiple affect states repeatedly negatively impact many other affect states, the risk of getting stuck in a persistent negative psychological state increases.

In this review, we have seen that findings differ between studies based on the macro-level and micro-level network theory of psychopathology. Now that we have outlined the underlying assumptions of both network theories, at both the macro- and micro-level, it may become clearer that differences in findings are simply due to inherent differences between these approaches. By separating the methodological approaches underlying both theories, and naming them the macro-level network theory and the MAD network theory, we aim to facilitate future systematic evaluations of empirical network research.

The Importance of Individual Models

So far only a few studies have modeled networks per individual (41, 64, 66, 73). These studies show a large heterogeneity in network structure between individuals. Interestingly, within-person network structures also appeared to change over a period in which vulnerability for depression changed (66, 73). At the group-level, connections in time-series networks reflect the average outcome of within-person effects across the entire group. However, the question is then what a group-level network would tell us about the network structure of each individual person within that group (89, 90). Vicious cycles, for example, arising from the dynamics between certain symptoms or momentary affective states, are assumed to be an important risk factor for psychopathology. For instance, an individual could be in a cycle where poor sleep leads to lower energy, which lowers cheerfulness, which triggers worrying, resulting even poorer sleep (see Figure 2). Models at the group-level, however, cannot offer insight into whether these network connections co-occur within every individual within that group. Indeed, some connections may occur only in certain individuals in the group while other connections occur only in other individuals. More idiographic research is thus needed (91), to verify whether presumed vicious cycles found at a group-level actually exist at an individual level. To conclude, there is still an important gap in the testing of assumptions of the network theory using individual models. This is not only a scientific issue, but also a clinical one, as individual models have the potential to bring novel personalized scientific insights into clinical practice.

The Importance of Studying Processes of Change Using Networks

All but two publications (66, 73) discussed here estimated network structures during a period in which the predicted network parameters were expected to remain constant. These networks were either modeled before symptoms developed or when the symptoms were already present. The process of change over time, thus during a period in which depressive symptoms increase or decrease, has hardly been mapped. If we want to know how symptoms develop and remit, we will have to focus on this dynamic process. This is achieved by creating movies of networks that allow parameters to change in order to visualize their developments (66). If we can demonstrate that network structures already change prior to a change in symptoms, this would strongly support the hypothesis that network structures expose processes that are important in the development of psychopathology. Essential questions remain as to whether or not network connectivity indeed increases shortly before the start of a depressive episode and whether the network structure changes as expected (e.g., that vicious cycles decrease in strength or disappear prior remission). These kinds of studies will be essential to enhance our understanding of the developmental process of psychopathology. This means we will need to collect the necessary longitudinal time-series data, which studies have shown to be feasible (66, 73), and develop data analytical models that can accommodate change. Importantly, the most commonly used network methods require stationarity, which implies that mean levels of symptoms and the associations under study are time-invariant (92) a difficult assumption in clinical psychology, where interventions and simply the passage of time will likely affect symptoms. Recently, models such as time-varying vector autoregressive models have been developed that can handle non-stationarity (93, 94) and enable us to study change in dynamics over time.

The Importance of the Selection of Network Nodes

Another point of discussion concerns the choice of nodes that are included in a network. First, when aiming to draw conclusions on whether or not symptoms trigger each other, we assume that network connections are not the result of reasons other than causality. Many studies have used the complete list of items from depressive symptom questionnaires or ESM diary questionnaires to construct networks. For a number of these items, however, it seems likely that a third variable is responsible for the co-occurrence of these symptoms (11, 95). For example, reward dysfunction is a likely latent cause for both loss of appetite and loss of pleasure or reduced interest (11). Therefore, to test hypothesis derived from network theory, both the macro-level theory and the MAD theory, it is desirable to prevent the inclusion of nodes with such a common latent cause in the network. Preferably, future studies devise new questionnaires that are designed for network modeling purposes (3) and only include symptom or affect states that really represent distinct facets of depression. The construction of such new questionnaires could be informed by important depressive constructs identified by patients and caregivers (96). Furthermore, networks should focus on the inclusion of contextual variables as nodes in the network, for example treatment (17, 97, 98), life events (99) or social activities (8). Although not the focus of the present review, depressive symptoms do not develop within a vacuum and are likely to be strongly influenced by context. Unfortunately, in micro-level studies, such variables are often measured with categorical response scales, making it difficult to include them in the network. Future research should develop ways to assess context alongside momentary states, to be able to assess their interplay.

The Importance of Statistical Choices and Pre-processing Steps

The operationalization of network theory faces several statistical challenges that will need to be resolved, some specific to cross-sectional macro-level networks and others specific to micro-level dynamic networks. First, pertaining to both methodological approaches, several empirical studies compared network connectivity between groups of individuals with and without depressive symptoms. However, this is problematic as floor effects in symptoms or affect states may bias estimates of associations between network nodes (84). Similarly, spurious associations may be uncovered when groups are selected based on sum scores of symptoms that are also used as the network nodes (termed Berkson's bias) (85). Floor effects can be circumvented by comparing network connectivity between individuals with equal levels of depression, but different future outcomes of symptom development (58, 69). Models that could address Berkson's bias are currently under development (100).

Second, estimates of network associations depend on the statistical method used (9, 64, 101). In recent years, a large variety of statistical methods have been developed to model networks. Cross-sectional macro-level networks are mostly based on partial correlations between symptoms, using regularization techniques to identify only significant associations (102). Other methods do exist however; for example, relative importance networks or directed acyclic graphs. It is yet unclear how results from these methods relate to one another (103). Since cross-sectional networks have received strong criticisms regarding their stability and generalizability (104, 105), many studies have since then increased their sample size and started to report measures of stability, refraining from reporting network estimates when they can be considered unstable (82, 106). Temporal networks, however, can be estimated on the basis of several techniques, such as multilevel VAR (13), the sparse VAR technique (12), impulse response functions (107), or the graphical VAR model (108). The choice of statistical method can greatly influence the results and thus the conclusions of a study. This is demonstrated by the study of de Vos and colleagues (64), who showed that multilevel VAR and sparse VAR techniques resulted in conflicting results regarding network connectivity in the same data. This inconsistency in statistical models hampers our understanding of the results of networks.

A third challenge is specific to micro-level MAD networks and pertains to the preprocessing steps to be carried out on the data. This occurs even before one has selected the statistical method of choice. These steps may seem trivial, and are often unfortunately left out in publications, but can greatly affect the conclusions drawn from network results (64, 101, 109). Examples of such preprocessing steps are decisions on person-mean centering and the removal of time trends. Others concern decisions regarding whether or not to impute missing values and apply regularization techniques for model optimization (110). To date, there is no gold standard for preprocessing decisions because optimal choices can differ per study design. Further, we simply do not yet know which choices result in network coefficients that adequately reflect psychopathological vulnerability. We therefore urge researchers to be transparent in their publications regarding preprocessing steps and analytical models, and their rationale. This will enable us to better understand conflicting or replicated results. Furthermore, systematic empirical research is needed as this may reveal what type of models and statistical choices yield networks that expose information with true clinical value.

Where do We Stand in Terms of the Application of Networks in Clinical Practice?

There is rightly a lot of enthusiasm about the network approach in psychopathology among mental health professionals (5, 6). Network theory could be an interesting clinical application in several ways. First, in clinical practice the intuitive idea prevails that the network theory is in line with how psychopathology is expressed, and that these networks can important processes contributing to the development of psychopathology. It is thus a natural fit to clinical frameworks (5). A second advantage of network models is that, hypothetically, they can identify how relevant contextual factors, such as physical activity or social behavior, influence a patient's well-being. Individual networks could also expose the presence of certain vicious cycles of psychological states and behaviors, as has been demonstrated in a patient with psychosis (4), and in a patient with a panic disorder (7). A final advantage lies in the potential of increased network connectivity to alert patients and clinicians of symptom relapse in the near future (66). Applications that use real-time monitoring and detection can thus be envisioned to apply these novel insights into clinical practice (73). In this way, the network approach could help patients and their caregivers to intervene more quickly and in a more focused manner. This is the promise of the network approach for clinical practice.

However, it is apparent from our review that network research is still in its infancy. The empirical testing of network theory has barely begun and still has many challenges. As stated above, it is still unclear which statistical network models can best inform clinical issues and how. Given that the choice of statistical models and preprocessing steps in data analyses can determine the conclusions drawn from networks, this is not a trivial question. Thus, at present, it is not yet possible to guarantee that individual network models based on the data of patients are valid and trustworthy. An illustrative example in this regard is the suggestion that highly central symptoms or momentary states represent good initial treatment targets. However, the use of centrality estimates in psychological networks has been criticized, since the flow process in psychopathology may be radically different from flow process in other types of networks from which it stems [e.g., social networks or virus infections; (78)]. It is therefore questionable whether centrality can be equated with influence, and whether it is indeed informative for intervention targets. Likewise, it is also highly difficult to empirically test whether intervening on a central symptom does directly improve the rest of the system (111). Such hypotheses will need to be empirically tested before they can be implemented in treatment.

Nonetheless, several studies have conducted proof-of-principle experiments to examine if networks could inform diagnostics and treatment (7), monitor the development of symptoms (4) and in early detection for the risk for depressive relapses (66). These applications might be considered successful in the sense that they facilitated the dialogue between patient and therapist, increased motivation for trying out different treatment techniques and improved the self-management of the patient (5, 6, 112). Therefore, despite the fact that the validity of networks is still under discussion, the network approach could be valuable in improving the patient-therapist alliance and encouraging more active involvement of patients in their treatment processes. However, the danger of current applications of network feedback in clinical practice is that patients may get the false impression that these personalized networks provide scientifically validated information, which cannot yet be guaranteed (113). It is therefore important that research clarifies, as quickly as possible, under what conditions networks make proper predictions about vulnerability to psychopathology and result in useful patient-specific knowledge on the best intervention targets. Furthermore, future studies will have to demonstrate the added value of using individual network models in treatment above and beyond existing evidence-based treatment protocols.

Conclusion

The potential of the network theory is large and cannot be denied. It has both scientific and clinical face validity. This justifies intensive scientific explorations into operationalizations of the network theory. In this narrative review, we have outlined the current state of empirical network studies within the field of depression. We made explicit that at least two conceptually different, but complementary, network theories have been investigated: the traditional macro-level network theory of psychopathology focusing on clinical symptoms, and the MAD network theory focusing on affective states at the micro-level. By systematically differentiating findings of these methodological approaches, we have structured the current empirical support for assumptions of the network theory. Importantly, we argue that we need more empirical studies, and careful systematic evaluation of their findings, to conclude whether network studies can be considered to illuminate the development of psychopathology of individual patients, and whether they provide novel and clinically useful information. Future research may focus on individual rather than group models, processes of change, defining relevant network nodes, and systematic testing of the impact of various statistical specifications on network models. These steps will ensure that the network theory is further consolidated as both a research methodology as well as a clinical instrument.

Author Contributions

MW designed and conceptualized the review, and wrote the first draft of the manuscript. MW and FB reviewed and selected the literature, and analyzed the results. TH assisted in the literature search and revised the article for grammar. FB, ES, and HR critically revised the review. All authors contributed to and have approved the final manuscript.

Funding

This study was financially supported by the Rob Giel Research Center, the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (ERC-CoG-2015; grant no. 681466 to MW), the Netherlands Organisation for Health Research and Development (ZonMw Off Road; project no. 451001029 to ES), the charitable foundation Stichting tot Steun VCVGZ (grant no. 239 to HR), and Innovation Fund “Stichting De Friesland” (grant no. DS81 to HR and MW).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^The term “cross-sectional” here refers to data observations collected at the same time point in the research design.

References

1. Borsboom D. A network theory of mental disorders. World Psychiatry. (2017) 16:5–13. doi: 10.1002/wps.20375

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Cramer AOJ, Waldorp LJ, van der Maas HLJ, Borsboom D. Comorbidity: a network perspective. Behav Brain Sci. (2010) 33:137–50. doi: 10.1017/S0140525X09991567

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Robinaugh DJ, Hoekstra RH, Toner ER, Borsboom D. The network approach to psychopathology: a review of the literature 2008–2018 and an agenda for future research. Psychol Med. (2020) 50:353–66. doi: 10.1017/S0033291719003404

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Bak M, Drukker M, Hasmi L, van Os J. An n=1 clinical network analysis of symptoms and treatment in psychosis. PLoS ONE. (2016) 11:e0162811. doi: 10.1371/journal.pone.0162811

PubMed Abstract | CrossRef Full Text | Google Scholar

5. von Klipstein L, Riese H, van der Veen DC, Servaas MN, Schoevers RA. Using person-specific networks in psychotherapy: challenges, limitations, and how we could use them anyway. BMC Med. (2020) 18:345. doi: 10.1186/s12916-020-01818-0

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Bos FM, Snippe E, Bruggeman R, Wichers M, van der Krieke L. Insights of patients and clinicians on the promise of the experience sampling method for psychiatric care. Psychiatr Serv. (2019) 70:983–91. doi: 10.1176/appi.ps.201900050

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Kroeze R, van der Veen DC, Servaas MN, Bastiaansen JA, Oude Voshaar R, Borsboom D, et al. Personalized feedback on symptom dynamics of psychopathology: a proof-of-principle study. J Pers Oriented Res. (2017) 3:1–7. doi: 10.17505/jpor.2017.01

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Wichers M. The dynamic nature of depression: a new micro-level perspective of mental disorder that meets current challenges. Psychol Med. (2014) 44:1349–60. doi: 10.1017/S0033291713001979

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Bos FM, Snippe E, de Vos S, Hartmann JA, Simons CJP, van der Krieke L, et al. Can we jump from cross-sectional to dynamic interpretations of networks? implications for the network perspective in psychiatry. Psychother Psychosom. (2017) 86:175–7. doi: 10.1159/000453583

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Bringmann LF, Eronen MI. Don't blame the model: reconsidering the network approach to psychopathology. Psychol Rev. (2018) 125:606–15. doi: 10.1037/rev0000108

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Wichers M, Wigman JTW, Bringmann LF, de Jonge P. Mental disorders as networks: some cautionary reflections on a promising approach. Soc Psychiatry Psychiatr Epidemiol. (2017) 52:143–5. doi: 10.1007/s00127-016-1335-z

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Abegaz F, Wit E. Sparse time series chain graphical models for reconstructing genetic networks. Biostatistics. (2013) 14:586–99. doi: 10.1093/biostatistics/kxt005

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Bringmann LF, Vissers N, Wichers M, Geschwind N, Kuppens P, Peeters F, et al. A network approach to psychopathology: new insights into clinical longitudinal data. PLoS ONE. (2013) 8:e60188. doi: 10.1371/journal.pone.0060188

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Contreras A, Nieto I, Valiente C, Espinosa R, Vazquez C. The study of psychopathology from the network analysis perspective: a systematic review. Psychother Psychosom. (2019) 88:71–83. doi: 10.1159/000497425

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Boschloo L, van Borkulo CD, Rhemtulla M, Keyes KM, Borsboom D, Schoevers RA. The network structure of symptoms of the diagnostic and statistical manual of mental disorders. PLoS ONE. (2015) 10:e0137621. doi: 10.1371/journal.pone.0137621

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Goekoop R, Goekoop JG. A network view on psychiatric disorders: network clusters of symptoms as elementary syndromes of psychopathology. PLoS ONE. (2014) 9:e112734. doi: 10.1371/journal.pone.0112734

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Blanken TF, van der Zweerde T, van Straten A, van Someren EJW, Borsboom D, Lancee J. Introducing network intervention analysis to investigate sequential, symptom-specific treatment effects: a demonstration in co-occurring insomnia and depression. Psychother Psychosom. (2019) 88:52–4. doi: 10.1159/000495045

PubMed Abstract | CrossRef Full Text | Google Scholar

18. van Loo HM, van Borkulo CD, Peterson RE, Fried EI, Aggen SH, Borsboom D, et al. Robust symptom networks in recurrent major depression across different levels of genetic and environmental risk. J Affect Disord. (2018) 227:313–22. doi: 10.1016/j.jad.2017.10.038

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Bakker JM, Goossens L, Lange I, Michielse S, Schruers K, Lieverse R, et al. Real-life validation of reduced reward processing in emerging adults with depressive symptoms. J Abnorm Psychol. (2017) 126:713–25. doi: 10.1037/abn0000294

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Afzali MH, Sunderland M, Teesson M, Carragher N, Mills K, Slade T. A network approach to the comorbidity between posttraumatic stress disorder and major depressive disorder: the role of overlapping symptoms. J Affect Disord. (2017) 208:490–6. doi: 10.1016/j.jad.2016.10.037

PubMed Abstract | CrossRef Full Text | Google Scholar

21. An MH, Park SS, You SC, Park RW, Kim HK, Son SJ. Depressive symptom network associated with comorbid anxiety in late-life depression. Front Psychiatry. (2019) 10:856. doi: 10.3389/fpsyt.2019.00856

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Beard C, Millner AJ, Forgeard MJC, Fried EI, Hsu KJ, Treadway MT, et al. Network analysis of depression and anxiety symptom relationships in a psychiatric sample. Psychol Med. (2016) 46:3359–69. doi: 10.1017/S0033291716002300

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Bekhuis E, Schoevers RA, van Borkulo CD, Rosmalen JGM, Boschloo L. The network structure of major depressive disorder, generalized anxiety disorder and somatic symptomatology. Psychol Med. (2016) 46:2989–98. doi: 10.1017/S0033291716001550

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Curtiss J, Klemanski DH. Taxonicity and network structure of generalized anxiety disorder and major depressive disorder: an admixture analysis and complex network analysis. J Affect Disord. (2016) 199:99–105. doi: 10.1016/j.jad.2016.04.007

PubMed Abstract | CrossRef Full Text | Google Scholar

25. De Haan A, Landolt MA, Fried EI, Kleinke K, Alisic E, Bryant R, et al. Dysfunctional posttraumatic cognitions, posttraumatic stress and depression in children and adolescents exposed to trauma: a network analysis. J Child Psychol. (2020) 61:77–87. doi: 10.1111/jcpp.13101

PubMed Abstract | CrossRef Full Text | Google Scholar

26. de la Torre-Luque A, Essau CA. Symptom network connectivity in adolescents with comorbid major depressive disorder and social phobia. J Affect Disord. (2019) 255:60–8. doi: 10.1016/j.jad.2019.05.015

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Djelantik AAAMJ, Robinaugh DJ, Kleber RJ, Smid GE, Boelen PA. Symptomatology following loss and trauma: Latent class and network analyses of prolonged grief disorder, posttraumatic stress disorder, and depression in a treatment-seeking trauma-exposed sample. Depress Anxiety. (2020) 37:26–34. doi: 10.1002/da.22880

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Garabiles MR, Lao CK, Xiong Y, Hall BJ. Exploring comorbidity between anxiety and depression among migrant Filipino domestic workers: a network approach. J Affect Disord. (2019) 250:85–93. doi: 10.1016/j.jad.2019.02.062

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Heeren A, Jones PJ, McNally RJ. Mapping network connectivity among symptoms of social anxiety and comorbid depression in people with social anxiety disorder. J Affect Disord. (2018) 228:75–82. doi: 10.1016/j.jad.2017.12.003

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Jones PJ, Mair P, Riemann BC, Mugno BL, McNally RJ. A network perspective on comorbid depression in adolescents with obsessive-compulsive disorder. J Anxiety Disord. (2018) 53:1–8. doi: 10.1016/j.janxdis.2017.09.008

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Lazarov A, Suarez-Jimenez B, Levi O, Coppersmith DD, Lubin G, Pine DS, et al. Symptom structure of PTSD and co-morbid depressive symptoms–a network analysis of combat veteran patients. Psychol Med. (2020) 50:2154–70. doi: 10.1017/S0033291719002034

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Levinson CA, Zerwas S, Calebs B, Forbush K, Kordy H, Watson H, et al. The core symptoms of bulimia nervosa, anxiety, and depression: a network analysis. J Abnorm Psychol. (2017) 126:340–54. doi: 10.1037/abn0000254

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Lorimer B, Delgadillo J, Kellett S, Brown G. Exploring relapse through a network analysis of residual depression and anxiety symptoms after cognitive behavioural therapy: a proof-of-concept study. Psychother Res. (2020) 30:650–61. doi: 10.1080/10503307.2019.1650980

PubMed Abstract | CrossRef Full Text | Google Scholar

34. McNally RJ, Mair P, Mugno BL, Riemann BC. Co-morbid obsessive–compulsive disorder and depression: a Bayesian network approach. Psychol Med. (2017) 47:1204–14. doi: 10.1017/S0033291716003287

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Park S-C, Kim D. The centrality of depression and anxiety symptoms in major depressive disorder determined using a network analysis. J Affect Disord. (2020) 271:19–26. doi: 10.1016/j.jad.2020.03.078

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Rogers ML, Hom MA, Joiner TE. Differentiating Acute Suicidal Affective Disturbance (ASAD) from anxiety and depression symptoms: a network analysis. J Affect Disord. (2019) 250:333–40. doi: 10.1016/j.jad.2019.03.005

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Shim E-J, Ha H, Park JE, Kim B-S, Chang SM, Hong JP, et al. Gender-based examination of the association between individual symptoms of alcohol use disorder, major depressive disorder, and suicidal behaviors: a network analysis of data from the Korean Epidemiologic Catchment Area Study. J Affect Disord. (2020) 272:432–9. doi: 10.1016/j.jad.2020.03.162

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Tundo A, Musetti L, Del Grande C, de Filippis R, Proietti L, Marazziti D, et al. The relationship between depression with anxious distress DSM-5 specifier and mixed depression: a network analysis. CNS Spectr. (2020) 26:251–7. doi: 10.1017/S1092852920000085

PubMed Abstract | CrossRef Full Text | Google Scholar

39. van Heijst BF, Deserno MK, Rhebergen D, Geurts HM. Autism and depression are connected: a report of two complimentary network studies. Autism. (2020) 24:680–92. doi: 10.1177/1362361319872373

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Wang Y, Hu Z, Feng Y, Wilson A, Chen R. Changes in network centrality of psychopathology symptoms between the COVID-19 outbreak and after peak. Mol Psychiatry. (2020) 25:3140–9. doi: 10.1038/s41380-020-00881-6

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Kaiser T, Laireiter A-R. Process-symptom-bridges in psychotherapy: an idiographic network approach. J Pers Oriented Res. (2018) 4:49–62. doi: 10.17505/jpor.2018.06

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Groen RN, Ryan O, Wigman JT, Riese H, Penninx BW, Giltay EJ, et al. Comorbidity between depression and anxiety: assessing the role of bridge mental states in dynamic psychological networks. BMC Med. (2020) 18:308. doi: 10.1186/s12916-020-01738-z

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Berlim MT, Richard-Devantoy S, dos Santos NR, Turecki G. The network structure of core depressive symptom-domains in major depressive disorder following antidepressant treatment: a randomized clinical trial. Psychol Med. 2020:1–15. doi: 10.1017/S0033291720001002

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Bos FM, Fried EI, Hollon SD, Bringmann LF, Dimidjian S, DeRubeis RJ, et al. Cross-sectional networks of depressive symptoms before and after antidepressant medication treatment. Soc Psychiatry Psychiatr Epidemiol. (2018) 53:617–27. doi: 10.1007/s00127-018-1506-1

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Boschloo L, van Borkulo CD, Borsboom D, Schoevers RA. A prospective study on how symptoms in a network predict the onset of depression. Psychother Psychosom. (2016) 85:183–4. doi: 10.1159/000442001

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Carney RM, Freedland KE, Steinmeyer BC, Rubin EH, Rich MW. Residual symptoms following treatment for depression in patients with coronary heart disease. Psychosom Med. (2018) 80:385–92. doi: 10.1097/PSY.0000000000000570

PubMed Abstract | CrossRef Full Text | Google Scholar

47. de la Torre-Luque A, Ojagbemi A, Caballero FF, Lara E, Moreno-Agostino D, Bello T, et al. Cross-cultural comparison of symptom networks in late-life major depressive disorder: Yoruba Africans and the Spanish Population. Int J Geriatr Psychiatry. (2020) 35:1060–8. doi: 10.1002/gps.5329

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Fried EI, Epskamp S, Nesse RM, Tuerlinckx F, Borsboom D. What are 'good' depression symptoms? comparing the centrality of DSM and non-DSM symptoms of depression in a network analysis. J Affect Disord. (2016) 189:314–20. doi: 10.1016/j.jad.2015.09.005

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Hakulinen C, Fried EI, Pulkki-Råback L, Virtanen M, Suvisaari J, Elovainio M. Network structure of depression symptomology in participants with and without depressive disorder: the population-based Health 2000–2011 study. Soc Psychiatry Psychiatr Epidemiol. (2020) 55:1273–82. doi: 10.1007/s00127-020-01843-7

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Kendler KS, Aggen SH, Flint J, Borsboom D, Fried EI. The centrality of DSM and non-DSM depressive symptoms in Han Chinese women with major depression. J Affect Disord. (2018) 227:739–44. doi: 10.1016/j.jad.2017.11.032

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Madhoo M, Levine SZ. Network analysis of the Quick Inventory of Depressive Symptomatology: reanalysis of the STAR* D clinical trial. Eur Neuropsychopharmacol. (2016) 26:1768–74. doi: 10.1016/j.euroneuro.2016.09.368

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Murri MB, Amore M, Respino M, Alexopoulos GS. The symptom network structure of depressive symptoms in late-life: results from a European population study. Mol Psychiatry. (2018) 25:1447–56. doi: 10.1038/s41380-018-0232-0

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Park SC, Jang EY, Xiang YT, Kanba S, Kato TA, Chong MY, et al. Network analysis of the depressive symptom profiles in Asian patients with depressive disorders: findings from the Research on Asian Psychotropic Prescription Patterns for Antidepressants (REAP-AD). Psychiatry Clin Neurosci. (2020) 74:344–53. doi: 10.1111/pcn.12989

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Santos H, Fried EI, Asafu-Adjei J, Ruiz RJ. Network structure of perinatal depressive symptoms in Latinas: relationship to stress and reproductive biomarkers. Res Nurs Health. (2017) 40:218–28. doi: 10.1002/nur.21784

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Santos H, Kossakowski JJ, Schwartz TA, Beeber L, Fried EI. Longitudinal network structure of depression symptoms and self-efficacy in low-income mothers. PLoS ONE. (2018) 13:e0191675. doi: 10.1371/journal.pone.0191675

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Semino L, Marksteiner J, Brauchle G, Danay E. Networks of depression and cognition in elderly psychiatric patients. Gero Psych. (2017) 30:89–96. doi: 10.1024/1662-9647/a000170

CrossRef Full Text | Google Scholar

57. van Borkulo C, Borsboom D, Epskamp S, Blanken TF, Boschloo L, Schoevers RA, et al. A new method for constructing networks from binary data. Sci Rep. (2014) 4:5918. doi: 10.1038/srep05918

PubMed Abstract | CrossRef Full Text

58. van Borkulo C, Boschloo L, Borsboom D, Penninx BWJH, Waldorp LJ, Schoevers RA. Association of symptom network structure with the course of longitudinal depression. JAMA Psychiatry. (2015) 72:1219–26. doi: 10.1001/jamapsychiatry.2015.2079

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Bringmann LF, Lemmens LHJM, Huibers MJH, Borsboom D, Tuerlinckx F. Revealing the dynamic network structure of the Beck Depression Inventory-II. Psychol Med. (2015) 45:747–57. doi: 10.1017/S0033291714001809

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Groen RN, Snippe E, Bringmann LF, Simons CJP, Hartmann JA, Bos EH, et al. Capturing the risk of persisting depressive symptoms: a dynamic network investigation of patients' daily symptom experiences. Psychiatry Res. (2019) 271:640–8. doi: 10.1016/j.psychres.2018.12.054

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Komulainen K, Airaksinen J, Savelieva K, Gluschkoff K, Velázquez RG, Elovainio M, et al. Network dynamics of depressive symptoms in antidepressant medication treatment: secondary analysis of eight clinical trials. Mol Psychiatry. (2020) 1–8. doi: 10.1038/s41380-020-00884-3

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Savelieva K, Komulainen K, Elovainio M, Jokela M. Longitudinal associations between specific symptoms of depression: network analysis in a prospective cohort study. J Affect Disord. (2020) 278:99–106. doi: 10.1016/j.jad.2020.09.024

PubMed Abstract | CrossRef Full Text | Google Scholar

63. David SJ, Marshall AJ, Evanovich EK, Mumma GH. Intraindividual dynamic network analysis–implications for clinical assessment. J Psychopathol Behav Assess. (2018) 40:235–48. doi: 10.1007/s10862-017-9632-8

PubMed Abstract | CrossRef Full Text | Google Scholar

64. de Vos S, Wardenaar KJ, Bos EH, Wit EC, Bouwmans MEJ, de Jonge P. An investigation of emotion dynamics in major depressive disorder patients and healthy persons using sparse longitudinal networks. PLoS ONE. (2017) 12:e0178586. doi: 10.1371/journal.pone.0178586

PubMed Abstract | CrossRef Full Text | Google Scholar

65. Fisher AJ, Reeves JW, Lawyer G, Medaglia JD, Rubel JA. Exploring the idiographic dynamics of mood and anxiety via network analysis. J Abnorm Psychol. (2017) 126:1044–56. doi: 10.1037/abn0000311

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Wichers M, Groot PC, Psychosystems, Group ESM, Group EWS. Critical slowing down as a personalized early warning signal for depression. Psychother Psychosom. (2016) 85:114–6. doi: 10.1159/000441458

PubMed Abstract | CrossRef Full Text | Google Scholar

67. Wigman JTW, van Os J, Borsboom D, Wardenaar KJ, Epskamp S, Klippel A, et al. Exploring the underlying structure of mental disorders: cross-diagnostic differences and similarities from a network perspective using both a top-down and a bottom-up approach. Psychol Med. (2015) 45:2375–87. doi: 10.1017/S0033291715000331

PubMed Abstract | CrossRef Full Text | Google Scholar

68. Baez LM, Heller AS. Impact of age at onset on the phenomenology of depression in treatment-seeking adults in the STAR* D trial. J Affect Disord. (2020) 262:381–8. doi: 10.1016/j.jad.2019.10.036

PubMed Abstract | CrossRef Full Text | Google Scholar

69. Schweren L, van Borkulo CD, Fried E, Goodyer IM. Assessment of symptom network density as a prognostic marker of treatment response in adolescent depression. JAMA Psychiatry. (2018) 75:98–100. doi: 10.1001/jamapsychiatry.2017.3561

PubMed Abstract | CrossRef Full Text | Google Scholar

70. Pe ML, Kircanski K, Thompson RJ, Bringmann LF, Tuerlinckx F, Mestdagh M, et al. Emotion-network density in major depressive disorder. Clin Psychol Sci. (2015) 3:292–300. doi: 10.1177/2167702614540645

PubMed Abstract | CrossRef Full Text | Google Scholar

71. Snippe E, Viechtbauer W, Geschwind N, Klippel A, de Jonge P, Wichers M. The impact of treatments for depression on the dynamic network structure of mental states: two randomized controlled trials. Sci Rep. (2017) 7:46523. doi: 10.1038/srep46523

PubMed Abstract | CrossRef Full Text | Google Scholar

72. van de Leemput IA, Wichers M, Cramer AOJ, Borsboom D, Tuerlinckx F, Kuppens P, et al. Critical slowing down as early warning for the onset and termination of depression. Proc Natl Acad Sci USA. (2014) 111:87–92. doi: 10.1073/pnas.1312114110

PubMed Abstract | CrossRef Full Text | Google Scholar

73. Wichers M, Smit AC, Snippe E. Early warning signals based on momentary affect dynamics can expose nearby transitions in depression: a confirmatory single-subject time-series study. J Pers Oriented Res. (2020) 6:1–15. doi: 10.17505/jpor.2020.22042

PubMed Abstract | CrossRef Full Text | Google Scholar

74. McGorry PD, Hartmann JA, Spooner R, Nelson B. Beyond the “at risk mental state” concept: transitioning to transdiagnostic psychiatry. World Psychiatry. (2018) 17:133–42. doi: 10.1002/wps.20514

PubMed Abstract | CrossRef Full Text | Google Scholar

75. Jones PJ, Ma R, McNally RJ. Bridge centrality: a network approach to understanding comorbidity. Multivariate Behav Res. (2021) 56:353–67. doi: 10.1080/00273171.2019.1614898

PubMed Abstract | CrossRef Full Text | Google Scholar

76. Jones P. Networktools: Tools for Identifying Important Nodes in Networks. R Package Version 12. (2017).

77. Opsahl T, Agneessens F, Skvoretz J. Node centrality in weighted networks: generalizing degree and shortest paths. Soc Netw. (2010) 32:245–51. doi: 10.1016/j.socnet.2010.03.006

CrossRef Full Text | Google Scholar

78. Bringmann LF, Elmer T, Epskamp S, Krause RW, Schoch D, Wichers M, et al. What do centrality measures measure in psychological networks. J Abnorm Psychol. (2019) 128:892–903. doi: 10.1037/abn0000446

PubMed Abstract | CrossRef Full Text | Google Scholar

79. Fredrickson BL. The broaden-and-build theory of positive emotions. Philos Trans R Soc Lond Ser B Biol Sci. (2004) 359:1367–77. doi: 10.1098/rstb.2004.1512

PubMed Abstract | CrossRef Full Text | Google Scholar

80. Garland EL, Fredrickson B, Kring AM, Johnson DP, Meyer PS, Penn DL. Upward spirals of positive emotions counter downward spirals of negativity: insights from the broaden-and-build theory and affective neuroscience on the treatment of emotion dysfunctions and deficits in psychopathology. Clin Psychol Rev. (2010) 30:849–64. doi: 10.1016/j.cpr.2010.03.002

PubMed Abstract | CrossRef Full Text | Google Scholar

81. van Roekel E, Heininga VE, Vrijen C, Snippe E, Oldehinkel AJ. Reciprocal associations between positive emotions and motivation in daily life: network analyses in anhedonic individuals and healthy controls. Emotion. (2019) 19:292–300. doi: 10.1037/emo0000424

PubMed Abstract | CrossRef Full Text | Google Scholar

82. Epskamp S. Bootnet: Bootstrap Methods for Various Network Estimation Routines. R Package. (2015).

83. Cramer AOJ, van Borkulo CD, Giltay EJ, van der Maas HLJ, Kendler KS, Scheffer M, et al. Major depression as a complex dynamic system. PLoS ONE. (2016) 11:e0167490. doi: 10.1371/journal.pone.0167490

PubMed Abstract | CrossRef Full Text | Google Scholar

84. Terluin B, de Boer MR, de Vet HCW. Differences in connection strength between mental symptoms might be explained by differences in variance: reanalysis of network data did not confirm staging. PLoS ONE. (2016) 11:e0155205. doi: 10.1371/journal.pone.0155205

PubMed Abstract | CrossRef Full Text | Google Scholar

85. De Ron J, Fried EI, Epskamp S. Psychological networks in clinical populations: investigating the consequences of Berkson's bias. Psychol Med. (2021) 51:168–76. doi: 10.1017/S0033291719003209

PubMed Abstract | CrossRef Full Text | Google Scholar

86. Snippe E, Simons CJP, Hartmann JA, Menne-Lothmann C, Kramer I, Booij SH, et al. Change in daily life behaviors and depression: within-person and between-person associations. Health Psychol. (2016) 35:433–41. doi: 10.1037/hea0000312

PubMed Abstract | CrossRef Full Text | Google Scholar

87. Scheffer M, Bascompte J, Brock WA, Brovkin V, Carpenter SR, Dakos V, et al. Early-warning signals for critical transitions. Nature. (2009) 461:53–9. doi: 10.1038/nature08227

PubMed Abstract | CrossRef Full Text | Google Scholar

88. Scheffer M, Carpenter SR, Lenton TM, Bascompte J, Brock W, Dakos V, et al. Anticipating critical transitions. Science. (2012) 338:344–8. doi: 10.1126/science.1225244

PubMed Abstract | CrossRef Full Text | Google Scholar

89. Wright AG, Woods WC. Personalized models of psychopathology. Annu Rev Clin Psychol. (2020) 16:49–74. doi: 10.1146/annurev-clinpsy-102419-125032

PubMed Abstract | CrossRef Full Text | Google Scholar

90. Fisher AJ, Medaglia JD, Jeronimus BF. Lack of group-to-individual generalizability is a threat to human subjects research. Proc Natl Acad Sci USA. (2018) 115:E6106–15. doi: 10.1073/pnas.1711978115

PubMed Abstract | CrossRef Full Text | Google Scholar

91. Zuidersma M, Riese H, Snippe E, Booij SH, Wichers M, Bos EH. Single-subject research in psychiatry: facts and fictions. Front Psychiatry. (2020) 11:539777. doi: 10.3389/fpsyt.2020.539777

PubMed Abstract | CrossRef Full Text | Google Scholar

92. Bringmann LF. Person-specific networks in psychopathology: past, present and future. Curr Opin Psychol. (2021) 41:59–64. doi: 10.1016/j.copsyc.2021.03.004

PubMed Abstract | CrossRef Full Text | Google Scholar

93. Haslbeck JM, Bringmann LF, Waldorp LJ. A tutorial on estimating time-varying vector autoregressive models. Multivariate Behav Res. (2021) 56:120–49. doi: 10.1080/00273171.2020.1743630

PubMed Abstract | CrossRef Full Text | Google Scholar

94. Bringmann LF, Ferrer E, Hamaker EL, Borsboom D, Tuerlinckx F. Modeling nonstationary emotion dynamics in dyads using a time-varying vector-autoregressive model. Multivariate Behav Res. (2018) 53:293–314. doi: 10.1080/00273171.2018.1439722

PubMed Abstract | CrossRef Full Text | Google Scholar

95. Fried EI, Cramer AOJ. Moving forward: challenges and directions for psychopathological network theory and methodology. Perspect Psychol Sci. (2017) 12:999–1020. doi: 10.1177/1745691617705892

PubMed Abstract | CrossRef Full Text | Google Scholar

96. Chevance A, Ravaud P, Tomlinson A, Le Berre C, Teufer B, Touboul S, et al. Identifying outcomes for depression that matter to patients, informal caregivers, and health-care professionals: qualitative content analysis of a large international online survey. Lancet Psychiatry. (2020) 7:692–702. doi: 10.1016/S2215-0366(20)30191-7

PubMed Abstract | CrossRef Full Text | Google Scholar

97. Bekhuis E, Schoevers R, de Boer M, Peen J, Dekker J, Van H, et al. Symptom-specific effects of psychotherapy versus combined therapy in the treatment of mild to moderate depression: a network approach. Psychother Psychosom. (2018) 87:121–3. doi: 10.1159/000486793

PubMed Abstract | CrossRef Full Text | Google Scholar

98. Boschloo L, Bekhuis E, Weitz ES, Reijnders M, DeRubeis RJ, Dimidjian S, et al. The symptom-specific efficacy of antidepressant medication vs. cognitive behavioral therapy in the treatment of depression: results from an individual patient data meta-analysis. World Psychiatry. (2019) 18:183–91. doi: 10.1002/wps.20630

PubMed Abstract | CrossRef Full Text | Google Scholar

99. Cramer AOJ, Borsboom D, Aggen SH, Kendler KS. The pathoplasticity of dysphoric episodes: Differential impact of stressful life events on the pattern of depressive symptom inter-correlations. Psychol Med. (2012) 42:957–65. doi: 10.1017/S003329171100211X

PubMed Abstract | CrossRef Full Text | Google Scholar

100. Haslbeck JM, Borsboom D, Waldorp LJ. Moderated network models. Multivariate Behav Res. (2019) 56:256–87. doi: 10.1080/00273171.2019.1677207

PubMed Abstract | CrossRef Full Text | Google Scholar

101. Bulteel K, Tuerlinckx F, Brose A, Ceulemans E. Using raw VAR regression coefficients to build networks can be misleading. Multivariate Behav Res. (2016) 51:330–44. doi: 10.1080/00273171.2016.1150151

PubMed Abstract | CrossRef Full Text | Google Scholar

102. Epskamp S, Fried EI. A tutorial on regularized partial correlation networks. Psychol Methods. (2018) 23:617–34. doi: 10.1037/met0000167

PubMed Abstract | CrossRef Full Text | Google Scholar

103. McNally RJ. Network analysis of psychopathology: controversies and challenges. Annu Rev Clin Psychol. (2020) 17:31–53. doi: 10.1146/annurev-clinpsy-081219-092850

PubMed Abstract | CrossRef Full Text | Google Scholar

104. Forbes MK, Wright AGC, Markon KE, Krueger RF. The network approach to psychopathology: promise versus reality. World Psychiatry. (2019) 18:272–3. doi: 10.1002/wps.20659

PubMed Abstract | CrossRef Full Text | Google Scholar

105. Forbes MK, Wright AGC, Markon KE, Krueger RF. Evidence that psychopathology symptom networks have limited replicability. J Abnorm Psychol. (2017) 126:969–88. doi: 10.1037/abn0000276

PubMed Abstract | CrossRef Full Text | Google Scholar

106. Epskamp S, Borsboom D, Fried EI. Estimating psychological networks and their accuracy: a tutorial paper. Behav Res Methods. (2017) 50:195–212. doi: 10.3758/s13428-017-0862-1

PubMed Abstract | CrossRef Full Text | Google Scholar

107. Brandt P, Williams JT. Multiple Time Series Models. Thousand Oaks, CA: Sage Publications Inc. (2007).

Google Scholar

108. Wild B, Eichler M, Friederich H-C, Hartmann M, Zipfel S, Herzog W. A graphical vector autoregressive modelling approach to the analysis of electronic diary data. BMC Med Res Methodol. (2010) 10:28. doi: 10.1186/1471-2288-10-28

PubMed Abstract | CrossRef Full Text | Google Scholar

109. Bastiaansen JA, Kunkels YK, Blaauw FJ, Boker SM, Ceulemans E, Chen M, et al. Time to get personal? the impact of researchers choices on the selection of treatment targets using the experience sampling methodology. J Psychosom Res. (2020) 137:110211. doi: 10.31234/osf.io/c8vp7

PubMed Abstract | CrossRef Full Text | Google Scholar

110. Tibshirani R. Regression shrinkage and selection via the lasso. J R Stat Soc Ser B. (1996) 58:267–88. doi: 10.1111/j.2517-6161.1996.tb02080.x

CrossRef Full Text | Google Scholar

111. Eronen MI. Causal discovery and the problem of psychological interventions. New Ideas Psychol. (2020) 59:100785. doi: 10.1016/j.newideapsych.2020.100785

CrossRef Full Text | Google Scholar

112. Frumkin MR, Piccirillo ML, Beck ED, Grossman JT, Rodebaugh TL. Feasibility and utility of idiographic models in the clinic: a pilot study. Psychother Res. (2020) 31:520–34. doi: 10.1080/10503307.2020.1805133

PubMed Abstract | CrossRef Full Text | Google Scholar

113. Bos FM, Snippe E, Bruggeman R, Doornbos B, Wichers M, van der Krieke L. Recommendations for the use of long-term experience sampling in bipolar disorder care: a qualitative study of patient and clinician experiences. Int J Bipolar Disord. (2020) 8:1–14. doi: 10.1186/s40345-020-00201-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Keywords: network theory, network analysis, major depressive disorder, experience sampling method (ESM)/ecological momentary assessment (EMA), review (article), momentary affect dynamics theory

Citation: Wichers M, Riese H, Hodges TM, Snippe E and Bos FM (2021) A Narrative Review of Network Studies in Depression: What Different Methodological Approaches Tell Us About Depression. Front. Psychiatry 12:719490. doi: 10.3389/fpsyt.2021.719490

Received: 02 June 2021; Accepted: 30 September 2021;
Published: 28 October 2021.

Edited by:

Gábor Gazdag, Jahn Ferenc Dél-Pesti Kórház és Rendelointézet, Hungary

Reviewed by:

Casimiro Cabrera Abreu, Queens University, Canada
Richard McNally, Harvard University, United States

Copyright © 2021 Wichers, Riese, Hodges, Snippe and Bos. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Fionneke M. Bos, f.m.bos01@umcg.nl

Download