Psychopathology networks replicate with stunning precision

By Denny Borsboom, Eiko Fried, Sacha Epskamp, Lourens Waldorp, Claudia van Borkulo, Han van der Maas, and Angélique Cramer

In a forthcoming paper in the Journal of Abnormal Psychology entitled “Evidence that psychopathology networks do not replicate”, Miriam Forbes, Aidan Wright, Kristian Markon, and Robert Krueger purport to show that network structures do not replicate across datasets. They estimate networks for symptoms of Major Depressive Episode (MDE) and Generalized Anxiety Disorder (GAD) in two large datasets – one from the National Comorbidity Survey-Replication (NCS) and one from the Australian National Survey of Mental Health and Well-Being (NSMHWB). As is evident from our published work (Fried & Cramer, in press; Fried, van Borkulo, Cramer, Boschloo, Schoevers, & Borsboom, 2017; Epskamp, Borsboom, & Fried, 2017) we see the reproducibility of network research as a top priority, and are happy to see that researchers are investigating this important issue.

The conclusion proposed by the authors is that “network analysis […] had poor replicability between […] samples”. This conclusion is not supported by the data for state-of-the-art network models, as we will argue in a commentary solicited by the Journal of Abnormal Psychology. Given the intense interest in the matter, however, we deemed it useful to post a short blog post in advance to state a fact that may not be obvious to most readers given the rhetorical style of Forbes et al. (in press). In one sentence: state-of-the-art networks don’t just replicate – they replicate with stunning precision.

Unfortunately, the authors of the paper did not share their data or code with us yet, so we cannot fully evaluate their work, but they did share the parameter matrices that they got out of the analysis, and these are sufficient to establish that the authors’ conclusion does not apply to state-of-the-art network modeling techniques (e.g., networks estimated using our R-package IsingFit; Van Borkulo et al., 2014). The authors of the paper suggest as much when they say that “[t]he replicability of the edges in the Ising models was remarkably similar between and within samples”, but this conclusion is easily lost in the rhetoric of the paper’s title, abstract, and discussion. In addition, the authors insufficiently articulate just how similar the networks are; as a result, users of our techniques may be wondering whether Ising networks really live up to their reputation as highly stable and secure network estimation techniques.

BlogPlotFigure 1. The networks estimated from the NCS and NSMHWB samples, and the scatterplot of network parameters as estimated in both samples (r=.95). The network representations use the default settings of qgraph (Epskamp, Cramer, Waldorp, Schmittmann, & Borsboom, 2011) and a common (average) layout to optimize the comparison between datasets.

So just how replicable are Ising networks? Figure 1 pictures the situation quite clearly: the IsingFit networks are almost indistinguishable, and the network parameters display a whopping correlation of .95 for network edges and .93 for node thresholds across samples (Spearman correlations equal .88 and .85, respectively). Even centrality indices, which we usually approach with considerable caution due to their sensitivity to sampling variation (Epskamp, Borsboom, & Fried, 2017), show surprisingly good replication performance with correlations of .94 (strength), .94 (betweenness), and .76 (closeness).

Nobody in our group had in fact expected such an accurate replication across two entirely distinct samples. As such, we argue that the authors’ conclusion that “the unique utility of network analysis …seems limited to visualizing complex multivariate relationships…” is unwarranted. Given our re-analysis of the results of the Forbes et al. (in press) paper one will wonder: how on earth can the authors of the paper interpret this result as “evidence that psychopathology networks do not replicate”? Well, if you want to find that out, keep an eye out for the upcoming issue of the Journal of Abnormal Psychology, in which we will provide a comprehensive dissection of their methodology and argumentation. We’ll keep you posted!



Epskamp, S., Borsboom, D. & Fried, E.I. (2017). Estimating psychological networks and their accuracy: a tutorial paper. Behavior Research Methods. doi:10.3758/s13428-017-0862-1

Epskamp, S., Cramer, A. O. J., Waldorp, L. J., Schmittmann, V. D., & Borsboom, D. (2012). qgraph: Network visualizations of relationships in psychometric data. Journal of Statistical Software, 48, 1-18.

Forbes, Wright, Markon, and Krueger (in press).  Evidence that psychopathology symptom networks do not replicate. Journal of Abnormal Psychology.

Fried, E. I. & Cramer, A. O. J. (in press). Moving forward: challenges and directions for psychopathological network theory and methodology. Perspectives on Psychological Science.

Fried, E. I.*, van Borkulo, C. D.*, Cramer, A. O. J., Lynn, B., Schoevers, R. A., Borsboom, D. (2016). Mental disorders as networks of problems: a review of recent insights. Social Psychiatry and Psychiatric Epidemiology, 52, 1-10.

Van Borkulo, C.D., Borsboom, D., Epskamp, S., Blanken, T.F., Boschloo, L., Schoevers, R.A. & Waldorp, L.J. (2014). A new method for constructing networks from binary data. Scientific Reports, 4: 5918. doi: 10.1038/srep05918




Paper on comparing networks of two groups of patients with MDD

Our paper on comparing networks of two groups of patients with Major Depressive Disorder was published in JAMA Psychiatry (PDF).

In this paper, we investigated the association between baseline network structure of depression symptoms and the course of depression. We compared the baseline network structure of persisters (defined as patients with MDD at baseline and depressive symptomatology at 2-year follow-up) and remitters (patients with MDD at baseline without depressive symptomatology at 2-year follow-up). To compare network structures we used the first statistical test that directly compares connectivity of two networks (Network Comparison Test; NCT). While both groups have similar symptomatology at baseline, persisters have a more densely connected network compared to remitters. More specific symptom associations seem to be an important determinant of persistence of depression.

A Dutch newspaper (NRC Handelsblad, November 21st, 2015) published a piece about this paper (Link).

Paper on network model of attitudes

Our paper on the Causal Attitude Network (CAN) model was published in Psychological Review (PDF).

In the paper, we introduce the CAN model, which conceptualizes attitudes as networks consisting of interacting evaluative reactions, such as beliefs (e.g., judging a presidential candidate as competent and charismatic), feelings (e.g., feeling proudness and hope about the candidate), and behaviors (e.g., voting for the candidate). Interactions arise through direct causal connections between the evaluative reactions (e.g., feeling hopeful about the candidate because one judges her as competent and charismatic). The CAN model assumes that causal connections between evaluative reactions serve to heighten the consistency of the attitude and we argue that the Ising model’s axiom of energy expenditure reduction represents a formalized account of consistency pressure. Because individuals not only strive for consistency but also for accuracy, network representations of attitudes have to deal with the tradeoff between consistency and accuracy. This tradeoff is likely to lead to a small-world structure and we show that attitude networks indeed have a small-world structure. We also discuss the CAN model’s implication for attitude change and stability. Furthermore, we show that connectivity of attitude networks provides a formalized and parsimonious account of the dynamical differences between strong and weak attitudes.

Dalege, J., Borsboom, D., van Harreveld, F., van den Berg, H., Conner, M., & van der Maas, H. L. J. (2015). Toward a formalized account of attitudes: The Causal Attitude Network (CAN) model. Psychological Review. Advance online publication.

HRQoL Paper published

Recently, our paper “The application of a network approach to health-related quality of life (HRQoL): introducing a new method for assessing hrqol in healthy adults and cancer patients” was published in Quality of Life Research.

The objective of this paper was to introduce a new approach for analyzing Health-Related Quality of Life (HRQoL) data, namely a network model.

The goal of this paper was to introduce the network approach in the analyzation of Health-Related Quality of Life (HRQoL) data. To show that the network approach can aid in the analysis of these kinds of data, we constructed networks of two samples: Dutch cancer patients (N = 485) and Dutch healthy adults (N = 1742). Both completed the 36-item Short Form Health Survey (SF-36), a commonly used instrument across different disease conditions and patient groups [1]. In order to investigate the influence of diagnostic status, we added this binary variable to a third network that was constructed using both samples. The SF-36 consists of 8 sub-scales (domains). We constructed so-called “sub-scale” networks to gain more insight into the dynamics of HRQoL on domain level.

Results showed that the global structure of the SF-36 is dominant in all networks, supporting the validity of questionnaire’s subscales. Furthermore, we found that the network structure of the individual samples were similar with respect to the basic structure (item level), and that the network structure of the individual samples were highly similar not only with respect to the basic structure, but also with respect to the strength of the connections (subscale level). Lastly, centrality analyses revealed that maintaining a daily routine despite one’s physical health predicts HRQoL levels best.

We concluded that the network approach offers an alternative view on Healt-Related Quality of Life. We showed that the HRQoL network is, in its basic structure, similar across samples. Moreover, by using the network approach, we are able to identify important characteristics in the structure, which may inform treatment decisions.

Kossakowski, J. J., Epskamp, S., Kieffer, J. M., Borkulo, C. D. van, Rhemtulla, M., & Borsboom, D. (in press). The Application of a Network Approach to Health-Related Quality of Life: Introducing a New Method for Assessing HRQoL in Healthy Adults and Cancer Patients. Quality of Life Research. DOI: 10.1007/s11136-015-1127-z.

[1] Ware, J. E, Jr, & Sherbourne, C. D. (1992). The MOS 36-item short-form health survey (SF-36): I. Conceptual framework and item selection. Medical Care, 30, 473–483.

ERC Consolidator grant for the Psychosystems Project

The European Research Council (ERC) has awarded a consolidator grant to Denny Borsboom to support the psychosystems project. The project, which is entitled “Psychosystems: Consolidating Network Approaches to Psychopathology”, is designed to further develop the theory and methodology of networks for mental disorders. The ERC will support the project for five years, allowing us to launch a number of new postdoc and PhD projects.