Psychopathology networks replicate with stunning precision

By Denny Borsboom, Eiko Fried, Sacha Epskamp, Lourens Waldorp, Claudia van Borkulo, Han van der Maas, and Angélique Cramer

Update: The pre-print of our official commentary is now online at PsyArXiv.

In a forthcoming paper in the Journal of Abnormal Psychology entitled “Evidence that psychopathology networks do not replicate”, Miriam Forbes, Aidan Wright, Kristian Markon, and Robert Krueger purport to show that network structures do not replicate across datasets. They estimate networks for symptoms of Major Depressive Episode (MDE) and Generalized Anxiety Disorder (GAD) in two large datasets – one from the National Comorbidity Survey-Replication (NCS) and one from the Australian National Survey of Mental Health and Well-Being (NSMHWB). As is evident from our published work (Fried & Cramer, in press; Fried, van Borkulo, Cramer, Boschloo, Schoevers, & Borsboom, 2017; Epskamp, Borsboom, & Fried, 2017) we see the reproducibility of network research as a top priority, and are happy to see that researchers are investigating this important issue.

The conclusion proposed by the authors is that “network analysis […] had poor replicability between […] samples”. This conclusion is not supported by the data for state-of-the-art network models, as we will argue in a commentary solicited by the Journal of Abnormal Psychology. Given the intense interest in the matter, however, we deemed it useful to post a short blog post in advance to state a fact that may not be obvious to most readers given the rhetorical style of Forbes et al. (in press). In one sentence: state-of-the-art networks don’t just replicate – they replicate with stunning precision.

Unfortunately, the authors of the paper did not share their data or code with us yet, so we cannot fully evaluate their work, but they did share the parameter matrices that they got out of the analysis, and these are sufficient to establish that the authors’ conclusion does not apply to state-of-the-art network modeling techniques (e.g., networks estimated using our R-package IsingFit; Van Borkulo et al., 2014). The authors of the paper suggest as much when they say that “[t]he replicability of the edges in the Ising models was remarkably similar between and within samples”, but this conclusion is easily lost in the rhetoric of the paper’s title, abstract, and discussion. In addition, the authors insufficiently articulate just how similar the networks are; as a result, users of our techniques may be wondering whether Ising networks really live up to their reputation as highly stable and secure network estimation techniques.

BlogPlotFigure 1. The networks estimated from the NCS and NSMHWB samples, and the scatterplot of network parameters as estimated in both samples (r=.95). The network representations use the default settings of qgraph (Epskamp, Cramer, Waldorp, Schmittmann, & Borsboom, 2011) and a common (average) layout to optimize the comparison between datasets.

So just how replicable are Ising networks? Figure 1 pictures the situation quite clearly: the IsingFit networks are almost indistinguishable, and the network parameters display a whopping correlation of .95 for network edges and .93 for node thresholds across samples (Spearman correlations equal .88 and .85, respectively). Even centrality indices, which we usually approach with considerable caution due to their sensitivity to sampling variation (Epskamp, Borsboom, & Fried, 2017), show surprisingly good replication performance with correlations of .94 (strength), .94 (betweenness), and .76 (closeness).

Nobody in our group had in fact expected such an accurate replication across two entirely distinct samples. As such, we argue that the authors’ conclusion that “the unique utility of network analysis …seems limited to visualizing complex multivariate relationships…” is unwarranted. Given our re-analysis of the results of the Forbes et al. (in press) paper one will wonder: how on earth can the authors of the paper interpret this result as “evidence that psychopathology networks do not replicate”? Well, if you want to find that out, keep an eye out for the upcoming issue of the Journal of Abnormal Psychology, in which we will provide a comprehensive dissection of their methodology and argumentation. We’ll keep you posted!



Epskamp, S., Borsboom, D. & Fried, E.I. (2017). Estimating psychological networks and their accuracy: a tutorial paper. Behavior Research Methods. doi:10.3758/s13428-017-0862-1

Epskamp, S., Cramer, A. O. J., Waldorp, L. J., Schmittmann, V. D., & Borsboom, D. (2012). qgraph: Network visualizations of relationships in psychometric data. Journal of Statistical Software, 48, 1-18.

Forbes, Wright, Markon, and Krueger (in press).  Evidence that psychopathology symptom networks do not replicate. Journal of Abnormal Psychology.

Fried, E. I. & Cramer, A. O. J. (in press). Moving forward: challenges and directions for psychopathological network theory and methodology. Perspectives on Psychological Science.

Fried, E. I.*, van Borkulo, C. D.*, Cramer, A. O. J., Lynn, B., Schoevers, R. A., Borsboom, D. (2016). Mental disorders as networks of problems: a review of recent insights. Social Psychiatry and Psychiatric Epidemiology, 52, 1-10.

Van Borkulo, C.D., Borsboom, D., Epskamp, S., Blanken, T.F., Boschloo, L., Schoevers, R.A. & Waldorp, L.J. (2014). A new method for constructing networks from binary data. Scientific Reports, 4: 5918. doi: 10.1038/srep05918




ERC Consolidator grant for the Psychosystems Project

The European Research Council (ERC) has awarded a consolidator grant to Denny Borsboom to support the psychosystems project. The project, which is entitled “Psychosystems: Consolidating Network Approaches to Psychopathology”, is designed to further develop the theory and methodology of networks for mental disorders. The ERC will support the project for five years, allowing us to launch a number of new postdoc and PhD projects.

Four new Ph. D. projects on networks

We are happy to welcome four new Ph.D. researchers into the network community:

  • Marie Deserno will investigate determinants of success and problems in Autism, in the context of a project organized by the Academische Werkplaats Autisme. The project is supervised by Hilde Geurts, Sander Begeer, and Denny Borsboom.
  • Riet van Bork will investigate methodologies suited to distinguish between network and latent variable explanations of symptom covariation. The project is funded partly by a European Commission Career Integration Grant awarded to Mijke Rhemtulla, who will supervise the project together with Denny Borsboom.
  • Jonas Dalege will develop network models for the structure and dynamics of attitudes in a project that involves a collaboration between the Psychological Methods and Social Psychology groups at the University of Amsterdam. The project is supervised by Frenk van Harreveld, Han van der Maas, and Denny Borsboom.
  • Merijn Mestdagh will develop time series models for nonlinear networks that involve large number of variables at the K.U. Leuven in a Ph. D. project awarded by the Flemish Organization for Scientific Research (FWO). The project is supervised by Francis Tuerlinckx and Denny Borsboom.

Three new NWO-Veni projects on networks

A very successful round of 2014 NWO grant applications for network researchers!

Our own Angélique Cramer received a Veni grant to further develop network methodology and apply that to mood disorders:

  • Network psychometrics: methods for exposing the architecture and dynamics of mood disorders
    Dr A.O.J. (Angélique) Cramer (f), University of Amsterdam – Psychometrics
    In the network approach a psychological disorder is a consequence of interactions between symptoms (gloomy and fretting). In this proposal methods to construct and analyse these networks will be further developed. Subsequently the networks of patients with mood disorders will be investigated: for example, can we predict when the patient will experience a relapse?

Hanneke Wigman of Groningen University received a Veni grant to investigate symptom networks in the etiology of psychosis:

  • Networks of symptoms in psychosis
    Dr J.T.W. (Hanneke) Wigman (f), University Medical Centre Groningen – Psychiatry
    The researcher will study the development of psychoses (a lost sense of reality) by describing networks of early symptoms and by examining per person how these symptoms influence each other. This will help to develop personalised treatment recommendations.

Annemarie Kalis of Utrecht University received a Veni grant to investigate the role of mental content in empirical psychological research, one subproject of which is devoted to network models:

  • A role for content and mental causation in empirical psychology
    Dr A. Kalis (f), Utrecht University – Philosophy
    Many mental states have content: our convictions, desires, intentions ‘are all about something’. Empirical psychology, however, scarcely offers space for the notion of content. This project will develop a philosophical analysis that shows how mental content can and should play a role in in empirical psychological research.