Most machine learning (ML) applications, e.g., healthcare, drug-discovery, etc., encounter dataset shift when operating in the real-world. The reason for this comes from the bias in the testing conditions compared to the training environment introduced by experimental design. It is well known that ML systems are highly susceptible to such dataset shifts, which often leads to unintended and potentially harmful behavior. For example, in ML-based electronic health record systems, input data is often characterized by shifting demographics, where clinical and operational practices evolve over time and a wrong prediction can threaten human safety.

Although dataset shift is a frequent cause of failure of ML systems, very few ML systems inspect incoming data for a potential distribution shift (Bulusu et al., 2020). While some practical methods such as (Rabanser et al., 2019) have been proposed for detecting shifts in applications with Euclidean structured data (speech, images, or video), there are limited efforts in solving such issues for graph structured data that naturally arises in several scientific and engineering applications. In recent years there has been a surge of interest in applying ML techniques to structured data, e.g. graphs, trees, manifolds etc. In particular, graph structured data is becoming prevalent in several high-impact applications including bioinformatics, neuroscience, healthcare, molecular chemistry and computer graphics. In this paper, we investigate the problem of detecting distribution shifts in graph-structured datasets for responsible deployment of ML in safety-critical applications. Specifically, we propose to solve the problem of detecting shifts in graph-structured data through the lens of statistical two-sample testing. Broadly, the objective in two-sample testing for graphs is to test whether two populations of random graphs are different or not based on the samples generated from each of them.

Two-sample testing has been of significant research interest due to its broad applicability. An important class of testing methods relies on summary metrics that quantify the topological differences between networks. For example, in brain network analysis, commonly adopted topological summary metrics include the global efficiency (Ginestet et al., 2011) and network modularity (Ginestet et al., 2014). An inherent challenge with these approaches is that the topological characteristics depend directly on the number of edges in the graph, and can be insufficient in practice. An alternative class of methods is based on comparing the structure of subgraphs to produce a similarity score (Shervashidze et al., 2009; Macindoe and Richards, 2010). For example, Shervashidze et al. (2009) used the earth mover’s distance between the distributions of feature summaries of their constituent subgraphs.

While these heuristic methods are reasonably effective for comparing real-world graphs, not until recently that a principled analysis of hypothesis testing with random graphs was carried out. In this spirit, Ginestet et al. (2017) developed a test statistic based on a precise geometric characterization of the space of graph Laplacian matrices. Most of these approaches for graph testing based on classical two-sample tests are only applicable to the restrictive low-dimensional setting, where the population size (number of graphs) is larger than the size of the graphs (number of vertices). To overcome this challenge, Tang et al. (2017a) proposed a semi-parametric two-sample test for a class of latent position random graphs, and studied the problem of testing whether two dot product random graphs are drawn from the same population or not. Other testing approaches that focused on hypothesis testing for specific scenarios, such as sparse networks (Ghoshdastidar et al., 2017a) and networks with a large number of nodes (Ghoshdastidar et al., 2017b), have been developed. More recently, Ghoshdastidar and von Luxburg (2018) developed a novel testing framework for random graphs, particularly for the cases with small sample sizes and the large number of nodes, and studied its optimality. More specifically, this test statistic was based on the asymptotic null distributions under certain model assumptions.

Unfortunately, all these approaches are limited to testing undirected graphs under the equal sample size (for two graph populations) setting. In real-world dataset shift detection problems, these assumptions are extremely restrictive, making existing approaches inapplicable to several applications. In order to circumvent these crucial shortcomings, we develop a novel approach based on hypothesis testing for detecting shifts in graph-structured data, which is more flexible (i.e., accommodates 1) both undirected and directed graphs and 2) unequal sample size cases). Moreover, it is highly effective even when the sample size grows. Notice that, similar to the setting in Ghoshdastidar and von Luxburg (2018), we also consider scenarios where all networks are defined from the same vertex set, which is common to several real-world applications. The main contributions of this paper are summarized below:

• We propose a new test statistic that can be applied to undirected graphs as well as directed graphs and/or unweighted graphs as well as weighted graphs, while eliminating the equal sample size requirement. The asymptotic distribution for the proposed statistic, based on the well-known U-statistic, is derived.

We consider the following two-sample setting. Let two random graph populations with d vertices be denoted as A 1 , … , A m from P ∈ [ 0,1 ] d × d and ℬ 1 , … , ℬ n from Q ∈ [ 0,1 ] d × d with their adjacency matrices A 1 , … , A m and B 1 , … , B n , respectively. We are concerned with testing hypotheses: H 0 : P = Q    vs     H 1 : P ≠ Q . (1)

Notice that we consider the cases where each population consists of independent and identically distributed samples, which encompasses a wide-range of network analysis problems, see, e.g., Holland et al. (1983), Newman and Girvan (2004), Newman (2006). In contrast to existing formulations, e.g., Ghoshdastidar and von Luxburg (2018), we consider a more flexible setup where 1) the sample sizes m and n are allowed to be different and 2) the graphs in p and Q can be weighted and/or directed.

While there have several efforts to two-sample testing of graphs (Bubeck et al., 2016; Gao and Lafferty, 2017; Maugis et al., 2017), recent works such as Tang et al. (2017a), Tang et al. (2017b); Ginestet et al. (2017) have focused on designing more general testing methods that are applicable to practical settings. For example, Ginestet et al. (2017) proposed a practical test statistic based on the correspondence between an undirected graph and its Laplacian under the inhomogeneous Erdős-Rényi (IER) assumption, which means all nodes are independently generated from a Bernoulli distribution (see details in Section 3). The test statistic, under the assumption of equal sample sizes m, can be described as follows: T g i n = ∑ i < j d [ ( A ¯ ) i j − ( B ¯ ) i j ] 2 a , (2) where a = 1 m ( m − 1 ) ∑ k = 1 m [ ( A k ) i j − ( A ¯ ) i j ] 2 + 1 m ( m − 1 ) ∑ k = 1 m [ ( B k ) i j − ( B ¯ ) i j ] 2 , ( A ¯ ) i j = 1 m ∑ k = 1 m ( A k ) i j ,     ( B ¯ ) i j = 1 m ∑ k = 1 m ( B k ) i j .

The authors showed that T g i n converges to a chi-square distribution as m → ∞ under H 0 . However, this statistic can be interpreted as Hotelling’s T 2 statistic for multivariate data, thus leading to no performance guarantees for “small m and large d” scenario. This is because the variance estimates used in Eq. 2 are not stable for small m and large d, especially when graphs are sparse.

Recently, Ghoshdastidar and von Luxburg (2018) proposed a new class of test statistics, designed for different scenarios under the IER model assumption. More specifically, they focused on cases with small m and large d. For cases with m > 1 , the following test statistic was used: T s p e c = | | ∑ k = 1 m ( A k − B k ) | | 2 max 1 ≤ i ≤ d ∑ j = 1 d ∑ j = 1 d ∑ k = 1 m [ ( A k ) i j + ( B k ) i j ] , (3)

While it was suggested by the authors to perform this test using bootstraps from the aggregated data, this could be challenging for sparse graphs, since it is difficult to construct bootstrapped statistics from an operator norm. Hence, they considered an alternate test statistic based on the Frobenius-norm as follows: T f r o = ∑ i < j d ( ∑ k ≤ m / 2 ( Δ k ) i j ) ( ∑ k > m / 2 ( Δ k ) i j ) ∑ i < j d ( ∑ k ≤ m / 2 ( S k ) i j ) ( ∑ k > m / 2 ( S k ) i j ) , (4) where ( Δ k ) i j = ( A k ) i j − ( B k ) i j and ( S k ) i j = ( A k ) i j + ( B k ) i j . It was shown that this test is provably effective and more reliable. Furthermore, they derived the asymptotic normality of T f r o as d → ∞ to make the method instantly applicable without the bootstrap procedure. Despite the good properties of this method, this test can be used only when the two sample sizes are equal, and when graphs are undirected. In the rest of this paper, we develop a new test statistic which addresses these two crucial limitations.

To carry out two-sample testing, we want to measure the distance between two populations. Here, we utilize the Frobenius distance as the evidence for discrepancy between two populations: T = ‖ P − Q ‖ F 2 . (5)

Next, we provide finite sample estimates of this quantity. To accommodate more general settings for random graphs, the new test statistic is defined as follows: T n e w = ∑ i = 1 d ∑ j = 1 d T i j , (6) where T i j = 1 m ( m − 1 ) ∑ k ≠ l m ( A k ) i j ( A l ) i j + 1 n ( n − 1 ) ∑ k ≠ l n ( B k ) i j ( B l ) i j − 2 m n ∑ k = 1 m ∑ l = 1 n ( A k ) i j ( B l ) i j .

Note that the proposed test statistic accommodates scenarios where 1 ) the sample sizes m and n are different and 2 ) the graphs in p and Q are weighted and/or directed.

Next, we analyze the theoretical properties of the proposed test. For the ease of theoretical analysis, we focus on the case where graphs are unweighted and undirected. However, the proposed test and algorithmic tools are applicable to weighted and/or directed graph scenarios which is the main focus of the paper and is considered in our experimental evaluations. More specifically, in our theoretical analysis, we assume that graphs are drawn from the inhomogeneous Erdős-Rényi (IER) random graph process, which is considered as an extended version of the Erdős-Rényi (ER) model from Bollobás et al. (2007). In other words, we consider unweighted and undirected random graphs, where edges occur independently without any additional structural assumption on the population adjacency matrix. Note, the IER model encompasses other models studied in the literature including random dot product graphs (Tang et al., 2017b) and stochastic block models (Lei et al., 2016). A graph G ∈ [ 0,1 ] d × d from a population symmetric adjacency p with zero diagonal is considered to be an IER graph if ( G ) i j i . i . d B e r n o u l l i ( P i j ) for all i , j ∈ { 1 , … , d } . Here, d denotes the cardinality of the vertex set. Next we analyze the theoretical properties of the proposed test under IER assumption.

LEMMA 3.1. T n e w is an unbiased empirical estimate of T, that is, E ( T n e w ) = T . (7)

PROOF. Under the IER assumptions, for all i , j = 1 , … , d , we have ( A k ) i j ( A l ) i j ∼ B e r n o u l l i ( P i j 2 ) , ∑ k ≠ l m ( A k ) i j ( A l ) i j ∼ B i n o m i a l ( m ( m − 1 ) , P i j 2 ) , ( B k ) i j ( B l ) i j ∼ B e r n o u l l i ( Q i j 2 ) , ∑ k ≠ l n ( B k ) i j ( B l ) i j ∼ B i n o m i a l ( n ( n − 1 ) , Q i j 2 ) , since A k and B l are mutually independent ( k = 1 , … , m ,       l = 1 , … , n ) . Then, E ( T ) = ∑ i = 1 d ∑ j = 1 d [ 1 m ( m − 1 ) m ( m − 1 ) P i j 2 + 1 n ( n − 1 ) n ( n − 1 ) Q i j 2 − 2 m n m n P i j Q i j ] = ∑ i = 1 d ∑ j = 1 d ( P i j − Q i j ) 2 = ∥ P − Q ∥ F 2 .

In the form of T i j , the first term and the second term represent a similarity (closeness) within two samples, and the last term represents similarity between two samples. Hence, a relatively large value of T n e w is the evidence against the null hypothesis. Note that the proposed statistic does not require equal sample sizes and undirected graphs assumptions.

When m = n , we have a simpler form of the estimate. Let Z = ( z 1 , … , z m ) be i . i . d random variables z k = ( A k , B k ) ∼ P × Q     ( k = 1 , … , m ) . Then, T n e w = ∑ i = 1 d ∑ j = 1 d T i j , (8) where T i j = 1 m ( m − 1 ) ∑ k ≠ l m h ( u k , u l ) i j , (9) and h ( z k , z l ) i j = ( A k ) i j ( A l ) i j + ( B k ) i j ( B l ) i j − ( A k ) i j ( B l ) i j − ( A l ) i j ( B k ) i j . Since the proposed estimate has a form of U-statistics, which provides a minimum-variance unbiased estimator for T (Hoeffding, 1992; Serfling, 2009), the asymptotic distribution of T n e w can be derived based on the asymptotic results of U-statistics.

Theorem 3.1 Assume E ( h 2 ) < ∞ . Under H 1 , we have m ( T n e w − T ) → d N ( 0 , d 2 σ 2 ) , (10) where σ 2 = var z ( E z ' h ( z , z ′ ) i j ) . Under H 0 , the U-statistic is degenerate and m T n e w → d ∑ u = 1 ∞ d 2 λ u ( ξ u 2 − 1 ) , (11) where ξ u i . i . d N ( 0,1 ) and λ u are the solutions of λ u ϕ u ( z ) = ∫ z ′ h ( z , z ′ ) i j ϕ u ( z ′ ) d P ( z ′ ) . (12)

PROOF. These results can be obtained by applying the asymptotic properties of U-statistics as given in Serfling (2009) and the IER assumptions.

Having devised the test statistic, our next aim is to determine whether the new test statistic T n e w is large enough to be outside the 1 − α quantile of the limiting null distribution in Eq. 11, where a is the significance level of the test. One difficulty in implementing this test is that the asymptotic null distribution 11) and its a quantile do not have an analytic form unless λ u = 0 or 1. Therefore, in order to estimate this quantile, we propose a permutation approach on the aggregated data. The main advantage of this method is that it yields a valid level a test in finite-sample scenarios (Lehmann and Romano, 2006). To this end, we first consider a simpler form of the test statistic (based on T n e w ) defined as follows: T ′ n e w = ∑ i = 1 d ∑ j = 1 d T ′ i j , (13) where T ′ i j = 1 m ( m − 1 ) ∑ k ≠ l m ( A k ) i j ( A l ) i j + 1 n ( n − 1 ) ∑ k ≠ l n ( B k ) i j ( B l ) i j . (14)

Although we do not use the last term of T i j in the definition of T ′ i j , the performance of the test statistic T ′ n e w achieved by incorporating similarities in two samples is still maintained in the permutation framework. The permutation test is summarized in Algorithm 1 ; its computational cost is O ( R ( m ∨ n ) 2 ) , where ( m ∨ n ) indicates the maximum among m and n.

Algorithm 1

Permutation test using T ′ n e w .

Input: Graph samples A 1 , … , A m and ℬ 1 , … , ℬ n ; Significance level α; Number of permutation R.

Unlike Ghoshdastidar and von Luxburg (2018) where the test is reliable even for a small number of samples, due to its asymptotic distribution, our test procedure needs a reasonable number of samples to implement the permutation test. Based on simulations, we see that as low as four samples are sufficient to obtain reliable results.

Here, we first examine the performance of the new test statistics under diverse settings through simulation studies. Later, we will apply the new test to real-world applications.

We propose the new two-sample test statistic for graph-structured data. Unlike the existing methods, the new test statistic is more versatile, which is applicable to directed graphs, imbalanced sample size cases, and even weighted graphs. The asymptotic distribution of the test statistic is presented and a practical testing procedure is proposed. The performance of the new method is studied under a number of settings. Experiments demonstrate that the new test in general outperforms state-of-the-art tests. The proposed test is also applied to two real datasets (including a safety-critical healthcare application), and we reveal that the new approach is effective to detecting the heterogeneity between disparate samples.