Algorithm 1 describes the overall structure of D-Cube . It first copies and assigns the given relation ℛ to ℛ o r i (line 1); and computes the sets of distinct attribute values composing ℛ (line 2). Then, it finds k dense subtensors one by one from ℛ (line 6) using its mass as a parameter (line 5). The detailed procedure for detecting a single dense subtensor from ℛ is explained in Section 3.1.2. After each subtensor ℬ is found, the tuples included in ℬ are removed from ℛ (line 7) to prevent the same subtensor from being found again. Due to this change in ℛ , subtensors found from ℛ are not necessarily the subtensors of the original relation ℛ o r i . Thus, instead of ℬ , the subtensor in ℛ o r i formed by the same attribute values forming ℬ is added to the list of k dense subtensors (lines 8–9). Notice that, due to this step, D-Cube can detect overlapping dense subtensors. That is, a tuple can be included in multiple dense subtensors.

Based on our assumption that the sets of distinct attribute values (i.e., { ℛ n } n = 1 N and { ℬ n } n = 1 N ) are stored in memory and can be randomly accessed, all the steps in Algorithm 1 can be performed by sequentially reading and writing tuples in relations (i.e., tensor entries) in disk without loading all the tuples in memory at once. For example, the filtering steps in lines 7–8 can be performed by sequentially reading each tuple from disk and writing the tuple to disk only if it satisfies the given condition.

Note that this overall structure of D-Cube is similar to that of M-Zoom (Shin et al., 2018) except that tuples are stored on disk. However, the methods differ significantly in the way each dense subtensor is found from ℛ , which is explained in the following section.

Algorithm 2 describes how D-Cube detects each dense subtensor from the given relation ℛ . It first initializes a subtensor ℬ to ℛ (lines 1–2) then repeatedly removes attribute values and the tuples of ℬ with those attribute values until all values are removed (line 5).

Specifically, in each iteration, D-Cube first chooses a dimension attribute A _i that attribute values are removed from (line 7). Then, it computes D _i , the set of attribute values whose masses are less than θ ( ≥ 1 ) times the average (line 8). We explain how the dimension attribute is chosen, in Section 3.1.3 and analyze the effects of θ on the accuracy and the time complexity, in Section 3.2. The tuples whose attribute values of A _i are in D _i are removed from ℬ at once within a single scan of ℬ (line 16). However, deleting a subset of D _i may achieve higher value of the metric ρ. Hence, D-Cube computes the changes in the density of ℬ (line 11) as if the attribute values in D _i were removed one by one, in an increasing order of their masses. This allows D-Cube to optimize ρ as if we removed attributes one by one, while still benefiting from the computational speedup of removing multiple attributes in each scan. Note that these changes in ρ can be computed exactly without actually removing the tuples from ℬ or even accessing the tuples in ℬ since its mass (i.e., M ℬ ) and the number of distinct attribute values (i.e., { | ℬ n | } n = 1 N ) are maintained up-to-date (11–12). This is because removing an attribute value from a dimension attribute does not affect the masses of the other values of the same attribute. The orders that attribute values are removed and when the density of ℬ is maximized are maintained (lines 13–15) so that the subtensor ℬ maximizing the density can be restored and returned (lines 17–18), as the result of Algorithm 2.

Note that, in each iteration (lines 5–16) of Algorithm 2, the tuples of ℬ , which are stored on disk, need to be scanned only twice, once in line 6 and once in line 16. Moreover, both steps can be performed by simply sequentially reading and/or writing tuples in ℬ without loading all the tuples in memory at once. For example, to compute attribute-value masses in line 6, D-Cube increases M ℬ ( t [ A n ] , n ) by t [ X ] for each dimension attribute A _n after reading each tuple t in ℬ sequentially from disk.

Algorithm_3

Algorithm_4

We discuss two policies for choosing a dimension attribute that attribute values are removed from. They are used in line 7 of Algorithm 2 offering different advantages.

Maximum Cardinality Policy (Algorithm 3): The dimension attribute with the largest cardinality is chosen, as described in Algorithm 3. This simple policy, however, provides an accuracy guarantee (see Theorem 3 in Section 3.2.2).

Maximum Density Policy (Algorithm 4): The density of ℬ when attribute values are removed from each dimension attribute is computed. Then, the dimension attribute leading to the highest density is chosen. Note that the tuples in ℬ , stored on disk, do not need to be accessed for this computation, as described in Algorithm 4. Although this policy does not provide the accuracy guarantee given by the maximum cardinality policy, this policy works well with various density measures and tends to spot denser subtensors than the maximum cardinality policy in our experiments with real-world data.

We present the optimization techniques used for the efficient implementation of D-Cube.

Combining Disk-Accessing Steps . The amount of disk I/O can be reduced by combining multiple steps involving disk accesses. In Algorithm 1, updating ℛ (line 7) in an iteration can be combined with computing the mass of ℛ (line 5) in the next iteration. That is, if we aggregate the values of the tuples of ℛ while they are written for the update, we do not need to scan ℛ again for computing its mass in the next iteration. Likewise, in Algorithm 2, updating ℬ (line 16) in an iteration can be combined with computing attribute-value masses (line 6) in the next iteration. This optimization reduces the amount of disk I/O in D-Cube about 30%.

Caching Tensor Entries in Memory . Although we assume that tuples are stored on disk, storing them in memory up to the memory capacity speeds up D-Cube up to 3 times in our experiments (see Section 4.4). We cache the tuples in ℬ , which are more frequently accessed than those in ℛ or ℛ o r i , in memory with the highest priority.

Theorem 1 states the worst-case time complexity, which equals to the worst-case I/O complexity, of D-Cube .

Lemma 1

(Maximum Number of Iterations in Algorithm 2). Let L = max n ∈ [ N ] | R n | . Then, the number of iterations (lines 5–16) in Algorithm 2 is at most N min ( log θ L , L ) .

Proof

In each iteration (lines 5–16) of Algorithm 2, among the values of the chosen dimension attribute A i , attribute values whose masses are at most θ M ℬ | ℬ i | , where θ ≥ 1 , are removed. The set of such attribute values is denoted by D i . We will show that, if | ℬ i | > 0 , then | ℬ i \ D i | < | ℬ i | / θ (1) Note that, when | ℬ i \ D i | = 0 , Eq. (1) trivially holds. When | ℬ i \ D i | > 0 , M ℬ can be factorized and lower bounded as M ℬ = ∑ a ∈ ℬ i \ D i M ℬ ( a , i ) + ∑ a ∈ D i M ℬ ( a , i ) ≥ ∑ a ∈ ℬ i \ D i i M ℬ ( a , i ) > | ℬ i \ D i | ⋅ θ M ℬ | ℬ i | , where the last strict inequality is from the definition of D i and that | ℬ i \ D i | > 0 . This strict inequality implies M ℬ > 0 , and thus dividing both sides by θ M ℬ | ℬ i | gives Eq. 1. Now, Eq. 1 implies that the number of remaining values of the chosen attribute after each iteration is less than 1 / θ of that before the iteration. Hence each attribute can be chosen at most log θ L times before all of its values are removed. Thus, the maximum number of iterations is at most N ⁡ log θ L . Also, by Eq. 1, at least one attribute value is removed per iteration. Hence, the maximum number of iterations is at most the number of attribute values, which is upper bounded by N L . Hence the number of iterations is upper bounded by N max ( log θ L , L ) .∎

Theorem 1 (Worst-case Time Complexity). Let L = max n ∈ [ N ] | ℛ n | . If θ = O ( e ( N | ℛ | L ) ) , which is a weaker condition than θ = O ( 1 ) , the worst-case time complexity of Algorithm 1 is O ( k N 2 | ℛ | min ( log θ L , L ) ) . (2)

Proof

From Lemma 1, the number of iterations (lines 5–16) in Algorithm 2 is O ( N min ( log θ L , L ) ) . Executing lines 6 and 16 O ( N min ( log θ L , L ) ) times takes O ( N 2 | ℛ | min ( log θ L , L ) ) , which dominates the time complexity of the other parts. For example, repeatedly executing line 9 takes O ( N L ⁡ log 2 ⁡ L ) , and by our assumption, it is dominated by O ( N 2 | ℛ | min ( log θ L , L ) ) . Thus, the worst-case time complexity of Algorithm 2 is O ( N 2 | ℛ | min ( log θ L , L ) ) , and that of Algorithm 1, which executes Algorithm 2, k times, is O ( k N 2 | ℛ | min ( log θ L , L ) ) .∎

However, this worst-case time complexity, which allows the worst distributions of the measure attribute values of tuples, is too pessimistic. In Section 4.4, we experimentally show that D-Cube scales linearly with k, N, and ℛ ; and sub-linearly with L even when θ is its smallest value 1.

Theorem 2 states the memory requirement of D-Cube . Since the tuples do not need to be stored in memory all at once in D-Cube , its memory requirement does not depend on the number of tuples (i.e., | ℛ | ).

Theorem 2

(Memory Requirements). The amount of memory space in Algorithm 1 is O ( ∑ n = 1 N | ℛ n | ​ ) .

Proof

In Algorithm 1, { { M ℬ ( a , n ) } a ∈ ℬ n } n = 1 N , { ℛ n } n = 1 N , and { ℬ n } n = 1 N need to be loaded into memory at once. Each has at most ∑ n = 1 N | ℛ n | values. Thus, the memory requirement is O ( ∑ n = 1 N | ℛ n | ​ ) . ∎

We show that D-Cube gives the same accuracy guarantee with in-memory algorithms proposed in Shin et al. (2018), if we set θ to 1, although accesses to tuples (stored on disk) are restricted in D-Cube to reduce disk I/Os. Specifically, Theorem 3 states that the subtensor found by Algorithm 2 with the maximum cardinality policy has density at least 1 θ N of the optimum when ρ a r i is used as the density measure.

Theorem 3

(θN-Approximation Guarantee). Let ℬ ∗ be the subtensor ℬ maximizing ρ a r i ( ℬ , ℛ ) in the given relation R. Let ℬ ˜ be the subtensor returned by Algorithm 2 with ρ a r i and the maximum cardinality policy. Then, ρ a r i ( ℬ ˜ , ℛ ) ≥ 1 θ N ρ a r i ( ℬ ∗ , ℛ ) .

Proof

First, the maximal subtensor ℬ ∗ satisfies that, for any i ∈ [ N ] and for any attribute value a ∈ ℬ i ∗ , its attribute-value mass M ℬ ∗ ( a , i ) is at least 1 N ρ a r i ( ℬ ∗ , ℛ ) . This is since the maximality of ρ a r i ( ℬ ∗ , ℛ ) implies ρ a r i ( ℬ ∗ − ℬ ∗ ( a , i ) , ℛ ) ≤ ρ a r i ( ℬ ∗ , ℛ ) , and plugging in Definition 1 to ρ a r i gives M ℬ ∗ − M ℬ ∗ ( a , i ) 1 N ( ( ∑ n = 1 N | ℬ n ∗ | ​ ) − 1 ) = ρ a r i ( ℬ ∗ − ℬ ∗ ( a , i ) , ℛ ) ≤ ρ a r i ( ℬ ∗ , ℛ ) = M ℬ ∗ 1 N ∑ n = 1 N | ℬ n ∗ | ​ , which reduces to M ℬ ∗ ( a , i ) ≥ 1 N ρ a r i ( ℬ ∗ , ℛ ) . (3)

Consider the earliest iteration (lines 5–16) in Algorithm 2 where an attribute value a of ℬ ∗ is included in D i . Let ℬ ′ be ℬ in the beginning of the iteration. Our goal is to prove ρ a r i ( ℬ ′ , ℛ ) ≥ 1 θ N ρ a r i ( ℬ ∗ , ℛ ) , which we will show as ρ a r i ( ℬ ˜ , ℛ ) ≥ ρ a r i ( ℬ ′ , ℛ ) ≥ M ℬ ′ ( a , i ) θ ≥ M ℬ ∗ ( a , i ) θ ≥ 1 θ N ρ a r i ( ℬ ∗ , ℛ ) .

First, ρ a r i ( ℬ ˜ , ℛ ) ≥ ρ a r i ( ℬ ′ , ℛ ) is from the maximality of ρ a r i ( ℬ ˜ , ℛ ) among the densities of the subtensors generated in the iterations (lines 1:line:single:order1-1:line:single:order2 in Algorithm 2). Second, applying | ℬ i ′ | ≥ 1 N ∑ n = 1 N | ℬ n ′ | ​ from the maximum cardinality policy (Algorithm 3) to Definition 1 of ρ a r i gives ρ a r i ( ℬ , ℛ ) = M ℬ 1 N ∑ n = 1 N | ℬ n ′ | ​ ≥ M ℬ ′ | ℬ i ′ | . And a ∈ D i gives θ M ℬ ′ | ℬ ′ | ≥ M ℬ ′ ( a , i ) . So combining these gives ρ a r i ( ℬ ′ , ℛ ) ≥ M ℬ ′ ( a , i ) θ . Third, M ℬ ′ ( a , i ) θ ≥ M ℬ ∗ ( a , i ) θ is from ℬ ′ ⊃ ℬ ∗ . Fourth, M ℬ ∗ ( a , i ) θ ≥ 1 θ N ρ a r i ( ℬ ∗ , ℛ ) is from Eq. (3). Hence, ρ a r i ( ℬ ˜ , ℛ ) ≥ 1 θ N ρ a r i ( ℬ ∗ , ℛ ) holds. ∎

While D-Cube requires only O ( ∑ n = 1 N | ℛ n | ​ ) memory space (see Theorem 2), which does not depend on the number of tuples (i.e., | ℛ | ), M-Zoom and M-Biz require additional O ( N | ℛ | ) space for storing all tuples in main memory. The worst-case time complexity of D-Cube is O ( k N 2 | ℛ | min ( log θ L , L ) ) (see Theorem 1), and it is slightly higher than that of M-Zoom, which is O ( k N | ℛ | log ⁡ L ) . Empirically, however, D-Cube is up to 7× faster than M-Zoom, as we show in Section 4. The main reason is that D-Cube reads and writes tuples only sequentially, allowing efficient caching based on spatial locality. On the other hand, M-Zoom requires tuples to be stored and accessed in hash tables, making efficient caching difficult. ¹ The time complexity of M-Biz depends on the number of iterations until reaching a local optimum, and there is no known upper bound on the number of iterations tighter than O ( 2 ( ∑ n = 1 N | ℛ n | ​ ) ) . If ρ a r i is used, M-Zoom and M-Biz ² give an approximation ratio of N, which is the approximation ratio of D-Cube when θ is set to 1 (see Theorem 3).

We ran all serial algorithms on a machine with 2.67GHz Intel Xeon E7-8837 CPUs and 1TB memory. We ran MapReduce algorithms on a 40-node Hadoop cluster, where each node has an Intel Xeon E3-1230 3.3GHz CPU and 32GB memory.

We describe the real-world and synthetic tensors used in our experiments. Real-world tensors are categorized into four groups: (a) Rating data (SWM, Yelp, Android, Netflix, and YahooM.), (b) Wikipedia revision histories (KoWiki and EnWiki), (c) Temporal social networks (Youtube and SMS), and (d) TCP dumps (DARPA and AirForce). Some statistics of these datasets are summarized in Table 3.

Rating data. Rating data are relations with schema (user, item, timestamp, score, #ratings). Each tuple (u,i,t,s,r) indicates that user u gave item i score s, r times, at timestamp t. In the SWM dataset (Akoglu et al., 2013), the timestamps are in dates, and the items are entertaining software from a popular online software marketplace. In the Yelp dataset, the timestamps are in dates, and the items are businesses listed on Yelp, a review site. In the Android dataset (McAuley et al., 2015), the timestamps are hours, and the items are Android apps on Amazon, an online store. In the Netflix dataset (Bennett and Lanning, 2007), the timestamps are in dates, and the items are movies listed on Netflix, a movie rental and streaming service. In the YahooM. dataset (Dror et al., 2012), the timestamps are in hours, and the items are musical items listed on Yahoo! Music, a provider of various music services.

Wikipedia revision history. Wikipedia revision histories are relations with schema (user, page, timestamp, #revisions). Each tuple (u,p,t,r) indicates that user u revised page p, r times, at timestamp t (in hour) in Wikipedia, a crowd-sourcing online encyclopedia. In the KoWiki dataset, the pages are from Korean Wikipedia. In the EnWiki dataset, the pages are from English Wikipedia.

Temporal social networks. Temporal social networks are relations with schema (source, destination, timestamp, #interactions). Each tuple (s,d,t,i) indicates that user s interacts with user d, i times, at timestamp t. In the Youtube dataset (Mislove et al., 2007), the timestamps are in hours, and the interactions are becoming friends on Youtube, a video-sharing website. In the SMS dataset, the timestamps are in hours, and the interactions are sending text messages.

TCP Dumps. The DARPA dataset (Lippmann et al., 2000), collected by the Cyber Systems and Technology Group in 1998, is a relation with schema (source IP, destination IP, timestamp, #connections). Each tuple (s,d,t,c) indicates that c connections were made from IP s to IP d at timestamp t (in minutes). The AirForce dataset, used for KDD Cup. 1999, is a relation with schema (protocol, service, src bytes, dst bytes, flag, host count, srv count, #connections). The description of each attribute is as follows:

protocol: type of protocol (tcp, udp, etc.).

Synthetic Tensors: We used synthetic tensors for scalability tests. Each tensor was created by generating a random binary tensor and injecting ten random dense subtensors, whose volumes are 10 ^N and densities (in terms of ρ a r i ) are between 10× and 100× of that of the entire tensor.

We implemented the following dense-subtensor detection methods for our experiments.

D-Cube (Proposed): We implemented D-Cube in Java with Hadoop 1.2.1. We set the mass-threshold parameter θ to 1 and used the maximum density policy for dimension selection, unless otherwise stated.

D-Cube detected network attacks from TCP dumps accurately by spotting corresponding dense subtensors. We consider two TCP dumps that are modeled differently. The DARPA dataset is a 3-way tensor where the dimension attributes are source IPs, destination IPs, and timestamps in minutes; and the measure attribute is the number of connections. The AirForce dataset, which does not include IP information, is a 7-way tensor where the measure attribute is the same but the dimension attributes are the features of the connections, including protocols and services. Both datasets include labels indicating whether each connection is malicious or not.

Figure 1C in Section 1 lists the five densest subtensors (in terms of ρ g e o ) found by D-Cube in each dataset. Notice that the dense subtensors are mostly composed of various types of network attacks. Based on this observation, we classified each connection as malicious or benign based on the density of the densest subtensor including the connection (i.e., the denser the subtensor including a connection is, the more suspicious the connection is). This led to high area under the ROC curve (AUROC) as seen in Table 4, where we report the AUROC when each method was used with the density measure giving the highest AUROC. In both datasets, using D-Cube resulted in the highest AUROC.

D-Cube spotted suspicious synchronized behavior accurately in rating data. Specifically, we assume an attack scenario where fraudsters in a review site, who aim to boost (or lower) the ratings of the set of items, create multiple user accounts and give the same score to the items within a short period of time. This lockstep behavior forms a dense subtensor with volume (# fake accounts × # target items × 1 × 1) in the rating dataset, whose dimension attributes are users, items, timestamps, and rating scores.

We injected 10 such random dense subtensors whose volumes varied from 15×15×1×1 to 60×60×1×1 in the Yelp and Android datasets. We compared the ratio of the injected subtensors detected by each dense-subtensor detection method. We considered each injected subtensor as overlooked by a method if the subtensor did not belong to any of the top-10 dense subtensors spotted by the method or it was hidden in a natural dense subtensor at least 10 times larger than the injected subtensor. That is, we measured the recall at top 10. We repeated this experiment 10 times, and the averaged results are summarized in Table 4. For each method, we report the results with the density measure giving the highest recall. In both datasets, D-Cube detected a largest number of the injected subtensors. Especially, in the Android dataset, D-Cube detected 9 out of the 10 injected subtensors, while the second best method detected only 7 injected subtensors on average.

D-Cube successfully spotted spam reviews in the SWM dataset, which contains reviews from an online software marketplace. We modeled the SWM dataset as a 4-way tensor whose dimension attributes are users, software, ratings, and timestamps in dates, and we applied D-Cube (with ρ = ρ a r i ) to the dataset. Table 6 shows the statistics of the top-3 dense subtensors. Although ground-truth labels were not available, as the examples in Table 5 show, all the reviews composing the first and second dense subtensors were obvious spam reviews. In addition, at least 48% of the reviews composing the third dense subtensor were obvious spam reviews.

D-Cube detected interesting anomalies in Wikipedia revision histories, which we model as 3-way tensors whose dimension attributes are users, pages, and timestamps in hours. Table 6 gives the statistics of the top-3 dense subtensors detected by D-Cube (with ρ = ρ a r i and the maximum cardinality policy) in the KoWiki dataset and by D-Cube (with ρ = ρ g e o and the maximum density policy) in the EnWiki dataset. All three subtensors detected in the KoWiki dataset indicated edit wars. For example, the second subtensor corresponded to an edit war where 4 users changed 4 pages, 1,011 times, within 5 h. On the other hand, all three subtensors detected in the Enwiki dataset indicated bot activities. For example, the third subtensor corresponded to 3 bots which edited 1,067 pages 973,747 times. The users composing the top-5 dense subtensors in the EnWiki dataset are listed in Table 7. Notice that all of them are bots.