A common term for runtime updating is Dynamic Software Updating (DSU) (Hicks and Nettles, 2005; Pina et al., 2014). With DSU, running processes are updated by rewriting the running code and process memory. Compared to traditional software updating, the process does not need to be stopped, although possibly it can be temporarily halted. Because of this, the state of the running process can be preserved. As a result, running sessions and connections can be kept active and no costly application boot is required.

Many early updating systems require a specialized program environment. Notably, Cook and Lee (1983) have described a system called DYMOS that encompasses nearly all aspects of a software system: a command interpreter, a source code manager, an editor, a compiler, and a runtime support system. By having control over all of these aspects, they have the ability to add and monitor synchronization systems allowing the updates to be performed seamlessly.

More recent models for DSU do not require an all-encompassing system. Instead, DURTS (Montgomery, 2004) requires only a custom linker and a module to load and synchronize replacement modules. The linked module is loaded into heap space from within the application and a pointer-to-function variable is used to execute the function, where the pointer value is updated to point to newer versions. Similarly, Hicks and Nettles (2005) have described an approach where they used the C-like language Popcorn, compiling code patches into Typed Assembly Language that can be dynamically linked and integrated. There are other works on a language level like (Mugarza et al., 2020), which are implemented on the Ada programming language. Alternatively, Bagherzadeh et al. (2020) present a language-independent approach based on model execution systems (Hojaji et al., 2019).

Some other systems use and extend the functionality provided by the platform an application runs on. For example, Kim et al. (2011) used the Java Virtual Machine (JVM) HotSwap capability to replace code, adding features such as bytecode rewriting to work around HotSwap limitations. One of the more complex systems is Rubah (Pina et al., 2014). It uses a manual definition of update points, combined with bytecode rewriting. Their update process consists of three steps. The first step is quiescence, reaching a stable state where it is safe to perform updates. In the next step, the running state is transformed, going over the objects in heap memory and changing the fields and methods from existing objects to their new versions. Next, the program threads are restarted at their equivalent location in the new version of the application. Later, Pina et al. (2019) improved system availability by warming up updates. They run old and new versions and perform the update if both versions converge. Neumann et al. (2017) and Gu et al. (2018) later introduced similar systems based on the same principles. Šelajev and Gregersen (2017) presented a runtime state analysis system to detect runtime issues caused by updating Java applications.

Clearly, many different DSU systems and types of systems exist. In fact, as early as 1993, Segal and Frieder (1993) have given a summary of on-the-fly updating. Some solutions require a complete restructuring of code to support updates and some even act as an entire operating system. Seifzadeh et al. (2013) have provided a more recent overview of different dynamic software update frameworks and approaches. They have also included a categorization for these updating frameworks and describe the metrics by which the frameworks are then compared. Mugarza et al. (2018) also analyzed the existing DSU techniques focusing on safety and security.

Most approaches require complete control over the environment in which the software is executed. However, developing applications in distributed systems is much more complicated, because process control has been handed to distributed data processing platforms such as Apache Spark (Zaharia et al., 2010), Flink (Carbone et al., 2015), and Storm (Toshniwal et al., 2014) instead of the user code. The platforms handle distribution, scheduling, and execution automatically, and users have only marginal influence in these areas. Implementing the approaches described above would require changes to the distributed processing platforms. However, modifying the existing distributed stream processing frameworks is undesirable as they are meant to act as a general core, where the user applications simply use existing functionality. Solutions should then be found that work within the current control paradigms to also work with the newer version of the application.

Updating variables of a pipeline is a common way of adding flexibility to distributed pipelines. Boyce and Leger (2020) provided a solution on top of Apache Spark for changing variables at the runtime by extending Broadcast variables. In Apache Flink, one can use the CoFlatMapFunction to get two streams of the original data and the parameters streams and assign parameters for each data record or use Apache Zookeeper (Hunt et al., 2010) for storing the configuration and let the Apache Storm application listen for an update. However, all these approaches have two significant limitations, 1) only the parameters can be updated; 2) the updates should be anticipated before launching an application.

Another approach is to use variation points as originally defined for Software Product Lines (SPLs) (Pohl et al., 2005). SPL is a concept where reusable components are created for a domain that can then be composed in multiple ways to develop new products. However, the variation points still need to be defined before launching the software. Overcoming this issue, Dynamic Software Product Lines (DSPLs) (Hallsteinsen et al., 2008; Eichelberger, 2016) extend SPL to allow the composition of predefined components to be done at runtime.

To apply DSPLs to distributed data processing pipelines, we must first be able to model such pipelines. Berger et al. (2014) and Dhungana et al. (2014) have described approaches to model topological variability, by which they mean connecting components in a specific order and in interconnected hierarchies. These hierarchies then have to be respected during reconfiguration. Qin and Eichelberger (2016) have previously implemented DSPLs on top of Apache Storm, allowing runtime switching between alternatives, but requiring that they already be implemented at design time.

In a previous work done by the authors (Lazovik et al., 2017), we have investigated the feasibility of dynamically updating the processing pipeline of an Apache Spark application. Apache Spark is one of the most popular big data processing platforms. It is a unified engine providing various operations, including SQL, Machine Learning, Streaming, and Graph Processing. Spark is based on the concept of Resilient Distributed Datasets (RDDs). An RDD is a read-only, distributed collection partitioned into distinct sets distributed over a computing cluster. RDDs form a pipeline which is a DAG where performing an operation over one RDD results in a new RDD. Edges of the Spark DAG are standard operations, e.g., map and reduce, while inside each operation, there is a user-defined function. Internally, the Spark task scheduler uses scopes instead of RDDs, where a single operation is represented by one scope but may internally create multiple (temporary) RDDs.

The spark-dynamic framework is an extension on top of Apache Spark, making it able to update both parameters and functions within pipeline steps during runtime. Variation points can be updated using a REST API, where functions are updated by providing a new byte code. Instead of the fixed alternatives of Qin and Eichelberger (2016), new algorithms and new version of algorithms can be used. The extension wraps each operation to pull the updated value for parameters and functions on every invocation. The wrapped methods are named dynamic(Operation) where operation is the original method name. For example, dynamicMap is a wrapper for the map operation.

Apart from updating parameters and functions, spark-dynamic can also change data sources on the fly. An intermediate Data Access Layer is introduced to intervene between the Spark processing pipeline and Spark Data Source Relation, which is responsible for preparing the RDD for the requested data.

The performance of the prototype was also measured as part of the feasibility study, with promising results (Lazovik et al., 2017). The solutions from this paper are applied on top of this earlier system.

With research showing the feasibility of modeling and updating distributed processing pipelines, given a distributed computational pipeline with placeholders, we should also be able to automatically select a component for each placeholder to satisfy the goal of the pipeline. To ensure that the newly generated pipeline configuration is valid and, for a running pipeline, that parameter updates do not introduce any errors, we must apply some form of consistency checking. When developers want to introduce an update, they are able to change both objects and functions as long as their signatures stay the same. However, these signatures do not describe all details of these updates. For example, a function may take the same types and number of arguments and yet provide different results. Consider f(a: Double, b: Double) → a∗b and g(a: Double, b: Double) → a/b that share the same signature but should be used differently. Since only signature checking is not enough, we must research other methods of consistency checking. We have focused on the topics of model checking, constraint programming, and automated planning since these techniques are relatively popular (Ghallab et al., 2004) and extensible and do not require the use of complex features such as a cost function or probability calculations.

Model checking is a brute-force method of examining all possible states of a system (Baier and Katoen, 2008), to determine if, given a program M and a specification h, the behavior of M meets the specification h (Emerson, 2008). The program is represented in specialized languages such as PROMELA or through analysis of source code such as C (Merz, 2001). The specification can among others be done through Linear Temporal Logic (LTL) and Computation Tree Logic (CTL).

We could use model checking to verify if a proposed pipeline configuration is valid. For the generation of configurations, however, we would need to brute-force the search space; i.e., we would need to iterate over every possible configuration until one is found that satisfies the property specification. This method of evaluation is not very efficient and therefore we would also have to investigate heuristics to speed up the process.

In constraint programming, the behavior of a system is specified through constraints, for example, by restricting the domains of individual variables or imposing constraints on groups of variables (Bockmayr and Hooker, 2005). This is done by having each subsequent constraint restrict the possible values in a constraint store. This way all possible combinations can be tested and a solution can be given if all constraints can be met. Basic constraints exist such as v ₁.gt(v ₂) that defines an arithmetic constraint over two variables v ₁ and v ₂, as well as global constraints such as model .allDifferent(v ₁,…,v _n ) (van Hoeve, 2001) that are defined over sets of variables.

Automated planning is a relatively broad subject, but classical planning is perhaps the most general. Classical planning is based on transition systems. States are connected through transitions, in which an action is applied that actually changes the one state into a consecutive one (Ghallab et al., 2004). An action typically has preconditions and effects. The preconditions are propositions required on a state to be able to apply the action, and the effects are the propositions set on the resulting state. Note that, in literature, an action is typically defined as a ground instance of a planning operator or action template (Ghallab et al., 2016). These operators or templates can include parameters that define the targets on which the preconditions and effects must apply. For simplicity, we will only reason about grounded planning operators in this work, and as such we will use only the term action instead of planning operator. Planning aims to solve a planning problem, which often consists of the transition system, one or more initial states, and one or more goal states.

Different types of transition systems are used, one of which is State Transition Systems (STSs) (Ghallab et al., 2004; Ghallab et al., 2016). In this model, states can be changed not only by actions but also by events, which cannot be controlled. They are defined as Σ = (S, A, E, γ), with S being the set of states, A being the set of actions, E being the set of events, and the state transition function γ: S × (A ∪ E) → 2 ^S (Nau, 2007). Further, restricted STSs do not allow events; thus, Σ = (S, A, γ). In this case, Ghallab et al. (2016) define the transition function as γ: S × A → S since there is only one resulting state for a transition. A planning problem on such a system is defined as P = ( Σ , s 0 , g ) , with s ₀ being the initial state and g being the set of goal states.

Ghallab et al. (2004) distinguish three representations of classical automated planning: set-theoretic representation, classical representation, and state-variable representation.

In the set-theoretic representation, each state is a set of propositions, and each action has propositions that are required to apply the action (preconditions), propositions that will be added to a new state when the action is applied (positive effects), and propositions that will be removed (negative effects). The classical representation is similar to the set-theoretic representation, except states are logical atoms that are either true or false, and actions change the truth values of these atoms. In the state-variable representation, each state contains the values for a set of variables, where different states contain different values for these variables and actions are partial functions that map between these tuples.

These base techniques are often extended. For example, some authors describe the use of model checking to solve planning problems (Cimatti et al., 1998; Giunchiglia and Traverso, 2000). Similarly, Kaldeli (2013) describes a planner built through constraint programming, based on earlier work from Lazovik et al. (2005), Lazovik et al. (2006). The planner definitions such as the actions and goals are first translated into constraints. Then, the constraints are evaluated using an off-the-shelf Constraint Satisfaction Problem (CSP) solver.

When comparing model checking and automated planning, both techniques have the same time complexity (EXPTIME or NEXPTIME for nondeterministic planning) (Ghallab et al., 2004; Meier et al., 2008). However, instead of the brute-force validation of configurations when using model checking, automated planning can use heuristics to reduce the search space, which could decrease the final planning time. Furthermore, automated planning techniques use proven concepts for the generation of plans which could serve as a basis on top of which we define our pipeline concepts, whereas for model checking, the basic configuration generation algorithms would still need to be designed.

Finally, we compare constraint programming and automated planning. The automated planning concepts are more abstract, separating the planning goal from the state and transition behaviors. Planning has a solid foundation with its transition systems, while constraint programming permits a lot of freedom in the behavior that can be built using constraints. Instead of choosing between these two techniques, we can combine them. We can use constraint programming as a basis to create an automated planner, by implementing the transition systems and other planning concepts using constraints. We then retain the flexibility of constraint programming in case we want to add features to the planner later.

Before we describe how to transform our pipeline information into a planning problem, we must first formulate the problem of pipeline reconfiguration as a planning problem.

We modify the classical definitions presented by Ghallab et al. (2004); since our planning problem does not contain a single initial state, our transition system can contain branches and joins and our plans must match the topology of the Spark pipeline we are planning for. Thus, we define our planning problem as P = ( Σ , Σ ≃ , S 0 , S g ) (1) With the transition system Σ = (S, A, γ) and with Σ ≃ to map the planning problem to our domain, we simplify the notation s _m ∈ γ(s _n , a), that is, the application of action a onto s _n resulting in s _m , as s n → a s m . Each state in S is represented by state variables; i.e., ( ∀ s ∈ S ) : s = { v ∈ V a r s | ( v , v a l ( s , v ) ⊆ D o m ( v ) ) } (2)

Here, Vars is the set of variables in our planning problem, Dom(v) is the domain of a specific variable v (i.e., all possible assignments to an instance of that variable), and val(s, v) represents the value or possible values of the state-variable representing variable v in state s. S ₀ ⊆ S is the set of initial states and S _g ⊆ S is the set of goal states. Note also that there are no states before the initial states, i.e., (∀s ₀ ∈ S ₀)(∀s _n ∈ S)(∀a ∈ A): s ₀∉γ(s _n , a), and there are no states after the goal states, i.e., (∀s _g ∈ S _g )(∀a ∈ A): γ(s _g , a) = ∅.

As in Ghallab et al. (2004), we define an action as a = ( n a m e ( a ) , p r e c o n d ( a ) , e f f e c t s ( a ) ) (3)

Supported preconditions and effects are variable equality (both with constant or other variables), constraint conjunction, constraint disjunction, and the negation of these constraints [e.g., “(a := 1) &:& (b !:= true)”]. Similar to (Kaldeli, 2013), each constraint is a propositional formula over a state variable or a combination of two constraints. To apply s n → a s m , we must ensure that s _n satisfies all preconditions of the action and that the effects of the action can be applied onto s _m . We can encode this with a generalized “state satisfies constraints” relation, where s _n satisfies precond(a) and s _m satisfies effects(a). We define this relation, using Cstrs as the set of all constraints in the planning problem, as ( ∀ c ∈ C s t r s ) : s a t f ( s t a t e , c ) (4) s a t f ( s , c o n s t r a i n t ( a , o p , b ) ) = v a l ( s , a ) = b , if  a ∈ V a r s ∧ b ∈ D o m ( a ) v a l ( s , a ) = v a l ( s , b ) , if  a ∈ V a r s ∧ b ∈ V a r s e q v a l ( s , a ) ≠ b , if  a ∈ V a r s ∧ b ∈ D o m ( a ) v a l ( s , a ) ≠ v a l ( s , b ) , if  a ∈ V a r s ∧ b ∈ V a r s n e q s a t f ( s , a ) ∧ s a t f ( s , b ) , if  a ∈ C s t r s ∧ b ∈ C s t r s a n d s a t f ( s , a ) ∨ s a t f ( s , b ) , if  a ∈ C s t r s ∧ b ∈ C s t r s o r ¬ s a t f ( s , a ) ∨ ¬ s a t f ( s , b ) , if  a ∈ C s t r s ∧ b ∈ C s t r s n a n d ¬ s a t f ( s , a ) ∧ ¬ s a t f ( s , b ) , if  a ∈ C s t r s ∧ b ∈ C s t r s n o r (5) where a constraint is described by a source a, a target b, and an operation op, with op translated from a constraint as defined by Table 1.

We also formulate the frame axiom similar to Barták et al. (2010) and Kaldeli (2013), which specifies that, unless the action being applied on a state modifies a state variable, the state variable will remain the same in the state following that action: ( ∀ s n ∈ S ) ( ∀ s m ∈ S ) ( ∀ v ∈ V a r s ) ( ∀ a ∈ A ) : s m ∈ γ ( s n , a ) ⇒ v a l ( s n ) , v = v a l ( s m , v ) ∨ v a f f e c t e d B y A c t i o n ( a ) (6) v a f f e c t e d B y A c t i o n ( a ) ≡ ( ∃ c ∈ e f f e c t s ( a ) ) : c a f f ( v ) (7) c o n s t r a i n t ( a , o p , b ) a f f ( v ) = a = v , if  a ∈ V a r s b = v , if  b ∈ V a r s a a f f ( v )   ∨   b a f f ( v ) , if  o p ∈ { a n d , o r , n a n d , n o r } f a l s e , otherwise (8) With the base planning model fully described, we extend it below to support the planning of pipelines.

Since the plans we generate must be applied onto a fixed Spark pipeline topology, we cannot simply generate any plan that satisfies the goal constraints. Instead, we must relate our STS (State → Action → State) to the Spark DAG (Scope → Operation → Scope). To implement this mapping, we first construct a separate transition system to which the final plan must be isomorphic. That is, any relation in the STS should be represented in the DAG and vice versa. We define this isomorphic system as Σ ≃ = ( R , T , r d d , V P , v p , γ ≃ ) (9)

We describe the parts of this equation in this section. The set R represents the RDD scopes in the pipeline, and T represents the Spark operations that transform one RDD scope to a new scope. The transition function γ ≃ : R × T → R describes how the transitions apply to the RDD scopes.

Besides adding constraints to actions, we also allow constraints to be added to the pipeline topology itself, both manually by the user and automatically inferred from the pipeline topology, such as the datatype of the initial RDD. Preconditions added by the user are applied to the scope they are defined on, while effects are applied to the scope(s) following it.

We connect these transition systems through the relation rdd: S → R, where ( ∀ s n ∈ S ) ( ∀ a ∈ A ) ( ∀ s m ∈ S ) : s m ∈ γ ( s n , a ) ⇒ ( ∃ t ∈ T ) ( ∃ r n ∈ R ) ( ∃ r m ∈ R ) : r n = r d d ( s n ) ∧ r m = r d d ( s m ) ∧ r m ∈ γ ≃ ( r n , t ) ∧ s m  satisfies  e f f e c t s ( r n ) ∪ p r e c o n d ( r m ) (10)

Figure 3 shows an example mapping between planning transition system Σ and a Spark DAG Σ ≃ , based on the scenario from Section 1. In this example, γ for some states contains multiple applicable actions (e.g., a ₁ and a ₂ can both be applied from the first state) or multiple possible result states based on an action (e.g., a ₁ from the first state leads to two next states if an or-constraint was used).

Since we only support planning for our variation points, we do not include in our planning problem any scopes that do not contain a variation point. On the other hand, we add a scope at the end of the pipeline so that we can attach the goal constraints, even if the final action of the pipeline does not contain a variation point.

We also add special handling for join operations. We show this in Figure 3 as a transition function in the form of γ(s _n , i _n ). These transitions are applied when there is a transition in the Spark DAG that the user has no control over. For example, when joining two RDDs, the user cannot provide their own function that will be applied during the join. Nevertheless, we want to encode these transitions in order to accurately represent the pipeline. We introduce these implicit transitions in Section 4.2.3.

We use the relation vp: T → VP to map our transitions to variation points. This allows us to take into account additional constraints relating to the topology, such as the input/output combinations from Table 2, which we will discuss in Section 4.2.1. Additionally, we encode a constraint that ensures that if a single variation point is used in multiple transitions in a pipeline, the actions applied in those transitions are the same. This is done because one variation point can only hold a single PlanningModule and therefore can only be assigned a single action. ( ∀ s n , s m , s k , s l ∈ S ) ( ∀ a i , a j ∈ A ) ( ∃ t i , t j ∈ T ) : s m ∈ γ ( s n , a i ) ∧ s l ∈ γ ( s k , a j ) ∧ r d d ( s m ) ∈ γ ≃ ( r d d ( s n ) , t i ) ∧ r d d ( s l ) ∈ γ ≃ ( r d d ( s k ) , t j ) ∧ v p ( t i ) = v p ( t j ) ⇒ a i = a j (11)

Expanding our definition of the transition function s n → a s m with the isomorphism requirement, we get s n → a s m ⇔ s n  satisfies  p r e c o n d ( a ) ∧ s m  satisfies  e f f e c t s ( a ) ∧ ( ∀ v ∈ V a r s ) : v a l ( s n , v ) = v a l ( s m , v ) ∨ v a f f e c t e d B y A c t i o n ( a ) ∧ ( ∃ t ∈ T ) : r d d ( s m ) ∈ γ ≃ ( r d d ( s n ) , t ) ∧ s m  satisfies  e f f e c t s ( r d d ( s n ) ) ∧ s m  satisfies  p r e c o n d ( r d d ( s m ) ) (12)

Now that we have a formal description of our planning model; we can describe how we have implemented it. We have used the Java-based Choco-solver (Prud’homme et al., 2017) library to implement our CSP-based planner, which has good interoperability with Scala.

We encode each “state variable” as a CSP variable [e.g., val(s ₁, y) is encoded as variable y@1]. The domain of the variable is the set of possible assignments found in the planning problem. All values are converted to integers in the CSP. For the Type variables, we first convert a type name into a fully qualified name [e.g., Seq(String) becomes “scala.collection.Seq(java.lang.String)”], and for each unique name, we assign a unique integer in the domain. The actions applied on transitions are tracked through “action variables” (e.g., a1@1 → 3), where a value of true indicates that the action (in this case a1) is applied in that transition (in this case 1 → 3). Figure 4C shows a CSP encoding of the join example discussed in subsection 4.2.3.

Encoding the constraints from Eq. 5 defined on the topology and on actions is also straightforward, since equality and inequality can be encoded as constraints {e.g., val(s, v) = b becomes arithm[v@s, “=,” encode(b)]}, with encode being the conversion of values described above. Conjunction and disjunction of constraints can also be encoded in the CSP [e.g., val(s, v ₁) ≠ val(s, v ₂) becomes arithm(v1@s, “!=” v2@s)].

By only creating transitions between the scopes following the Spark DAG, we automatically fulfill part of the requirement of the plan to be isomorphic to Σ ≃ such that s n → a s m ⇒ ( ∃ t ∈ T ) : r d d ( s m ) ∈ γ ≃ ( r d d ( s n ) , t ) .

Subsequently, we add a constraint stating that exactly one of the action variables in each transition can be used. If we treat a false Boolean value as zero and a true Boolean value as one, we can add a constraint that sums each action variable in a transition and ensures the sum is equal to one.

Finally, we add an optimization objective that maximizes the occurrences of the no-op actions, to ensure that we do not perform extremely unnecessary processing (e.g., performing some function f, undoing it, and then redoing it).

From the action variables defined on the transitions in the CSP, we can then extract the actions assigned on those transitions, which correspond to user code that should be assigned to variation points.

Having defined our planning system, we must first know that it provides correct results before it can be used in practice. This is based on the concepts of soundness and completeness (Ghallab et al., 2016). A planning system is sound if, for any solution plan it returns, the plan is a solution for the planning problem. A system is complete if, given a solvable planning problem, the system will return at least one solution plan.

Because we represent our planning problem as a CSP and use an existing constraint solver, we do not evaluate the planning problems ourselves. Nevertheless, we can guarantee that the planning process is complete; i.e., it will eventually stop and result in either a generated configuration or a failure. This is true because our planning problem is finite (the Spark DAG is of finite size, each state in Σ has a finite number of state variables, and each state variable has a finite domain) and as a result, the encoded CSP also contains a finite number of variables with a finite number of domains. Since the CSP solver works through piecemeal reduction of the variable domains, given a correct solving algorithm (Kondrak and van Beek, 1997), eventually, a solution will be reached or the solving will fail in case the CSP is unsatisfiable. The result is also sound since we encode all aspects of our planning problem as constraints as described in the previous section (i.e., every possible variable and its type and all possible actions), and the constraint solver will ensure every constraint on the CSP is met. Therefore, any solution to the CSP is a solution to the planning problem.

Soundness and completeness of the planning problem itself are based on both the construction of the transition system Σ as well as the representation of the distributed processing pipeline Σ ≃ . This is again focused on the Spark DAG but can be modified for other distributed processing frameworks. Since our planning system is based on the state-variable representation as described by Ghallab et al. (2004), we know that this approach can yield correct results. Therefore, we will only discuss the correctness of the Spark DAG translation, i.e., Σ ≃ . We do this based on our definitions in Section 4.2, of which the most important is the final definition of the transition function in Eq. 12.

Since Spark uses a (directed) acyclic graph, the transition system must also be acyclic. We represent the DAG as γ ≃ : R × T → R . Since γ ≃ is created from the Spark DAG, these properties (such as it being acyclic) also hold for γ ≃ . Next, recall the rdd relation between γ ≃ and γ, defined as ( ∀ s n ∈ S ) ( ∀ a ∈ A ) ( ∀ s m ∈ S ) ( ∃ t i ∈ T ) : s m ∈ γ ( s n , a ) ⇒ r d d ( s m ) ∈ γ ≃ ( r d d ( s n ) , t i ) . If there If there isa cycle between states in γ, there must then also be a cycle between RDD scopes in γ ≃ . This is not possible; therefore, γ must also be acyclic.

Apart from relating γ with γ ≃ as described above, the rdd relation also ensures that topology constraints that specify the behavior of the pipeline are met. This is achieved through the encoding of effects(rdd(s _n )) and precond(rdd(s _m )) as constraints that must be satisfied, given (∃t ∈ T) such that r d d ( s m ) ∈ γ ≃ ( r d d ( s n ) , t ) . If there is no transition t, that means there is no s _n ∈ γ that results in s _m . In this case, s _m is an initial state and as such, it is already described by the initial conditions S ₀.

Further complications arise from the possibility of multiple RDDs to be joined, for which our solution is described in Section 4.2.3. We support two strategies that reconcile the state variables between the two branches. For the “require equality” strategy, all state variables are related through an equality constraint. If a state variable in one branch is different from the other, this constraint would be violated and the configuration would not be provided as a solution. For the “accept either” strategy, the user explicitly states that such conflicting state variables are still acceptable in a solution for the planning problem. Nevertheless, the final Spark configuration is still limited in that an RDD can only be assigned a single type. As a result, a solution where multiple possible types are assigned to a single state could not be applied to the pipeline. As mentioned in the description of the “accept either” strategy, we slightly restrict the planning model by specifying that the Type variables should always be equal between branches regardless of the chosen strategy.

Another complication of join operations is that they are the only operations supported by our planning system that do not accept a user function (in which an action could be applied). A configuration would therefore be invalid if an action is assigned to such an operation. Through the implicit transitions mentioned in Section 4.2.3, we enforce a constraint that ensures a no-op action is applied in those transitions; therefore, no invalid action can be assigned.

Finally, through the vp relation, we ensure that a module is of the same category as the variation point in an operation. Furthermore, we ensure that the module is compatible with the operation. If the user attempts to use a variation point in an operation of a type that is not compatible, the chosen module must be of the same incompatible type. As this is prevented in our action encoding, no plan can be generated.

The plan generation time experiments explore three different properties of a pipeline to determine their impact on the time it takes to generate a configuration.

The first property we examine is pipeline length. We measure how long it takes to generate a plan for a pipeline with n sequential operations, as shown in Figure 5A. We first measure the planning time for only one operation, followed by the planning time for two operations, up until n = 16. For each step, only a single action is applicable. This is achieved by generating an action specifically for each step in the pipeline, as shown in Listing 1.

Listing 1

Overview of the generator for variable length pipelines.

The second property being examined is the number of alternatives per step, as shown in Figure 5B. We generate a pipeline with four steps, but instead of having exactly one action for each Spark operation, we generate multiple actions with the same preconditions and effects on the Step variable, as shown in Listing 1 with ALTERNATIVES > 1.

The last property under examination is the number of joins in a pipeline. We again allow only one possible module per step, but instead of applying a single operation to the initial RDD, we apply two separate map operations on the same RDD, followed by a union. The resulting pipeline topology is shown in Figure 5C. We determine the time it takes to generate such pipelines for both join strategies mentioned in Section 4.2.3: “require equality” and “accept either.” We otherwise generate the pipeline in the same way as Listing 1.

The results of all measurements in this experiment are shown in Figure 6.

Let us first discuss the results for the varying number of operations and number of alternative actions per operation. For both measurements, the growth is exponential, although the planning generation time for the variable pipeline length grows considerably faster.

We attribute this higher growth rate to multiple factors. First, the domain for the Step variable increases, as the final goal value is increased. Next, the system also needs to test the applicability of more possible actions, since we generate one action for each step in the pipeline. For the steps themselves, we need to encode more constraints (described in Section 4), and we also have more steps where we try to apply the no-op actions. The measurements for the varying number of alternatives are also affected by the increase of possible actions to be tested; however, this introduces much fewer constraints to the CSP.

The results for the experiments using a variable number of joins show a higher growth rate when using the “accept either” strategy compared to the “all equal” strategy. The planning time when using the “all equal” strategy still grows faster than that of the varying pipeline size and varying number of alternatives, since for each join, two actions need to be applied, one for each branch. Using the “accept either” strategy results in even bigger planning time growth, since the system must accept either value as a result of one join. This uncertainty propagates through every step in the pipeline, meaning that we cannot reduce the search space as quickly as for the other experiments.

Next, we compare the adaptive framework with the spark-dynamic framework (Lazovik et al., 2017) and with regular Spark implementations. These experiments are based on three implemented scenarios inspired by real projects, each built using commonly used operations and increasing in complexity.