REVIEW article

Front. Comput. Sci., 15 October 2026

Sec. Software

Volume 8 - 2026 | https://doi.org/10.3389/fcomp.2026.1946723

Constructing t-way test suites: a systematic survey and qualitative synthesis of the One-Parameter-at-a-Time approach

  • 1. Department of Computing, Universiti Teknologi PETRONAS, Seri Iskandar, Perak, Malaysia

  • 2. Department of Computer Engineering Technology, Federal Polytechnic Daura, Daura, Katsina, Nigeria

  • 3. Department of Electrical and Electronic Engineering, Federal Polytechnic Damaturu, Damaturu, Yobe, Nigeria

Abstract

Parameter interactions are a major source of software failures, yet the exponential growth of possible configurations makes exhaustive testing impractical. Combinatorial Interaction Testing (CIT) is a highly effective sampling technique for identifying software defects caused by parameter interactions. Over the past two decades, the One-Parameter-at-a-Time (OPAT) constructions paradigm has emerged as a cornerstone of CIT test suite generation. Although numerous OPAT strategies have been proposed to optimize test suite size and execution speed, the literature remains fragmented, and direct side-by-side quantitative benchmarking has been impeded by a lack of open-source tooling and standardized platforms. To address this, this study presents a systematic survey and qualitative synthesis of 40 primary OPAT studies published between 2000 and 2026, retrieved from four major digital libraries. Rather than attempting a direct quantitative benchmark comparison of execution times, which is shown to be mathematically invalid due to highly heterogeneous self-reported hardware environments and the closed-source nature of most academic tools, we establish a unified taxonomy that classifies OPAT research into five major categories: deterministic refinements, scaling and distribution strategies, advanced feature-enhanced strategies, metaheuristic and hybrid strategies, and specialized OPAT adaptations. Our analysis synthesizes qualitative trends, structural characteristics, parameter-order growth mechanisms, and feature-integration gaps across the literature. We conclude with a critical assessment of the tooling and reproducibility challenges in the field, outlining vital future pathways, including diversified metaheuristic and hyper-heuristic techniques, deterministic parallel algorithms, and AI-driven adaptive test generation, to transition OPAT research into industrial-grade, open-source software engineering pipelines.

1 Introduction

Software testing plays a vital role in ensuring software quality, reliability, and safety by identifying defects before deployment (Ebert et al., 2022; Lopuha et al., 2023). As modern software systems continue to grow in size, functionality, and configurability, the number of possible input combinations grows exponentially, making exhaustive testing impractical (Fadhil et al., 2022). This challenge is particularly evident in configurable systems, where failures often stem not from individual input values but from unexpected interactions among multiple parameters. Consequently, efficient testing strategies that detect interaction faults while minimizing testing effort have become increasingly important (Villoth et al., 2025; Zivkovic and Zivkovic, 2026).

Traditional software testing techniques, including static and dynamic testing, provide valuable mechanisms for detecting a wide range of software defects (Kathiresan, 2024; Zhang et al., 2024). Static testing examines software artifacts, such as requirements, design documents, and source code, without executing the program, whereas dynamic testing evaluates software behavior during execution using white-box, black-box, or gray-box techniques. Although these approaches are effective in many testing scenarios, they are generally insufficient for systematically identifying faults arising from interactions among multiple input parameters (Niu et al., 2020a,b). As software configurations become more complex, the number of possible parameter combinations increases combinatorially, making comprehensive validation increasingly difficult.

Combinatorial Interaction Testing (CIT) addresses this challenge by systematically selecting a relatively small yet highly effective set of test cases that cover interactions among input parameters (Mercan et al., 2020; Ramli et al., 2024). Rather than executing every possible configuration, CIT exploits the observation that most software failures are triggered by interactions involving only a small number of parameters. Pairwise testing (t = 2) is the most widely adopted form of CIT because it covers every pair of parameter values at least once while significantly reducing the number of required test cases. For systems in which failures depend on higher-order interactions, CIT can be extended to arbitrary interaction strengths (t-way testing), providing a practical balance between testing cost and fault-detection effectiveness.

Existing t-way combinatorial testing approaches can be classified from two complementary perspectives. The first classifies strategies according to interaction strength, including uniform-strength, variable-strength, and input-output-based testing (Aminu and Hashim, 2024). The second classifies test generation strategies according to their construction mechanisms, namely One-Test-at-a-Time (OTAT) and One-Parameter-at-a-Time (OPAT) (Muazu et al. 2023a,b). OTAT strategies construct complete test cases individually by satisfying uncovered interaction requirements, whereas OPAT strategies incrementally build test suites by introducing one parameter at a time through two complementary phases: horizontal growth, which extends existing test cases to maximize interaction coverage, and vertical growth, which generates additional test cases to cover any remaining uncovered interactions. This incremental construction enables OPAT to reuse existing test cases effectively while maintaining good scalability as new parameters are introduced.

Since the introduction of the In-Parameter-Order (IPO) algorithm (Tai and Lei, 2002) and its generalized variant IPOG (Lei et al., 2007), the OPAT approach has evolved considerably. Numerous extensions have been proposed to improve scalability, reduce test suite size, support higher interaction strengths, enable test case seeding and constraint handling, exploit parallel and distributed computing environments, and integrate metaheuristic optimization techniques. These developments demonstrate the OPAT framework's adaptability to increasingly complex software testing problems and diverse application domains.

Despite these advances, a comprehensive synthesis of the OPAT literature remains absent from current review studies. Existing studies typically focus on individual algorithms or specific optimization techniques, while broader review articles primarily examine combinatorial testing from search-based, interaction-based, or metaheuristic perspectives. To the best of our knowledge, no previous review has provided a comprehensive synthesis dedicated exclusively to OPAT-based combinatorial interaction testing. Consequently, there remains a lack of a unified understanding of the evolution of OPAT strategies, their underlying optimization mechanisms, practical capabilities, research trends, and remaining challenges.

To address this gap, this paper presents a comprehensive review of OPAT-based combinatorial interaction testing strategies published between 2000 and 2026. The selected studies are systematically analyzed and organized into five major categories: deterministic refinements, scaling and distribution strategies, feature-enhanced approaches, metaheuristic and hybrid strategies, and specialized OPAT adaptations. Based on this classification, the review examines the strengths, limitations, optimization characteristics, and practical applicability of existing approaches while identifying emerging research directions and open challenges.

The major contributions of this review are summarized as follows:

  • i. It presents what, to the best of our knowledge, is the first systematic literature review dedicated to One-Parameter-at-a-Time (OPAT) strategies for combinatorial interaction testing, a claim supported by our systematic search of 1,300 publications, which did not identify any other review with this specific focus.

  • ii. It proposes a unified taxonomy, illustrated in Figure 1, that classifies existing OPAT strategies into five major research categories based on their underlying design objectives and optimization mechanisms.

Figure 1

  • iii. It provides a comparative analysis of OPAT strategies with respect to scalability, computational efficiency, test suite optimality, feature support, and practical applicability.

  • iv. It identifies current research directions, including unified optimization frameworks, intelligent parameter adaptation, broader integration of metaheuristics, deterministic parallelization, and large-scale industrial validation.

The remainder of this paper is organized as follows. Section 2 introduces the OPAT approach and reviews related studies. Section 3 describes the review methodology. Section 4 presents and analyzes the findings. Section 5 discusses the major observations, future research directions, and threats to validity. Finally, Section 6 concludes the paper.

2 Study context and related work

This section provides the technical background necessary to understand the One-Parameter-at-a-Time (OPAT) approach and its role in combinatorial interaction testing (CIT). It summarizes the fundamental principles of OPAT and its incremental test-construction process, and reviews existing survey and review studies to establish the research gap that motivates this work.

2.1 One-Parameter-at-a-Time (OPAT) approach

The One-Parameter-at-a-Time (OPAT) approach is one of the widely adopted construction strategies for combinatorial interaction testing. Unlike one-test-at-a-time (OTAT) strategies, which generate complete test cases independently. OPAT incrementally constructs a test suite by introducing one parameter at each generation stage. This incremental strategy enables the reuse of previously generated test cases while maintaining comprehensive interaction coverage, thereby improving computational efficiency and scalability.

The OPAT approach originated from the In-Parameter-Order (IPO) algorithm and was subsequently generalized through the In-Parameter-Order General (IPOG) algorithm to support arbitrary interaction strengths. Since then, numerous extensions have been proposed to improve scalability, reduce test suite size, support constraint handling, facilitate test case seeding, exploit parallel computing environments, and incorporate optimization techniques.

Test generation in OPAT is performed through complementary mechanisms: horizontal growth and vertical growth. Horizontal growth extends existing test cases by assigning values to a newly introduced parameter. The selected value maximizes the number of uncovered interactions that each existing test case covers, thereby reducing the need for additional test cases. Vertical growth is subsequently applied to cover any remaining interactions after horizontal expansion. Additional test cases are generated only when necessary, ensuring complete interaction coverage while minimizing redundancy.

The combination of these two mechanisms enables OPAT to balance interaction coverage, computational efficiency, and test suite compactness. Consequently, OPAT has become the foundation for many subsequent combinatorial testing algorithms and remains one of the most influential paradigms in CIT research.

2.2 Related work

Several review studies have investigated different aspects of combinatorial interaction testing; however, none of these existing reviews has provided a dedicated synthesis of the One-Parameter-at-a-Time (OPAT) approach. To situate our study, we analyze the specific focus and boundaries of these prior works, as summarized in Table 1.

Table 1

S/NoReview studyTypeFocus
1Abdul Manan et al. (2021)SLRTest case prioritization and sequencing metrics
2Muazu et al. (2022)ReviewSurvey of nature-inspired metaheuristic algorithms selection
3Muazu et al. (2024c)Scoping reviewBroad categorization of CIT perspective features
4Yahaya et al. (2025)SLRExploration and exploitation mechanism in pairwise test case generation
5This work (2026)SLROne-Parameter-at-a-Time (OPAT) approach

Comparative analysis of existing CIT reviews against our work.

Abdul Manan et al. (2021) conducted a systematic literature review on test case prioritization in combinatorial testing. However, their work was strictly bound by post-generation sequence optimization techniques to accelerate early fault detection and did not examine the actual generation algorithm or the horizontal or vertical parameter growth mechanisms of OPAT.

(Muazu et al. 2022) reviewed nature-inspired metaheuristic algorithms for combinatorial testing, identifying more than one hundred optimization algorithms and discussing their potential applications to t-way testing. Although their review provides valuable insights into metaheuristic search engines, it analyzes them as black-box optimization frameworks without categorizing them by construction mechanisms (such as OPAT vs. OTAT), thereby completely omitting the foundational deterministic, distributed, and parallelized parameter-order growth strategies (such as IPOG, IPOG-D, or MC-MIPOG).

More recently, Muazu et al. (2024c) presented a scoping review of combinatorial interaction testing strategies, categorizing existing approaches by methodological characteristics. While their study offers a broad overview of the field, the OPAT approach represents only a single, high-level category within a much wider class of occupational frameworks and lacks a detailed evaluation of its horizontal and vertical growth heuristics, parameter ordering, and trade-offs.

Yahaya et al. (2025) subsequently performed a systematic literature review on exploration and exploitation mechanisms in pairwise test generation. Their analysis concentrated on metaheuristic optimization techniques used in pairwise testing. However, their review was restricted exclusively to pairwise testing (t = 2) and did not investigate the broader, multi-way (t ≄ 3) capabilities, deterministic models, parallel execution frameworks, or specialized adaptations that characterize the wider OPAT framework. provided an SLR focusing on metaheuristic optimization mechanisms specifically for pairwise testing.

3 Methodology

This section outlines the systematic review process adopted to investigate the evolution of combinatorial interaction testing (CIT) strategies within the One-Parameter-at-a-Time (OPAT) approach. The methodology follows a structured and transparent set of guidelines established in the literature (Farina et al., 2022; Hemmat et al., 2025). The review process is organized into four main stages: (i) review design and scoping, (ii) research question formulation, (iii) search strategy and study selection, and (iv) data extraction and synthesis. This structured approach ensures methodological rigor, minimizes selection bias, and supports a comprehensive and reproducible analysis.

3.1 Review design and scope

This review aims to systematically identify, classify, and analyze OPAT-based strategies for combinatorial test generation, with particular emphasis on their optimization mechanisms, scalability, and practical applicability. The scope of this review is defined using the PICOC framework (see Table 2), which guides the formulation of the search strategy.

Table 2

ElementDescription
PopulationStudies focusing on One-Parameter-at-a-Time (OPAT)-based Combinatorial Interaction Testing (CIT) strategies
InterventionIdentification and analysis of OPAT techniques, optimization strategies, and supporting mechanisms, including deterministic refinements, scalability improvements, constraint handling, metaheuristic integration, and specialized adaptations
ComparisonDual-level comparative analysis: (a) evaluating the performance and functional advancements of OPAT strategies relative to each other to track methodological evolution, and (b) synthesizing evidence from the original studies that benchmark OPAT variants against established non-OPAT baselines
OutcomesDevelopment of a unified taxonomy of OPAT-based CIT strategies, identification of major research trends, analysis of strengths and limitations, evaluation of trade-offs among approaches, identification of existing research gaps and future research directions
ContextA structured review of existing academic contributions on OPAT-based CIT, covering algorithmic developments, empirical evaluations, and practical applications to understand the evolution and future advancement of OPAT techniques

PICOC framework for defining the scope and objectives of the review.

Preliminary exploration of the literature indicated a diverse range of OPAT-based strategies, including deterministic refinements, parallel and distributed implementations, feature-enhanced strategies (e.g., constraint handling and seeding), metaheuristic and hybrid methods, and specialized adaptations. This diversity necessitated a structured synthesis to consolidate fragmented findings and provide a unified understanding of the field.

Accordingly, the selected studies were systematically categorized into these five thematic groups, forming the basis for the analytical framework used in the Results section. This categorization enables a comparative evaluation of strategies across key dimensions, including scalability, test-suite optimality, computational cost, and feature integration.

3.2 Research questions

The review was designed to investigate the evolution of One-Parameter-at-a-Time (OPAT) strategies in combinatorial interaction testing. The following research questions guided the study.

  • i. What OPAT-based strategies have been proposed for combinatorial interaction testing?

  • ii. What optimization techniques and enhancement mechanisms have been incorporated into OPAT strategies?

  • iii. What are the strengths, limitations, and application characteristics of existing OPAT approaches?

  • iv. What research gaps and future directions remain for OPAT-based combinatorial interaction testing?

These research questions collectively provide a comprehensive understanding of the development, capabilities, and future prospects of OPAT strategies.

3.3 Search strategy and study selection

A comprehensive literature search was conducted to identify studies relevant to OPAT-based combinatorial interaction testing. The search covered publications from 2000 to 2026, corresponding to the period from the introduction of the IPO algorithm to the most recent developments in OPAT research. Four major academic databases were selected because of their extensive coverage of software engineering literature: ACM Digital Library, IEEE Xplore, ScienceDirect, and Scopus (see Table 3).

Table 3

No.LibrariesSearch string
1ACM Digital Library[All: ā€œcombinatorial interaction testingā€] OR [All: ā€œcombinatorial testingā€] OR [All: ā€œpairwise test case generationā€] OR [All: ā€œpairwise testingā€] OR [All: ā€œtest case generationā€] AND [E-Publication Date: (01/01/2000 TO 12/31/2026)]
2IEEE Xplore(ā€œCombinatorial Interaction Testingā€ OR ā€œCombinatorial Testingā€ OR ā€œPairwise Test Case Generationā€ OR ā€œPairwise Testingā€ OR ā€œTest Case Generationā€)
3Science Direct(ā€œCombinatorial Interaction Testingā€ OR ā€œCombinatorial Testingā€ OR ā€œPairwise Test Case Generationā€ OR ā€œPairwise Testingā€ OR ā€œTest Case Generationā€)
4Scopus(((ā€œCombinatorial Interaction Testingā€) OR (ā€œCombinatorial Testingā€) OR (ā€œPairwise Test Case Generationā€) OR (ā€œPairwise Testing) OR (ā€Test Case Generationā€œ)) AND PUBYEAR > 2001 AND PUBYEAR < 2026 AND (LIMIT-TO (SUBJAREA, ā€ENGIā€œ) OR LIMIT-TO (SUBJAREA, ā€COMPā€œ)) AND (LIMIT-TO (DOCTYPE, ā€arā€œ) OR LIMIT-TO (DOCTYPE, ā€cpā€œ)) AND (LIMIT-TO (LANGUAGE, ā€Englishā€œ)) AND (LIMIT-TO (EXACTKEYWORD, ā€Software Testingā€œ) OR LIMIT-TO (EXACTKEYWORD, ā€Testingā€œ) OR LIMIT-TO (EXACTKEYWORD, ā€Test Case Generationā€œ) OR LIMIT-TO (EXACTKEYWORD, ā€Software Engineeringā€œ))

Libraries and their related search string.

The search strategy was developed using the PICOC framework and involved three steps:

  • i. Identification of core concepts for combinatorial interaction testing and OPAT strategies.

  • ii. Expansion of keywords using synonyms, related terms, and controlled vocabulary.

  • iii. Construction of search strings using Boolean operators (AND, OR, NOT), quotation marks, and parentheses.

The search strings were applied consistently across all selected databases. The inclusion and exclusion criteria (see Table 4) were defined to ensure relevance by focusing on studies that propose, improve, or evaluate OPAT-based combinatorial testing strategies.

Table 4

No.CharacteristicsInclusionExclusion
1Year2000–2026Before or after the specified period
2Document typeArticle and Conference paperOther related document type
3Subject areaComputer Science and EngineeringNot related to the mentioned subject
4LibrariesACM Digital Library, IEEE Xplore, Science Direct, and ScopusNot mentioned library
5LanguageEnglishOther languages

Inclusion and exclusion.

The initial search yielded 1,300 studies. After removing duplicates (N = 320) using Mendeley and Microsoft Excel, the remaining studies were screened based on titles and abstracts, followed by full-text assessment (see Figure 2). To minimize selection bias and ensure a highly reliable selection process, a multi-stage screening and selection protocol was executed independently by multiple researchers.

Figure 2

During the Title and Abstract Screening phase, two reviewers (Yahaya and Mamman) independently evaluated the titles and abstracts of the N = 980 unique records against the eligibility criteria. To quantify the level of agreement between the screeners before any consensus discussions, Cohen's Kappa (k) coefficient was calculated. The inter-reviewer agreement for this initial screening phase was k = 0.83, representing substantial methodological agreement. Disagreements at this stage were resolved through collaborative discussion, resulting in N = 760 records progressing to the next stage.

During the full-text eligibility phase, the same reviewers independently read and assessed the full texts of N = 675, while 85 studies could not be retrieved. The inter-reviewer agreement for the full-text selection phase was k = 0.89, indicating almost perfect agreement. Any persistent discrepancies or borderline cases regarding study inclusion were resolved through a structured consensus-building protocol; if a consensus could not be reached, a third senior reviewer (Abdullahi) acted as an arbitrator to make the final decision. This rigorous process ultimately yielded the final set of N = 40 primary studies included in the systematic review.

During the search, it was observed that the earliest relevant study was published in 2002, while the most recent appeared in 2026. The year-wise distribution of the selected studies is illustrated in Figure 3, showing that the highest number of publications occurred in 2008 (five studies), followed by 2013, 2014, and 2018 (four studies each), and 2020 and 2024 (three studies each).

Figure 3

3.4 Quality assessment of primary studies

To ensure the methodological rigor and reliability of the synthesized findings, a formal Quality Assessment (QA) procedure was conducted on the final set of N = 40 included primary studies. The QA checklist was designed following established guidelines for systematic literature reviews in software engineering (Kitchenham and Charters, 2007) and tailored to the technical characteristics of Combinatorial Interaction Testing (CIT).

Each primary study was evaluated against four criteria:

  • 1) QA1: Does the study clearly describe the OPAT strategy's algorithm, horizontal/vertical growth mechanisms, or optimization search logic?

  • 2) QA2: Are the experimental settings, benchmark configurations (such as TCAS or uniform/mixed covering arrays), and hardware environments explicitly defined?

  • 3) QA3: Are the primary outcomes (test suite sizes, execution/generation time, or interaction coverage) quantitatively measured and reported?

  • 4) QA4: Does the study compare the proposed strategy against established baselines (e.g., IPOG, Jenny, or other OTAT/algebraic approaches) using statistical validation or diverse configurations?

Each study was scored quantitatively for each criterion as follows: yes (1.0 point) if the criterion was fully met; Partial (0.5 points) if the criterion was only partially met or lacked minor details; and No (0.0 points) if the criterion was not addressed. The total QA score for any individual study ranged from 0.0 (lowest quality) to 4.0 (highest quality).

To prevent selection bias, low-scoring studies were not excluded from the review, provided they satisfied the strict inclusion and exclusion criteria outlined in Table 3. Rather, the QA procedures were used to evaluate the strength of the evidence and to ensure that major performance claims (such as near-linear speedup or superior scalability) were supported by robust experimental designs.

The quality profiling revealed high methodological quality across the selected literature (see Appendix Table A1), with 82.5% (33 out of 40) of the primary studies scoring ≄3.0 out of 4.0. The remaining studies scored between 2.0 and 2.5, primarily due to partial metric reporting (missing execution times or hardware specifications) or a lack of statistical significance testing. The high overall quality scores confirm that the synthesized performance claims presented in Section 4 are derived from scientifically credible sources.

3.5 Data extraction and synthesis

A standard data extraction form was designed to ensure consistent collection of information from each selected study. The extracted information included bibliographic details, publication year, proposed OPAT strategy, optimization technique, interaction strength, constraint-handling capability, reported strengths and limitations, and application domain.

The extracted data were synthesized using both qualitative and quantitative analyses. The Qualitative synthesis was employed to classify OPAT strategies according to their methodological characteristics and optimization mechanisms. The quantitative synthesis summarized publication trends, geographical distribution, publication venues, optimization techniques, interaction strength, and application domains.

The synthesized findings were subsequently organized into comparative tables and thematic categories to identify research trends, technological evolution, existing limitations, and future research opportunities.

3.6 Analytical framework

The five-category taxonomy presented in this review was developed inductively rather than established a priori. Following the initial screening of the N = 40 included studies, the authors performed an interactive coding process to identify recurring themes, technical mechanisms, and specific research objectives. The construction of the taxonomy followed a three-phase qualitative synthesis:

  • i. Iterative Coding: studies were first grouped by their foundational algorithmic logic, such as those that directly refined the original IPO/IPOG mechanisms (e.g., VIPO, IIPO).

  • ii. Thematic Clustering: new categories emerged as specific research trends were identified, including a distinct cluster of strategies designed for parallel or grid environments (e.g., MC-MIPOG, TS_OP) and those integrating global optimization metaheuristics (e.g., eHS, OPAT-HS).

  • iii. Refinement: the ā€œSpecialized OPATā€ category was inductively defined to capture strategies that adapted the approach to niche domains like sequence testing or input-output relations (e.g., DEO, ReqOrder).

To ensure the reliability of the classification, disagreements were resolved through a collaborative, consensus-based protocol. As detailed in the ā€œAuthor Contributionā€ section, multiple researchers (Yahaya, Mamman, and Abdullahi) independently reviewed the technical features of each strategy. In instances of overlapping characteristics, the research team conducted peer-review sessions to determine the ā€œprimary contributionā€ of the study (e.g., whether a strategy was primarily a scaling refinement or an advanced feature support) until reaching 100% consensus. This rigorous internal cross-checking was adopted to minimize subjective bias and ensure methodological rigor.

3.7 Criteria for practical and industrial evaluation

The evaluation of Industrial Relevance and Practical Applicability within this review is based on a structured four-tier evidence framework. Rather than inferring applicability from algorithmic design alone, strategies are rated according to documented tool maturity, industrial adoption, and validation against real-world systems:

  • i. Very High: strategies integrated into widely distributed research or commercial tools (e.g., ACTS/FireEye) that have documented adoption by a large number of organizations (e.g., 1,200 plus organizations for ACTS). These strategies must also be validated against large-scale, heterogeneous real-life systems such as Apache, Bugzilla, and GCC.

  • ii. High: strategies that are supported by an available standalone tool (e.g., VIPO, PMC, or Jenny) and have been benchmarked against established real-world modules, most notably the FAA's Traffic Collision Avoidance System (TCAS). These ratings also require the presence of critical industrial features such as constraint handling and test case seeding.

  • iii. Moderate: strategies validated primarily through standard academic benchmarks (e.g., uniform covering arrays) but which demonstrate a clear path to practical use by addressing real-world problems like sequence-dependent events (e.g., DEO) or unit testing in specific languages like C/C++.

  • iv. Low: conceptual strategies or foundational algorithms that lack available tool support or validation beyond small-scale synthetic configurations.

4 Result analysis

The evolution of combinatorial interaction testing (CIT) strategies within the one-parameter-at-a-time (OPAT) approach reflects a sustained effort to balance test-suite optimality and computational scalability. To provide a comprehensive evaluation, the subsequent synthesis analyzes these strategies across two dimensions. First, it examines how individual OPAT variants refine the foundational IPO/IPOG framework. Second, it highlights the comparative performance of these variants as evidenced by the experimental results reported in the selected primary studies.

The In-Parameter-Order (IPO) strategy forms the foundation for subsequent OPAT developments, which establish a structured, incremental mechanism for generating pairwise test suites through horizontal and vertical growth. Its extension, In-Parameter-Order General (IPOG), generalized this strategy to support t-way interactions. While both approaches offer significant advantages (i.e., incremental extensibility, reuse of existing test configurations, and relatively lower computational complexity compared to one-test-at-a-time (OTAT) strategies), they suffer from scalability limitations due to the exponential growth of interaction combinations at higher strengths (Lei et al., 2007; Tai and Lei, 2002).

To address these limitations, this review demonstrates a diversification of OPAT-based strategies into five major categories: (i) deterministic refinements, (ii) scaling and distribution strategies, (iii) feature-enhanced approaches, (iv) metaheuristic and hybrid strategies, and (v) specialized OPAT adaptations. Each category reflects a distinct research direction aimed at mitigating specific shortcomings of the original IPO/IPOG strategies.

4.1 Performance objectives and empirical evidence in OPAT-based CIT

A systematic assessment of OPAT strategies requires consideration of multiple performance dimensions because no single metric adequately captures the effectiveness of a CIT strategy. Accordingly, the review examined five principal dimensions: test-suite size, generation time, memory consumption, interaction strength, and fault-detection capability. These dimensions are interdependent, and improvements in one may involve trade-offs in another. Importantly, the qualitative results reported in this section are study-specific and are presented according to the experimental conditions of the original studies. They are therefore not interpreted as a controlled cross-study ranking.

Test-suite size is a primary objective in CIT because a smaller suite reduces test execution costs while maintaining the required interaction coverage. However, minimizing a covering array is computationally difficult, and approaches that invest additional search effort in minimization may incur greater generation costs. The quantitative results reported in the primary studies illustrate this trade-off. For example, Lei et al. (2008) reported comparisons between IPOG and IPOG-D (see Table 5) as domain size and interaction strength increased. The results show that IPOG-D substantially reduced generation time relative to IPOG while producing larger test suites. These results demonstrate the trade-off between computational efficiency and test-suite compactness within the experimental setting of the original study.

Table 5

No. of valuesIPOG (size)IPOG (Time (s))IPOG-D (Size)IPOG-D (Time (s))Size ratio (IPOG-D/IPOG)Time ratio (IPOG-D/IPOG)
21344.081881.401.400.34
31,12348.081,5522.391.380.05
44,531234.086,73912.311.490.05
515,095996.8922,19755.061.470.06
637,7483,273.4256,619203.131.490.06
781,8149,040.25124,186694.741.520.08

Original self-reported performance of IPOG and IPOG-D for 5-way testing.

Similarly, Younis and Zamli (2011) compared MIPOG and IPOG for five-way covering arrays as shown in Table 6. MIPOG generally produced smaller test suites, but its computational advantage was not consistent across all configurations; at larger domain sizes, the reported generation time exceeded that of IPOG. This provides further evidence that test-suite optimality and generation efficiency should be considered jointly rather than treated as interchangeable indicators of algorithmic superiority.

Table 6

ValuesMIPOG (size)MIPOG (Time (s))IPOG (Size)IPOG (Time (s))Size ratio (MIPOG/IPOG)
2920.172980.2210.939
36260.5937511.3610.834
42,9115.0463,0576.3750.952
58,16945.95010,11131.8750.879
623,557766.58025,247126.1560.933
749,5971,365.45054,567430.9370.909

Original self-reported performance of MIPOG and IPOG for 5-way testing.

The evidence from these representative studies, therefore, indicates that OPAT strategies exhibit a fundamental size-time trade-off. More aggressive optimization can reduce the number of generated tests, whereas strategies designed to reduce computational effort may accept a larger test suite. Nevertheless, because the reported results originate from different studies and experimental settings, these numerical values should be interpreted as evidence of within-study behavior rather than as a basis for universal ranking.

Memory requirements become increasingly important as the number of parameters and interactions increases. Explicit enumeration of all t-way combinations can produce substantial memory requirements because of the combinatorial growth of the interaction space. IPOG-D addresses this limitation through its construction mechanism, while distributed approaches such as G_MIPOG partition computation across multiple nodes.

Interaction strength determines the required degree of coverage, ranging from pairwise testing (t = 2) to higher-strength testing. Increasing t substantially increases the number of interactions to be covered and, consequently, affects test-suite size, generation time, and memory requirements. Most reviewed OPAT strategies were evaluated at relatively low interaction strengths, whereas MC-MIPOG and GAMIPOG demonstrate applicability at substantially higher strengths, reaching approximately t =12 and t = 15, respectively. High-strength scalability, therefore, remains an important differentiating characteristic of OPAT approaches.

4.2 Deterministic refinement of OPAT strategies

Deterministic refinements enhance the efficiency and effectiveness of the original IPO/IPOG algorithms by improving heuristics, optimizing search strategies, and making structural modifications. Rather than changing the fundamental OPAT paradigm, these approaches focus on improving the trade-offs between execution time, scalability, and test suite size.

A major characteristic of deterministic strategies is the balance between scalability and optimality. IPOG-D (Lei et al., 2008) extends IPOG by introducing a recursive doubling (D-construction) mechanism and explicit enumeration, thereby improving scalability for larger configurations. Experimental evidence showed that for 5-way testing of 20 4-valued parameters, IPOG-D required only 3% of the standard IPOG's execution time, at the cost of generating 51% more test cases. These tests were conducted on a laptop with a 1.6 GHz CPU and 1 GB of RAM.

Subsequent refinements shifted toward heuristic optimization. IPOF1 and IPOF2 (Wang et al., 2008) enhance the horizontal growth process through greedy heuristics, controlled randomization, and dynamic programming. These improvements achieve faster test generation while maintaining competitive test suite sizes. IPOF1 generally emphasizes test suite compactness, whereas IPOF2 prioritizes execution speed.

MIPOG (Younis et al., 2008a) further improves test suite minimization by applying enhanced greedy optimization during both the horizontal and vertical growth phases. Although this strategy consistently produces smaller test suites than IPOG, Jenny, and ITCH, particularly at t ≄ 4, the additional search effort increases computational cost. While VIPO (Cui et al., 2009) accelerates test generation by simplifying horizontal growth decisions, producing smaller test suites than IPO, TConfig, and AllPairs across 11 benchmark configurations (e.g., 1010 and 1510 uniform configurations) in a running environment with a 3.0 GHz CPU and 1 GB of RAM.

In contrast, IIPO (Prasad Mudarakola et al., 2014) introduces local optimization and incremental reuse mechanisms to reduce ā€œdon't careā€ values and improve adaptability when new parameters are introduced. While this improves flexibility and generation efficiency, its scalability gains remain moderate. Study 20 by Gao et al. (2014a) proposes general optimization strategies for the IPO algorithmic family by applying cyclic value selection during horizontal growth and by filling ā€œdon't careā€ positions with the least-frequent values, thereby significantly reducing generation runtime and test suite size as domain sizes increase.

Building on S20, Study 22 by Gao et al. (2015) refines the frequency-balancing mechanism and pairs it with a post-generation test-reordering technique, achieving an average speed-up of two times in test-suite generation time compared to standard IPO variants. Overall, deterministic OPAT strategies demonstrate a clear evolution from improving scalability to enhancing test suite optimality. As summarized in Table 7, each strategy represents a different compromise among computational cost, scalability, and test suite minimization, indicating that no single deterministic refinement can simultaneously optimize all performance objectives.

Table 7

StrategyOptimization strategyScalabilityTest suite size (optimality)Computational costImplications
IPOG-DRecursive D-construction (reduce interaction enumeration)HighLargerModerateImproves scalability for large configurations
IPOF1Greedy heuristic with expanded searchModerateSmallerLowBalances efficiency and compactness
IPOF2Dynamic programming with heuristic refinementModerateSlightly larger than IPOF1LowPrioritizes faster generation
MIPOGEnhanced greedy optimization during horizontal and vertical growthModerateSmallestHighMinimizes test suite size
VIPOSimplified horizontal growth heuristicReduces execution timeMay reduce test suite optimalityModerateProduces smaller test suites
IIPOLocal optimization and incremental reuseModerateModerateLowImproves adaptability to evolving parameters.
S20Offset/cyclic value search during horizontal growthHighSmallerHighImproves reduce run time and produce smaller test suite
S22Frequency balancing, prioritization, and reorderingHighsmallerHighShift benchmark paradigm to practical execution cost and rate of fault detection

Comparative characteristics of deterministic OPAT refinement strategies.

4.3 Scaling and distribution strategies

As combinatorial interaction testing problems scale to industrial-level configurations, the limitations of single-machine execution (particularly memory constraints and generation time) become increasingly prohibitive. This has led to the emergence of scaling and distribution strategies that adapt the OPAT approach to parallel and distributed computing environments.

A dominant trend in this category is the decomposition of the test generation process into independent or semi-independent computational units, enabling concurrent execution. For example, distributed approaches such as G_MIPOG (Younis et al., 2008b) partition both memory and computation across multiple nodes, effectively addressing the out-of-memory limitations inherent in high-dimensional interaction spaces. It demonstrated logarithmic speedups as coverage strength increased, specifically tested with 6-way testing of 7–10 5-valued parameters using a configuration of six computers.

Similarly, MC-MIPOG (Younis and Zamli, 2011) demonstrates the adaptation of OPAT strategies to shared-memory parallel architectures, leveraging multi-core processors to accelerate test generation. By reducing control and data dependencies within the algorithm, MC-MIPOG enables efficient thread-level parallelism, resulting in near-linear speedup as the number of parameters increases. This was evidence using a 2.4 GHz Core 2 Quad CPU on Linux CentOS, where speedup gains increased linearly for parameter counts from 5 to 16 at t = 4.

In contrast, TS_OP (Soh et al., 2014) introduces a more reasonable speedup in a distributed environment using MapReduce and Tuple Space. Testing on a network of five identical 2.2 GHz workstations showed that, while speedup was not perfectly linear due to network overhead, distributing the workload significantly reduced main memory utilization per machine.

As shown in Table 8, distributed approaches primarily address memory constraints, whereas multicore-based strategies focus on execution speed, highlighting distinct optimization priorities within scaling solutions.

Table 8

StrategyParallelization modelBenefitsLimitationApplication
G_MIPOGDistributed (Grid-based)Improves memory scalabilityRequires distributed infrastructureHigh-dimensional configurations
MC-MIPOGShared memory (Multicore)Accelerates test generationLimited by available processor coresModern multicore systems
TS_OPDistributed (Tuple Space/MapReduce)High scalability and workload distributionNon-deterministic outputsLarge distributed environments

Scalability and parallelization characteristics of OPAT strategies.

4.4 Advanced features and applicability

Beyond scalability, modern combinatorial testing increasingly demands support for real-world constraints, domain-specific requirements, and flexible interaction models. As a result, OPAT-based strategies have evolved to incorporate advanced features such as constraint handling, seeding, and mixed-strength interactions, enhancing their practical applicability.

A major focus in this category is the integration of constraint-handling mechanisms that ensure generated test cases exclude invalid or infeasible parameter combinations. Strategies such as ACTS (Yu et al., 2013a) exemplify this evolution by embedding IPOG and its variants within a flexible system that supports constraint specification, mixed-strength testing, and multiple user interfaces. This reflects a shift from standalone algorithms to comprehensive testing ecosystems.

More specialized approaches, such as IPOG-C (Yu et al., 2013b) and IPO-SAT (Gao et al., 2014b), focus on improving the efficiency of constraint evaluation. IPOG-C achieves this by minimizing redundant checks and selectively evaluating only relevant constraints, while IPO-SAT leverages SAT solvers to transform constraint handling into a satisfiability problem. These approaches demonstrate that constraint processing is a major computational bottleneck and optimizing it can yield substantial performance gains.

Recently, Shwetha (2017) proposed RIPOG-D, which integrates incremental construction with hybrid optimization techniques and variable-strength constraint support, providing a flexible solution for complex constrained systems. Study 23 by Wang (2017) proposed advanced constraint-handling mechanisms that use logical input-domain validation to prune invalid combinations of web form inputs. In contrast, Study 29 by Rao et al. (2019) introduces parameter-to-column mapping optimization, which maps system parameters onto pre-existing covering arrays (CAs) to minimize the number of discarded test rows when enforcing forbidden-tuple constraints.

In addition to constraints, seeding mechanisms have emerged as a critical feature for aligning test generation with domain knowledge. Strategies such as SCIPOG (Muazu et al., 2023b) and SCIPOG-VS (Muazu et al., 2023a) incorporate seeded values into the generation process, ensuring that critical or high-risk configurations are included in the test suite. When combined with constraint handling and support for variable interaction strengths, these approaches offer a highly flexible and expressive testing framework.

As summarized in Table 9, constraint-handling techniques and seeding mechanisms emerge as the dominant enhancements, with SAT-based and heuristic-based approaches offering different trade-offs between computational cost and flexibility.

Table 9

StrategyFeature supportStrengthsLimitationsIndustrial relevance
ACTSConstraint handling, mixed-strength testingComprehensive testing frameworkConstraint evaluation overheadHigh
IPOG-COptimized constraint evaluationReduces redundant constraint checkingAdditional processing complexityHigh
IPO-SATSAT-based constraint solvingEfficient validation of complex constraintsSolver overheadHigh
S23Logical parameter constraints and web testingInput domain validation and parameter reductionParallelized growth mechanismsHigh
RIPOG-DAlgebraic doubling construction with constraint handling for VSCAHandles mixed hardware and OS constraintsRedundant ā€œdon't careā€ paddingHigh
S29Parameter mapping optimization with constraint handlingPreserves base CA optimalityComputationally expensive for large data setsHigh
SCIPOGTest case seeding and constraintsIncorporates domain knowledge into test generationIncreased algorithmic complexityHigh
SCIPOG-VSSeeding, constraints, and variable strength testingSupports realistic industrial testing scenariosHigher implementation complexityVery High

Feature integration and constraint-handling capabilities in OPAT strategies.

4.5 Metaheuristic and hybrid strategies

Although deterministic OPAT strategies provide systematic and predictable test generation, they often struggle to achieve optimal solutions when applied to large or highly complex configuration spaces. To address these limitations, recent research has increasingly integrated metaheuristic optimization techniques to enhance exploration of the search space while preserving OPAT's structured construction process.

A prominent direction within this category is the adoption of Harmony Search Algorithm (HSA)-based strategies, such as OPAT-HS (Alsewari et al., 2018) and its derivatives. These approaches model test generation as an optimization problem in which candidate solutions are iteratively refined to maximize interaction coverage while minimizing redundancy. By embedding OPAT principles within a metaheuristic framework, these strategies combine systematic construction with stochastic optimization to achieve improved test suite compactness and generation efficiency.

Subsequent extensions, including HCATS (Muazu and Muazu, 2018), OPATHS (Aminu Muazu and Aminu Muazu, 2018) and PWiseHA (Muazu and Danjuma Maiwada, 2020), further refine this integration by enhancing support for mixed covering arrays and simplifying the optimization process. These approaches demonstrate that metaheuristic techniques can be effectively tailored to OPAT structures, particularly for low-to-moderate interaction strengths.

A hybrid strategy, such as GAMIPOG (Younis, 2020b), represents a more advanced integration that combines a genetic algorithm with the OTAT approach and deterministic OPAT components (e.g., MIPOG). This allows the algorithm to leverage both global search capabilities and structured incremental construction, resulting in improved scalability and support for higher interaction strengths (up to t = 15). It outperformed most existing strategies in terms of size (except Simulated Annealing for t = 3) when tested on a 2.2 GHz Core i7 with 8 GB of RAM.

Within the metaheuristic category, PHOSC (Muazu et al., 2024b), SCHOP (Muazu et al., 2024a), and eIPOG (Aminu and Hashim, 2024) represent a family of near-equivalent strategies that concurrently integrate seeding and constraint handling into the Harmony Search Algorithm (HSA) using the OPAT framework. Despite their shared functional objectives, they are distinguished by their depth and underlying (t = 2), utilizing high-probability local improvisation to achieve test-suite compactness. SCHOP provides the broadest interaction support (up to t = 6) and employs an optimized balance of intensification and diversification to maximize the test suite's validity in complex industrial configurations. Finally, eIPOG refines the IPOG mechanism by modifying its internal bit structures to enhance uniform interaction testing up to t = 5.

More recent developments further extend this direction toward adaptive and multi-stage optimization. The eHS strategy (Muazu and Hashim, 2025) enhances Harmony Search by incorporating OPAT-based initialization within the Harmony Memory and dynamically adjusting key parameters (HMCR and PAR) to balance exploration and exploitation, thereby improving coverage efficiency and reducing redundancy. It reported a 90.91% success rate in test suite size reduction across 33 configurations compared to IPOG. However, it achieved 0% success in execution time superiority over IPOG, as the metaheuristic search process is inherently slower than the foundational deterministic greedy algorithm.

Similarly, the Hybrid One-Parameter-at-a-Time Strategy (HOS) (Yahaya et al., 2026) combines multiple metaheuristics within the IPO framework by applying a GA for horizontal growth and Simulated Annealing (SA) for vertical growth. This dual-phase optimization leverages GA's global search capability and SA's local refinement strength. It demonstrated improved test-suite compactness by achieving optimal sizes for 61 of 71 benchmark configurations when tested on an i7-5500U (2.40 GHz) with 8 GB of RAM.

A quantitative analysis of the metaheuristic-based OPAT strategies identified in this review confirms a significant methodological imbalance within the field. The Harmony Search Algorithm (HSA) is the overwhelmingly dominant optimization family, appearing in 77.8% of the analyzed stochastic OPAT studies. In contrast, other established metaheuristics, such as Genetic Algorithms (GA) and Simulated Annealing (SA), appear almost exclusively as components of hybrid models (e.g., GAMIPOG and HOS), rather than as standalone OPAT frameworks.

Table 10 provides a comparative overview of the optimization techniques employed, illustrating how different metaheuristic integrations balance global search capability with the structured nature of OPAT-based construction.

Table 10

StrategyOptimization techniqueStrengthLimitation
OPAT-HSPioneer strategy to integrate Harmony Search into OPATSupport both Covering arrays (CA) and Mixed CA under uniform interaction strength (t ≤ 3).Performance decreases for high interaction strengths
HCATSHarmony searchCompact test suites and improve search efficiencyLimited scalability at higher interaction strengths
OPATHSEnhanced harmony searchSupports mixed covering arraysLimited to mixed covering arrays
PWiseHASimplified harmony searchStable and efficient optimizationLess effective for high-dimensional problems
GAMIPOGGenetic algorithm (GA) and MIPOGHigh scalability and global optimizationParameter tuning and computational overhead
PHOSCHSA-based OPAT for pairwise searchMulti-objective optimizationIncreased computational cost
SCHOPFeature-aware HSA with balanced intensification/diversificationImproves test validity and relevanceHigher algorithm complexity
eIPOGHSA-enhanced IPOG with bit-structure modificationBalances optimization and constraint satisfactionAdditional execution overhead
eHSAdaptive harmony searchDynamic exploration-exploitation balanceSensitive parameter tuning
HOSGenetic algorithm and Simulated Annealing (SA)Global and local optimization with compact test suitesHybrid coordination overhead

Optimization strategies in metaheuristic and hybrid OPAT strategies.

4.6 Specialized OPAT adaptations

Beyond improvements in scalability and feature integration, several OPAT strategies have been developed to address specialized testing requirements. Unlike the previous categories, these approaches target specific computational challenges, application domains, or heuristic enhancements while preserving the OPAT framework's incremental construction mechanism.

One research direction focuses on reducing computational overhead through simplified heuristic design. For example, whereas IPO-s (Calvagna and Gargantini, 2009) exploit symmetry among parameter interactions to improve coverage efficiency by introducing informed approximations that maintain acceptable interaction coverage while reducing processing time.

Other studies explore alternative structural representations to improve test generation. TBGCC (Klaib et al., 2010) employs a tree-based model to guide interaction coverage, while N-IPO (Gupta and Rana, 2013) utilizes a Fibonacci-based selection mechanism to reduce test suite size for large input domains. Although these strategies demonstrate improved efficiency in specific scenarios, their effectiveness depends on assumptions regarding parameter relationships or data distributions, which may limit their applicability to more general testing environments.

Specialized OPAT adaptations have extended combinatorial testing beyond conventional parameter interaction problems. DEO (Younis, 2020a) applies the OPAT paradigm to sequence covering arrays for event-driven systems, while PMC (Tran et al., 2022)automates test data generation directly from source code, reducing manual testing effort in software unit testing. In contrast, Study 2 by Maity and Nayak (2005) integrates advanced, specialized combinatorial models and algebraic structures such as mutually orthogonal Latin squares (MOLS) and Orthogonal Arrays (OA) for direct, instantaneous test generation. ReqOrder (Ziyuan et al., 2007) and ParaOrder (Wang et al., 2008) efficiently support input/output, variable, and overlapping interaction strengths.

Building on this, Study 18 by (Uthaileang et al. 2014) presents a practical pairwise testing methodology that combines Equivalent Partitioning (EP) with combinatorial test generation to manage systems with large input spaces. Study 26 by (Gupta et al. 2017) introduces N-IPO (Novel-In-Parameter-Order) algorithms that integrate boundary value analysis by leveraging the mathematical properties of Fibonacci-based value analysis.

These studies demonstrate the flexibility of OPAT in supporting dynamic and domain-specific testing applications. As summarized in Table 11, these strategies extend OPAT through heuristic simplification, alternative structural models, hybrid optimization, and domain-specific adaptations, highlighting the framework's flexibility beyond general-purpose combinatorial interaction testing.

Table 11

StrategySpecializationStrengthLimitationApplication domain
S2Algebraic combinatorial design using MOLS and OAsDirect computation and mathematical optimalityDomain restrictions and difficulty with complex parameterCompiler design and hardware module testing
IPO-sSymmetry-based optimizationImproves interaction coverage efficiencyLimited generalizationGeneral combinatorial testing
TBGCCTree-based interaction modelingProduces compact test suitesHigher structural complexityCost-aware combinatorial testing
N-IPOFibonacci-based parameter selectionEfficient for large input domainsAssumes representative parameter distributionLarge-scale configurations
DEOSequence covering array generationEfficient sequence-based generationSpecialized application scopeEvent-driven systems
PMCSource code-driven test generationAutomates test data generationLanguage and tool dependentC/C++ unit testing
ReqOrderInput-output relation combinatorial testingTargeted coverage and requirement mappingIt assumes factors are independent and free of forbidden tupleObject-oriented systems and class cluster interaction
ParaOrderGeneralized variable strength combinatorial testingIts support overlapping and non-disjoint interaction requirementSusceptible to local optimaComplex software architectures
S18Pairwise testing integrated with equivalent partitioning and boundary value analysisInput space reduction and high practical utilityCase-study focusWeb-based application testing
S26Novel In-Parameter-Order with Fibonacci samplingFibonacci sequence sampling/ordering and multi-layered trial run strategyLarger execution timePhased software testing

Heuristic and application-specific variants of OPAT strategies.

5 Discussion and future research directions

The systematic literature review (SLR) identified 40 primary studies and revealed a progressive evolution of One-Parameter-at-a-Time (OPAT) combinatorial interaction testing (CIT) from deterministic construction toward optimization-driven and application-oriented approaches. Based on the technical mechanisms and principal contributions reported in the selected studies, this study presents a five-category taxonomy: deterministic refinements; scaling and distribution; advanced feature-enhanced strategies; metaheuristic/hybrid strategies; and specialized adaptations. The taxonomy provides a structural basis for examining how OPAT strategies have addressed limitations in test suite size, generation efficiency, scalability, constraint handling, and application-specific requirements.

The temporal distribution of the reviewed studies reveals a historical peak in 2008, representing an important transition in the development of CIT. Prior to 2008, the state of the art was dominated by foundational pairwise (t = 2) testing strategies, such as the In-Parameter-Order (IPO) framework. The 2008 peak marks the transition toward multi-way (t ≄ 3) interaction strengths, driven by algorithms such as IPOG, IPOG-D, and MIPOG, which introduced structural optimizations to mitigate combinatorial explosion in high-dimensional parameter spaces.

Evaluating these strategies requires balancing test suite compactness against generation speed, as performance claims depend heavily on benchmark configurations and hardware constraints. For example, IPOG-D reduces execution time for high-strength configurations to 3% of standard IPOG but yields 51% larger test suites. Conversely, parallel and distributed approaches such as Multi-Core MIPOG (MC-MIPOG) and Grid-based MIPOG (G_MIPOG) achieve near-linear and algorithmic speedups, respectively, at higher interaction strengths (t ≄ 4), although multi-core variants suffer from initialization overhead at lower strengths (t = 2).

Metaheuristic approaches illustrate this trade-off particularly clearly. The enhanced Harmony Search algorithm (eHS), for example, demonstrates substantial improvements in test-suite minimization relative to IPOG under reported experimental conditions, but the iterative search process introduces additional overhead. These findings suggest that no single OPAT strategy simultaneously optimizes all performance objectives. Instead, strategy selection depends on whether the primary requirements are compactness, generation speed, computational scalability, or a combination of these objectives.

Metaheuristic search strategies represent a major advancement in test suite minimization, but their distribution across optimization families remains heavily unbalanced. Harmony Search is particularly dominant among the reviewed approaches, appearing in HCATS, OPAT-HS, OPATHS, PWiseHA, elPOG, PHOSC, SCHOP, and eHS, leaving alternative search frameworks largely unexplored. While metaheuristics consistently achieve superior minimization compared to greedy deterministic baselines, future work should incorporate underutilized families such as Particle Swarm Optimization, Tabu Search, and hyper-heuristics to improve convergence speeds.

The reviewed studies also demonstrate increasing attention to practical features that are not adequately addressed by basic OPAT construction. Constraint handling, variable strength testing, and test-case seeding enable generated suites to better represent realistic system configurations. These capabilities are particularly important because unconstrained covering arrays may contain combinations that are valid or meaningless in real-world applications.

However, the industrial applicability of existing OPAT strategies remains limited, as several approaches have been evaluated primarily using synthetic covering-array benchmarks rather than realistic industrial configurations. Consequently, improvements demonstrated on benchmark arrays do not necessarily establish effectiveness in operational testing environments, where constraints, heterogeneous parameter domains, evolving configurations, and domain-specific requirements may substantially affect algorithm performance. Future evaluations should therefore include realistic industrial configurations alongside standard benchmark suites.

A further finding of this review concerns the difficulty of establishing reliable cross-study performance comparisons. Although the primary studies frequently report test-suite size and generation time, these measurements are generally obtained under different experimental conditions, including hardware configuration, implementation environments, benchmark characteristics, interaction strengths, and parameter settings. Consequently, numerical results reported by different studies should not be interpreted as evidence of a universal performance ranking without controlling for these differences.

The reported results from representative studies illustrate this issue. For example, the quantitative results reported for IPOG and IPOG-D demonstrate a clear trade-off between test-suite size and generation time under the experimental environment used by the original authors. Similarly, the reported comparison between MIPOG and IPOG shows that MIPOG's relative advantage varies across parameter-domain sizes and performance measures. These findings provide useful evidence about the behavior of the strategies within their respective experimental settings, but they should not be interpreted as controlled comparisons across the broader OPAT literature.

The problem is further compounded by the limited availability of executable implementations, source code, and standardized experimental configurations. Without accessible implementations, it is difficult to independently reproduce the reported experiments or to execute competing strategies under identical conditions. Therefore, the limited reproducibility of existing OPAT studies is not merely a methodological limitation of this SLR; it constitutes a substantive research gap in the OPAT field. The current evidence base supports systematic identification of algorithmic mechanisms and qualitative characterization of performance trade-offs, but it does not consistently support statistically controlled cross-paper benchmarking.

These findings also change how comparative claims in OPAT research should be interpreted. Rather than treating reported test-suite sizes or execution times from separate studies as directly interchangeable measurements, future evaluations should distinguish between within-study empirical comparisons and across-study evidence synthesis. Controlled benchmarking should use common hardware and software environments, identical benchmark configurations, clearly reported parameter settings, and standardized performance measures. Where stochastic algorithms are evaluated, repeated independent runs and appropriate statistical tests should be reported.

5.1 Future research directions

The synthesis of existing OPAT strategies highlights several promising research directions that emerge for advancing combinatorial testing.

  • i. Toward Unified Optimization Strategies: current research largely treats scalability, optimality, and feature integration as separate concerns. Future work should focus on developing unified strategies that simultaneously balance these objectives, potentially through adaptive or multi-objective optimization models that dynamically adjust strategy behavior based on problem characteristics.

  • ii. Beyond Harmony Search Algorithm: the dominance of Harmony Search in current metaheuristic OPAT strategies indicates a lack of exploration of alternative optimization techniques. Future research should investigate diverse metaheuristics and hybrid models, including Particle Swarm Optimization, Tabu Search, and hyper-heuristic approaches, to improve robustness and reduce dependence on algorithm-specific tuning.

  • iii. Intelligent and Adaptive Test Generation: emerging trends in artificial intelligence present opportunities to enhance OPAT strategies with learning-based adaptation. Future systems could leverage historical testing data, fault patterns, or runtime feedback to dynamically guide parameter selection, prioritize critical interactions, and improve overall testing effectiveness.

  • iv. Greater attention should be given to reproducible benchmarking through publicly available source code, executable implementations, benchmark configurations, and generated test suites.

  • v. End-to-End Integration and Industrial Validation: despite the availability of tools such as ACTS, a gap persists between academic advancements and industrial adoption. Future research should focus on integrating OPAT-based test generation into automated testing pipelines, including execution, monitoring, and result analysis. Additionally, large-scale empirical studies on real-world systems are needed to validate the effectiveness and scalability of advanced OPAT strategies for fault detection.

5.2 Threats to validity

Although this systematic literature review was designed and conducted following a rigorous, transparent methodology to minimize bias and maximize coverage, several threats to its construct, internal, and external validity must be addressed:

  • i. Study Selection and Screening Validity: to mitigate selection bias and prevent the accidental omission of relevant publications, a multi-stage screening and selection protocol was established. Two reviewers independently screened the titles and abstracts of all retrieved studies against the inclusion and exclusion criteria and subsequently conducted independent full-text evaluations of the candidate papers. Any screening disagreements were resolved through formal consensus-building discussions, with a third reviewer acting as an arbitrator to make final determinations. To quantitatively measure the reliability of this screening process, Cohen's Kappa (k) was calculated prior to consensus discussions, yielding k = 0.83 for the title and abstract screening phase and k = 0.89 for the full-text eligibility phase, indicating substantial to almost perfect inter-reviewer agreement.

  • ii. English-Only Publication Restriction: a potential external threat to the comprehensiveness of our study pool is the language restriction in our eligibility criteria, which limits our primary studies strictly to those published in English.

  • iii. Exclusion of Gray Literature: our search strategy focused strictly on peer-reviewed academic journal papers and conference proceedings. This high-quality threshold successfully filtered out conceptually weak or unverified publications.

6 Conclusion

This systematic literature review provides a comprehensive synthesis of One-Parameter-at-a-Time (OPAT) strategies for combinatorial interaction testing (CIT), based on 40 primary studies published between 2000 and 2026. The review establishes a five-category taxonomy that tracks OPAT's evolution from early deterministic foundations to parallel/distributed scaling, feature-enhanced handling (constraints, seeding, and variable strengths), metaheuristic/hybrid optimization models, and domain-specific adaptations. The findings indicate persistent trade-offs among test suite compactness, generation time, memory consumption, scalability, and interaction strength. Deterministic and distributed approaches primarily address computational and scalability challenges, whereas metaheuristic and hybrid approaches can improve test-suite minimization but generally incur greater computation cost.

An important finding of this review is that the quantitative performance results reported across OPAT studies cannot generally be combined to establish a reliable universal ranking of strategies. Differences in hardware, implementation environments, benchmark configurations, parameter settings, interaction strengths, and experimental procedures limit direct cross-study comparison. Moreover, the limited availability of publicly accessible source code and executable implementations restricts independent reproduction and controlled head-to-head evaluation. Thus, the reported quantitative results provide valuable evidence of within-study performance but should not be interpreted as a definitive cross-study comparison of OPAT strategies.

These findings highlight benchmarking and reproducibility as important methodological gaps in OPAT research. Future research should provide publicly accessible implementations, complete experimental configurations, benchmark instances, generated test suites, parameter settings, and reproducible execution environments to facilitate independent replication and controlled evaluation. Further research should also explore underutilized metaheuristics and hybrid optimization approaches, scalable, parallel, and distributed OPAT frameworks, adaptive AI-driven test generation, and validation on large-scale industrial systems. Overall, advancing OPAT-based CIT requires not only more effective optimization mechanisms but also standardized, transparent, and reproducible empirical evaluation practices.

Statements

Author contributions

MY: Formal analysis, Writing – original draft, Software, Project administration, Supervision, Methodology, Investigation, Conceptualization, Validation, Data curation, Writing – review & editing. AH: Conceptualization, Resources, Validation, Funding acquisition, Writing – review & editing, Supervision, Writing – original draft. HM: Methodology, Writing – original draft, Validation, Data curation, Formal analysis. SA: Writing – review & editing, Methodology, Formal analysis, Data curation, Validation. UU: Writing – original draft, Validation, Formal analysis, Visualization. DA: Formal analysis, Data curation, Writing – original draft, Validation. IG: Data curation, Writing – review & editing, Validation, Formal analysis.

Funding

The author(s) declared that financial support was received for this work and/or its publication. This work was supported by the Universiti Teknologi PETRONAS Foundation Fundamental Research under Grant (YUTPR-FRG: 015LC0-604).

Conflict of interest

The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declared that Generative AI was not used in the creation of this manuscript.

Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

  • 1

    Abdul MananM. S.Abang JawawiD. N.AhmadJ. (2021). ā€œA systematic literature review on test case prioritization in combinatorial testing,ā€ in ACM international conference proceeding series, 55–61. doi: 10.1145/3490700.3490710

  • 2

    AlsewariA. A. Mu'aza, A. A.RassemT. H.TairanN. M.ShahH.ZamliK. Z. (2018). One-parameter-at-a-time combinatorial testing strategy based on harmony search algorithm OPAT-HS. Adv. Sci. Lett.24, 7273–7277. doi: 10.1166/asl.2018.12927

  • 3

    Aminu MuazuA.Aminu MuazuA. (2018). ā€œOne-parameter-at-a-time combinatorial testing strategy based on harmony search algorithm supporting mixed covering array mathematical notation (OPATHS),ā€ in 1st international conference on information technology in education & development (ITED) (At: Abuja), 64–70.

  • 4

    AminuA. M.HashimA. S. (2024). Comparative analysis of IPOG strategy variants for enhanced combinatorial T-way testing efficiency. UMYU Sci.3, 141–150. doi: 10.56919/usci.2433.016

  • 5

    CalvagnaA.GargantiniA. (2009). ā€œIPO-s: incremental generation of combinatorial interaction test data based on symmetries of covering arrays,ā€ in IEEE international conference on software testing, verification, and validation workshops, 10–18.

  • 6

    CuiY.LiL.YaoS. (2009). ā€œA new strategy for pairwise test case generation,ā€ in 3rd international symposium on intelligent information technology application, IITA 2009, Vol. 3, 303–306. doi: 10.1109/IITA.2009.416

  • 7

    EbertC.BajajD.WeyrichM. (2022). Testing software systems. IEEE Softw.39, 8–17. doi: 10.1109/MS.2022.3166755

  • 8

    FadhilH. M.AbdullahM. N.YounisM. I. (2022). Combinatorial testing approaches : a systematic review. Iraqi J. Comput. Commun. Control Syst. Eng.22, 60–79. doi: 10.33103/uot.ijccce.22.4.6

  • 9

    FarinaM.FedorovskayaA.PolivtsevE.SucciG. (2022). Software engineering and filmmaking: a literature review. Front. Comput. Sci. 4, 1–13. doi: 10.3389/fcomp.2022.884533

  • 10

    ForbesM.LawrenceJ.LeiY.KackerR. N.KuhnD. R. (2008). Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Natl. Inst. Stand. Technol. 113:287.

  • 11

    GaoS.LvJ.DuB.JiangY.MaS. (2014a). ā€œGeneral optimization strategies for refining the In-Parameter-order algorithm,ā€ in Proceedings - international conference on quality software, 21–26. doi: 10.1109/QSIC.2014.15

  • 12

    GaoS.DuB.JiangY.LvJ.MaS. (2014b). ā€œAn efficient algorithm for pairwise test case generation in presence of constraints,ā€ in The 2014 2nd international conference on systems and informatics (ICSAI 2014), November (IEEE), 406–410.

  • 13

    GaoS. W.LvJ. H.DuB. L.ColbournC. J.MaS. L. (2015). Balancing frequencies and fault detection in the in-parameter-order algorithm. J. Comput. Sci. Technol.30, 957–968. doi: 10.1007/s11390-015-1574-6

  • 14

    GuptaD.RanaA. (2013). ā€œFibonacci driven novel test generation strategy for constrained testing,ā€ in Proceedings of the 2013 3rd IEEE international advance computing conference, IACC 2013, 1475–1478. doi: 10.1109/IAdCC.2013.6514444

  • 15

    GuptaD.RanaA.TyagiS. (2017). ā€œSequence generation of test case using pairwise approach methodology,ā€ in Proceedings of advances in intelligent systems and computing, September (Springer), 79–85. doi: 10.1007/978-981-10-3773-3_9

  • 16

    HemmatA.SharbafM.Kolahdouz-RahimiS.LanoK.TehraniS. Y. (2025). Research directions for using LLM in software requirement engineering: a systematic review. Front. Comput. Sci.7, 1–18. doi: 10.3389/fcomp.2025.1519437

  • 17

    KathiresanG. (2024). A multimodal approach to software quality assurance: integrating static analysis, dynamic testing, and AI-based anomaly detection. Int. J. Innov. Res. Comput. Commun. Eng. 12. doi: 10.15680/IJIRCCE.2024.1202003

  • 18

    KitchenhamB.ChartersS. (2007). Guidelines for performing systematic literature reviews in software engineering (Version 2.3, EBSE Technical Report EBSE-2007-01). Keele University and Durham University

  • 19

    KlaibM. F.MuthuramanS.AhmadN.SidekR. (2010). Tree based test case generation and cost calculation strategy for uniform parametric pairwise testing. J. Comput. Sci.6, 425–430. doi: 10.3844/jcssp.2010.542.547

  • 20

    LeiY.KackerR.KuhnD. R.OkunV.LawrenceJ. (2007). ā€œIPOG: a general strategy for T-way software testing,ā€ in Proceedings of the international symposium and workshop on engineering of computer based systems, 549–556. doi: 10.1109/ECBS.2007.47

  • 21

    LeiY.KackerR.KuhnD. R.OkunV.LawrenceJ. (2008). IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing. Softw. Test. Verif. Reliab.18, 125–148. doi: 10.1002/stvr.381

  • 22

    LopuhaO.TsiutsiuraS.PoplavskyiO.LysytsinO.BondarO.KrukP. (2023). ā€œTest design methodology for software verification,ā€ in 2023 IEEE international conference on smart information systems and technologies (SIST) (IEEE), 241–245. doi: 10.1109/SIST58284.2023.10223573

  • 23

    MaityS.NayakA. (2005). ā€œImproved test generation algorithms for pair-wise testing,ā€ in Proceedings - international symposium on software reliability engineering, ISSRE 2005, 235–244. doi: 10.1109/ISSRE.2005.23

  • 24

    MercanH.JaveedA.YilmazC. (2020). ā€œFlexible combinatorial interaction testing,ā€ in IEEE transactions on software engineering, Vol. 48 (IEEE), 1030–1066. doi: 10.1109/TSE.2020.3010317

  • 25

    MuazuA. A.Danjuma MaiwadaU. (2020). ā€œPWiseHA: application of harmony search algorithm for test suites generation using pairwise techniques,ā€ in International journal of computer and information technology, Vol. 9 (Number 4). doi: 10.24203/ijcit.v9i4.23

  • 26

    MuazuA. A.HashimA. S. (2025). ā€œEnhancing harmony search metaheuristic algorithm for coverage efficiency, test suite reduction, and running time in combinatorial interaction testing,ā€ in IEEE access, Vol. 13 (IEEE), 110828–110852. doi: 10.1109/ACCESS.2025.3583176

  • 27

    MuazuA. A.HashimA. S.AudiU. I. Aliyu Y YahayaM. S. (2024c). Combinatorial interaction testing for T-way test case generation: a scoping review of the perspective features. Zhongguo Kuangye Daxue Xuebao, 29, 1–21.

  • 28

    MuazuA. A.HashimA. S.AudiU. I.MaiwadaU. D. (2024a). ā€œRefining a one-parameter-at-a-time approach using harmony search for optimizing test suite size in combinatorial T-way testing,ā€ in IEEE access, Vol. 12 (IEEE), 137373–137398. doi: 10.1109/ACCESS.2024.3463953

  • 29

    MuazuA. A.HashimA. S.MaiwadaU. D.Isma'ilaU. A.YakubuM. M.IbrahimM. A. (2024b). Pairwise test case generation with harmony search, one-parameter-at-at-time, seeding, and constraint mechanism integration. Int. J. Electr. Comput. Eng.14, 3137–3149. doi: 10.11591/ijece.v14i3.pp3137-3149

  • 30

    MuazuA. A.HashimA. S.MaiwadaU. D.MuppidiA. (2023a). Enhanced version of seeding and constraint support in IPOG strategy for variable strength interaction t-way testing. Malays. J. Comput. Sci.36, 381–403. doi: 10.22452/mjcs.vol36no4.4

  • 31

    MuazuA. A.HashimA. S.SarlanA. (2022). ā€œReview of nature inspired metaheuristic algorithm selection for combinatorial t-way testing,ā€ in IEEE access, Vol. 10 (IEEE), 27404–27431. doi: 10.1109/ACCESS.2022.3157400

  • 32

    MuazuA. A.HashimA. S.SarlanA.AbdullahiM. (2023b). SCIPOG: seeding and constraint support in IPOG strategy for combinatorial t-way testing to generate optimum test cases. J. King Saud Univ. Comput. Inform. Sc.35, 185–201. doi: 10.1016/j.jksuci.2022.11.010

  • 33

    MuazuA. A.MuazuA. A. (2018). ā€œDesign of a harmony search algorithm based on covering array T-way testing strategy,ā€ in 1st international conference on information technology in education & development (ITED), 33–38.

  • 34

    NiuX.NieC.LeiJ. Y.LeungH.WangX. (2020a). ā€œIdentifying failure-causing schemas in the presence of multiple faults,ā€ in IEEE transactions on software engineering, Vol. 46 (IEEE), 141–162. doi: 10.1109/TSE.2018.2844259

  • 35

    NiuX.NieC.LeungH.LeiY.WangX.XuJ.et al. (2020b). ā€œAn interleaving approach to combinatorial testing and failure-inducing interaction identification,ā€ in IEEE transactions on software engineering, Vol. 46 (IEEE), 584–615. doi: 10.1109/TSE.2018.2865772

  • 36

    Prasad MudarakolaL.SuneethaP.ShahidaS.SomasekharaK. (2014). An efficient and improved in parameter strategy (IIPO) for generating the pairwise test cases. Int. J. Sci. Eng. Technol. Res.3, 9375–9380.

  • 37

    RamliN.OthmanR. R.HendradiR.ZakariaH. L.IsmailI.Mohd ZakiN. A. N. A.et al. (2024). Input-output based relations T-way test suite generation strategy based on ant colony optimization algorithm (iTTSGA). J. Adv. Res. Appl. Sci. Eng. Technol.63, 1–16. doi: 10.37934/araset.63.1.116

  • 38

    RaoC.LiN.LeiY.KackerR. N.KuhnD. R.GuoJ. (2019). ā€œUsing parameter mapping to avoid forbidden tuples in a covering array,ā€ in Proceedings - 2019 IEEE 12th international conference on software testing, verification and validation workshops, ICSTW 2019, 251–259.

  • 39

    ShwethaM. S. (2017). ā€œRecast IPOG-D algorithm with constraint handling for combinatorial testing,ā€ in Proceedings - 2017 international conference on recent advances in electronics and communication technology, ICRAECT 2017 (Bangalore: IEEE), 179–185. doi: 10.1109/ICRAECT.2017.62

  • 40

    SohZ. H. C.AbdullahS. A. C.ZamliK. Z. (2014). A distributed t-way test suite generation using ā€œone-parameter-at-a-timeā€ approach. Int. J. Adv. Soft Comput. Appl.5, 91–103.

  • 41

    TaiK. C.LeiY. (2002). ā€œA test generation strategy for pairwise testing,ā€ in IEEE transactions on software engineering, Vol. 28 (IEEE), 109–111. doi: 10.1109/32.979992

  • 42

    TranH. V.TungL. N.HungP. N. (2022). ā€œA pairwise based method for automated test data generation for C/C++ projects,ā€ in Proceedings - 2022 RIVF international conference on computing and communication technologies, RIVF 2022, 352–357. doi: 10.1109/RIVF55975.2022.10013824

  • 43

    UthaileangW.KiattisinS.LeelasantithamA. (2014). ā€œPair-wise testing applied with online registration,ā€ in JICTEE 2014 : the fourth joint international conference on information and communication technology, electronic and electrical engineering. March 5-8, 2014 (Chiang Rai: IEEE). doi: 10.1109/JICTEE.2014.6804077

  • 44

    VillothJ. P.ZivkovicM.ZivkovicT.Abdel-salamM.HammadM.JovanovicL.et al. (2025). Two-tier deep and machine learning approach optimized by adaptive multi-population firefly algorithm for software defects prediction. Neurocomputing630:129695. doi: 10.1016/j.neucom.2025.129695

  • 45

    WangL. (2017). ā€œA reduction method of combinatorial testing based on input parameter constraints,ā€ in 2017 IEEE 9th international conference on communication software and networks (ICCSN), 1482–1485.

  • 46

    WangZ.XuB.NieC. (2008). ā€œGreedy heuristic algorithms to generate variable strength combinatorial test suite,ā€ in Proceedings - international conference on quality software, 155–162.

  • 47

    YahayaM. S.HashimA. S.TijjaniS.SalisuJ. A.UsmanF. S.ElhuseiniG. (2026). ā€œHOS: a hybrid metaheuristic one-parameter-at-a-time strategy for T-way test case generation,ā€ in IEEE access, Vol. 14 (IEEE), 30387–30410. doi: 10.1109/ACCESS.2026.3666942

  • 48

    YahayaM. S.HashimA. S. B.Oluwagbemiga BalogunA.Aminu MuazuA.Sabo UsmanF.Adamu AliyuD.et al. (2025). ā€œExploration and exploitation mechanism in pairwise test case generation: a systematic literature review,ā€ in IEEE access, Vol. 13 (IEEE), 82342–82377. doi: 10.1109/ACCESS.2025.3566163

  • 49

    YounisM. I. (2020a). DEO: a dynamic event order strategy for t-way sequence covering array test data generation. Baghdad Sci. J.17:0575. doi: 10.21123/bsj.2020.17.2.0575

  • 50

    YounisM. I. (2020b). GAMIPOG: A deterministic genetic multi-parameter-order strategy for the generation of variable strength covering arrays. J. Eng. Sci. Technol.15, 3142–3162.

  • 51

    YounisM. I.ZamliK. Z. (2011). MIPOG - An Efficient t-Way Minimization Strategy for Combinatorial Testing.Int. J. Comput. Theory Eng. 3, 388–397. doi: 10.7763/IJCTE.2011.V3.337

  • 52

    YounisM. I.ZamliK. Z.IsaN. A. M. (2008a). ā€œMIPOG-Modification of the IPOG strategy for T-way software testing,ā€ in Proceedings of the distributed frameworks and applications (DFmA 2008) (Penang, Malaysia), 1–6.

  • 53

    YounisM. I.ZamliK. Z.IsaN. A. M. (2008b). ā€œA strategy for grid based T-way test data generation,ā€ in 2008 First international conference on distributed framework and application, 73–78. doi: 10.1109/ICDFMA.2008.4784416

  • 54

    YuL.LeiY.KackerR. N.KuhnD. R. (2013a). ā€œACTS: a combinatorial test generation tool,ā€ in Proceedings - IEEE 6th international conference on software testing, verification and validation, ICST 2013, 370–375. doi: 10.1109/ICST.2013.52

  • 55

    YuL.LeiY.NourozborazjanyM.KackerR. N.KuhnD. R. (2013b). ā€œAn efficient algorithm for constraint handling in combinatorial test generation,ā€ in Proceedings - IEEE 6th international conference on software testing, verification and validation, ICST 2013, 242–251. doi: 10.1109/ICST.2013.35

  • 56

    ZhangX.ShenW.LiangZ.CuiL.WangY. (2024). ā€œResearch and application of automated testing technology for data security vulnerabilities,ā€ in 2024 4th international conference on mobile networks and wireless communications (ICMNWC) (Tumkuru: IEEE), 01–05. doi: 10.1109/ICMNWC63764.2024.10872029

  • 57

    ZivkovicT.ZivkovicM. (2026). Metaheuristic optimization of ML for software testing defect prediction: a systematic review and critical analysis of methods, datasets, and research gaps. Evol. Intell.19:103. doi: 10.1007/s12065-026-01225-z

  • 58

    ZiyuanW.ChanghaiN.BaowenX. (2007). ā€œGenerating combinatorial test suite for interaction relationship,ā€ in Fourth international workshop on software quality assurance: in conjunction with the 6th ESEC/FSE Joint Meeting, 55–61. doi: 10.1145/1295074.1295085

Appendix

Table A1

IDsLiteratureQuality assessmentTaxonomy classification
ReferencesStrategyt-wayQA1QA2QA3QA4Total score
S1Tai and Lei (2002)IPOt = 211114Foundational
S2Maity and Nayak (2005)–t = 211114Specialized OPAT strategies
S3Lei et al. (2007)IPOGt ≄ 711114Deterministic refinement
S4Ziyuan et al. (2007)ReqOrdert ≤ 310.5113.5Specialized OPAT strategies
S5Lei et al. (2008)IPOG-Dt ≤ 611114Deterministic refinement
S6Forbes et al. (2008)IPOG-Ft ≤ 31110.53.5Deterministic refinement
S7Younis et a. (2008a)MIPOGt ≤ 611114Deterministic refinement
S8Wang et al. (2008)ParaOrdert ≤ 3110.50.53Specialized OPAT strategies
S9Younis et al. (2008b)G_MIPOGt ≤ 61110.53.5Scaling and distribution strategies
S10Calvagna and Gargantini (2009)IPO-st = 2110.513.5Specialized OPAT strategies
S11Cui et al. (2009)VIPOt = 211114Deterministic refinement
S12Klaib et al. (2010)TBGCCt = 210.50.50.52.5Specialized OPAT strategies
S13Younis and Zamli (2011)MC-MIPOGt ≄ 1211114Scaling and distribution strategies
S14Yu et al. (2013b)IPOG-Ct ≤ 311114Feature-enhanced strategies
S15Gupta and Rana (2013)N-IPOt = 2110.502.5Specialized OPAT strategies
S16Yu et al. (2013a)ACTSt ≤ 611002Feature-enhanced strategies
S17Soh et al. (2014)TS_IPOt ≤ 610.5113.5Scaling and distribution strategies
S18Uthaileang et al. (2014)–t = 21110.53.5Specialized OPAT strategies
S19Prasad Mudarakola et al. (2014)IIPOt = 2110.50.53Deterministic refinement
S20Gao et al. (2014a)t ≤ 51110.53.5Deterministic refinement
S21Gao et al. (2014b)IPO-SATt = 21110.53.5Feature-enhanced strategies
S22Gao et al. (2015)–t ≤ 61110.53.5Deterministic refinement
S23Wang (2017)–t = 210.50.502Feature-enhanced strategies
S24Shwetha (2017)RIPOG-Dt ≤ 610.510.53Feature-enhanced strategies
S25Aminu Muazu and Aminu Muazu (2018)HCATSt = 2110.502.5Metaheuristic and hybrid strategies
S26Gupta et al. (2017)–t = 210.50.502Specialized OPAT strategies
S27Alsewari et al. (2018)OPAT-HSt = 311114Metaheuristic and hybrid strategies
S28Muazu and Muazu (2018)OPATHSt ≤ 310.5113.5Metaheuristic and hybrid strategies
S29Rao et al. (2019)–t ≤ 6110.50.53Feature-enhanced strategies
S30Younis (2020b)GAMIPOGt ≄ 711114Metaheuristic and hybrid strategies
S31Muazu and Danjuma Maiwada (2020)PWiseHAt ≤ 411114Metaheuristic and hybrid strategies
S32Younis (2020a)DEOt = 711114Specialized OPAT Strategies
S33Tran et al. (2022)PMCt = 210.510.53Specialized OPAT strategies
S34Muazu et al. (2023b)SCIPOGt ≄ 711114Feature-enhanced strategies
S35Muazu et al. (2023a)SCIPOG-VSt ≄ 711114Feature-enhanced strategies
S36Muazu et al. (2024a)SCHOPt ≤ 611114Metaheuristic and hybrid strategies
S37Muazu et al. (2024b)PHOSCt = 211114Metaheuristic and hybrid strategies
S38Aminu and Hashim (2024)eIPOGt ≤ 611114Metaheuristic and hybrid strategies
S39Muazu and Hashim (2025)eHSt ≤ 611114Metaheuristic and hybrid strategies
S40Yahaya et al. (2026)HOSt = 211114Metaheuristic and hybrid strategies

Publication trend.

Summary

Keywords

combinatorial interaction testing, metaheuristic optimization, One-Parameter-at-a-Time, software testing, test suite generation, t-way testing

Citation

Yahaya MS, Hashim AS, Mamman H, Abdullahi S, Umer U, Adamu Aliyu D and Godowoli IM (2026) Constructing t-way test suites: a systematic survey and qualitative synthesis of the One-Parameter-at-a-Time approach. Front. Comput. Sci. 8:1946723. doi: 10.3389/fcomp.2026.1946723

Received

23 July 2026

Revised

10 September 2026

Accepted

29 September 2026

Published

15 October 2026

Volume

8 - 2026

Edited by

Martin Shepperd, Brunel University London, United Kingdom

Reviewed by

Miodrag Zivkovic, Singidunum University, Serbia

Heba Fadhil, University of Baghdad, Iraq

Updates

Copyright

*Correspondence: Muhammad Sabo Yahaya,

Disclaimer

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Outline

Figures

Cite article

Copy to clipboard


Export citation file


Share article

Article metrics