Abstract
Parameter interactions are a major source of software failures, yet the exponential growth of possible configurations makes exhaustive testing impractical. Combinatorial Interaction Testing (CIT) is a highly effective sampling technique for identifying software defects caused by parameter interactions. Over the past two decades, the One-Parameter-at-a-Time (OPAT) constructions paradigm has emerged as a cornerstone of CIT test suite generation. Although numerous OPAT strategies have been proposed to optimize test suite size and execution speed, the literature remains fragmented, and direct side-by-side quantitative benchmarking has been impeded by a lack of open-source tooling and standardized platforms. To address this, this study presents a systematic survey and qualitative synthesis of 40 primary OPAT studies published between 2000 and 2026, retrieved from four major digital libraries. Rather than attempting a direct quantitative benchmark comparison of execution times, which is shown to be mathematically invalid due to highly heterogeneous self-reported hardware environments and the closed-source nature of most academic tools, we establish a unified taxonomy that classifies OPAT research into five major categories: deterministic refinements, scaling and distribution strategies, advanced feature-enhanced strategies, metaheuristic and hybrid strategies, and specialized OPAT adaptations. Our analysis synthesizes qualitative trends, structural characteristics, parameter-order growth mechanisms, and feature-integration gaps across the literature. We conclude with a critical assessment of the tooling and reproducibility challenges in the field, outlining vital future pathways, including diversified metaheuristic and hyper-heuristic techniques, deterministic parallel algorithms, and AI-driven adaptive test generation, to transition OPAT research into industrial-grade, open-source software engineering pipelines.
1 Introduction
Software testing plays a vital role in ensuring software quality, reliability, and safety by identifying defects before deployment (Ebert et al., 2022; Lopuha et al., 2023). As modern software systems continue to grow in size, functionality, and configurability, the number of possible input combinations grows exponentially, making exhaustive testing impractical (Fadhil et al., 2022). This challenge is particularly evident in configurable systems, where failures often stem not from individual input values but from unexpected interactions among multiple parameters. Consequently, efficient testing strategies that detect interaction faults while minimizing testing effort have become increasingly important (Villoth et al., 2025; Zivkovic and Zivkovic, 2026).
Traditional software testing techniques, including static and dynamic testing, provide valuable mechanisms for detecting a wide range of software defects (Kathiresan, 2024; Zhang et al., 2024). Static testing examines software artifacts, such as requirements, design documents, and source code, without executing the program, whereas dynamic testing evaluates software behavior during execution using white-box, black-box, or gray-box techniques. Although these approaches are effective in many testing scenarios, they are generally insufficient for systematically identifying faults arising from interactions among multiple input parameters (Niu et al., 2020a,b). As software configurations become more complex, the number of possible parameter combinations increases combinatorially, making comprehensive validation increasingly difficult.
Combinatorial Interaction Testing (CIT) addresses this challenge by systematically selecting a relatively small yet highly effective set of test cases that cover interactions among input parameters (Mercan et al., 2020; Ramli et al., 2024). Rather than executing every possible configuration, CIT exploits the observation that most software failures are triggered by interactions involving only a small number of parameters. Pairwise testing (t = 2) is the most widely adopted form of CIT because it covers every pair of parameter values at least once while significantly reducing the number of required test cases. For systems in which failures depend on higher-order interactions, CIT can be extended to arbitrary interaction strengths (t-way testing), providing a practical balance between testing cost and fault-detection effectiveness.
Existing t-way combinatorial testing approaches can be classified from two complementary perspectives. The first classifies strategies according to interaction strength, including uniform-strength, variable-strength, and input-output-based testing (Aminu and Hashim, 2024). The second classifies test generation strategies according to their construction mechanisms, namely One-Test-at-a-Time (OTAT) and One-Parameter-at-a-Time (OPAT) (Muazu et al. 2023a,b). OTAT strategies construct complete test cases individually by satisfying uncovered interaction requirements, whereas OPAT strategies incrementally build test suites by introducing one parameter at a time through two complementary phases: horizontal growth, which extends existing test cases to maximize interaction coverage, and vertical growth, which generates additional test cases to cover any remaining uncovered interactions. This incremental construction enables OPAT to reuse existing test cases effectively while maintaining good scalability as new parameters are introduced.
Since the introduction of the In-Parameter-Order (IPO) algorithm (Tai and Lei, 2002) and its generalized variant IPOG (Lei et al., 2007), the OPAT approach has evolved considerably. Numerous extensions have been proposed to improve scalability, reduce test suite size, support higher interaction strengths, enable test case seeding and constraint handling, exploit parallel and distributed computing environments, and integrate metaheuristic optimization techniques. These developments demonstrate the OPAT framework's adaptability to increasingly complex software testing problems and diverse application domains.
Despite these advances, a comprehensive synthesis of the OPAT literature remains absent from current review studies. Existing studies typically focus on individual algorithms or specific optimization techniques, while broader review articles primarily examine combinatorial testing from search-based, interaction-based, or metaheuristic perspectives. To the best of our knowledge, no previous review has provided a comprehensive synthesis dedicated exclusively to OPAT-based combinatorial interaction testing. Consequently, there remains a lack of a unified understanding of the evolution of OPAT strategies, their underlying optimization mechanisms, practical capabilities, research trends, and remaining challenges.
To address this gap, this paper presents a comprehensive review of OPAT-based combinatorial interaction testing strategies published between 2000 and 2026. The selected studies are systematically analyzed and organized into five major categories: deterministic refinements, scaling and distribution strategies, feature-enhanced approaches, metaheuristic and hybrid strategies, and specialized OPAT adaptations. Based on this classification, the review examines the strengths, limitations, optimization characteristics, and practical applicability of existing approaches while identifying emerging research directions and open challenges.
The major contributions of this review are summarized as follows:
i. It presents what, to the best of our knowledge, is the first systematic literature review dedicated to One-Parameter-at-a-Time (OPAT) strategies for combinatorial interaction testing, a claim supported by our systematic search of 1,300 publications, which did not identify any other review with this specific focus.
ii. It proposes a unified taxonomy, illustrated in Figure 1, that classifies existing OPAT strategies into five major research categories based on their underlying design objectives and optimization mechanisms.
Figure 1
iii. It provides a comparative analysis of OPAT strategies with respect to scalability, computational efficiency, test suite optimality, feature support, and practical applicability.
iv. It identifies current research directions, including unified optimization frameworks, intelligent parameter adaptation, broader integration of metaheuristics, deterministic parallelization, and large-scale industrial validation.
The remainder of this paper is organized as follows. Section 2 introduces the OPAT approach and reviews related studies. Section 3 describes the review methodology. Section 4 presents and analyzes the findings. Section 5 discusses the major observations, future research directions, and threats to validity. Finally, Section 6 concludes the paper.
2 Study context and related work
This section provides the technical background necessary to understand the One-Parameter-at-a-Time (OPAT) approach and its role in combinatorial interaction testing (CIT). It summarizes the fundamental principles of OPAT and its incremental test-construction process, and reviews existing survey and review studies to establish the research gap that motivates this work.
2.1 One-Parameter-at-a-Time (OPAT) approach
The One-Parameter-at-a-Time (OPAT) approach is one of the widely adopted construction strategies for combinatorial interaction testing. Unlike one-test-at-a-time (OTAT) strategies, which generate complete test cases independently. OPAT incrementally constructs a test suite by introducing one parameter at each generation stage. This incremental strategy enables the reuse of previously generated test cases while maintaining comprehensive interaction coverage, thereby improving computational efficiency and scalability.
The OPAT approach originated from the In-Parameter-Order (IPO) algorithm and was subsequently generalized through the In-Parameter-Order General (IPOG) algorithm to support arbitrary interaction strengths. Since then, numerous extensions have been proposed to improve scalability, reduce test suite size, support constraint handling, facilitate test case seeding, exploit parallel computing environments, and incorporate optimization techniques.
Test generation in OPAT is performed through complementary mechanisms: horizontal growth and vertical growth. Horizontal growth extends existing test cases by assigning values to a newly introduced parameter. The selected value maximizes the number of uncovered interactions that each existing test case covers, thereby reducing the need for additional test cases. Vertical growth is subsequently applied to cover any remaining interactions after horizontal expansion. Additional test cases are generated only when necessary, ensuring complete interaction coverage while minimizing redundancy.
The combination of these two mechanisms enables OPAT to balance interaction coverage, computational efficiency, and test suite compactness. Consequently, OPAT has become the foundation for many subsequent combinatorial testing algorithms and remains one of the most influential paradigms in CIT research.
2.2 Related work
Several review studies have investigated different aspects of combinatorial interaction testing; however, none of these existing reviews has provided a dedicated synthesis of the One-Parameter-at-a-Time (OPAT) approach. To situate our study, we analyze the specific focus and boundaries of these prior works, as summarized in Table 1.
Table 1
| S/No | Review study | Type | Focus |
|---|---|---|---|
| 1 | Abdul Manan et al. (2021) | SLR | Test case prioritization and sequencing metrics |
| 2 | Muazu et al. (2022) | Review | Survey of nature-inspired metaheuristic algorithms selection |
| 3 | Muazu et al. (2024c) | Scoping review | Broad categorization of CIT perspective features |
| 4 | Yahaya et al. (2025) | SLR | Exploration and exploitation mechanism in pairwise test case generation |
| 5 | This work (2026) | SLR | One-Parameter-at-a-Time (OPAT) approach |
Comparative analysis of existing CIT reviews against our work.
Abdul Manan et al. (2021) conducted a systematic literature review on test case prioritization in combinatorial testing. However, their work was strictly bound by post-generation sequence optimization techniques to accelerate early fault detection and did not examine the actual generation algorithm or the horizontal or vertical parameter growth mechanisms of OPAT.
(Muazu et al. 2022) reviewed nature-inspired metaheuristic algorithms for combinatorial testing, identifying more than one hundred optimization algorithms and discussing their potential applications to t-way testing. Although their review provides valuable insights into metaheuristic search engines, it analyzes them as black-box optimization frameworks without categorizing them by construction mechanisms (such as OPAT vs. OTAT), thereby completely omitting the foundational deterministic, distributed, and parallelized parameter-order growth strategies (such as IPOG, IPOG-D, or MC-MIPOG).
More recently, Muazu et al. (2024c) presented a scoping review of combinatorial interaction testing strategies, categorizing existing approaches by methodological characteristics. While their study offers a broad overview of the field, the OPAT approach represents only a single, high-level category within a much wider class of occupational frameworks and lacks a detailed evaluation of its horizontal and vertical growth heuristics, parameter ordering, and trade-offs.
Yahaya et al. (2025) subsequently performed a systematic literature review on exploration and exploitation mechanisms in pairwise test generation. Their analysis concentrated on metaheuristic optimization techniques used in pairwise testing. However, their review was restricted exclusively to pairwise testing (t = 2) and did not investigate the broader, multi-way (t ā„ 3) capabilities, deterministic models, parallel execution frameworks, or specialized adaptations that characterize the wider OPAT framework. provided an SLR focusing on metaheuristic optimization mechanisms specifically for pairwise testing.
3 Methodology
This section outlines the systematic review process adopted to investigate the evolution of combinatorial interaction testing (CIT) strategies within the One-Parameter-at-a-Time (OPAT) approach. The methodology follows a structured and transparent set of guidelines established in the literature (Farina et al., 2022; Hemmat et al., 2025). The review process is organized into four main stages: (i) review design and scoping, (ii) research question formulation, (iii) search strategy and study selection, and (iv) data extraction and synthesis. This structured approach ensures methodological rigor, minimizes selection bias, and supports a comprehensive and reproducible analysis.
3.1 Review design and scope
This review aims to systematically identify, classify, and analyze OPAT-based strategies for combinatorial test generation, with particular emphasis on their optimization mechanisms, scalability, and practical applicability. The scope of this review is defined using the PICOC framework (see Table 2), which guides the formulation of the search strategy.
Table 2
| Element | Description |
|---|---|
| Population | Studies focusing on One-Parameter-at-a-Time (OPAT)-based Combinatorial Interaction Testing (CIT) strategies |
| Intervention | Identification and analysis of OPAT techniques, optimization strategies, and supporting mechanisms, including deterministic refinements, scalability improvements, constraint handling, metaheuristic integration, and specialized adaptations |
| Comparison | Dual-level comparative analysis: (a) evaluating the performance and functional advancements of OPAT strategies relative to each other to track methodological evolution, and (b) synthesizing evidence from the original studies that benchmark OPAT variants against established non-OPAT baselines |
| Outcomes | Development of a unified taxonomy of OPAT-based CIT strategies, identification of major research trends, analysis of strengths and limitations, evaluation of trade-offs among approaches, identification of existing research gaps and future research directions |
| Context | A structured review of existing academic contributions on OPAT-based CIT, covering algorithmic developments, empirical evaluations, and practical applications to understand the evolution and future advancement of OPAT techniques |
PICOC framework for defining the scope and objectives of the review.
Preliminary exploration of the literature indicated a diverse range of OPAT-based strategies, including deterministic refinements, parallel and distributed implementations, feature-enhanced strategies (e.g., constraint handling and seeding), metaheuristic and hybrid methods, and specialized adaptations. This diversity necessitated a structured synthesis to consolidate fragmented findings and provide a unified understanding of the field.
Accordingly, the selected studies were systematically categorized into these five thematic groups, forming the basis for the analytical framework used in the Results section. This categorization enables a comparative evaluation of strategies across key dimensions, including scalability, test-suite optimality, computational cost, and feature integration.
3.2 Research questions
The review was designed to investigate the evolution of One-Parameter-at-a-Time (OPAT) strategies in combinatorial interaction testing. The following research questions guided the study.
i. What OPAT-based strategies have been proposed for combinatorial interaction testing?
ii. What optimization techniques and enhancement mechanisms have been incorporated into OPAT strategies?
iii. What are the strengths, limitations, and application characteristics of existing OPAT approaches?
iv. What research gaps and future directions remain for OPAT-based combinatorial interaction testing?
These research questions collectively provide a comprehensive understanding of the development, capabilities, and future prospects of OPAT strategies.
3.3 Search strategy and study selection
A comprehensive literature search was conducted to identify studies relevant to OPAT-based combinatorial interaction testing. The search covered publications from 2000 to 2026, corresponding to the period from the introduction of the IPO algorithm to the most recent developments in OPAT research. Four major academic databases were selected because of their extensive coverage of software engineering literature: ACM Digital Library, IEEE Xplore, ScienceDirect, and Scopus (see Table 3).
Table 3
| No. | Libraries | Search string |
|---|---|---|
| 1 | ACM Digital Library | [All: ācombinatorial interaction testingā] OR [All: ācombinatorial testingā] OR [All: āpairwise test case generationā] OR [All: āpairwise testingā] OR [All: ātest case generationā] AND [E-Publication Date: (01/01/2000 TO 12/31/2026)] |
| 2 | IEEE Xplore | (āCombinatorial Interaction Testingā OR āCombinatorial Testingā OR āPairwise Test Case Generationā OR āPairwise Testingā OR āTest Case Generationā) |
| 3 | Science Direct | (āCombinatorial Interaction Testingā OR āCombinatorial Testingā OR āPairwise Test Case Generationā OR āPairwise Testingā OR āTest Case Generationā) |
| 4 | Scopus | (((āCombinatorial Interaction Testingā) OR (āCombinatorial Testingā) OR (āPairwise Test Case Generationā) OR (āPairwise Testing) OR (āTest Case Generationā)) AND PUBYEAR > 2001 AND PUBYEAR < 2026 AND (LIMIT-TO (SUBJAREA, āENGIā) OR LIMIT-TO (SUBJAREA, āCOMPā)) AND (LIMIT-TO (DOCTYPE, āarā) OR LIMIT-TO (DOCTYPE, ācpā)) AND (LIMIT-TO (LANGUAGE, āEnglishā)) AND (LIMIT-TO (EXACTKEYWORD, āSoftware Testingā) OR LIMIT-TO (EXACTKEYWORD, āTestingā) OR LIMIT-TO (EXACTKEYWORD, āTest Case Generationā) OR LIMIT-TO (EXACTKEYWORD, āSoftware Engineeringā)) |
Libraries and their related search string.
The search strategy was developed using the PICOC framework and involved three steps:
i. Identification of core concepts for combinatorial interaction testing and OPAT strategies.
ii. Expansion of keywords using synonyms, related terms, and controlled vocabulary.
iii. Construction of search strings using Boolean operators (AND, OR, NOT), quotation marks, and parentheses.
The search strings were applied consistently across all selected databases. The inclusion and exclusion criteria (see Table 4) were defined to ensure relevance by focusing on studies that propose, improve, or evaluate OPAT-based combinatorial testing strategies.
Table 4
| No. | Characteristics | Inclusion | Exclusion |
|---|---|---|---|
| 1 | Year | 2000ā2026 | Before or after the specified period |
| 2 | Document type | Article and Conference paper | Other related document type |
| 3 | Subject area | Computer Science and Engineering | Not related to the mentioned subject |
| 4 | Libraries | ACM Digital Library, IEEE Xplore, Science Direct, and Scopus | Not mentioned library |
| 5 | Language | English | Other languages |
Inclusion and exclusion.
The initial search yielded 1,300 studies. After removing duplicates (N = 320) using Mendeley and Microsoft Excel, the remaining studies were screened based on titles and abstracts, followed by full-text assessment (see Figure 2). To minimize selection bias and ensure a highly reliable selection process, a multi-stage screening and selection protocol was executed independently by multiple researchers.
Figure 2
During the Title and Abstract Screening phase, two reviewers (Yahaya and Mamman) independently evaluated the titles and abstracts of the N = 980 unique records against the eligibility criteria. To quantify the level of agreement between the screeners before any consensus discussions, Cohen's Kappa (k) coefficient was calculated. The inter-reviewer agreement for this initial screening phase was k = 0.83, representing substantial methodological agreement. Disagreements at this stage were resolved through collaborative discussion, resulting in N = 760 records progressing to the next stage.
During the full-text eligibility phase, the same reviewers independently read and assessed the full texts of N = 675, while 85 studies could not be retrieved. The inter-reviewer agreement for the full-text selection phase was k = 0.89, indicating almost perfect agreement. Any persistent discrepancies or borderline cases regarding study inclusion were resolved through a structured consensus-building protocol; if a consensus could not be reached, a third senior reviewer (Abdullahi) acted as an arbitrator to make the final decision. This rigorous process ultimately yielded the final set of N = 40 primary studies included in the systematic review.
During the search, it was observed that the earliest relevant study was published in 2002, while the most recent appeared in 2026. The year-wise distribution of the selected studies is illustrated in Figure 3, showing that the highest number of publications occurred in 2008 (five studies), followed by 2013, 2014, and 2018 (four studies each), and 2020 and 2024 (three studies each).
Figure 3
3.4 Quality assessment of primary studies
To ensure the methodological rigor and reliability of the synthesized findings, a formal Quality Assessment (QA) procedure was conducted on the final set of N = 40 included primary studies. The QA checklist was designed following established guidelines for systematic literature reviews in software engineering (Kitchenham and Charters, 2007) and tailored to the technical characteristics of Combinatorial Interaction Testing (CIT).
Each primary study was evaluated against four criteria:
1) QA1: Does the study clearly describe the OPAT strategy's algorithm, horizontal/vertical growth mechanisms, or optimization search logic?
2) QA2: Are the experimental settings, benchmark configurations (such as TCAS or uniform/mixed covering arrays), and hardware environments explicitly defined?
3) QA3: Are the primary outcomes (test suite sizes, execution/generation time, or interaction coverage) quantitatively measured and reported?
4) QA4: Does the study compare the proposed strategy against established baselines (e.g., IPOG, Jenny, or other OTAT/algebraic approaches) using statistical validation or diverse configurations?
Each study was scored quantitatively for each criterion as follows: yes (1.0 point) if the criterion was fully met; Partial (0.5 points) if the criterion was only partially met or lacked minor details; and No (0.0 points) if the criterion was not addressed. The total QA score for any individual study ranged from 0.0 (lowest quality) to 4.0 (highest quality).
To prevent selection bias, low-scoring studies were not excluded from the review, provided they satisfied the strict inclusion and exclusion criteria outlined in Table 3. Rather, the QA procedures were used to evaluate the strength of the evidence and to ensure that major performance claims (such as near-linear speedup or superior scalability) were supported by robust experimental designs.
The quality profiling revealed high methodological quality across the selected literature (see Appendix Table A1), with 82.5% (33 out of 40) of the primary studies scoring ā„3.0 out of 4.0. The remaining studies scored between 2.0 and 2.5, primarily due to partial metric reporting (missing execution times or hardware specifications) or a lack of statistical significance testing. The high overall quality scores confirm that the synthesized performance claims presented in Section 4 are derived from scientifically credible sources.
3.5 Data extraction and synthesis
A standard data extraction form was designed to ensure consistent collection of information from each selected study. The extracted information included bibliographic details, publication year, proposed OPAT strategy, optimization technique, interaction strength, constraint-handling capability, reported strengths and limitations, and application domain.
The extracted data were synthesized using both qualitative and quantitative analyses. The Qualitative synthesis was employed to classify OPAT strategies according to their methodological characteristics and optimization mechanisms. The quantitative synthesis summarized publication trends, geographical distribution, publication venues, optimization techniques, interaction strength, and application domains.
The synthesized findings were subsequently organized into comparative tables and thematic categories to identify research trends, technological evolution, existing limitations, and future research opportunities.
3.6 Analytical framework
The five-category taxonomy presented in this review was developed inductively rather than established a priori. Following the initial screening of the N = 40 included studies, the authors performed an interactive coding process to identify recurring themes, technical mechanisms, and specific research objectives. The construction of the taxonomy followed a three-phase qualitative synthesis:
i. Iterative Coding: studies were first grouped by their foundational algorithmic logic, such as those that directly refined the original IPO/IPOG mechanisms (e.g., VIPO, IIPO).
ii. Thematic Clustering: new categories emerged as specific research trends were identified, including a distinct cluster of strategies designed for parallel or grid environments (e.g., MC-MIPOG, TS_OP) and those integrating global optimization metaheuristics (e.g., eHS, OPAT-HS).
iii. Refinement: the āSpecialized OPATā category was inductively defined to capture strategies that adapted the approach to niche domains like sequence testing or input-output relations (e.g., DEO, ReqOrder).
To ensure the reliability of the classification, disagreements were resolved through a collaborative, consensus-based protocol. As detailed in the āAuthor Contributionā section, multiple researchers (Yahaya, Mamman, and Abdullahi) independently reviewed the technical features of each strategy. In instances of overlapping characteristics, the research team conducted peer-review sessions to determine the āprimary contributionā of the study (e.g., whether a strategy was primarily a scaling refinement or an advanced feature support) until reaching 100% consensus. This rigorous internal cross-checking was adopted to minimize subjective bias and ensure methodological rigor.
3.7 Criteria for practical and industrial evaluation
The evaluation of Industrial Relevance and Practical Applicability within this review is based on a structured four-tier evidence framework. Rather than inferring applicability from algorithmic design alone, strategies are rated according to documented tool maturity, industrial adoption, and validation against real-world systems:
i. Very High: strategies integrated into widely distributed research or commercial tools (e.g., ACTS/FireEye) that have documented adoption by a large number of organizations (e.g., 1,200 plus organizations for ACTS). These strategies must also be validated against large-scale, heterogeneous real-life systems such as Apache, Bugzilla, and GCC.
ii. High: strategies that are supported by an available standalone tool (e.g., VIPO, PMC, or Jenny) and have been benchmarked against established real-world modules, most notably the FAA's Traffic Collision Avoidance System (TCAS). These ratings also require the presence of critical industrial features such as constraint handling and test case seeding.
iii. Moderate: strategies validated primarily through standard academic benchmarks (e.g., uniform covering arrays) but which demonstrate a clear path to practical use by addressing real-world problems like sequence-dependent events (e.g., DEO) or unit testing in specific languages like C/C++.
iv. Low: conceptual strategies or foundational algorithms that lack available tool support or validation beyond small-scale synthetic configurations.
4 Result analysis
The evolution of combinatorial interaction testing (CIT) strategies within the one-parameter-at-a-time (OPAT) approach reflects a sustained effort to balance test-suite optimality and computational scalability. To provide a comprehensive evaluation, the subsequent synthesis analyzes these strategies across two dimensions. First, it examines how individual OPAT variants refine the foundational IPO/IPOG framework. Second, it highlights the comparative performance of these variants as evidenced by the experimental results reported in the selected primary studies.
The In-Parameter-Order (IPO) strategy forms the foundation for subsequent OPAT developments, which establish a structured, incremental mechanism for generating pairwise test suites through horizontal and vertical growth. Its extension, In-Parameter-Order General (IPOG), generalized this strategy to support t-way interactions. While both approaches offer significant advantages (i.e., incremental extensibility, reuse of existing test configurations, and relatively lower computational complexity compared to one-test-at-a-time (OTAT) strategies), they suffer from scalability limitations due to the exponential growth of interaction combinations at higher strengths (Lei et al., 2007; Tai and Lei, 2002).
To address these limitations, this review demonstrates a diversification of OPAT-based strategies into five major categories: (i) deterministic refinements, (ii) scaling and distribution strategies, (iii) feature-enhanced approaches, (iv) metaheuristic and hybrid strategies, and (v) specialized OPAT adaptations. Each category reflects a distinct research direction aimed at mitigating specific shortcomings of the original IPO/IPOG strategies.
4.1 Performance objectives and empirical evidence in OPAT-based CIT
A systematic assessment of OPAT strategies requires consideration of multiple performance dimensions because no single metric adequately captures the effectiveness of a CIT strategy. Accordingly, the review examined five principal dimensions: test-suite size, generation time, memory consumption, interaction strength, and fault-detection capability. These dimensions are interdependent, and improvements in one may involve trade-offs in another. Importantly, the qualitative results reported in this section are study-specific and are presented according to the experimental conditions of the original studies. They are therefore not interpreted as a controlled cross-study ranking.
Test-suite size is a primary objective in CIT because a smaller suite reduces test execution costs while maintaining the required interaction coverage. However, minimizing a covering array is computationally difficult, and approaches that invest additional search effort in minimization may incur greater generation costs. The quantitative results reported in the primary studies illustrate this trade-off. For example, Lei et al. (2008) reported comparisons between IPOG and IPOG-D (see Table 5) as domain size and interaction strength increased. The results show that IPOG-D substantially reduced generation time relative to IPOG while producing larger test suites. These results demonstrate the trade-off between computational efficiency and test-suite compactness within the experimental setting of the original study.
Table 5
| No. of values | IPOG (size) | IPOG (Time (s)) | IPOG-D (Size) | IPOG-D (Time (s)) | Size ratio (IPOG-D/IPOG) | Time ratio (IPOG-D/IPOG) |
|---|---|---|---|---|---|---|
| 2 | 134 | 4.08 | 188 | 1.40 | 1.40 | 0.34 |
| 3 | 1,123 | 48.08 | 1,552 | 2.39 | 1.38 | 0.05 |
| 4 | 4,531 | 234.08 | 6,739 | 12.31 | 1.49 | 0.05 |
| 5 | 15,095 | 996.89 | 22,197 | 55.06 | 1.47 | 0.06 |
| 6 | 37,748 | 3,273.42 | 56,619 | 203.13 | 1.49 | 0.06 |
| 7 | 81,814 | 9,040.25 | 124,186 | 694.74 | 1.52 | 0.08 |
Original self-reported performance of IPOG and IPOG-D for 5-way testing.
Similarly, Younis and Zamli (2011) compared MIPOG and IPOG for five-way covering arrays as shown in Table 6. MIPOG generally produced smaller test suites, but its computational advantage was not consistent across all configurations; at larger domain sizes, the reported generation time exceeded that of IPOG. This provides further evidence that test-suite optimality and generation efficiency should be considered jointly rather than treated as interchangeable indicators of algorithmic superiority.
Table 6
| Values | MIPOG (size) | MIPOG (Time (s)) | IPOG (Size) | IPOG (Time (s)) | Size ratio (MIPOG/IPOG) |
|---|---|---|---|---|---|
| 2 | 92 | 0.172 | 98 | 0.221 | 0.939 |
| 3 | 626 | 0.593 | 751 | 1.361 | 0.834 |
| 4 | 2,911 | 5.046 | 3,057 | 6.375 | 0.952 |
| 5 | 8,169 | 45.950 | 10,111 | 31.875 | 0.879 |
| 6 | 23,557 | 766.580 | 25,247 | 126.156 | 0.933 |
| 7 | 49,597 | 1,365.450 | 54,567 | 430.937 | 0.909 |
Original self-reported performance of MIPOG and IPOG for 5-way testing.
The evidence from these representative studies, therefore, indicates that OPAT strategies exhibit a fundamental size-time trade-off. More aggressive optimization can reduce the number of generated tests, whereas strategies designed to reduce computational effort may accept a larger test suite. Nevertheless, because the reported results originate from different studies and experimental settings, these numerical values should be interpreted as evidence of within-study behavior rather than as a basis for universal ranking.
Memory requirements become increasingly important as the number of parameters and interactions increases. Explicit enumeration of all t-way combinations can produce substantial memory requirements because of the combinatorial growth of the interaction space. IPOG-D addresses this limitation through its construction mechanism, while distributed approaches such as G_MIPOG partition computation across multiple nodes.
Interaction strength determines the required degree of coverage, ranging from pairwise testing (t = 2) to higher-strength testing. Increasing t substantially increases the number of interactions to be covered and, consequently, affects test-suite size, generation time, and memory requirements. Most reviewed OPAT strategies were evaluated at relatively low interaction strengths, whereas MC-MIPOG and GAMIPOG demonstrate applicability at substantially higher strengths, reaching approximately t =12 and t = 15, respectively. High-strength scalability, therefore, remains an important differentiating characteristic of OPAT approaches.
4.2 Deterministic refinement of OPAT strategies
Deterministic refinements enhance the efficiency and effectiveness of the original IPO/IPOG algorithms by improving heuristics, optimizing search strategies, and making structural modifications. Rather than changing the fundamental OPAT paradigm, these approaches focus on improving the trade-offs between execution time, scalability, and test suite size.
A major characteristic of deterministic strategies is the balance between scalability and optimality. IPOG-D (Lei et al., 2008) extends IPOG by introducing a recursive doubling (D-construction) mechanism and explicit enumeration, thereby improving scalability for larger configurations. Experimental evidence showed that for 5-way testing of 20 4-valued parameters, IPOG-D required only 3% of the standard IPOG's execution time, at the cost of generating 51% more test cases. These tests were conducted on a laptop with a 1.6 GHz CPU and 1 GB of RAM.
Subsequent refinements shifted toward heuristic optimization. IPOF1 and IPOF2 (Wang et al., 2008) enhance the horizontal growth process through greedy heuristics, controlled randomization, and dynamic programming. These improvements achieve faster test generation while maintaining competitive test suite sizes. IPOF1 generally emphasizes test suite compactness, whereas IPOF2 prioritizes execution speed.
MIPOG (Younis et al., 2008a) further improves test suite minimization by applying enhanced greedy optimization during both the horizontal and vertical growth phases. Although this strategy consistently produces smaller test suites than IPOG, Jenny, and ITCH, particularly at t ā„ 4, the additional search effort increases computational cost. While VIPO (Cui et al., 2009) accelerates test generation by simplifying horizontal growth decisions, producing smaller test suites than IPO, TConfig, and AllPairs across 11 benchmark configurations (e.g., 1010 and 1510 uniform configurations) in a running environment with a 3.0 GHz CPU and 1 GB of RAM.
In contrast, IIPO (Prasad Mudarakola et al., 2014) introduces local optimization and incremental reuse mechanisms to reduce ādon't careā values and improve adaptability when new parameters are introduced. While this improves flexibility and generation efficiency, its scalability gains remain moderate. Study 20 by Gao et al. (2014a) proposes general optimization strategies for the IPO algorithmic family by applying cyclic value selection during horizontal growth and by filling ādon't careā positions with the least-frequent values, thereby significantly reducing generation runtime and test suite size as domain sizes increase.
Building on S20, Study 22 by Gao et al. (2015) refines the frequency-balancing mechanism and pairs it with a post-generation test-reordering technique, achieving an average speed-up of two times in test-suite generation time compared to standard IPO variants. Overall, deterministic OPAT strategies demonstrate a clear evolution from improving scalability to enhancing test suite optimality. As summarized in Table 7, each strategy represents a different compromise among computational cost, scalability, and test suite minimization, indicating that no single deterministic refinement can simultaneously optimize all performance objectives.
Table 7
| Strategy | Optimization strategy | Scalability | Test suite size (optimality) | Computational cost | Implications |
|---|---|---|---|---|---|
| IPOG-D | Recursive D-construction (reduce interaction enumeration) | High | Larger | Moderate | Improves scalability for large configurations |
| IPOF1 | Greedy heuristic with expanded search | Moderate | Smaller | Low | Balances efficiency and compactness |
| IPOF2 | Dynamic programming with heuristic refinement | Moderate | Slightly larger than IPOF1 | Low | Prioritizes faster generation |
| MIPOG | Enhanced greedy optimization during horizontal and vertical growth | Moderate | Smallest | High | Minimizes test suite size |
| VIPO | Simplified horizontal growth heuristic | Reduces execution time | May reduce test suite optimality | Moderate | Produces smaller test suites |
| IIPO | Local optimization and incremental reuse | Moderate | Moderate | Low | Improves adaptability to evolving parameters. |
| S20 | Offset/cyclic value search during horizontal growth | High | Smaller | High | Improves reduce run time and produce smaller test suite |
| S22 | Frequency balancing, prioritization, and reordering | High | smaller | High | Shift benchmark paradigm to practical execution cost and rate of fault detection |
Comparative characteristics of deterministic OPAT refinement strategies.
4.3 Scaling and distribution strategies
As combinatorial interaction testing problems scale to industrial-level configurations, the limitations of single-machine execution (particularly memory constraints and generation time) become increasingly prohibitive. This has led to the emergence of scaling and distribution strategies that adapt the OPAT approach to parallel and distributed computing environments.
A dominant trend in this category is the decomposition of the test generation process into independent or semi-independent computational units, enabling concurrent execution. For example, distributed approaches such as G_MIPOG (Younis et al., 2008b) partition both memory and computation across multiple nodes, effectively addressing the out-of-memory limitations inherent in high-dimensional interaction spaces. It demonstrated logarithmic speedups as coverage strength increased, specifically tested with 6-way testing of 7ā10 5-valued parameters using a configuration of six computers.
Similarly, MC-MIPOG (Younis and Zamli, 2011) demonstrates the adaptation of OPAT strategies to shared-memory parallel architectures, leveraging multi-core processors to accelerate test generation. By reducing control and data dependencies within the algorithm, MC-MIPOG enables efficient thread-level parallelism, resulting in near-linear speedup as the number of parameters increases. This was evidence using a 2.4 GHz Core 2 Quad CPU on Linux CentOS, where speedup gains increased linearly for parameter counts from 5 to 16 at t = 4.
In contrast, TS_OP (Soh et al., 2014) introduces a more reasonable speedup in a distributed environment using MapReduce and Tuple Space. Testing on a network of five identical 2.2 GHz workstations showed that, while speedup was not perfectly linear due to network overhead, distributing the workload significantly reduced main memory utilization per machine.
As shown in Table 8, distributed approaches primarily address memory constraints, whereas multicore-based strategies focus on execution speed, highlighting distinct optimization priorities within scaling solutions.
Table 8
| Strategy | Parallelization model | Benefits | Limitation | Application |
|---|---|---|---|---|
| G_MIPOG | Distributed (Grid-based) | Improves memory scalability | Requires distributed infrastructure | High-dimensional configurations |
| MC-MIPOG | Shared memory (Multicore) | Accelerates test generation | Limited by available processor cores | Modern multicore systems |
| TS_OP | Distributed (Tuple Space/MapReduce) | High scalability and workload distribution | Non-deterministic outputs | Large distributed environments |
Scalability and parallelization characteristics of OPAT strategies.
4.4 Advanced features and applicability
Beyond scalability, modern combinatorial testing increasingly demands support for real-world constraints, domain-specific requirements, and flexible interaction models. As a result, OPAT-based strategies have evolved to incorporate advanced features such as constraint handling, seeding, and mixed-strength interactions, enhancing their practical applicability.
A major focus in this category is the integration of constraint-handling mechanisms that ensure generated test cases exclude invalid or infeasible parameter combinations. Strategies such as ACTS (Yu et al., 2013a) exemplify this evolution by embedding IPOG and its variants within a flexible system that supports constraint specification, mixed-strength testing, and multiple user interfaces. This reflects a shift from standalone algorithms to comprehensive testing ecosystems.
More specialized approaches, such as IPOG-C (Yu et al., 2013b) and IPO-SAT (Gao et al., 2014b), focus on improving the efficiency of constraint evaluation. IPOG-C achieves this by minimizing redundant checks and selectively evaluating only relevant constraints, while IPO-SAT leverages SAT solvers to transform constraint handling into a satisfiability problem. These approaches demonstrate that constraint processing is a major computational bottleneck and optimizing it can yield substantial performance gains.
Recently, Shwetha (2017) proposed RIPOG-D, which integrates incremental construction with hybrid optimization techniques and variable-strength constraint support, providing a flexible solution for complex constrained systems. Study 23 by Wang (2017) proposed advanced constraint-handling mechanisms that use logical input-domain validation to prune invalid combinations of web form inputs. In contrast, Study 29 by Rao et al. (2019) introduces parameter-to-column mapping optimization, which maps system parameters onto pre-existing covering arrays (CAs) to minimize the number of discarded test rows when enforcing forbidden-tuple constraints.
In addition to constraints, seeding mechanisms have emerged as a critical feature for aligning test generation with domain knowledge. Strategies such as SCIPOG (Muazu et al., 2023b) and SCIPOG-VS (Muazu et al., 2023a) incorporate seeded values into the generation process, ensuring that critical or high-risk configurations are included in the test suite. When combined with constraint handling and support for variable interaction strengths, these approaches offer a highly flexible and expressive testing framework.
As summarized in Table 9, constraint-handling techniques and seeding mechanisms emerge as the dominant enhancements, with SAT-based and heuristic-based approaches offering different trade-offs between computational cost and flexibility.
Table 9
| Strategy | Feature support | Strengths | Limitations | Industrial relevance |
|---|---|---|---|---|
| ACTS | Constraint handling, mixed-strength testing | Comprehensive testing framework | Constraint evaluation overhead | High |
| IPOG-C | Optimized constraint evaluation | Reduces redundant constraint checking | Additional processing complexity | High |
| IPO-SAT | SAT-based constraint solving | Efficient validation of complex constraints | Solver overhead | High |
| S23 | Logical parameter constraints and web testing | Input domain validation and parameter reduction | Parallelized growth mechanisms | High |
| RIPOG-D | Algebraic doubling construction with constraint handling for VSCA | Handles mixed hardware and OS constraints | Redundant ādon't careā padding | High |
| S29 | Parameter mapping optimization with constraint handling | Preserves base CA optimality | Computationally expensive for large data sets | High |
| SCIPOG | Test case seeding and constraints | Incorporates domain knowledge into test generation | Increased algorithmic complexity | High |
| SCIPOG-VS | Seeding, constraints, and variable strength testing | Supports realistic industrial testing scenarios | Higher implementation complexity | Very High |
Feature integration and constraint-handling capabilities in OPAT strategies.
4.5 Metaheuristic and hybrid strategies
Although deterministic OPAT strategies provide systematic and predictable test generation, they often struggle to achieve optimal solutions when applied to large or highly complex configuration spaces. To address these limitations, recent research has increasingly integrated metaheuristic optimization techniques to enhance exploration of the search space while preserving OPAT's structured construction process.
A prominent direction within this category is the adoption of Harmony Search Algorithm (HSA)-based strategies, such as OPAT-HS (Alsewari et al., 2018) and its derivatives. These approaches model test generation as an optimization problem in which candidate solutions are iteratively refined to maximize interaction coverage while minimizing redundancy. By embedding OPAT principles within a metaheuristic framework, these strategies combine systematic construction with stochastic optimization to achieve improved test suite compactness and generation efficiency.
Subsequent extensions, including HCATS (Muazu and Muazu, 2018), OPATHS (Aminu Muazu and Aminu Muazu, 2018) and PWiseHA (Muazu and Danjuma Maiwada, 2020), further refine this integration by enhancing support for mixed covering arrays and simplifying the optimization process. These approaches demonstrate that metaheuristic techniques can be effectively tailored to OPAT structures, particularly for low-to-moderate interaction strengths.
A hybrid strategy, such as GAMIPOG (Younis, 2020b), represents a more advanced integration that combines a genetic algorithm with the OTAT approach and deterministic OPAT components (e.g., MIPOG). This allows the algorithm to leverage both global search capabilities and structured incremental construction, resulting in improved scalability and support for higher interaction strengths (up to t = 15). It outperformed most existing strategies in terms of size (except Simulated Annealing for t = 3) when tested on a 2.2 GHz Core i7 with 8 GB of RAM.
Within the metaheuristic category, PHOSC (Muazu et al., 2024b), SCHOP (Muazu et al., 2024a), and eIPOG (Aminu and Hashim, 2024) represent a family of near-equivalent strategies that concurrently integrate seeding and constraint handling into the Harmony Search Algorithm (HSA) using the OPAT framework. Despite their shared functional objectives, they are distinguished by their depth and underlying (t = 2), utilizing high-probability local improvisation to achieve test-suite compactness. SCHOP provides the broadest interaction support (up to t = 6) and employs an optimized balance of intensification and diversification to maximize the test suite's validity in complex industrial configurations. Finally, eIPOG refines the IPOG mechanism by modifying its internal bit structures to enhance uniform interaction testing up to t = 5.
More recent developments further extend this direction toward adaptive and multi-stage optimization. The eHS strategy (Muazu and Hashim, 2025) enhances Harmony Search by incorporating OPAT-based initialization within the Harmony Memory and dynamically adjusting key parameters (HMCR and PAR) to balance exploration and exploitation, thereby improving coverage efficiency and reducing redundancy. It reported a 90.91% success rate in test suite size reduction across 33 configurations compared to IPOG. However, it achieved 0% success in execution time superiority over IPOG, as the metaheuristic search process is inherently slower than the foundational deterministic greedy algorithm.
Similarly, the Hybrid One-Parameter-at-a-Time Strategy (HOS) (Yahaya et al., 2026) combines multiple metaheuristics within the IPO framework by applying a GA for horizontal growth and Simulated Annealing (SA) for vertical growth. This dual-phase optimization leverages GA's global search capability and SA's local refinement strength. It demonstrated improved test-suite compactness by achieving optimal sizes for 61 of 71 benchmark configurations when tested on an i7-5500U (2.40 GHz) with 8 GB of RAM.
A quantitative analysis of the metaheuristic-based OPAT strategies identified in this review confirms a significant methodological imbalance within the field. The Harmony Search Algorithm (HSA) is the overwhelmingly dominant optimization family, appearing in 77.8% of the analyzed stochastic OPAT studies. In contrast, other established metaheuristics, such as Genetic Algorithms (GA) and Simulated Annealing (SA), appear almost exclusively as components of hybrid models (e.g., GAMIPOG and HOS), rather than as standalone OPAT frameworks.
Table 10 provides a comparative overview of the optimization techniques employed, illustrating how different metaheuristic integrations balance global search capability with the structured nature of OPAT-based construction.
Table 10
| Strategy | Optimization technique | Strength | Limitation |
|---|---|---|---|
| OPAT-HS | Pioneer strategy to integrate Harmony Search into OPAT | Support both Covering arrays (CA) and Mixed CA under uniform interaction strength (t ⤠3). | Performance decreases for high interaction strengths |
| HCATS | Harmony search | Compact test suites and improve search efficiency | Limited scalability at higher interaction strengths |
| OPATHS | Enhanced harmony search | Supports mixed covering arrays | Limited to mixed covering arrays |
| PWiseHA | Simplified harmony search | Stable and efficient optimization | Less effective for high-dimensional problems |
| GAMIPOG | Genetic algorithm (GA) and MIPOG | High scalability and global optimization | Parameter tuning and computational overhead |
| PHOSC | HSA-based OPAT for pairwise search | Multi-objective optimization | Increased computational cost |
| SCHOP | Feature-aware HSA with balanced intensification/diversification | Improves test validity and relevance | Higher algorithm complexity |
| eIPOG | HSA-enhanced IPOG with bit-structure modification | Balances optimization and constraint satisfaction | Additional execution overhead |
| eHS | Adaptive harmony search | Dynamic exploration-exploitation balance | Sensitive parameter tuning |
| HOS | Genetic algorithm and Simulated Annealing (SA) | Global and local optimization with compact test suites | Hybrid coordination overhead |
Optimization strategies in metaheuristic and hybrid OPAT strategies.
4.6 Specialized OPAT adaptations
Beyond improvements in scalability and feature integration, several OPAT strategies have been developed to address specialized testing requirements. Unlike the previous categories, these approaches target specific computational challenges, application domains, or heuristic enhancements while preserving the OPAT framework's incremental construction mechanism.
One research direction focuses on reducing computational overhead through simplified heuristic design. For example, whereas IPO-s (Calvagna and Gargantini, 2009) exploit symmetry among parameter interactions to improve coverage efficiency by introducing informed approximations that maintain acceptable interaction coverage while reducing processing time.
Other studies explore alternative structural representations to improve test generation. TBGCC (Klaib et al., 2010) employs a tree-based model to guide interaction coverage, while N-IPO (Gupta and Rana, 2013) utilizes a Fibonacci-based selection mechanism to reduce test suite size for large input domains. Although these strategies demonstrate improved efficiency in specific scenarios, their effectiveness depends on assumptions regarding parameter relationships or data distributions, which may limit their applicability to more general testing environments.
Specialized OPAT adaptations have extended combinatorial testing beyond conventional parameter interaction problems. DEO (Younis, 2020a) applies the OPAT paradigm to sequence covering arrays for event-driven systems, while PMC (Tran et al., 2022)automates test data generation directly from source code, reducing manual testing effort in software unit testing. In contrast, Study 2 by Maity and Nayak (2005) integrates advanced, specialized combinatorial models and algebraic structures such as mutually orthogonal Latin squares (MOLS) and Orthogonal Arrays (OA) for direct, instantaneous test generation. ReqOrder (Ziyuan et al., 2007) and ParaOrder (Wang et al., 2008) efficiently support input/output, variable, and overlapping interaction strengths.
Building on this, Study 18 by (Uthaileang et al. 2014) presents a practical pairwise testing methodology that combines Equivalent Partitioning (EP) with combinatorial test generation to manage systems with large input spaces. Study 26 by (Gupta et al. 2017) introduces N-IPO (Novel-In-Parameter-Order) algorithms that integrate boundary value analysis by leveraging the mathematical properties of Fibonacci-based value analysis.
These studies demonstrate the flexibility of OPAT in supporting dynamic and domain-specific testing applications. As summarized in Table 11, these strategies extend OPAT through heuristic simplification, alternative structural models, hybrid optimization, and domain-specific adaptations, highlighting the framework's flexibility beyond general-purpose combinatorial interaction testing.
Table 11
| Strategy | Specialization | Strength | Limitation | Application domain |
|---|---|---|---|---|
| S2 | Algebraic combinatorial design using MOLS and OAs | Direct computation and mathematical optimality | Domain restrictions and difficulty with complex parameter | Compiler design and hardware module testing |
| IPO-s | Symmetry-based optimization | Improves interaction coverage efficiency | Limited generalization | General combinatorial testing |
| TBGCC | Tree-based interaction modeling | Produces compact test suites | Higher structural complexity | Cost-aware combinatorial testing |
| N-IPO | Fibonacci-based parameter selection | Efficient for large input domains | Assumes representative parameter distribution | Large-scale configurations |
| DEO | Sequence covering array generation | Efficient sequence-based generation | Specialized application scope | Event-driven systems |
| PMC | Source code-driven test generation | Automates test data generation | Language and tool dependent | C/C++ unit testing |
| ReqOrder | Input-output relation combinatorial testing | Targeted coverage and requirement mapping | It assumes factors are independent and free of forbidden tuple | Object-oriented systems and class cluster interaction |
| ParaOrder | Generalized variable strength combinatorial testing | Its support overlapping and non-disjoint interaction requirement | Susceptible to local optima | Complex software architectures |
| S18 | Pairwise testing integrated with equivalent partitioning and boundary value analysis | Input space reduction and high practical utility | Case-study focus | Web-based application testing |
| S26 | Novel In-Parameter-Order with Fibonacci sampling | Fibonacci sequence sampling/ordering and multi-layered trial run strategy | Larger execution time | Phased software testing |
Heuristic and application-specific variants of OPAT strategies.
5 Discussion and future research directions
The systematic literature review (SLR) identified 40 primary studies and revealed a progressive evolution of One-Parameter-at-a-Time (OPAT) combinatorial interaction testing (CIT) from deterministic construction toward optimization-driven and application-oriented approaches. Based on the technical mechanisms and principal contributions reported in the selected studies, this study presents a five-category taxonomy: deterministic refinements; scaling and distribution; advanced feature-enhanced strategies; metaheuristic/hybrid strategies; and specialized adaptations. The taxonomy provides a structural basis for examining how OPAT strategies have addressed limitations in test suite size, generation efficiency, scalability, constraint handling, and application-specific requirements.
The temporal distribution of the reviewed studies reveals a historical peak in 2008, representing an important transition in the development of CIT. Prior to 2008, the state of the art was dominated by foundational pairwise (t = 2) testing strategies, such as the In-Parameter-Order (IPO) framework. The 2008 peak marks the transition toward multi-way (t ā„ 3) interaction strengths, driven by algorithms such as IPOG, IPOG-D, and MIPOG, which introduced structural optimizations to mitigate combinatorial explosion in high-dimensional parameter spaces.
Evaluating these strategies requires balancing test suite compactness against generation speed, as performance claims depend heavily on benchmark configurations and hardware constraints. For example, IPOG-D reduces execution time for high-strength configurations to 3% of standard IPOG but yields 51% larger test suites. Conversely, parallel and distributed approaches such as Multi-Core MIPOG (MC-MIPOG) and Grid-based MIPOG (G_MIPOG) achieve near-linear and algorithmic speedups, respectively, at higher interaction strengths (t ā„ 4), although multi-core variants suffer from initialization overhead at lower strengths (t = 2).
Metaheuristic approaches illustrate this trade-off particularly clearly. The enhanced Harmony Search algorithm (eHS), for example, demonstrates substantial improvements in test-suite minimization relative to IPOG under reported experimental conditions, but the iterative search process introduces additional overhead. These findings suggest that no single OPAT strategy simultaneously optimizes all performance objectives. Instead, strategy selection depends on whether the primary requirements are compactness, generation speed, computational scalability, or a combination of these objectives.
Metaheuristic search strategies represent a major advancement in test suite minimization, but their distribution across optimization families remains heavily unbalanced. Harmony Search is particularly dominant among the reviewed approaches, appearing in HCATS, OPAT-HS, OPATHS, PWiseHA, elPOG, PHOSC, SCHOP, and eHS, leaving alternative search frameworks largely unexplored. While metaheuristics consistently achieve superior minimization compared to greedy deterministic baselines, future work should incorporate underutilized families such as Particle Swarm Optimization, Tabu Search, and hyper-heuristics to improve convergence speeds.
The reviewed studies also demonstrate increasing attention to practical features that are not adequately addressed by basic OPAT construction. Constraint handling, variable strength testing, and test-case seeding enable generated suites to better represent realistic system configurations. These capabilities are particularly important because unconstrained covering arrays may contain combinations that are valid or meaningless in real-world applications.
However, the industrial applicability of existing OPAT strategies remains limited, as several approaches have been evaluated primarily using synthetic covering-array benchmarks rather than realistic industrial configurations. Consequently, improvements demonstrated on benchmark arrays do not necessarily establish effectiveness in operational testing environments, where constraints, heterogeneous parameter domains, evolving configurations, and domain-specific requirements may substantially affect algorithm performance. Future evaluations should therefore include realistic industrial configurations alongside standard benchmark suites.
A further finding of this review concerns the difficulty of establishing reliable cross-study performance comparisons. Although the primary studies frequently report test-suite size and generation time, these measurements are generally obtained under different experimental conditions, including hardware configuration, implementation environments, benchmark characteristics, interaction strengths, and parameter settings. Consequently, numerical results reported by different studies should not be interpreted as evidence of a universal performance ranking without controlling for these differences.
The reported results from representative studies illustrate this issue. For example, the quantitative results reported for IPOG and IPOG-D demonstrate a clear trade-off between test-suite size and generation time under the experimental environment used by the original authors. Similarly, the reported comparison between MIPOG and IPOG shows that MIPOG's relative advantage varies across parameter-domain sizes and performance measures. These findings provide useful evidence about the behavior of the strategies within their respective experimental settings, but they should not be interpreted as controlled comparisons across the broader OPAT literature.
The problem is further compounded by the limited availability of executable implementations, source code, and standardized experimental configurations. Without accessible implementations, it is difficult to independently reproduce the reported experiments or to execute competing strategies under identical conditions. Therefore, the limited reproducibility of existing OPAT studies is not merely a methodological limitation of this SLR; it constitutes a substantive research gap in the OPAT field. The current evidence base supports systematic identification of algorithmic mechanisms and qualitative characterization of performance trade-offs, but it does not consistently support statistically controlled cross-paper benchmarking.
These findings also change how comparative claims in OPAT research should be interpreted. Rather than treating reported test-suite sizes or execution times from separate studies as directly interchangeable measurements, future evaluations should distinguish between within-study empirical comparisons and across-study evidence synthesis. Controlled benchmarking should use common hardware and software environments, identical benchmark configurations, clearly reported parameter settings, and standardized performance measures. Where stochastic algorithms are evaluated, repeated independent runs and appropriate statistical tests should be reported.
5.1 Future research directions
The synthesis of existing OPAT strategies highlights several promising research directions that emerge for advancing combinatorial testing.
i. Toward Unified Optimization Strategies: current research largely treats scalability, optimality, and feature integration as separate concerns. Future work should focus on developing unified strategies that simultaneously balance these objectives, potentially through adaptive or multi-objective optimization models that dynamically adjust strategy behavior based on problem characteristics.
ii. Beyond Harmony Search Algorithm: the dominance of Harmony Search in current metaheuristic OPAT strategies indicates a lack of exploration of alternative optimization techniques. Future research should investigate diverse metaheuristics and hybrid models, including Particle Swarm Optimization, Tabu Search, and hyper-heuristic approaches, to improve robustness and reduce dependence on algorithm-specific tuning.
iii. Intelligent and Adaptive Test Generation: emerging trends in artificial intelligence present opportunities to enhance OPAT strategies with learning-based adaptation. Future systems could leverage historical testing data, fault patterns, or runtime feedback to dynamically guide parameter selection, prioritize critical interactions, and improve overall testing effectiveness.
iv. Greater attention should be given to reproducible benchmarking through publicly available source code, executable implementations, benchmark configurations, and generated test suites.
v. End-to-End Integration and Industrial Validation: despite the availability of tools such as ACTS, a gap persists between academic advancements and industrial adoption. Future research should focus on integrating OPAT-based test generation into automated testing pipelines, including execution, monitoring, and result analysis. Additionally, large-scale empirical studies on real-world systems are needed to validate the effectiveness and scalability of advanced OPAT strategies for fault detection.
5.2 Threats to validity
Although this systematic literature review was designed and conducted following a rigorous, transparent methodology to minimize bias and maximize coverage, several threats to its construct, internal, and external validity must be addressed:
i. Study Selection and Screening Validity: to mitigate selection bias and prevent the accidental omission of relevant publications, a multi-stage screening and selection protocol was established. Two reviewers independently screened the titles and abstracts of all retrieved studies against the inclusion and exclusion criteria and subsequently conducted independent full-text evaluations of the candidate papers. Any screening disagreements were resolved through formal consensus-building discussions, with a third reviewer acting as an arbitrator to make final determinations. To quantitatively measure the reliability of this screening process, Cohen's Kappa (k) was calculated prior to consensus discussions, yielding k = 0.83 for the title and abstract screening phase and k = 0.89 for the full-text eligibility phase, indicating substantial to almost perfect inter-reviewer agreement.
ii. English-Only Publication Restriction: a potential external threat to the comprehensiveness of our study pool is the language restriction in our eligibility criteria, which limits our primary studies strictly to those published in English.
iii. Exclusion of Gray Literature: our search strategy focused strictly on peer-reviewed academic journal papers and conference proceedings. This high-quality threshold successfully filtered out conceptually weak or unverified publications.
6 Conclusion
This systematic literature review provides a comprehensive synthesis of One-Parameter-at-a-Time (OPAT) strategies for combinatorial interaction testing (CIT), based on 40 primary studies published between 2000 and 2026. The review establishes a five-category taxonomy that tracks OPAT's evolution from early deterministic foundations to parallel/distributed scaling, feature-enhanced handling (constraints, seeding, and variable strengths), metaheuristic/hybrid optimization models, and domain-specific adaptations. The findings indicate persistent trade-offs among test suite compactness, generation time, memory consumption, scalability, and interaction strength. Deterministic and distributed approaches primarily address computational and scalability challenges, whereas metaheuristic and hybrid approaches can improve test-suite minimization but generally incur greater computation cost.
An important finding of this review is that the quantitative performance results reported across OPAT studies cannot generally be combined to establish a reliable universal ranking of strategies. Differences in hardware, implementation environments, benchmark configurations, parameter settings, interaction strengths, and experimental procedures limit direct cross-study comparison. Moreover, the limited availability of publicly accessible source code and executable implementations restricts independent reproduction and controlled head-to-head evaluation. Thus, the reported quantitative results provide valuable evidence of within-study performance but should not be interpreted as a definitive cross-study comparison of OPAT strategies.
These findings highlight benchmarking and reproducibility as important methodological gaps in OPAT research. Future research should provide publicly accessible implementations, complete experimental configurations, benchmark instances, generated test suites, parameter settings, and reproducible execution environments to facilitate independent replication and controlled evaluation. Further research should also explore underutilized metaheuristics and hybrid optimization approaches, scalable, parallel, and distributed OPAT frameworks, adaptive AI-driven test generation, and validation on large-scale industrial systems. Overall, advancing OPAT-based CIT requires not only more effective optimization mechanisms but also standardized, transparent, and reproducible empirical evaluation practices.
Statements
Author contributions
MY: Formal analysis, Writing ā original draft, Software, Project administration, Supervision, Methodology, Investigation, Conceptualization, Validation, Data curation, Writing ā review & editing. AH: Conceptualization, Resources, Validation, Funding acquisition, Writing ā review & editing, Supervision, Writing ā original draft. HM: Methodology, Writing ā original draft, Validation, Data curation, Formal analysis. SA: Writing ā review & editing, Methodology, Formal analysis, Data curation, Validation. UU: Writing ā original draft, Validation, Formal analysis, Visualization. DA: Formal analysis, Data curation, Writing ā original draft, Validation. IG: Data curation, Writing ā review & editing, Validation, Formal analysis.
Funding
The author(s) declared that financial support was received for this work and/or its publication. This work was supported by the Universiti Teknologi PETRONAS Foundation Fundamental Research under Grant (YUTPR-FRG: 015LC0-604).
Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declared that Generative AI was not used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisherās note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
Abdul MananM. S.Abang JawawiD. N.AhmadJ. (2021). āA systematic literature review on test case prioritization in combinatorial testing,ā in ACM international conference proceeding series, 55ā61. doi: 10.1145/3490700.3490710
2
AlsewariA. A. Mu'aza, A. A.RassemT. H.TairanN. M.ShahH.ZamliK. Z. (2018). One-parameter-at-a-time combinatorial testing strategy based on harmony search algorithm OPAT-HS. Adv. Sci. Lett.24, 7273ā7277. doi: 10.1166/asl.2018.12927
3
Aminu MuazuA.Aminu MuazuA. (2018). āOne-parameter-at-a-time combinatorial testing strategy based on harmony search algorithm supporting mixed covering array mathematical notation (OPATHS),ā in 1st international conference on information technology in education & development (ITED) (At: Abuja), 64ā70.
4
AminuA. M.HashimA. S. (2024). Comparative analysis of IPOG strategy variants for enhanced combinatorial T-way testing efficiency. UMYU Sci.3, 141ā150. doi: 10.56919/usci.2433.016
5
CalvagnaA.GargantiniA. (2009). āIPO-s: incremental generation of combinatorial interaction test data based on symmetries of covering arrays,ā in IEEE international conference on software testing, verification, and validation workshops, 10ā18.
6
CuiY.LiL.YaoS. (2009). āA new strategy for pairwise test case generation,ā in 3rd international symposium on intelligent information technology application, IITA 2009, Vol. 3, 303ā306. doi: 10.1109/IITA.2009.416
7
EbertC.BajajD.WeyrichM. (2022). Testing software systems. IEEE Softw.39, 8ā17. doi: 10.1109/MS.2022.3166755
8
FadhilH. M.AbdullahM. N.YounisM. I. (2022). Combinatorial testing approaches : a systematic review. Iraqi J. Comput. Commun. Control Syst. Eng.22, 60ā79. doi: 10.33103/uot.ijccce.22.4.6
9
FarinaM.FedorovskayaA.PolivtsevE.SucciG. (2022). Software engineering and filmmaking: a literature review. Front. Comput. Sci. 4, 1ā13. doi: 10.3389/fcomp.2022.884533
10
ForbesM.LawrenceJ.LeiY.KackerR. N.KuhnD. R. (2008). Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Natl. Inst. Stand. Technol. 113:287.
11
GaoS.LvJ.DuB.JiangY.MaS. (2014a). āGeneral optimization strategies for refining the In-Parameter-order algorithm,ā in Proceedings - international conference on quality software, 21ā26. doi: 10.1109/QSIC.2014.15
12
GaoS.DuB.JiangY.LvJ.MaS. (2014b). āAn efficient algorithm for pairwise test case generation in presence of constraints,ā in The 2014 2nd international conference on systems and informatics (ICSAI 2014), November (IEEE), 406ā410.
13
GaoS. W.LvJ. H.DuB. L.ColbournC. J.MaS. L. (2015). Balancing frequencies and fault detection in the in-parameter-order algorithm. J. Comput. Sci. Technol.30, 957ā968. doi: 10.1007/s11390-015-1574-6
14
GuptaD.RanaA. (2013). āFibonacci driven novel test generation strategy for constrained testing,ā in Proceedings of the 2013 3rd IEEE international advance computing conference, IACC 2013, 1475ā1478. doi: 10.1109/IAdCC.2013.6514444
15
GuptaD.RanaA.TyagiS. (2017). āSequence generation of test case using pairwise approach methodology,ā in Proceedings of advances in intelligent systems and computing, September (Springer), 79ā85. doi: 10.1007/978-981-10-3773-3_9
16
HemmatA.SharbafM.Kolahdouz-RahimiS.LanoK.TehraniS. Y. (2025). Research directions for using LLM in software requirement engineering: a systematic review. Front. Comput. Sci.7, 1ā18. doi: 10.3389/fcomp.2025.1519437
17
KathiresanG. (2024). A multimodal approach to software quality assurance: integrating static analysis, dynamic testing, and AI-based anomaly detection. Int. J. Innov. Res. Comput. Commun. Eng. 12. doi: 10.15680/IJIRCCE.2024.1202003
18
KitchenhamB.ChartersS. (2007). Guidelines for performing systematic literature reviews in software engineering (Version 2.3, EBSE Technical Report EBSE-2007-01). Keele University and Durham University
19
KlaibM. F.MuthuramanS.AhmadN.SidekR. (2010). Tree based test case generation and cost calculation strategy for uniform parametric pairwise testing. J. Comput. Sci.6, 425ā430. doi: 10.3844/jcssp.2010.542.547
20
LeiY.KackerR.KuhnD. R.OkunV.LawrenceJ. (2007). āIPOG: a general strategy for T-way software testing,ā in Proceedings of the international symposium and workshop on engineering of computer based systems, 549ā556. doi: 10.1109/ECBS.2007.47
21
LeiY.KackerR.KuhnD. R.OkunV.LawrenceJ. (2008). IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing. Softw. Test. Verif. Reliab.18, 125ā148. doi: 10.1002/stvr.381
22
LopuhaO.TsiutsiuraS.PoplavskyiO.LysytsinO.BondarO.KrukP. (2023). āTest design methodology for software verification,ā in 2023 IEEE international conference on smart information systems and technologies (SIST) (IEEE), 241ā245. doi: 10.1109/SIST58284.2023.10223573
23
MaityS.NayakA. (2005). āImproved test generation algorithms for pair-wise testing,ā in Proceedings - international symposium on software reliability engineering, ISSRE 2005, 235ā244. doi: 10.1109/ISSRE.2005.23
24
MercanH.JaveedA.YilmazC. (2020). āFlexible combinatorial interaction testing,ā in IEEE transactions on software engineering, Vol. 48 (IEEE), 1030ā1066. doi: 10.1109/TSE.2020.3010317
25
MuazuA. A.Danjuma MaiwadaU. (2020). āPWiseHA: application of harmony search algorithm for test suites generation using pairwise techniques,ā in International journal of computer and information technology, Vol. 9 (Number 4). doi: 10.24203/ijcit.v9i4.23
26
MuazuA. A.HashimA. S. (2025). āEnhancing harmony search metaheuristic algorithm for coverage efficiency, test suite reduction, and running time in combinatorial interaction testing,ā in IEEE access, Vol. 13 (IEEE), 110828ā110852. doi: 10.1109/ACCESS.2025.3583176
27
MuazuA. A.HashimA. S.AudiU. I. Aliyu Y YahayaM. S. (2024c). Combinatorial interaction testing for T-way test case generation: a scoping review of the perspective features. Zhongguo Kuangye Daxue Xuebao, 29, 1ā21.
28
MuazuA. A.HashimA. S.AudiU. I.MaiwadaU. D. (2024a). āRefining a one-parameter-at-a-time approach using harmony search for optimizing test suite size in combinatorial T-way testing,ā in IEEE access, Vol. 12 (IEEE), 137373ā137398. doi: 10.1109/ACCESS.2024.3463953
29
MuazuA. A.HashimA. S.MaiwadaU. D.Isma'ilaU. A.YakubuM. M.IbrahimM. A. (2024b). Pairwise test case generation with harmony search, one-parameter-at-at-time, seeding, and constraint mechanism integration. Int. J. Electr. Comput. Eng.14, 3137ā3149. doi: 10.11591/ijece.v14i3.pp3137-3149
30
MuazuA. A.HashimA. S.MaiwadaU. D.MuppidiA. (2023a). Enhanced version of seeding and constraint support in IPOG strategy for variable strength interaction t-way testing. Malays. J. Comput. Sci.36, 381ā403. doi: 10.22452/mjcs.vol36no4.4
31
MuazuA. A.HashimA. S.SarlanA. (2022). āReview of nature inspired metaheuristic algorithm selection for combinatorial t-way testing,ā in IEEE access, Vol. 10 (IEEE), 27404ā27431. doi: 10.1109/ACCESS.2022.3157400
32
MuazuA. A.HashimA. S.SarlanA.AbdullahiM. (2023b). SCIPOG: seeding and constraint support in IPOG strategy for combinatorial t-way testing to generate optimum test cases. J. King Saud Univ. Comput. Inform. Sc.35, 185ā201. doi: 10.1016/j.jksuci.2022.11.010
33
MuazuA. A.MuazuA. A. (2018). āDesign of a harmony search algorithm based on covering array T-way testing strategy,ā in 1st international conference on information technology in education & development (ITED), 33ā38.
34
NiuX.NieC.LeiJ. Y.LeungH.WangX. (2020a). āIdentifying failure-causing schemas in the presence of multiple faults,ā in IEEE transactions on software engineering, Vol. 46 (IEEE), 141ā162. doi: 10.1109/TSE.2018.2844259
35
NiuX.NieC.LeungH.LeiY.WangX.XuJ.et al. (2020b). āAn interleaving approach to combinatorial testing and failure-inducing interaction identification,ā in IEEE transactions on software engineering, Vol. 46 (IEEE), 584ā615. doi: 10.1109/TSE.2018.2865772
36
Prasad MudarakolaL.SuneethaP.ShahidaS.SomasekharaK. (2014). An efficient and improved in parameter strategy (IIPO) for generating the pairwise test cases. Int. J. Sci. Eng. Technol. Res.3, 9375ā9380.
37
RamliN.OthmanR. R.HendradiR.ZakariaH. L.IsmailI.Mohd ZakiN. A. N. A.et al. (2024). Input-output based relations T-way test suite generation strategy based on ant colony optimization algorithm (iTTSGA). J. Adv. Res. Appl. Sci. Eng. Technol.63, 1ā16. doi: 10.37934/araset.63.1.116
38
RaoC.LiN.LeiY.KackerR. N.KuhnD. R.GuoJ. (2019). āUsing parameter mapping to avoid forbidden tuples in a covering array,ā in Proceedings - 2019 IEEE 12th international conference on software testing, verification and validation workshops, ICSTW 2019, 251ā259.
39
ShwethaM. S. (2017). āRecast IPOG-D algorithm with constraint handling for combinatorial testing,ā in Proceedings - 2017 international conference on recent advances in electronics and communication technology, ICRAECT 2017 (Bangalore: IEEE), 179ā185. doi: 10.1109/ICRAECT.2017.62
40
SohZ. H. C.AbdullahS. A. C.ZamliK. Z. (2014). A distributed t-way test suite generation using āone-parameter-at-a-timeā approach. Int. J. Adv. Soft Comput. Appl.5, 91ā103.
41
TaiK. C.LeiY. (2002). āA test generation strategy for pairwise testing,ā in IEEE transactions on software engineering, Vol. 28 (IEEE), 109ā111. doi: 10.1109/32.979992
42
TranH. V.TungL. N.HungP. N. (2022). āA pairwise based method for automated test data generation for C/C++ projects,ā in Proceedings - 2022 RIVF international conference on computing and communication technologies, RIVF 2022, 352ā357. doi: 10.1109/RIVF55975.2022.10013824
43
UthaileangW.KiattisinS.LeelasantithamA. (2014). āPair-wise testing applied with online registration,ā in JICTEE 2014 : the fourth joint international conference on information and communication technology, electronic and electrical engineering. March 5-8, 2014 (Chiang Rai: IEEE). doi: 10.1109/JICTEE.2014.6804077
44
VillothJ. P.ZivkovicM.ZivkovicT.Abdel-salamM.HammadM.JovanovicL.et al. (2025). Two-tier deep and machine learning approach optimized by adaptive multi-population firefly algorithm for software defects prediction. Neurocomputing630:129695. doi: 10.1016/j.neucom.2025.129695
45
WangL. (2017). āA reduction method of combinatorial testing based on input parameter constraints,ā in 2017 IEEE 9th international conference on communication software and networks (ICCSN), 1482ā1485.
46
WangZ.XuB.NieC. (2008). āGreedy heuristic algorithms to generate variable strength combinatorial test suite,ā in Proceedings - international conference on quality software, 155ā162.
47
YahayaM. S.HashimA. S.TijjaniS.SalisuJ. A.UsmanF. S.ElhuseiniG. (2026). āHOS: a hybrid metaheuristic one-parameter-at-a-time strategy for T-way test case generation,ā in IEEE access, Vol. 14 (IEEE), 30387ā30410. doi: 10.1109/ACCESS.2026.3666942
48
YahayaM. S.HashimA. S. B.Oluwagbemiga BalogunA.Aminu MuazuA.Sabo UsmanF.Adamu AliyuD.et al. (2025). āExploration and exploitation mechanism in pairwise test case generation: a systematic literature review,ā in IEEE access, Vol. 13 (IEEE), 82342ā82377. doi: 10.1109/ACCESS.2025.3566163
49
YounisM. I. (2020a). DEO: a dynamic event order strategy for t-way sequence covering array test data generation. Baghdad Sci. J.17:0575. doi: 10.21123/bsj.2020.17.2.0575
50
YounisM. I. (2020b). GAMIPOG: A deterministic genetic multi-parameter-order strategy for the generation of variable strength covering arrays. J. Eng. Sci. Technol.15, 3142ā3162.
51
YounisM. I.ZamliK. Z. (2011). MIPOG - An Efficient t-Way Minimization Strategy for Combinatorial Testing.Int. J. Comput. Theory Eng. 3, 388ā397. doi: 10.7763/IJCTE.2011.V3.337
52
YounisM. I.ZamliK. Z.IsaN. A. M. (2008a). āMIPOG-Modification of the IPOG strategy for T-way software testing,ā in Proceedings of the distributed frameworks and applications (DFmA 2008) (Penang, Malaysia), 1ā6.
53
YounisM. I.ZamliK. Z.IsaN. A. M. (2008b). āA strategy for grid based T-way test data generation,ā in 2008 First international conference on distributed framework and application, 73ā78. doi: 10.1109/ICDFMA.2008.4784416
54
YuL.LeiY.KackerR. N.KuhnD. R. (2013a). āACTS: a combinatorial test generation tool,ā in Proceedings - IEEE 6th international conference on software testing, verification and validation, ICST 2013, 370ā375. doi: 10.1109/ICST.2013.52
55
YuL.LeiY.NourozborazjanyM.KackerR. N.KuhnD. R. (2013b). āAn efficient algorithm for constraint handling in combinatorial test generation,ā in Proceedings - IEEE 6th international conference on software testing, verification and validation, ICST 2013, 242ā251. doi: 10.1109/ICST.2013.35
56
ZhangX.ShenW.LiangZ.CuiL.WangY. (2024). āResearch and application of automated testing technology for data security vulnerabilities,ā in 2024 4th international conference on mobile networks and wireless communications (ICMNWC) (Tumkuru: IEEE), 01ā05. doi: 10.1109/ICMNWC63764.2024.10872029
57
ZivkovicT.ZivkovicM. (2026). Metaheuristic optimization of ML for software testing defect prediction: a systematic review and critical analysis of methods, datasets, and research gaps. Evol. Intell.19:103. doi: 10.1007/s12065-026-01225-z
58
ZiyuanW.ChanghaiN.BaowenX. (2007). āGenerating combinatorial test suite for interaction relationship,ā in Fourth international workshop on software quality assurance: in conjunction with the 6th ESEC/FSE Joint Meeting, 55ā61. doi: 10.1145/1295074.1295085
Appendix
Table A1
| IDs | Literature | Quality assessment | Taxonomy classification | ||||||
|---|---|---|---|---|---|---|---|---|---|
| References | Strategy | t-way | QA1 | QA2 | QA3 | QA4 | Total score | ||
| S1 | Tai and Lei (2002) | IPO | t = 2 | 1 | 1 | 1 | 1 | 4 | Foundational |
| S2 | Maity and Nayak (2005) | ā | t = 2 | 1 | 1 | 1 | 1 | 4 | Specialized OPAT strategies |
| S3 | Lei et al. (2007) | IPOG | t ā„ 7 | 1 | 1 | 1 | 1 | 4 | Deterministic refinement |
| S4 | Ziyuan et al. (2007) | ReqOrder | t ⤠3 | 1 | 0.5 | 1 | 1 | 3.5 | Specialized OPAT strategies |
| S5 | Lei et al. (2008) | IPOG-D | t ⤠6 | 1 | 1 | 1 | 1 | 4 | Deterministic refinement |
| S6 | Forbes et al. (2008) | IPOG-F | t ⤠3 | 1 | 1 | 1 | 0.5 | 3.5 | Deterministic refinement |
| S7 | Younis et a. (2008a) | MIPOG | t ⤠6 | 1 | 1 | 1 | 1 | 4 | Deterministic refinement |
| S8 | Wang et al. (2008) | ParaOrder | t ⤠3 | 1 | 1 | 0.5 | 0.5 | 3 | Specialized OPAT strategies |
| S9 | Younis et al. (2008b) | G_MIPOG | t ⤠6 | 1 | 1 | 1 | 0.5 | 3.5 | Scaling and distribution strategies |
| S10 | Calvagna and Gargantini (2009) | IPO-s | t = 2 | 1 | 1 | 0.5 | 1 | 3.5 | Specialized OPAT strategies |
| S11 | Cui et al. (2009) | VIPO | t = 2 | 1 | 1 | 1 | 1 | 4 | Deterministic refinement |
| S12 | Klaib et al. (2010) | TBGCC | t = 2 | 1 | 0.5 | 0.5 | 0.5 | 2.5 | Specialized OPAT strategies |
| S13 | Younis and Zamli (2011) | MC-MIPOG | t ā„ 12 | 1 | 1 | 1 | 1 | 4 | Scaling and distribution strategies |
| S14 | Yu et al. (2013b) | IPOG-C | t ⤠3 | 1 | 1 | 1 | 1 | 4 | Feature-enhanced strategies |
| S15 | Gupta and Rana (2013) | N-IPO | t = 2 | 1 | 1 | 0.5 | 0 | 2.5 | Specialized OPAT strategies |
| S16 | Yu et al. (2013a) | ACTS | t ⤠6 | 1 | 1 | 0 | 0 | 2 | Feature-enhanced strategies |
| S17 | Soh et al. (2014) | TS_IPO | t ⤠6 | 1 | 0.5 | 1 | 1 | 3.5 | Scaling and distribution strategies |
| S18 | Uthaileang et al. (2014) | ā | t = 2 | 1 | 1 | 1 | 0.5 | 3.5 | Specialized OPAT strategies |
| S19 | Prasad Mudarakola et al. (2014) | IIPO | t = 2 | 1 | 1 | 0.5 | 0.5 | 3 | Deterministic refinement |
| S20 | Gao et al. (2014a) | t ⤠5 | 1 | 1 | 1 | 0.5 | 3.5 | Deterministic refinement | |
| S21 | Gao et al. (2014b) | IPO-SAT | t = 2 | 1 | 1 | 1 | 0.5 | 3.5 | Feature-enhanced strategies |
| S22 | Gao et al. (2015) | ā | t ⤠6 | 1 | 1 | 1 | 0.5 | 3.5 | Deterministic refinement |
| S23 | Wang (2017) | ā | t = 2 | 1 | 0.5 | 0.5 | 0 | 2 | Feature-enhanced strategies |
| S24 | Shwetha (2017) | RIPOG-D | t ⤠6 | 1 | 0.5 | 1 | 0.5 | 3 | Feature-enhanced strategies |
| S25 | Aminu Muazu and Aminu Muazu (2018) | HCATS | t = 2 | 1 | 1 | 0.5 | 0 | 2.5 | Metaheuristic and hybrid strategies |
| S26 | Gupta et al. (2017) | ā | t = 2 | 1 | 0.5 | 0.5 | 0 | 2 | Specialized OPAT strategies |
| S27 | Alsewari et al. (2018) | OPAT-HS | t = 3 | 1 | 1 | 1 | 1 | 4 | Metaheuristic and hybrid strategies |
| S28 | Muazu and Muazu (2018) | OPATHS | t ⤠3 | 1 | 0.5 | 1 | 1 | 3.5 | Metaheuristic and hybrid strategies |
| S29 | Rao et al. (2019) | ā | t ⤠6 | 1 | 1 | 0.5 | 0.5 | 3 | Feature-enhanced strategies |
| S30 | Younis (2020b) | GAMIPOG | t ā„ 7 | 1 | 1 | 1 | 1 | 4 | Metaheuristic and hybrid strategies |
| S31 | Muazu and Danjuma Maiwada (2020) | PWiseHA | t ⤠4 | 1 | 1 | 1 | 1 | 4 | Metaheuristic and hybrid strategies |
| S32 | Younis (2020a) | DEO | t = 7 | 1 | 1 | 1 | 1 | 4 | Specialized OPAT Strategies |
| S33 | Tran et al. (2022) | PMC | t = 2 | 1 | 0.5 | 1 | 0.5 | 3 | Specialized OPAT strategies |
| S34 | Muazu et al. (2023b) | SCIPOG | t ā„ 7 | 1 | 1 | 1 | 1 | 4 | Feature-enhanced strategies |
| S35 | Muazu et al. (2023a) | SCIPOG-VS | t ā„ 7 | 1 | 1 | 1 | 1 | 4 | Feature-enhanced strategies |
| S36 | Muazu et al. (2024a) | SCHOP | t ⤠6 | 1 | 1 | 1 | 1 | 4 | Metaheuristic and hybrid strategies |
| S37 | Muazu et al. (2024b) | PHOSC | t = 2 | 1 | 1 | 1 | 1 | 4 | Metaheuristic and hybrid strategies |
| S38 | Aminu and Hashim (2024) | eIPOG | t ⤠6 | 1 | 1 | 1 | 1 | 4 | Metaheuristic and hybrid strategies |
| S39 | Muazu and Hashim (2025) | eHS | t ⤠6 | 1 | 1 | 1 | 1 | 4 | Metaheuristic and hybrid strategies |
| S40 | Yahaya et al. (2026) | HOS | t = 2 | 1 | 1 | 1 | 1 | 4 | Metaheuristic and hybrid strategies |
Publication trend.
Summary
Keywords
combinatorial interaction testing, metaheuristic optimization, One-Parameter-at-a-Time, software testing, test suite generation, t-way testing
Citation
Yahaya MS, Hashim AS, Mamman H, Abdullahi S, Umer U, Adamu Aliyu D and Godowoli IM (2026) Constructing t-way test suites: a systematic survey and qualitative synthesis of the One-Parameter-at-a-Time approach. Front. Comput. Sci. 8:1946723. doi: 10.3389/fcomp.2026.1946723
Received
23 July 2026
Revised
10 September 2026
Accepted
29 September 2026
Published
15 October 2026
Volume
8 - 2026
Edited by
Martin Shepperd, Brunel University London, United Kingdom
Updates
Copyright
Ā© 2026 Yahaya, Hashim, Mamman, Abdullahi, Umer, Adamu Aliyu and Godowoli.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Muhammad Sabo Yahaya, msy5657@fedpolydaura.edu.ng
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.