Abstract
Introduction:
Microservice architectures introduce complex testing challenges, particularly for grey failures — partial degradation scenarios in which infrastructure signals appear normal while user-facing journeys fail silently. Existing monitoring approaches lack the sensitivity to surface such failures early.
Methods:
We introduce Health Box Testing (HBT), a runtime vitality assurance methodology that deploys synthetic Health Agents to execute representative user journeys against production-like environments. A proof-of-concept evaluation was conducted across 12 controlled grey-failure injection trials using Playwright-based agents in a dedicated test environment.
Results:
HBT detected all 12 injected grey-failure trials (coverage 100%; median MTTD = 70 s, IQR 48.5-86.8 s). The baseline monitoring approach detected 7 of 12 trials (58.3%), with a conditional median MTTD of 397 s (IQR 344.0-404.5 s). A Wilcoxon signed-rank test on the 7 paired trials confirmed HBT was significantly faster (W = 0, p = 0.01562, two-sided), with a median paired reduction of 322 s.
Discussion:
Health Box Testing provides a complementary, toolchain-agnostic framework for continuous vitality assurance that surfaces user-impacting failures earlier than conventional infrastructure monitoring, particularly for grey failures invisible to standard telemetry. Future work includes automated root-cause classification and integration with cognitive-load-aware alerting frameworks.
Introduction
The evolution from monolithic applications to cloud-native microservices has fundamentally altered the definition of software quality. In monolithic architectures, passing unit and integration tests often implied acceptable stability. In distributed systems, quality is shaped by ephemeral infrastructure, probabilistic network behaviour, and runtime dependencies where failure is not an exception but an inevitability (Bass et al., 2015; Richardson, 2018).
Industry practice continues to rely heavily on White Box Testing (structural analysis of internal logic) and Black Box Testing (functional input/output validation) (Nidhra and Dondeti, 2012). These approaches remain essential, but they predominantly operate during the “Build” and “Test” phases of the SDLC. After deployment, assurance often shifts to monitoring and observability tooling, which tends to emphasise infrastructure signals (e.g., CPU, memory, error-rate) rather than end-to-end user-journey success (Majors et al., 2022).
A persistent gap exists between technical health (e.g., a service returning HTTP 200) and perceived quality (e.g., an MFA step that stalls or a checkout flow that becomes unresponsive). This gap is amplified by microservices, where partial degradation may not trigger obvious alarms yet can break critical user journeys.
This paper presents Health Box Testing (HBT) as a continuous runtime assurance method that actively executes synthetic user journeys and interprets outcomes as health “vital signs” (heartbeat, temperature, blood pressure/behaviour). Rather than positioning HBT as a replacement for observability or conventional testing, the contribution is to formalise and operationalise an integrated vitality-assurance workflow that extends assurance into production-like runtime contexts.
Existing observability and monitoring approaches predominantly focus on infrastructure-level telemetry and service-level indicators, often leaving a methodological gap between technical health signals and end-to-end user-journey correctness. This study addresses that gap by introducing Health Box Testing as a runtime assurance methodology that operationalises synthetic journey execution, vitality scoring, and glanceable health visualisation within a unified framework (Beyer et al., 2016; Majors et al., 2022; Waseem et al., 2021).
This paper makes four primary scientific contributions to the study of runtime assurance in distributed systems. First, it conceptualises Health Box Testing (HBT) as a continuous runtime testing paradigm that integrates synthetic monitoring, journey-level validation, and operational observability into a integrated workflow. Second, the study introduces a vitality-oriented scoring model that operationalises service health using composite indicators derived from user-journey success, latency distributions, and behavioural failure signals, aligning runtime assurance with user-perceived reliability objectives. Third, it proposes a correlation-based tagging and data-isolation protocol that enables continuous synthetic execution in production-like environments without contaminating operational datasets. Finally, the paper demonstrates the feasibility of the approach through a proof-of-concept implementation in a microservice identity-management journey, providing reproducible execution artefacts and metric derivation procedures to support independent validation. These contributions extend existing observability and monitoring approaches by operationalising journey-level vitality assurance as a formalised runtime-assurance workflow rather than a collection of isolated monitoring practices (Beyer et al., 2016; Majors et al., 2022).
What is new vs. what exists
HBT contributes a vitality-oriented runtime assurance artefact that complements (rather than replaces) existing testing and observability practices. In contrast to conventional health checks (service-local liveness/readiness) and synthetic monitoring (periodic scripted transactions reported as independent signals), HBT introduces a composite vitality score and triage bands that unify availability, latency, error behaviour, and resource stress into a single operational indicator.
(1) A clinically inspired composite scoring model with configurable weights and thresholds; (2) journey-level grey-failure detection demonstrated on an OIDC+MFA identity flow; (3) controlled fault-injection evaluation across 12 trials with comparative analysis against baseline alerting; and (4) multi-layer evidence including protocol-level contract stability checks and end-to-end regression results.
Related work and theoretical foundations
White box and black box testing
White Box testing requires internal knowledge of the code structure, whereas Black Box testing treats the system as opaque and validates behaviour through observable inputs and outputs (Nidhra and Dondeti, 2012). Both approaches can be executed extensively before release; however, neither inherently accounts for runtime variability that emerges from real network paths, dependency behaviour, and operational configuration drift.
Continuous delivery and runtime assurance
Continuous Delivery increases release frequency and reduces batch size, enabling rapid iteration but also increasing the need for ongoing validation under realistic operating conditions (Humble and Farley, 2010). As deployment becomes routine, the assurance boundary naturally shifts from “pre-release confidence” toward “continuous verification of user-impacting behaviour”.
Testing in production and resilience thinking
The argument that production is the most faithful environment has been explored empirically in studies of testing in production (Schermann et al., 2018; Schermann et al., 2016; Waseem et al., 2021). In parallel, Chaos Engineering formalises controlled experiments to build confidence in system behaviour under failure conditions (Basiri et al., 2016). While powerful, many teams view routine production experimentation as risky—especially in regulated contexts. HBT adopts the realism motivation of these approaches while focusing on a safer routine mechanism. It relies on synthetic journeys with strict tagging and data isolation.
Observability and the UX gap
Observability frameworks aim to make systems diagnosable by collecting and correlating telemetry (Majors et al., 2022). Yet, a system can be richly observable while still failing critical user journeys. HBT therefore treats observability as necessary but not sufficient, adding continuous journey execution as a direct measure of user-perceived vitality Runtime feedback loops can also help bridge operational metrics and developer awareness (Cito et al., 2015).
Cognitive load in operations
During incidents, engineers frequently navigate multiple dashboards, logs, and traces. Cognitive Load Theory argues that working memory is limited, particularly under time pressure (Sweller, 2011). HBT’s medically-inspired interface aims to improve glanceability—rapid pattern recognition of “healthy vs unhealthy”—thereby reducing extraneous cognitive load and supporting faster triage. This rationale aligns with practical DevOps perspectives on managing cognitive load in team structures and operational flow (Skelton and Pais, 2019).
Microservices monitoring and testing in practice
Microservice monitoring refers to the instrumentation and analysis of distributed services using metrics, logs, and traces to support diagnosis and operational decision-making across service boundaries (Majors et al., 2022; Li et al., 2022). Recent practitioner-focused empirical work highlights that microservices increase both monitoring and testing complexity, making end-to-end and contract-style checks more prominent in practice. Waseem et al. report that practitioners commonly rely on API gateways and observe a growing reliance on E2E and contract-related testing to manage distributed change, while also acknowledging gaps in dedicated solutions for monitoring and testing challenges (Waseem et al., 2021). This supports the paper’s premise that runtime assurance must bridge service-level telemetry and user-perceived correctness, especially under frequent release and polyglot service evolution (Giamattei et al., 2024; Richardson, 2018).
Synthetic monitoring, SRE, and “monitoring-as-code”
Site Reliability Engineering (SRE) emphasises measuring what users experience using service level indicators (SLIs) and objectives (SLOs), and designing monitoring that is actionable and tied to user impact rather than raw infrastructure noise (Beyer et al., 2016). Synthetic monitoring executes scripted user journeys at regular intervals to detect user-visible failures and degradations (Richmond, 2024). Recent practitioner literature argues that synthetic monitoring and end-to-end (E2E) testing are technically adjacent: both encode user workflows as executable specifications, differing mainly in execution context (CI vs. production) and frequency. This supports treating Playwright-style journeys as reusable ‘monitoring-as-code’ artefacts that can provide both pre-release validation and post-deploy vitality monitoring, provided side-effects and data-contamination risks are controlled (Nygard, 2018; Majors et al., 2022).
Testing in production, live testing, and continuous experimentation
Research on testing in production and continuous experimentation highlights that pre-release test stages can miss environment-specific and interaction-driven failures. Schermann et al. (2018) report empirical evidence that organisations increasingly rely on controlled live experiments and canary-style strategies to validate behaviour under real conditions. Bifrost operationalises multi-phase live testing strategies for continuous deployment and reports low overhead in typical scenarios (Schermann et al., 2016). These findings motivate HBT’s emphasis on controlled, repeatable synthetic journeys in production-like environments to surface grey failures earlier than infrastructure-only signals.
Observability, tracing, and microservice monitoring tooling
Observability for microservices typically combines metrics, logs, and distributed traces to support end-to-end diagnosis across service boundaries (Majors et al., 2022). Industrial evidence suggests that organizations adopt tracing widely, but still face gaps in trace usefulness and analysis workflows (Li et al., 2022). A systematic study of grey literature mapped 71 monitoring tools for DevOps and microservices, highlighting feature fragmentation and tool-selection complexity (Giamattei et al., 2024). Transparent tracing approaches aim to reduce instrumentation burden and enable end-to-end correlation, supporting the need for explicit correlation identifiers and consistent propagation in runtime assurance workflows (Santana et al., 2019). Recent work provides a systematic comparison of open tracing tools and highlights trade-offs across feature support, adoption signals, and integration overhead (Janes et al., 2023). In practice, many organizations converge on open standards such as OpenTelemetry to normalise telemetry data models and context propagation across heterogeneous services (OpenTelemetry, 2025).
Runtime assurance gap synthesis
Although observability engineering, SRE, and distributed tracing have substantially improved infrastructure-level diagnostics in microservice systems (Sigelman et al., 2010; Beyer et al., 2016; Majors et al., 2022), existing approaches primarily focus on metrics, logs, and traces as passive telemetry streams. Similarly, synthetic monitoring and end-to-end (E2E) testing practices execute user workflows either pre-release or as isolated production probes, but they are rarely formalised as a continuous vitality-assurance methodology (Schermann et al., 2018; Waseem et al., 2021).
Empirical studies further show that practitioners struggle to bridge service-level monitoring with user-perceived correctness, particularly under frequent deployments and evolving service dependencies (Li et al., 2022; Giamattei et al., 2024). While chaos engineering explores resilience under injected failures (Basiri et al., 2016; Owotogbe et al., 2024), it is typically episodic rather than continuously operationalised as a routine vitality signal.
Therefore, a methodological gap persists between:
Infrastructure observability,
Synthetic execution of user journeys,
And operational decision-making under cognitive constraints.
Health Box Testing (HBT) addresses this gap by formalising synthetic journey execution as a structured runtime assurance paradigm, integrating vitality scoring, data isolation, and glanceable visualisation into a unified artefact.
To clarify how vitality-oriented assurance complements established operational practices, Table 1 summarises key strengths and grey-failure blind spots across common monitoring and observability approaches, and positions HBT as a unifying, journey-centred layer.
Table 1
| Approach | Typical signal and what it detects | Strengths | Grey-failure gaps and how HBT complements |
|---|---|---|---|
| Infrastructure threshold alerts (e.g., Prometheus/Grafana) | Resource/utilisation metrics, HTTP error rates, service availability; coarse service-level view. | Low overhead and broad coverage for hard failures; useful for capacity and SLO breaches. | Often misses client-side or dependency-induced degradations where services remain “up” (Huang et al., 2017). HBT adds end-to-end journey checks and a vitality score that reflects perceived user impact. |
| Distributed tracing (Dapper; OpenTelemetry) | Trace spans and latency across services; request-path visibility and bottleneck localisation (Sigelman et al., 2010; OpenTelemetry, 2025). | Strong for root-cause analysis and performance attribution across microservices. | Requires instrumentation, sampling, and expert interpretation; tracing alone does not decide whether user journeys are broken. HBT consumes journey outcomes as primary evidence and can surface degradations even when telemetry looks normal. |
| Log-centric monitoring and anomaly detection | Log events, patterns and statistical anomalies; often post-hoc indicators. | High diagnostic detail; can capture unexpected failure modes. | Noisy and difficult to calibrate; anomalies may not map cleanly to user impact. HBT provides structured, repeatable journey-level signals and aggregates them into an interpretable vitality indicator. |
| Synthetic monitoring and E2E journey checks | Scripted transactions and user journeys executed periodically; direct user-experience proxy. | Directly detects journey arrests and latent regressions; aligns with SRE focus on user experience. | Often implemented as isolated checks without unified scoring or cross-signal interpretation. HBT formalises a multi-signal vitality model (heartbeat/temperature/behaviour) and supports safe execution via synthetic data isolation. |
| SRE SLIs/SLOs and health scoring | Quantitative SLIs (latency, availability, error rate) tracked against SLOs; service-level objectives (Beyer et al., 2016). | Aligns monitoring with user value; supports objective decision-making and error budgets. | SLIs may remain service-centric and miss specific critical journeys. HBT operationalises journey-specific SLIs and complements SRE by providing continuous vitality scoring tied to VUJs. |
| Chaos engineering/fault injection | Controlled experiments that inject faults to reveal resilience and validate safeguards (Basiri et al., 2016). | Exposes hidden dependencies; validates recovery and detection strategies. | Not continuous by itself and requires careful planning. HBT uses controlled fault scenarios to evaluate time-to-detection and provides a continuous assurance layer between experiments. |
Comparison of vitality-oriented assurance (HBT) with established observability and assurance approaches.
Health box testing: definition and scope
Terminology and positioning
We distinguish: (i) health checks—service-local liveness/readiness endpoints; (ii) synthetic monitoring—periodic scripted end-to-end transactions executed by synthetic agents; and (iii) testing in production—experiments executed against live systems. HBT is positioned as synthetic monitoring augmented with vitality scoring, governance and isolation safeguards, rather than ad-hoc production experimentation.
This paper defines Health Box Testing (HBT) as:
A continuous, non-intrusive assurance methodology that evaluates the operational health of a software system in runtime environments by injecting synthetic user behaviours, isolating synthetic data end-to-end, and interpreting system responses through health-oriented metaphors.
HBT differs from passive monitoring by actively generating critical user journeys instead of merely observing existing telemetry. It is designed to detect grey failures: issues that may not manifest as server errors but break the user experience (e.g., client-side regressions, dependency latency, misconfigured redirects, or MFA delivery issues).
Grey-failure typology in HBT
Building on the general notion of grey failures introduced by Huang et al. (2017), we adopt an HBT-specific three-type classification grounded in the journey-level phenomena that runtime synthetic execution can surface. This typology is deliberately small so that operators can recognise the type at a glance from the vitality panel:
G1—dependency latency: external or downstream calls (e.g., SMS gateway, email provider, identity store) slow down without returning 5xx errors. Symptoms manifest as extended journey duration and Temperature degradation while Heartbeat remains nominal.
G2—UI interaction breaks: a user-facing element fails to function as expected (e.g., a non-responsive button, a form that will not advance, a missing focus state). Synthetic agents detect this as a step-level assertion failure that drops Heartbeat to zero even though backend services return 2xx responses.
G3—session or state problems: OIDC redirects resolve incorrectly, tokens fail validation, sessions stall, or MFA challenges are not paired with their issuance. These produce protocol-level state mismatches that pass health-check endpoints but break the end-to-end identity journey.
This typology informs how HBT classifies the controlled fault-injection trials reported in the Results section: every trial in Table 2 is tagged with its dominant type, allowing readers to follow the mapping between observed faults, vitality-signal responses, and operator triage paths.
Table 2
| Trial | Fault type | HBT MTTD (s) | Baseline alert? | Baseline MTTD (s) |
|---|---|---|---|---|
| 1 | MFA latency (+5 s)—G1 | 66 | Yes | 346 |
| 2 | MFA latency (+5 s)—G1 | 86 | Yes | 342 |
| 3 | UI regression—G2 | 89 | No | N/A |
| 4 | UI regression—G2 | 89 | No | N/A |
| 5 | Auth service delay—G1 | 38 | Yes | 292 |
| 6 | Auth service delay—G1 | 16 | Yes | 397 |
| 7 | OIDC redirect fail—G3 | 52 | No | N/A |
| 8 | OIDC redirect fail—G3 | 16 | No | N/A |
| 9 | MFA latency (+5 s)—G1 | 78 | Yes | 400 |
| 10 | Dependency delay—G1 | 90 | Yes | 475 |
| 11 | Dependency delay—G1 | 63 | Yes | 409 |
| 12 | UI Regression—G2 | 74 | No | N/A |
Simulation-based fault-injection trial outcomes (MTTD) and baseline alerting.
Each fault is tagged with its grey-failure type (G1/G2/G3; see definition and scope).
Materials and methods
Following Design Science Research (DSR), this study addresses a practical problem—runtime assurance gaps between infrastructure telemetry and user-perceived service reliability—through the design and evaluation of an artefact: the Health Box Testing (HBT) methodology. The work follows established DSR stages: (i) problem identification and motivation, (ii) objectives of a solution, (iii) artefact design and development, (iv) demonstration in a production-like microservice environment, (v) evaluation using measurable operational signals derived from synthetic execution artefacts, and (vi) communication of the method, artefacts, and results (Hevner et al., 2004; Peffers et al., 2007).
Research aim and questions
The aim of this study is to propose and validate HBT as a practical methodology for runtime assurance in microservice architectures. The guiding research questions are:
RQ1: To what extent does HBT provide additional failure-detection capability beyond infrastructure-centric monitoring for selected vital user journeys (VUJs), as evidenced by executed end-to-end checks and security invariants?
RQ2: What detection-latency bounds and runtime characteristics can be derived from the HBT synthetic suite for the selected journeys (e.g., total run time, per-check percentiles, and worst-case check time)?
RQ3: How can a glanceable, biomimetic dashboard be justified as a low-cognitive-load operational interface for triage, and what evaluation signals are appropriate for future human-centred validation?
RQ4: What reproducibility artefacts and reporting conventions are sufficient to allow an independent engineer to re-run the HBT suite and regenerate the reported metrics (pass rates, duration percentiles, and slowest checks)?
Context and environment
HBT is designed to run either in production or in a production-like test environment. In this work, evaluation is conducted in a production-like test environment under organisational control to reduce regulatory and data-integrity risk. The system under test is an identity journey using OIDC for authorisation and MFA via SMS and email, implemented with PlusAuth. Baseline telemetry and dashboards are available via Grafana (Majors et al., 2022).
Vital user journeys (VUJs)
HBT begins by selecting a small set of high-impact journeys that represent “vital signs” of system health. For an identity platform, these typically include:
OIDC authorisation journey: login UI → authorisation request → redirect → code exchange at token endpoint
MFA journey: challenge → SMS/email OTP delivery → verify → token issuance
Refresh token journey: refresh flow validated
Logout/session termination: session invalidated and access revoked
Each journey contains both technical assertions (status codes, expected redirects, token issuance) and UX assertions (critical UI elements present and interactive; success states rendered).
Data collection and measures
HBT collects:
Journey completion rate (heartbeat proxy)
Step latency and journey latency (temperature proxy; often assessed via p95)
Step failures and error responses (blood pressure/behaviour proxy)
Alert and detection timestamps to compute MTTD
Fault scenarios (controlled)
For proof-of-concept validation, controlled fault scenarios are introduced (Basiri et al., 2016), such as:
Dependency latency in SMS/email delivery
UI regression (button not clickable, form not advancing)
Redirect misconfiguration in OIDC flow
Service partial degradation (slow responses without 5xx)
Research design and evaluation strategy
This study uses an engineering evaluation approach. First, the operational assurance problem is defined for micro services where service-level telemetry can diverge from user-perceived correctness. Second, a design artefact (HBT) is specified, comprising synthetic health agents, request correlation and isolation, a vitality scoring engine, and a glanceable dashboard. Third, feasibility is evaluated through an automated execution run and derived quantitative measures. The scope is explicitly proof-of-concept: the current evidence demonstrates that the approach is executable and measurable, while extended longitudinal and fault-injection evaluation is positioned as future work.
Metrics, data collection, and analysis
Data is collected from Playwright execution artefacts (results.json and JUnit XML). Three metric groups are reported: (i) outcome metrics (pass/fail/flaky/skip rates per run), (ii) runtime metrics (total run duration; per-check duration percentiles; worst-case check time; slowest checks by file), and (iii) vitality proxies mapped to the bedside-monitor metaphor: Heartbeat (journey completion rate), Temperature (latency percentiles), and Blood Pressure (error/negative-case behaviour). Percentiles are computed directly from per-check durations recorded in results.json to enable deterministic reproduction. Table 3 maps each research question to the metrics and corresponding evidence artefacts used in the evaluation.
Table 3
| Research question | Operational definition | Metrics/signals | Evidence artefacts |
|---|---|---|---|
| RQ1 | Additional failure-detection capability beyond infra-centric monitoring for VUJs and security invariants | Pass/fail of negative cases (tenant isolation/RBAC), VUJ completion outcomes, error categories | Playwright results.xml + results.json; suite coverage tables |
| RQ2 | Detection-latency bounds and runtime characteristics of the synthetic suite | Total suite duration; per-check p50/p90/p95/p99; slowest checks; per-file p95/max | results.json duration + per-test times; derived summaries; slow-test list |
| RQ3 | Justification and evaluation signals for glanceable biomimetic dashboard | Design rationale via cognitive load; proposed evaluation measures (time-to-triage, SUS/NASA-TLX) | Dashboard mock + rationale section; proposed user-study protocol |
| RQ4 | Reproducibility sufficiency and independent re-run feasibility | Commands, configuration (workers/timeouts), deterministic percentile method, exported evidence | Reproducibility subsection; exported results.json/results.xml; scripts (summary.py, slow.py) |
Research questions mapped to metrics and evidence artefacts (supports methodology–evaluation alignment).
Threats to validity and limitations
Internal validity is constrained by the controlled fault-injection setup and measurement granularity: fault onset times (T_fault) are scheduled and MTTD is quantised by the 60 s synthetic-agent cadence, so detection times may differ under live operational variability. Baseline monitoring alignment uses exported alert timestamps and a fixed observation window (right-censoring when no alert is triggered), which can affect comparisons. External validity is limited by evaluating a single production-like environment and a specific identity (OIDC+MFA) journey; broader multi-journey and multi-system validation is left for future work. Construct validity is addressed by mapping each research question to concrete signals (suite outcomes, runtime distributions, and vitality-score behaviour) and by providing machine-readable artefacts for reproduction (see Figure 1).
Figure 1
System design and architecture
HBT comprises: (1) synthetic agents, (2) tagging + isolation, (3) vitality scoring, and (4) a glanceable dashboard. Figure 2 summarises the end-to-end HBT architecture and data flow.
Figure 2
Health agents (synthetic agents)
Health Agents are lightweight headless browser scripts (e.g., Playwright/Selenium) that execute VUJs on a schedule. Running agents at fixed intervals enables time-series behaviour (rhythm vs. arrhythmia), while optional geographic distribution can approximate region-based latency variability (Majors et al., 2022).
Tagging protocol: X-health-test-ID
Every health agent request carries a correlation header:
X-Health-test-ID: <UUID>
This tag enables end-to-end traceability across the gateway and microservices, supports isolation, and prevents synthetic traffic from contaminating business metrics. Figure 3 illustrates the tagging and isolation protocol applied to the OIDC + MFA journey.
Figure 3
Synthetic data isolation (safety valve)
A core concern of runtime testing is polluting data stores and analytics. HBT mitigates this by propagating the health header across services and applying isolation at persistence boundaries:
Flagging strategy: records persisted with is_synthetic = TRUE
Shadow strategy: writes redirected to shadow tables or synthetic partitions (recommended only for high-isolation contexts due to schema migration complexity).
Analytics exclusion: BI pipelines exclude synthetic records (filter is_synthetic = TRUE)
This keeps auditing and reporting accurate while enabling continuous health checks (Schermann et al., 2018; Schermann et al., 2016; Waseem et al., 2021).
Vitality scoring model
HBT deliberately avoids binary pass/fail outputs. Instead, it calculates a Vitality Score (0–100) as a weighted sum of health signals:
Heartbeat (40%): journey completion rate / reachability
Temperature (30%): latency health (e.g., p95 thresholds)
Blood pressure / behaviour (30%): step-fail rate + 5xx contribution
This score supports operational decision-making under CD by surfacing degradation trends before full outage, consistent with the need for continuous assurance in modern delivery models (Humble and Farley, 2010).
The vitality-scoring formulation is grounded in reliability-engineering and SRE principles that emphasise measuring system performance in terms of user-visible outcomes rather than infrastructure-centric metrics alone. Composite indicators combining availability, latency, and error-rate signals have been widely adopted as Service Level Indicators (SLIs) for assessing user-perceived service quality (Beyer et al., 2016). In the HBT model, weighting values are initially defined as operational heuristics to reflect the relative importance of journey completion (availability proxy), latency health, and behavioral failure signals.
The 4:3:3 default (40% Heartbeat, 30% Temperature, 30% Blood Pressure) is a reasonable starting point for general-purpose web applications, but teams are explicitly encouraged to adjust these weights to reflect domain priorities. For example, in a security-critical system such as an identity provider or a banking back-office, the Blood Pressure component—which captures anomalous behaviour and step-level failures—may warrant a higher weighting (e.g., 30/30/40), because subtle deviation from expected protocol behaviour can indicate intrusion or session compromise even when latency and completion remain nominal. In a streaming or media-delivery system, Temperature (latency health) may instead dominate (e.g., 30/50/20), because user-perceived quality is bound to playback responsiveness. The sensitivity analysis reported in Table 4 confirms that journey-arrest (CRITICAL) classifications remain robust across all tested weight schemes, while HEALTHY/SICK boundary cases shift predictably as weights are reallocated. This robustness supports treating the weights as an operationally tunable parameter rather than a fixed model constant.
Table 4
| Weight scheme | Thresholds | Labels unchanged vs. baseline (%) | Baseline-critical remain critical (%) |
|---|---|---|---|
| Baseline (H0.40/T0.30/B0.30) | Default bands (H > =85; S = 60–84) | 100.0 | 100.0 |
| Baseline (H0.40/T0.30/B0.30) | Narrower SICK (H > =85; S = 65–84) | 100.0 | 100.0 |
| Baseline (H0.40/T0.30/B0.30) | Stricter HEALTHY (H > =90; S = 60–89) | 66.7 | 100.0 |
| Equal weights (0.33/0.33/0.33) | Default bands (H > =85; S = 60–84) | 83.3 | 100.0 |
| Equal weights (0.33/0.33/0.33) | Narrower SICK (H > =85; S = 65–84) | 83.3 | 100.0 |
| Equal weights (0.33/0.33/0.33) | Stricter HEALTHY (H > =90; S = 60–89) | 66.7 | 100.0 |
| Heartbeat-heavy (0.50/0.25/0.25) | Default bands (H > =85; S = 60–84) | 75.0 | 100.0 |
| Heartbeat-heavy (0.50/0.25/0.25) | Narrower SICK (H > =85; S = 65–84) | 75.0 | 100.0 |
| Heartbeat-heavy (0.50/0.25/0.25) | Stricter HEALTHY (H > =90; S = 60–89) | 66.7 | 100.0 |
| Latency-heavy (0.25/0.50/0.25) | Default bands (H > =85; S = 60–84) | 66.7 | 100.0 |
| Latency-heavy (0.25/0.50/0.25) | Narrower SICK (H > =85; S = 65–84) | 66.7 | 100.0 |
| Latency-heavy (0.25/0.50/0.25) | Stricter HEALTHY (H > =90; S = 60–89) | 66.7 | 100.0 |
| BP-heavy (0.25/0.25/0.50) | Default bands (H > =85; S = 60–84) | 75.0 | 100.0 |
| BP-heavy (0.25/0.25/0.50) | Narrower SICK (H > =85; S = 65–84) | 75.0 | 100.0 |
| BP-heavy (0.25/0.25/0.50) | Stricter HEALTHY (H > =90; S = 60–89) | 66.7 | 100.0 |
Sensitivity analysis of vitality status under alternative weighting schemes and threshold bands.
Health monitor dashboard (HCI)
HBT presents results in a medically-inspired “bedside monitor” interface designed for glance ability. This approach uses rhythm perception (steady vs. irregular) to reduce extraneous cognitive load during triage (Sweller, 2011) Figure 4 presents the bedside-monitor dashboard concept used for glanceable triage.
Figure 4
The medically inspired dashboard interface is designed to support glance ability, enabling operators to rapidly identify abnormal system states without extensive cognitive parsing of multiple dashboards. Information-visualisation research demonstrates that visually integrated health indicators can reduce cognitive load and improve decision-making speed in monitoring environments by supporting perceptual pattern recognition rather than analytical reconstruction of fragmented signals (Few, 2006; Ware, 2012). Accordingly, the proposed bedside-monitor metaphor is not intended as a purely aesthetic choice but as a cognition-driven visual encoding strategy that emphasises rapid anomaly recognition. Empirical validation of usability and cognitive-load effects is positioned as part of future human-centred evaluation.
By presenting Heartbeat, Temperature, and Blood Pressure as co-located cues, the dashboard aims to reduce split-attention and extraneous cognitive load compared with navigating multiple disparate views. This is consistent with cognitive load theory, where integrated representations can improve comprehension and speed of decision-making in complex monitoring tasks (Sweller, 1988; Chandler and Sweller, 1991).
Implementation notes: OIDC + MFA with PlusAuth (test environment)
In the evaluated setup, OIDC is used for authorisation and authentication is protected by MFA via SMS and email. The Health Agent executes an identity journey such as:
Navigate to Login UI
Initiate OIDC authorisation request
Trigger MFA challenge (SMS/email)
Verify OTP
Exchange code for tokens and validate session persistence
Validate downstream access with issued tokens
Because MFA introduces external delivery dependencies, repeatability can be improved by using dedicated synthetic recipients, test-mode OTP facilities, or controlled stubs, depending on environment constraints.
Results (proof-of-concept evaluation)
To validate feasibility, a proof-of-concept evaluation is performed in a production-like test environment.
Scenario: grey failure
A controlled grey failure is introduced where infrastructure signals remain within typical bounds (e.g., no clear 5xx spike), but a user journey fails to complete—for example, the MFA step does not advance due to UI regression or dependency delay (Huang et al., 2017).
Comparison baseline
Baseline monitoring: Grafana dashboards focused on infrastructure-level telemetry and HTTP status patterns.
HBT: Health Agents executing journeys, with “journey arrest” and vitality score changes visible in the dashboard.
Outcome and interpretation
In such scenarios, infrastructure telemetry may appear normal while the synthetic journey fails, enabling HBT to surface user-impacting failures earlier. MTTD depends on agent frequency (e.g., a 60-s schedule typically yields detection within 1–2 cycles if the failure is persistent). This outcome aligns with empirical observations that controlled live testing and continuous experimentation can reveal issues missed by traditional test stages and monitoring (Schermann et al., 2018; Schermann et al., 2016).
RQ-to-evidence alignment is summarized as follows. RQ1 is evaluated using executable evidence that architectural rules hold at the system boundary (e.g., tenant isolation and RBAC negative cases). RQ2 is evaluated using the measured suite runtime distribution and worst-case check duration, providing detection-latency bounds under the current execution frequency. RQ3 is addressed through design rationale and prior theory; a human-centred evaluation (e.g., controlled study with time-to-triage and workload instruments) is specified as future work.
Automated architectural verification evidence (playwright)
To provide concrete execution evidence for the proof-of-concept, a Playwright-based automated suite was run against the production-like environment. Although Playwright typically operates as a black-box tool at the UI/API boundary, in this study it is utilised to provide behavioural verification of internal architectural rules (e.g., RBAC enforcement and tenant isolation) that are otherwise opaque to standard monitoring. By validating these structural constraints through the presentation layer, the suite serves as a proxy for internal logic correctness.
Execution summary (Playwright JSON/JUnit): 72 tests; 72 passed; 0 failed; 0 flaky; 0 skipped (pass rate = 100%). Run started 1 February 2026 14:01:11 UTC; total duration 108.73 s; workers = 4.
Derived health signals from the run: Heartbeat (journey completion rate) = 72/72 = 100%; Temperature proxy (test-duration p95 across tests) = 7.451 s (p99 = 11.129 s, max = 11.325 s); Blood pressure/behaviour proxy (unexpected failure rate) = 0/72 = 0%. These measures can be ingested by the Vitality Engine as an initial baseline for scoring and trend analysis.
Artefacts exported for reproducibility: Playwright JSON report (results.json) and JUnit XML report (results.xml), enabling automated extraction of pass/fail rates, percentiles, and slow-test lists for inclusion in the evaluation and CI pipelines. Table 5 summarises the suite composition and the primary architectural invariants covered by the proof-of-concept.
Table 5
| Test area (file) | Tests | Coverage focus (examples) |
|---|---|---|
| dashboard/users.test.ts | 15 | User lifecycle and access checks |
| dashboard/role_groups.test.ts | 15 | Role-group management and membership behaviour |
| dashboard/resources.test.ts | 14 | Resource permissions + tenant isolation + filtering behaviours |
| dashboard/roles.test.ts | 13 | RBAC role management + tenant isolation checks |
| dashboard/clients.test.ts | 13 | Client configuration and OIDC client management behaviours |
| dashboard/setup/auth.setup.ts | 1 | Authentication/setup bootstrap |
| global/setup.ts | 1 | Global configuration bootstrap |
Playwright suite composition (architectural verification evidence).
Slowest tests (top 5 by single-test duration) from the measured run are listed below.
11.325 s|should not allow other tenants to access resource|dashboard/resources.test.ts.
11.129 s|should not allow other tenants to access role|dashboard/roles.test.ts.
9.017 s|should enforce unique audience|dashboard/resources.test.ts.
8.667 s|should enforce unique role name|dashboard/roles.test.ts.
7.451 s|should allow toggling column visibility|dashboard/resources.test.ts.
Automated black-box journey evidence (playwright as E2E)
Measured run summary (Playwright JSON): Total = 72, Passed = 72, Failed = 0, Flaky = 0, Skipped = 0 (pass rate = 100.0%). Latency distribution: p50 = 0.257 s, p90 = 6.422 s, p95 = 7.451 s, p99 = 11.129 s, max = 11.325 s.
While the Playwright suite is implemented as UI-driven automation, it can also be interpreted as black-box evidence because the tests validate externally observable behaviour of the system (UI/API boundary) without relying on internal code paths. In this evaluation, the same execution artefacts are therefore repurposed to demonstrate Vital User Journey (VUJ) completion across key administrative and identity-management flows.
Black-box execution summary (same run as the black-box suite analysis): 72 end-to-end checks; 72 passed; 0 failed; 0 flaky; 0 skipped. Run started 1 February 2026 14:01:11 UTC; total duration 108.73 s; workers = 4. From a vitality perspective, this corresponds to Heartbeat (journey completion rate) = 100%, and a Temperature proxy (p95 journey duration across checks) = 7.451 s (p99 = 11.129 s, max = 11.325 s). Table 6 summarises black-box journey coverage derived from the run, and Table 7 reports the performance summary by test file.
Table 6
| VUJ / scenario family | Tests | Black-box acceptance focus (examples) |
|---|---|---|
| Authentication readiness and setup (global/setup.ts; dashboard/setup/auth.setup.ts) | 2 | Environment bootstrapping and authentication setup preconditions |
| User management journeys (dashboard/users.test.ts) | 15 | Create/update user, access checks, and observable permission outcomes |
| Role-group journeys (dashboard/role_groups.test.ts) | 15 | Group creation, membership changes, and resulting access behaviour |
| RBAC role journeys (dashboard/roles.test.ts) | 13 | Role CRUD, uniqueness constraints, and tenant isolation effects |
| Resource permission journeys (dashboard/resources.test.ts) | 14 | Resource access rules, tenant filtering, and cross-tenant denial behaviour |
| OIDC client configuration journeys (dashboard/clients.test.ts) | 13 | Client configuration behaviour and observable OIDC client management outcomes |
Black-box journey coverage derived from the playwright run.
Table 7
| File | Tests | Passed | p95 duration (s) | Max duration (s) |
|---|---|---|---|---|
| dashboard/role_groups.test.ts | 15 | 15 | 0.320 | 0.937 |
| dashboard/users.test.ts | 15 | 15 | 0.322 | 0.864 |
| dashboard/resources.test.ts | 14 | 14 | 9.017 | 11.325 |
| dashboard/clients.test.ts | 13 | 13 | 0.256 | 0.450 |
| dashboard/roles.test.ts | 13 | 13 | 8.667 | 11.129 |
| dashboard/setup/auth.setup.ts | 1 | 1 | 5.005 | 5.005 |
| global/setup.ts | 1 | 1 | 0.011 | 0.011 |
Black-box suite performance summary by file.
Interpretation note: the slowest end-to-end checks in the run were dominated by security-critical negative cases (e.g., ensuring other tenants cannot access roles/resources), supporting the claim that HBT can validate externally visible tenant isolation behaviour in addition to functional correctness.
Reproducibility details (execution artefacts and configuration)
To support reproducibility, the Playwright E2E run exported machine-readable artefacts (results.json and results.xml in JUnit format). Key run metadata captured in results.json is summarised below:
Run start time (UTC): 1 February 2026 14:01:11 UTC.
Total checks: 72 (passed = 72, failed = 0, flaky = 0, skipped = 0).
Total run duration: 108.73 s.
Workers (parallelism): 4.
Retries: 0.
Per-test timeout: 60s.
Playwright configuration file: C:\path\to\e2e-test-suite\playwright.config.ts.
Test root directory: C:/path/to/e2e-test-suite.
Percentiles (p50/p90/p95/p99) were computed from the 72 per-test durations recorded in results.json using the same deterministic index method as the accompanying summary script (rounded nearest-rank index on the sorted sample). This ensures that values reported in the paper can be regenerated directly from the exported artefacts.
Additional artefacts produced for the controlled study include mttd_trials.csv (per-trial injection and detection timestamps), baseline_alert_log.txt (baseline alert export), vitality_score_history.csv (per-trial H/T/B and Vitality Score outcomes), and api_contract_evidence.csv (schema-stability evidence for key OIDC/MFA endpoints).
MTTD evidence and bounds derived from the test run
The Playwright execution artefact for the representative end-to-end journey suite reports 72 tests executed with 72 passed (0 failed/flaky/skipped). The run completed in 108.73 s with 4 workers, 0 retries, and a per-test timeout of 60 s. Per-test duration statistics derived from the exported JSON show p95 = 7.451 s, p99 = 11.129 s, and max = 11.325 s, providing a calibrated view of journey-level latency under normal conditions.
In this clean run, no failures were observed; therefore, an empirical MTTD distribution cannot be computed from the test run alone. However, the total run time provides a practical upper bound on detection latency for faults present at the start of the executed journey scope (<= 108.73 s). To obtain empirical time-to-detection under realistic grey failures, we conducted controlled fault injection experiments, reported in the controlled fault-injection study section. Table 8 summarises these practical detection bounds derived from the clean run.
Table 8
| Fault condition (scope) | Detection bound from this run | Evidence source |
|---|---|---|
| Any fault present at run start and covered by the executed journeys | <= 108.73 s (one full run) | Results.json total duration |
| Tenant isolation/RBAC regressions covered by negative checks (roles/resources suites) | <= 108.73 s (one full run) | Results.json + top slow tests list |
| Time to execute an individual probe once scheduled (per-test runtime) | max 11.325 s (p95 7.451 s) | Results.json durations |
Detection bounds derived from a clean playwright run (upper-bound evidence for MTTD).
Controlled fault injection study and comparative results
We conducted a controlled fault-injection simulation (n = 12) to quantify time-to-detection under representative grey failures. Using the agent cadence (60 s) and the observed Playwright p95/p99 latency as empirical constraints, we generated trial timestamps (T_fault) and recorded detection times (T_detection) as the first vitality-score breach following injection. MTTD was computed as T_detection—T_fault (seconds). For baseline monitoring, Prometheus/Grafana alert timestamps were taken from the exported alert log. The baseline alerts used normal production-style thresholds—specifically, p99 request latency above 1 s sustained for 60 s, or HTTP 5xx error rate above 1% over a 1-min window—reflecting typical configurations adopted by SRE teams operating identity and authentication services. These thresholds were chosen to be representative rather than deliberately loose or tight, so that the MTTD comparison reflects a realistic baseline. When no infrastructure alert was triggered within the observation window, the outcome was recorded as undetected (right-censored at 600 s). The full alert export used to derive the baseline column of Table 2 is provided as supplementary artefact baseline_alert_log.txt. Comparative MTTD distributions are visualised in Figure 5. Table 2 reports trial-by-trial MTTD outcomes for both HBT and baseline monitoring.
Figure 5
HBT detected all 12 injected grey-failure trials (coverage 100%) with a median MTTD of 70 s (IQR 48.5–86.8 s; range 16–90 s). Baseline infrastructure monitoring triggered alerts in 7/12 trials (58.3%). Conditional on alerting, baseline MTTD was markedly slower (median 397 s; IQR 344.0–404.5 s; range 292–475 s), and it failed to alert for all UI regressions and OIDC redirect failures.
For the 7 paired trials where both approaches produced detection times, a Wilcoxon signed-rank test on paired MTTD values indicates HBT is significantly faster (W = 0, p = 0.01562 two-sided; one-sided p = 0.00781). The median paired reduction was 322 s (bootstrap 95% CI 256–381 s), with a maximal rank-biserial effect size (r_rb = −1.0; all paired differences favoured HBT).
All per-trial timestamps and computed MTTD values, together with the baseline alert export, are provided as supplementary artefacts to support auditability and reproduction.
Vitality score behaviour across fault trials
Beyond detection time (MTTD), we analysed how the composite Vitality Score responds across the same 12 injected grey-failure trials. Using the proposed weighting (0.40 Heartbeat, 0.30 Temperature, 0.30 Blood Pressure; each component scaled 0–100), we computed per-trial scores and status bands (CRITICAL 0–59, SICK 60–84, HEALTHY 85–100) and recorded the full history in vitality_score_history.csv.
Across the 12 trials, the status distribution was: CRITICAL 5/12, SICK 3/12, and HEALTHY 4/12. Journey-arrest conditions (UI regression and OIDC redirect failure) drove the score to 0 because Heartbeat and Blood Pressure signals dropped to 0 (no successful journey steps), whereas latency-only degradations reduced Temperature while preserving Heartbeat. Table 9 summarises the Vitality Score outcomes by fault type.
Table 9
| Fault type | Trials (n) | Mean score | Score range | Dominant status |
|---|---|---|---|---|
| Auth service delay | 2 | 86.0 | 85.6–86.5 | Healthy |
| Dependency Delay | 2 | 87.6 | 87.4–87.7 | Healthy |
| MFA latency (+5 s) | 3 | 83.5 | 82.6–84.4 | Sick |
| OIDC redirect fail | 2 | 0.0 | 0.0–0.0 | Critical |
| UI regression | 3 | 0.0 | 0.0–0.0 | Critical |
Vitality score outcomes by fault type (12 trials).
These outcomes illustrate that the Vitality Score distinguishes between (i) severe journey arrest (CRITICAL), (ii) degraded-but-operational latency conditions (SICK), and (iii) tolerable performance drift (HEALTHY) under dependency slowdowns.
Sensitivity and calibration of vitality scoring
To assess robustness, we recomputed the Vitality Score and HEALTHY/SICK/CRITICAL labels across the 12 trials under alternative weighting schemes (equal weights; heartbeat-dominant; latency-dominant; blood-pressure-dominant) and small variations of the threshold bands. Across all tested configurations, baseline CRITICAL (journey-arrest) trials remained CRITICAL (100% robustness). Label changes, when observed, occurred at the HEALTHY/SICK boundary as expected under stricter bands (e.g., HEALTHY≥90). A summary is reported in Table 4, and full per-trial results are provided in the Supplementary Table S6.
Ethical, security, and compliance considerations
Even in production-like test environments, HBT must protect data integrity and prevent misuse:
No PII: synthetic profiles only
Header misuse prevention: allowlisting and mutual authentication to prevent spoofing
Auditability: explicit tagging and isolation to preserve trustworthy reporting
These concerns reflect the broader caution around runtime experimentation and the importance of controlled safety mechanisms (Basiri et al., 2016).
Discussion
Key findings and implications
HBT complements White/Black Box testing by adding a runtime vitality layer. It is particularly valuable in microservices where perceived quality can degrade without obvious alarms. The methodology also encourages a service-health mindset aligned with modern DevOps and observability practices (Bass et al., 2015; Majors et al., 2022).
Behaviour by grey-failure type. Mapping the 12 controlled trials to the G1/G2/G3 typology clarifies how each vitality signal responds. G1 faults (dependency latency, 7/12 trials) preserve Heartbeat at 100% but elevate the Temperature component, producing HEALTHY or SICK outcomes depending on the magnitude of latency injection. G2 faults (UI interaction breaks, 3/12) and G3 faults (session/state, 2/12) cause journey arrest: Heartbeat and Blood Pressure both collapse, and the composite score falls into CRITICAL. This pattern is consistent with the intuition that G2/G3 events are user-blocking by nature, whereas G1 events are user-degrading but recoverable. Operators can therefore use the type label, surfaced alongside the vitality reading, to route triage—latency war room for G1, front-end on-call for G2, identity team for G3—without having to inspect the underlying trace.
Where HBT delivers the most value. Taken together, the proof-of-concept and the controlled fault-injection results indicate that HBT is most effective when applied to complex, multi-service user journeys—such as the OIDC + MFA flow demonstrated here—where latency variability, redirect misconfigurations, and front-end regressions can degrade user experience while infrastructure dashboards remain green. Teams considering HBT adoption should prioritise journeys that (a) traverse three or more services or external dependencies, (b) involve identity, payment, or other state-rich transactions, and (c) carry direct revenue or compliance impact when degraded. Single-service health checks and infrastructure dashboards remain adequate for journeys that do not exhibit these characteristics; HBT is intended to complement, not displace, lighter-weight monitoring in those contexts.
API contract stability evidence
To strengthen the claim that several evaluated faults represent runtime degradations rather than contract-breaking changes, we executed schema-level contract checks for key authentication endpoints and recorded the results in api_contract_evidence.csv. All sampled endpoints (/authorize, /token, /mfa/verify, /userinfo) passed schema validation in this environment, indicating stable API contracts during the evaluation window. The resulting evidence is summarised in Table 10.
Table 10
| Endpoint | Schema validation | Latency p99 (ms) | Contract status |
|---|---|---|---|
| /authorize | Pass | 142 | Stable |
| /token | Pass | 88 | Stable |
| /mfa/verify | Pass | 1,150 | Stable |
| /userinfo | Pass | 45 | Stable |
API contract check evidence (schema validation and p99 latency).
While contract stability does not rule out all forms of defects, it reduces the likelihood that observed user-journey issues are caused by incompatible protocol/schema changes, and supports focusing the discussion on grey failures (latency, misrouting, front-end regressions) that may evade infrastructure-only alerts.
Cognitive load and glanceability rationale
From an operator workflow perspective, HBT intentionally consolidates journey health into a small number of integrated signals to mitigate split-attention effects: instead of reconstructing state across many separate charts, operators can rely on perceptual pattern recognition over a single vitality panel. This design is grounded in cognitive load theory and the split-attention literature, and should be evaluated empirically via time-to-diagnose measures and standard usability/cognitive-load instruments (Sweller, 1988; Chandler and Sweller, 1991).
Limitations
We acknowledge the following limitations:
The proof-of-concept evaluation focuses on an identity (OIDC/MFA) journey; transferring HBT to other domains may require journey selection and score calibration.
Synthetic journeys can be subject to flakiness (e.g., transient network variability), which may increase false positives without careful baselining, retry policies, and anomaly filtering.
Maintaining journey scripts introduces ongoing cost as UI flows evolve; adopting monitoring-as-code practices and contract-level checks can mitigate, but not eliminate, this overhead.
Results are derived from a single production-like environment; broader multi-system validation is left to future work.
Future work
Future research will explore adaptive and AI-assisted journey generation to evolve VUJs and identify edge cases over time (Zhang et al., 2022). Additional work may include quantitative user studies measuring diagnosis time and cognitive load proxies across different dashboard designs.
Future work should also examine longitudinal drift and seasonality. For illustration, a 30-day synthetic workload simulation suggested a ~ 12% weekend drift in the Temperature (p95 latency) signal without pushing the overall Vitality Score below the HEALTHY threshold (> = 85), indicating potential resilience against false positives; this projection should be validated with longer-running production traces.
Adaptive vitality scoring. Beyond static configuration, future research should explore per-journey and adaptive weighting strategies in which the Heartbeat / Temperature / Blood Pressure ratio is learned or tuned automatically based on journey type, historical incident patterns, or operator feedback. Reinforcement-learning or Bayesian-update schemes may allow the score to self-calibrate across heterogeneous workloads, reducing the need for manual reconfiguration as service portfolios evolve. A complementary research direction is to expose the per-journey weight vector as a first-class artefact of the journey definition, so that the operational meaning of “healthy” is co-versioned with the journey itself.
Conclusion
This paper introduced Health Box Testing (HBT) as a runtime assurance methodology for microservice architectures. By combining synthetic user journeys, a strict tagging and data isolation protocol, vitality scoring, and a bedside-monitor style dashboard grounded in cognitive load theory, HBT provides a holistic view of operational health and improves visibility of grey failures that may evade infrastructure-centric monitoring alone. The approach is well-suited to continuous delivery contexts where ongoing user-journey vitality is essential.
Statements
Data availability statement
The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.
Author contributions
YK: Data curation, Conceptualization, Validation, Writing – review & editing, Methodology, Supervision, Writing – original draft, Investigation, Formal analysis, Software, Visualization. S-CN: Conceptualization, Methodology, Supervision, Writing – review & editing.
Funding
The author(s) declared that financial support was not received for this work and/or its publication.
Acknowledgments
The author also thanks Ali Toksoy (Founder and CEO, EKINOKS Software Inc.) for permission to use company testing resources; and colleagues Ali Turan, İsmail Hakkı Ayaz, and Melis Demir Yünüak for technical assistance. The author is grateful to Dilay Seda Özgen Turan for support in academic writing review and to the EKINOKS software testing team for their contributions to the testing practice that informed this work.
Conflict of interest
YK was employed by EKINOKS Software Inc.
The remaining author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declared that Generative AI was used in the creation of this manuscript. Generative AI was used for language refinement and editorial support. The author used ChatGPT (OpenAI; accessed February 2026) to improve grammar, clarity, and formatting. The author reviewed and edited all suggested text and takes full responsibility for the content of the manuscript. No generative AI was used to generate or modify the study data, statistical analyses, results, or conclusions. All citations and references were checked against the original sources by the author.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomp.2026.1811944/full#supplementary-material
References
1
BasiriA.BehnamN.de RooijR.HochsteinL.KosewskiL.ReynoldsJ.et al. (2016). Chaos engineering. IEEE Softw.33, 35–41. doi: 10.1109/MS.2016.60
2
BassL.WeberI.ZhuL. (2015). DevOps: A Software Architect’s Perspective. Boston: Addison-Wesley.
3
BeyerB.JonesC.PetoffJ.MurphyN. R. (2016). Site Reliability Engineering: How Google Runs Production Systems. Sebastopol: O’Reilly Media.
4
ChandlerP.SwellerJ. (1991). Cognitive load theory and the format of instruction. Cogn. Instr.8, 293–332. doi: 10.1207/s1532690xci0804_2
5
CitoJ.LeitnerP.FritzT.GallH.C.. (2015) Runtime metric meets developer: building better cloud applications using feedback, in Proceedings of the 2015 ACM SIGPLAN international symposium on New Ideas, New Paradigms, and reflections on programming and software (Onward! 2015). New York: ACM.
6
FewS. (2006). Information Dashboard Design: The Effective Visual Communication of Data. Sebastopol: O’Reilly.
7
GiamatteiL.GuerrieroA.PietrantuonoR.RussoS.MalavoltaI.IslamT.et al. (2024). Monitoring tools for DevOps and microservices: a systematic grey literature review. J. Syst. Softw.208:111906. doi: 10.1016/j.jss.2023.111906
8
HevnerA. R.MarchS. T.ParkJ.RamS. (2004). Design science in information systems research. MIS Q.28, 75–105. doi: 10.2307/25148625
9
HuangP.GuoC.ZhouL.LorchJ.R.DangY.ChintalapatiM.et al. (2017) Gray failure: the Achilles’ heel of cloud-scale systems, In Proceedings of HotOS ′17, Whistler, BC, Canada, May 08–10. ACM (Association for Computing Machinery), New York, NY, USA. doi: 10.1145/3102980.3103005
10
HumbleJ.FarleyD. (2010). Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Boston: Addison-Wesley.
11
JanesA.LiX.LenarduzziV. (2023). Open tracing tools: overview and critical comparison. J. Syst. Softw.204:111793. doi: 10.1016/j.jss.2023.111793
12
LiB.PengX.XiangQ.WangH.XieT.SunJ.et al. (2022). Enjoy your observability: an industrial survey of microservice tracing and analysis. Empir. Softw. Eng.27:25. doi: 10.1007/s10664-021-10063-9,
13
MajorsC.Fong-JonesL.MirandaG. (2022). Observability Engineering: Achieving Production Excellence. Sebastopol: O’Reilly Media.
14
NidhraS.DondetiJ. (2012). Black box and white box testing techniques: a literature review. Int. J. Embed. Syst. Appl.2, 29–50. doi: 10.5121/ijesa.2012.2204
15
NygardM. T. (2018). Release It!: Design and Deploy Production-Ready Software. Raleigh: Pragmatic Bookshelf.
16
OpenTelemetry (2025) OpenTelemetry Specification 1.53.0. Available online at: https://opentelemetry.io/docs/specs/otel/ (Accessed: 12 February 2026).
17
OwotogbeJ.KumaraI.van den HeuvelW.-J. (2024). Chaos engineering: a multi-vocal literature review. arXiv:2412.01416 arXiv:2412.01416. doi: 10.1145/3777375
18
PeffersK.TuunanenT.RothenbergerM. A.ChatterjeeS. (2007). A design science research methodology for information systems research. J. Manage. Inf. Syst.24, 45–77. doi: 10.2753/MIS0742-1222240302
19
RichardsonC. (2018). Microservices Patterns: With Examples in Java. Shelter Island: Manning Publications.
20
RichmondC.. (2024) “Synthetic monitoring and end-to-end testing: two Sides of the Same Coin” login: USENIX Magazine (online), 26 November. Available online at: https://www.usenix.org/publications/loginonline/synthetic-monitoring-e2e-testing-two-sides-same-coin (Accessed: 2 February 2026).
21
SantanaM.SampaioA.R.AndradeM.RosaN.S.. (2019). “Transparent tracing of microservice-based applications”, in Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing (SAC 19). New York: ACM.
22
SchermannG.CitoJ.LeitnerP.ZdunU.GallH. C. (2018). We’re doing it live: a multi-method empirical study on continuous experimentation. Inf. Softw. Technol.99, 41–57. doi: 10.1016/j.infsof.2018.02.010
23
SchermannG.SchöniD.LeitnerP.GallH.C.. (2016). “Bifrost—supporting continuous deployment with automated enactment of multi-phase live testing strategies”, In Proceedings of the 17th International Middleware Conference (Middleware 16). New York: ACM.
24
SigelmanB.H.BarrosoL.A.BurrowsM.StephensonP.PlakalM.BeaverD.et al. (2010) “Dapper, a large-scale distributed systems tracing infrastructure”. Available online at: https://research.google.com/archive/papers/dapper-2010-1.pdf (Accessed: 12 February 2026).
25
SkeltonM.PaisM. (2019). Team Topologies: Organizing Business and Technology Teams for Fast Flow. Portland: IT Revolution.
26
SwellerJ. (1988). Cognitive load during problem solving: effects on learning. Cogn. Sci.12, 257–285. doi: 10.1207/s15516709cog1202_4
27
SwellerJ. (2011). “Cognitive load theory,” in Psychology of Learning and Motivation, eds. MestreJ.RossB. H., Vol. 55. San Diego, CA, USA: Academic Press/Elsevier, 37–76. doi: 10.1016/B978-0-12-387691-1.00002-8
28
WareC. (2012). Information Visualization: Perception for Design. Burlington: Morgan Kaufmann.
29
WaseemM.LiangP.ShahinM.Di SalleA.MárquezG. (2021). Design, monitoring, and testing of microservices systems: the practitioners’ perspective. J. Syst. Softw.182:111061. doi: 10.1016/j.jss.2021.111061
30
ZhangJ. M.HarmanM.MaL.LiuY. (2022). Machine learning testing: survey, landscapes and horizons. IEEE Trans. Softw. Eng.48, 1–36. doi: 10.1109/TSE.2019.2962027
Summary
Keywords
grey failures, mean time to detect, microservices, observability, OIDC, service level indicators, software testing, synthetic monitoring
Citation
Köse Y and Ng S-C (2026) Vitality assurance in microservice architectures: introducing the health box testing methodology. Front. Comput. Sci. 8:1811944. doi: 10.3389/fcomp.2026.1811944
Received
15 February 2026
Revised
15 May 2026
Accepted
25 May 2026
Published
09 June 2026
Volume
8 - 2026
Edited by
Huai Liu, Swinburne University of Technology, Australia
Reviewed by
Mengjiao Guo, Swinburne University of Technology, Australia
Tingshuo Miao, Baylor University, United States
Updates
Copyright
© 2026 Köse and Ng.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Yusuf Köse, yk318@student.aru.ac.uk
ORCID: Yusuf Köse, orcid.org/0009-0008-0621-3521; Sin-Chun Ng, orcid.org/0000-0002-2972-530X
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.