The role of explainability throughout the MLOps lifecycle: review and research agenda

Abstract

As Machine Learning Operations (MLOps) adoption accelerates, systematic integration of explainability is imperative for reliability, transparency, and continuous quality assurance. This paper presents a scoping review examining how explainability is integrated across the MLOps lifecycle, encompassing data handling, model development, and deployment. Each phase is further analyzed through its subareas: data handling (data quality, data pre-processing, and data management), model development (training and pre-deployment auditing), and deployment (developer oversight and end-user interfacing). We identified several key touchpoints within each subarea where XAI methods address specific technical and operational challenges. The synthesis covers a wide range of topics, from explainable imputation and data filtering to fairness auditing in high-stakes decision-making. Findings reveal that although explainability is widely applied, it remains fragmented, with insufficiently validated reliability, and limited operationalization for regulatory compliance. Building on this analysis, we propose a research agenda for embedding continuous explainability throughout MLOps pipelines. Key directions include connecting explainability touchpoints across lifecycle phases, validating the reliability of XAI methods, and operationalizing explainability to meet regulatory requirements such as those defined in the EU AI Act. By framing explainability as an infrastructural mechanism for assurance rather than a post-hoc diagnostic feature, this work advances a lifecycle-spanning perspective on trustworthy and governable AI systems.

1 Introduction

Recent advances in artificial intelligence (AI) and machine learning (ML) have driven significant innovation across domains, accompanied by growing public attention to their benefits and risks. While the technical capabilities of turning large volumes of data into operational decision support with human-like characteristics are overwhelming, the black-box “magic” of ML methods and their non-determinism lead to hesitation in deploying them in critical, autonomous applications. The EU AI Act exemplifies the growing societal and regulatory demand for transparency, accountability, and oversight in AI systems (European Parliament and European Council, 2024).

Explainable AI (XAI) has emerged as a response to such requirements, referring to methods that make ML-driven decision-making processes transparent and understandable to humans. This is critical for improving model performance, building trust, and enabling AI adoption in high-stakes domains (Barredo Arrieta et al., 2020). However, as Wang et al. (2024) emphasize, explainability is not a one-size-fits-all concept. XAI serves different purposes at different points in time (when) with different information (what) to different stakeholders (whom) in different presentation formats (how), depending on the needs and conditions. In this work, we adopt this perspective as a motivating principle and focus on the functional roles of explainability in practice.

To translate these explainability principles into practice, they must be embedded within the engineering and operational processes that govern the entire lifecycle of ML systems. The concept of MLOps (Machine Learning Operations) is a general framework encompassing such a multitude of aspects of developing and operating an ML system. MLOps represents a paradigm that combines best practices, conceptual foundations, and a development culture for machine learning products. It merges best practices from data engineering, machine learning, and software engineering, especially DevOps, into a unified engineering discipline (Kreuzberger et al., 2023).

While MLOps and XAI have each been extensively studied (e.g., Kreuzberger et al., 2023 and Wang et al., 2024; Longo et al., 2024; Angelov et al., 2021, respectively), the intersection between them remains fragmented and underexplored. This paper addresses that gap by presenting a scoping review that examines how explainability is integrated throughout the MLOps lifecycle. We adopt a lifecycle-wide conception of explainability that provides transparency into the ML-driven decision-making processes distributed across three main MLOps phases: data handling, model development, and deployment (Figure 1). This encompasses several key explainability touchpoints such as data quality issues (e.g., why data is missing, how to handle it), preprocessing decisions (e.g., which features were selected and why), model behavior (e.g., prediction rationale), and operational decisions (e.g., when to retrain or intervene under performance drift). We map XAI research across these touchpoints, examine cross-phase connections between explainability practices, analyse how XAI reliability and trustworthiness are evaluated, and assess alignment with emerging regulatory requirements. Building on this synthesis, we propose a three-part research agenda focused on (i) advancing from isolated stages to continuous explainability in MLOps, (ii) assuring the reliability of XAI methods, and (iii) enabling regulatory compliance through explainability.

Figure 1

The paper is organized as follows. Section 2 describes the methodology for the scoping review. Section 3 presents the functional roles of explainability across the MLOps lifecycle, structured by Data Handling (Section 3.1), Model Development (Section 3.2), and Deployment (Section 3.3). Section 4 analyses lifecycle-wide connections between explainability touchpoints. Section 5 examines how XAI reliability and trustworthiness are assessed, while Section 6 evaluates prospective alignment with regulatory and compliance requirements. Section 7 outlines a research agenda for integrating explainability across the MLOps lifecycle, and Section 8 concludes the paper.

2 Methodology

The study adopts a semi-systematic scoping review methodology (Arksey and O'Malley, 2005), also referred to as a mapping study (Kitchenham et al., 2011) (see Figure 2). Unlike systematic literature reviews (SLRs), which address tightly scoped empirical questions and aim to aggregate evidence quantitatively, mapping studies prioritize breadth over depth and focus primarily on qualitative synthesis of research trends (Kitchenham et al., 2011). Consequently, search strategies are intentionally less restrictive, and no formal quality appraisal is applied. The findings should therefore be interpreted as indicative of structural patterns, thematic gaps, and emerging directions rather than as exhaustive or statistically representative evidence. While this approach is well-suited to continuously growing fields, such as explainable AI, where contributions are dispersed and research efforts are often unbalanced, it may overlook certain relevant studies.

Figure 2

2.1 XAI-MLOps mapping study

In this section, we establish the analytical framework guiding our literature mapping. We define explainability as the set of methods and artifacts that make data transformations, model behaviors, and operational decisions transparent and interpretable across the MLOps phases. Critically, end-to-end MLOps pipelines often employ multiple ML models for different tasks, from data preprocessing (e.g., feature selection, imputation) to operational monitoring (e.g., anomaly detection). Under this lifecycle-wide definition, explainability must therefore address transparency at each stage where ML is employed, rather than focusing solely on final model outputs. This comprehensive view of explainability provides the foundation for mapping XAI methods across data handling, model development, and deployment.

To translate this definition of explainability, we distilled each MLOps phase into operational subareas. This provides a finer granularity into where and how transparency needs arise (Figure 1). In the data handling phase, explainability spans data quality, data pre-processing, and data management, which are well-established downstream processes in data engineering (Saltz and Shamshurin, 2016). In model development, we distinguish explainability in training to support design choices and in pre-deployment auditing for diagnostics and validation of model behavior, reflecting the division between development and verification activities (Breck et al., 2017). In deployment, explainability differentiates between technical transparency for developer oversight and understandability for end-user interfacing, emphasizing human-centered XAI frameworks (Miller, 2019).

These subareas provide the organizational structure for identifying where explainability is applied within MLOps pipelines. Within each subarea, we identified several explainability touchpoints, i.e., specific subprocesses where XAI methods enhance transparency, reliability, and traceability. Touchpoints emerged from preliminary scoping searches that revealed where XAI research concentrates, highlighting recurring themes and application contexts. This preliminary mapping revealed uneven distribution of explainability across stages, weak connections across touchpoints, variation in how XAI methods are evaluated, and emerging regulatory considerations. These observations informed the following research questions:

• RQ1:How is explainability applied across the MLOps lifecycle in terms of functional roles, application domains, and emerging trends?

• RQ2:How are explainability touchpoints connected across lifecycle stages?

• RQ3:How are XAI methods assessed for reliability and trustworthiness?

• RQ4:To what extent do current XAI practices consider regulatory requirements?

2.2 Search strategy and study selection

A comprehensive literature search was conducted in Google Scholar covering the period 2020–2025. The search employed a structured query combining explainability with touchpoint-specific terms (e.g., “explainable”, “explainability”, “XAI”, “clustering”). Google Scholar was selected to capture the breadth of explainability practices across interdisciplinary venues rather than conduct a database-specific systematic review. Its coverage spans publications indexed in IEEE Xplore, ACM Digital Library, and Springer. We screened the first ten pages of results for each query, as relevance and citation impact drop substantially beyond this range. While this approach may not guarantee exhaustiveness, and replicability may be affected by Google Scholar' dynamic ranking and indexing mechanisms, it provides broad coverage across the most relevant and highly cited works, which is appropriate for a scoping review. Titles and abstracts were screened to identify papers directly addressing explainability in relation to the defined touchpoints. Only peer-reviewed journal articles and conference papers focusing on empirical and methodological studies were included. While much ML and XAI research is disseminated through non-peer-reviewed open platforms (e.g., arXiv), this review relies on established peer-review processes to ensure scholarly rigor. Included studies were open access or available through institutional access. This process yielded 219 papers included in the final analysis (see Figure 3). The selected literature was mapped and conceptually analyzed to address the research questions, examining functional roles of explainability, cross-phase connections, assessment practices, and regulatory alignment across the MLOps lifecycle.

Figure 3

3 Role of explainability throughout the MLOps lifecycle (RQ1)

To examine how explainability is integrated in machine learning operations, we analyzed the literature across the three main phases of the MLOps lifecycle: data handling, model development, and deployment. In unpacking each phase, we identified several explainability touchpoints. These are specific subprocesses where XAI methods enhance transparency, reliability, and traceability (see Figure 1). Explainability in the data handling phase spans data quality, data pre-processing, and data management; in model development, it includes training and pre-deployment auditing; and in deployment, it covers developer oversight and end-user interfacing.

Figure 4 summarizes how explainability is distributed across these phases, revealing a clear imbalance in current research. Most studies focus on explainability at deployment time, particularly on monitoring for developer oversight and on decision support for end-user interfacing. Within data handling, explainability is primarily applied during data pre-processing, with a strong emphasis on the clustering task. In contrast, activities in intermediate stages such as training and pre-deployment auditing receive substantially less attention. Overall, explainability is predominantly treated as a post-hoc interpretability tool rather than as a design-time mechanism embedded throughout the lifecycle.¹ The following subsections analyse the explainability touchpoints in detail, highlighting dominant functional roles, application contexts, and emerging trends.

Figure 4

3.1 Data handling

3.1.1 Data quality

This section discusses three key areas: how missing data, data imbalance, and automated data labeling are handled through explainability approaches. Table 1 summarizes the corresponding publications.

Handling missing data and explainability

Missing or incomplete data is a pervasive challenge in real-world systems, often degrading both the predictive accuracy of models and the interpretability of their outcomes. Hakim and Darari (2021) demonstrated that different types of data reduction impact datasets in distinct ways, showing that missing data not only reduces recommendation accuracy but also undermines the quality of system explanations. To address this issue, imputation methods² have been widely adopted to fill missing values. However, ensuring that such imputations are themselves explainable is critical to enable traceability (Adhikari et al., 2022). Our analysis reveals that research on explainability in handling missing data addresses two distinct roles: explaining imputation decisions and understanding missingness.

Research on explainable imputation has progressed significantly. Early contributions introduced likelihood-based imputations (Song and Sun, 2020), followed by feature-importance explanations combined with imputation methods (Cinquini et al., 2023). Subsequent work proposed constraint-driven imputations with attribute-level explanations (Hans et al., 2023), while recent approaches have explored interpretable diffusion-based resampling strategies (Jiang and Zhang, 2025). A meta-learning framework that jointly addresses imputation and explainability was also proposed (Baysal Erez et al., 2025).

Beyond imputation, a parallel line of research emphasizes understanding the causes and implications of missingness. Song et al. (2021) proposed presenting explanations as logical chains (e.g., “The customer ID data is missing because the transaction-to-customer rule was violated, indicating an orphaned transaction”), thereby framing missingness as a domain-specific reasoning task rather than just a technical issue. Zamanian et al. (2023) further highlighted that assumptions about missingness can vary across settings. Azimi and Pahl (2021) showed that data incorrectness has a more damaging effect on model performance than incompleteness. Accordingly, Azimi and Pahl argued for prioritizing data accuracy over mere imputation. In safety-critical domains, missing values caused by sensor failures, communication breakdowns, or power outages cannot simply be imputed without risk. To address this, research proposed models that handle missingness during training while remain interpretable (Scutari, 2020; McTavish et al., 2024). Such approaches not only enhance robustness but also make it possible to identify how missing features influence model outputs.

Handling imbalanced data and explainability

In many real-world applications, the number of samples across classes is not uniformly distributed, but rather one class contains a larger number of instances compared to others. Such imbalanced datasets often lead to overfitting, where models become biased toward the majority class, resulting in poor performance on minority class predictions. To address this challenge, data-level methods such as undersampling and oversampling are commonly employed, with oversampling generally preferred as it avoids information loss. Nevertheless, the effect of oversampling on performance can vary across datasets and tasks (Mujahid et al., 2024). We found that research on explainability in handling imbalanced data addresses three distinct roles: assessing oversampling effects, guiding data augmentation, and understanding class properties.

Synthetic Minority Over-sampling Technique (SMOTE) is widely adopted to generate synthetic samples of the minority class. While effective in improving class balance, prior studies caution that SMOTE may affect interpretability by skewing feature importance. Oversampling can artificially inflate certain features, raising concerns about the stability of explanations. To investigate such issues, XAI methods have been applied alongside oversampling to provide transparency into model behavior. Patil et al. (2020) and Nobel et al. (2024) studied explanations on oversampled datasets in fraud detection, while Sghaireen et al. (2022) applied a similar approach to medical data for metabolic syndrome diagnosis. These studies showed that oversampling did not distort feature correlations and that the most important predictors remained consistent while improving model performance.

Recent advances extend beyond evaluation to XAI-guided augmentation strategies. Mersha et al. (2025) proposed a context-aware data augmentation framework that modifies less critical features while preserving task-relevant ones based on explainability-driven insights. Similarly, Kwon and Lee (2023) leveraged word-importance scores for text data augmentation, using them to create “soft labels” when mixing words across sentences. These approaches illustrate how XAI can move beyond post-hoc explanation to directly inform the data generation process.

Finally, understanding class imbalance requires not only corrective measures but also the interpretation of data properties themselves. Dablain et al. (2024) proposed a framework that reveals data complexities inherent in imbalanced datasets to better understand class exemplars that are key to model performance. By examining global data properties, XAI methods can offer a more holistic view of how spurious features cause model confusion.

Data labeling and explainability

A central challenge in MLOps is the difficulty of accurately and efficiently labeling the massive datasets required to train models. Manual annotation is costly and time-consuming, particularly in safety-critical domains. Automated labeling aims to address this bottleneck, but its utility depends not only on accuracy and scalability, but also on explainability. Research in this area demonstrates how explainability can validate auto-labeling quality and build trust in automated annotations.

Medical domain exemplifies this challenge, where clinicians require interpretable rationales to trust automated labels. Two notable contributions in medical data analysis highlight this intersection. De Oliveira et al. (2020a) proposed incorporating explainability as an inherent part of the automated labeling process for medical event logs. This approach leverages an autoencoder's decoder to reconstruct data from a compressed representation. Analysis of the reconstructed output helps to identify the most representative medical events and codes of a given label, offering an interpretable rationale for the auto-labeled outputs. Building on this, Kim et al. (2022) introduced an automated labeling approach for chest X-ray images using a pre-existing, validated XAI model as an “atlas.” In this framework, unlabelled images are compared against known explainable features from the reference dataset. A probability-of-similarity metric quantifies the degree of alignment between the new image and the known features, and labels are assigned only when this similarity is sufficiently high. These studies underscore the potential of explainability-enabled auto-labeling frameworks, which not only accelerate dataset creation but also preserve transparency in the early stages of the MLOps pipeline.

3.1.2 Data pre-processing

Within data pre-processing, we examine how explainability ensures transparency across dimensionality reduction, clustering, feature engineering, and feature selection processes. Related publications for each touchpoint are summarized in Table 2.

Dimensionality reduction and explainability

The primary role of dimensionality reduction is to decrease data complexity, improving model performance and enabling more efficient computation, management, and visualization, all while retaining much of the original information. Beyond efficiency, recent work has explored how dimensionality reduction itself can be explainable. Our analysis reveals two distinct roles of explainability in dimensionality reduction: explaining the results and explaining the process.

Explaining the results focuses on interpreting the output of dimensionality reduction and clarifying why data points are positioned in specific locations within the reduced space. Marćılio-Jr and Eler (2021) employed Shapley values to reveal which original features most influenced each point's embedding position, while Bardos et al. (2022) developed a local explanation method to interpret neighborhood structures in reduced representations.³Mylonas et al. (2024) further analyzed interpretability patterns across different dimensionality reduction techniques to validate if reduced representations preserve meaningful structure.

Explaining the process focuses on making the dimensionality reduction mechanism itself transparent, revealing how and why certain features or dimensions are selected or transformed. Das et al. (2023) used feature importance insights to guide which dimensions to preserve during reduction to maintain prediction accuracy while reducing complexity. Zang et al. (2022) designed an inherently interpretable deep network where the reduction process itself is transparent, generating explanations that reveal how the algorithm transforms high-dimensional data into lower-dimensional representations.

Clustering and explainability

Clustering methods group data points without predefined labels, and their utility often depends on providing a rationale for why a point belongs to a particular cluster. We observe that research on explainable clustering addresses two functional roles: explaining clustering results and explaining clustering processes.

Explaining clustering results applies XAI techniques to elucidate why data points are grouped together and what characterizes each cluster. A common approach is to use decision trees to explain the results of clustering algorithms such as k-means and k-medians. The core idea is to train a decision tree to predict cluster assignments, with each path representing logical rules that describe cluster membership. This method has been studied from a theoretical perspective (Moshkovitz et al., 2020; Makarychev and Shan, 2021; Bandyapadhyay et al., 2023) and applied in practice, e.g., to explain hydrochemical time series in water quality analysis (Thrun et al., 2021), multidimensional prototypes in steel manufacturing (Bobek et al., 2022), and customer segmentation in retail (Khan et al., 2021). These applications show how decision trees provide human-understandable explanations of cluster characteristics. Complementing decision trees, rule-based methods generate “if–then” statements that summarize the characteristics of each cluster. Loetsch and Malkusch (2021) used such rules to explain pain phenotypes, while Prabhakaran et al. (2022) applied to occupancy patterns, translating multidimensional data into simple rules that are easily understandable by domain experts. One line of research has shifted toward advanced clustering techniques leveraging deep learning algorithms. Ahmed et al. (2022) employed an attention-based neural network for mental health treatment, where attention weights highlight influential features for each cluster. Guan et al. (2020) and Kauffmann et al. (2022) applied deep learning models to text and image data, identifying the most important features in latent space to explain cluster structures. These methods aim to capture deeper insights into complex, non-linear cluster structures.

Explaining clustering processes embeds explainability directly into clustering algorithms, making the grouping mechanism itself transparent. Some works propose inherently interpretable clustering models that embed explainability into the algorithm's design. Methods by Saisubramanian et al. (2020) and Sambaturu et al. (2020) are built to optimize both clustering performance and interpretability by generating a concise set of cluster descriptions. Similarly, Gad-Elrab et al. (2020) and Wickramasinghe et al. (2021) developed graph-based clustering models with explainability mechanisms integrated into the learning objective.

Feature engineering and explainability

Feature engineering, the process of creating new features from existing ones, has long been used to incorporate domain knowledge into training data (Roscher et al., 2020). Recent research integrates explainability into this process, addressing two functional roles: guiding feature engineering and assessing engineered features.

The first line of work integrates explainability directly into the creation of new features. Instead of treating feature engineering as a black-box that just improves model performance, these approaches aim to make the process and the resulting features understandable. Li and Yang (2021) used domain knowledge in the ironmaking industry to construct inherently explainable features, while Chatzimparmpas et al. (2022) introduced a visual analytics tool that allows users to observe and interpret feature construction step by step. Similarly, Leung et al. (2021) applied explainable data analytics in healthcare to ensure engineered features remain clinically interpretable.

The second line of work applies explainability methods to evaluate the contribution of new features. Hrnjica and Softic (2020) and Trost et al. (2025) used XAI to assess how engineered features, such as sensor readings or material properties, influences predictions in manufacturing and materials science. Waldis et al. (2020) focused on text-based concept recognition, showing how new text features affected a model's ability to identify specific concepts.

Feature selection and explainability

Feature selection aims to identify the most informative variables for a task while reducing redundancy and improving efficiency. The literature reveals two main roles: ranking feature importance and enabling transparent feature selection.

Among approaches for ranking feature importance, SHAP has emerged as the most prominent method, leveraging a trained model's own learned patterns to guide feature selection. The process typically involves training a model, computing SHAP values, ranking features, and then selecting a subset (Marćılio and Eler, 2020; Zacharias et al., 2022; Shafin, 2024). This yields leaner, faster, and often more accurate models trained on only the most influential features. SHAP-based feature selection has been applied across domains, including DNA promoter prediction (Le et al., 2022), data center optimization (Gebreyesus et al., 2023), and fault diagnosis in rolling bearings (Santos et al., 2024). It also serves as a tool to validate domain-specific hypotheses, confirming that models rely on features meaningful to experts. Comparative studies show SHAP explainability can also be used for model optimization by eliminating redundant features without sacrificing accuracy (Li et al., 2024). However, (Fryer et al. 2021) cautioned that SHAP is not a guaranteed solution, highlighting issues such as computational cost, instability, and feature correlation.

To enable transparent selection, feature selection methods integrate explainability into the selection process itself. Renau et al. (2021) demonstrated that extreme feature selection can produce simple, human-understandable rules by restricting models to only a few highly relevant features. Rabiee et al. (2024) incorporated expert-augmented domain-specific knowledge to ensure that selected features are both statistically significant and meaningful. Vlahek and Mongus (2021) introduced an iterative process that reveals feature importance dynamically, while Ma et al. (2023) embedded explainability into task-oriented communications by selecting only semantically relevant features. Collectively, these works highlight a move away from a single, universal explanation method toward a more diverse toolkit of explainable feature selection strategies.

3.1.3 Data management

Data management encompasses two touchpoints: data privacy protection with explainability and data sharing with explainability. Table 3 summarizes the corresponding publications.

Data privacy protection and explainability

As AI systems are increasingly deployed in sensitive domains, balancing fairness, interpretability, and privacy has become a critical challenge for both research and practice. Grant and Wischik (2020) argued that in many contexts, meeting one legal requirement prevents full compliance with another, creating a “zero-sum game”+ where gains in explainability often come at the expense of privacy. Recent work emphasizes that ethical AI, responsible data engineering, and privacy preservation must be addressed collectively from the very beginning of the data pipeline (Muvva, 2021; Ferry et al., 2025). In this context, explainability serves two functional roles: assessing privacy risks and enabling privacy-preserving explanations.

A central paradox emerges as the very properties that make explanations useful (e.g., revealing influential features or decision boundaries) can also expose sensitive information. Explanations act as a potential side channel for privacy attacks, particularly membership inference attacks (Shokri et al., 2021; Liu et al., 2024), where adversaries infer whether specific data points were included in training. Some explanation types carry a higher risk than others, e.g., counterfactual explanations show the smallest change needed to alter a prediction. As shown by Goethals et al. (2023), these can be exploited through explanation linkage attacks, which leverage the proximity of the counterfactual to the original data point to infer sensitive attributes about it. Similarly, Naretto et al. (2025) showed that local explainers, which provide detailed insights into individual predictions, pose greater privacy risks than global explainers offering aggregated summaries.

Protecting sensitive information without compromising explanation quality is a difficult trade-off. Data anonymization through masking can enhance privacy but also distort explanation outputs. Bozorgpanah et al. (2022) showed that applying such mechanisms reduces the stability and accuracy of Shapley-value explanations, illustrating a measurable tension between privacy and interpretability. Several strategies have been proposed to reconcile this trade-off. Federated and distributed learning allow model training without centralizing sensitive data, applied in domains from fraud detection (Awosika et al., 2024) to healthcare (Raza et al., 2022). These methods, however, face the challenge of providing consistent explanations, which Bogdanova et al. (2023) addressed with a specialized DC-SHAP method. Another strategy involves case-based and generative approaches, where privacy-preserving GANs are used to produce representative but non-sensitive data points for explanation (Montenegro et al., 2021, 2022). Data-level protection has also been explored through techniques such as explainable image compression in medical imaging (Puiu et al., 2021; Gaudio et al., 2023). Finally, holistic frameworks attempt to integrate fairness, privacy, and explainability into unified models, such as proposed by Franco et al. (2021) for face recognition.

Data sharing and explainability

Explainability in data sharing ensures that the data-sharing process is transparent and justifiable to all stakeholders. It provides clarity on why data is needed and how it will be used. Supporting this requires technical, procedural, and legal safeguards, including formal agreements and data lineage tools for auditability, anonymization, and secure transfer protocols. Nevertheless, reluctance to share data remains a key obstacle in domains such as healthcare and industry, where concerns over misuse, liability, and confidentiality hinder collaboration. Starke et al. (2023) cautioned that explainability alone cannot compensate for inadequate data sharing. They note that without reliable and accessible datasets, explanations risk becoming a superficial “fig leaf” that masks deeper methodological weaknesses. Qian et al. (2023) highlighted tools such as federated learning and differential privacy to bridge the gap between data accessibility and protection. Within these frameworks, explainability serves two functional roles: explaining federated models and supporting secure data sharing.

Federated learning enables collaborative model training without sharing raw data, thereby preserving privacy. Sánchez et al. (2024) introduced FederatedTrust, a framework that combines federated learning with explainability to ensure reliability, accountability, and transparency in federated systems. Healthcare has been a major application domain where Raza et al. (2022) developed an ECG monitoring system using federated transfer learning, allowing hospitals to jointly train robust models without exchanging patient data, and the explainability module ensures interpretability for diagnoses for clinicians. Zhao et al. (2024) proposed a general explainable federated learning scheme for secure clinical data sharing, further addressing the dual need for privacy and interpretability. Similar ideas extend to finance, where Mavrogiorgou et al. (2023) presented FAME, a decentralized data marketplace using federated learning for embedded finance, enabling institutions to share data securely while maintaining transparency and interpretability of derived insights. Beyond federated learning, alternative frameworks also integrate secure sharing with explainability. Olaya et al. (2022) proposed a container-based platform to enhance trust through data traceability and explainable outputs in earth science research.

3.2 Model development

Within model development, we examine how explainability informs model selection and supports pre-deployment auditing through robustness testing and bias detection. Related publications for each touchpoint are summarized in Table 4.

3.2.1 Training

Model selection and explainability

Explainability in model selection extends evaluation beyond predictive accuracy, ensuring that models are both effective and interpretable. This involves generating explanations for candidate models and using them to inform final selection. The literature highlights two main functional roles of explainability: guiding offline and real-time model selection.

In offline settings, Haagen et al. (2023) and Fischer and Saadallah (2024) proposed automated frameworks that integrate explainability into model selection. Haagen et al. (2023) measured interpretability through metrics such as compactness and explanation stability, flagging models with unstable explanations as unreliable, which can be used to guide the debugging process. Fischer and Saadallah (2024) introduced a resource-aware method that leverages prior knowledge to estimate model suitability, thereby reducing the computational burden of multi-objective selection. Shekkizhar and Ortega (2021) applied a polytope interpolation framework to explain neural networks, using instance-based explanations to identify problematic patterns and improve generalization. In healthcare, Berloco et al. (2024) showed that when models perform similarly, those with more reliable explanations are preferred by clinicians. Tavares et al. (2022) and Tavares et al. (2025) further advanced adaptive, variability-aware approaches that ensure selected models remain robust across different tasks, while also identifying model limitations for debugging.

In real-time contexts, candidate models are deployed in parallel and continuously evaluated as new data arrive. This is particularly important for non-stationary time series data, where patterns evolve over time. Explainability provides transparency for why a specific model is favored at a given moment. Tomar et al. (2022) introduced a prequential selection method using saliency maps to identify the best forecaster for specific data segments. Given the computational demands of maintaining explanations in streaming environments, Saadallah (2023) and Jakobs and Saadallah (2023) proposed adaptive frameworks that reduce latency by identifying the “regions of competence” for each model, focusing evaluation only on relevant candidates. These systems enable dynamic switching between models while maintaining accuracy and interpretability.

3.2.2 Pre-deployment auditing

Robustness and explainability

Robustness and explainability operate as mutually reinforcing pillars of trustworthy AI (Chander et al., 2025; Hamon et al., 2020). While explainability has primarily been leveraged as a tool to identify and address vulnerabilities, thereby improving robustness, evidence also suggests that robustness itself can contribute to more meaningful and reliable explanations. This bidirectional relationship manifests in two functional roles: detecting model weaknesses and enhancing explanation reliability.

Regarding detecting model weaknesses, explanations act as diagnostic tools, revealing why models fail on certain inputs and exposing reliance on irrelevant or spurious features, which are commonly exploited by adversarial attacks. Hartl et al. (2020) demonstrated this in the context of Recurrent Neural Networks (RNNs), applying a relevance attribution method to identify critical input sequence elements. The analysis revealed vulnerabilities that could then be addressed through retraining, leading to more robust models. Sun R. et al. (2023) extended this approach by proposing an explainability-guided, model-agnostic framework for testing malware detectors against adversarial attacks, where explanations serve as a means of stress-testing and improving system robustness. Similarly, Awal et al. (2025) employed explainability to investigate adversarial attacks in software analytics, showing how explanations can reveal an over-reliance on non-critical code features that attackers exploit to cause misclassifications. Other studies highlight explainability as a method for error analysis and feature robustness. Singla et al. (2021) argued that models fail when they rely on “brittle” features, i.e., unstable representations that are easily manipulated. Their framework uses explainability to identify and analyze such features, showing how explainability can guide training with more robust representations. Taesiri et al. (2022) established a more direct link between explainability and robustness by embedding the explanations into the prediction process itself. This integration forces the model to learn deeper, more meaningful representations of visual data.

The reverse relationship, i.e., enhancing explanation reliability, has received comparatively less attention but holds significant potential. The central idea is that by making a model robust, forces it to learn more meaningful and stable features that in turn yield higher-quality explanations that faithfully reflect the underlying data. Augustin et al. (2020) hypothesized that improvements in adversarial robustness naturally enhance explainability because robust models are compelled to develop more human-like representations. As a result, the explanations they generate are more aligned with meaningful aspects of the input.

Bias detection and explainability

Explainability plays a crucial role in detecting and mitigating bias in AI systems. By providing transparency into decision-making mechanisms, explainability techniques can uncover subtle biases that might otherwise go unnoticed, thereby supporting fair and equitable outcomes (Para, 2024). At the same time, Deck et al. (2024) cautioned that claims about XAI's fairness benefits are often vague and overly simplistic. They argue that XAI should be understood as one tool within a broader fairness toolkit, emphasizing the importance of specifying which method is used, which fairness objective it addresses, and which stakeholders it is intended to benefit. Without such specificity, the claimed advantages of XAI risk being overstated or misinterpreted. With this critical lens in mind, research on bias detection through XAI addresses two functional roles: detecting data bias and detecting model bias.

Data-level bias arises from imbalances or spurious correlations present in the training data itself. Explainability methods can expose these issues even before model training by highlighting which features drive predictions in proxy or simplified models. Mikołajczyk et al. (2021) used counterfactuals and global explanations to uncover spurious correlations in skin lesion images, where models relied on irrelevant artifacts (e.g., black frames) instead of lesion features. Similarly, Palatnik de Sousa et al. (2021) applied XAI techniques to COVID-19 CT-scan classifiers and found that models with high accuracy metrics were still susceptible to biases toward spurious artifacts in the images. Together, these studies underscore the dual role of explainability in identifying both model-level and data-level biases.

Model-level bias refers to biases inherent in a model's decision-making logic. XAI methods reveal whether a model disproportionately relies on sensitive attributes (e.g., gender, age). Although explanation methods do not directly measure bias, they provide evidence of biased behavior that standard fairness metrics may overlook. Van Stein et al. (2023) introduced Deep-BIAS, a framework that predicts the type of structural bias in optimization algorithms and explains its causes, enabling developers to correct design flaws. Tavares et al. (2023) extended model selection to include bias detection, ensuring that chosen models remain fair across varying data contexts. In natural language processing (NLP), Manerba and Guidotti (2021) presented FairShades, which applies counterfactual explanations to audit abusive language detection systems, illustrating how XAI can proactively mitigate fairness concerns in NLP.

3.3 Deployment

3.3.1 Developer oversight

Developer oversight addresses three explainability touchpoints: logging, monitoring, and data filtering. Table 5 summarizes the corresponding publications.

Logging and explainability

The scale of modern distributed systems produces log data at overwhelming volume and velocity, making storage, processing, and analysis both technically and financially demanding. As Cândido et al. (2021) noted, voluminous, unstructured, and context-poor records are a pervasive challenge across industries. A persistent tension exists between logging too much data, which overwhelms systems, and logging too little, which undermines debugging and monitoring. While automated approaches to logging have been explored, the role of explainability in logging remains largely overlooked.

Khosravi Tabrizi et al. (2024) addressed this gap with an Adaptive Logging System (ALS) that uses reinforcement learning to decide what information should be logged. The system balances the need for sufficient detail to support debugging and performance analysis against the costs of excessive logging. Although this adaptive approach marks progress toward more intelligent and efficient logging practices, its decision-making remains opaque. While it offers some degree of insight into system decisions, the reinforcement learning mechanism functions as a black-box, limiting transparency into why specific information is logged or omitted.

Monitoring and explainability

Post-deployment monitoring with explanations involves tracking both model predictions and their explanations over time. Integrating explainability into monitoring enables the detection and diagnosis of hidden failures, offering transparency into why performance changes occur (Karval and Singh, 2023). This diagnostic capability emerges through four functional roles: explaining silent failures, explaining concept drift, explaining data drift, and explaining anomaly detection.

Silent failures encompass a range of issues that cause a model's performance to degrade without an explicit error message. Concept drift, data drift, and anomalies are among the most common causes of such failures. De Oliveira et al. (2020b) demonstrated this principle in mining by applying explainability to classify and correct incomplete event logs. Their work illustrates how explainable AI can enhance system integrity by making hidden errors detectable and correctable. Similarly, Koebler et al. (2023) proposed Explanatory Performance Estimation (XPE), monitoring performance while attributing performance changes to interpretable features, thereby enabling targeted interventions rather than default retraining. Shahad and Raj (2024) advanced further with an interpretability-based virtual drift detection and adaptation algorithm, which uses interpretability to detect subtle changes in data distribution, diagnose root causes, and adapt in real time.

Concept drift arises when the relationship between inputs and outputs changes, meaning the “concept” a model has learned is no longer valid. The survey by Hinder et al. (2024) confirmed that detecting and explaining concept drift is a growing research focus, with the community moving beyond simple detection to diagnosis and explanation. Vieira et al. (2021) introduced a multi-agent framework, Driftage, to analyse data streams and diagnose drift. While the work is not primarily focused on explainability, their approach offers an indirect interpretability. Shaer and Shami (2024) applied explainable concept drift to thwart cybersecurity attacks. In contrast, work on data drift focuses specifically on changes in the statistical properties of input data over time. Duckworth et al. (2021) provided a real-world application of this approach in monitoring emergency department admissions during COVID-19. They used explainable machine learning to not only identify that patient data was changing (e.g., demographics, symptoms) but also to characterize what was changing and how that data drift correlated with emergent health risks. Decker et al. (2024) expanded this perspective with a general framework for explanatory model monitoring, using XAI to clarify how shifts in input features drive changes in model performance.

Finally, anomaly detection involves identifying unusual or unexpected data points that may indicate data corruption, malicious activity, or critical system errors. The literature shows a strong trend toward integrating post-hoc explainability, particularly SHAP values, into anomaly detection. Park et al. (2020) demonstrated how SHAP could explain anomalies in district heating systems, laying the foundation for its adoption across domains such as sensor fault detection (Hwang and Lee, 2021), hot-rolling processes (Jakubowski et al., 2021), maritime main engines (Kim et al., 2021), predictive maintenance (Choi et al., 2022), and industrial control systems (Hoang et al., 2022; Huong et al., 2022). More recent work has extended explainable anomaly detection to complex data types and domains, including video (Wu et al., 2022), healthcare monitoring (Abououf et al., 2023), IoT (Gummadi et al., 2024), and autonomous driving (Nazat et al., 2024). Some approaches move beyond post-hoc methods by embedding explainability into the model architecture itself. Szymanowicz et al. (2022) and Cho et al. (2023) proposed neural networks that generate discrete or prototypical representations, allowing anomalies to be explained by their resemblance to known concepts. Similarly, Lee et al. (2024) introduced masked latent generative modeling, and Jin et al. (2025) developed LogicAD, both of which build inherently interpretable internal representations. Lee et al. (2023) further contributed DuoGAT, a dual graph attention network that enhances both accuracy and interpretability in time-series anomaly detection by highlighting relationships between data points.

Data filtering and explainability

Typical MLOps pipelines often rely on scheduled retraining, which may require acquiring new data alongside fine-tuning. However, collecting every new data point can be both expensive and time-consuming. Explainability offers a more strategic solution by guiding data filtering to inform retraining decisions. Research in this area addresses two functional roles: informing data acquisition decisions and explaining data filtering decisions.

Research focusing on informing data acquisition uses explainability to optimize which data to collect, addressing the challenge of determining which sensors, features, or measurements are most informative before or during data collection. This helps identify which new data points should be prioritized for model updates in cases of concept or data drift. Pandiyan et al. (2023) used feature importance for in-situ monitoring in manufacturing, identifying the most critical sensor data for quality control. This allows for more efficient and focused monitoring, reducing the amount of data that needs to be collected and processed. Guney et al. (2025) proposed an explainability-driven framework for dynamic feature acquisition. Their method employs local explanations to rank features for a given instance, with a policy network determining which features to acquire next. This approach filters out irrelevant data, leading to more efficient and accurate predictions.

Research focusing on explaining data filtering decisions applies explainability to validate data that has undergone filtering or preprocessing. Maulana et al. (2023) combined explainable data-driven methods with Bayesian filtering for remaining useful lifetime prediction in aircraft engines, using explainability to interpret how filtered sensor readings contribute to degradation predictions and ensuring that the filtering process preserves critical prognostic information.

3.3.2 End-user interfacing

This section discusses four key areas of explainability within end-user interfacing: decision support, post-deployment auditing, trust calibration, and human-in-the-loop systems. Table 6 summarizes the corresponding publications.

Decision support and explainability

Explainability in decision support systems (DSS) provide interpretable evidence for machine-generated outcomes, enabling human decision-makers to validate or override model outputs. Recent literature reflects a growing interest in applying XAI-enabled DSS across domains such as medicine, manufacturing, and education (Kostopoulos et al., 2024). However, as Antoniadi et al. (2021) highlighted, there remains a considerable gap between the theoretical benefits of XAI and its practical validation, particularly in clinical settings, with respect to understanding the needs of clinicians and their interactions with these systems. On the technical side, Ezeji et al. (2024) underscored the trade-offs between explainability and computational efficiency, stating that explanations often impose costs in processing power, memory, and latency. For real-time decision support, computationally lightweight explainers may therefore be more practical than more resource-intensive methods. Despite these challenges, research demonstrates diverse applications of XAI in DSS, which we categorize by their primary decision-support function: knowledge-driven, communication-driven, data-driven, and document-driven decisions.

Knowledge-driven DSS, also known as expert systems, embed domain expertise through rules, facts, and procedures, with ML models acting as the core of the system. Early work by Sachan et al. (2020); Hamrouni et al. (2021) focused on integrating explainability into ontology- and rule-based systems for domains such as loan underwriting and sustainable business modeling. This has been prominent in clinical applications, where explainable models support trust, personalization, and evidence-based reasoning for conditions ranging from gestational diabetes to neurological disorders (Schoonderwoerd et al., 2021; Amann et al., 2022; Du et al., 2022; Pierce et al., 2022; Valente et al., 2022; Niu et al., 2024; Gombolay et al., 2024). Aiosa et al. (2023) further demonstrated the utility of explainable diagnosis support in managing comorbidities. Beyond healthcare, XAI-enabled DSS have been applied to industrial and supply chain contexts (Tiensuu et al., 2021; Olan et al., 2025; Sadeghi et al., 2024), where interpretable recommendations have been shown to improve quality management, resilience, and agility in decision-making.

Communication-driven DSS facilitate collaboration among decision-makers and emphasize explanations that are interpretable across diverse stakeholders. Research in this area examined how explanations enhance collaboration in patient education and long-term behavior change (Woensel et al., 2022), as well as understanding how a user's domain expertise influences their trust in the system's recommendations (Bayer et al., 2022). Other work explored novel communication channels, such as augmented reality for smart environments (Zheng et al., 2022) or visual approaches to explain clinical decisions (Müller et al., 2020). Studies also highlight the role of explainability in team dynamics, e.g., improving efficiency in rapid triage decisions (Laxar et al., 2023) and the importance of communicating confidence measures (van der Waa et al., 2020). A co-design perspective has been advocated (Panigutti et al., 2023), complemented by evidence-based approaches to ensure explanations remain human-centered and effective in a collaborative setting (Famiglini et al., 2024).

Data-driven DSS emphasize the analysis of large-scale data in data mining, predictive analytics, and forecasting, and explainability transforms predictive outputs into actionable and trustworthy insights. Applications span critical domains such as aviation safety (Midtfjord et al., 2022) and IoT-based predictive maintenance (Sayed-Mouchaweh and Rajaoarisoa, 2022), where operators rely on interpretable outputs to validate forecasts in dynamic conditions. In business contexts, explainable DSS enhance strategic decision-making in areas such as credit risk assessment (Nallakaruppan et al., 2024), employee attrition analysis (Maŕın D́ıaz et al., 2023), customer development (Onari et al., 2024), energy management (Panagoulias et al., 2023), and business process analytics (Galanti et al., 2023).

Finally, document-driven DSS address decision-making challenges in unstructured data found in documents, web pages, and other text-based sources. These systems analyse documents to extract, classify, and summarize relevant information by leveraging natural language processing (NLP) models. Kim et al. (2020) investigated explainability in CNN-based document analysis, enabling users to see which words, phrases, or textual patterns influenced the system's conclusions.

Post-deployment auditing and explainability

Explainability plays a central role in error analysis by making model failures more interpretable and actionable. Explanations for misclassified or high-loss predictions reveal which features contributed most to errors, helping practitioners diagnose systematic patterns that inform feature engineering, data collection, and model refinement. Beyond technical improvement, these explanations establish accountability by documenting decision rationale for stakeholders and regulators. Research on post-deployment auditing addresses these dual needs through two functional roles: enabling error analysis and enabling accountability.

Several studies explore the role of explainability in post-deployment error analysis across diverse domains to diagnose the root causes of model mistakes. Gerling et al. (2022) applied AutoML tools in manufacturing to trace erroneous predictions back to feature contributions, uncovering how certain variables systematically influenced failures. Hallé (2020) focused on explainable queries over event logs, enabling more transparent debugging of process-related errors. In education, Hlosta et al. (2022) applied predictive learning analytics with explanations to identify systematic patterns in student misclassifications and performance issues. Pai et al. (2024) extended this to healthcare by using explainable analytics to investigate misdiagnoses, showing how explanations pinpoint which features most influenced incorrect outcomes. Similarly, Mortezaagha et al. (2025) developed methods for detecting inconsistencies in cancer data classification, using XAI to pinpoint errors tied to specific feature contributions.

A growing body of work situates explainability within the broader context of accountability, emphasizing its role in ensuring transparency, fairness, and compliance in post-deployment AI systems. This is particularly crucial for decisions in regulated sectors such as finance and healthcare, where decisions must be supported by formal, auditable rationales. Several studies propose iterative and learning-theoretic approaches to online fairness auditing, where explanations guide refinement and verification of algorithmic behavior (Maneriker et al., 2023; Yadav et al., 2024). Shulner-Tal et al. (2022) and Shin (2020) have examined how different explanation methods influence non-expert users' perceptions of fairness and accountability, highlighting the importance of human-centered design in auditing frameworks. In regulated domains, explainability supports accountability in critical decision-making, from interpretable classifiers for cancer diagnosis (Sabol et al., 2020) to fairness and transparency in banking and credit scoring (Mariotti et al., 2021; Bücker et al., 2022). Extensions into forecasting and risk detection by Buczak et al. (2022) further illustrated how explainable auditing can provide actionable evidence for anticipating disruptive events. Smith-Renner et al. (2020) and Webb et al. (2021) linked accountability to interactive feedback and learning, arguing that explanations must be actionable and adaptable to shifting contexts.

Parallel research explored explainability as a tool for institutionalizing accountability within AI governance. Li and Goel (2025); Zhang et al. (2022a); Cen and Alur (2024), and Zhong and Goel (2024) emphasized the auditability of AI systems, showing how explainable outputs can be harnessed for formal oversight, compliance, and assurance in high-stakes settings. At the regulatory level, Nannini et al. (2024) discussed the operationalization of XAI in the European Un ion (EU), while Simuni (2024) framed explainability as a pathway toward enforceable accountability standards.

Trust calibration and explainability

Trust calibration refers to the process by which users learn to trust an AI system in proportion to its actual reliability, aligning their confidence with the system's true capabilities. In this context, explainability is not only a technical feature but also a socio-technical mechanism for fostering appropriately calibrated trust. Atf and Lewis (2025) confirmed that explainability and trust are positively, though variably, correlated across domains, while Ferrario and Loi (2022) cautioned that transparency should not be treated as a blanket justification but rather as a tool for supporting trust alignment. Research on trust calibration through explainability addresses two functional roles:understanding cognitive factors and supporting team collaboration.

From a cognitive perspective, explainability plays a crucial role in shaping how users perceive and adjust their trust in AI systems, independent of the systems' technical properties. The key contribution of XAI in this area is not simply to increase trust, but to enable users to form more accurate and robust mental models of the AI's capabilities. Naiseh et al. (2021a,b) demonstrated that explainable recommendations reveal how cognitive biases and systematic user errors affect trust calibration, underscoring the importance of designing explanations that prevent both over- and under-trust. Xu and Wang (2024) further showed that explainability enhances trust resilience, enabling users to maintain appropriate confidence even when AI systems make mistakes, as explanations clarify the reasons behind errors.

A parallel line of research situates explainability within the context of human–AI collaboration, with a particular emphasis on trust calibration and its impact on team effectiveness. Tomsett et al. (2020) introduced interpretable, uncertainty-aware approaches for rapid trust calibration, demonstrating that explanations combined with confidence cues align user trust with system reliability. Zhang et al. (2020) examined how confidence information and explanations jointly influence accuracy and trust calibration in AI-assisted decision-making. Chen et al. (2024) added a temporal dimension, demonstrating that the timing of explanations affects user perceptions and trust, highlighting that explanations must be integrated dynamically rather than as static outputs. Wintersberger et al. (2020) explored adaptive interfaces for automated driving, showing how personalized interactions foster effective human–automation collaboration. Naiseh et al. (2023) expanded the discussion by comparing different classes of explanations, evaluating their differential impact on trust calibration. Finally, Zafari et al. (2024) have taken a longitudinal perspective, studying how explainability supports the development and evolution of trust in personalized assistive systems over time.

Human in the loop systems and explainability

A human in the loop (HITL) system integrates human intelligence into an AI or machine learning process to improve algorithmic outcomes. Explainability plays a bidirectional role in such systems: enabling correction ((human-to-AI) and facilitating knowledge transfer (AI-to-human).

For enabling correction, explanation helps humans understand why an error occurred, enabling human interventions that both rectify immediate mistakes and provide targeted feedback, which in turn can be used to improve model performance. Li et al. (2020) introduced a probabilistic model checking approach that allows humans to meaningfully intervene “on the loop,” while Coma-Puig and Carmona (2021) applied explainability to enhance non-technical loss detection by allowing humans to validate and refine suspicious cases. Similarly, Nakao et al. (2022) explored fairness in interactive HITL systems, fostering end-user engagement in identifying and mitigating bias. Estivill-Castro et al. (2022a,b) advanced this idea by embedding interpretability directly into classifiers and enabling iterative human feedback during model training. Other contributions extend these principles across domains: Sun T. S. et al. (2023) designed a feedback loop for recalibrating convolutional neural networks via local explanations, Zhang et al. (2022b) applied HITL explainability to dynamic digital twins, Kotsiopoulos et al. (2024) demonstrated its use in industrial defect recognition, Assadi and Safaei (2024) improved interpretable recommendation engines through user corrections, and Shin et al. (2025) explored strategies to sustain user engagement in interactive learning loops.

HITL systems also leverage explainability for knowledge transfer. Holzinger et al. (2023) demonstrated this by integrating of domain knowledge graphs into explainable federated deep learning, allowing experts to both validate and learn from model reasoning. Young et al. (2025) proposed SPARC, a HITL framework where spatial concepts are explained in interpretable ways, enabling humans to understand new spatial concepts learn by the model. Similarly, Tsiakas and Murray-Rust (2022) highlighted how HITL systems, supported by explainability, can foster new work practices by transferring machine-discovered insights into human decision-making.

4 Lifecycle-wide connection of XAI (RQ2)

This section examines how explainability touchpoints are connected across stages of the MLOps lifecycle. We classified the reviewed studies according to whether cross-phase connections were explicitly addressed (clear integration across lifecycle stages), implicitly suggested (cross-phase effects inferred but not formalized), or isolated (no cross-phase consideration). Figure 5 visualizes these relationships, mapping explainability touchpoints to MLOps phases, where explanatory information propagates across stages. Touchpoints on the left are linked to the primary lifecycle phases; information from each phase flows back to the touchpoints (indicated by asterisks) to capture how it flows back to relevant touchpoints. Overall, the literature is dominated by single-phase analyses, with limited attention to lifecycle-wide integration. When cross-phase links are discussed, they mainly reflect how upstream processes influence downstream decisions, with limited focus on how deployment-time observations trigger data and model updates.

Figure 5

In regard to data handling, literature shows that methods, such as missing data imputation and dimensionality reduction, significantly influence final model accuracy and performance (Azimi and Pahl, 2021; Das et al., 2023). Explanations generated at this stage, therefore, provide actionable signals for subsequent modeling decisions, rather than merely retrospective justification. Decisions made during feature selection and feature engineering establish the feature dependency baseline (Li and Yang, 2021; Leung et al., 2021). Feature importance estimates derived from explainable feature selection methods (Shafin, 2024; Le et al., 2022) directly inform model selection.

In the model development phase, recent works demonstrate model selection as a bridge between development and deployment. Tomar et al. (2022)'s prequential selection uses real-time explanations to trigger adaptive model switching in production, establishing continuous model-deployment feedback. Fischer and Saadallah (2024) embedded explainability as a multi-objective optimization criterion during offline selection, ensuring deployed models meet operational transparency requirements from the outset. Similarly, Haagen et al. (2023) integrated interpretability constraints into AutoML pipelines, creating cohesive development-to-deployment workflows where explainability requirements inform model choice rather than being added as an afterthought. Robustness further illustrates the lifecycle-spanning role of explainability. Augustin et al. (2020) demonstrated that adversarial robustness directly impacts explanation fidelity and deployment trust, while Taesiri et al. (2022) showed that explanation-driven analysis of deployment failures can expose upstream data weaknesses, prompting corrective interventions in earlier lifecycle stages. Robustness analyses further reveal features vulnerable to adversarial perturbation (Hartl et al., 2020; Sun R. et al., 2023), yet auditing and monitoring remain largely disconnected in practice.

Explanations in deployment are particularly crucial for supporting the principles of Continuous Integration and Continuous Delivery (CI/CD) in MLOps. Studies on monitoring in production suggest that explanations derived from drift detection, performance evaluation, and user feedback can provide signals for data updates, feature engineering, and retraining (Vieira et al., 2021; Cho et al., 2023; Panagoulias et al., 2023). Explanations generated during monitoring can thus act as triggers for the CI pipeline. When a feature importance drops sharply, it signals the need to integrate a fix or deploy a new model. In most cases, these cross-phase connections are implied rather than explicitly operationalized. Explanations are interpreted as diagnostic signals, but the feedback loop into automated CI pipelines is rarely formalized.

In relation to other missing connections, literature on post-hoc interpretability highlights that explanations are widely used for debugging and verifying model reasoning, yet these insights are typically treated in isolation (Berloco et al., 2024; Shekkizhar and Ortega, 2021). Although such explanations could, in principle, inform upstream data handling by revealing redundant features, unstable representations, or data deficiencies, these insights are rarely operationalized into data pipelines. Similarly, by extending prior work on explainability in pre-deployment auditing (Awal et al., 2025; Tavares et al., 2025; Manerba and Guidotti, 2021; Sun R. et al., 2023), these outputs can directly guide deployment decisions, such as defining targeted monitoring strategies for sensitive features, setting confidence thresholds for human intervention, or adjusting specific alert mechanisms to mitigate identified robustness or bias risks. However, existing studies rarely formalize these connections. Likewise, decision-support explanations further illustrate unrealized opportunities for upstream feedback. Explanations in deployment highlight what should change in data pipelines, but do not define mechanisms by which deployment-time signals are translated into automated or governed data filtering and acquisition processes.

5 Assessment of XAI reliability and trustworthiness (RQ3)

This section examines how the reviewed studies assess the reliability and trustworthiness of explainability methods, focusing on the types of evaluation employed, their consistency across the MLOps lifecycle, and the remaining validation gaps that limit operational assurance. Following established XAI evaluation frameworks (Tekkesinoglu, 2023), we analyse the literature along two complementary dimensions: technical assessment and human-centric assessment. Technical assessment concerns properties of explanations, such as fidelity, stability, consistency, robustness, efficiency, complexity, and scalability, which are quantifiable metrics targeting technical users. Human-centric assessment captures how explanations are understood, trusted, and acted upon, encompassing comprehensibility, trust calibration, interactivity, actionability, correctability, user acceptance, and task performance. We evaluated each study based on which of these dimensions it addresses, providing a systematic identification of imbalances in evaluation practices and unresolved challenges in validating XAI for real-world deployment.

Figure 6 summarizes how technical assessment is conducted across the reviewed studies. Existing research predominantly focuses on robustness and efficiency, particularly within the data handling and model development phases. In contrast, fidelity, i.e., the degree to which explanations accurately reflect the model's internal reasoning, remains comparatively underexamined, with many studies implicitly assuming explanation correctness rather than explicitly validating it. A recurring evaluation strategy concerns feature attribution validation, which examines whether the features identified as important by explanation methods reflect those the model has learned. Methods such as sensitivity analysis, perturbation-based robustness, and ground-truth alignment recur frequently (Decker et al., 2024; Zamanian et al., 2023). Comparative analyses assessing feature importance consistency across ML models (Van Zyl et al., 2024; Cinquini et al., 2023; Mersha et al., 2025) and across XAI methods (Patil et al., 2020; Nobel et al., 2024) are commonly discussed. On the other hand, inherently interpretable models are evaluated through direct inspection of input-output relationships rather than relying on quantitative metrics (McTavish et al., 2024). Several studies further propose domain-specific evaluation methods (Kim et al., 2022; Song et al., 2021), while others introduce new metrics to address the limitations of conventional evaluation approaches. For instance, (Šimić et al., 2025) proposed the Consistency-Magnitude Index, which facilitates a faithful assessment of feature importance attribution.

Figure 6

Figure 7 summarizes the distribution of human-centric evaluation practices across the MLOps lifecycle. Compared to technical assessment, human-centered validation is more prevalent, particularly in the deployment phase. Comprehensibility is the most frequently assessed dimension, reflecting a strong focus on whether explanations are understandable at decision time. In contrast, deeper forms of human engagement, such as interactivity, actionability, and correctability, remain largely under-evaluated across all phases. Recent work increasingly incorporates human feedback into post-hoc validation, examining not only how explanations reflect model behavior but also how they shape user interpretation and decision-making (Famiglini et al., 2024; Thrun et al., 2021). These studies surface critical risks, such as misleading explanations, over-reliance, and misplaced trust. Overly confident explanations can induce over-reliance (Zhang et al., 2020), while inconsistent or delayed explanations undermine perceived competence (Tomsett et al., 2020; Xu and Wang, 2024). Human-centered studies show that explanation design directly affects trust calibration, cognitive workload, and decision quality (Schoonderwoerd et al., 2021; Laxar et al., 2023; Zafari et al., 2024; Papagni et al., 2023; Zhang et al., 2022b). Research further shows that poorly designed explanations may mislead users and amplify systematic errors (Atf and Lewis, 2025; Bayer et al., 2022; Naiseh et al., 2021b). Collectively, these works treat explainability not as transparency alone but as human cognitive ergonomics, where poor calibration can reduce rather than enhance system reliability.

Figure 7

Moreover, the literature increasingly recognizes the need for standardized evaluation frameworks to support compliance-aligned validation of XAI. The most concrete developments are evident in auditing research, where explainability is positioned as certifiable evidence rather than informal insight. XAudit framework demonstrates how explanations can serve as certifiable evidence for fairness and compliance, marking a shift from interpretability toward verifiable accountability (Yadav et al., 2024). Complementary studies (Li and Goel, 2025; Cen and Alur, 2024; Zhong and Goel, 2024) emphasize procedural transparency and institutional readiness for explainable, auditable AI systems. Broader discussions highlight moves toward standardized evaluation pipelines, though the lack of unified protocols remains a significant barrier (Nannini et al., 2024; Akhtar et al., 2024; Hussain and Hussain, 2025).

6 XAI practices for regulatory requirements (RQ4)

This section examines the degree of prospective alignment between the reviewed literature and EU AI Act requirements across the MLOps lifecycle, along with the level of operationalization, and remaining gaps (see Table 7). The selected AI Act provisions were chosen because they impose explicit or implicit transparency, traceability, robustness, and oversight requirements that directly intersect with explainability. We group these provisions into data handling, lifecycle-wide governance, and deployment. The analysis reveals alignment in terms of design intent and operational practice rather than demonstrated legal compliance. Regulatory alignment is assessed qualitatively based on whether explainability is framed as addressing AI Act requirements explicitly, indirectly, or not at all. Operational embeddability is characterized by whether explanations are ad hoc and ephemeral (no logging or reuse), partially instrumented (repeatable, loggable, and auditable), or embedded as directly enabling operational interventions. As most reviewed work predates the EU AI Act, alignment is often implicit and reflects earlier GDPR principles, particularly transparency, accountability, and data protection at the data and decision levels. Overall, our findings indicate that the AI Act's lifecycle-wide obligations remain only weakly supported by current XAI research.

Regarding AI Act provisions on data handling (Art. 10-Data Governance), the literature shows limited alignment, with most reviewed studies not explicitly addressing data governance requirements. Operationalization is mostly ephemeral with isolated conceptual examples, with a notable absence of instrumented and actionable implementations. Existing research supports data quality assessment (Azimi and Pahl, 2021; Dablain et al., 2024; Kim et al., 2022); however, these contributions remain largely disconnected from the explanations presented to end-users or operators. The main gaps concern explanations confined to isolated subprocesses and the lack of data lineage tracking to support explainability in downstream decisions.

Studies addressing lifecycle-wide governance reveal substantial gaps in the operational infrastructure required for continuous regulatory compliance. Current XAI research largely neglects foundational compliance mechanisms, including risk management (Art. 9), technical documentation (Art. 11), record-keeping (Art. 12), and quality management (Art. 17). These performance-related requirements cannot be addressed through explainability alone but benefit substantially from explainability-driven quality assurance. While a handful of works align toward these requirements, they lack systematic logging, traceability, and actionable integration, limiting their regulatory relevance. As a result, the field has yet to establish auditable risk identification processes, standardized documentation practices, or continuous validation frameworks for explanations. These deficiencies fundamentally constrain the use of XAI for compliance, as explanations cannot be reliably audited, maintained, or verified across the system lifecycle without supporting infrastructure. In contrast, accuracy and robustness (Art. 15) receive comparatively greater attention, with moderate levels of operationalization across embeddability levels. Existing studies demonstrate how XAI supports robustness testing (Hartl et al., 2020; Sun R. et al., 2023), bias detection (Van Stein et al., 2023; Tavares et al., 2023), and runtime model selection (Jakobs and Saadallah, 2023; Saadallah, 2023), all of which are relevant for compliance maintenance. However, critical gaps persist, including that explanation fidelity is rarely validated and robustness assessments typically target models rather than explanations themselves, offering no reliability guarantees for XAI‘outputs.

Among the AI Act provisions relevant to deployment, a central requirement is that AI systems be “sufficiently transparent to enable deployers to interpret a system's output and use it appropriately” (Art. 13(1)). This requirement receives the highest level of coverage in the reviewed literature. Most studies address transparency in some form, though predominantly at moderate or limited alignment levels. A critical paradox emerges at the level of operationalization. While conceptual support for transparency is high, implementations are largely ephemeral or, at best, instrumented, with few actionable mechanisms. The literature aligns with Art. 13 across several explainability touchpoints. Explainable model selection supports deployers in interpreting system outputs, understanding model limitations, and applying them responsibly within operational settings (Haagen et al., 2023; Fischer and Saadallah, 2024; Berloco et al., 2024). Post-deployment monitoring studies further demonstrate how explanations can be used to track predictions over time and diagnose silent failures, drift, and performance degradation (Koebler et al., 2023; Shahad and Raj, 2024; Decker et al., 2024; De Oliveira et al., 2020b; Vieira et al., 2021; Hinder et al., 2024). Nevertheless, the field provides transparency conceptually, lacking persistent explanation storage, traceability, and auditable integration required for compliance.

The AI Act's human oversight requirement mandates that human overseers “correctly interpret the high-risk AI system's output, taking into account [] the interpretation tools and methods available” (Art. 14(4)). This requirement aligns with research that provides interpretable evidence to support human validation or override of model outputs (Aiosa et al., 2023; Midtfjord et al., 2022; Sayed-Mouchaweh and Rajaoarisoa, 2022). However, it remains largely unexplored in the literature, and operationalization is predominantly ephemeral, with few actionable examples. Human oversight is typically supported only at decision time, without mechanisms for systematic intervention, override, or feedback integration into operational workflows. A similar gap is evident for post-market monitoring (Art. 72). Most reviewed studies do not address continuous monitoring requirements, and those that do, offer limited and fragmented operational support. Explainability is rarely integrated into post-market monitoring infrastructures, and runtime monitoring of explanations themselves is largely absent. As a result, neither human oversight nor post-market monitoring is supported in a lifecycle-consistent or auditable manner, undermining their regulatory effectiveness.

Finally, the AI Act establishes a right to explanation of individual decision-making, granting affected persons “clear and meaningful” explanations of “the role of the AI system in the decision-making procedure and the main elements of the decision taken” (Art. 86(1)). This requirement remains largely conceptual in the literature, with most reviewed studies addressing explainability only indirectly and context outside “the right to explanation”. Operationalization is emerging as actionable, primarily through end-user-facing explanations in decision-support and accountability contexts, yet lacks persistence, traceability, or mechanisms for reuse. Relevant work appears mainly in healthcare and credit scoring (Schoonderwoerd et al., 2021; Amann et al., 2022; Mariotti et al., 2021) and in trust calibration research (Naiseh et al., 2023; Zafari et al., 2024), overlapping conceptually with human oversight requirements. However, several critical gaps remain, including that individual-level explanations remain rare, recourse mechanisms are largely absent, and explanations seldom provide affected persons with actionable insights.

7 Research agenda

The synthesis of findings across the RQs reveals three structural gaps that define our proposed research agenda for explainability in MLOps. The analysis of lifecycle-wide connections shows that explainability remains fragmented, with minimal mechanisms linking data handling, model development, and deployment (Section 4). This fragmentation prevents explanations from supporting continuous oversight, underscoring the need for lifecycle-integrated explainability. The assessment of XAI reliability and trustworthiness exposes the absence of standardized validation frameworks, as most studies evaluate explainability through either selected technical metrics or human-centered studies in isolation, rarely combining both perspectives (Section 5). The regulatory analysis demonstrates that explainability is rarely operationalized in ways that support auditability, traceability, or sustained compliance with emerging governance requirements (Section 6). Together, these gaps point to the need for research that reconceptualizes explainability as a continuous, validated, and compliance-aware capability embedded throughout the MLOps lifecycle. Accordingly, we outline a research agenda structured around these three challenges.

7.1 From isolated stages to continuous explainability in MLOps

Despite the growing use of XAI across the MLOps pipeline from data pre-processing (Kauffmann et al., 2022; Trost et al., 2025; Shafin, 2024; Zang et al., 2022) and model debugging (Tavares et al., 2025; Haagen et al., 2023) to trust calibration (Naiseh et al., 2023; Zafari et al., 2024) explainability remains largely fragmented. Most studies examine each phase in isolation, applying XAI tools to discrete tasks such as clustering, bias detection, or anomaly detection (Bandyapadhyay et al., 2023; Van Stein et al., 2023; Jin et al., 2025), rather than as a continuous mechanism that generates insights to inform subsequent stages.

Several structural factors contribute to this fragmentation. Research on explainability is siloed across data engineering, machine learning, and MLOps communities, limiting methodological cross-pollination. Tooling limitations further inhibit integration, as explanations are not represented in standardized, reusable formats that allow propagation across pipeline stages. Academically, incentives tend to prioritize methodological novelty over systems-level integration. Additionally, the benefits of lifecycle-wide explainability, such as long-term oversight and governance, are difficult to evaluate empirically. Finally, as MLOps itself is a relatively recent paradigm, integration-focused XAI research has not matured at the same pace as standalone explanationtechniques.

In practice, however, the MLOps lifecycle is nonlinear and interdependent (Figure 8). Decisions made during data handling influence model design; explanations from model development can guide deployment strategies; and deployment-time insights can trigger data acquisition, filtering, and retraining, closing the feedback loop. While these connections are theoretically acknowledged, current research treats them as isolated observations rather than as components of an integrated quality-assurance framework. Explanations generated in one stage rarely propagate systematically to inform decisions in another, and no established methodology exists for tracing how explanations evolve across the pipeline or interact across multiple XAI touchpoints. To bridge this gap, future research should address:

• i. How do XAI touchpoints at different stages of the MLOps pipeline influence or connect to one another?

• ii. How can explainability be embedded into every phase of MLOps in a structured way so that it actively contributes to overall workflow quality?

Figure 8

Answering these questions requires mapping inter-stage dependencies, identifying feedback mechanisms, and developing frameworks that trace the flow of explanations across the entire lifecycle. By structuring XAI in this way, explainability shifts from being a retrospective diagnostic tool to a proactive quality-assurance mechanism.

7.2 Assuring the reliability of XAI methods

While explainability methods are widely applied across MLOps, the reliability and robustness of the explanations themselves remains underexplored. Existing literature highlights that explanations themselves can introduce new risks, such as over-reliance, misplaced trust, or misleading explanations, which may compromise the very transparency they aim to promote. Ensuring that explanation methods provide faithful and trustworthy explanations requires rigorous validation.

Our analysis reveals that current evaluation practices are fragmented and rarely lifecycle-linked. Model selection explanations are seldom validated beyond offline settings, drift detection explanations are rarely assessed under evolving data conditions, and privacy-preserving explanations are almost never evaluated for practical usefulness. Most studies rely on either selected technical metrics or human-centered evaluations in isolation, with little integration across phases or stakeholder roles. Developer-facing explanations are rarely subjected to user studies, while very few study examined end-user explanations for their impact on accountability or correctability. More broadly, compliance-aligned validity is largely absent. Explanations are not evaluated for persistence, auditability, or suitability for regulatory oversight.

Despite growing methodological contributions, XAI evaluation research remains dispersed and domain-specific, lacking systematic frameworks for validating the reliability of explanations across the ML pipeline. Addressing this gap requires benchmark suites, perturbation analyses, cross-method consistency checks, and human-in-the-loop evaluations that jointly assess robustness, usability, and downstream impact. Future research should therefore aim to operationalize multi-layered validation frameworks that treat explanations as first-class system artifacts subject to continuous testing and auditing. Key guiding research questions include:

• i. How can we systematically evaluate and validate XAI methods to ensure that explanations faithfully reflect model reasoning, remain stable under diverse conditions, and maintain integrity across data and model variations?

• ii. How can standardized metrics, benchmarks, and validation frameworks for XAI reliability be developed and operationalized within MLOps pipelines to support continuous explainability?

Answering these questions requires a shift from narrow, task-specific validation toward lifecycle-aware, auditable evaluation frameworks that integrate technical robustness with human trust calibration and accountability. The lack of shared metrics and protocols remains a fundamental barrier to establishing coherent, comparable standards for XAI reliability and trustworthiness.

7.3 Regulatory compliance through explainability

As AI regulation matures, aligning explainable AI with legal requirements has become a central challenge. In the EU, the AI Act (AIA) (European Parliament and European Council, 2024), in force since August 2024, introduces binding obligations for AI systems placed on the European market under a risk-based framework. Explainability-related requirements primarily apply to high-risk AI systems. Prior work has analyzed these provisions from software engineering and governance perspectives, highlighting interpretability and explainability as key compliance mechanisms (Wagner et al., 2025; Nisevic et al., 2024; Nannini, 2025).

Our review shows that while XAI research aligns conceptually with regulatory transparency and oversight goals, it falls short operationally. Most contributions provide ephemeral or loosely instrumented explanations that are neither persistently logged nor integrated into documentation, record-keeping, quality management, or post-market monitoring processes. As a result, explanations rarely serve as auditable compliance artifacts that support continuous regulatory assurance throughout the MLOps lifecycle.

This gap reflects a broader limitation. Current research addresses specific explainability touchpoints but lacks frameworks that operationalize explainability as part of lifecycle-wide compliance infrastructure. Moreover, the AIA's explainability and interpretability requirements remain somewhat ambiguous, with details to be specified through upcoming harmonized standards and implementing acts. Addressing these challenges requires moving beyond explanation generation toward explanation systems that support persistence, traceability, logging, and intervention. To address these gaps, future research should investigate:

• i. How can explainability be systematically implemented in MLOps pipelines to address regulatory requirements on explainability and interpretability across phases?

• ii. How can explainability-driven quality assurance facilitate compliance with performance-related AIA requirements?

Advancing this agenda requires compliance-oriented frameworks that treat explainability not as a post-hoc transparency aid, but as a continuous, auditable quality assurance layer throughout the MLOps lifecycle, capable of evolving alongside regulatory guidance while ensuring trustworthy AI in practice.

8 Conclusion

We presented a scoping review of explainability throughout the MLOps lifecycle, examining how XAI methods is integrated across data handling, model development, and deployment. Our systematic analysis reveals that explainability has permeated virtually every stage of MLOps, with methods emerging for explainable imputation, interpretable clustering, bias detection, robustness testing, drift detection, and decision support. Despite this extensive application, three critical limitations constrain current explainability potentials. First, explainability remains fragmented across the lifecycle. XAI methods operate largely in isolation within individual stages, preventing explanations from functioning as an integrated mechanism. Yet explanations generated in one stage can inform decision-making in another, creating a feedback loop that connects data handling, model development, and deployment, ultimately cycling back to data handling. Second, the reliability of XAI methods themselves lacks rigorous validation. As explanations increasingly drive operational decisions, standardized evaluation frameworks for assessing their fidelity, stability, and human interpretability become essential. Third, operationalizing explainability for regulatory compliance remains an open challenge, particularly under frameworks like the EU AI Act that impose explicit requirements for transparency and continuous quality management. The proposed research agenda calls for systematically connecting MLOps phases through explainability, developing robust validation frameworks to ensure the reliability of XAI methods, and operationalizing explainability to meet regulatory and performance-related obligations. Ultimately, these represent necessary steps toward building AI systems that are auditable, accountable, and sustainable across their entire lifecycle.

References

• 1

  AbououfM.SinghS.MizouniR.OtrokH. (2023). Explainable AI for event and anomaly detection and classification in healthcare monitoring systems. IEEE Intern. Things J. 11, 3446–3457. doi: 10.1109/JIOT.2023.3296809

  • CrossRef
  • Google Scholar

• 2

  AdhikariD.JiangW.ZhanJ.HeZ.RawatD. B.AickelinU.et al. (2022). A comprehensive survey on imputation of missing data in internet of things. ACM Comp. Surv. 55, 1–38. doi: 10.1145/3533381

  • CrossRef
  • Google Scholar

• 3

  AhmedU.SrivastavaG.YunU.LinJ. C.-W. (2022). EANDC: An explainable attention network based deep adaptive clustering model for mental health treatment. Future Generat. Comp. Syst. 130, 106–113. doi: 10.1016/j.future.2021.12.008

  • CrossRef
  • Google Scholar

• 4

  AiosaG. V.PalesiM.SapuppoF. (2023). Explainable AI for decision support to obesity comorbidities diagnosis. IEEE Access11, 107767–107782. doi: 10.1109/ACCESS.2023.3320057

  • CrossRef
  • Google Scholar

• 5

  AkhtarM. A. K.KumarM.NayyarA. (2024). “Transparency and accountability in explainable AI: Best practices,” in Towards Ethical and Socially Responsible Explainable AI: Challenges and Opportunities (Cham: Springer), 127–164.

  • Google Scholar

• 6

  Alvarez-GarciaM.Ibar-AlonsoR.Arenas-ParraM. (2024). A comprehensive framework for explainable cluster analysis. Inf. Sci. 663:120282. doi: 10.1016/j.ins.2024.120282

  • CrossRef
  • Google Scholar

• 7

  AmannJ.VetterD.BlombergS. N.ChristensenH. C.CoffeeM.GerkeS.et al. (2022). To explain or not to explain?–Artificial intelligence explainability in clinical decision support systems. PLOS Digital Health1:e0000016. doi: 10.1371/journal.pdig.0000016

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8

  AngelovP. P.SoaresE. A.JiangR.ArnoldN. I.AtkinsonP. M. (2021). Explainable artificial intelligence: an analytical review. Wiley Interdiscip. Rev.: Data Mining Knowl. Discov. 11:e1424. doi: 10.1002/widm.1424

  • CrossRef
  • Google Scholar

• 9

  AnjomshoaeS.OmeizaD.JiangL. (2021). Context-based image explanations for deep neural networks. Image Vis. Comput. 116:104310. doi: 10.1016/j.imavis.2021.104310

  • CrossRef
  • Google Scholar

• 10

  AntoniadiA. M.DuY.GuendouzY.WeiL.MazoC.BeckerB. A.et al. (2021). Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: a systematic review. Appl. Sci. 11:5088. doi: 10.3390/app11115088

  • CrossRef
  • Google Scholar

• 11

  ArkseyH.O'MalleyL. (2005). Scoping studies: towards a methodological framework. Int. J. Soc. Res. Methodol. 8:19–32. doi: 10.1080/1364557032000119616

  • CrossRef
  • Google Scholar

• 12

  AssadiP.SafaeiN. (2024). “Interpretable AI in human-machine systems: insights from human-in-the-loop product recommendation engines,” in Interpretable AI: Past, Present and Future.

  • Google Scholar

• 13

  AssafR.GiurgiuI.PfefferleJ.MonneyS.PozidisH.SchumannA. (2021). “An anomaly detection and explainability framework using convolutional autoencoders for data storage systems,” in Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence IJCAI-PRICAI 2020 (Yokohama), 5228–5230.

  • Google Scholar

• 14

  AtfZ.LewisP. R. (2025). Is trust correlated with explainability in AI? A meta-analysis. IEEE Trans. Technol. Soc. 7, 70–77. doi: 10.1109/TTS.2025.3558448

  • CrossRef
  • Google Scholar

• 15

  AugustinM.MeinkeA.HeinM. (2020). “Adversarial robustness on in-and out-distribution improves explainability,” in European Conference on Computer Vision (Cham: Springer), 228–245.

  • Google Scholar

• 16

  AwalM. A.RochanM.RoyC. K. (2025). Investigating adversarial attacks in software analytics via machine learning explainability. Softw. Qual. J. 33, 1–42. doi: 10.1007/s11219-025-09727-2

  • CrossRef
  • Google Scholar

• 17

  AwosikaT.ShuklaR. M.PranggonoB. (2024). Transparency and privacy: the role of explainable AI and federated learning in financial fraud detection. IEEE Access12, 64551–64560. doi: 10.1109/ACCESS.2024.3394528

  • CrossRef
  • Google Scholar

• 18

  AzimiS.PahlC. (2021). “The effect of IOT data completeness and correctness on explainable machine learning models,” in International Conference on Database and Expert Systems Applications (Cham: Springer), 151–160.

  • Google Scholar

• 19

  BandyapadhyayS.FominF. V.GolovachP. A.LochetW.PurohitN.SimonovK. (2023). How to find a good explanation for clustering?Artif. Intell. 322:103948. doi: 10.1016/j.artint.2023.103948

  • CrossRef
  • Google Scholar

• 20

  BardosA.MollasI.BassiliadesN.TsoumakasG. (2022). “Local explanation of dimensionality reduction,”? in Proceedings of the 12th Hellenic Conference on Artificial Intelligence (New York, NY: ACM), 1–9.

  • Google Scholar

• 21

  Barredo ArrietaA.D-az-Rodr-guezN.Del SerJ.BennetotA.TabikS.BarbadoA.et al. (2020). Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inform. Fusion58, 82–115. doi: 10.1016/j.inffus.2019.12.012

  • CrossRef
  • Google Scholar

• 22

  BayerS.GimpelH.MarkgrafM. (2022). The role of domain expertise in trusting and following explainable AI decision support systems. J. Decis. Syst. 32, 110–138. doi: 10.1080/12460125.2021.1958505

  • CrossRef
  • Google Scholar

• 23

  Baysal ErezI.FlokstraJ.PoelM.van KeulenM. (2025). Extended metalirs: Meta-learning for imputation and regression selection model with explainability for different missing data mechanisms. Int. J. Data Sci. Analyt. 20, 5895–5920. doi: 10.1007/s41060-025-00808-w

  • CrossRef
  • Google Scholar

• 24

  BerlocoF.MarvulliP. M.SugliaV.ColucciS.PaganoG.PalazzoL.et al. (2024). Enhancing survival analysis model selection through XAI(t) in healthcare. Appl. Sci. 14:6084. doi: 10.3390/app14146084

  • CrossRef
  • Google Scholar

• 25

  BhuyanH. K.ChakrabortyC. (2022). Explainable machine learning for data extraction across computational social system. IEEE Trans. Comput. Soc. Syst. 11, 3131–3145. doi: 10.1109/TCSS.2022.3164993

  • CrossRef
  • Google Scholar

• 26

  BobekS.KukM.SzelkażekM.NalepaG. J. (2022). Enhancing cluster analysis with explainable AI and multidimensional cluster prototypes. IEEE Access10, 101556–101574. doi: 10.1109/ACCESS.2022.3208957

  • CrossRef
  • Google Scholar

• 27

  BogdanovaA.ImakuraA.SakuraiT. (2023). DC-SHAP method for consistent explainability in privacy-preserving distributed machine learning. Human-Centric Intellig. Syst. 3, 197–210. doi: 10.1007/s44230-023-00032-4

  • CrossRef
  • Google Scholar

• 28

  BozorgpanahA.TorraV.AliahmadipourL. (2022). Privacy and explainability: the effects of data protection on shapley values. Technologies10:125. doi: 10.3390/technologies10060125

  • CrossRef
  • Google Scholar

• 29

  BreckE.CaiS.NielsenE.SalibM.SculleyD. (2017). “The ML test score: a rubric for ML production readiness and technical debt reduction,” in 2017 IEEE International Conference on Big Data (Piscataway, NJ: IEEE), 1123–1132.

  • Google Scholar

• 30

  BückerM.SzepannekG.GosiewskaA.BiecekP. (2022). Transparency, auditability, and explainability of machine learning models in credit scoring. J. Operat. Res. Soc. 73, 70–90. doi: 10.1080/01605682.2021.1922098

  • CrossRef
  • Google Scholar

• 31

  BuczakA. L.BaugherB. D.BerlierA. J.ScharfsteinK. E.MartinC. S. (2022). “Explainable forecasts of disruptive events using recurrent neural networks,” in 2022 IEEE International Conference on Assured Autonomy (ICAA) (Fajardo, PR: IEEE), 64–73.

  • Google Scholar

• 32

  CândidoJ.AnicheM.Van DeursenA. (2021). Log-based software monitoring: a systematic mapping study. PeerJ Comp. Sci. 7:e489. doi: 10.7717/peerj-cs.489

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 33

  CenS. H.AlurR. (2024). “From transparency to accountability and back: a discussion of access and evidence in AI auditing,” in Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (New York, NY: ACM), 1–14.

  • Google Scholar

• 34

  ChanderB.JohnC.WarrierL.GopalakrishnanK. (2025). Toward trustworthy artificial intelligence (TAI) in the context of explainability and robustness. ACM Comp. Surv. 57, 1–49. doi: 10.1145/3675392

  • CrossRef
  • Google Scholar

• 35

  ChatzimparmpasA.MartinsR. M.KucherK.KerrenA. (2022). Featureenvi: Visual analytics for feature engineering using stepwise selection and semi-automatic extraction approaches. IEEE Trans. Vis. Comput. Graph. 28, 1773–1791. doi: 10.1109/TVCG.2022.3141040

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36

  ChenC.LiaoM.SundarS. S. (2024). “When to explain? Exploring the effects of explanation timing on user perceptions and trust in AI systems,” in Proceedings of the Second International Symposium on Trustworthy Autonomous Systems (New York, NY: ACM), 1–17.

  • Google Scholar

• 37

  ChenZ.TanS.ChajewskaU.RudinC.CarunaR. (2023). “Missing values and imputation in healthcare data: can interpretable machine learning help?,” in Conference on Health, Inference, and Learning (new York: PMLR), 86–99.

  • Google Scholar

• 38

  ChoW.ParkJ.ChooJ. (2023). “Training auxiliary prototypical classifiers for explainable anomaly detection in medical image segmentation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (Waikoloa, HI: IEEE), 2624–2633.

  • Google Scholar

• 39

  ChoiH.KimD.KimJ.KimJ.KangP. (2022). Explainable anomaly detection framework for predictive maintenance in manufacturing systems. Appl. Soft Comput. 125:109147. doi: 10.1016/j.asoc.2022.109147

  • CrossRef
  • Google Scholar

• 40

  CinquiniM.GiannottiF.GuidottiR.MatteiA. (2023). “Handling missing values in local post-hoc explainability,” in World Conference on Explainable Artificial Intelligence (Cham: Springer), 256–278.

  • Google Scholar

• 41

  Coma-PuigB.CarmonaJ. (2021). “A human-in-the-loop approach based on explainability to improve NTL detection,” in 2021 International Conference on Data Mining Workshops (ICDMW) (Auckland: IEEE), 943–950.

  • Google Scholar

• 42

  CuéllarS.SantosM.AlonsoF.FabregasE.FariasG. (2024). Explainable anomaly detection in spacecraft telemetry. Eng. Appl. Artif. Intell. 133:108083. doi: 10.1016/j.engappai.2024.108083

  • CrossRef
  • Google Scholar

• 43

  DablainD.BellingerC.KrawczykB.AhaD. W.ChawlaN. (2024). Understanding imbalanced data: XAI & interpretable ML framework. Mach. Learn. 113, 3751–3769. doi: 10.1007/s10994-023-06414-w

  • CrossRef
  • Google Scholar

• 44

  DasS.SultanaM.BhattacharyaS.SenguptaD.DeD. (2023). XAI-reduct: accuracy preservation despite dimensionality reduction for heart disease classification using explainable ai. J. Supercomput. 79, 18167–18197. doi: 10.1007/s11227-023-05356-3

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 45

  De OliveiraH.AugustoV.JouanetonB.LamarsalleL.ProdelM.XieX. (2020a). Automatic and explainable labeling of medical event logs with autoencoding. IEEE J. Biomed. Health Inform. 24, 3076–3084. doi: 10.1109/JBHI.2020.3021790

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 46

  De OliveiraH.AugustoV.JouanetonB.LamarsalleL.ProdelM.XieX. (2020b). “An optimization-based process mining approach for explainable classification of timed event logs,” in 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE) (Hong Kong: IEEE), 43–48.

  • Google Scholar

• 47

  DeckL.SchoefferJ.De-ArteagaM.KühlN. (2024). “A critical survey on fairness benefits of explainable AI,”? in Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (New York, NY: ACM), 1579–1595.

  • Google Scholar

• 48

  DeckerT.KoeblerA.LebacherM.ThonI.TrespV.BuettnerF. (2024). “Explanatory model monitoring to understand the effects of feature shifts on performance,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (New York, NY: ACM), 550–561.

  • Google Scholar

• 49

  DeshmukhS.BeheraB. K.MulayP.AhmedE. A.Al-KuwariS.TiwariP.et al. (2023). Explainable quantum clustering method to model medical data. Knowl.-Based Syst. 267:110413. doi: 10.1016/j.knosys.2023.110413

  • CrossRef
  • Google Scholar

• 50

  DuY.RaffertyA. R.McAuliffeF. M.WeiL.MooneyC. (2022). An explainable machine learning-based clinical decision support system for prediction of gestational diabetes mellitus. Sci. Rep. 12:1170. doi: 10.1038/s41598-022-05112-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 51

  DuckworthC.ChmielF. P.BurnsD. K.ZlatevZ. D.WhiteN. M.DanielsT. W.et al. (2021). Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during covid-19. Sci. Rep. 11:23017. doi: 10.1038/s41598-021-02481-y

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 52

  EsfandiariH.MirrokniV.NarayananS. (2022). “Almost tight approximation algorithms for explainable clustering,” in Proceedings of the 2022 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) (Philadelphia, PA: SIAM (Society for Industrial and Applied Mathematics)), 2641–2663.

  • Google Scholar

• 53

  Estivill-CastroV.GilmoreE.HexelR. (2022a). Constructing explainable classifiers from the start–enabling human-in-the loop machine learning. Information13:464. doi: 10.3390/info13100464

  • CrossRef
  • Google Scholar

• 54

  Estivill-CastroV.GilmoreE.HexelR. (2022b). “Interpretable decisions trees via human-in-the-loop-learning,” in Australasian Conference on Data Mining (Cham: Springer), 115–130.

  • Google Scholar

• 55

  European Parliament and European Council (2024). AI Act. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonised Rules on Artificial Intelligence and Amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act).

  • Google Scholar

• 56

  EzejiI. N.AdigunM.OkiO. (2024). Computational complexity in explainable decision support system: a review. J. Intellig. Fuzzy Syst. doi: 10.3233/JIFS-219407

  • CrossRef
  • Google Scholar

• 57

  FamigliniL.CampagnerA.BarandasM.La MaidaG.GallazziE.CabitzaF.et al. (2024). Evidence-based XAI: an empirical approach to design more effective and explainable decision support systems. Comput. Biol. Med. 170:108042. doi: 10.1016/j.compbiomed.2024.108042

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 58

  FerrarioA.LoiM. (2022). “How explainability contributes to trust in AI,” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (New York, NY: ACM), 1457–1466.

  • Google Scholar

• 59

  FerryJ.AïvodjiU.GambsS.HuguetM.-J.SialaM. (2025). Taming the triangle: On the interplays between fairness, interpretability, and privacy in machine learning. Comput. Intellig. 41:e70113. doi: 10.1111/coin.70113

  • CrossRef
  • Google Scholar

• 60

  FischerR.SaadallahA. (2024). “AutoXPCR: automated multi-objective model selection for time series forecasting,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (New York, NY: ACM), 806–815.

  • Google Scholar

• 61

  FrancoD.OnetoL.NavarinN.AnguitaD. (2021). Toward learning trustworthily from data combining privacy, fairness, and explainability: an application to face recognition. Entropy23:1047. doi: 10.3390/e23081047

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 62

  FryerD.StrümkeI.NguyenH. (2021). Shapley values for feature selection: The good, the bad, and the axioms. IEEE Access9, 144352–144360. doi: 10.1109/ACCESS.2021.3119110

  • CrossRef
  • Google Scholar

• 63

  Gad-ElrabM. H.StepanovaD.TranT.-K.AdelH.WeikumG. (2020). “Excut: Explainable embedding-based clustering over knowledge graphs,” in International Semantic Web Conference (Cham: Springer), 218–237.

  • Google Scholar

• 64

  GalantiR.de LeoniM.MonaroM.NavarinN.MarazziA.Di StasiB.et al. (2023). An explainable decision support system for predictive process analytics. Eng. Appl. Artif. Intell. 120:105904. doi: 10.1016/j.engappai.2023.105904

  • CrossRef
  • Google Scholar

• 65

  GaudioA.SmailagicA.FaloutsosC.MohanS.JohnsonE.LiuY.et al. (2023). Deepfixcx: Explainable privacy-preserving image compression for medical image analysis. Wiley Interdiscipl. Rev.: Data Mining and Knowl. Discov. 13:e1495. doi: 10.1002/widm.1495

  • CrossRef
  • Google Scholar

• 66

  GebreyesusY.DaltonD.NixonS.De ChiaraD.ChinniciM. (2023). Machine learning for data center optimizations: feature selection using shapley additive explanation (SHAP). Future Intern. 15:88. doi: 10.3390/fi15030088

  • CrossRef
  • Google Scholar

• 67

  GerlingA.KamperO.SeifferC.ZiekowH.SchreierU.HessA.et al. (2022). “Results from using an automl tool for error analysis in manufacturing,” in ICEIS (Setúbal: SciTePress), 100–111.

  • Google Scholar

• 68

  GoethalsS.SörensenK.MartensD. (2023). The privacy issue of counterfactual explanations: explanation linkage attacks. ACM Trans. Intellig. Syst. Technol. 14, 1–24. doi: 10.1145/3608482

  • CrossRef
  • Google Scholar

• 69

  GombolayG. Y.SilvaA.SchrumM.GopalanN.Hallman-CooperJ.DuttM.et al. (2024). Effects of explainable artificial intelligence in neurology decision support. Annals Clini. Transl. Neurol. 11, 1224–1235. doi: 10.1002/acn3.52036

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 70

  GrantT. D.WischikD. J. (2020). Show us the data: Privacy, explainability, and why the law can't have both. Geo. Wash. L. Rev. 88:1350.

  • Google Scholar

• 71

  GuanR.ZhangH.LiangY.GiunchigliaF.HuangL.FengX. (2020). Deep feature-based text clustering and its explanation. IEEE Trans. Knowl. Data Eng. 34, 3669–3680. doi: 10.1109/TKDE.2020.3028943

  • CrossRef
  • Google Scholar

• 72

  GummadiA. N.NapierJ. C.AbdallahM. (2024). Xai-iot: an explainable AI framework for enhancing anomaly detection in IoT systems. IEEE Access12, 71024–71054. doi: 10.1109/ACCESS.2024.3402446

  • CrossRef
  • Google Scholar

• 73

  GuneyO. B.SaichandranK. S.ElzokmK.ZhangZ.KolachalamaV. B. (2025). “Active feature acquisition via explainability-driven ranking,” in Forty-Second International Conference on Machine Learning (Cambridge: Proceedings of Machine Learning Research).

  • Pubmed Abstract
  • Google Scholar

• 74

  HaagenT.KayaH.SnijderJ.NiermanM. (2023). “Autoxplain: towards automated interpretable model selection,” in The 1st World Conference on eXplainable Artificial Intelligence (CEUR WS), 18–23.

  • Google Scholar

• 75

  HakimD. L.DarariF. (2021). “Data completeness impact on deep learning based explainable recommender systems,” in 2021 4th International Conference on Information and Communications Technology (ICOIACT) (Yogyakarta: IEEE), 262–267.

  • Google Scholar

• 76

  HalléS. (2020). “Explainable queries over event logs,” in 2020 IEEE 24th International Enterprise Distributed Object Computing Conference (EDOC) (Piscataway, NJ: IEEE), 171–180.

  • Google Scholar

• 77

  HamonR.JunklewitzH.SanchezI. (2020). Robustness and explainability of artificial intelligence.Luxembourg: Publications Office of the European Union, 207.

  • Google Scholar

• 78

  HamrouniB.BourouisA.KorichiA.BrahmiM. (2021). Explainable ontology-based intelligent decision support system for business model design and sustainability. Sustainability13:9819. doi: 10.3390/su13179819

  • CrossRef
  • Google Scholar

• 79

  HansS.SahaD.AggarwalA. (2023). “Explainable data imputation using constraints,” in Proceedings of the 6th Joint International Conference on Data Science & Management of Data (10th ACM IKDD CODS and 28th COMAD) (New York, NY: ACM), 128–132.

  • Google Scholar

• 80

  HartlA.BachlM.FabiniJ.ZsebyT. (2020). “Explainability and adversarial robustness for RNNs,” in 2020 IEEE Sixth International Conference on Big Data Computing Service and Applications (BigDataService) (Oxford: IEEE), 148–156.

  • Google Scholar

• 81

  HasanM. J.SohaibM.KimJ.-M. (2021). An explainable AI-based fault diagnosis model for bearings. Sensors21:4070. doi: 10.3390/s21124070

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 82

  HinderF.VaquetV.HammerB. (2024). One or two things we know about concept drift–a survey on monitoring in evolving environments. part b: locating and explaining concept drift. Front. Artif. Intellig. 7:1330258. doi: 10.3389/frai.2024.1330258

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 83

  HlostaM.HerodotouC.PapathomaT.GillespieA.BergaminP. (2022). Predictive learning analytics in online education: a deeper understanding through explaining algorithmic errors. Comp. Educ.: Artif. Intellig. 3:100108. doi: 10.1016/j.caeai.2022.100108

  • CrossRef
  • Google Scholar

• 84

  HoangN. X.HoangN. V.DuN. H.HuongT. T.TranK. P.et al. (2022). Explainable anomaly detection for industrial control system cybersecurity. IFAC-PapersOnLine55, 1183–1188. doi: 10.1016/j.ifacol.2022.09.550

  • CrossRef
  • Google Scholar

• 85

  HolzingerA.SarantiA.HauschildA.-C.BeineckeJ.HeiderD.RoettgerR.et al. (2023). “Human-in-the-loop integration with domain-knowledge graphs for explainable federated deep learning,”? in International Cross-Domain Conference for Machine Learning and Knowledge Extraction (Cham: Springer), 45–64.

  • Google Scholar

• 86

  HooshmandM. K.HuchaiahM. D.AlzighaibiA. R.HashimH.AtlamE.-S.GadI. (2024). Robust network anomaly detection using ensemble learning approach and explainable artificial intelligence (XAI). Alexand. Eng. J. 94, 120–130. doi: 10.1016/j.aej.2024.03.041

  • CrossRef
  • Google Scholar

• 87

  HrnjicaB.SofticS. (2020). “Explainable AI in manufacturing: a predictive maintenance case study,” in IFIP International Conference on Advances in Production Management Systems (Cham: Springer), 66–73.

  • Google Scholar

• 88

  HuongT. T.BacT. P.HaK. N.HoangN. V.HoangN. X.HungN. T.et al. (2022). Federated learning-based explainable anomaly detection for industrial control systems. IEEE Access10, 53854–53872. doi: 10.1109/ACCESS.2022.3173288

  • CrossRef
  • Google Scholar

• 89

  HussainA.HussainA. (2025). Transparency and accountability: unpacking the real problems of explainable AI. AI & Society40, 5587–5588. doi: 10.1007/s00146-025-02302-0

  • CrossRef
  • Google Scholar

• 90

  HwangC.LeeT. (2021). E-sfd: Explainable sensor fault detection in the ics anomaly detection system. IEEE Access9, 140470–140486. doi: 10.1109/ACCESS.2021.3119573

  • CrossRef
  • Google Scholar

• 91

  HwangH.WhangS. E. (2023). “Xclusters: explainability-first clustering,” in Proceedings of the AAAI Conference on Artificial Intelligence, 37, 7962–7970.

  • Google Scholar

• 92

  JakobsM.SaadallahA. (2023). “Explainable adaptive tree-based model selection for time-series forecasting,”? in 2023 IEEE International Conference on Data Mining (ICDM) (Shanghai: IEEE), 180–189.

  • Google Scholar

• 93

  JakubowskiJ.StaniszP.BobekS.NalepaG. J. (2021). “Explainable anomaly detection for hot-rolling industrial process,” in 2021 IEEE 8th international conference on data science and advanced analytics (DSAA) (Porto: IEEE), 1–10.

  • Google Scholar

• 94

  JiangD.ZhangS. (2025). An explainable missing data imputation method and its application in soft sensing. Measurement253:117692. doi: 10.1016/j.measurement.2025.117692

  • CrossRef
  • Google Scholar

• 95

  JinE.FengQ.MouY.LakemeyerG.DeckerS.SimonsO.et al. (2025). “LogicAD: Explainable anomaly detection via VLM-based text feature extraction,” in Proceedings of the AAAI Conference on Artificial Intelligence (Washington, DC: AAAI Press), 39, 4129–4137.

  • Google Scholar

• 96

  KarvalR.SinghK. N. (2023). “Catching silent failures: a machine learning model monitoring and explainability survey,” in 2023 OITS International Conference on Information Technology (OCIT) (Raipur: IEEE), 526–532.

  • Google Scholar

• 97

  KauffmannJ.EsdersM.RuffL.MontavonG.SamekW.MüllerK.-R. (2022). From clustering to cluster explanations via neural networks. IEEE trans. Neural Netw. Learn. Syst. 35, 1926–1940. doi: 10.1109/TNNLS.2022.3185901

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 98

  KhanR. H.DofadarD. F.AlamM. G. R. (2021). “Explainable customer segmentation using k-means clustering,” in 2021 IEEE 12th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) (New York, NY: IEEE), 0639–0643.

  • Google Scholar

• 99

  Khosravi TabriziA.Ezzati-JivanN.TetreaultF. (2024). “An adaptive logging system (ALS): Enhancing software logging with reinforcement learning techniques,” in Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering (New York, NY: ACM), 37–47.

  • Google Scholar

• 100

  KimB.ParkJ.SuhJ. (2020). Transparency and accountability in AI decision support: explaining and visualizing convolutional neural networks for text information. Decis. Support Syst. 134:113302. doi: 10.1016/j.dss.2020.113302

  • CrossRef
  • Google Scholar

• 101

  KimD.AntariksaG.HandayaniM. P.LeeS.LeeJ. (2021). Explainable anomaly detection framework for maritime main engine sensor data. Sensors21:5200. doi: 10.3390/s21155200

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 102

  KimD.ChungJ.ChoiJ.SucciM. D.ConklinJ.LongoM. G. F.et al. (2022). Accurate auto-labeling of chest x-ray images based on quantitative similarity to an explainable AI model. Nat. Commun. 13:1867. doi: 10.1038/s41467-022-29437-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 103

  KitchenhamB. A.BudgenD.BreretonO. P. (2011). Using mapping studies as the basis for further research-a participant-observer case study. Inform. Softw. Technol. 53:638–651. doi: 10.1016/j.infsof.2010.12.011

  • CrossRef
  • Google Scholar

• 104

  KoeblerA.DeckerT.LebacherM.ThonI.TrespV.BuettnerF. (2023). “Towards explanatory model monitoring,” in XAI in Action: Past, Present, and Future Applications.

  • Google Scholar

• 105

  KostopoulosG.DavrazosG.KotsiantisS. (2024). Explainable artificial intelligence-based decision support systems: a recent review. Electronics13:2842. doi: 10.3390/electronics13142842

  • CrossRef
  • Google Scholar

• 106

  KotsiopoulosT.PapakostasG.VafeiadisT.DimitriadisV.NizamisA.BolzoniA.et al. (2024). Revolutionizing defect recognition in hard metal industry through AI explainability, human-in-the-loop approaches and cognitive mechanisms. Expert Syst. Appl. 255:124839. doi: 10.1016/j.eswa.2024.124839

  • CrossRef
  • Google Scholar

• 107

  KreuzbergerD.KühlN.HirschlS. (2023). Machine learning operations (MLOPS): overview, definition, and architecture. IEEE Access11, 31866–31879. doi: 10.1109/ACCESS.2023.3262138

  • CrossRef
  • Google Scholar

• 108

  KumarR.JaveedD.AljuhaniA.JolfaeiA.KumarP.IslamA. N. (2023). Blockchain-based authentication and explainable AI for securing consumer iot applications. IEEE Trans. Consumer Electron. 70, 1145–1154. doi: 10.1109/TCE.2023.3320157

  • CrossRef
  • Google Scholar

• 109

  KwonS.LeeY. (2023). Explainability-based mix-up approach for text data augmentation. ACM Trans. Knowl. Discov. Data17, 1–14. doi: 10.1145/3533048

  • CrossRef
  • Google Scholar

• 110

  LaberE.MurtinhoL.OliveiraF. (2023). Shallow decision trees for explainable k-means clustering. Pattern Recognit. 137:109239. doi: 10.1016/j.patcog.2022.109239

  • CrossRef
  • Google Scholar

• 111

  LaberE. S.MurtinhoL. (2021). “On the price of explainability for some clustering problems,”? in International Conference on Machine Learning (New York: PMLR), 5915–5925.

  • Google Scholar

• 112

  LawlessC.KalagnanamJ.NguyenL. M.PhanD.ReddyC. (2022). “Interpretable clustering via multi-polytope machines,” in Proceedings of the AAAI Conference on Artificial Intelligence (Washington, DC: AAAI Press), 36, 7309–7316.

  • Google Scholar

• 113

  LaxarD.EitenbergerM.MaleczekM.KaiderA.HammerleF. P.KimbergerO. (2023). The influence of explainable vs non-explainable clinical decision support systems on rapid triage decisions: a mixed methods study. BMC Med. 21:359. doi: 10.1186/s12916-023-03068-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 114

  LeN. Q. K.HoQ.-T.NguyenV.-N.ChangJ.-S. (2022). BERT-promoter: an improved sequence-based predictor of dna promoter using bert pre-trained model and shap feature selection. Comput. Biol. Chem. 99:107732. doi: 10.1016/j.compbiolchem.2022.107732

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 115

  LeeD.MalacarneS.AuneE. (2024). Explainable time series anomaly detection using masked latent generative modeling. Pattern Recognit. 156:110826. doi: 10.1016/j.patcog.2024.110826

  • CrossRef
  • Google Scholar

• 116

  LeeJ.ParkB.ChaeD.-K. (2023). “DuoGAT: dual time-oriented graph attention networks for accurate, efficient and explainable anomaly detection on time-series,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (New York, NY: ACM), 1188–1197. doi: 10.1145/3583780.3614857

  • CrossRef
  • Google Scholar

• 117

  LeungC. K.FungD. L.MaiD.WenQ.TranJ.SouzaJ. (2021). “Explainable data analytics for disease and healthcare informatics,” in Proceedings of the 25th International Database Engineering & Applications Symposium (New York, NY: ACM), 65–74.

  • Google Scholar

• 118

  LiM.SunH.HuangY.ChenH. (2024). Shapley value: from cooperative game to explainable artificial intelligence. Autonomous Intellig. Syst. 4:2. doi: 10.1007/s43684-023-00060-8

  • CrossRef
  • Google Scholar

• 119

  LiN.AdepuS.KangE.GarlanD. (2020). “Explanations for human-on-the-loop: a probabilistic model checking approach,” in Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (Seoul: IEEE), 181–187.

  • Google Scholar

• 120

  LiY.GoelS. (2025). Artificial intelligence auditability and auditor readiness for auditing artificial intelligence systems. Int. J. Account. Inform. Syst. 56:100739. doi: 10.1016/j.accinf.2025.100739

  • CrossRef
  • Google Scholar

• 121

  LiY.YangC. (2021). Domain knowledge based explainable feature construction method and its application in ironmaking process. Eng. Appl. Artif. Intell. 100:104197. doi: 10.1016/j.engappai.2021.104197

  • CrossRef
  • Google Scholar

• 122

  LiuH.WuY.YuZ.ZhangN. (2024). “Please tell me more: Privacy impact of explainability through the lens of membership inference attack,” in 2024 IEEE Symposium on Security and Privacy (SP) (San Francisco, CA: IEEE), 4791–4809.

  • Google Scholar

• 123

  LoetschJ.MalkuschS. (2021). Interpretation of cluster structures in pain-related phenotype data using explainable artificial intelligence (XAI). Eur. J. Pain25, 442–465. doi: 10.1002/ejp.1683

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 124

  LongoL.BrcicM.CabitzaF.ChoiJ.ConfalonieriR.Del SerJ.et al. (2024). Explainable artificial intelligence (XAI) 2.0: a manifesto of open challenges and interdisciplinary research directions. Inform. Fusion106:102301. doi: 10.1016/j.inffus.2024.102301

  • CrossRef
  • Google Scholar

• 125

  Loyola-GonzalezO.Gutierrez-RodríguezA. E.Medina-PérezM. A.MonroyR.Martínez-TrinidadJ. F.Carrasco-OchoaJ. A.et al. (2020). An explainable artificial intelligence model for clustering numerical databases. IEEE Access8, 52370–52384. doi: 10.1109/ACCESS.2020.2980581

  • CrossRef
  • Google Scholar

• 126

  MaS.QiaoW.WuY.LiH.ShiG.GaoD.et al. (2023). Task-oriented explainable semantic communications. IEEE Trans. Wireless Commun. 22, 9248–9262. doi: 10.1109/TWC.2023.3269444

  • CrossRef
  • Google Scholar

• 127

  MakarychevK.ShanL. (2021). “Near-optimal algorithms for explainable k-medians and k-means,” in International Conference on Machine Learning (New York: PMLR), 7358–7367.

  • Google Scholar

• 128

  ManerbaM. M.GuidottiR. (2021). “Fairshades: Fairness auditing via explainability in abusive language detection systems,” in 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) (Atlanta, GA: IEEE), 34–43.

  • Google Scholar

• 129

  ManerikerP.BurleyC.ParthasarathyS. (2023). “Online fairness auditing through iterative refinement,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (New York, NY: ACM) 1665–1676.

  • Google Scholar

• 130

  MarcílioW. E.ElerD. M. (2020). “From explanations to feature selection: assessing shap values as feature selection mechanism,” in 2020 33rd SIBGRAPI conference on Graphics, Patterns and Images (SIBGRAPI) (Porto de Galinhas: IEEE), 340–347.

  • Google Scholar

• 131

  Marcílio-JrW. E.ElerD. M. (2021). Explaining dimensionality reduction results using shapley values. Expert Syst. Appl. 178:115020. doi: 10.1016/j.eswa.2021.115020

  • CrossRef
  • Google Scholar

• 132

  Marín DíazG.Galán HernándezJ. J.Galdón SalvadorJ. L. (2023). Analyzing employee attrition using explainable AI for strategic hr decision-making. Mathematics11:4677. doi: 10.3390/math11224677

  • CrossRef
  • Google Scholar

• 133

  MariottiE.AlonsoJ. M.ConfalonieriR. (2021). “A framework for analyzing fairness, accountability, transparency and ethics: a use-case in banking services,” in 2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) (Luxembourg: IEEE), 1–6.

  • Google Scholar

• 134

  MaulanaF.StarrA.OmpusungguA. P. (2023). Explainable data-driven method combined with Bayesian filtering for remaining useful lifetime prediction of aircraft engines using nasa cmapss datasets. Machines11:163. doi: 10.3390/machines11020163

  • CrossRef
  • Google Scholar

• 135

  MavrogiorgouA.KiourtisA.MakridisG.KotiosD.KoukosV.KyriazisD.et al. (2023). “FAME: federated decentralized trusted data marketplace for embedded finance,” in 2023 International Conference on Smart Applications, Communications and Networking (SmartNets) (Istanbul: IEEE), 1–6.

  • Google Scholar

• 136

  McTavishH.DonnellyJ.SeltzerM.RudinC. (2024). Interpretable generalized additive models for datasets with missing values. Adv. Neural Inf. Process. Syst. 37, 11904–11945. doi: 10.52202/079017-0380

  • CrossRef
  • Google Scholar

• 137

  MershaM. A.YigezuM. G.TonjaA. L.ShakilH.IskandarS.KolesnikovaO.et al. (2025). Explainable AI: XAI-guided context-aware data augmentation. Expert Syst. Appl. 2025:128364. doi: 10.1016/j.eswa.2025.128364

  • CrossRef
  • Google Scholar

• 138

  MidtfjordA. D.De BinR.HusebyA. B. (2022). A decision support system for safer airplane landings: predicting runway conditions using XGBoost and explainable AI. Cold Regions Sci. Technol. 199:103556. doi: 10.1016/j.coldregions.2022.103556

  • CrossRef
  • Google Scholar

• 139

  MikolajczykA.GrochowskiM.KwasigrochA. (2021). Towards explainable classifiers using the counterfactual approach: global explanations for discovering bias in data. J. Artif. Intellig. Soft Comp. Res. 11, 51–67. doi: 10.2478/jaiscr-2021-0004

  • CrossRef
  • Google Scholar

• 140

  MillerT. (2019). Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38. doi: 10.1016/j.artint.2018.07.007

  • CrossRef
  • Google Scholar

• 141

  MontenegroH.SilvaW.CardosoJ. S. (2021). Privacy-preserving generative adversarial network for case-based explainability in medical image analysis. IEEE Access9, 148037–148047. doi: 10.1109/ACCESS.2021.3124844

  • CrossRef
  • Google Scholar

• 142

  MontenegroH.SilvaW.GaudioA.FredriksonM.SmailagicA.CardosoJ. S. (2022). Privacy-preserving case-based explanations: enabling visual interpretability by protecting privacy. IEEE Access10, 28333–28347. doi: 10.1109/ACCESS.2022.3157589

  • CrossRef
  • Google Scholar

• 143

  MortezaaghaP.Makarand JoshiA.RahgozarA. (2025). Inconsistency detection in cancer data classification using explainable-ai. BMC Artif. Intellig. 1:5. doi: 10.1186/s44398-025-00005-6

  • CrossRef
  • Google Scholar

• 144

  MoshkovitzM.DasguptaS.RashtchianC.FrostN. (2020). “Explainable k-means and k-medians clustering,” in International Conference on Machine Learning (New York: PMLR), 7055–7065.

  • Google Scholar

• 145

  MujahidM.KinaE.RustamF.VillarM. G.AlvaradoE. S.De La Torre DiezI.et al. (2024). Data oversampling and imbalanced datasets: an investigation of performance for machine learning and feature engineering. J. Big Data11:87. doi: 10.1186/s40537-024-00943-4

  • CrossRef
  • Google Scholar

• 146

  MüllerJ.StoehrM.OeserA.GaebelJ.StreitM.DietzA.et al. (2020). A visual approach to explainable computerized clinical decision support. Comp. Graphics91, 1–11. doi: 10.1016/j.cag.2020.06.004

  • CrossRef
  • Google Scholar

• 147

  MuvvaS. (2021). Ethical AI and responsible data engineering: a framework for bias mitigation and privacy preservation in large-scale data pipelines. Int. J. Scient. Res. Eng. Managem. 5:09. doi: 10.55041/IJSREM10633

  • CrossRef
  • Google Scholar

• 148

  MylonasN.MollasI.BassiliadesN.TsoumakasG. (2024). Exploring local interpretability in dimensionality reduction: analysis and use cases. Expert Syst. Appl. 252:124074. doi: 10.1016/j.eswa.2024.124074

  • CrossRef
  • Google Scholar

• 149

  NaisehM.Al-ThaniD.JiangN.AliR. (2021a). Explainable recommendation: when design meets trust calibration. World Wide Web. 24, 1857–1884. doi: 10.1007/s11280-021-00916-0

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 150

  NaisehM.Al-ThaniD.JiangN.AliR. (2023). How the different explanation classes impact trust calibration: the case of clinical decision support systems. Int. J. Hum. Comput. Stud. 169:102941. doi: 10.1016/j.ijhcs.2022.102941

  • CrossRef
  • Google Scholar

• 151

  NaisehM.CemilogluD.Al ThaniD.JiangN.AliR. (2021b). Explainable recommendations and calibrated trust: two systematic user errors. Computer54, 28–37. doi: 10.1109/MC.2021.3076131

  • CrossRef
  • Google Scholar

• 152

  NakaoY.StumpfS.AhmedS.NaseerA.StrappelliL. (2022). Toward involving end-users in interactive human-in-the-loop AI fairness. ACM Trans. Interact. Intellig. Syst. 12, 1–30. doi: 10.1145/3514258

  • CrossRef
  • Google Scholar

• 153

  NallakaruppanM.ChaturvediH.GroverV.BalusamyB.JarautP.BahadurJ.et al. (2024). Credit risk assessment and financial decision support using explainable artificial intelligence. Risks12:164. doi: 10.3390/risks12100164

  • CrossRef
  • Google Scholar

• 154

  NanniniL. (2025). “Habemus a right to an explanation: so what? – A framework on transparency-explainability functionality and tensions in the EU AI act,” in Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society (Washington, DC: AAAI Press), 1023–1035.

  • Google Scholar

• 155

  NanniniL.Alonso-MoralJ. M.CataláA.LamaM.BarroS. (2024). Operationalizing explainable artificial intelligence in the european union regulatory ecosystem. IEEE Intell. Syst. 39, 37–48. doi: 10.1109/MIS.2024.3383155

  • CrossRef
  • Google Scholar

• 156

  NarettoF.MonrealeA.GiannottiF. (2025). Evaluating the privacy exposure of interpretable global and local explainers. Trans. Data Priv. 18, 67–93. doi: 10.1109/CogMI56440.2022.00012

  • CrossRef
  • Google Scholar

• 157

  NazatS.LiL.AbdallahM. (2024). XAI-ADS: An explainable artificial intelligence framework for enhancing anomaly detection in autonomous driving systems. IEEE Access12, 48583–48607. doi: 10.1109/ACCESS.2024.3383431

  • CrossRef
  • Google Scholar

• 158

  NguyenM.-D.BouazizA.ValdesV.Rosa CavalliA.MallouliW.Montes De OcaE. (2023). “A deep learning anomaly detection framework with explainability and robustness,” in Proceedings of the 18th International Conference on Availability, Reliability and Security (New York, NY: ACM), 1–7.

  • Google Scholar

• 159

  NiloofarP.Lazarova-MolnarS.OmitaomuF.XuH.LiX. (2023). “A general framework for human-in-the-loop cognitive digital twins,” in 2023 winter simulation conference (WSC) (San Antonio, TX: IEEE), 3202–3213.

  • Google Scholar

• 160

  NisevicM.CuypersA.De BruyneJ. (2024). “Explainable AI: can the AI Act and the GDPR Go Out for a Date?,” in 2024 International Joint Conference on Neural Networks (IJCNN) (Piscataway, NJ: IEEE), 1–8.

  • Google Scholar

• 161

  NiuS.YinQ.MaJ.SongY.XuY.BaiL.et al. (2024). Enhancing healthcare decision support through explainable AI models for risk prediction. Decis. Support Syst. 181:114228. doi: 10.1016/j.dss.2024.114228

  • CrossRef
  • Google Scholar

• 162

  NobelS. N.SultanaS.SinghaS. P.ChakiS.MahiM. J. N.JanT.et al. (2024). Unmasking banking fraud: Unleashing the power of machine learning and explainable AI (XAI) on imbalanced data. Information15:298. doi: 10.3390/info15060298

  • CrossRef
  • Google Scholar

• 163

  OlanF.SpanakiK.AhmedW.ZhaoG. (2025). Enabling explainable artificial intelligence capabilities in supply chain decision support making. Prod. Plann. Cont. 36, 808–819. doi: 10.1080/09537287.2024.2313514

  • CrossRef
  • Google Scholar

• 164

  OlayaP.KennedyD.LlamasR.ValeraL.VargasR.LofsteadJ.et al. (2022). Building trust in earth science findings through data traceability and results explainability. IEEE Trans. Parallel Distrib. Syst. 34, 704–717. doi: 10.1109/TPDS.2022.3220539

  • CrossRef
  • Google Scholar

• 165

  OnariM. A.RezaeeM. J.SaberiM.NobileM. S. (2024). An explainable data-driven decision support framework for strategic customer development. Knowl.-Based Syst. 295:111761. doi: 10.1016/j.knosys.2024.111761

  • CrossRef
  • Google Scholar

• 166

  PaiH.-T.ChungW.-C.FangX.-H.HsuY.-H.HuangS.-T. (2024). “The explainable analytics for exploring misdiagnoses,” in Proceedings of the 2024 8th International Conference on Medical and Health Informatics (New York, NY: ACM), 238–243.

  • Google Scholar

• 167

  Palatnik de SousaI.VellascoM. M.Costa da SilvaE. (2021). Explainable artificial intelligence for bias detection in COVID CT-scan classifiers. Sensors21:5657. doi: 10.3390/s21165657

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 168

  PanagouliasD. P.SarmasE.MarinakisV.VirvouM.TsihrintzisG. A.DoukasH. (2023). Intelligent decision support for energy management: a methodology for tailored explainability of artificial intelligence analytics. Electronics12:4430. doi: 10.3390/electronics12214430

  • CrossRef
  • Google Scholar

• 169

  PandiyanV.WróbelR.LeinenbachC.ShevchikS. (2023). Optimizing in-situ monitoring for laser powder bed fusion process: deciphering acoustic emission and sensor sensitivity with explainable machine learning. J. Mater. Proc. Technol. 321:118144. doi: 10.1016/j.jmatprotec.2023.118144

  • CrossRef
  • Google Scholar

• 170

  PaniguttiC.BerettaA.FaddaD.GiannottiF.PedreschiD.PerottiA.et al. (2023). Co-design of human-centered, explainable AI for clinical decision support. ACM Trans. Interact. Intellig. Syst. 13, 1–35. doi: 10.1145/3587271

  • CrossRef
  • Google Scholar

• 171

  PapagniG.de PagterJ.ZafariS.FilzmoserM.KoeszegiS. T. (2023). Artificial agents' explainability to support trust: considerations on timing and context. AI & Soc. 38, 947–960. doi: 10.1007/s00146-022-01462-7

  • CrossRef
  • Google Scholar

• 172

  ParaR. K. (2024). The role of explainable AI in bias mitigation for hyper-personalization. J. Artif. Intellig. General Sci. 6, 625–635. doi: 10.60087/jaigs.v6i1.289

  • CrossRef
  • Google Scholar

• 173

  ParkS.MoonJ.HwangE. (2020). “Explainable anomaly detection for district heating based on shapley additive explanations,” in 2020 International Conference on Data Mining Workshops (ICDMW) (Sorrento: IEEE), 762–765.

  • Google Scholar

• 174

  PatilA.FramewalaA.KaziF. (2020). “Explainability of smote based oversampling for imbalanced dataset problems,” in 2020 3rd international conference on information and computer technologies (ICICT) (San Jose, CA: IEEE), 41–45.

  • Google Scholar

• 175

  PengX.LiY.TsangI. W.ZhuH.LvJ.ZhouJ. T. (2022). XAI beyond classification: Interpretable neural clustering. J. Mach. Learn. Res. 23, 1–28.

  • Google Scholar

• 176

  PierceR. L.Van BiesenW.Van CauwenbergeD.DecruyenaereJ.SterckxS. (2022). Explainability in medicine in an era of AI-based clinical decision support systems. Front. Genet. 13:903600. doi: 10.3389/fgene.2022.903600

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 177

  PrabhakaranK.DridiJ.AmayriM.BouguilaN. (2022). Explainable k-means clustering for occupancy estimation. Procedia Comput. Sci. 203, 326–333. doi: 10.1016/j.procs.2022.07.041

  • CrossRef
  • Google Scholar

• 178

  PuiuA.VizitiuA.NitaC.ItuL.SharmaP.ComaniciuD. (2021). Privacy-preserving and explainable AI for cardiovascular imaging. Stud. Inform. Cont. 30, 21–32. doi: 10.24846/v30i2y202102

  • CrossRef
  • Google Scholar

• 179

  QianC.LiuY.Barnett-NeefsC.SalgiaS.SerbetciO.AdaljaA.et al. (2023). A perspective on data sharing in digital food safety systems. Crit. Rev. Food Sci. Nutr. 63, 12513–12529. doi: 10.1080/10408398.2022.2103086

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 180

  RabieeM.MirhashemiM.PangburnM. S.PiriS.DelenD. (2024). Towards explainable artificial intelligence through expert-augmented supervised feature selection. Decis. Support Syst. 181:114214. doi: 10.1016/j.dss.2024.114214

  • CrossRef
  • Google Scholar

• 181

  RadB.SongF.JacobV.DiaoY. (2021). “Explainable anomaly detection on high-dimensional time series data,” in Proceedings of the 15th ACM International Conference on Distributed and Event-based Systems (New York, NY: ACM), 2–14.

  • Google Scholar

• 182

  RazaA.TranK. P.KoehlL.LiS. (2022). Designing ECG monitoring healthcare system with federated transfer learning and explainable AI. Knowl.-Based Syst. 236:107763. doi: 10.1016/j.knosys.2021.107763

  • CrossRef
  • Google Scholar

• 183

  RenauQ.DréoJ.DoerrC.DoerrB. (2021). “Towards explainable exploratory landscape analysis: extreme feature selection for classifying BBOB functions,” in International Conference on the Applications of Evolutionary Computation (Part of EvoStar) (Cham: Springer), 17–33.

  • Google Scholar

• 184

  RoscherR.BohnB.DuarteM. F.GarckeJ. (2020). Explainable machine learning for scientific insights and discoveries. IEEE Access8, 42200–42216. doi: 10.1109/ACCESS.2020.2976199

  • CrossRef
  • Google Scholar

• 185

  SaadallahA. (2023). “Online explainable model selection for time series forecasting,” in 2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA) (Thessaloniki: IEEE), 1–10.

  • Google Scholar

• 186

  SabolP.SinčákP.HartonoP.KočanP.BenetinováZ.BlichárováA.et al. (2020). Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images. J. Biomed. Inform. 109:103523. doi: 10.1016/j.jbi.2020.103523

  • CrossRef
  • Google Scholar

• 187

  SachanS.YangJ.-B.XuD.-L.BenavidesD. E.LiY. (2020). An explainable AI decision-support-system to automate loan underwriting. Expert Syst. Appl. 144:113100. doi: 10.1016/j.eswa.2019.113100

  • CrossRef
  • Google Scholar

• 188

  SadeghiK.OjhaD.KaurP.MahtoR. V.DhirA. (2024). Explainable artificial intelligence and agile decision-making in supply chain cyber resilience. Decis. Support Syst. 180:114194. doi: 10.1016/j.dss.2024.114194

  • CrossRef
  • Google Scholar

• 189

  SaisubramanianS.GalhotraS.ZilbersteinS. (2020). “Balancing the tradeoff between clustering value and interpretability,” in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (New York, NY: ACM), 351–357.

  • Google Scholar

• 190

  SaltzJ. S.ShamshurinI. (2016). “Big data team process methodologies: A literature review and the identification of key factors for a project's success,” in 2016 IEEE International Conference on Big Data (Big Data) (Washington, DC: IEEE), 2872–2879.

  • Google Scholar

• 191

  SambaturuP.GuptaA.DavidsonI.RaviS.VullikantiA.WarrenA. (2020). “Efficient algorithms for generating provably near-optimal cluster descriptors for explainability,” in Proceedings of the AAAI Conference on Artificial Intelligence (Washington, DC: AAAI Press), 34, 1636–1643.

  • Google Scholar

• 192

  SánchezP. M. S.CeldránA. H.XieN.BovetG.PérezG. M.StillerB. (2024). Federatedtrust: a solution for trustworthy federated learning. Future Generat. Comp. Syst. 152, 83–98. doi: 10.1016/j.future.2023.10.013

  • CrossRef
  • Google Scholar

• 193

  SantosM. R.GuedesA.Sanchez-GendrizI. (2024). Shapley additive explanations (SHAP) for efficient feature selection in rolling bearing fault diagnosis. Mach. Learn. Knowl. Extract. 6, 316–341. doi: 10.3390/make6010016

  • CrossRef
  • Google Scholar

• 194

  Sayed-MouchawehM.RajaoarisoaL. (2022). “Explainable decision support tool for IOT predictive maintenance within the context of industry 4.0,” in 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA) (Nassa: IEEE), 1492–1497.

  • Google Scholar

• 195

  SchoonderwoerdT. A.JorritsmaW.NeerincxM. A.Van Den BoschK. (2021). Human-centered XAI: Developing design patterns for explanations of clinical decision support systems. Int. J. Hum. Comput. Stud. 154:102684. doi: 10.1016/j.ijhcs.2021.102684

  • CrossRef
  • Google Scholar

• 196

  ScutariM. (2020). Bayesian network models for incomplete and dynamic data. Stat. Neerl. 74, 397–419. doi: 10.1111/stan.12197

  • CrossRef
  • Google Scholar

• 197

  SghaireenM. G.Al-SmadiY.Al-QeremA.SrivastavaK. C.GanjiK. K.AlamM. K.et al. (2022). Machine learning approach for metabolic syndrome diagnosis using explainable data-augmentation-based classification. Diagnostics12:3117. doi: 10.3390/diagnostics12123117

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 198

  ShaerI.ShamiA. (2024). “Thwarting cybersecurity attacks with explainable concept drift,” in 2024 International Wireless Communications and Mobile Computing (IWCMC) (Ayia Napa: IEEE), 1785–1790.

  • Google Scholar

• 199

  ShafinS. S. (2024). An explainable feature selection framework for web phishing detection with machine learning. Data Sci. Managem. doi: 10.1016/j.dsm.2024.08.004

  • CrossRef
  • Google Scholar

• 200

  ShahadP.RajE. D. (2024). “Interpretability-based virtual drift detection and adaptation algorithm: A case study on Tetouan's energy data,” in 2024 International Conference on Intelligent Computing and Emerging Communication Technologies (ICEC) (Guntur: IEEE), 1–6.

  • Google Scholar

• 201

  ShekkizharS.OrtegaA. (2021). “Model selection and explainability in neural networks using a polytope interpolation framework,” in 2021 55th Asilomar Conference on Signals, Systems, and Computers (Pacific Grove, CA: IEEE), 177–181.

  • Google Scholar

• 202

  ShinD. (2020). User perceptions of algorithmic decisions in the personalized AI system: Perceptual evaluation of fairness, accountability, transparency, and explainability. J. Broadcast. Elect. Media64, 541–565. doi: 10.1080/08838151.2020.1843357

  • CrossRef
  • Google Scholar

• 203

  ShinH.ParkJ.YuJ.KimJ.KimH. Y.OhC. (2025). Looping in: Exploring feedback strategies to motivate human engagement in interactive machine learning. Int. J. Hum. Comput. Interact. 41, 8666–8683. doi: 10.1080/10447318.2024.2413293

  • CrossRef
  • Google Scholar

• 204

  ShokriR.StrobelM.ZickY. (2021). “On the privacy risks of model explanations,” in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (New York, NY: ACM), 231–241.

  • Google Scholar

• 205

  Shulner-TalA.KuflikT.KligerD. (2022). Fairness, explainability and in-between: Understanding the impact of different explanation methods on non-expert users' perceptions of fairness toward an algorithmic system. Ethics Inf. Technol. 24:2. doi: 10.1007/s10676-022-09623-4

  • CrossRef
  • Google Scholar

• 206

  Siddique AyonS.Ebrahim HossainM.Ullah MiahM. S.RahmanM. M.MahmudM. (2024). “Explainable AI in feature selection: Improving classification performance on imbalanced datasets,” in International Conference on Neural Information Processing (Cham: Springer), 303–318.

  • Google Scholar

• 207

  ŠimićI.VeasE.SabolV. (2025). A comprehensive analysis of perturbation methods in explainable AI feature attribution validation for neural time series classifiers. Sci. Rep. 15:26607. doi: 10.1038/s41598-025-09538-2

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 208

  SimuniG. (2024). Explainable AI in ML: The path to transparency and accountability. Int. J. Recent Adv. Multidiscipl. Res. 11, 10531–10536.

  • Google Scholar

• 209

  SinglaS.NushiB.ShahS.KamarE.HorvitzE. (2021). “Understanding failures of deep networks via robust feature extraction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (Nashville, TN: IEEE), 12853–12862.

  • Google Scholar

• 210

  Smith-RennerA.FanR.BirchfieldM.WuT.Boyd-GraberJ.WeldD. S.et al. (2020). “No explainability without accountability: an empirical study of explanations and feedback in interactive ml,” in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (New York, NY: ACM), 1–13.

  • Google Scholar

• 211

  SongQ.LinP.MaH.WuY. (2021). “Explaining missing data in graphs: a constraint-based approach,” in 2021 IEEE 37th International Conference on Data Engineering (ICDE) (Chania: IEEE), 1476–1487.

  • Google Scholar

• 212

  SongS.SunY. (2020). “Imputing various incomplete attributes via distance likelihood maximization,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (New York, NY: ACM), 535–545.

  • Google Scholar

• 213

  StarkeG.SchmidtB.De ClercqE.ElgerB. S. (2023). Explainability as fig leaf? An exploration of experts' ethical expectations towards machine learning in psychiatry. AI and Ethics3, 303–314. doi: 10.1007/s43681-022-00177-1

  • CrossRef
  • Google Scholar

• 214

  SunR.XueM.TysonG.DongT.LiS.WangS.et al. (2023). “Mate! are you really aware? An explainability-guided testing framework for robustness of malware detectors,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (New York, NY: ACM), 1573–1585.

  • Google Scholar

• 215

  SunT. S.GaoY.KhaladkarS.LiuS.ZhaoL.KimY.-H.et al. (2023). Designing a direct feedback loop between humans and convolutional neural networks through local explanations. Proc. ACM Human-Comp. Interact. 7, 1–32. doi: 10.1145/3610187

  • CrossRef
  • Google Scholar

• 216

  SzymanowiczS.CharlesJ.CipollaR. (2022). “Discrete neural representations for explainable anomaly detection,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 148–156.

  • Google Scholar

• 217

  TaesiriM. R.NguyenG.NguyenA. (2022). Visual correspondence-based explanations improve AI robustness and human-AI team accuracy. Adv. Neural Inf. Process. Syst. 35, 34287–34301. doi: 10.52202/068431-2485

  • CrossRef
  • Google Scholar

• 218

  TavaresC.NascimentoN.AlencarP.CowanD. (2022). “Adaptive method for machine learning model selection in data science projects,” in 2022 IEEE International Conference on Big Data (Big Data) (Osaka: IEEE), 2682–2688.

  • Google Scholar

• 219

  TavaresC.NascimentoN.AlencarP.CowanD. (2023). “Extending variability-aware model selection with bias detection in machine learning projects,” in 2023 IEEE International Conference on Big Data (BigData) (Piscataway, NJ: IEEE), 2441–2449.

  • Google Scholar

• 220

  TavaresC.NascimentoN.AlencarP.CowanD. (2025). Variability-aware machine learning model selection: Feature modeling, instantiation, and experimental case study. IEEE Access. doi: 10.1109/ACCESS.2025.3558218

  • CrossRef
  • Google Scholar

• 221

  TekkesinogluS. (2023). Exploring Evaluation Methodologies for Explainable AI: Guidelines for Objective and Subjective Assessment.

  • Google Scholar

• 222

  TekkesinogluS.PudasS. (2024). Explaining graph convolutional network predictions for clinicians–an explainable AI approach to Alzheimer's disease classification. Front. Artif. Intellig. 6:1334613. doi: 10.3389/frai.2023.1334613

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 223

  ThrunM. C.UltschA.BreuerL. (2021). Explainable AI framework for multivariate hydrochemical time series. Mach. Learn. Knowl. Extract. 3, 170–204. doi: 10.3390/make3010009

  • CrossRef
  • Google Scholar

• 224

  TiensuuH.TamminenS.PuukkoE.RöningJ. (2021). Evidence-based and explainable smart decision support for quality improvement in stainless steel manufacturing. Appl. Sci. 11:10897. doi: 10.3390/app112210897

  • CrossRef
  • Google Scholar

• 225

  TomarS.TirupathiS.SalwalaD. V.DusparicI.DalyE. (2022). “Prequential model selection for time series forecasting based on saliency maps,” in 2022 IEEE International Conference on Big Data (Big Data) (IEEE), 3383–3392.

  • Google Scholar

• 226

  TomsettR.PreeceA.BrainesD.CeruttiF.ChakrabortyS.SrivastavaM.et al. (2020). Rapid trust calibration through interpretable and uncertainty-aware AI. Patterns1:4. doi: 10.1016/j.patter.2020.100049

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 227

  TripathyS. M.ChouhanA.DixM.KotriwalaA.KlöpperB.PrabhuneA. (2022). “Explaining anomalies in industrial multivariate time-series data with the help of eXplainable AI,” in 2022 IEEE International Conference on Big Data and Smart Computing (BigComp) (Daegu: IEEE), 226–233.

  • Google Scholar

• 228

  TrostC.ŽákS.SchafferS.WalchL.ZitzJ.KlünsnerT.et al. (2025). Explainable machine learning and feature engineering applied to nanoindentation data. Mater. Design253:113897. doi: 10.1016/j.matdes.2025.113897

  • CrossRef
  • Google Scholar

• 229

  TsiakasK.Murray-RustD. (2022). “Using human-in-the-loop and explainable AI to envisage new future work practices,” in Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments (New York, NY: ACM), 588–594.

  • Google Scholar

• 230

  ValenteF.ParedesS.HenriquesJ.RochaT.de CarvalhoP.MoraisJ. (2022). Interpretability, personalization and reliability of a machine learning based clinical decision support system. Data Min. Knowl. Discov. 36, 1140–1173. doi: 10.1007/s10618-022-00821-8

  • CrossRef
  • Google Scholar

• 231

  van der WaaJ.SchoonderwoerdT.van DiggelenJ.NeerincxM. (2020). Interpretable confidence measures for decision support systems. Int. J. Hum. Comput. Stud. 144:102493. doi: 10.1016/j.ijhcs.2020.102493

  • CrossRef
  • Google Scholar

• 232

  Van SteinB.VermettenD.CaraffiniF.KononovaA. V. (2023). “Deep bias: Detecting structural bias using explainable AI,” in Proceedings of the Companion Conference on Genetic and Evolutionary Computation (New York, NY: ACM), 455–458.

  • Google Scholar

• 233

  Van ZylC.YeX.NaidooR. (2024). Harnessing explainable artificial intelligence for feature selection in time series energy forecasting: a comparative analysis of grad-cam and shap. Appl. Energy353:122079. doi: 10.1016/j.apenergy.2023.122079

  • CrossRef
  • Google Scholar

• 234

  VieiraD. M.FernandesC.LucenaC.LifschitzS. (2021). Driftage: a multi-agent system framework for concept drift detection. GigaScience10:giab030. doi: 10.1093/gigascience/giab030

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 235

  VijayanM.SridharS.VijayalakshmiD. (2022). A deep learning regression model for photonic crystal fiber sensor with XAI feature selection and analysis. IEEE Trans. Nanobiosci. 22, 590–596. doi: 10.1109/TNB.2022.3221104

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 236

  VlahekD.MongusD. (2021). An efficient iterative approach to explainable feature learning. IEEE Trans. Neural Netw. Learn. Syst. 34, 2606–2618. doi: 10.1109/TNNLS.2021.3107049

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 237

  WagnerM.WittmannM.-T.BorgM. (2025). “A critical look at the EU AI act's requirements and affected systems-who must explain what?,” in 2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW), 494–503.

  • Google Scholar

• 238

  WaldisA.MazzolaL.DenzlerA. (2020). “Towards explainable AI in text features engineering for concept recognition,” in International Conference on Statistical Language and Speech Processing (Cham: Springer), 122–133.

  • Google Scholar

• 239

  WangZ.HuangC.YaoX. (2024). A roadmap of explainable artificial intelligence: explain to whom, when, what and how?ACM Trans. Autonom. Adapt. Syst. 19, 1–40. doi: 10.1145/3702004

  • CrossRef
  • Google Scholar

• 240

  WebbM. E.FluckA.MagenheimJ.Malyn-SmithJ.WatersJ.DeschênesM.et al. (2021). Machine learning for human learners: opportunities, issues, tensions and threats. Educ. Technol. Res. Dev. 69, 2109–2130. doi: 10.1007/s11423-020-09858-2

  • CrossRef
  • Google Scholar

• 241

  WesterskiA.KanagasabaiR.ShahamE.NarayananA.WongJ.SinghM. (2021). Explainable anomaly detection for procurement fraud identification–lessons from practical deployments. Int. Trans. Operat. Res. 28, 3276–3302. doi: 10.1111/itor.12968

  • CrossRef
  • Google Scholar

• 242

  WickramasingheC. S.AmarasingheK.MarinoD. L.RiegerC.ManicM. (2021). Explainable unsupervised machine learning for cyber-physical systems. IEEE Access9, 131824–131843. doi: 10.1109/ACCESS.2021.3112397

  • CrossRef
  • Google Scholar

• 243

  WintersbergerP.NicklasH.MartlbauerT.HammerS.RienerA. (2020). “Explainable automation: Personalized and adaptive UIs to foster trust and understanding of driving automation systems,” in 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 252–261.

  • Google Scholar

• 244

  WoenselW. V.SciosciaF.LosetoG.SeneviratneO.PattonE.AbidiS.et al. (2022). “Explainable clinical decision support: towards patient-facing explanations for education and long-term behavior change,” in International Conference on Artificial Intelligence in Medicine (Cham: Springer), 57–62.

  • Google Scholar

• 245

  WoodbrightM. D.MorshedA.BrowneM.RayB.MooreS. (2024). “Optimised hybrid-model selection for the autonomous relevance technique explainable framework,” in 2024 International Conference on Data Science and Its Applications (ICoDSA) (Kuta: IEEE), 380–385.

  • Google Scholar

• 246

  WuC.ShaoS.TuncC.SatamP.HaririS. (2022). An explainable and efficient deep learning framework for video anomaly detection. Cluster Comput. 25, 2715–2737. doi: 10.1007/s10586-021-03439-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 247

  XuM.WangY. (2024). Explainability increases trust resilience in intelligent agents. Br. J. Psychol. 117, 528–547. doi: 10.1111/bjop.12740

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 248

  YadavC.MoshkovitzM.ChaudhuriK. (2024). XAudit: A learning-theoretic look at auditing with explanations. Trans. Mach. Learn. Res.

  • Google Scholar

• 249

  YeboahD.SteinmeisterL.HierD. B.HadiB.WunschD. C.OlbrichtG. R.et al. (2020). An explainable and statistically validated ensemble clustering model applied to the identification of traumatic brain injury subgroups. IEEE Access8, 180690–180705. doi: 10.1109/ACCESS.2020.3027453

  • CrossRef
  • Google Scholar

• 250

  YoungB.AndersonD. T.KellerJ.PetryF.MichaelC. J. (2025). Sparc: a human-in-the-loop framework for learning and explaining spatial concepts. Information16:252. doi: 10.3390/info16040252

  • CrossRef
  • Google Scholar

• 251

  ZachariasJ.von ZahnM.ChenJ.HinzO. (2022). Designing a feature selection method based on explainable artificial intelligence. Elect. Markets32, 2159–2184. doi: 10.1007/s12525-022-00608-1

  • CrossRef
  • Google Scholar

• 252

  ZafariS.de PagterJ.PapagniG.RosensteinA.FilzmoserM.KoeszegiS. T. (2024). Trust development and explainability: a longitudinal study with a personalized assistive system. Multimodal Technol. Interact. 8:20. doi: 10.3390/mti8030020

  • CrossRef
  • Google Scholar

• 253

  ZamanianA.AhmidiN.DrtonM. (2023). Assessable and interpretable sensitivity analysis in the pattern graph framework for nonignorable missingness mechanisms. Stat. Med., 42, 5419–5450. doi: 10.1002/sim.9920

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 254

  ZangZ.ChengS.XiaH.LiL.SunY.XuY.et al. (2022). DMT-EV: an explainable deep network for dimension reduction. IEEE Trans. Vis. Comput. Graph., 30, 1710–1727. doi: 10.1109/TVCG.2022.3223399

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 255

  ZhangC. A.ChoS.VasarhelyiM. (2022a). Explainable artificial intelligence (XAI) in auditing. Int. J. Accoun. Inform. Syst. 46:100572. doi: 10.1016/j.accinf.2022.100572

  • CrossRef
  • Google Scholar

• 256

  ZhangN.BahsoonR.TziritasN.TheodoropoulosG. (2022b). “Explainable human-in-the-loop dynamic data-driven digital twins,” in International Conference on Dynamic Data Driven Applications Systems (Cham: Springer), 233–243.

  • Google Scholar

• 257

  ZhangY.LiaoQ. V.BellamyR. K. (2020). “Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making,” in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, (New York, NY: ACM), 295–305.

  • Google Scholar

• 258

  ZhaoL.XieH.ZhongL.WangY. (2024). Explainable federated learning scheme for secure healthcare data sharing. Health Inform. Sci. Syst. 12:49. doi: 10.1007/s13755-024-00306-6

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 259

  ZhengM.PanX.BermeoN. V.ThomasR. J.CoyleD.O'hareG. M.et al. (2022). Stare: Augmented reality data visualization for explainable decision support in smart environments. IEEE Access10, 29543–29557. doi: 10.1109/ACCESS.2022.3156697

  • CrossRef
  • Google Scholar

• 260

  ZhongC.GoelS. (2024). Transparent AI in auditing through explainable AI. Curr. Issues Audit. 18, A1–14. doi: 10.2308/CIIA-2023-009

  • CrossRef
  • Google Scholar