ACCRUE—An Integral Index for Measuring Experimental Relevance in Support of Neutronic Model Validation

Abstract

A key challenge for the introduction of any design changes, e.g., advanced fuel concepts, first-of-a-kind nuclear reactor designs, etc., is the cost of the associated experiments, which are required by law to validate the use of computer models for the various stages, starting from conceptual design, to deployment, licensing, operation, and safety. To achieve that, a criterion is needed to decide on whether a given experiment, past or planned, is relevant to the application of interest. This allows the analyst to select the best experiments for the given application leading to the highest measures of confidence for the computer model predictions. The state-of-the-art methods rely on the concept of similarity or representativity, which is a linear Gaussian-based inner-product metric measuring the angle—as weighted by a prior model parameters covariance matrix—between two gradients, one representing the application and the other a single validation experiment. This manuscript emphasizes the concept of experimental relevance which extends the basic similarity index to account for the value accrued from past experiments and the associated experimental uncertainties, both currently missing from the extant similarity methods. Accounting for multiple experiments is key to the overall experimental cost reduction by prescreening for redundant information from multiple equally-relevant experiments as measured by the basic similarity index. Accounting for experimental uncertainties is also important as it allows one to select between two different experimental setups, thus providing for a quantitative basis for sensor selection and optimization. The proposed metric is denoted by ACCRUE, short for Accumulative Correlation Coefficient for Relevance of Uncertainties in Experimental validation. Using a number of criticality experiments for highly enriched fast metal systems and low enriched thermal compound systems with accident tolerant fuel concept, the manuscript will compare the performance of the ACCRUE and basic similarity indices for prioritizing the relevance of a group of experiments to the given application.

1 Introduction

Model validation is one of the key regulatory requirements to develop a scientifically-defendable process in support of establishing confidence in the results of computerized physics models for the various developmental stages starting from conceptual design to deployment, licensing, operation, and safety. To ensure that model predictions can be trusted for a given application, e.g., the domain envisaged for code usage, the regulatory process requires the consolidation of two independent sources of knowledge, one from measurements collected from experimental conditions that are similar to the application, and the other from code predictions that model the same experimental conditions. For criticality safety applications, representing the focus of this manuscript, model validation plays a critical role in supporting design changes, e.g., the introduction of high burnup fuel, high assay low enrichment fuel, etc., or new fuel designs, e.g., accident tolerant fuel, both typically challenged by the scarcity of experimental data.

It is thus paramount to devise a methodology that can consolidate knowledge from both the experimental and computational domains in some optimal manner. The optimality of this consolidation process needs to recognize the possible scarcity of relevant experimental data expected with new designs, the cost for constructing new validation experiments, and the infeasibility of duplicating of all application conditions in the experimental domain. Ideally, the consolidation methodology should be able to optimally leverage existing experimental data in order to minimize the need for new experiments.

In our context, model validation entails a mapping process in which the experimental biases (differences between measurements and model predictions) are to be mapped to the application’s responses of interest in the form of calculational (i.e., best-estimate) biases along with their uncertainties. The goal is to improve the analyst’s confidence in the calculated application response. Mathematically, the confidence is measured in terms of the response uncertainty. The initial uncertainty propagated throughout the model is referred to as the prior uncertainty which accounts for parameter uncertainties, modeling assumptions, numerical approximations, initial, and boundary conditions, etc. The consolidation of experimental biases with the prior uncertainties results in a calculational bias that is intended to correct for the prior uncertainties. A successful consolidation process would result in a reduced bias uncertainty, i.e., as compared to the prior uncertainty, implying increased confidence in the calculated response.

The prior uncertainties are often grouped into two categories, aleatory, and epistemic. This manuscript will focus on epistemic uncertainties resulting from the lack of knowledge of the true values of the nuclear cross-section data. The implied assumption here is that cross-sections constitute the major source of uncertainty in neutronic calculations (Glaeser, 2008; Avramova and Ivanov, 2010; Abdel-Khalik et al., 2013; Wieselquist, 2016). Specifically, we focus on a single consolidation methodology for reducing the impact of epistemic uncertainties, the so-called generalized linear least-squares (GLLS) methodology which may be derived using Bayesian estimation theory (Williams et al., 2016). It is designed to calculate an optimal bias for any calculated response based on an optimal adjustment of the nuclear cross-sections.

In the neutronic community, the GLLS methodology has been independently developed by various researchers (Gandini, 1967; Salvatores, 1973; Broadhead et al., 2004; Cacuci and Ionescu-Bujor, 2010) with varying levels of generalization and mathematical formulation, e.g., Gaussianity assumption of the uncertainty source, degree of nonlinearity of the response variations with cross-sections, mathematical formulation in the cross-section space or the response space, etc. Under the same set of assumptions however, one can show the equivalence of these various formulations. For example, for Gaussian prior cross-section uncertainties and assumed linear approximations, the noted GLLS optimality criterion reduces to an L₂ minimization of the sum of two terms, see Eq. 4. The first term minimizes the L₂ norm of the adjustments of the cross-sections to ensure their consistency with their prior values, and the second term minimizes the discrepancy between the measurements and predictions for the selected experimental responses. The GLLS methodology is briefly discussed in Section 2.

A prerequisite for the GLLS methodology is to select the experiments that are most relevant to the application conditions¹. The premise is that with higher relevance biases with higher confidence, i.e., reduced uncertainties, can be calculated. In the neutronic community, sensitivity methods have been adopted to determine experimental relevance using a scalar quantity, denoted by the similarity index c_k (Broadhead et al., 2004)—also called representativity factor by other researchers (Palmiotti and Salvatores, 2012)—which can be used to prioritize/rank the experiments and possibly judge the value of a new experiment before it is constructed.

To measure the similarity index c_k, sensitivity methods are first employed to calculate the first-order variations in select quantities of interest that can be experimentally measured, e.g., critical eigenvalue, reaction rate ratios, etc., with respect to the cross-section variations by isotope, reaction type, and incident neutron energy. This is done with both the experimental models as well as the application model of interest, e.g., calculating the criticality conditions for a new fuel design, resulting in one sensitivity vector per model. The sensitivity vector comprises the first-order derivatives, i.e., sensitivity coefficients, of a given response with respect to all cross-sections. Next, the sensitivity vector of the experiment is folded with that of the application and the prior cross-section covariance matrix to calculate the similarity index.

The result of this folding process, see Eq. 5, is an integral quantity (c_k) taken to measure the degree of similarity between the first order derivatives of a single quantity of interest with respect to all cross-sections as calculated from both the experiment and the application models². The prior uncertainties are used as weighting parameters, assigning more weight to cross-sections with higher uncertainties. The resulting similarity index c_k is thus expected to be heavily influenced by cross-sections exhibiting both high prior uncertainties as well as strong sensitivities. This helps the GLLS methodology find the optimal adjustments for cross-sections with strong sensitivities as well as high uncertainties. This is justified as follows: cross-sections with weak sensitivities would require large adjustments to change the response, potentially rendering them statistically inconsistent with their prior values. Similarly, adjusting low-uncertainty cross-sections would violate their prior values, also considered a form of fudging that cannot be mathematically justified as it violates the basic assumption of the GLLS methodology, that is the observed discrepancies are mainly originating from the prior cross-section uncertainties.

The resulting similarity index c_k is a scalar quantity which lies between −1.0 and 1.0 and may be interpreted as follows: a zero value implies no correlations, i.e., cross-sections with strong sensitivities and high uncertainties, exist between the application, and the experimental conditions. This implies that experimental bias cannot be used to infer the application bias, i.e., it cannot be used to improve the prior estimate of the application response and hence the experiment is judged to have no value to the given application. Conversely, a high similarity value, i.e., close to 1.0, implies that the associated experimental bias can be reliably used to infer the application bias. More important, the bias uncertainty can be reduced with highly relevant experiments. Theoretically, the inclusion of a zero-similarity experiment would keep the prior uncertainty for the application unchanged—not increasing the confidence—while a perfectly similar experiment, e.g., c_k 1.0, would result in the minimum bias uncertainty, i.e., the maximum attainable confidence³.

One key limitation of the similarity index is that it does not account for the impact of measurement uncertainties. Essentially, the c_k value is obtained by normalizing the covariance matrix for the calculated responses. This further implies that the measurements uncertainties have no impact on the c_k value calculation. To explain this, consider two experiments with analogous similarity as measured by the c_k value but with different measurement uncertainties. The experiment with the lower uncertainty would result in the calculation of lower bias uncertainty, i.e., more confidence. This implies that an experiment with a lower c_k value and a low measurement uncertainty could result in a lower bias uncertainty than that obtained from an experiment with higher c_k value and higher measurement uncertainty. Thus, it is important to include the measurement uncertainty in the definition of relevance. This brings value to the design of future experiments, often involving an optimization of sensors’ types and placements. Inclusion of measurement uncertainty would allow the analyst to compare the value of different experiments (and sensors selection) prior to the conduction of the experiment.

Another limitation of the similarity index is that it does not consider the impact of past experiments. As the c_k value is calculated by normalizing the weighted inner product of two sensitivity vectors, with more experiments involved in the relevance evaluation process, the c_k value cannot be employed to capture a weighted relevance between two subspaces. To explain this, consider that the analyst has calculated the application bias using ideal conditions, i.e., with a highly relevant experiment and near zero measurement uncertainty. In this scenario, the inclusion of additional experiments, even if highly relevant, is unlikely to lead to further noticeable reduction in the bias uncertainty. Thus, two experiments with the same c_k value should be assigned different relevance depending on which experiment is employed first. This provides a lot of value when designing new experiments by quantifying the maximum possible increase in confidence while accounting for past experiments. Addressing these two limitations will help analysts determine the minimum number of experiments required to meet a preset level of increased confidence as well as compare the value of planned experiments, providing a quantitative approach for their optimization.

In response to these limitations, this manuscript employs the concept of experimental relevance as opposed to similarity in order to distinguish between the possible added value of a new experiment, if any, and the value available from past experiments. This is possible by extending the definition of the c_k similarity index⁴via a new analytical expression for experimental relevance, denoted by the ACCRUE index, designed to account for the experiment’s measurement uncertainty and the prior confidence associated with past experiments. The symbol j_k is used to distinguish the ACCRUE index from the similarity index c_k, where the j denotes the ability to jointly assess the relevance of an experiment with past experiments. The ACCRUE index is short for Accumulated Correlation Coefficient for Relevance of Uncertainties in Experimental validation.

The TSURFER code, representing the GLLS rendition under the ORNL’s SCALE code suite, is employed to exemplify the application of the ACCRUE index j_k. Specifically, we develop three sorting methods for the available experiments based respectively on the similarity index c_k, the ACCRUE index j_k, and pure random sampling. Two different sets of experiments are employed to compare the performance of the various sorting methods. The first set involves low-enriched uranium thermal compound systems with the accident-tolerant fuel (ATF) concept BWR assembly, and the second comprises highly enriched uranium fast metal systems. Finally, numerical experiments will be employed to verify the analytically-calculated values for j_k.

This manuscript is organized as follows. Section 2 provides a background on sensitivity theory and the details of mathematical description of the GLLS nuclear data adjustment methodology. Section 3 introduces ACCRUE algorithm and an extension of the non-intrusive stochastic verification with mathematical details. Section 4 verifies the performance of the proposed algorithm by numerical experiments to compare the results made by one of the conventional integral similarity indices, c_k¸ and the ACCRUE index, j_k. Concluding remarks and further research are summarized in Section 5.

2 Background and Relevance

This section presents a brief background on three key topics: 1) sensitivity methods employed for the calculation of first-order derivatives; 2) the GLLS adjustment theory, employed to calculate the application bias; and 3) the extant similarity index c_k definition. The material in this section may be found in the literature, however compiled here to help set the stage for the proposed ACCRUE index.

2.1 Sensitivity Theory

Sensitivity coefficients are the key ingredients for the GLLS methodology, as they are used to relate the response variations to the model parameter variations, with the latter assumed to represent the dominant sources of uncertainties. A sensitivity coefficient measures the first-order variation of a response that is caused by a change in one input model parameter. For the numerical experiments employed in this paper, we focus on the multiplication factor, k_eff, i.e., critical eigenvalue, as the response of interest and the reaction-wise cross-sections by isotope and energy group as the model parameters.

While sensitivity coefficients can be readily evaluated using finite differencing, the adjoint-based perturbation theory methodology (Usachev, 1964; Gandini, 1967; Stacey, 1974; Oblow, 1976; Cacuci and Ionescu-Bujor, 2010) has been adopted as the most efficient way to calculate derivatives. This follows because adjoint-based sensitivity theory requires one adjoint solution per response, implying that one can calculate the first-order derivatives of the given response with respect to all cross-sections using a single adjoint model evaluation, whereas finite differencing requires an additional forward model evaluation for each cross-section. The general idea behind adjoint-based sensitivity analysis is summarized below for the evaluation of the first order derivatives of the critical eigenvalue.

The Boltzmann transport equation for an assembly containing fissionable material, referred to as the forward model, can be symbolically expressed as (Williams, Broadhead and Parks, 2001):where

Multigroup form of the Boltzmann loss operator.

Multigroup form of the Boltzmann production operator.

Multigroup angular flux. where is a function of

position.

neutron moving direction.

energy group.

The two operators and are functions of the reference multi-group cross-section data which may be described by an n-dimensional vector, whose n components are the reaction-wise cross-sections by energy-group and isotope⁵. Thus, the solution of this equation yielding the eigenvalue may be compactly written as follows:where is the code-predicted eigenvalue⁶, i.e., k_eff, of jth critical experiment model and is a compact representation of the solution of Eq. 1 implying its dependence on . Analytically, the first-order derivatives of k_eff with respect to each cross-section can be expressed in a relative sense—referred to as the sensitivity coefficients—using a first-order Taylor series expansion:and aggregated in a vector (referred to as the sensitivity vector or profile) with the superscript “T” representing vector/matrix transpose:

Eq. 2 implies that a finite-difference-based sensitivity analysis would require forward model executions, one with the reference cross-section values, and one additional execution per cross-section. For most practical neutronic problems, this is computationally infeasible because cross-sections often number in the 10⁴ to 10⁵ range.

The adjoint formulation of sensitivity coefficients may be described by the following equation:

The brackets represent an inner product operation corresponding to an integration over entire phase-space (e.g., energy groups, direction of neutron travel, and space) using the forward solution obtained from Eq. 1 and a new quantity, called the adjoint solution, obtained from:where and are the adjoint operators of the forward operators and , evaluated at the reference cross-section values. The term differentiates the operators and with respect to the ith cross-section, with the derivative taken around the reference cross-section values. Since the mathematical expressions for and are known, these changes can be analytically calculated. More importantly, they do not require re-execution of the forward or the adjoint model. The implication is that one can evaluate the derivatives with respect to all cross-sections using a single forward and a single adjoint model evaluation.

Several computer codes have embodied the adjoint methodology to calculate sensitivity coefficients (Lucius et al., 1981; Becker et al., 1982; Gerald; Rimpault et al., 2002). Of interest to us is the SCALE TSUNAMI methodology (Rearden and Jessee, 2016) which is used as a basis for the evaluation of the sensitivity coefficients for the GLLS-based TSURFER code, discussed in the next section.

2.2 GLLS Adjustment Methodology

As discussed earlier, the main goal of GLLS is to consolidate knowledge from computations and experiments. This is illustrated in Figure 1 using two representative PDFs describing the best available knowledge about the k_eff from the experiments (shown in red) and model predictions (blue). The spread of each PDF is taken as a measure of confidence. The confidence in the model predictions is determined by the propagated prior uncertainties, and the experiment’s confidence is tied to its measurement uncertainties. The GLLS methodology represents a disciplined mathematical approach to consolidate these two PDFs into one (yellow) that provides higher confidence for the calculated response as compared to the prior confidence from model predictions.

FIGURE 1

To achieve that, GLLS assumes that the uncertainties originate from the cross-sections. Therefore, it attempts to identify the optimal cross-section adjustments which minimize the discrepancies between measured and predicted responses. Based on the optimal adjustments, one can calculate the corresponding change in the application’s response, with the application representing the conditions for which no experimental values exist. The change in the code-calculated application response, i.e., from its prior value, is denoted by the application bias.

Considering that the analyst is interested in calculating the bias for the k_eff value for a given application, and there exist available experiments, the corresponding prior values may be aggregated in a vector such that:where the last component is the prior value for the application k_eff. The corresponding measurements for the first values are designated by another vector . In this formulation, the last element of is set to the prior value of k_eff, assumed to have no corresponding experimental value.

The prior cross-section uncertainties are described by a multi-variable Gaussian PDF with a vector of means representing the reference multi-group cross-sections and a covariance matrix given by:

The adjusted cross-sections are calculated as the minimizer of the following minimization problem subject to the constraint :where is the covariance matrix for the measured k_eff. The constraint implies that the adjusted cross-sections will update the best-estimated values (the components of ) for all experiments as well as the application. The last element of the vector is taken to represent the best-estimate for the application k_eff value, and the last component of is referred to as the application bias. Note that represents the initial discrepancy between the prior code-calculated and measured k_eff values for the jth experiment, and represents the discrepancy after the cross-sections are adjusted.

The objective function in Eq. 4 may be re-written in terms of the calculated and adjusted k_eff values as:where is the -degrees of freedom chi-square value describing the discrepancies between the prior and adjusted k_eff values. The matrix denotes the prior covariance matrix for the calculated k_eff values given by:and matrix aggregates the sensitivity profiles for all experiments and the application:where (j, i) element represents the relative sensitivity coefficient of the jth experiment [or application, th row] with respect to the ith multi-group cross-section.

Assuming that the linearization of the constraint is sufficiently accurate within the limitations of first-order sensitivity theory, the minimizer of the objective function in Eq. 4 may be given by:where and is the discrepancy vector,

The posterior (i.e., post the consolidation of experimental and prior values) covariance matrix for the k_eff values is given by:

The diagonal elements of this matrix describe the confidence one has in the posterior k_eff values. The th diagonal element of the matrix () measures the prior confidence one has in the calculated application k_eff value. If the experiments are relevant to the application, the posterior confidence, () should provide higher confidence, i.e., .

2.3 Similarity Index, c_k

The definition of the similarity index c_k naturally appears in the GLLS formulation of the prior covariance matrix. Specifically, one can expand Eq. 5 as follows:

The diagonal entries of this matrix represent the uncertainty (in the units of variance) of the prior k_eff values and the off-diagonal entries are the correlations between these uncertainties. Ideally, one would want to maximize the correlations between the application and all experiments, described by the last row or last column of the matrix. A standardized form of this matrix may be obtained by multiplying it from both sides by the inverse of the square root of its diagonal elements to produce the matrix :

In this representation, the value of any off-diagonal terms is standardized between .0 and .0 which is equivalent to the definition of the standard correlation coefficient between two random variables. Appearing naturally in the GLLS formulation, these off-diagonal terms have been adopted as to measure experimental relevance. Specifically, the similarity index c_k between the application and the jth experiment is given by:

This equation may be used to pre-calculate the similarities of all available experiments with respect to the given application. To achieve that one needs to calculate the corresponding sensitivity profiles for the experiments and the application which are readily calculated using the adjoint sensitivity theory presented in Section 2.1. In our work, the SCALE TSUNAMI code is employed to calculate the sensitivity profiles as well as the similarity indices.

Next, it is instructive to give a geometric interpretation of the similarity index. To do that, rewrite the expressions in Eq. 8 using the Cholesky decomposition of as:where , where the inner-product of any of two columns of gives:

Performing this transformation for both the numerator and denominator in Eq. 8, the c_k index reduces to:

As illustrated in Figure 2, this expression is interpreted as cosine angle between two vectors, one related to the application and the other to an experiment.

FIGURE 2

Further, it has been shown in earlier research (Huang et al., 2020) that one can calculate c_k using randomized forward model evaluations taking advantage of the equivalence between the c_k definition and the standard correlation coefficient. This requires sampling of the cross-sections within their prior uncertainties a few hundred times to obtain sufficiently approximate estimates of the similarity index, as shown below. As demonstrated in earlier work using the Sampler code under the SCALE environment (Wieselquist, 2016), this forward-based approach provides two advantages; first, it allows one to calculate similarity indices when the adjoint solver is not available; and second, it provides a way to verify the results of adjoint-based calculations.

It has been shown in earlier work that these two vectors may be interpreted as the directional sensitivity profiles with respect to the dominant eigen directions of the prior covariance matrix. To illustrate the mechanics of the forward-based approach for calculating the similarity index, first consider re-writing the cross-section covariance matrix decomposed by Cholesky methodology as follows:where a unitary matrix; and is a diagonal matrix whose elements are square root of the singular values of .

If is a Gaussian variable, one can generate random samples for the cross-sections which respect their covariance structure, such that:

By the law of large numbers, one can show that as increases (Stark and Woods, 2012)

This limit is readily reached with a few hundred samples. Verification with numerical experiments is provided in the following section.

Then, Eq. 10 can be re-written by the cross-section samples, such as:With the linearity assumption valid, e.g., , samples for the code-calculated responses also can be calculated by the sandwich rule in Eq. 5, and thus the covariance matrix for calculated responses can be re-written as:where transforms the sensitivity matrix using the Chelosky factor of .

Thus, the construction of the matrix effectively reduces to the following three steps; first, it transforms the original variables into a set that are uncorrelated; second, it calculates the sensitivities along the directions of the transformed variables (referred to as directional sensitivity in the calculus literature); and third, it weighs each directional sensitivity by its corresponding prior uncertainty.

The deviation vector of jth experiment (or application denoted by subscript ) code-calculated samples from its reference value can be written as:

Each term in may be considered as a sample resulting from executing the forward model with a random cross-section perturbation. Thus, is a vector of sampled perturbations of the jth response. Graphically, these perturbation vectors can be displayed via a scatter plot. For example, Figure 3 shows a representative scatter plot of the perturbation vectors for the jth experiment and the application.

FIGURE 3

Their similarity index thus reduces to the standard correlation coefficient between the two vectors and similarly to Eq. 9 such that:With a large number of samples, the inner-products of any two sample vectors reduce to the elements of the matrix, such that:

Thus,

In this manuscript, the ACCRUE index will be calculated using both the analytical definition, presented in Section 3, as well as the noted forward-based approach for verification.

3 ACCRUE Index and Verification Algorithm

This section details the theoretical derivation of the ACCRUE index j_k, discusses its relationship to the c_k index, and shows how it can be calculated both analytically using the adjoint approach and statistically using the forward approach. The j_k index is designed to address two limitations of the c_k index, first the impact of measurement uncertainty on the relevance of a given experiment, and second, the diminished value of an experiment resulting from its similarity with previously consolidated experiments. With regard to the first limitation, the c_k index bases the similarity on the code-calculated values only. In practice however, an experiment with a high c_k index could prove less valuable to estimating the application bias if its measurements have high uncertainties. The second limitation calls for an approach to identify experimental redundancy. The high level premise of any inference procedure is that additional measurements will result in more confidence in the calculated application bias. In practice however, the confidence gleaned from multiple highly relevant experiments could be equally obtained from a smaller number of experiments if high level of redundancy exists between the experiments, a common phenomenon observed in many fields, often referred to as the law of diminished return. The c_k index does not capture this effect because it is based on a single experiment.

3.1 Impact of Measurement Uncertainty

Different from the c_k index which relies on the matrix, the j_k index leverages the matrix which appears in the GLLS procedure to weigh the prior values and the experimental measurements. This matrix can be shown to be the covariance matrix for the discrepancies between the calculated and measured values, whereas the matrix is the covariance matrix for the calculated values only.

The discrepancies can be aggregated in a vector such that:where the last element, , of the discrepancy vector remains the same as , since measurement uncertainty for the application is not applicable.

Then the covariance matrix for the discrepancy vector, , is:where and describes the possible correlations between the experimental measurements. The elements of the last column or the last row of the are the same as those of the matrix since no measurement is applicable for the application. If the experiments are independent, then the matrix reduces to a diagonal matrix.

Similarly to before, consider the expressions in Eq. 11 using the Cholesky decomposition of as:where , where the inner-product of any of two columns of gives:

Geometrically, the ACCRUE index j_k with a single experiment thus is defined as the cosine angle defined by two vectors of , specifically one related to the application and the other to an experiment as:

Figure 4 graphically shows how the measurement uncertainty impacts on the j_k value with a single experiment (the jth experiment denoted by subscript “j”) as compared to the c_k value. Due to the measurement uncertainty, the cosine angles estimated by c_k and j_k respectively change from to . If the associate measurement uncertainty of the given experiment is zero, the j_k value reduces to the c_k value. In any other cases where the measurement uncertainty of the given experiment is not zero, the cosine angle given by j_k is always smaller than that by c_k as illustrated, e.g., or . This can be readily proved by comparing the analytical expressions for each c_k and j_k value shown in Eqs. 9, 12, respectively.

FIGURE 4

With regard to the matrix notations of both equations, and , and are the same respectively since measurement is not applicable for the application, while is always greater than since . As a result, the cosine angle is reduced proportional to with experimental measurement uncertainty present.

As discussed in Section 2.3, the covariance matrix, , can be written as:where a low triangular matrix.

Then the measurement sample vector for jth experiment, , is defined as:

Thus, the discrepancy sample vector for jth experiment, , is:where is a scalar quantity representing a ratio of experimental to calculated response values of jth experimentThus,

3.2 Impact of Multiple Experiments

The ACCRUE index may be viewed as the similarity between a group of experiments and the application of interest when the experimental uncertainties are excluded from the analysis. Its true value however is in assessing the relevance of a new experiment by taking into account both the experiment’s measurement uncertainty and the value gleaned from past experiments. In this section, the detailed analytical derivation for j_k value with multiple experiments is provided with the vectors. However, if analysts are not interested in including the impact of measurement uncertainties, they can work directly with the vectors instead of vectors.

Analytically, the j_k index for the first experiments is given using the elements of the matrix in Eq. 11 by:where the terms are defined by the following recurring relation:

These terms can be evaluated analytically given access to the sensitivity coefficients, the prior cross-section covariance matrix, and the measurement uncertainties. By way of an example, consider the simple case with , i.e., a single experiment, where the j_k reduces to:

This expression equivalent to the c_k index assuming zero measurement uncertainty, i.e., . The implication is the j_k will always have a lower value than the corresponding c_k value for any realistic non-zero experimental uncertainties. With , and assuming the measurement uncertainties are uncorrelated, i.e., , the j_k value becomes:

Eq. 15 shows that the relevance of two experiments may be expressed as the sum of two terms, one very similar to the c_k index representing the relevance of the first experiment, and the other subtracting away the impact of the first experiment. To help interpret the j_k index for experiments, we resort to the geometrical and statistical interpretations provided earlier in Section 3.1.

Consider the case with two experiments (denoted by the subscripts “1” and “2”, respectively) as illustrated in Figure 5, where j_k calculates the angle between the vector (representing the application) and the subspace formed by and (representing the two experiments). As long as the second experiment provides additional information which is not duplicated by the first experiment, the j_k value is expected to increase as compared to the value obtained with a single experiment, i.e., or as shown in Figure 5. In the case that the second experiment carries no additional information, e.g., the j_k value remains the same. The implication is that the second experiments provides no value to the GLLS inference procedure, and hence can be excluded.

FIGURE 5

The cosine angle between any two subspaces can be calculated by orthogonal projection techniques, e.g., QR decomposition or Singular Value Decomposition (SVD). For example, consider the j_k value of the first experiments and the application, the corresponding vectors can be aggregated as:

Then, the associated j_k expression can be written as:where is jth vector of the matrix from QR decomposition and is a normalized vector for the application such as .

For example, consider the case where contains only vector corresponding to the first experiment, i.e., , then the j_k value is calculated as:where is a normalized directional vector of :

Thus, the j_k value iswhich is equivalent to Eq. 14

If contains two vectors ( and ) corresponding to the first two experiments, then the j_k value is calculated as:where is defined by Gram-Schmidt process such as:

Thus, the j_k value is:which is equivalent to Eq. 15

Consequently, the general expression for j_k value can be written as:wherewhich evaluates the same scalar quantity as that calculated by the matrix element notations in earlier this section.

3.3 Overall Process

Figure 6 shows the overall process for evaluating the similarity or relevance of an experiment to a given application. The similarity accounts for the correlation between two responses, e.g., one from the application and the other from the experiment, as calculated by a computer code. The ACCRUE index extends the concept of similarity to quantify the relevance, as measured by the added value of the experiment, taking into account both the experiment’s measurement uncertainties as well as past experiments. As shown in this figure, the first step is to check if an adjoint solver is available which allows one to employ sensitivity coefficients. If no experimental measurement uncertainty is available and only a single experiment is available, the ACCRUE index j_k reduces to the similarity index c_k. If the experimental uncertainty is available, Eq. 12 is employed to evaluate the j_k index. If additional experiments are available, then the most general expression for j_k is employed per Eq. 13.

FIGURE 6

4 Numerical Experiments

Numerical experiments have been conducted with two different case studies. The first case study assumes that the applications have low c_k values for all available experiments which are in the order of 0.7, referred to as the low relevance applications, and the second case study considers applications with high c_k values that are greater than 0.9. An application with low relevant experiments represents a realistic scenario expected with first-of-a-kind designs, i.e., advanced reactor designs and new fuel concepts, with no prior or rare strongly relevant experiments. It also highlights the expected high cost of new experiments, and the need to employ modeling and simulation to design a minimal number of targeted validation experiments. When the c_k values are low for the given application, it is important to figure out a way to prioritize the selection of past available experiments, as well as judge the value of new/planned experiments. We compare the performance of the j_k and c_k indices for both the high relevance and low relevance applications for a range of assumed values for the experimental uncertainties. This is also important when designing new experiments, as it provides quantitative value for different types of instrumentations with varying levels of measurement uncertainties.

The low relevance case study employs 17 critical benchmark experiments from the low enriched uranium thermal compound systems (LCT-008-001—LCT-008-017) in the ICSBEP handbook (NEA, 2011) as experiments and the accident-tolerant fuel (ATF) concept BWR assembly as an application. The selected LCT benchmark experiment set (LCT-008) cases are critical lattices with UO₂ fuel rods of 2.9% enrichment and perturbing rods in borated water in common, but have different boron concentrations and various rods arrangements. Their similarity indices, c_k, to the BWR ATF model are all around 0.7 as reported by TSUNAMI-IP. The BWR ATF model is a GE14 dominant lattice with UO₂ fuel and FeCrAl cladding. This model comprises 92 UO₂ fuel pins, 78 out of which are fuel with various enrichments from 2 to 4% and the remaining are 14 rods with gadolinium, surrounded by water coolant in a channel box structure (Jessee, 2020). The layouts of a representative LCT benchmark experiment (LCT-008-001) and the application model are shown in Figure 7. The high relevance case study employs 42 critical experiments from highly enriched uranium fast metal systems (HMF set, short for HEU-MET-FAST) in ICSBEP handbook. In this study, application models are selected among these benchmark experiments and similarity indices are in the 0.9 range (NEA, 2011). All the critical benchmarks employ highly enriched uranium fuel with a variety of design configurations and different enrichments. Figure 8 shows two representative benchmark experiments one of which is used as an experiment (HMF-020-001, Polyethylene-reflected spherical assembly of ²³⁵U) and the other as an application (HMF-019-001, Graphite-reflected spherical assembly of ²³⁵U).

FIGURE 7

FIGURE 8

To identify the possible impact of initial discrepancies, i.e., differences between calculated and measured responses, on the GLLS-estimated biases, three critical experiments having different biases (high, intermediate, and low) are selected as applications and the remaining experiments are used as validation experiments. The high bias application is selected to have an initial k_eff discrepancy in the order of 1,000 pcm, while the intermediate in the order of 300–500 pcm, and the low 200 pcm.

4.1 Comparison of j_kvs. c_k-Sorting

Figure 9 shows representative results for the high relevance case study for three different applications with high, intermediate, and low biases. The x-axis explores the change in the GLLS-estimated biases when adding one experiment at a time, wherein the experiments are ordered according to their c_k values (top graphs), and j_k values (bottom graphs). The c_k-sorting is straightforward as each experiment is employed once in conjunction with the given application. Markedly different, the j_k value depends on the number and order of experiments included, hence each sorting is expected to give rise to different profiles for j_k and the associated bias and bias uncertainty. The goal is thus to identify the order that allows the analyst to reach a certain level of confidence in the calculated bias with minimal number of experiments. In the current study, the search for this optimal order is initiated with the experiment having the highest c_k value, with the second experiment selected to maximize the j_k value among all remaining experiments, and so on. In practice, one may start with any experiment, and adds experiments as they become available, or may employ the j_k value to quantify the value of new/planned experiments. For each added experiment, the GLLS bias and bias uncertainty are calculated to help compare the c_k and j_k-sorting. The goal is to achieve a stable prediction of the bias with minimal uncertainty.

FIGURE 9

These results show that the bias stabilizes quicker with the j_k sorting. More important, the c_k-sorting could show sudden changes after a period of stable behavior. For all considered applications using the c_k-sorting, the biases continue to experience sudden or gradual changes following a period of stable behavior. For example, for the low bias application, the estimated bias exhibits an upward trend after the 20th experiment. The implication is that the additional experiments continue to provide value to the GLLS procedure despite their lower relevance. With the j_k-sorting, a more explainable trend is achieved whereby the bias trend shows stable behavior after the 20th experiment.

Regarding the bias uncertainties, shown in Figure 9 as error bars, they are plotted in Figures 10, 11 as a function of the number of experiments using both the c_k and j_k-sorting for, respectively, one application with high relevance experiments and one application with low relevance experiments. The results highlight a key limitation of the c_k-sorting, that is the addition of low relevance experiments could change the trend of both the bias and the bias uncertainty. The j_k-sorting however does not suffer from this limitation, implying that one can select the minimal number of experiments required to reach a certain pre-determined level of confidence for the calculated bias. Comparison of the uncertainty values in both figures indicate that the impact is much less pronounced when highly relevant experiments are available. This highlights the value of the j_k-sorting when limited number of experiments are available, as is the case with first-of-a-kind designs.

FIGURE 10

FIGURE 11

These results are compared in Figure 12 with randomized orders (shown as multi-color solid lines) for one of the low relevance applications at different levels of measurement uncertainty, specifically 10 pcm (representing a highly accurate measurement), 100 pcm (a realistic measurement), and 500 pcm (a low confidence measurement) for the measured k_eff values. The results show that pure random sampling could be superior to c_k-sorting, with the j_k sorting still exhibiting the best behavior in terms of reducing the bias uncertainty with minimal number of experiments. When the measurement uncertainty is too high, the ordering of the experiments is no longer important. This is a key observation highlighting the value of ordering experiments as experimentalists continue to improve their measurements in support of model validation.

FIGURE 12

To help understand the changes in the bias and bias uncertainty associated with the c_k-sorting, the left plot in Figure 13 orders the experiments according to their c_k values, and the middle plot calculates the corresponding j_k values using the c_k-sorting. Notice that although the first few experiments, i.e., #15, #13, #12, …, to #4 have higher relevance than later experiments, they do not change the j_k value, and hence the bias and bias uncertainty as shown in the earlier figures. The j_k values start to show larger increase when additional lower relevance experiments are added, explaining the sudden or gradual change in the bias and its uncertainty. However, when the experiments are ordered according to their j_k values, as shown in the right plot, a smoother j_k profile is obtained which is consistent with the bias and bias uncertainty profiles obtained using the GLLS procedure. The implication is that one can employ the j_k profile to develop insight into the amount of experimental effort necessary to reach a target confidence in the calculated bias, even before the actual measurements are recorded. This follows because j_k only requires access to the prior uncertainties and the expected measurement uncertainty, not the actual measured bias.

FIGURE 13

4.2 Stochastic Non-Intrusive Verification

As mentioned earlier, the analytical expressions for the similarity or relevance metrics such as c_k and j_k require access to derivatives which may not be readily available. To address this challenge, earlier work has developed an alternative to the estimation of c_k value using a non-intrusive forward-based stochastic method (Huang et al., 2020). In this section, we present numerical results comparing the results of the analytically-calculated j_k value to that estimated by the noted stochastic method. This will serve two purposes: the first is to provide a simple approach for the calculation of the j_k value when derivatives are unavailable, and the second to help verify the calculated j_k value using the stochastic method by comparing it to the analytically-calculated value. To achieve that, two benchmark experiments (HMF-005-001 and 005-002) are selected to calculate the j_k value. Their calculated response uncertainties are 1,492 pcm and 1721 pcm respectively, and measurement uncertainties are 360 pcm. A total of 1,000 samples are generated to examine the convergence of forward-based j_k value, whose corresponding analytical value is 0.9445 as given by Eq. 15, shown in Figure 14 as a horizontal line. The results indicate that the forward-based approach produces acceptable approximation of the analytical value using few hundred simulations, which is consistent with the results reported previously for the c_k value (Huang et al., 2020).

FIGURE 14

5 Conclusion and Further Research

This manuscript has introduced an extension of the basic similarity metric, denoted by the ACCRUE metric and mathematically symbolized by the j_k index to distinguish it from the c_k similarity metric. The ACCRUE metric takes into account the impact of multiple experiments and the associated experimental uncertainties, both currently missing from the extant similarity definition. The results show that the ACCRUE metric is capable of finding the optimal sorting of the experiments, the one that leads to the most stable variation in the GLLS calculated bias and bias uncertainty. When the experiments available are highly relevant and numerous, the performance of the c_k metric approaches that of the j_k metric. However, when highly-relevant experiments are scarce and when experimental uncertainties are low, the j_k metric is capable of identifying the minimal number of experiments required to reach a certain confidence for the calculated bias, whereas the c_k metric may be outperformed by random sorting of the experiments. The results of this work are expected to be valuable to the validation of computer models for first-of-a-kind designs where little or rare experimental data exist. Another important value for the j_k metric is that it can be calculated using forward samples of the model responses, thereby precluding the need for derivatives, which allows one to extend the concept to non-k_eff responses. This will allow one to extend its definition to models exhibiting nonlinear behavior often resulting from multi-physics coupling, e.g., different geometry, compositions, and types of reactor. This will pave the way to the development of relevance metrics that goes beyond the first-order variations currently captured by the extant similarity analysis.

References

• 1

  Abdel-KhalikH. S.BangY.WangC. (2013). Overview of Hybrid Subspace Methods for Uncertainty Quantification, Sensitivity Analysis. Ann. Nucl. Energ.52, 28–46. 10.1016/j.anucene.2012.07.020

  • CrossRef
  • Google Scholar

• 2

  AvramovaM. N.IvanovK. N. (2010). Verification, Validation and Uncertainty Quantification in Multi-Physics Modeling for Nuclear Reactor Design and Safety Analysis. Prog. Nucl. Energ.52, 601–614. 10.1016/j.pnucene.2010.03.009

  • CrossRef
  • Google Scholar

• 3

  BeckerM.HenleyE. J.OshimaK. (1982). Advances in Nuclear Science and Technology.

  • Google Scholar

• 4

  BroadheadB. L.ReardenB. T.HopperC. M.WagschalJ. J.ParksC. V. (2004). Sensitivity- and Uncertainty-Based Criticality Safety Validation Techniques. Nucl. Sci. Eng.146, 340–366. 10.13182/NSE03-2

  • CrossRef
  • Google Scholar

• 5

  CacuciD. G.Ionescu-BujorM. (2010). Best-Estimate Model Calibration and Prediction through Experimental Data Assimilation-I: Mathematical Framework. Nucl. Sci. Eng.165, 18–44. 10.13182/NSE09-37B

  • CrossRef
  • Google Scholar

• 6

  GandiniA. (1967). A Generalized Perturbation Method for Bi-linear Functionals of the Real and Adjoint Neutron Fluxes. J. Nucl. Energ.21, 755–765. 10.1016/0022-3107(67)90086-X

  • CrossRef
  • Google Scholar

• 7

  GlaeserH. (20082008). GRS Method for Uncertainty and Sensitivity Evaluation of Code Results and Applications. Sci. Tech. Nucl. Installations2008, 1–7. 10.1155/2008/798901

  • CrossRef
  • Google Scholar

• 8

  HuangD.MertyurekU.Abdel-KhalikH. (2020). Verification of the Sensitivity and Uncertainty-Based Criticality Safety Validation Techniques: ORNL's SCALE Case Study. Nucl. Eng. Des.361, 110571. 10.1016/j.nucengdes.2020.110571

  • CrossRef
  • Google Scholar

• 9

  JesseeM. (2020). SCALE Lattice Physics Code Assessments of Accident-Tolerant Fuel.

  • Google Scholar

• 10

  LuciusJ. L.WeisbinC. R.MarableJ. H.DrischlerJ. D.WrightR. Q.WhiteJ. E. (1981). Users Manual for the FORSS Sensitivity and Uncertainty Analysis Code System. Oak Ridge, TN. 10.2172/6735933

  • CrossRef
  • Google Scholar

• 11

  NEA (2011). International Handbook of Evaluated Criticality Safety Benchmark Experiments 2015.

  • Google Scholar

• 12

  OblowE. M. (1976). Sensitivity Theory from a Differential Viewpoint. Nucl. Sci. Eng.59, 187–189. 10.13182/nse76-a15688

  • CrossRef
  • Google Scholar

• 13

  PalmiottiG.SalvatoresM. (20122012). Developments in Sensitivity Methodologies and the Validation of Reactor Physics Calculations. Sci. Tech. Nucl. Installations2012, 1–14. 10.1155/2012/529623

  • CrossRef
  • Google Scholar

• 14

  ReardenB. T.JesseeM. A. (2016). TSUNAMI Utility Modules. Ornl/Tm-2005/39.

  • Google Scholar

• 15

  RimpaultG.PlissonD.TommasiJ.JacqminR.RieunierJ-M.VerrierD.et al (2002). The ERANOS Code and Data System for Fast Reactor Neutronic Analysis.

  • Google Scholar

• 16

  SalvatoresM. (1973). Adjustment of Multigroup Neutron Cross Sections by a Correlation Method. Nucl. Sci. Eng.50, 345–353. 10.13182/nse73-a26569

  • CrossRef
  • Google Scholar

• 17

  StaceyW. B. (1974). Variational Methods in Nuclear Reactor Physics. Academic Press. 10.1016/b978-0-12-662060-3.x5001-2

  • CrossRef
  • Google Scholar

• 18

  StarkH.WoodsJ. W. (2012). Probability, Statistics, and Random Processes for Engineers. Fourth Edition4th editio. Upper Saddle River: Pearson.

  • Google Scholar

• 19

  UsachevL. N. (1964). Perturbation Theory for the Breeding Ratio and for Other Number Ratios Pertaining to Various Reactor Processes. J. Nucl. Energ. Parts A/B. Reactor Sci. Tech.18, 571–583. 10.1016/0368-3230(64)90142-9

  • CrossRef
  • Google Scholar

• 20

  WieselquistW. A., 2016. SAMPLER: A Module for Statistical Uncertainty Analysis with SCALE Sequences, 6.2.3. ed. Ornl/Tm-2005/39.

  • Google Scholar

• 21

  WilliamsM. L.BroadheadB. L.JesseeM. A.WagschalJ. J.LefebvreR. A., 2016. TSURFER: An Adjustment Code to Determine Biases and Uncertainties in Nuclear System Responses by Consolidating Differential Data and Benchmark Integral Experiments, 6.2.3. ed.

  • Google Scholar