Predictive state representation (PSR) is a model to predict observations in stochastic environments, including POMDPs. A POMDP is defined as a hextuple {S,A,T,O,Ω,R}, where S is the set of underlying MDP states; A is the set of actions; T is the transition function, T:A×S×S→[0,1], which gives the probability of transitioning from one state to another given the action taken; R is the reward function; O is the set of observations; and Ω is the set of conditional observation probabilities. POMDPs differ from MDPs in that the current observation is not sufficient for an agent to be able to determine its underlying state.

Let t be a finite stream of action-observation pairs. Then t is a test, and we let T be the set of all tests. Let the history h_i ∈ H of the agent at time i be the stream of action-observation pairs aj∈A,oj∈O,∀j∈[0,i)∩ℕ observed up to time i.

A PSR (Littman et al., 2002) of an environment consists of a set of core tests, Q; a set of |Q|-dimensional m_a,o,t vectors for all a∈A,o∈O,t∈Q; and an initial state.

The set of core tests, Q, is a finite subset of T, with the property that P(t ∣ h) for all t∈T,h∈H can be found as some linear combination of the probabilities P(q ∣ h) for all q ∈ Q. The empty test, ϵ = {}, is always included in Q, such that P(ϵ ∣ h) = 1 for all h which are possible under the PSR model. The PSR state vector after observing history h, y(h), is a (1 × |Q|) vector which holds P(q ∣ h) for each q ∈ Q. By stacking the rows of m_a,o,t for all t ∈ Q, we obtain a (|Q| × |Q|) projection vector M_a,o for every length 1 test (a,o). For all h ∈ H we have that P(o∣h,a)=y(h)×Ma,oT. Projection vectors for longer tests can be created by multiplying projection vectors for shorter tests. For example, M_a1,o1,a2,o2 = M_a2,o2 × M_a1,o1, where × is matrix multiplication.

To maintain an accurate state vector, m_a,o,t must be available for all a∈A,o∈O,t∈Q. Let Q = {t₁, t₂, …, t_n}. This can be used to obtain the state vector at time i, y(h_i), given the a_i, o_i action observation pair observed at timestep i, and y(h_i−1). Recall that the PSR state vector contains the probabilities of each core test, and let y_j(h) denote the element in y(h) corresponding to core test t_j. Then, the probability of each core test can be found as follows. For all t_j ∈ Q: (1)yj(hi)=P(tj∣hi-1,ai,oi)=y(hi-1)×mai,oi,tjTy(hi-1)×mai,oiT . Equivalently, we can use the previously defined projection vectors: (2)y(hi)=y(hi-1)×Mai,oiTy(hi-1)×mai,oiT . From the definition above, it follows that PSRs can give an indication of the probability of certain transitions to occur. In particular, the following theorem specifies the probability of observing particular action observation pairs:

THEOREM 1. Given that the state vector is y(h_i−1) at time i − 1, the probability of seeing observation o_i after taking action a_i at time step i is given by the following equation (3)P(oi|hi-1,ai)=y(hi-1)×mai,oiT . Proof: By construction. Note that the state vector contains enough information to accurately predict future observations even in partially observable environments. The state vector acts not only as a prediction, but also as an internal state for the PSR model.

A natural question to ask is, ‘what is the predictive state of an empty history y(ϵ)?’. If the environment is known to always start in a given underlying state, the corresponding predictive state may be used. However, this is a strong assumption in general. If we assume the agent is likely to start in each underlying state proportionally to the amount of time previously spent in that state, y(ϵ) can be set to the stationary distribution. The stationary distribution, which is the weighted expected value of y over all time steps, is given by (4)y(ϵ)=∑h∈Hy(h)|H| where H is the set of all histories. As H is an infinite set, this is calculated instead from the histories that the agent has seen. When calculated this way, the stationary distribution may change depending on the policy followed by the agent. Additionally, as the stationary distribution represents a distribution over all states, it may not be a state that the agent can reach through normal exploration; however, it represents a positive probability for all states previously visited.

The state space of the PSR is the set of states that can be generated recursively by y=y′×Ma,oTy′×ma,oT, for all a∈A,o∈O, where y′ is known to be in the state space of the PSR, and y×ma,oT is non-zero. In order to generate the set of states, an initial state y′ must be assumed to be in the state space. Sometimes an initial state is provided when the starting state of the environment is known. However, when no initial state is provided, the stationary distribution may be used.

There are several algorithms for learning PSRs offline, but relatively few for learning and discovering tests online. One such algorithm is the constrained gradient algorithm (McCracken and Bowling, 2005), which we use in our experiments for learning PSRs.

A simple neural network (Bishop, 1995) can be trained to predict observations in a given environment. Given a time window of duration i, a neural network can take the observations o∈O and the actions a∈A and predict the new observation o at time i + 1. If the softmax transfer function is used to produce output probabilities, the network can also be trained to predict the probability of each observation at i + 1 in stochastic environments. The difference between the prediction and the observation can be used to train such a system with gradient descent. As for the PSR model explained in the previous section, a neural network predictor can be trained effectively only if stationary conditions are assumed during the training phase. Thus, changes in the environment such as those occurring in dynamic POMDPs (see next section) are challenging conditions that this study addresses.

Neural networks, PSRs, and other predictive models have one thing in common: they give as output a prediction of the probability of seeing each possible observation next. The sum of probabilities of seeing each observation is 1. Let K be a predictive model, O be the set of possible observations, and A be the set of possible actions. Then, we define P(o ∣ K_i) to be the predicted probability of observing o at time i given by K. Additionally, we define P(O∣Ki) as the probability distribution over all observations at time i given by K.

Assume that an environment remains stationary for a certain amount of time. Under this assumption, it is possible to learn a model of the environment (e.g., a PSR, as detailed in McCracken and Bowling, 2005). If, after a certain amount of time, the environment changes, the continual training of the same model will lead to inaccurate predictions at first, and catastrophic forgetting of the first environment in the long term. This is because the assumption of stationary conditions is not valid, and one model tries to learn an average distribution of two or more distributions that occur in different environments. These changing environments are similar to switching hidden Markov models (SHMMs) investigated in Chuk et al. (2020) and Höffken et al. (2009).

Let 𝕍 = {V₁, V₂, …, V_m} be a set of distinct POMDP environments. A dynamic POMDP environment, D, behaves as a single environment V_i ∈ 𝕍 for a number of time steps, n₀, before changing its behavior to another environment, V_j ∈ 𝕍. The dynamic environment D may switch to any environment in 𝕍 every n_k time steps, with k ∈ ℕ and n_k ≫ 1. Effectively, these dynamics can be seen as hierarchical large stationary models where a state variable z ∈ ℕ determines the specific environment V_z at a given time. However, z is not observable and can only be derived inferring which specific environment from the set 𝕍 matches the current stream of data. We assume that transitions between environments in 𝕍 occur with considerably lower frequency than state transitions within the environments V_z. This assumption reflects two points: (1) in real world scenarios, generally, two environments can be thought of as being distinct when transitions from one to another occur rarely, otherwise it makes more sense to consider them as one environment; (2) for a model to learn and predict one environment, it is necessary to experience the environment for a minimum number of steps that capture transitions within it.

Such dynamic conditions occur typically when an agent learns to predict an environment, e.g., to navigate in a house, and is then required to learn a new somewhat different environment, e.g., to navigate in an office. The new environment might bear a similarity with the previous one, but also significant differences. Desirable skills of an agent include the ability to detect that there is a new environment, the ability to learn the new environment quickly, possibly exploiting previous knowledge, and also retain the knowledge of the previous environment (avoiding catastrophic forgetting). The aim of this study is precisely to achieve such capabilities as explained in the next section.

Given a set of statistical models that each predict an environment, a question that can be asked is: what model is best at predicting a given stream of data? Several approaches have been proposed in the literature to perform model selection (Cox, 2006). Approaches for model selections are based on the information theory such as the Akaike information criterion (Akaike, 1974) and the Bayesian (or Schwarz) information criterion (Schwarz, 1978). The idea is to measure the information that is lost due to the difference in statistical distributions between the model and the data using the Kullback-Leibler divergence (Kullback, 1997). By doing so, it is possible to select a best fitting model that minimizes this difference. Different statistical methods, however, may be more or less appropriate or accurate according to a number of factors including the number of data points and the assumptions on the statistical distributions that are being observed.

Given a recent window of data from one environment, hypothesis testing can be used to accept or reject the null hypothesis that the distribution of the environment data matches the distribution of the model-generated data (Lehmann and Romano, 2006). The ability to accept or reject such a null hypothesis is a valuable tool, particularly in the context of explainable AI applications in which a model is used to predict a data stream. If the current data stream has a very low p-value, it is reasonable to reject the null hypothesis that the model is correct. On the contrary, if the null hypothesis cannot be rejected, the model offers a good approximation and it is therefore reasonable to (1) consider it as a good-enough predictor and (2) use new data to further improve it via a training process.

We assume that (1) a data stream is generated by one environment V_i from the set 𝕍; (2) that we want to learn and identify which model K_i ∈ 𝕂 describes the current data stream, including if none of the current models describe the data; (3) the data stream produces a set of finite discrete or categorical outcomes. Therefore, we use hypothesis testing instead of model selection, so that we can determine when none of the existing models fit the data, allowing us to create a new model. To identify which specific V_i is generating the data, the problem can be formulated as selecting the corresponding model K_i that maximizes the likelihood (5)argmaxiL^=P(h|Ki) where h is a limited time window of recent input data.

It is important to note that there are no guarantees that a current set of models, 𝕂, can accurately predict a corresponding set of environments 𝕍. However, the key idea tested is: assuming we can select the most fitting model given by Equation (5), then we can associate a particular time window of the data stream and use it to improve the model K_i to maximize L^. Therefore, such an approach can be used both to learn and to find the best set of models that fit a set of environments.

The data generated by the set of actions and observations in a POMDP as described in the previous section form a multinomial distribution. Thus, the χ² test can be used to accept or reject the null hypothesis that a recent time window of data is generated by a given model. In the next sections, the procedure to compute the degrees of freedom and the χ² p-values is presented.

The degrees of freedom (DF) of a statistical model K corresponds to the number of independent parameters in the model, and is required to perform statistical tests such as χ². Predictive models have in common the ability to predict the probability distribution of the next observation given a history or internal state. Note that we are not trying to find the number of independent parameters in the predictive agent, but in the underlying model the agent predicts. Let o′∈O. If, for all o∈{O\o′} we know the value of P(o ∣ K_i), then (6)P(o′∣Ki)=1-∑o∈{O\o′}P(o∣Ki) . Therefore, the number of independent parameters for each prediction is |O|-1 as the final observation's probability can be inferred from the others. Each prediction may be independent in the predictive model, thus, we assume that the predictive model is an approximation to a statistical model (the underlying POMDP) with a number of independent states that is much smaller than the length of a history of data points. To estimate the number of independent states in the underlying model, we cluster together similar predictions in the predictive model. These can be clustered according to the predictive model's hidden states, the prediction of the next observation, or the prediction of several next observations.

The adaptive model detection algorithm (AMD) keeps a history window W of up to L recent prediction observation pairs, where L is a parameter of the algorithm. The choice of L affects the behavior of the algorithm as shown in the results section with an analysis of different values for L.

Knowing how many times each prediction has been made is necessary to determine whether the environment behaves as the predictive model expects. To count the number of times different predictions are given by a learning model, it is necessary to cluster such predictions that are expressed as vectors with possibly slightly different probabilities values. Let CK be the set of all clusters made by AMD for the predictive model. For a given cluster c∈CK, let c̄ be the mean of the distributions P(O∣Ki)∈c in the cluster, and c̄(o) therefore be the mean probability of observing observation o.

AMD uses the DBSCAN clustering algorithm (Ester et al., 1996) to form clusters of predictions given by a model. The scikit-learn (Pedregosa et al., 2011) vanilla implementation of the algorithm is used. As the data changes over time, the DBSCAN algorithm may form clusters differently on each timestep. This does not pose an issue, as c̄ can be recalculated at each timestep, meaning that even when the clusters change, the expected and observed frequencies will be similar in an accurate agent. DBSCAN does not include into clusters those outlier points which seem to not fit into larger clusters. Such a property is advantageous in our context because: (1) the mean distribution of a cluster c̄ is therefore not affected by an outlier that was forced into it, and (2) clusters must be formed of a certain minimum size, the advantage of which is explained in section 3.5.

To compute the χ², AMD counts as X_c,o the number of times in the history that each observation o follows predictions in each cluster c for all c ∈ C and o∈O. The number of times each observation o is expected to follow predictions in a given cluster is given by Ec,o=|c|×c̄(o).

Thus, from Pearson (1900), (7)χ2=∑c∈C∑o∈OXc,o-Ec,oEc,o . Effectively, χ² measures how well the actual data matches the expected data, with higher values meaning that the observed data does not match the expected data well. For a general model, some values of X_c,o and E_c,o may be equal or close to 0, corresponding to impossible (or never observed) transitions. For these cases when X_c,o = E_c,o = 0, the value Xc,o-Ec,oEc,o is set to 0.

Knowing the χ² value and DF allows us to find the p-value, which represents the probability of the observed data coming from the expected data distribution. This function is available in most statistical programs, code libraries, and toolkits as part of the χ² implementation¹.

Following the guidelines in (Yates et al., 1999), a χ² test is considered to be sufficiently accurate when “no more than 20% of the expected counts are less than five, and all individual expected counts are one or greater.” Accordingly, a minimum history length is necessary to be able to perform the test, and the longer the history, the more accurate the test is expected to be. Unfortunately, with an arbitrary large POMDP, even a long history does not guarantee that all possible prediction observation pairs have been seen. Additionally, while a long history length makes p-values more accurate, a long history means that the assessment of the probability for a model can be done only over a long period of time, potentially losing granularity on the precise point of the transition.

Even with a long history length, if the agent encounters a sequence of observations it does not expect, it may produce an internal state or prediction unlike any it has generated before. Due to this, the cluster containing the unusual state or prediction could be small enough that the expected number of times a given observation follows states in the cluster is lower than 1. Conveniently, DBSCAN has a minimum cluster size, so such small clusters can be avoided entirely, unless of course the probability of observing a particular observation from states in a cluster are very low.

An AMD that tests data against multiple predictive models K₁, K₂, …, K_n keeps track of the p-value of each one, and uses this tracked p-value to determine which environment it is most likely to be in. This allows AMD to select which model is trained at any point in time in a statistically motivated way, contributing to explainable decisions in AI. Additionally, detecting which environments are likely at given points in time opens up the possibility of applying a different policy based on the current environment.

Note that as long as the p-value is above the threshold at which the null hypothesis is rejected, it does not matter whether the value is low, high, or fluctuating. The AMD algorithm is summarized with pseudo code in Algorithm 1.

The proposed algorithm was tested on a set of POMDPs of various size and complexity. The first set of problems (Figures 1A–D) is a series of uncontrolled POMDP environments, i.e., Markov chain environments, with only 2 observations (as there are more states than observations, the observation is given by the color of the state in Figure 1). These environments appear simple at first, however, they have different states and transition probabilities, and, due to their stochastic nature, some environments could be mistaken for others based on the data generated by exploration. For example, environment C creates a data stream that can be produced by environment D. However, when interacting with the environment over a longer period, the observations reveal the data stream is more likely to be generated by environment C than environment D.

The next set of environments, Figures 1E1,E2, represents a decision process originating in S0 where one sequence of actions takes the agent to S7, and all other sequences lead to S9. The challenge in this set is that the observations do not reveal the distance from S0, and thus make it hard for an agent to locate itself along the graph as it progresses from left to right. E1 and E2 have two further variations, E3 and E4 (not shown). E3 is the same as E1 but the transitions from S1 have inverted actions. Similarly, E4 is the same as E2 but transitions from S1 have inverted actions. These four environments are very similar in structure, but they require different policies to be traversed from S0 to S7.

Finally, environments Figures 1F1,F2 have most of their transition probabilities being exactly the same. This means that transitions in the data stream distinguishing the two environments occur less frequently than in some other environments.

The experiments in this section assess overall stability of the detection and the impact of the history length expressed by the parameter L. Figure 2 shows the effect of different history lengths when tracking the probability of an accurate PSR model for environment C. In Figure 2 (left), the values of L for 40, 60, 80, 100, and 120 are shown. In all cases, the AMD shows consistent p-values of 0 and 1, indicating that the model can confidently determine which model the data stream belongs to. The longer the history, the slower the change in p-value, confirming the intuitive notion that longer histories require more time steps to reveal a change in the environment. When assessed on the stochastic environments A and B (Figure 2, right), the dropping p-values indicate that stochasticity is a significant confounding factor in the detection of the environment.

It can be concluded that a small L is advantageous to detect changes more readily only if the environment is predominantly deterministic. When the environment has stochastic transitions, a longer history might be necessary to guarantee the stability of the detection. To further assess these dynamics, Figure 3 shows the comparison of a short and a long history window (L = 20 and L = 120) on the deterministic environment C and the stochastic environment D. The AMD p-values show that while C can be tracked reliably with both L = 20 and L = 120, environment D (orange line) causes the p-value to oscillate, although with less amplitude, even with L = 120.

Other factors that can affect the stability of the p-values could include the complexity and the similarity with other environments. In Figure 4 (left), the tracking of the environments E1 and E2 is shown with L varying from 60 to 220. The faster settings (L = 60 and L = 100) appear to detect E2 when still in E1. With L = 140, the detection becomes more reliable, and with L = 180 and L = 220 the detection is accurate, although slower after step 2,000 to detect the transition from E1 to E2. Figure 4 (right) shows the p-values for all four E models tracked simultaneously. Despite the similarity of these four environments, the p-values show high confidence in determining which model currently matches the data stream. A similar result is also observed in Figure 5 where the environments of the set F are tested. The stochasticity in this set does not affect significantly the stability of the p-values. This is because the environment is primarily deterministic, and although the points where the data streams differ do not occur often, the p-value drops significantly when they do occur.

The ability to detect which environment the current stream of data belongs to allows the system to train different models independently, and thus implement continual learning and avoid catastrophic forgetting. The scenario devised in this section is when two unknown environments X and Y alternate while a learning system is trying to learn them from the data stream. This condition is particularly challenging because the data stream is generated by two different environments, both unknown. Therefore, there is an obvious bootstrap problem. How can the AMD know when the environment changes before any environment has been learned?

A reasonable assumption to overcome this problem is to assume that the data stream is initially generated by a single environment for a certain amount of time, so that one single model can be at least partially trained on the initial data stream.