Suppose that a statistical model M={p(x;θ): θ∈Θ⊂Rp} is given, which is regarded as a p-dimensional differential manifold and called a statistical model manifold (though it will be called simply a model where no confusion is possible). As usual, all necessary regularity conditions are assumed.

We also define the Riemannian metric and affine connections on the manifold M. Let l = log p(x; θ) denote the log-likelihood function.

Definition 1. The Riemannian metric g_ij = g(∂_i, ∂_j) is defined as

where ∂il=∂l∂θi=∂logp(x;θ)∂θi and E[·] denotes expectation with respect to observation x. The above quantities are also called the Fisher information matrix in statistics. Thus, we often call the above metric the Fisher metric.

The statistical cubic tensor and the coefficients of the e-connection are defined as

Definition 2. For every real α, p³ quantities

define an affine connection, which is called the α-connection.

We identify an affine connection with its coefficients below. Connection coefficients with upper indices are obtained by

where g^ij is the inverse matrix of the Fisher metric g_ij, and we have used Einstein's summation convention [see, e.g., Amari and Nagaoka [14] for details].

Conventionally, when α = 1, we call it the e-connection and when α = −1, we call it the m-connection and denote it as Γ(m)ij,k, i.e.,

It is well-known that α-connection and −α-connection are mutually dual with respect to the Fisher metric. (In a Riemannian manifold with an affine connection Γ, another affine connection Γ^* is said to be dual with respect to Γ if it satisfies ∂kgij=Γki,j+Γkj,i*. For equivalent definitions, see, e.g., Amari and Nagaoka [14], Chap. 3.) When α = 0, the self-dual connection is called the Levi-Civita connection, which defines a parallel transport that keeps the Riemannian metric invariant. The Levi-Civita connection is defined by the sum of the partial derivative of the metric, and its explicit form is given by

In the present study, the following identities are useful. They are obtained in a straightforward manner; thus, their proofs are omitted.

Lemma 1. Let m_ijk = E[∂_i∂_j∂_kl]. Then,

and

hold.

The first equation yields relation (Equation 1). The last equation shows the duality of e- and m-connections directly and is generalized to ±α-connections.

Lemma 2. For mutually dual connections, the following identities hold.

Using Lemma 1 and Equation (1), we obtain the Bartlett identity, which is well-known in mathematical statistics.

Lemma 3. For m_ijk, T_ijk, and, the first derivative of Fisher metric, g_ij, the following holds.

In Bayesian statistics, for a given statistical model M, we need a probability distribution over the model parameter space, which is called a prior distribution, or simply a prior. We often denote a prior density as π (π(θ) ≥ 0 and ∫Θπ(θ) θ=1).

A volume element on a p-dimensional model manifold corresponds to a prior density function over the parameter space (θ ∈ Θ ⊂ R^p) in a one-to-one manner. For a prior π(θ), its corresponding volume element ω is a p-form (differential form of degree p) and is written as

in the local coordinate system.

For example, in two-dimensional Euclidian space (p = 2), the volume element is given by ω = dx ∧ dy in Cartesian coordinates (x, y). In polar coordinates (r, θ), it is written as ω = rdr ∧ dθ.

Then, under the coordinate transformation θ → ξ, how do the probability density on the parameter space and its ratio change? From the formula for the p-dimensional volume element, it is written as

where |∂θ∂ξ| denotes the Jacobian. In differential geometry, such quantities are called tensor densities.

From the above Equation (5), we see that the ratio of two probability densities, say π1(θ)π2(θ), is invariant under reparametrization.

We briefly summarize some of the prior studies on noninformative priors in Bayesian statistics. Basically, a noninformative prior is often defined as the solution of a partial differential equation (PDE) derived from fundamental principles. If it is independent of parametrization, then it usually has a geometrical meaning. The defining equation itself is expected to be invariant under every coordinate transformation.

As an underlying principle, Bernardo [5] adopted construction of the minimax code in information theory to derive noninformative priors. After that, the noninformative prior is defined as the input source distribution that maximizes the mutual information between the parameter and the outcome. This prior is called a (Bernardo's) reference prior. Under some conditions, his idea has been strictly formulated by several authors [18, 19] (for a review, see, e.g., Berger et al. [6]).

In one of the many studies and variants of reference priors, recently Liu et al. [13] adopted the α-divergence instead of the KL-divergence in Bernardo's argument and obtained a generalized result.

Definition 4. Let p(x) and q(x) be probability densities. For a fixed real parameter α, the α-divergence from p to q is defined as

Remark 2. In the textbook on information geometry by Amari [20], the following parametrization is used because of the emphasis on the duality:

where we write β instead of α. We adopt the parametrization of α in Equation (10). For example, χ²-divergence corresponds to α = −1 in Equation (10) and β = 3 in Equation (11). More explicitly, the relation α=1-β2 (and thus, 1-α=1+β2) holds.

When α = 0, 1, taking the limit, the α-divergence reduces to the KL-divergence.

Now, let us see the definition of the noninformative prior proposed by Liu et al. [13]. Under regularity conditions (e.g., the compactness of the parameter space Θ), they considered the maximization of the following functional of a prior density π as follows:

where E[·|θ] denotes expectation with respect to p(X|θ), and the expression emphasizes that the parameter θ is fixed in the integral. Under their criterion, the maximizer of J[π] is adopted as a noninformative prior.

Following Liu et al. [13], we rewrite the above functional Equation (12) in a more simple form as follows:

Depending on the sign of α(1 − α), our problem reduces to maximization or minimization of the expectation E[π(θ|X)^−α| θ]. Clearly, it is not solved explicitly for general cases. Thus, as usual, we adopt the approximation of the expectation term under the assumption that X=(X1,…,Xn)~i.i.d.p(x|θ) with n → ∞.

Except for α = −1 (χ²-divergence), the maximization of J[π] reduces to that of the first-order term in the following expansion (Theorem 2), which yields the Jeffreys prior for −1 < α < 1. However, for χ²-divergence, we need to evaluate the second-order term since the first-order term is constant.

First, we present a key result in Liu et al. [13]. Some notation in their result follows ours. For example, the Fisher information matrix and its determinant are denoted as g_ij and g, respectively. The dimension of the parameter θ is denoted as p. Please refer to the original article for technical details.

Theorem 2. [Liu et al. [13], Theorem 3.1] The expectation term E[π(θ|X)^−α| θ] in the functional J[π] can be rewritten as

where the 1/n part in braces {⋯} is given by

The last term s(θ) does not include the prior density π.

From Theorem 2, for a positive constant C_n and a sufficiently large n, the functional Equation (12) is approximated by

When −1 < α < 1, the maximization yields π∝g12, that is, the Jeffreys prior. When α < −1, rather, the Jeffreys prior minimizes the functional J[π].

However, at the boundary point α = −1 (χ²-divergence), the above first-order term becomes a constant independent of π. In this case, we need to evaluate the second-order term more carefully.

Now let us rewrite the second-order term of the asymptotic expansion in Theorem 2 in terms of information geometry. We fix α = −1, and from here on, consider only the case for χ²-divergence.

Although our approach differs from that in the original article, the final PDE agrees with their result. The difference and our contribution are discussed in the next subsection.

We summarize how we rewrite each term to obtain the final result (Theorem 3) later. First, we rewrite ∂i∂jππ by using the following relation:

After that, we replace a prior density π with the density ratio h = π/π_J, where πJ=g. The terms including the prior density π and its derivatives are expected to be written using the scalar function log h. Indeed, this expectation is correct, and we obtain the final form after tedious, lengthy, and straightforward calculation. Because we use partial integrals in transforming the original form of the asymptotic expansion, the integral symbol remains in the expression below.

Theorem 3. [Liu et al. [13]], Corollary of Theorem 3.1.

where the 1/n part in square brackets is given by

in which, we set T:=Tidθi and the norm of one-form A is defined as ||A||2:=AiAjgij.

The above one-form T is called the Tchebychev form in affine geometry [see, e.g., p. 58 in Simon et al. [21]].

From Theorem 3, maximizing J[π] over the set of all prior densities is equivalent to maximizing the above integral with respect to a scalar function h when n → ∞. Since the second and third terms inside braces {⋯} in Equation (13) are independent of h, the expression achieves the maximum if the first term vanishes, that is,

holds. Thus, we obtain an equation of a differential one-form which determines χ²-prior. In a proper coordinate system, the component-wise form of equation (14) is given by

which agrees with the original PDE (Equation 9) derived in the previous study.

Finally, we discuss the existence of χ²-prior. Generally, χ²-prior does not necessarily exist on a statistical model. The existence of a χ²-prior on a given model is equivalent to the existence of the solution of PDE (Equation 9).

A solution of PDE (Equation 9) exists if and only if T_i satisfies the following condition:

which is called an integrability condition and is well-known. As a simplification, we may write dT = 0.

A bit surprisingly, the above condition (Equation 15) agrees with the condition that the α(≠0)-parallel prior exists [11]. This implies a certain relationship between χ²-prior and an α-parallel prior. Indeed, its expectation is correct and χ²-prior is shown to be the 12-parallel prior, which is the theme in the next section.

We here discuss the difference between the original result obtained by Liu et al. [13] and the present study.

First, the PDE they obtained for χ²-prior is not in the form of tensor equations. They gave a PDE for log π as follows

instead of our PDE (Equation 9) [Liu et al. [13], p. 357, Equation (48)]. Both sides of Equation (16) are not tensors, i.e., not invariant under coordinate transformation.

Second, although Liu et al. obtained the asymptotic expansion as in Theorem 2 (Theorem 3.1 in the original article), their approach to derive the PDE (Equation 16) is not sufficient. Strictly speaking, they only show that πχ2 satisfying the PDE (Equation 16) achieves the extreme value asymptotically. They did not organize messy terms and utilized the variational method in an ad hoc manner to derive the PDE (Equation 16). Moreover, their approach does not exclude the possibility of achieving the minimum.

Our approach shows more directly that πχ2 satisfying the PDE (Equation 16)achieves the maximum of the functional asymptotically. Using the square completion for the one-form d log h, we show that πχ2 maximizes the functional J[π] when n → ∞.

In addition, our underlying philosophy is the invariance principle under coordinate transformation. Clearly, the expected χ²-divergence from a prior to its posterior is independent of parametrization. Thus, we naturally expect that the O(n⁻¹) term is independent of parametrization, i.e., represented by geometrical quantities. As a result, we obtain a simpler expression (Equation 13) in Theorem 3. This is a good example of how organizing from the viewpoint of information geometry can simplify various terms and make the structure of the problem easier to understand.

As for derivation of fundamental PDEs, we point out a formal analogy between general relativity and ours. Historically speaking, Hilbert showed that the Einstein equation is derived from Einstein–Hilbert action integral S[g_ab], where g_ab is the pseudo-Riemannian metric on the time-space manifold [see, e.g., Wald [22], Appendix E.1]. In our problem, we take the expected χ²-divergence from a prior to its posterior instead of S[g_ab]. The maximization of J[π] and minimization of S[g_ab] yield the tensor equation (Equation 16) and the Einstein equation, respectively.

Takeuchi and Amari [11] introduced a family of geometric priors called α-parallel priors, which include the well-known Jeffreys prior and maximum likelihood (ML) prior [23]. We briefly review basic definitions and related results on α-parallel priors below.

Now let us consider a relation between α-parallel prior and χ²-prior. To compare them, we focus on the following PDE for an α-parallel prior:

[Takeuchi and Amari [11], Proposition 1, p. 1016, Equation (7)].

Since both sides of the PDE (Equation 20) are not tensors, its invariance under coordinate transformation is not clear. Thus, we introduce a one-form (geometrical quantity) derived from a scalar function h = π/π_J and modify the equation.

Theorem 5. The above PDE (Equation 20) is equivalent to the following tensor equation:

When we set T=Tidθi, then the above equation (21) can be rewritten as

which is the equation of a differential one-form.

Proof. Using Equation (8), we rewrite the PDE (Equation 20) as follows:

Therefore, using Lemma 2, we modify the RHS of Equation (22):

Surprisingly, the PDE defining πχ2 (Equation 9) agrees with Equation (21) with α=12. Thus, χ²-prior derived by Liu et al. [13] is the 12-parallel prior. This finding is interesting in two ways.

First, to Bayesian statistics, it is a new example where the formulation in terms of information geometry is useful to research on noninformative priors [for several examples, see Komaki [7] and Tanaka and Komaki [9]]. Liu et al. [13] derived the PDE (Equation 9) by considering one extension of the reference prior with χ²-divergence. Their starting point is completely independent of the geometry of statistical models. In spite of this, χ²-prior has a good geometrical interpretation: it is volume element invariant under the parallel transport with respect to the 12-connection.

Second, to information geometry, it would be the first specific example where only the 12-connection makes sense in statistical applications. In information geometry, the meaning of each connection among α-connections has not been clarified enough except for specific alphas (α = 0, ±1). In Takeuchi and Amari [11], α-parallel priors were not proposed as noninformative priors. Rather, they regarded the Jeffreys prior as the 0-parallel prior and extended it to every α. Except for α = 0, only the 12-parallel prior is interpreted as a noninformative prior.

Let us derive a general form of α-parallel priors in statistically equiaffine models. In the following, we denote an α-parallel prior as π_α. For example, π1/2=πχ2 and π₀ = π_J.

First, we briefly review some formulas for α-parallel priors derived by several authors [11, 25]. According to Matsuzoe et al. [25], there exists a scalar function ϕ that satisfies T_a = ∂_aϕ on a statistically equiaffine model manifold. Therefore, using this function ϕ, a general solution of the PDE (Equation 21) is given by h(θ)∝exp{-α2ϕ(θ)}. Thus, we obtain α-parallel prior π_α as having the following form:

In the exponential family (e-flat model), Takeuchi and Amari show that α-parallel priors are representable as a power of Jeffreys prior π_J for every α [Takeuchi and Amari [11], Example 2, p. 1017].

Here, we extend their result for the exponential family to a γ(γ ≠ 1)-flat model. We use the parameter γ instead of α because the two parameters may be different.

Theorem 6. Let γ ≠ 0. Suppose that a statistical model manifold (M,g,∇(γ),∇(-γ)) is γ-flat. Then, there exists an α-parallel prior π_α for every α. In a γ-affine coordinate system, {θⁱ}, it is written as a power of Jeffreys prior π_J, that is,

holds.

Proof. Since the model is γ-flat, we take a γ-affine coordinate system, say, {θⁱ}. Then, Γ(γ)ij,k=0 and from Equation (3) in Lemma 2,

holds.

On the contrary, from a result in Amari and Nagaoka [14], Section 3.3, there exists a scalar function ψ(θ) such that

for the γ-affine coordinates {θⁱ}. This implies that

Therefore, using Eqs (23) and (24), we can rewrite T_i as follows:

Clearly, T satisfies the condition (Equation 19), and thus, the model is statistically affine. Therefore, Theorem 4 implies that there exists an α-parallel prior π_α for every α.

Now, let us obtain an explicit form by using π_J. Substituting the above T_i into the RHS of Equation (21),

and thus, we obtain

In particular, we get

It is true only in the γ-affine coordinate system {θⁱ} that π_α is equal to a power of the Jeffreys prior. Since the above argument is not invariant under coordinate transformation, we take the Jacobian into consideration in another coordinate.

Theorem 6 includes previous results. Liu et al. [13], Example 1, corresponds to the case when α = 1/2 and γ = 1. Takeuchi and Amari [11], Example 2 (p. 1017), corresponds to the case when γ = 1.