Before going on to lay out the theory for the extraction of motion invariant visual features, we need to recall some facts about the optical flow condition Eq. 2. Given a video stream described by its brightness intensity I as it is defined as in Section 1, the problem of finding for each pixel of the frame spatial support at each time instant the velocity field u(x, t) satisfying ∂ t I x , t + u x , t ⋅ ∇ I x , t = 0 ∀ x , t ∈ Ω × 0 , T (3) is clearly ill posed since a scalar equation is not sufficient to properly constrain the two components of u. Locally, we can unequivocally determine only the component of u along ∇I.

Although many methods have been proposed to overcome this issue (see for example the work of Aubert and Kornprobst (2006)), that is usually referred to as the aperture problem, here we are interested in the class of approaches that aims at regularizing the optical flow: inf v A I v + S v + H I v (4) where A _I is a functional that enforces constraint Eq. 3, hence a standard choice is A I v ≔ ∫ Ω ∂ t I x , t + v ⋅ ∇ I x , t 2 d x , (5)

S imposes smoothness and H may be used to condition the extraction of the flow over spatially homogeneous regions. Depending on the regularity assumptions on I and the properties that we want to impose on the solution of this regularized problem (namely if we want to admit solutions that preserve discontinuities or not) the exact form of S and the form and presence of H _I may vary. For example, in the original approach proposed in Horn and Schunck (1981) we find: Horn–Schunck regularization : S v = ∫ Ω | ∇ v 1 | 2 + | ∇ v 2 | 2 d x , H I ≡ 0 . (6)

Note that since we are interested in extracting the optical flow for any frame of a video, namely the field u(x, t), it is useful for any function f : Ω × [ 0 , T ] → R p to define f t : Ω → R p as f ^t (x)≔f(x, t). With this notation, when the infimum in Eq. 4 is attained, we can pose. u x , t = u t x = arg min v A I v + S v + H I v . (7)

Notice that the brightness might not necessarily be the ideal signal to track. Since the brightness can be expressed as a weighed average of the red R, green G, and blue B components, one could think of tracking each single color component of the video signal by using the same invariance principle stated by Eq. 3. It could in fact be the case that one or more of the components R, G, B are more invariant in the sense of Eq. 3 during the motion of the corresponding material point, see Figure 2. In that case, in general, each color can be associated with corresponding optical flows v _R , v _G , v _B that might differ. In doing so, instead of tracking the brightness, one can track the single colors. Instead, under the assumption that each color component has the same optical flow v = v _R = v _G = v _B we have. ∂ ∂ t R G B + v ⋅ ∇ R G B = 0 , (8) where v ⋅∇(R,G,B)′≔(v ⋅∇R,v ⋅∇G,v ⋅∇B)′. It is worth mentioning that the simultaneous tracking of different channels might contribute to a better positioning of the problem since, in general, rank∇(R, G, B)′ = 2 and the system Eq. 8 admits an unique solution. ¹ One can think of the color components as features that, unlike classical convolutional spatial features, are temporal features.

Before proceeding further, let us underline that some other optical flow methods try to directly solve the brightness invariance condition Eq. 1 without differentiating it. This is the case, for example, of the Gunnar Farnebäck’s algorithm (Farnebäck, 2003): the basic idea here is to approximate the brightness of the input images through polynomial expansions with variable coefficients, and the brightness invariance condition Eq. 1 is then solved under this assumption. Figure 3 shows the optical flows extracted by the Horn-Schunck and Farnebäcks methods in the barber’s pole case.

In the next section, we will discuss how to use a very similar approach, based on the consistency of features along apparent motion trajectories on the frame spatial support, to derive visual features φ _i along with the corresponding conjugate optical flows v φ i . We anticipate that this motion consistency condition will also play a prominent role in defining affordance features, as described in Section 3.2.

As already noticed, when we consider color images, what is done in the case of brightness invariance can be applied to the separated components R,G,B. Interestingly, for a material point of a certain color, given by a mixture of the three components, we can establish the same brightness invariance principle, since those components move with the same velocity. Said in other words, there could be group of different visual features φ _i , i = 1, …, m that share the same velocity (v _i = v) and are consistent with it, that is ∂ _t φ _i + v ⋅∇φ _i = 0 ∀i = 1, …, m. Thus, we can promptly see that any feature φ of span(φ ₁, …, φ _m ) is still conjugated with v; we can think of span(φ ₁, …, φ _m ) as a functional space conjugated with v.

Let us now consider the feature group ϕ = ( φ 1 , … , φ m ) ′ and the corresponding invariance condition. ∂ t ϕ + v ⋅ ∇ ϕ = 0 , (11) where ∇ ϕ ∈ R m × 2 is the matrix with elements ( ∇ ϕ ) i j = ( ∇ ϕ i ) j and v ⋅ ∇ ϕ ≔ v ⋅ φ 1 , … , v ⋅ φ m ′ . An important observation, very related to the discussion about color tracking of Eq. 8, is the following one. Notice that, if we consider the case in which the only scalar feature we are dealing with is the brightness, then Eq. 11 boils down to a single equation with two unknowns (the velocity components). Differently, in the case of the feature group ϕ, we have m equations and still two unknowns. The dimension m of matrix ∇ϕ can enforce the increment of its rank, which leads to a better posedness of the problem of estimating the optical flow v. Because of the two-dimensional structure of the frame spatial support, which leads to v ∈ R 2 , and since ∇ ϕ ∈ R m × 2 , with m ≥ 2, it turns out that feature grouping regularizes the velocity discovery. In order to understand the effect of feature grouping we can in fact simply notice that, under the assumption rank∇ϕ = rank(∇ϕ∣ − ϕ _t ), a random choice of the features yields rank∇ϕ = 2. As a consequence, by Rouché-Capelli theorem, linear Eq. 11 admits a unique solution in v. However, this regularization effect of feature grouping does not prevent ill-posedness, since ϕ is far from being a random map. On the opposite, it is supposed to extract a uniform value in portions of the frame spatial support that are characterized by the same feature. Hence, rank∇ϕ = 1 is still possible whenever the features of the group are somewhat dependent.

Feature groups, that are characterized by their common velocity, can give rise to more structured features belonging to the same group. This can promptly be understood when we go beyond linear spaces and consider for a set of indices F . α = ∑ j ∈ F w j φ j η = σ α . (12) Evaluating ∂ _t η + v ⋅∇η we obtain indeed ∂ t η + v ⋅ ∇ η = σ ′ α ∂ t α + v ⋅ ∇ α = σ ′ α ∑ j ∈ F ω j ∂ t φ j + v ⋅ ∇ φ j . (13) and we conclude that if ∀ j ∈ F we have ∂ _t φ _j + v ⋅∇φ _j = 0 then also the feature η defined by Eq. 12 is conjugated with v, that is ∂ _t η + v ⋅∇η = 0. However, the vice versa does not hold true. Basically, the inheritance of conjugation with v holds in the direction towards more abstract features. Of course, the feedforward-like recursive application of the derivation stated by Eq. 12 yields a feature that is still conjugated with v.

Any learning process that relies on the motion of a given object can only aspire to discover the identity of that object, along with its characterizing visual features such as pose and shape. The motion invariance process is in fact centered around the object itself and, as such, it does reveal its own features in all possible expositions that are gained during motion. Humans, and likely most animals, also conquer a truly different understanding of visual scenes that goes beyond the conceptualization with single object identities. In the early Sixties, James J. Gibson coined the notion of affordance in (Gibson, 1966), even though a more refined analysis came later in (Gibson, 1979). In his own words: “The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. The verb to afford is found in the dictionary, the noun affordance is not. I have made it up. I mean by it something that refers to both the environment and the animal in a way that no existing term does. It implies the complementarity of the animal and the environment.” Considering this animal-centric view, we gain the understanding that affordance can be interpreted as what characterizes the “interaction” between animals and their surrounding environment. In more general terms, the way an agent interacts with a particular object is what defines its affordance, and this is strictly related to their relative motion. In the last decades, computer scientists have also being working on this general idea, trying to quantitatively implement it in the fields of computer vision and robotics (Ardón et al., 2020; Hassanin et al., 2021). As far as visual affordance is concerned, that is, extracting affordance information from still images and videos, different cognitive tasks have been considered so far, as for example affordance recognition and affordance segmentation, see (Hassanin et al., 2021) for a recent review.

In the spirit of the previous section, we will consider a more abstract notion of affordance, characterizing the interaction between different visual features along with their corresponding conjugate velocity fields. We will focus our attention on actions that are perceivable from single pictures ² and on the related local notion of affordance, that will be defined by some function characterizing the interaction between feature φ _j and feature φ _i when considering the pixel x at the specific time instant t. As we will see, the principle of motion invariance can be extended to naturally define (explicitly or implicitly) this generalized notion of affordance. A natural choice is to consider what we will denote as the affordance field ψ _ij as a function of space and time. To implicitly codify the interaction between features i and j, ψ _ij (x, t) has to be constrained by some relation of the form g ψ i j , ∂ t ψ i j , ∇ ψ i j , φ i , v i , φ j , v j = 0 , where we are considering only first order derivatives of the affordance field and g is a scalar function. In the lower order approximation (we also need quadratic terms to build scalars from vectors): g ψ i j , ∂ t ψ i j , ∇ ψ i j , φ i , v i , φ j , v j = a 1 + a 2 ψ i j + a 3 φ i + a 4 φ j + a 5 ∂ t ψ i j + a 6 v i ∇ ψ i j + a 7 v j ∇ ψ i j , (14) where a ₁, …, a ₇ are scalars. Considering the case in which the motion field associated to feature φ _j is everywhere null, the affordance field ψ _ij will codify a property only related to φ _i itself and strictly related to its identity, that has to be invariant with respect to v _i . Thus, from this observation, we can infer a ₁ = a ₂ = a ₃ = a ₄ = 0 and a ₅ = a ₆ so that the above constraint becomes: ∂ t ψ i j + v i + b j v j ⋅ ∇ ψ i j = 0 (15) where b _j = a ₇/a ₅. Requiring b _j = −1 this constraint assumes a very reasonable physical meaning, that is the motion invariance of the affordance field ψ _ij in the reference of feature φ _j . Within this choice, the affordance field is conjugated with the velocity v _i − v _j indeed, which is in fact the relative velocity of feature φ _i in the reference of feature φ _j . Considering points at the border of φ _i , this can lead to slightly expand ψ _ij outside the region defined by φ _i itself, as shown in Figure 4. In the case v _i = 0, the motion consistency is forced “backward” along the pixels’ trajectories defined by − v _j . In the case b _j = 1 Eq. 15 becomes symmetric under permutations instead so that ψ _ij and ψ _ji will be developed exploiting the same constraint. This will likely result in the same affordance feature unless some other factor (let us think for example to different initializations in neural architectures) breaks that symmetry. From the classic affordance perspective this is not a desirable property as we can easily understand considering, for example, a knife that is used to slice bread: the affordance transmitted by the knife to the bread would be strictly related to the possibility of being cut or sliced, that is clearly a property that could not be attached to the knife.

Another viable and different alternative to codify the interaction between features may be the one of directly evaluating the affordance as function of the feature fields and their respective velocities: ψ ̃ i j x , t , φ i , v i , φ j , v j . Here, we are giving up the previous field theory approach, being explicitly codifying the interaction between features in the computational scheme of the ψ ̃ i j function. On the other hand, since we have already distinguished ψ ̃ i j and φ _i by the different computational structure, requiring the same motion invariance property of φ _i with respect to v _i for the affordance function ψ ̃ i j appears a very natural choice, see also Figure 5: ∂ t ψ ̃ i j + v i ⋅ ∇ ψ ̃ i j = 0 . (16)

Given the possible great variability of velocity fields in a visual scene, let us underline that within this second approach some problems in the learning of the affordance function ψ ̃ i j x , t , φ i , v i , φ j , v j may emerge. Moreover, to pursue the fascinating idea to describe all the visual processes entirely through visual fields defined on the frame spatial support, in the following we will only consider the affordance field ψ _ij (x, t) and the related motion invariance property Eq. 15 with b _j = −1.

Given a certain visual environment we can easily realize that, as time goes by, object interactions begin obeying statistical regularities and the interactions of feature φ _i with the others become very well defined. Hence, the notion of ψ _ij can be evolved towards the inherent affordance ψ _i of feature φ _i , which is in fact a property associated with φ _i while living in a certain visual environment. For example, thinking in terms of the classic notion of affordance, when considering a knife the related inherent affordance property is gained by being manipulated, in a certain way, by a virtually unbounded number of different people. Based on Eq. 15 (b _j = −1) we define the inherent feature affordance as the function ψ _i (x, t) which satisfies. ∂ t ψ i + v i − v j ⋅ ∇ ψ i = 0 , 1 ≤ i , j ≤ n . (17)

Let us note that the above formula can also be interpreted as the motion invariance property of ψ _i with respect to the velocity v i − ∑ j = 1 n v j / n . The identification feature φ _i pairs with the corresponding affordance feature ψ _i , and the visual scene turns out to be effectively described by the collection of visual fields V = { ( φ i , ψ i , v i ) } . In a sense, ψ _i can be thought of as the abstraction of φ _i , as it arises from its environmental interactions. A few comments are in order concerning these visual fields.

• The pairing of φ _i and ψ _i relies on the same optical flow which comes from φ _i . This makes sense, since the inherent affordance is a feature that is expected to gain abstraction coming from the interactions with other features, whereas the actual optical flow can only come from identifiable entities that are naturally defined by φ _i .

In order to abstract the notion of affordance even further we can, for instance, proceed as follows: for each κ = 1,…, n we can consider another set of fields χ κ : Γ → R each of which satisfies the following condition ∂ t χ κ + v j ⋅ ∇ χ κ = 0 , j = 1 , … , n . (18)

In this way the variables χ _κ do not depend, unlike for ψ, on a particular v _i , which contributes to lose the link with its firing feature. Moreover, they need to take into account, during their development, multiple motion fields which results in a motion invariant property with respect to the average velocity ∑ j = 1 n v j / n and in a greater level of abstraction.

Once the set of the χ _κ is given, a method to select the most relevant affordances could simply be achieved through a linear combination. In other words, a subselection of χ ₁, …, χ _n can be performed by considering for each l = 1, …, n _χ < n the linear combinations X l ≔ ∑ κ = 1 n a l κ χ κ , (19) where ( a l κ ) ∈ R n χ × n is a matrix of learnable parameters. Notice that since X _l ∈ span(χ ₁, …, χ _n ), as we remarked in Section 3.4, then ∂ _t X _l + v _j ⋅∇X _l = 0 for all j = 1, …, n. It is worth mentioning that the learning of coefficients a _lκ does not involve motion invariance principles. Interestingly, they can be used for additional developmental steps like that of object recognition. For example, they can be learned under the classic supervised framework along with the correspondent regularization.

We have already discussed the ill-posed definition of features conjugated with their corresponding optical flow. Interestingly, we have also shown that a feature group ϕ = ( φ i , … , φ m ) ′ with conjugate velocity v exhibits an inherent regularization that, however, does not prevent ill-positioning, especially when one is interested in developing abstract features that are likely constant over large regions of the frame spatial support.

Let us assume that we are given n feature groups ϕ _i , i = 1, …n, each one composed of m _i single features (m _i -dimensional feature vector) ϕ i ∈ R m i . Furthermore, let v _i be the velocity field shared by each component of feature group ϕ _i and let us also denote with ϕ = (ϕ ₁, …, ϕ _n ) and with v = (v ₁, …, v _n ). We can then impose the following generalization of the smoothness term (6), used for the classical optical flow estimation, to the velocities and the corresponding visual features: E = 1 2 ∑ i = 1 n ∫ Γ ∇ v i x , t 2 + λ φ ϕ i x , t 2 + λ ∇ ∇ ϕ i x , t 2 d μ x , t . (20)

Here, the notation Z 2 (generic argument matrix Z) means ∑ i , j Z i j 2 , while λ _φ , λ _∇ are positive constants that express the relative weight of the regularization terms. First of all, notice that E is a functional of the pairs {(ϕ _i , v _i )}, that is, once they are given, we can compute E( ϕ , v ). On the contrary, the index used to regularize the classical Horn–Schunck optical flow Eq. 6 only depends on v. The dependence on visual features and their temporal dynamic in Eq. 20 is explained considering that, while the brightness is given, the features are learned as time goes by, which is just another facet of the feature-velocity conjugation. Moreover, it is worth mentioning that E only involves spatial smoothness whereas it doesn’t contain any time regularization term. There is also another difference with respect to the classic optical flow regularization Eq. 6, that is the penalizing term ( 1 / 2 ) | ϕ i | 2 which favors the development of ϕ _i = 0. Of course, there is no such requirement in classic optical flow since, as already stated, the brightness is given. On the opposite, the discovery of visual features is expected to be driven by motion information, but their “default value” is expected to be null. We can promptly see that the introduction of the regularization term Eq. 20 does not suffice to achieve a well-posed learning problem. The motion invariance condition Eq. 9 is still satisfied by the trivial constant solution φ = c _φ indeed.

Important additional information comes from the need of exhibiting the human visual skill of reconstructing pictures from our symbolic representation. At a certain level of abstraction, the features that are gained by motion invariance possess a certain degree of semantics that is needed to interpret the scene. However, visual agents are also expected to deal with actions and react accordingly. As such, a uniform cognitive task that visual agents are expected to carry out is that of predicting what will happen next, which is translated into the capability of guessing the next incoming few frames in the scene. We can think of a predictive computational scheme based on the φ _i codes y ( x , t ) = α y ϕ ( ⋅ , t ) , t ( x ) , where α y : R Ω n × [ 0 , T ] → R Ω and the prediction y needs to satisfy the condition established by the index R = 1 2 ∫ Γ y x , t − I t x , t 2 d μ x , t . (21)

Of course, as the visual agent gains the capability of predicting what will come next, it means that the developed internal representation based on features ϕ cannot correspond with the mentioned trivial solution. Interestingly, it looks like visual perception does not come alone: the typical paired skill of visual prediction that animals exhibit helps regularizing the problem of developing invariant features. Clearly, the φ = c _φ which satisfies motion invariance is no longer acceptable since it does not reconstruct the input. This motivates the involvement of prediction skills typical of action that, again, seems to be interwound with perception.

Having described all the regularization terms necessary to the well-posedness of the learning problem, we can introduce the following functional. S ϕ , v = A ϕ , v + λ E E ϕ , v + λ R R ϕ , v (22) where A( ϕ , v ) is the direct generalization to feature groups of Eq. 10, that is A ( ϕ , v ) = 1 2 ∑ i = 1 n ∫ Ω × [ 0 , T ] | ∂ t ϕ i + v ⋅ ∇ ϕ i | 2 d μ ( x , t ) . Here, λ _R and λ _E > 0 are the regularization parameters. Learning to see means to discover the indissoluble pair ( ϕ ^⋆, v ^⋆) such that ϕ ⋆ , ν ⋆ = arg min ϕ , ν S ϕ , ν . (23)

Basically, the minimization is expected to return the pair ( ϕ , v ), whose terms should nominally be conjugated. The case in which we reduce to consider only the brightness, that is when the only ϕ is I, corresponds with the classic problem of optical flow estimation. Of course, in this case the term R is absent and the problem has a classic solution. Another special case is when there is no motion, so as the integrand in the definition Eq. 10 of A is simply null ∀i = 1, …, n. In this case, the learning problem reduces to the unsupervised extraction of features ϕ .

The learning of ψ _i can be based on a formulation that closely resembles what has been done for φ _i , for which we have already considered the regularization issues. In the case of ψ _i we can get rid of the trivial constant solution by minimizing. I ψ = ∑ i = 1 n ∫ Γ 1 − ψ i x , t φ i x , t d μ x , t , (24) which comes from the p-norm translation (Gori and Melacci, 2013; Gnecco et al., 2015) of the logic implication φ _i ⇒ ψ _i . Here we are assuming that φ _i , ψ _i range in [0, 1], so as whenever φ _i gets close to 1, it forces the same for ψ _i . This yields a well-posed formulation thus avoiding the trivial solution.

As far as the χ are concerned, like for ψ, we ask for the minimization of I χ = ∑ k = 1 n ∫ Γ 1 − χ κ x , t ψ κ x , t d μ x , t , (25) that comes from the p-norm translation of the logic implication ψ _κ ⇒ χ _κ . While this regularization term settles the value of χ _κ on the corresponding ψ _κ , notice that the motion invariance condition (18) does not assume any privilege with respect to the firing feature ψ _κ .

In the previous sections we discussed invariance properties of visual features that lead to model the processes of computational vision as transport equations on the visual fields, see Eqs 9, 15, 17, 18. Some of those properties are based on the concept of consistency under motion, others lead to a generalization of the concept of affordance. In this section we will discuss how the features φ, ψ and χ, along with the velocity fields v _φ , can be represented in terms of neural networks that operate on a visual stream and how the above theory can be interpreted in a classical framework of machine learning.

The first step we need to perform consists in moving to a discretized frame spatial support Ω ⋄ = { ( i , j ) ∈ N : 0 < i ≤ w , 0 < i ≤ h } . As a consequence, the fields φ _i and the velocities can be viewed as vector-valued functions of time t ↦ φ i ( t ) ∈ R Ω ⋄ and t ↦ v φ i ( t ) ∈ ( R Ω ⋄ ) 2 ; similarly the discretized brightness can be seen as a map t ↦ I ( t ) ∈ R Ω ⋄ . ³ Then, features φ _i (and similarly the fields ψ and χ) can be modelled as neural networks Φ i : R N × R Ω ⋄ → R Ω ⋄ that given the brightness I at a certain instant and a set of N weights w Φ i ∈ R N yield the value Φ i ( I , w Φ i ) of φ _i . Similarly, the velocities v _φ can be estimated by a neural network V i : R M × R Ω ⋄ × ( R Ω ⋄ ) 2 → ( R Ω ⋄ ) 2 that takes as inputs the temporal partial derivative I ̇ (that is the discrete version of the term ∂ _t I), the discrete spatial gradient ∇I of I and a given a set of M weights w v i in order to predict V i ( w v i , I ̇ , ∇ I ) as the velocity field v φ i .

It should be noted that, within this framework, the learning problem for the fields φ, χ, η and v _φ , that is based on the principles described in this paper and that is defined by the optimization problem of the form described in Eq. 23, becomes a finite-dimensional learning problem on the weights of the neural models.

Thus, learning will be affected by the structure (i.e. the architecture) of the network that we choose. Recent successes of deep learning within the realm of computer vision suggest that natural choices for Φ would be Deep Convolutional Neural Networks (DCNN). More precisely, the features extracted at level ℓ of a DCNN can be identified with a group of Φ _i ; in this way we are establishing a hierarchy between features that, in turn, suggests a natural way in which we could perform the grouping operation that we discussed in Section 3.1. In this way, features that are at the same level of a CNN share the same velocity (see Figure 6). In the case of velocities, CNN-based architectures like the one employed by FlowNet (see Fischer et al. (2015)) have been already proven to be suitable to model velocity fields.

It is also important to bear in mind that the choice of a specific neural architecture has strong repercussions on the way the invariance conditions are satisfied. For instance, let us consider the case of Convolutional Networks together with the fundamental condition expressed by Eq. 9. In this case, since CNN are equivariant under translations, any feature that tracks a uniformly translating motion of the brightness will automatically satisfy Eq. 9 with the same velocity of the translation of the input.