An Event-Based Solution to the Perspective-n-Point Problem

The goal of the Perspective-n-Point problem (PnP) is to find the relative pose between an object and a camera from a set of n pairings between 3D points and their corresponding 2D projections on the focal plane. Current state of the art solutions, designed to operate on images, rely on computationally expensive minimization techniques. For the first time, this work introduces an event-based PnP algorithm designed to work on the output of a neuromorphic event-based vision sensor. The problem is formulated here as a least-squares minimization problem, where the error function is updated with every incoming event. The optimal translation is then computed in closed form, while the desired rotation is given by the evolution of a virtual mechanical system whose energy is proven to be equal to the error function. This allows for a simple yet robust solution of the problem, showing how event-based vision can simplify computer vision tasks. The approach takes full advantage of the high temporal resolution of the sensor, as the estimated pose is incrementally updated with every incoming event. Two approaches are proposed: the Full and the Efficient methods. These two methods are compared against a state of the art PnP algorithm both on synthetic and on real data, producing similar accuracy in addition of being faster.

with w 0 < 1. Then it is true that: Since the weights decay towards zero with j, we can approximate A k by taking into account every past 2 event e k−j with j = 0, 1, ... , k − 1: Developing this expression A k becomes: In the case of B k , it becomes: some recent events for which the estimated pose is approximately constant. Under this assumptions we can 11 approximate the value of B k in an analogous manner, making:

B JUSTIFICATION OF THE ROTATION
When we apply a resulting torque Γ to a body, the Newton's Second Law for rotation states that Taylor where H is the angular momentum of the body that takes the value: Here, ω is the angular velocity and J is a 3 × 3 symmetric matrix known as the inertia tensor. The form of 16 J depends on how the mass of a body is distributed, and it gives an idea of how hard it is to accelerate the 17 object around each one of the axis. However, since we are not modeling the behavior of a real mechanical 18 system, we can imagine the mass of our virtual system to be equally distributed in all directions. This is 19 equivalent to imagining the object to be embedded in a uniform sphere. The inertia tensor of a sphere has 20 the form: where α is a real value, equal in the case of the sphere to 2ml 2 /5, l being the radius of the sphere and m its 22 mass.

23
After these considerations (7) becomes: which is equivalent to the Newton's Second Law for translational motion. If we integrate this equation for a 25 small period of time ∆t, during which we suppose Γ to remain constant, and assuming zero initial angular 26 velocity, we obtain the following rotation r: where we make λ r = ∆t 2 2α .

C MAXIMUM TORQUE AND OPTIMAL λ R
Developing the expression for the torque associated with event e k−j , we get: which is expressed as the addition of two vectors. The first one of them is equal to the cross product of a 30 vector with its own projection. Taking into account that RV = V , its norm is thus bounded by the 31 expression: where the maximum value is attained when R * V i(k−j) and the line of sight form an angle of π/4 radians.

33
Next, (L k−j − I 3 )T * is equal to the distance between T * and the corresponding line of sight. If we 34 assume that the estimated translation is close to its true value T * ≈ T , then it follows that: which happens when RV i(k−j) is perpendicular to the line of sight.

36
The maximum torque will therefore be produced when both vectors in (12) take their maximum value, that is to say when RV i(k−j) is perpendicular to the line of sight, and R * V i(k−j) forms an angle of π/4 radians with the line of sight. Fig. 1 shows the state of the system in this case. From the geometry it follows that: If we call ρ max to the maximum norm ρ max = max i { V i }, then the resulting torque is bounded by the 37 expression: which depends on the dimensions of the considered object.

39
From the geometry, it follows that the angle formed by R * V i(k−j) and RV i(k−j) is in this case 3π/4 40 radians. We will accept the optimal value of λ r , that we denote λ opt r , to be the one that causes the angle of 41 the rotation applied in this case to be equal to 3π/4 radians. λ opt r is thus given by the expression: Let us note that the maximum value of the torque is very unlikely to be produced. Even if the geometry 43 matches the right one, different points of the object are likely to produce events, usually yielding lower 44 values of the torque. Consequently, λ opt r is a conservative value, and we can have stable systems with λ r 45 greater than λ opt r . However, we consider this expression to be an useful tool to determine the order of 46 magnitude for this parameter.
47 Figure 1. State of the system when the maximum torque is produced: R * V i(k−j) forms an angle of π/4 radians with the line of sight, causing R * V i(k−j) × L k R * V i(k−j) to be maximum. At the same time, RV i(k−j) is perpendicular to this same line of sight, making the distance between T and the line of sight maximum: (L k−j − I 3 )T = V i(k−j) .