As shown in Figure 2, the LGMD-1 and the LGMD-2 possess the same visual processing in the first two pre-synaptic layers. The first layer is composed of photoreceptors arranged in a matrix sensing time-varying, single-channel luminance (grey-scale in our case). The photoreceptors compute temporal derivative of every pixel to get motion information. Let L ( x , y , t ) ∈ R 3 denote the input image streams, where x, y, and t are spatial and temporal positions, respectively. The current motion can be retrieved by P x , y , t = L x , y , t − L x , y , t − 1 + ∑ i = 1 n p a i P x , y , t − i , where a i = 1 + e i − 1 . (1) The motion persistence is constituted by n _p frames.

In addition, the P-layer also indicates the whole-field luminance change with respect to time. This indicator is applied as the FFI in the LGMD-1 neural network, which can be obtained by averaging the overall luminance change at time t. That is, F t = ∑ x = 1 R ∑ y = 1 C | P x , y , t | ⋅ C ⋅ R − 1 , (2) where C and R denote columns and rows of the visual field in pixels. In addition to that, the FFI goes through a time delay unit (see TD in Figure 2), defined as F ̂ t = α 1 F t + 1 − α 1 F t − 1 , α 1 = τ i / τ f + τ i . (3) τ _f stands for a time constant and τ _i is the time interval between consecutive frames of digital signals, both in milliseconds. Notably, here the FFI can directly shut down the LGMD-1 neuron if it overshoots T _ffi , i.e., a predefined threshold.

While modelling the LGMD-2, we propose a temporal tuning mechanism, the PM in Figure 2, to adjust local inhibitions in the medulla layer of the LGMD-2. The computations of PM conform to Eqs. 2, 3 which are not restated here.

Motion information inevitably induces luminance increment or decrement over time. As shown in Figure 2, the second lamina layer separates the relayed signals into parallel ON and OFF channels, at each node. More precisely, the luminance increment flows into the ON channel, whilst the decrement streams to the OFF channel with a sign-inverting operation. Both the channels retain positive inputs. That is, P o n x , y , t = P x , y , t + + α 2 P o n x , y , t − 1 , P o ff x , y , t = − P x , y , t − + α 2 P o ff x , y , t − 1 . (4) [x]⁺ and [x]⁻ denote max (0, x) and min (x, 0). In addition, a small fraction (α ₂) of previous signal is allowed to pass through.

The medulla layer is the place where different collision selectivity between LGMD-1 and LGMD-2 is shaped. The visual processing thus differs in this layer. First, in the LGMD-1’s medulla, the delayed information varies in different polarity pathways. More precisely, in the ON channels, the local inhibition (I _on1 ) is formed by convolving surrounding delayed excitations (E _on1 ). The whole spatiotemporal computation can be defined as the following: E o n 1 x , y , t = P o n x , y , t , (5) E ̂ o n 1 x , y , t = α 3 E o n 1 x , y , t + 1 − α 3 E o n 1 x , y , t − 1 , α 3 = τ i / τ 1 + τ i , (6) I o n 1 x , y , t = ∑ i = − r r ∑ j = − r r E ̂ o n 1 x + i , y + j , t W 1 i + r , j + r . (7) τ ₁ stands for the latency of excitatory signal. r indicates the radius of convolving area [W ₁] denotes the convolution kernel in the LGMD-1 that meets the following matrix: W 1 = 1 / 8 1 / 4 1 / 8 1 / 4 0 1 / 4 1 / 8 1 / 4 1 / 8 (8)

In the LGMD-1’s OFF channels, the delay is nevertheless put forth on the inhibitory signal; the excitation is thus formed by convolving delayed lateral inhibitions. That is, I o ff 1 x , y , t = P o ff x , y , t , (9) I ̂ o ff 1 x , y , t = α 4 I o ff 1 x , y , t + 1 − α 4 I o ff 1 x , y , t − 1 , α 4 = τ i / τ 2 + τ i , (10) E o ff 1 x , y , t = ∑ i = − r r ∑ j = − r r I ̂ o ff 1 x + i , y + j , t ⋅ W 2 i + r , j + r . (11) Here the convolution kernel [W ₂] is set equally to W ₁ in Eq. 8.

Second, in the LGMD-2’s medulla, much stronger local inhibitions are put forth in all the ON channels forming a biased-ON pathway in order to achieve its specific selectivity to only darker objects (see dashed lines in Figure 2). More specifically, the generation of local excitation (E _on2 ) and inhibition (I _on2 ) in the LGMD-2’s ON channels conforms to the LGMD-1 yet with a different latency τ ₃. To implement the ‘bias’, the convolution kernel matrix [W ₃] is increased with self-inhibition as W 3 = 1 / 4 1 / 2 1 / 4 1 / 2 2 1 / 2 1 / 4 1 / 2 1 / 4 (12)

In the LGMD-2’s OFF pathway, the neural computation is defined as E o ff 2 x , y , t = P o ff x , y , t , (13) E ̂ o ff 2 x , y , t = α 5 E o ff 2 x , y , t + 1 − α 5 E o ff 2 x , y , t − 1 , α 5 = τ i / τ 4 + τ i , (14) I o ff 2 x , y , t = ∑ i = − r r ∑ j = − r r E ̂ o ff 2 x + i , y + j , t W 4 i + r , j + r . (15)

[W ₄] fits the following matrix: W 4 = 1 / 8 1 / 4 1 / 8 1 / 4 1 1 / 4 1 / 8 1 / 4 1 / 8 (16)

As illustrated in Figure 2, following the generation of local ON/OFF excitation and inhibition, there are local summation units in either the medulla layer. For the LGMD-1, the calculation is as the following: S o n 1 x , y , t = E o n 1 x , y , t − w 1 ⋅ I o n 1 x , y , t + , S o ff 1 x , y , t = E o ff 1 x , y , t − w 2 ⋅ I o ff 1 x , y , t + . (17) w ₁ and w ₂ are the local biases. Note that only the positive S unit signals will pass through to the subsequent circuit. Compared to the LGMD-1, the two local biases are time varying, adjusted by the PM pathway in the LGMD-2. That is, w 3 t = max 1 , P M t T ffi , w 4 t = max 0.5 , P M t T ffi , (18) S o n 2 x , y , t = E o n 2 x , y , t − w 3 t ⋅ I o n 2 x , y , t + , S o ff 2 x , y , t = E o ff 2 x , y , t − w 4 t ⋅ I o ff 2 x , y , t + . (19)

In both the LGMDs, the polarity summation cells interact with each other in a supra-linear manner as S x , y , t = S o n x , y , t + S o ff x , y , t + S o n x , y , t S o ff x , y , t . (20)

Cascaded the S unit, a grouping unit is introduced to reduce isolated motion and enhance the extraction of expanding edges of colliding objects in cluttered backgrounds. This is implemented with a passing coefficient matrix [Ce] determined by another convolving process with an equally weighted kernel [W _g ]. That is, C e x , y , t = ∑ i = − r r ∑ j = − r r S x + i , y + j , t ⋅ W g i + r , j + r , (21) W g = 1 9 × 1 1 1 1 1 1 1 1 1 (22) G x , y , t = S x , y , t C e x , y , t ω t − 1 , where ω t = max C e t C ω − 1 + Δ C . (23) ω is a scale parameter computed at every discrete time step. C _ω is a constant. Δ _C stands for a small real number. Here only the non-negative grouped signals can get through.

After the signal processing of pre-synaptic neural networks, the LGMD-1 and LGMD-2 neurons integrate corresponding local excitations from the medulla layer to generate membrane potentials. Here we apply a sigmoid transformation as the neuron activation function. The whole process can be defined as k t = ∑ x = 1 R ∑ y = 1 C G x , y , t , K t = 1 + e − k t ⋅ C ⋅ R − 1 − 1 , (24)

Subsequently, a spike frequency adaptation mechanism is applied to sculpt the neural response to moving objects threatening collision. The computation is defined as follows: K ̂ t = α 6 K ̂ t − 1 + K t − K t − 1 , if K t − K t − 1 ≤ 0 α 6 K t , otherwise , (25) α 6 = τ s / τ s + τ i , (26) where α ₆ is a coefficient indicating the adaptation rate to visual movements calculated by the time constant τ _s . Generally speaking, such a mechanism reduces neuronal firing rate to stimuli with constant or decreasing intensity, e.g., objects recede or translate at a constant speed; while it has little effect on accelerating motion with increasing intensity like the approaching.

As the time interval between frames of digital signals is much longer than the reaction time of real visual neurons, we map the membrane potentials exponentially to spikes by an integer-valued function. That is, S s p i k e t = e α 7 ⋅ K ̂ t − T s p i , (27) where T _spi denotes the spiking threshold and α ₇ is a scale coefficient affecting the firing rate, i.e., raising it will bring about more spikes within a specified time window.

As illustrated in Figure 2, the elicited spikes are conveyed to their post-synaptic target neurons. Differently from previous modelling on single neuron computation of either the LGMD-1 or the LGMD-2, we herein put forward a non-linear hybrid spiking mechanism aiming at improving the selectivity to darker objects that only threaten direct collision by suppressing the response to other categories of visual stimuli. As a result, the specific selectivity of LGMD-2 well complements the LGMD-1’s where the hybrid spiking frequency will be amplified merely when both neurons are activated. The computation is defined as S h s p i k e = S 2 s p i k e , if F ̂ t ≥ T ffi S 1 s p i k e × S 2 s p i k e , otherwise (28)

Finally, the detection of potential collision threat can be indicated by C o l t = True , if ∑ i = t − n t t S h s p i k e i × 1000 / n t ⋅ τ i ≥ T c o l False , otherwise (29) n _t denotes a short time window in frames. T _col stands for the collision warning threshold.

Table 2 elucidates the parameters. In this study, we set up the parameters of neural networks depending on 1) prior knowledges from neuroscience (Rind and Bramwell, 1996; Simmons and Rind, 1997; Rind et al., 2016), 2) previous experience on modelling and experimenting of the LGMD-1 and the LGMD-2 neuron models (Fu et al., 2018b; Fu et al., 2019b), 3) considerations of fast implementation with optimisation as embedded vision systems for online visual processing (Hu et al., 2018). More concretely, the convolutional matrices W ₁, W ₂, W ₃, and W ₄ are not only based on previous biological and computational studies, but also optimised by “bitwise operation” on the embedded system. There is currently no feedback pathways and learning steps involved in the proposed hybrid neural networks. The parameters given in Table 2 have been systematically investigated in our previous bio-robotic studies with optimisation (Fu et al., 2018b; Fu et al., 2019b; Fu et al., 2017). In addition to that, the very limited computational resources in the micro-robot is restricted for online learning algorithms. Therefore, aligned with previous settings, the emphasis herein is laid on investigating the integration of both LGMDs inspired visual systems in robotic implementation of dynamic visual scenes.

Firstly, the multiple pheromones module was originally developed to conduct the swarm robotic experiments mimicking behaviours of social insects with interactions between multiple pheromones representing different biochemical substances (Sun et al., 2019). More specifically, as illustrated in Figure 3, the module consists of a camera system, a computer, and an arena with an LCD screen acting as the ground. The computer runs a pattern recognition algorithm in real time (Krajník et al., 2014), which is feasible to track and localise many ID-specific patterns with images at 1920 (pixels in width) × 1,080 (pixels in height) from the top-down facing camera, simultaneously, so as to record coordinates of robots with respect to time. In addition, the computer can render the virtual pheromone components, optically, represented by colour tracks or spots on the LCD screen indicating meaningful fields for mobile robots. Here the virtual pheromones are applied to render road maps and signals in the context. As shown in Figure 4, since the nature of pheromone field displayed on the LCD screen is a colour image, different traffic paradigms can be formed in which the roads are drawn by white tracks with boundaries, and the traffic lights and signals are embodied by green/red colour spots with appropriate size on the roads. Accordingly, different traffic sections like intersections, junctions, and even more complex road network can be established with scalability. Figure 4 shows some examples in our experiments. Together with periphery patterned walls (urban skyline), the arena is well constructed for our specific goal of simulating robotic traffic to test the proposed bio-inspired visual systems.

As illustrated in Figure 3, the autonomous mobile robot used in this study is called Colias-IV (Hu et al., 2018), which includes mainly two components that provide different functions, namely the Colias Basic Unit (CBU), and the Colias Sensing Unit (CSU).

The CBU serves preliminary robot features such as motion, power management and some basic sensing like the bumper infra-red (IR) sensors in Figure 3. The more detailed configuration can be found in our recent paper (Hu et al., 2018), which is not reiterated here. Specifically for the proposed tasks, the CBU is assembled with four colour sensors with high sensitivity on its bottom (see Figure 3). When the robot is running in the arena, these sensors can pick up optical pheromone information on the LCD screen, and then the robot behaviours are adjusted accordingly.

A key factor herein is tightly following the paths. We propose a control strategy that the two-side colour sensors are applied to bind the robot trajectory on the roads, as explained in Figure 5. Moreover, the front and rear light sensors play roles of recognising traffic signals including red (stop) and green (go) lights in the city traffic system, as well as accelerating and decelerating fields in the highway traffic system. More concrete control logic will be presented in the following Section 3.

The proposed LGMDs inspired visual systems are implemented for online visual processing. Here the CSU supports this where an ARM Cortex-M4F core micro controller is deployed as the main processor to handle intensive image processing. A monocular camera system with a low voltage CMOS image sensor OV7670 module is utilised in the CSU. With compact size, the camera is capable of operating up to 30 frames per second (fps) in VGA mode with output support for various colour formats. The horizontal viewing angle is approximately 70°. As a trade-off between processing efficiency and image quality, the resolution is configured at 72 × 99 pixels on 30 fps, with output format of 8-bit YUV422. Since the LGMDs only process grey-scale images, the camera setting fits it well, i.e., the proposed image format separates each pixel’s colour channels from the brightness channel; thus no additional colour transformation is required. More details of the CSU can be found in (Hu et al., 2018). Importantly, when assessing the proposed LGMDs inspired visual systems, the optical sensor is applied as the only collision detector.

Furthermore, the micro-robot can communicate with a host computer via a Bluetooth device connecting the CSU (Hu et al., 2018). Here we use it for retrieving the hybrid spiking frequency. With limited processing memory space and transmission ability, a current drawback of the robot is that it cannot send back real-time image views accompanied by motion.

In the city traffic system, we carry out systematic and comparative experiments involving several cases. In the first case, the robot traffic has no signal controls at intersections. We also look deeper into the density effect on collision prediction performance. Figure 11 illustrates the event and density maps of all micro-robot agents engaging in the ring road traffic for 1-h implementation.

In the second case, the red and green switching traffic lights are used as the auxiliary signals for robot flows control at intersections. An interesting episode is plotted in this case by mixing unruly robot agents not obeying the law of traffic signals, i.e., red to stop and green to go, that mimic the drivers who always break the traffic rules at intersections leading to immense on-road safety issue. Figure 12 illustrates the corresponding results at this point.

Together with the statistical results in Tables 3, T4, we have the following observations on the experiments of city traffic system:

1) Table 4 shows that more than half critical events take place at intersections in all the imitations of city ring road traffic (see EPro₁ in Table 4) indicating that our robot traffic could reflect real world road challenges (Colombo and Vecchio, 2012).

Generally speaking, the proposed bio-inspired hybrid neural networks work effectively and consistently on collision prediction in the city traffic system despite that the intersections are still posing challenges on timely detection-and-avoidance using the visual approach as the only modality. However, we believe this can be improved by increasing the view angle of optical sensor as the current view of frontal camera can only reach approximately 70°. The risk of intersection could also be alleviated by sensor fusion strategy, or other algorithms in intelligent transportation system (Colombo and Vecchio, 2012). With discrepancies amongst forward velocities of multi-robots (refer to the setting in Algorithm 1), the robot vehicles well demonstrate queueing effect guided by the collision prediction systems. On the straight and curving roads, the LGMDs inspired visual systems perform more robustly and consistently on collision alert in comparison with the intersections. Additionally, the robot density in the traffic system dose not greatly affect the overall performance of visual systems, which imply the proposed bio-inspired computation is robust and flexible to more dynamic visual scenes.

In the critical highway traffic system, two lanes separate the speed of robots into two ranges, as presented in Section 3. The overall CAR and PR are given in Tables 3, T4. In addition, Figure 13 illustrates the results with event and density maps. Here the most noticeable observation is that in comparison with the city traffic system barring the 10-agents case, more than twofold critical events take place in the highway traffic within the 1-h implementation. Table 4 clearly shows that nearly half (43.2%) critical events concentrate at the junction where the high-speed and low-speed robot flows merge. In spite of that, the overall CAR remains fairly high that is in consistent with the city traffic system without signals control (79.98%); the PR at either the junction (75.29%) or the other road sections (83.55%) is slightly lower than the city traffic results. In general, the LGMDs inspired visual systems are robust to cope with collision prediction in high-speed, dynamic visual scene, in the micro-robot under constrained computation cost.

On the other hand, we also find challenges through the experiments. The event density maps in Figure 13 demonstrate that it is still difficult to address the crash avoidance problems at the junction where the low-speed robot vehicles are accelerating to merge into the high-speed flow. At this point, the robots are required to form a queue to pass the junction free of collision. The similar situations happen at the deceleration zone where the high-speed vehicles are shunted to queue into the low-speed flow. In addition to that, compared to the city traffic results, the PR in other sections is relatively lower, i.e., more crashes between robots occur on the high-speed curving road (see Figure 13C).