Abstract
Autonomous flight for large aircraft appears to be within our reach. However, launching autonomous systems for everyday missions still requires an immense interdisciplinary research effort supported by pointed policies and funding. We believe that concerted endeavors in the fields of neuroscience, mathematics, sensor physics, robotics, and computer science are needed to address remaining crucial scientific challenges. In this paper, we argue for a bio-inspired approach to solve autonomous flying challenges, outline the frontier of sensing, data processing, and flight control within a neuromorphic paradigm, and chart directions of research needed to achieve operational capabilities comparable to those we observe in nature. One central problem of neuromorphic computing is learning. In biological systems, learning is achieved by adaptive and relativistic information acquisition characterized by near-continuous information retrieval with variable rates and sparsity. This results in both energy and computational resource savings being an inspiration for autonomous systems. We consider pertinent features of insect, bat and bird flight behavior as examples to address various vital aspects of autonomous flight. Insects exhibit sophisticated flight dynamics with comparatively reduced complexity of the brain. They represent excellent objects for the study of navigation and flight control. Bats and birds enable more complex models of attention and point to the importance of active sensing for conducting more complex missions. The implementation of neuromorphic paradigms for autonomous flight will require fundamental changes in both traditional hardware and software. We provide recommendations for sensor hardware and processing algorithm development to enable energy efficient and computationally effective flight control.
Introduction
Autonomous flying capability in aerospace is gathering momentum with the needs for enhanced safety, sustainability, and new missions. The latest developments in data science and machine learning have accelerated the relevance of autonomy for many areas, such as space operations, unmanned aerial vehicles (UAVs), (passenger) transport, manned-unmanned teaming, and air traffic management (; ). In future scenarios, coordination of a vast number of participants in contested air space is needed, which can be answered by autonomous and self-organizing vehicles only with unique sensing capabilities.
The latest (pilot-controlled) Airbus A350 has 50,000 sensors on board collecting 2.5 terabytes of data every day (). Processing the amount of sensory data needed for autonomous flight by current processing technologies requires high-performance computers consuming many kilowatts of energy. In contrast, a bird has two eyes, two ears, two vestibular organs, skin, several hundreds of olfactory receptors, 25–500 taste buds, and is capable of magneto reception (; ). Considering that the largest bird brains weigh 20 g, have 3 billion neurons and consume ∼0.2 W (), we need a new paradigm in bio-inspired sensors and processing, to overcome the computational and energy constraints required for autonomous operation in complex three-dimensional high-speed environments.
Neuromorphic engineering describes a large-scale system of integrated circuits that mimic neurobiological architectures present in the nervous system (; ; ). It aims to harness the efficiencies of biological brains by developing essential computation analogs of neuronal circuits. To explore the potential of neuromorphic engineering for aviation sensors, a virtual workshop on the topic of “Neuromorphic Sensing and Processing Paradigm” was organized by Airbus, U.S. Office of Naval research Global (ONRG) and the U.S. DEVCOM Army Research Laboratory (ARL) in July 2020.
This perspective paper is one output of the workshop and aims to describe the needs, current advances and provide directions for future research related to neuromorphic sensing and processing enablers for autonomous flight missions. The goal is to call upon the scientific community and funding agencies to enable and perform the required multidisciplinary research in this area.
Missions and Needs
Autonomous flight for passenger aircraft has the potential to deliver increased fuel savings, reduce operating costs, and allow pilots to focus on strategic decision-making and mission management rather than aircraft management. This is enabled by image-processing, decision-making support, speech processing and interpretation, and mutual human-machine trust. Urban air mobility (UAM) also requires new roles and capabilities, such as safe ground control of multiple autonomous pilot-less vehicles. Moreover, autonomy is extremely important for the next generation of defense and security air systems, where one aim is to connect manned and unmanned components in operations. Another (part of this) system can be a “swarm” of unmanned vehicles that independently coordinate to perform missions. This requires robust navigation in potentially obscure conditions, for potential missions such as reconnaissance, disaster management after catastrophes, search and rescue at sea or on ground, surveillance, automatic recognition of objects and/or people, border security, imaging, and optimized air traffic control.
General vital capabilities to achieve autonomous flight are attitude control (), height control and landing (; Baird et al., 2013), collision avoidance (; ), and navigating between places of interest (). For search and rescue and security missions, high-speed vehicles able to operate in complex environments require fast response times. On the other end are long-term (>1 day) missions with a great need for low energy consumption where swarms of flying autonomous vehicles could bring benefits in terms of cost, safety, and performance. A wide range of capabilities is required, such as communication channels, predictive tracking, control of own-drone space (situational awareness), mission-dependent information, visual/spectral tracking, navigation and control. For human-autonomy teaming missions, sharing of environmental sensory information, including temperature, humidity, airflow, and olfactory inputs (smoke, biohazards, explosives, etc.) is of major importance. For space missions, autonomous navigation and situational awareness are required in, for example, space-debris collision prevention. These exquisite capabilities can be enabled by a combination of multiple centralized and distributed sensors, such as observed in biology. The multi-domain missions as described above require simultaneous sensing modalities where neuromorphic sensors may need to be combined with conventional ones, such as hyperspectral imaging.
To enable sensing and efficient real-time processing for future missions and to address real world challenges, foundational advances in neuromorphic computing are needed at the theoretical, computational, algorithmic, and hardware implementation levels to overcome today’s scalability limit of e.g. finite state machines. For example, just as insects integrate information from vision and mechanoreceptors to instantaneously correct for changes in e.g., airspeed (), autonomous flying vehicles also require efficient and adaptive sensorimotor computations for instantaneous flight correction. Another challenge is navigating through complex dynamic environments with obstacle avoidance, such as flying through treetops with wind blowing through leaves and twigs. This requires luminance normalization (), coupled with contextual processing of higher order features, environmental and task-related visual statistics, as well as feedback and recurrent processing for visual prediction. Whereas in biology such normalization, contextualization, and prediction are interdependent processes, there still exists a major gap in computational implementation today.
An ultimate task of autonomy, the ability to deal with unexpected situations, requires the system to learn and act upon sensory inputs, which is the embodiment of cognition. This requirement represents another computational challenge that may be met by neuromorphic implementation. A robust computational framework must be developed to realize learning in autonomy that mimics, to some degree, even simple animals. The inability to provide training data for all scenarios poses a challenge for full autonomous flight to meet commercial, defense, security and space mission needs of the future. Given the vast area of research in neuromorphic processing, we only treat learning and decision-making in the context of sensing here in our discussion.
Current Advances in Neuromorphic Sensing
Neuromorphic sensing occurs through a device that perceives changes in a certain parameter and outputs them into a stream of events (“spikes”) (). Thus, essentially any sensing modality can be converted into an event-based (EB) detector. For example, pixels in EB cameras can signal brightness changes exceeding some threshold; microphones in EB audio sensors can react to sound intensity variations in certain frequency ranges (; ; ); EB chemical or olfactory sensor arrays or bio-hybrid sensors can fire as a result of chemical element concentration deviations (; ; ; Anderson et al., 2020); EB tactile sensors respond to changes in force and movement (; ).
The most prevalent example of EB sensors are video cameras (; ; ; ; ; ; ), which despite their relatively recent birth have already experienced significant evolution. Event based cameras have the necessary properties to become the foundational sensors for autonomous flight, solving the challenges of power, computing, and timing requirements and enabling local decision-making. In addition to being affordable and fast, they offer independence from lighting conditions and a large dynamic range. Event based sensors can process the equivalent of several hundred kHz frames using conventional adaptive CPU hardware for computations that are impossible to carry out with classic frame-based sensors. Examples are: real-time optical flow (), pose estimation (, ), time-based machine learning (), aperture free optical flow (), and many other applications that have been now developed using pure temporal mechanisms performed at almost real-time speeds.
Another neuromorphic approach to active sensing, includes the world’s first spiking neural network-based chip that was announced recently for radar signal processing. The first application was reported to encompass the creation of a low-power smart anti-collision radar system for drone collision avoidance; future plans are to process a variety of active sensor data including electrocardiogram, speech, sonar, radar and LIDAR streams (). Reportedly, the chip consumes 100 times less power than traditional implementations and provides 10X reduction in latency.
The algorithmic and hardware transitions to EB sensing platforms are driven by the desire to reduce latency, to achieve orders of magnitude improvement in energy efficiency, dynamic range, and sensitivity, to solve complex control problems with limited computing resources and to attain autonomous system’s capability of adapting to operation in unpredictable dynamic environments. Hence recently, they have been applied successfully in space surveillance applications (Roffe et al., 2021) and for controlled landing of micro-air vehicles (Dupeyroux et al., 2020). However, despite the progress achieved in the last decade by state-of-the-art neuromorphic sensors, there are several fundamental barriers separating them from real life applications. For example, visual EB sensors have limited ability to handle high focal plane array utilization due to complex illumination or clutter as well as pixel response inhomogeneity. In terms of sensor data processing, a major current challenge is to develop spike neural network learning principles, concurrently advancing both the algorithms and hardware, to enable the disparate sensor data fusion inherent to biological sensing.
Bio-Inspiration From Flying Animals and Insects
To address the challenges of energy-efficient real-time processing of multiple sensing modalities, and the ability to deal with unexpected situations as described above, we can look toward flying animals and insects. For example, spatial navigation builds upon a network of cognitive functions. Animals that rely on active sensing present particularly powerful models to guide the implementation of cognitive functions in the design of autonomous navigation systems, as their actions reflect cognitive states and directly influence signals used to represent the environment, which, in turn, guide 3D movement. Echolocating bats, for example, transmit sonar cries, and use information carried by returning echoes to determine the 3D position, size, and other features of objects in the environment (Simmons and Vernon, 1971; ; ; Busnel and Fish, 1980; Nachtigall and Moore, 1988; Moss and Schnitzler, 1989; Thomas et al., 2004; Moss and Surlykke, 2010). Central to successful 3D navigation of autonomous vehicles in cluttered environments is spatial attention. Attention invokes mechanisms that allocate computational resources to selectively process and enhance relevant information from the environment (Broadbent, 1957). It has been demonstrated in bats that sonarguided attention drives state-dependent processing of echo returns in brain regions that play key roles in spatial navigation. Specifically, attention invokes the sharpening of auditory midbrain neurons that encode an object’s 3D location in egocentric coordinates and hippocampal place cells that encode an animal’s 3D location in allocentric coordinates (Yartsev and Ulanovsky, 2013; Kothari et al., 2018; Wohlgemuth et al., 2018). Moreover, bats adjust the directional aim and temporal patterning of echolocation signals to inspect objects in the environment, which reveals adaptive sonar signal design, tailored to the task at hand. Current sonar technologies do not yet implement adaptive signal transmission and processing, which may explain superior performance of animals over artificial devices. New technologies that incorporate more complete knowledge of animal echolocation systems pave the way for advanced 3D sonar-guided navigation of autonomous vehicles in dark and cluttered environments.
For barn owls, as efficient predators adapted for hunting rodents in extremely low light conditions, attention is also of key importance to prevent overload of information processing capacity and for stable behavioral control in face of multiple distractors in cluttered and noisy environments (Posner, 2016). Neurons in the optic tectum of barn owls respond preferentially to visual and auditory stimuli that break a regular pattern of their background (Zahar et al., 2018) or that are categorically stronger from competing stimuli (Mysore et al., 2011). Unique neural adaptation (Netser et al., 2011), circuit motifs (Mysore and Knudsen, 2012) and top down modulation (Winkowski and Knudsen, 2008) that facilitate the stimulus selection process at the neuronal level have been identified. Such insights on the mechanisms of the barn owl’s neural activity may teach us information-processing strategies that are efficient and behaviorally useful.
The barn owl’s intriguing capability to avoid obstacles in dark conditions seems to rely on spatial memory and strong sense of self position in the memorized map of space (Payne, 1971). Preliminary results have begun to reveal the neural representation of the owl’s location and direction in space and provide a promising avenue for new inspirations about autonomous aviation in extreme light conditions (Agarwal and Gutfreund, 2019).
Another entry point is to focus on insect brains that are known to be of lesser complexity and therefore more adapted to modeling. Insect brains are particularly adapted to resource-constrained scenarios as in the case of flying machines while showing amazing functionalities that allow them to perform complicated tasks with great ease such as (visual) collision avoidance, localization, communication, navigation, odor source localization and social interaction in unknown unpredictable environments (Smith, 2007; Schneider and Levine, 2014; Weir and Dickinson, 2015; Fu et al., 2019; Huang et al., 2019; Hu et al., 2020).
For example, Drosophila melanogaster, or fruit fly, has a small brain with ∼100,000 neurons that are highly efficient in sensory processing. Although having an extremely poor spatial resolution (only ∼800 pixels in each compound eye), the visual system of fruit flies and other type of flies are highly sensitive to movements (Borst et al., 2010; Borst, 2018), inspiring the development of EB cameras described earlier. Moreover, there are several downstream layers of neural circuits that compute and process optical flow (Weir and Dickinson, 2015; Mauss and Borst, 2020). This innate neural “computation” endows a fly with the abilities to detect threats (high-speed moving objects), avoid obstacles, control its flight course (Mauss and Borst, 2020), and estimate its orientation (Su et al., 2017), which is exactly what we need for our autonomous flying vehicles, without the need for more than rudimentary object recognition and classification, which are computationally expensive in today’s artificial neural network architectures. Other examples are represented by locusts and flies that can detect visual collisions by a special neural structure (Fu et al., 2019). Recently, it has been modified into bio-plausible neural models that were applied on autonomous mobile robots and also UAVs with constrained computational resources (Huang et al., 2019; Hu et al., 2020).
Outlook Toward Advancing Neuromorphic Applications
So, how can we translate nature’s amazing capabilities into autonomous flying vehicles with limited energy supply? For example, flying nano-drones (Ma et al., 2013; Karásek et al., 2018) mimicking capabilities of fruit flies will unlock novel opportunities, such as development of nano-drone swarms that can explore and monitor unknown indoor or outdoor environments (Brambilla et al., 2013; McGuire et al., 2019). Recently, a neuromorphic chip was used in the control loop of a flying drone able to perform smooth optical flow landings, like honeybees, which immediately illustrated the energy efficiency and speed promised for such neuromorphic applications (Dupeyroux et al., 2020). Computational principles of dynamic vision of fruit flies or other insects can be implemented together with EB cameras, and used in parallel with the slower and energetically demanding computer vision systems that are designed for object recognition. This way a UAV can detect obstacles or potential threats even before they are recognized while using low-powered passive vision. For larger aircraft, advances in neuromorphic computing could lead to improved sensing and sensory fusion, including real-world resilience and prediction, inspired by the role of biological recurrent networks in solving such challenges (Tang et al., 2018; Kubilius et al., 2019).
Autonomous flight with neuromorphic efficiency requires a cross-disciplinary effort, and EB temporal computation requires new thinking. One missing piece of the puzzle to create truly neuromorphic systems is the computational hardware. It is expected that such architecture will be extremely low power while allowing to truly operate in real-time at the native resolution of the sensor. An optimal solution would be to approach the problem by considering the timing of events and developing techniques where each event incrementally adds information to what has already been computed (). This idea of local computation is not new and has been described to be present in most real neural networks where the layers of neurons process information at their own independent time scales based on the received sensory data rather than relying upon any form of “global” clock or memory.
Potential advantages of neuromorphic computing for active vs. passive imaging (e.g., bat echo-location vs. owl vision) should also be explored. Active vision (e.g., dense LIDAR) can provide limited 3D sensing but is challenged by objects such as trees, poles, and traffic lights (), whereas passive imaging is preferred for maintaining stealth but is energetically expensive. Both areas of research have been dominated by convolutional approaches, and an open question is how to fuse active and passive sensors, including antennae, and their data for efficient, resilient, and adaptive sensory-decision-motor loops. Sensory information in biological brains is represented in the same way for all signals, there is occurrence and temporal correlation, no distinction between inputs, and a generic way of generating events triggered by the data. This saves a lot of time and energy and EB sensors could aid in emulating biology in this sense.
Top-down (internal guidance – mission driven) and bottom-up (externally driven) attention (Katsuki and Constantinidis, 2014) are the neural processes that may solve the bottleneck issue in sensory information processing. With these two types of attention mechanisms, our brain’s central executive unit is able to focus on mission-relevant sensory signals while maintaining flexibility in rapidly switching to other sensory signals that occur unexpectedly. Predictive coding might play a crucial role here because it generates and updates an internal representation (or mental model) of the environment, and attention is required only when a prediction error occurs, which causes the system to shift to a high-power mode.
An additional layer of complexity is presented by neuromorphic computing inspired by biological principles for learning, which is needed for adaptive, resilient, and resource-efficient distributed sensing and learning (i.e., by swarms and other sensors) (; ), e.g., of target signatures and harsh environmental conditions, and for assured low-bandwidth communication. Progress on these challenges would create a framework of foundational principles, e.g., for predicting patterns and decisions from complex dynamics.
We have identified a clear need to enhance understanding of neurosensory systems in nature’s flying creatures, which shall result in new and better mathematical models needed for autonomous flying vehicles, see Figure 1. The long-term goal is hardware and software design and prototyping for interacting autonomous vehicles. Our target is neuromorphic hardware that aims at mimicking the functions of neural cells in custom synthetic hardware that is analog, digital, and asynchronous in its nature of information processing and is vastly more energy-efficient and lighter than classical silicon circuitry. It is expected that such a neuromorphic technology will disrupt existing solutions and be a key enabler for real-time processing of different sensor modalities by lower cost, lower energy consumption, lower weight, adaptable to changing missions, while providing enhanced and resilient performance and saving human lives.
FIGURE 1
In Table 1, we have created an overview of the current challenges toward autonomous flight and how the biology of flying creatures can inspire us in the coming years to reach the desired milestones. To summarize our recommendations:
TABLE 1
| Desired functions | Current challenges | Bio-inspiration | Estimated timeline (years) |
| High-speed in complex environment (obstacle and collision avoidance, mission performance) | Dynamic range and sensitivity, response times, sensory fusion, multiple agents in congested space | Fruit fly innate flight control and survival capabilities, swarming | 5 (UAM) – 20 (combat systems) |
| Robust navigation | Obscuration and glare conditions, GPS denied environment | Visual and magnetoreceptive capabilities of flying animals, spatial memory | 5–10 |
| Increasing complexity of flight control, air/ground transition efficiency | Sensorimotor integration, translation between small and large platforms and different degrees of complexity | Resilience to wind gusts, innate landing/perching/take-off, differences in brain processing between insects and birds | 5–10 |
| Computing and sensing efficiency | Scalability, power, weight | Resource-limitation in biology (few neurons in small low-weight brain), sensory fusion | 10–15 |
| Multi-sensory awareness | Sensory fusion and energy consumption, reliable automated object recognition | No distinction between neurons signalling in different sensory processing systems, learning, attention and recognition | 5–15 |
| Cognition/adaptability | Deep understanding of brain learning mechanism | Learning, attention, decision and reward systems | 30+ |
Neuromorphic sensing for autonomous capabilities – roadmap.
Please note that the mentioned challenges involve a mix of both algorithms and hardware.
- 1.
Develop EB neuromorphic sensor hardware and processing algorithms to support resilient and efficient navigation and collision avoidance
- 2.
Develop computationally efficient flight control with fast sensor-to-actuator responses to support resilience
- 3.
Develop neuromorphic attentional, sensory fusion, and representational mechanisms to increase efficiency and goal-directed performance in complex scenarios
- 4.
Develop neuromorphic approaches to learning for adaptive and predictive sensing and control
- 5.
Develop principles to integrate neuromorphic and convolutional approaches to harness their mutual advantages
Disclaimer
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the DEVCOM Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Statements
Data availability statement
The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author/s.
Author contributions
PP created the outline and wrote introduction, collected inputs, and integrated them in the full manuscript. AK and CH wrote sections and reviewed manuscript. YG, C-CL, RB, CM, and GC reviewed the manuscript and contributed sections that were integrated in the manuscript by PP. AS and FG reviewed the manuscript. All authors contributed to the article and approved the submitted version.
Funding
The barn owl research is partly funded by grants from the Israel Science Foundation (ISF) and the Rappaport foundation for Biomedical research. C-CL is partially supported by the Ministry of Science and Technology (Taiwan) grant number 109-2218-E-007-021. NSF Brain Initiative [NCS-FO 1734744 (2017-2021)], Air Force Office of Scientific Research (FA9550-14-1-0398NIFTI), and Office of Naval Research (N00014-17-1-2736) grants to CM supported research reported in this review.
Conflict of interest
PP and AS are employed by the company Airbus Defence and Space GmbH. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
Airbus (2020a). Autonomous Flight – Towards a World of More Autonomous Air Travel. Available online at: https://www.airbus.com/innovation/autonomous-and-connected/autonomous-flight.html,, [accessed on Oct 5, th Oct 2020].
2
National Research Council [NRC] (2014). Autonomy Research for Civil Aviation: Toward a New Era of Flight.Washington, DC: The National Academies Press, 10.17226/18815
3
Ornithology.com (2020). The Nervous System and Senses. Available online at: https://ornithology.com/ornithology-lectures/nervous-system-senses/#:~:text=A%20bird’s%20skin%20possess%20sensory,birds%20have%20concentrated%20nerve%20endings, [accessed on 11 Nov 11, 2020].
4
Airbus (2020b). Data Revolution in Aviation. Available online at: https://www.airbus.com/public-affairs/brussels/our-topics/innovation/data-revolution-in-aviation.html, [accessed on 11 Nov 11, 2020].
5
OlkowiczS.KocourekM.LuèanR. K.PortešM.FitchW. T.Herculano-HouzelS.et al (2016). Birds have primate-like numbers of neurons in the forebrain.PNAS1137255–7260. 10.1073/pnas.1517131113
6
MorabitoF. C.AndreouA. G.ChiccaE. (2013). Editorial – Neuromorphic Engineering: from neural systems to brain-like engineered systems.Neural. Networks451–3. 10.1016/j.neunet.2013.07.001
7
MaZ.KringsA. W. (2009). Insect sensory systems inspired computing and communications.Ad. Hoc. Networks7742–755. 10.1016/j.adhoc.2008.03.003
8
ZhangW.GaoB.TangJ.YaoP.YuS.ChangM.-F.et al (2020). Neuro-inspired computing chips.Nat. Electr.3371–382. 10.1038/s41928-020-0435-7
9
LichtsteinerP.PoschC.DelbruckT. (2008). A 128x128 120 dB 15μs Latency Asynchronous Temporal Contrast Vision Sensor.IEEE J. Solid State Cir.43566–576. 10.1109/JSSC.2007.914337
10
PoschC.MatolinD.WohlgenanntR. (2011). A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor With Lossless Pixel-Level Video Compression and Time-Domain CDS.IEEE J. Solid State Cir.46259–275. 10.1109/JSSC.2010.2085952
11
BrandliC.BernerR.YangM.LiuS.DelbruckT. (2014). A 240 × 180 130 dB 3 μs Latency Global Shutter Spatiotemporal Vision Sensor.IEEE J. Solid State Cir.492333–2341. 10.1109/JSSC.2014.2342715
12
TaverniG.MoeysD. P.LiC. (2018). Front and Back Illuminated Dynamic and Active Pixel Vision Sensors Comparison.IEEE Trans. Cir. Syst. II65677–681. 10.1109/TCSII.2018.2824899
13
ClarkeP. (2020). Swiss Event-Based Vision Startup Launches Next-Gen Chip. Available online at: https://www.eenewsanalog.com/news/swiss-event-based-vision-startup-launches-next-gen-chip, [accessed on Aug 21, 2019Oct 2020]
14
SonB.SuhY.JungH.KimJ.-S. (2017). “A 640×480 dynamic vision sensor with a 9μm pixel and 300Meps address-event representation,” in 2017 IEEE International Solid-State Circuits Conference (ISSCC), (San Francisco: IEEE), 66–67. 10.1109/ISSCC.2017.7870263
15
ChenS.TangW.ZhangX. (2012). “A 64x 64 Pixels UWB Wireless Temporal-Difference Digital Image Sensor,” in IEEE Transactions on Very Large Scale Integration (VLSI) Systems, (San Francisco: IEEE), 2232–2240. 10.1109/TVLSI.2011.2172470
16
LiC.DelbruckT.LiuS. (2012). “Real-time speaker identification using the AEREAR2 event-based silicon cochlea,” in 2012 IEEE International Symposium on Circuits and Systems (ISCAS), Seoul, (San Francisco: IEEE), 1159–1162. 10.1109/ISCAS.2012.6271438
17
LiuS.van SchaikA.MinchB. A.DelbruckT. (2014). Asynchronous Binaural Spatial Audition Sensor With 2x64x4 Channel Output.IEEE Trans. Biomed. Cir. Syst.8453–464. 10.1109/TBCAS.2013.2281834
18
Inilabs.com (2020). Dynamic Audio Sensor. URL: https://inilabs.com/products/dynamic-audio-sensor/. Accessed Oct 2020
19
KoickalT. J.HamiltonA.TanS. L.CovingtonJ. A.GardnerJ. W. (2007). “Analog VLSI circuit implementation of an adaptive neuromorphic olfaction chip,” in IEEE Transactions on Circuits and Systems I: Regular Papers, (San Francisco: IEEE), Vol. 5460–73. 10.1109/TCSI.2006.888677
20
ChiuS.-W.TangK.-T. (2013). Towards a chemiresistive sensor-integrated electronic nose: a review.Sensors1314214–14247. 10.3390/s131014214
21
VanarseA.OsseiranA.RassauA. (2017). An Investigation into Spike-Based Neuromorphic Approaches for Artificial Olfactory Systems.Sensors17:2591. 10.3390/s17112591
22
Baghaei NaeiniF.AlAliA. M.Al-HusariR.RigiA. (2020). A Novel Dynamic-Vision-Based Approach for Tactile Sensing Applications.IEEE Trans. Instrum. Measur.691881–1893. 10.1109/TIM.2019.2919354
23
KimY.ChortosA.XuW.LiuY.OhJ. Y.SonD.et al (2018). A bioinspired flexible organic artificial afferent nerve.Science360998–1003. 10.1126/science.aao0098
24
LiuJ. (2020). Press Release: IMEC Builds World’s First Spiking Neural Network-Based Chip for Radar Signal Processing. Available online at: https://www.imec-int.com/en/articles/imec-builds-world-s-first-spiking-neural-network-based-chip-for-radar-signal-processing, accessed Oct 2020[accessed on April 28, 2020].
25
Buckminster FullerS.StrawA. D.PeekM. Y.MurrayR. M.DickinsonM. H. (2014). Flying Drosophila stabilise vision-based velocity controller by sensing wind with their antennae.PNAS111E1182–E1191. 10.1073/pnas.1323529111
26
HungC. P.Callahan-FlintoftC.FedeleP. D.FluittK. F.OdoemeneO.WalkerA. J.et al (2020). Abrupt darkening under high dynamic range (HDR) luminance invokes facilitation for high-contrast targets and grouping by luminance similarity.J. Vis.201–16. 10.1167/jov.20.7.9
27
ChauhanI.BrennerC.GargR. D.ParidaM. (2014). A new approach to 3D dense LiDAR data classification in urban environment.J. Ind. Soc. Remote Sens.42673–678. 10.1007/s12524-013-0354-4
28
AbdelzaherT.AyanianN.BasarT. (2018). Toward an internet of battlefield things: a resilience perspective.Computer5124–36. 10.1109/MC.2018.2876048
29
ThomasA. L. R. (1997). On the tails of birds.BioScience47215–225. 10.2307/1313075
30
WiltschoR.WiltschoW. (2019). Magnetoreception in birds.J. R. Soc. Interf.16:20190295. https://royalsocietypublishing.org/doi/doi: 10.1098/rsif.2019.0295. 10.1098/rsif.2019.0295
31
BenosmanR. B.ClercqC.LagorceX.IengS. H.BartolozziC. (2014). Event-based visual flow.IEEE Trans. Neural Netw. Learn. Syst.25407–417. 10.1109/tnnls.2013.2273537
32
Reverter ValeirasD.OrchardG.IengS. H.BenosmanR. B. (2018). Neuromorphic event-based 3d pose estimation.Front. Neurosci.9:522. 10.3389/fnins.2015.00522
33
Reverter ValeirasD.KimeS.IengS. H.BenosmanR. B. (2016). An event-based solution to the perspective-n-point problem.Front. Neurosci.10:208. 10.3389/fnins.2016.00208
34
LagorceX.OrchardG.GalluppiF.ShiB. E.BenosmanR. B. (2017). Hots: a hierarchy of event-based time-surfaces for pattern recognition.IEEE Trans. Patt. Analys. Mach. Intell.391346–1359. 10.1109/tpami.2016.2574707
35
AkolkarH.IengS. H.BenosmanR. B. (2020). Real-time high speed motion prediction using fast aperture-robust event-driven visual flow.arXiv preprint arXiv:1811.11135 [preprint].
36
GoulardR.VercherJ. L.ViolletS. (2016). To crash or not to crash: how do hoverflies cope with free-fall situations and weightlessness?J. Exp. Biol.2192497–2503. 10.1242/jeb.141150
37
BairdE.BoeddekerN.IbbotsonM. R.SrinivasanM. V. (2013). universal strategy for visually guided landing.Proc. Natl. Acad. Sci. U. S. A.11018686–18691. 10.1073/pnas.1314311110
38
SrinivasanM. V.ZhangS. W.ChahlJ. S.BarthE.VenkateshS. (2000). How honeybees make grazing landings on flat surfaces.Biol. Cyber.83171–183. 10.1007/s004220000162
39
RindF. C.WernitznigS.PöltP.ZankelA.GütlD.SztarkerJ.et al (2016). Two identified looming detectors in the locust: ubiquitous lateral connections among their inputs contribute to selective responses to looming objects.Scient. Rep.6:35525.
40
TammeroL. F.DickinsonM. H. (2002). Collision-avoidance and landing responses are mediated by separate pathways in the fruit fly, Drosophila melanogaster.J. Exp. Biol.2052785–2798.
41
WebbB. (2019). The internal maps of insects.J. Exp. Biol.222:jeb188094. 10.1242/jeb.188094
42
GriffinD. R. (1974). Listening in the Dark: The Acoustic Orientation of Bats and Men.USA: Dover Publications.
43
SimmonsJ. A. (1973). The Resolution of Target Range by Echolocating Bats.J. Acoust. Soc. Am.54157–173. 10.1121/1.1913559
44
SimmonsJ. A.VernonJ. A. (1971). Echolocation: discrimination of Targets by the Bat, Eptesicus Fuscus.J. Exp. Zool.176315–328. 10.1002/jez.1401760307
45
BusnelR. G.FishJ. F. (eds) (1980). General Bibliography. Animal Sonar Systems.New York: Plenum Publishing.
46
NachtigallP. E.MooreP. W. B. (eds) (1988). Animal Sonar: Processes and Performance.Boston, MA: Springer.
47
MossC. F.SchnitzlerH. U. (1989). Accuracy of Target Ranging in Echolocating Bats: acoustic Information Processing.J. Comp. Physiol. A165383–393. 10.1007/bf00619357
48
ThomasJ. A.MossC. F.VaterM. (eds) (2004). Echolocation in Bats and Dolphins.Chicago: University of Chicago Press.
49
MossC. F.SurlykkeA. (2010). Probing the Natural Scene by Echolocation in Bats.Front. Behav. Neurosci.4:33. 10.3389/fnbeh.2010.00033
50
KothariN. B.WohlgemuthM. J.MossC. F. (2018). Dynamic Representation of 3D Auditory Space in the Midbrain of the Free-Flying Echolocating Bat.eLife7:e29053. 10.7554/eLife.29053
51
WohlgemuthM. J.YuC.MossC. F. (2018). 3D Hippocampal Place Field Dynamics in Free-Flying Echolocating Bats.Front. Cell. Neurosci.12:270. 10.3389/fncel.2018.00270
52
YartsevM. M.UlanovskyN. (2013). Representation of Three-Dimensional Space in the Hippocampus of Flying Bats.Science.340367–72. 10.1126/science.1235338
53
BorstA.HaagJ.ReiffD. F. (2010). Fly Motion Vision.Ann. Rev. Neurosci.3349–70.
54
BorstA. (2018). Biophysical mechanism for preferred direction enhancement in fly motion vision.PLoS Comput. Biol.14:e1006240. 10.1371/journal.pcbi.1006240
55
MaussA. S.BorstA. (2020). Optic flow-based course control in insects.Curr. Opin. Neurobiol.6021–27. 10.1016/j.conb.2019.10.007
56
WeirP. T.DickinsonM. H. (2015). Functional divisions for visual processing in the central brain of flying Drosophila.Proc. Natl. Acad. Sci. U. S. A.112E5523–E5532.
57
SuT. S.LeeW. J.HuangY. C.WangC. T.LoC. C. (2017). Coupled symmetric and asymmetric circuits underlying spatial orientation in fruit flies.Nat. Commun.8:139.
58
PosnerM. I. (2016). Orienting of attention: then and now.Q. J. Exp. Psychol.691864–1875. 10.1080/17470218.2014.937446
59
NetserS.ZaharY.GutfreundY. (2011). Stimulus-specific adaptation: can it be a neural correlate of behavioral habituation?J. Neurosci.3117811–17820. 10.1523/jneurosci.4790-11.2011
60
MysoreS. P.KnudsenE. I. (2012). Reciprocal inhibition of inhibition: a circuit motif for flexible categorization in stimulus selection.Neuron73193–205. 10.1016/j.neuron.2011.10.037
61
WinkowskiD. E.KnudsenE. I. (2008). Distinct mechanisms for top-down control of neural gain and sensitivity in the owl optic tectum.Neuron60698–708. 10.1016/j.neuron.2008.09.013
62
SmithD. P. (2007). Odor and pheromone detection in Drosophila melanogaster.Pflug. Arch. Eur. J. Physiol.454749–758. 10.1007/s00424-006-0190-2
63
SchneiderJ.LevineJ. D. (2014). Automated identification of social interaction criteria in Drosophila melanogaster.Biol. Lett.10:20140749. 10.1098/rsbl.2014.0749
64
ZaharY.Lev-AriT.WagnerH.GutfreundY. (2018). Behavioral Evidence and Neural Correlates of Perceptual Grouping by Motion in the Barn Owl.J. Neurosci.386653–6664. 10.1523/jneurosci.0174-18.2018
65
MysoreS. P.AsadollahiA.KnudsenE. I. (2011). Signaling of the strongest stimulus in the owl optic tectum.J. Neurosci.315186–5196. 10.1523/jneurosci.4592-10.2011
66
PayneR. S. (1971). Acoustic location of prey by barn owls (Tyto alba).J. Exp. Biol.54535–573.
67
AgarwalA.GutfreundY. (2019). “Space and flight-direction representation in the dorsal pallium of barn owls,” in Neuroscience Meeting Planner, Chicago, Il: Society for Neuroscience.
68
MaK. Y.ChirarattananonP.FullerS. B.WoodR. J. (2013). Controlled flight of a biologically inspired, insect-scale robot.Science340603–607. 10.1126/science.1231806
69
KarásekM.MuijresF. T.de WagterC.RemesB. D.de CroonG. C. (2018). tailless aerial robotic flapper reveals that flies use torque coupling in rapid banked turns.Science3611089–1094. 10.1126/science.aat0350
70
McGuireK. N.de WagterC.TuylsK.KappenH. J.de CroonG. C. (2019). Minimal navigation solution for a swarm of tiny flying robots to explore an unknown environment.Sci. Rob.4:eaaw9710. 10.1126/scirobotics.aaw9710
71
DupeyrouxJ.HagenaarsJ.Paredes-VallésF.de CroonG. C. (2020). Neuromorphic control for optic-flow-based landings of MAVs using the Loihi processor.arXiv preprint arXiv:2011.00534. [preprint]
72
BrambillaM.FerranteE.BirattariM.DorigoM. (2013). Swarm robotics: a review from the swarm engineering perspective.Swarm Intell.71–41. 10.1007/s11721-012-0075-2
73
AndersonM. J.SullivanJ. G.HoriuchiT. K.FullerS. B.DanielT. L. (2020). A bio-hybrid odor-guided autonomous pal-sized air vehicle.Bioinspir. Biomimet.10.1088/1748-3190/abbd81 [Epub online ahead of print].
74
KatsukiF.ConstantinidisC. (2014). Bottom-up and top-down attention: different processes and overlapping neural systems.Neuroscientist20509–521. 10.1177/1073858413514136
75
FuQ.HuC.PengJ.RindF. C.YueS. (2019). robust collision perception visual neural network with specific selectivity to darker objects.IEEE Trans. Cybernet.505074–5088. 10.1109/tcyb.2019.2946090
76
HuC.XiongC.PengJ.YueS. (2020). Coping With Multiple Visual Motion Cues Under Extremely Constrained Computation Power of Micro Autonomous Robots.IEEE Access8159050–159066. 10.1109/access.2020.3016893
77
HuangJ.WeiY.KrappH. (2019). A biohybrid fly-robot interface system that performs active collision avoidance.Bioinspir. Biomimetics.14:065001. 10.1088/1748-3190/ab3b23
78
KubiliusJ.SchrimpfM.KarK.HongH.MajajN. J.RajalinghamR.et al (2019). Brain-like object recognition with high-performing shallow recurrent ANNs.arXiv preprint arXiv:1909.06161 [preprint].
79
TangH.SchrimpfM.LotterW.MoermanC.ParedesA.CaroJ. O.et al (2018). Recurrent computations for visual pattern completion.Proc. Natl. Acad. Sci. U. S. A.1158835–8840. 10.1073/pnas.1719397115
80
RoffeS.AkolkarH.GeorgeA. D.Linares-BarrancoB.BenosmanR. (2021). Neutron-Induced, Single-Event Effects on Neuromorphic Event-based Vision Sensor: a First Step Towards Space Applications.arXiv preprint arXiv [preprint]. https://arxiv.org/abs/2102.00112.
81
BroadbentD. E. (1957). Mechanical Model for Human Attention and Immediate Memory.Psychol. Rev.64205–215. 10.1037/h0047313
Summary
Keywords
neuromorphic sensing, autonomous flight, bio-inspiration, flying animals, learning, flight control, energy efficiency
Citation
Parlevliet PP, Kanaev A, Hung CP, Schweiger A, Gregory FD, Benosman R, de Croon GCHE, Gutfreund Y, Lo C-C and Moss CF (2021) Autonomous Flying With Neuromorphic Sensing. Front. Neurosci. 15:672161. doi: 10.3389/fnins.2021.672161
Received
25 February 2021
Accepted
07 April 2021
Published
14 May 2021
Volume
15 - 2021
Edited by
Michael Schmuker, University of Hertfordshire, United Kingdom
Reviewed by
Cheng Hu, Guangzhou University, China; Shizuko Hiryu, Doshisha University, Japan; Luca Patanè, University of Messina, Italy
Updates
Copyright
© 2021 Parlevliet, Schweiger, Benosman, de Croon, Gutfreund, Lo and Moss.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Patricia P. Parlevliet, patricia.parlevliet@airbus.com
This article was submitted to Neuromorphic Engineering, a section of the journal Frontiers in Neuroscience
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.