A Review of Current Neuromorphic Approaches for Vision, Auditory, and Olfactory Sensors

Conventional vision, auditory, and olfactory sensors generate large volumes of redundant data and as a result tend to consume excessive power. To address these shortcomings, neuromorphic sensors have been developed. These sensors mimic the neuro-biological architecture of sensory organs using aVLSI (analog Very Large Scale Integration) and generate asynchronous spiking output that represents sensing information in ways that are similar to neural signals. This allows for much lower power consumption due to an ability to extract useful sensory information from sparse captured data. The foundation for research in neuromorphic sensors was laid more than two decades ago, but recent developments in understanding of biological sensing and advanced electronics, have stimulated research on sophisticated neuromorphic sensors that provide numerous advantages over conventional sensors. In this paper, we review the current state-of-the-art in neuromorphic implementation of vision, auditory, and olfactory sensors and identify key contributions across these fields. Bringing together these key contributions we suggest a future research direction for further development of the neuromorphic sensing field.


INTRODUCTION
The field of neuromorphic engineering has been developing rapidly over the last decade. With the growing trend toward embedding intelligence in day-to-day devices, we are constantly making our surroundings smarter and more adaptive to our behavior. However, this technological progression requires an ever increasing number of sensors and associated data storage (Tenore and Etienne-Cummings, 2011). Along with the data processing challenges, factors such as power consumption and financial viability limit the development of smart devices. Realizing these limitations in the late 1980s, Carver Mead introduced the concept of neuromorphic engineering. This interdisciplinary field addresses the underlying concepts of neurobiological architecture and mimics its implementation using aVLSI. Neurobiological architecture is a low power consuming system which learns through exposure; these attributes, along with sparse output are crucial design criteria for neuromorphic systems (Mead, 1990).
Neuromorphic approaches have been applied in implementing neural processors, developing neural networks, and particularly in electronic sensing where novel methodologies have been developed (Chicca et al., 2014). In the mid 1980s, Max Delbrück, John Hopfield, Carver Mead, and Richard Feynman collaborated to exploit the non-linear current characteristics of transistors (Indiveri and Horiuchi, 2011). Carver Mead, further highlighted the excessive dissipation of energy through conventional computing methods and the limitation of using transistors merely as digital switching components. He proposed that the analog physical properties of transistors could be exploited to design adaptive and low power consuming sensors like silicon retina and cochlea (Mead, 1990). With these models as inspiration, neuromorphic concepts have been applied to vision sensors, auditory sensors, and olfactory sensors. Current sensor advances have been supported by massive parallelism, asynchronous processing and self-organization (Douglas et al., 1995;Liu et al., 2015).
As described in Hasler and Marr (2013), analog implementation of neural-like systems are capable of approaching equivalence to biological systems in terms of power consumption and efficiency. Most of the research in neuromorphic sensors has involved aVLSI; however, with rapidly changing technology, digital electronics has also been applied to implement neuromorphic concepts, particularly as it has proved to be robust to internal and external noise (Sarpeshkar, 1998). Systems implemented using digital electronics are easily programmed and upscaled (Sarpeshkar, 2006). Regardless of whether analog or digital implementations are used, the lack of standards and benchmarks for the output of neuromorphic sensors may limit their development and adoption. In the same way that the interfacing for neuromorphic sensors uses standard Address Event Representation (AER; Boahen, 2000), a standardized method to evaluate sensor outputs could help establish appropriate benchmarks for further improvement. In this paper we review significant recent contributions to neuromorphic vision, auditory, and olfactory sensing and compare them to identify potential benchmarks for neuromorphic sensors.

NEUROMORPHIC VISION SENSORS
Neuromorphic engineering concepts have been successfully implemented to emulate biological sensory systems with silicon retinas being a prominent example (Tenore and Etienne-Cummings, 2011). Currently, vision sensing depends on the conventional frame-based approach but regardless of whether the scene changes, these frames are captured continuously and this generates significant volumes of redundant data (Lichtsteiner et al., 2008;Brandli et al., 2014). However, reducing the frame capture rate may cause excessive information loss between consecutive frames, particularly for real-time applications such as machine vision and robotics. Such frame-based approaches also consume substantial power and make data management challenging (Posch et al., 2014). Attempts were made to control the data output from these sensors by relaying information only for changed values of pixels. However, off-sensor processing and complex control strategies increased the overall power consumption of the system (Lichtsteiner et al., 2008).
Mahowald and Mead implemented the first silicon retina model in Mahowald and Mead (1991), that was both adaptive and energy efficient, by emulating retina functionalities, especially the cone cells, through analog properties of transistors and introducing adaptive vision sensing., improved this model by adding the functionalities of inner retina and parvo-magno cells (Zaghloul and Boahen, 2004a,b). These attempts could only model the retina in silicon, however, and did not provide a realistic implementation for practical use. In response, the neuromorphic community focused on the operating principle of the neurobiological architecture rather than modeling the overall sensory system. Specifically, this problem could be solved by realizing the difference between temporal and spatial contrasts. With the developments in AER, pixels could operate individually as processing units and report any deviations in temporal contrast. The spiking output is similar to the action potentials generated by ganglion cells and consequently most retinomorphic sensors now use AER communication (Posch et al., 2014).
Tobi Delbruck built on the idea of adaptive photoreceptor circuits developed in Delbruck and Mead (1994) and introduced strategies for enhancing retinomorphic sensors. The 128 × 128 pixel Dynamic Vision Sensor (DVS) can be considered as the product of several improvements through (Lichtsteiner et al., 2004(Lichtsteiner et al., , 2006Lichtsteiner and Delbruck, 2005) where the concepts of differentiating ON/OFF events with respect to lumosity change and relative lumosity change were implemented. DVS established a benchmark in neuromorphic vision sensing with its AER-based approach in which each individual pixel processed the normalized time derivative of the sensed light and provided an output in the form of spikes of the pixel addresses that detect lumosity change. As an alternative approach, frame-based temporal detection imagers (Gottardi et al., 2009;Cottini et al., 2013) were developed. The operating principle of these imagers was based on integration of the photocurrent between successive frames and computing of the difference between them. However these implementations have a limited speed response and a low-dynamic range, 100 dB for Gottardi et al. (2009) and 52 dB for Cottini et al. (2013). With features such as sub-millisecond precision, dynamic range >120 dB, and low power consumption of 23 mW, DVS was a path-breaking discovery and was used in various robotic and real-time systems (Rüedi et al., 2003;Drazen et al., 2011;Brandli et al., 2013).
With DVS, a benchmark was established for essential characteristics that a neuromorphic vision sensor should possess and gave a clear direction for further research in vision sensing. Developing on the basic idea of DVS, (Serrano-Gotarredona and Linares-Barranco, 2013) enhanced the capabilities of DVS by improving the contrast sensitivity by one order of magnitude (down to 1.5%) and reducing power consumption to 4 mW and fixed pattern noise to 0.9% and thus the overall pixel size of the sensor down to 30 × 31 µm 2 per pixel. The QVGA (304 × 240 pixel) ATIS (Asynchronous Time-based Image Sensor) by Posch et al. (2011), implemented PWM (Pulse Width Modulation) based intensity readout that improved the dynamic range (143-125 dB) at the cost of increased pixel area of 30 × 30 µm 2 . The inclusion of DVS pixel and PWM intensity readout triples the sensor output data. Brandli et al. (2014) proposed a hybrid approach between frame-based and framefree vision sensing. The 240 × 180 pixel DAVIS (Dynamic and Active-pixel Vision Sensor) arose in part from contributions by Berner et al. (2013), in which the concept of integrating a synchronous active-pixel sensor with asynchronous DVS pixels was implemented.
The successful implementation of DAVIS set another benchmark that will inspire future work toward neuromorphic vision sensors that provides spatial details of static scenes while also responding to dynamic temporal changes with minimum latency. This will also drive efforts to improve the fill factor and dynamic range.

NEUROMORPHIC AUDITORY SENSORS
The conventional method of sensing auditory signals is sampling the data continuously for auditory input at an application specific Nyquist frequency. This data undergoes Analog-to-Digital Conversion (ADC) and further digital processing to generate auditory frames. There is a significant power cost for high resolution ADC and digital processing of auditory frames. Although, this sampling rate can be altered dynamically to reduce power consumption, there is a risk of losing critical information due to low sampling rate. For applications such as auditory scene analysis, it is necessary that such sensors use less power and generate sparse data .
Lyon and Mead, proposed an auditory sensor in Lyon and Mead (1988) that models the human cochlea using aVLSI. This work addressed key concepts like automatic gain control, the use of cascaded second-order resonant lowpass filters and the necessary quality factors for auditory applications such as delay, high-gain pseudoresonance, and sharp roll-off. Although, these researchers implemented the bio-physics of both outer and inner hair cells of the basilar membrane, this analog cochlea model did not incorporate any biasing circuits for process, voltage, and temperature variations. However, it was further improved by addressing issues such as device mismatch, stability, and dynamic range (Watts et al., 1992). By introducing "overlapping cochlear cascades" , established a novel approach to the design of an aVLSI cochlea with dynamic range of 61 dB that consumes 0.5 mW. These early works underpinned further studies in silicon cochlea design.
Crucially, the efforts in silicon cochlea research led to the development of an auditory processor for cochlear implants that operated on minimal power. Sarpeshkar and his colleagues extending their work in Wang et al. (1997) and , and taking inspiration from other contributions Salthouse and Sarpeshkar, 2003), developed an ultra-low power auditory processor for a bionic ear. This processor can theoretically operate on a 100 mAh rechargeable battery for several years and features automatic gain control and microphone pre-amplifier audio front end, so that the processor converts the input signal to the desired dynamic range. The digital output of the processor ensures its independence from voltage and temperature variations. It operates over 16 channels that are comprised of independently programmable bandpass filters. Sarpeshkar et al. (2005) claimed that such processors can be applied in systems needing low-powered portable speech recognition front ends.
Along with cochlear implants, auditory scene analysis is a crucial application for the silicon cochlea. The implementation of AER to communicate output spikes stimulated further research on developing a spatial auditory sensor. Building on the silicon cochlea described by Van Schaik et al. (1996) and a neuromorphic front end MEMS (Micro-Electro-Mechanical Systems) microphone explained in Van Schaik and Shamma (2004), Chan et al. (2007) developed AER EAR, a matched pair of silicon cochlea with an AER interface. This auditory sensor models the basilar membrane bio-physics by cascading low-pass filters to provide output over 32 channels. The simplified inner hair cell circuit and spiking neurons ensured sparse asynchronous output; the design was tested for localization applications by computing the interaural time delay between the matched pair of silicon cochleas (Yu et al., 2009).
Further improvements including microphone pre-amplifiers and per-channel capability led to the development of AEREAR2 , a 64 channel binaural audition sensor that set a benchmark in neuromorphic audition. By integrating local Digital-to-Analog Converters (DAC) that enable the quality factors of individual channels to be adjusted, this sensor overcomes most of the drawbacks of AEREAR. With improved dynamic range, binaural structure, integrated microphone preamplifiers, and biasing circuits for stability against voltage and temperature variance, this sensor provides precise timing of spikes over a USB interface. This approach was used in complex applications like speaker identification (Li et al., 2012). A thorough comparison between conventional cross-correlation approaches and spike-based sound localization algorithms shows that event-driven methods are about 40 times less computationally demanding . Even more precise and efficient neuromorphic auditory systems will be developed by applying interesting approaches such as spike based audio front ends described in Koickal et al. (2011).

NEUROMORPHIC OLFACTORY SENSORS
The development of artificial olfaction devices started with Moncrieff (1961), who applied mechanical concepts to measuring and determining odors. Since the development of the electronic nose (Persaud and Dodd, 1982), emphasis has been placed on developing olfactory sensors that are portable and precise. Conventional olfactory sensors are large with restricted portability and also impede reliability as the chemical constituents in a target gas vary rapidly. These factors along with high manufacturing costs have largely restricted the use of such sensors to laboratory experiments and industries (Chiu and Tang, 2013).
The electronic nose has benefited from CMOS and MEMS technologies, advanced pattern matching methods and new sensing materials (Gardner et al., 2010). Biologically based olfaction systems have inspired a general structure for electronic nose systems that are composed of a sensor array, signal conditioning circuitry, and a pattern recognition unit (Raman et al., 2008). By applying neuromorphic concepts, improvements were made to this structure by integrating all these units on a single chip and implementing neural networks for pattern recognition. Thomas Koickal and colleagues made a notable contribution to developing an adaptive neuromorphic olfaction chip. The CB polymer sensor array used in that system was fabricated using the AMS 0.6 µm CMOS process (Koickal et al., 2007). A novel design was implemented to cancel the baseline sensor variations due to sensor poisoning and variation in operating current specifications across different sensors. The neuromorphic implementation simplified the odor detection especially in the presence of background odor signals. This design proved to be a technological benchmark to stimulate further study in neuromorphic olfaction by introducing features like on-chip Spike Timing Dependent Plasticity (STDP) learning, reduced power consumption, and temporal spiking signals output. Covington et al. (2007) developed a biomimetic mucosa that can generate spatio-temporal output but improvements were needed for reduced response times and odor delivery channel size in this design.
Researchers have focused on implementing the critical olfaction characteristics rather than emulating the entire biological olfactory pathway. A 4 × 4 tin oxide gas sensor array was designed (Ng et al., 2011) such that each row forms a group of sensors showing similar drift behavior. It is possible to detect a wide range of chemical gases by assigning the same catalyst to each group of sensors. The firing delay in the spiking output from these sensors generates a unique sequence of drift-insensitive spikes (Ng et al., 2009). This output represents a signature for a specific gas which is determined by matching it in a library of 2-D spatio-temporal spike signatures. This approach reduces the computation challenges involved in pattern matching (Ng et al., 2010). The entire gas recognition circuit is fabricated and implemented on a single CMOS chip and the power consumption is as low as 6.6 mW with 94.9% identification accuracy.
E-Nose described in Tang et al. (2011), consists of a conducting polymer sensor chip, interface circuitry, ADC, and a microprocessor with a pattern recognition algorithm and an associated memory module. The output of this chip is a unique signature of the target gas, but the inclusion of a pattern matching algorithm instead of a neural network makes this approach computationally expensive. This approach was also used by Tang et al. (2010) to identify and classify fruity odors and led to the development of a spiking neural network chip that implements the Spike Timing Dependent Plasticity (STDP) learning rule (Hsieh and Tang, 2012). The sensor array used for sampling odor data is a commercial electronic nose (Cyranose 320). This work focused on the backend computation to identify odor and the developed chip can identify three different odors concurrently. The average power consumption is as low as 3.6 µW and mean testing accuracy is 87.59%. NEUROCHEM is an important project lead by European universities that is focused on developing a large sensor array for neuromorphic olfactory systems (Bernabei et al., 2012). This conductive polymer sensor array mimics the essential characteristics of biological Odor Receptor Neurons (ORNs) including redundancy and sensitivity to a wide range of volatile compounds.
The neuromorphic olfactory sensor literature indicates that there is considerable scope for improving these sensors. The CMOS chip by Thomas Koickal is a notable contribution in neuromorphic olfaction and can be considered as a highly cited research contribution in neuromorphic olfaction. Although, there are several shortcomings in this implementation, it proposes a novel architecture for olfactory sensors. Improvements proposed in Ng et al. (2011) are promising if the response time can be improved further. Chiu and Tang (2013) also exposes gaps in interfacing, signal conditioning, and pattern matching computations in neuromorphic olfaction.

TRENDS IN SENSOR FUSION APPLICATION
While neuromorphic sensors offer benefits of low-power consumption and sparse output data generation, the means to process the spike-based data format is still limited. Decades of research in digital image processing and digital signal processing, has led to the development of advanced algorithms and hardware architectures that allowed efficient processing of conventional outputs (e.g., frame-based and audio samples). As techniques for high-level processing of event-based data are still under development, the large-scale application of neuromorphic sensors depends on the introduction of these techniques. However, advanced research in neuromorphic sensors has increased the application scope of these sensors in intelligent embedded systems. Prototypes of neuromorphic vision and auditory sensors evolved into commercial products such as the DVS128 PAER and DAS1. However, most of the systems that make use of neuromorphic sensing implement only a single type of sensor such as vision or auditory. Rajapakse and Acharya (1991) and Bečanović et al. (2005) were among the first who targeted the development of platforms for interfacing multiple neuromorphic sensors. Chan et al. (2012) implemented sensor fusion of Audio-Video (AV) neuromorphic sensors and presented an advanced version of the Koala robot that was first developed by Bečanović et al. (2005) for object tracking. Development of neuromorphic processing boards under large scale projects such as CAVIAR, BrainScaleS and SpiNNaker, promotes the idea of sensor fusion, and data correlation. By utilizing concepts like the spiking Deep Belief Network (DBN; O'Connor et al., 2013), the idea of multi-sensor neuromorphic systems can be brought to fruition. Such systems can have numerous applications in fields such as robotics, biosecurity and environmental monitoring to name a few.

CONCLUSION
In this paper, we have reviewed some of the most significant research contributions toward the improvement of neuromorphic vision, auditory, and olfactory sensors. The distinctive properties of neuromorphic sensors, such as sparse data output and low power consumption have led to extensive research and commercialization. The concept of developing neuromorphic sensors by emulating neuro-biological sensing in silicon has been progressing for many years. More recently, path-breaking research in biological sensing has provided an impetus to developments in neuromorphic sensing, especially in vision and auditory sensors. Pioneering contributions such as DVS and DAVIS, and AEREAR2 have provided considerable progress toward a sensor design that simulates neuro-biological vision and auditory sensing. Accordingly, these have led to the development of several applications for these sensors aiming at replacing conventional sensors in vision and audition. What is lacking is research that provides benchmarks for olfactory sensor implementation and its performance evaluation. Subsequently, future development in neuromorphic sensing should focus on the correlation of inputs from different sensors and efficient pre-processing. With this review we have identified challenges for future research on neuromorphic olfaction, building on the advancements made in vision and audition. In addition to neuromorphic olfaction, future research directions should target neuromorphic sensing of parameters such as pressure, vibration, thermal, and magnetic field as well as their intercorrelated sensor fusion functions which would be ideal for applications such as the Internet of Things (IoT).

AUTHOR CONTRIBUTIONS
AV has contributed more than 75% to this article and is the first author. AO is the principal supervisor of AV and has contributed to this article as a second author. AR is the co-supervisor of AV and has contributed toward content editing and refining. AR is the third author of this article.