We conducted non-intrusive behavioral measurements on tethered male silk moths. Although similar systems to measure the olfactory behavior of insects have been used in the past (Shigaki et al., 2018a, 2019b), in this study we ensure that odor stimuli are accurately presented by using optogenetic silk moths. Using genetically modified specimens that react to blue light stimuli in the same way as normal specimens react to the pheromone bombykol, allowed us to present stimuli accurately and with reproducibility. This is because gaseous pheromones diffuse in the air; therefore, not all stimuli present the same amount of pheromone molecules to the antennae of the moth. Furthermore, in this case, the response of the antennae is measured using an electroantennogram (EAG), which is technically challenging and subjected to electrical noise; in addition, damage to the antennae may occur. Our non-invasive behavior measurement system for the silk moth is shown in Figure 2, and fulfills the following purposes:

To measure the pose of the silk moth, we fixed its back to a thin aluminum rod (Ø 2 mm; length 150 mm) with glue (G17 Bond, Konishi K.K., Osaka, Japan) and placed it on a polysterene sphere (Ø 60 mm), which served as a two-dimensional treadmill. When the moth walked, the sphere moved in response because it was being levitated by the flow of wind from a small fan (FW1251-1051C2ALARX, ARX, Wanchai, Hong Kong). The movements of the sphere were detected using two optical sensors, such as those found in a computer mouse (ADNS-5030, Avago Technologies, California, USA), at a sampling rate of 20 Hz. They were then translated into translational and rotational movements of the moth, that is, the pose.

We developed a virtual representation of an odor plume by modeling the dispersion of white smoke in a wind tunnel. First, we recorded videos of the dispersion of smoke. We also calculated the statistics of the position and intensity of the pixels in the smoke video. Based on these statistics, we programmed a random process that generates virtual circular puffs that match the intensity and transit the positions of the real smoke puffs in the video. An example of a virtual plume is shown in Figure 2A. In addition to the virtual representation of the odor plume, we also programmed a virtual representation of a silk moth. As in the real world, the virtual moth reacts to the virtual plume and travels toward its source. By using a virtual odor plume environment, we can tune parameters such as wind speed, emission rate, and particle lifetime. Tuning such parameters is particularly useful in infotaxis-based behavior modeling because it allows for faster testing of various plume structures and higher reproducibility; compared with real plume experiments. In summary, the following process describes the operation of our experimental device:

The presentation of accurate stimuli is important for the applied infotaxis-based analysis because updating the probability distribution of the source position; as well as the calculation of the expected entropy decrease, are directly affected by whether the agent experiences a hit or not at a given time step. In addition, reproducible odor stimuli are an overall useful property for an olfactory behavior measurement system because their duration and frequency can be finely tuned. Both properties have been reported to directly influence the olfactory behavior of moths (Celani et al., 2014) and other animals (Ache et al., 2016). To present olfactory stimuli to the moth, previous studies have presented pheromones from glass tubes placed directly in front of the antennae of the moth. However, the amount of pheromone particles that effectively reach the antenna varies owing to their gaseous nature.

To ensure that each stimulus has the same intensity and is accurately sensed by the antennae, we utilized genetically modified moths. These BmOR1-GAL4/UAS-ChR2 silk moths (ChR2 hereinafter); express channelrhodopsin-2 in their olfactory receptor neurons. As a result, they execute their olfactory search behavior when their antennae encounter blue light, rather than pheromone particles. This property has been used in previous studies to ensures that all stimuli are reproducible with the same intensity and duration (Shigaki et al., 2018b, 2019a). To activate channelrhodopsin-2, i.e., blue light sensitivity in these moths, we injected all-trans retinal (ATR) into their abdomen on the day before the experiments; because insects do not intrinsically possess ATR. All behavior measurement experiments were conducted from 9:00 to 17:00 to reduce circadian effects (Tomioka et al., 1993). It is reported that brain serotonin level increases in the daytime and that serotonin enhances pheromones sensitivities in the silk moth (Gatellier et al., 2004).

We generated stimuli for the ChR2 silk moths with LEDs (LBW5AP-JYKY-35-Z; Osram Opto Semiconductors), which produced blue light with a 470 nm wavelength and a light intensity of more than 1.6 mW/mm². Such values of wavelength and light intensity have been reported to reliably produce olfactory search responses in ChR2 moths (Tabuchi et al., 2013). On each LED, we attached optical fibers of 3 mm in diameter to ensure that blue light was directed only to each antenna, as seen in Figure 2B. In addition, moths are unable to make yaw turns because their back is glued to an aluminum rod. The only rotation they are able to make is on their neck (see Supplementary Video). However, this neck rotation is very small and it does not decrease the sensibility or the amount of stimulation to the antennae.

Infotaxis was first proposed by Vergassola et al. (2007) as an odor source search algorithm for turbulent environments. In this algorithm, a point-mass agent is located at a position r and searches for an odor source by iteratively reducing its uncertainty about the distribution of possible source locations r_src. The agent has knowledge of its trajectory, Tt, which contains its sequence of positions as well as the odor “hits” it has experienced throughout the search. The agent also maintains a probability map P(r_src|Tt) or “belief” (Thrun et al., 2005) about the location of the source. This belief spans all possible locations of the source r_src that in both the original infotaxis study and the present paper, consist of a two-dimensional lattice of discrete locations. The certainty of the belief P(r_src|Tt) is represented by Shannon's entropy as in Equation (1):

The goal of infotaxis is to minimize the entropy of the belief P(r_src|Tt); therefore, at every time step, the agent calculates the expected change of entropy by moving from its current position r_t to a future position r′ as defined in Equation (2).

Where p^* is the probability of finding the source at r′, and ΔS^* and ΔS are the change in entropy if the source is found or not found at r′, respectively. The agent then executes the move rt↦r′ with the largest negative value of E[ΔS], or; in other words, the move that causes the greatest reduction of uncertainty in the agent's probability map of the possible source locations. Figures 3A,B show conceptual representations of the agent's belief as well as the effect of odor detections on such belief. Detailed derivations of the infotaxis formulae are presented in Appendix A of this paper.

We modeled the body of the silk moth as a point agent with a radius of 10 mm (half of its average body length). We reduced the three degrees of freedom of the moth to (x, y) coordinates because an infotaxis agent moves in a two-dimensional grid ignoring the orientation. Furthermore, we considered as odor hits only those that occurred when the moth was facing upwind, that is, when cos(π − θ + θ_src) > 0 (see Figure 3C), where θ and θ_src are the angle of the moth and the plume's centerline, respectively. We considered this capture region because real moths limit the odor hits to those coming from the front by flapping their wings (Loudon and Koehl, 2000).

We determined whether the behavior of the silk moth matches the definition of the programmed behavior (Kanzaki et al., 1992) by comparing it to the definition of Minegishi et al. (2012). Accordingly, we classified the maneuvers of the silk moth by simply considering the time elapsed since the last odor hit, which we call “blank duration” τ_b as in Celani et al. (2014). We also classified maneuvers according to both τ_b and the moth's linear and angular velocities (v and ω, respectively) based on Minegishi et al. (2012). We denote the first and second classification as “temporal” and “kinematic,” respectively. Table 1 shows a comparison of both schemes used to classify maneuvers and Figure 4 shows the result of using each scheme. The blank duration threshold of 500 ms in the “temporal” classification of Table 1 was selected because this is the average duration of surge motions after an odor hit as reported in Kanzaki et al. (1992). Throughout all olfactory search experiments, we classified the moth maneuvers by both schemes and labeled the state of the moth at each time step as “matching” if it matches the criteria of both schemes and “mismatching” if it only matches the “kinematic” criteria.

To determine whether “mismatching” behaviors are motivated by higher information gains, we analyze the value of the entropy change ΔS and the expected entropy change E[ΔS] regarding the rate of odor hits and the cumulative odor hits experienced by moths over a search. We are particularly interested in these variables because recent studies identified that they influence the decision-process of olfactory behaviors (Celani et al., 2014; Pang et al., 2018). We also evaluate whether the distribution of ΔS is different for “matching” and “mismatching” behaviors with a two-sample Kolmogorov-Smirnov test and by comparing their histograms. In addition, we calculate the cumulative density function (CDF) of ΔS and E[ΔS] to specifically determine whether “mismatching” behaviors have a higher probability of obtaining larger negative values of those variables, that is, greater information gains. Finally, we calculate the root mean squared error (RMSE) between the values of ΔS and E[ΔS] to determine what type of behavior is more similar to infotaxis, regarding the rate of odor hits and the cumulative sum of hits, which are our variables of interest. The following section presents the results of the calculations of ΔS and E[ΔS] regarding hit rate and cumulative hits, the histograms and CDFs, and the RMSE of “matching” and “mismatching” behaviors.

We subjected ChR2 silkmoths to olfactory search experiments. We conducted 20 trials in which the moth searched for a pheromone source in a 350 mm long by 200 mm wide virtual environment where the wind was blowing in the positive x-direction at a mean speed of 0.1 m/s. The initial position of the moth in the virtual environment was (x, y, θ) = (180, 0, −π/6), where θ is in radians. Moths searched for a source located at (x, y) = (0, 0) by entering a radius of 35 mm around it under a time limit of 180 s. The mean ± std. dev. of the time required to reach the source was 73.92 ± 46.5 s. Figure 5A shows the information entropy for the experiments where moths found the pheromone source. The solid line represents the average value, the shaded range represents the standard deviation, and the gray lines show the value for each trial. Figure 5B shows the moth trajectories of these successful trials. The color gradient represents the value of the information entropy. Table 2 shows the statistics of the matching and mismatching moth states. Surge (temporal) and Rotate (kinematic) represent the proportion of time taken when the silk moths exhibited a mismatching state over the entire duration of the search experiments. In total we conducted 20 experiments with 10 specimens. Out of these, 12 trials from six specimens successfully found the odor source under the time limit; thus achieving a success rate of 60.0%. We considered only the data from the successful trials for the classification of matching and mismatching behaviors.

We investigated whether there is a relationship between mismatching maneuvers and a higher expected decrease in entropy E[ΔS(rt↦r′)]. Figure 6 shows the actual rewards ΔS and expected rewards E[ΔS] of the match and mismatch behaviors. As can be seen in Figures 6A,C, matching and mismatching behaviors generate large decreases in entropy at low or high hit rates, respectively. In addition, the matching behaviors generated penalties (entropy increase) at high numbers of accumulated hits. Please note that entropy is non-monotonous (Hajieghrary et al., 2016; Rodríguez et al., 2017) and can increase on detection to non-detection sequences since the agent's belief is narrowed by the detection but broadens again at the non-detection. Figures 6B,D show that the expected rewards are greater at low or high hit rates for mismatching and matching behaviors, respectively. Figures 7A,C show histograms of the actual and expected rewards, respectively. We validated the statistical difference in the distributions of the matching and mismatching states (Kolmogorov-Smirnov test p < 0.01).

Figure 7B shows the cumulative density function of the actual rewards ΔS for matching and mismatching behaviors. As shown in the figure, mismatching behaviors have a higher probability of greater entropy reductions (particularly values of approximately 10⁻⁴ and 10⁻¹). Mismatching behaviors also have a higher probability of a larger decrease in entropy (values of approximately -4 × 10⁻³) as shown in Figure 7D. Figures 8A,B show the cumulative odor hits and hit rate against the root mean squared error between the actual ΔS and the expected reward E[ΔS]. This was calculated as shown in Equation (3), where N is 20 because the sampling frequency of the behavioral measurement system is 20 Hz.

In a recent study, Shigaki et al. (2019b) simultaneously measured the odor search behavior of male silkmoths and the neural activity from their lateral accessory lobe (LAL). The LAL generates motor commands in response to odor stimuli. That study found that silkmoths are less likely to “surge” (move forward) as the frequency of odor hits increases. In terms of infotaxis, this can be interpreted as moths preferring rotations (exploration) because, at high odor encounter rates, the expected decrease in entropy is less than at low rates. Our results found that matching and mismatching behaviors generate rewards at high and low hit rates, respectively (Figure 6). Thus, this leads us to believe that at high hit rates, silk moths prefer reactive or more exploitative behaviors, and at low rates, they prefer more stochastic or explorative behaviors such as rotations instead of straight forward moves. Furthermore, this tendency was observed on all specimens that reached the odor source.

An interesting interpretation of these results can also be made from the viewpoint of reinforcement learning (RL). In this field, an agent learns to behave according to an optimal policy with the highest expected accumulated reward over a time horizon. Nonetheless, many RL algorithms face the exploration and exploitation dilemma in which greedily selecting the actions with the highest reward can lead to suboptimal policies stuck in the local maxima. A common way to avoid this is to add stochasticity in the selection of actions; thus balancing exploration and exploitation, using methods such as ϵ-greedy algorithms (Sutton and Barto, 2018). An analogy can be made to the behavior of the silkmoth in the sense that some randomness in the selection of the “surge” maneuver leads to higher information gains and possibly a better odor source search performance. This can be clearly seen in Figures 7B,D, where the probability of obtaining better rewards is higher for the mismatching behaviors.

We found that maneuvers that deviate from the programmed behavior model correspond to a larger expected decrease in entropy, that is, a higher expected reward in the terminology of reinforcement learning. Therefore, we demonstrated the capability of the infotaxis strategy to quantitatively express maneuvers that deviate from the programmed behavior as explorative and those that match it as exploitative.

Another interesting point to note is the relationship between matching and mismatching behaviors with the root mean squared error (RMSE) of the real vs. expected rewards. As shown in Figure 8A, the error decreases proportionally to the accumulation of odor hits. This is relatively intuitive because more detections narrow down the belief of the source location. However, more RMSE occurs between real and expected rewards at times of high hit rates. Furthermore, the matching behaviors have a lower error than the mismatching behaviors. One possible interpretation for this is that matching behaviors are more exploitative; thus they are more similar to the greedy infotaxis policy, whereas the mismatching behaviors are more explorative; hence, they differ from the expected reward of the infotaxis strategy.

We believe that being able to represent animal olfactory behavior through a method such as infotaxis is an important contribution to the fields of ethology and robotics because having a representation of the decision process of animals in terms of probabilistic beliefs and expected rewards facilitates the algorithmic implementation of these processes in robots. Furthermore, it allows for the refinement of these decision processes using tools such as machine and reinforcement learning.