# Trimodal Race Model Inequalities in Multisensory Integration: I. Basics

^{1}Cognitive Psychology Lab, Department of Psychology, University of Oldenburg, Oldenburg, Germany^{2}Cluster of Excellence ‘Hearing4All,’ University of Oldenburg, Oldenburg, Germany^{3}Cognitive Science Lab, Life Sciences and Chemistry, Jacobs University Bremen, Bremen, Germany

The race model inequality has become an important testing tool for the analysis of redundant signals tasks. In crossmodal reaction time experiments, the strength of violation of the inequality is taken as measure of multisensory integration occurring beyond probability summation. Here we extend previous results on trimodal race model inequalities and specify the underlying context invariance assumptions required for their validity. Some simulation results comparing the race model and the superposition model for Erlang distributed random variables illustrate the trimodal inequalities.

## 1. Introduction

When stimulus information, perceived via several sensory modalities, indicates the occurrence of some event, an observer is faster detecting and responding to the stimulus compared to receiving only unimodal information, given a background of noisy signals. As a daily-life example consider the warning lights and siren of an ambulance in a traffic environment, a common audio-visual signal that allows, e.g., a driver to initiate an adequate reaction like giving way faster than if only acoustic or only visual information was available. Since the pioneering study by Todd (1912), this *redundant signals effect* (RSE) has frequently been replicated under laboratory conditions for crossmodal redundant signals combining different modalities, for both manual and saccadic reaction times (RT), and under different experimental conditions (e.g., divided vs. focused attention) (e.g., Miller, 1982; Gielen et al., 1983; Diederich and Colonius, 1987; Corneil et al., 2002).

A number of different models for the mechanisms underlying the RSE have been suggested. Raab (1962) proposed that race models could explain the speedup of responses. Race models assume that (a) each individual stimulus elicits a modality-specific process performed in parallel with the others, and (b) the winners time determines the observable RT, which will also consist of other components like motor execution time. This model implies that the RSE is generated by *statistical facilitation*, or *probability summation*: If latencies are interpreted as (non-negative) random variables, the time to respond to the first of several redundant signals is faster, on average, than the response time to a single signal. More generally, Miller (1982) observed that for the race model with stimuli *x* and *y*, the following inequality should hold:

for all non-negative time points *t*, with *F*_{xy}, *F*_{x}, and *F*_{y} denoting the distribution function for the redundant-signals condition and the single-stimulus conditions, respectively. Literature on this race model inequality (RMI) test involving different sensory modalities is huge (for a recent review, see Gondan and Minakata, 2016), likely due to the following reason: a statistically significant violation of the inequality for some value of *t* suggests that the observed speedup of the response cannot be accounted for entirely by probability summation and some, additional or alternative, *coactivation* mechanism has to be postulated.

Numerous modeling approaches for a coactivation mechanism have been proposed (Diederich, 1995) (see also Diederich and Colonius, 2012, for an overview). In contrast to the race model, to our knowledge they all are based on assuming some specific probability distribution or stochastic process. The one adopted for the simulations described below is introduced in Section 3.

Sometimes, instead of inequality (1), the race model is tested using inequality

However, this is not generally recommended since it is more restrictive than (1) by assuming stochastic independence between the random latencies. While this assumption is not required by the general race model, there is another, essential assumption hidden in any version of the model, known as “context independence” or “context invariance”: the processing of a stimulus of a given modality does not depend on which and how many stimuli from other modalities are presented concurrently (e.g., Luce, 1986, pp. 128–129; Colonius, 1990; Colonius and Vorberg, 1994; Townsend and Wenger, 2004; Gondan and Minakata, 2016). For a more general discussion using coupling theory, see Colonius (2016). A formal definition of this assumption is presented in the next section.

In the majority of RT studies on the RSE, only the bimodal case has been tested. Exceptions are, without claiming exhaustiveness, (Diederich, 1992a, 1995; Diederich and Colonius, 2004; Hecht et al., 2008; Oskarsson et al., 2012; Wang et al., 2012, 2013; Pomper et al., 2014; Hagmann and Russo, 2016), but it seems that no systematic investigation of all possible bimodal and trimodal RMIs and their interdependencies has been performed so far. Here we first specify the context invariance assumptions underlying the inequalities and then discuss various types of trimodal RMIs that can be tested assuming different patterns of (non-) violation of the inequalities. This is illustrated with a first set of simulations comparing the race model with a coactivation model.

## 2. Generalized Race Model Inequalities

In most cases, the stimuli being tested for multisensory integration are from the visual, auditory, or somatosensory modality. Many notable studies also involve other modalities (e.g., Gu et al., 2008; Hoechenberger et al., 2015; Kaliuzhna et al., 2016), like vestibular and olfactory stimuli but, for simplicity, we refer to the first three here only. We write ${V}$ for the unimodal visual, ${A}{V}$ for the bimodal visual-auditory, and ${A}{V}{S}$ for the trimodal visual-auditory-somatosensory condition, with the remaining obvious uni- and bimodal cases denoted accordingly.

### 2.1. The “Context Invariance” Assumption of the Race Model

Let *A, V, S* be random latencies corresponding to stimulus modalities ${A},{V}$, and ${S}$, respectively. *F*_{A}, *F*_{V}, and *F*_{S}, denote the distribution function for the unimodal conditions ${A},{V}$, and ${S}$ respectively. For bimodal condition ${A}{V}$, *H*_{AV} is the bivariate distribution function of random vector (*A, V*), and for trimodal condition ${A}{V}{S}$, *H*_{AVS} stands for the trivariate distribution function of (*A, V, S*). Thus, for example,

for all *s, t, u* ≥ 0. Moreover, for marginal distributions we write, e.g.,

The formal definition of “context invariance” is as follows:

**DEFINITION 1**. For all *s, t, u* ≥ 0, necessary and sufficient conditions for (trimodal) complete context invariance are

and

In other words, complete context invariance holds in the trimodal case if the distributions in the unimodal and bimodal conditions are identical to the corresponding univariate and bivariate marginal distributions of the trivariate distribution. As recently argued in Miller (2016), context invariance is an essential part of the race model concept.

### 2.2. Proving Race Model Inequalities

The proof of Inequality 1 relies on a simple probability inequality. Rewrite (1) for the ${A}{V}$ condition as

For any *t* ≥ 0, we define events

Because of context invariance, *F*_{AV}(*t*) = *P*(*B*_{t} ∪ *C*_{t}), and the inequality follows from

where the last probability inequality is known as a special case of “Boole's inequality” (e.g., Diederich, 1992b). It is important to recognize the role of the context invariance assumption here: it guarantees that events *B*_{t} and *C*_{t} are defined on the same probability space for all *t* or, more generally, that there exists a bivariate distribution *H*_{AV}(*s, t*) with marginals equal to *F*_{A}(*s*) and *F*_{V}(*t*), respectively: *H*_{AV}(*s*, ∞) = *F*_{A}(*s*) and *H*_{AV}(∞, *t*) = *F*_{V}(*t*).

### 2.3. Generalized Race Model Inequalities: Complete Context Invariance

In the following, we assume that trimodal context invariance holds unless indicated otherwise. In order to avoid trivial upper bounds larger than 1, the right-hand side of all inequalities presented may be replaced by, e.g., min{*F*_{A}(*t*) + *F*_{V}(*t*), 1}, etc. For simplicity, we do not state this explicitly.

For further reference, let us start with a listing of all possible bimodal RMIs:

A straightforward generalization to the trimodal case is

which follows again as special case of Boole's inequality.

As shown in Diederich (1992b), Bonferroni-type probability inequalities (Worsley, 1982) can be used to derive further trimodal RMIs:

A sharper^{1} bound for *F*_{AVS}(*t*) results by taking the minimum (at each value of *t*) across all three bounds in 7–9:

Given the bimodal inequalities 3–5 hold, the trimodal inequalities 7–9 are sharper than Inequality 6. For example, if Inequalities 3 and 4 are satisfied, Inequality 9 implies Inequality 6:

### 2.4. Generalized Race Model Inequalities with Restricted Context Invariance

Next we consider a situation where, for example, data from conditions ${A}{V}{S}$, ${A}{V}$, and ${S}$ are available but none from ${V}$ or ${A}$. Responses under the bimodal condition ${A}{V}$ can then be conceived as representing reaction times to a “combined” visual-auditory modality formally equivalent to a unimodal condition. The underlying distribution function is bivariate,

for all *w, z* ≥ 0, with *AV* denoting the RT in condition ${A}{V}$. Context invariance is restricted to the bivariate case, that is:

The race model then implies

Obviously, with adding the following two, there are three inequalities in total:

with the corresponding, mutually incompatible, restricted context invariance assumptions.

An alternative situation for considering Inequalities 13–15 is when all three univariate distributions are available but violations occur for some of the univariate pairs. For example, there may be one or more values *t*′ such that

While the race model for condition ${A}{V}$ would be ruled out in this case, the joint processing of *AV* and *S* may still be consistent with a race.

## 3. An Illustration with Simulated Data Sets

In this section, we (i) illustrate a possible simulation approach and (ii) point to a specific aspect of dependency occurring for trimodal race models.

### 3.1. Erlang Distribution Simulation

Simulating the race model and comparing it with a coactivation model requires specifying some RT distributions. Here we select distribution functions derived from the most basic stochastic counting process, i.e., the *Poisson process*. The time between two randomly occurring events follows an exponential distribution with intensity rate λ. Stimulus processing time is defined as the waiting time of the Poisson process for *c*-th event. Empirically, criterion *c* may be influenced by the experimental condition. For example, rewarding high detection accuracy would increase the threshold (Luce, 1986) and higher values of *c* will result in longer detection time, denoted *D*. The distribution of *D* is known as *Erlang* distribution, a special case of the *gamma* distribution with an integer-valued shape parameter *c* and rate λ:

A rate parameter λ^{x}, *x* ∈ {*A, V, S*}, has to be specified for each single stimulus condition, auditory (A), visual (V), and somato-sensory (S), respectively, while threshold parameter *c* is assumed to be constant across the modalities. For simplicity, we neglect residual processes, like motor time, and assume that *D* equals the observed reaction time.

For *x, y, z* pairwise different modalities chosen from {*A, V, S*}, detection time in the race model for trimodal redundant stimulation, *D*_{xyz}, or *D*_{xy} for bimodal stimulation, is defined as the minimum of the corresponding single stimulus detection times *D*_{x}, *D*_{y}, *D*_{z} :

For a coactivation model, we choose the *superposition model* proposed in Schwarz (1989) (see also Diederich and Colonius, 1991). While a number of alternative coactivation models are available, choosing one based on the Poisson process has the advantage that, with one and the same set of values for the parameters, detection time *D* can be simulated either for the race model or for the superposition model. For the latter, detection time for redundant stimuli follows again an Erlang/Gamma distribution with the intensity rate given by the *sum* of the single stimulus intensity rates λ_{x}, λ_{y}, λ_{z}:

The prediction of the race model, an exponential distribution of RTs, is of course not consistent with typical data. It is taken here just for illustration; for fitting empirical data, it would be easy to add a Gaussian component, resulting in an ex-Gaussian distribution. For an empirical evaluation of race and superposition models we refer to Diederich (1992a).

Figures 1, 2 depict empirical distribution functions obtained from simulations of race and superposition models with parameter values *c* = 2, λ_{A} = λ_{V} = λ_{S} = 0.01 with sample size *n* = 2,000. In both figures the trimodal distribution was obtained from the superposition model, i.e., a Gamma (2, 0.03) distribution. Figure 1 compares it to the bounds described by Inequality 6 and 10 (denoted as “sharp RMI”). Both bounds are violated for a large range of percentiles, with the latter bound being violated even for percentiles beyond 80%. Figure 2 compares the trimodal distribution of the superposition model with data from simulating the bounds described by Inequalities 6 and 13 for the race model. Again, there are large-range violations of the bounds. *F*_{AV} in bound 13 was obtained from the race model but, in principle, it could be made arbitrarily close to 1 by choosing some coactivation model for condition ${A}{V}$.

**Figure 1**. It shows the empirical distribution function of the simulated data for condition ${A}{V}{S}$ and the bounds described by Inequality 6 and 10 (denoted as “sharp RMI”). RTs for condition ${A}{V}{S}$ were simulated according to the superposition model, whereas RTs for all other conditions were generated according to the race model (*n* = 2,000, *c* = 2, λ_{A} = λ_{V} = λ_{S} = 0.01).

**Figure 2**. It shows the empirical distribution function of the simulated data for condition ${A}{V}{S}$ and the bounds described by Inequalities 6 and 13. RTs for condition ${A}{V}{S}$ were simulated according to a superposition model, whereas RTs for condition ${A}{V}$ were generated according to the race model (*n* = 2,000, *c* = 2, λ_{A} = λ_{V} = λ_{S} = 0.01).

### 3.2. Trimodal Dependency: Adjusting Correlations

One characteristic of the race model is the possibility to increase the size of the redundant signals effect by tweaking the correlations between the involved detection processes. Given a race between two detection processes, consider their random processing times, *X* and *Y*, varying from trial to trial. By definition, the shortest processing time min(*X, Y*) determines the detection time for a trial. Negative correlation means that, when one process is fast, the other tends to be slow in a given trial. Thus, there would only be few trials where both processes are rather slow. This results in decreased average detection times, since long times for one process are replaced by shorter times of the other process. One can show that a race model with two processes and maximal negative correlation yields the maximum redundant signals effect (Miller, 1982; Colonius, 1990, 2016).

However, a situation with three “competing” processes *X, Y, Z*, e.g., audio, visual, and somato-sensory, is more complicated. First, instead of one there are three correlation coefficients *r*_{xy}, *r*_{xz}, *r*_{yz}. For a fixed set of two-sample Neyman–Pearson correlation coefficients *r*_{xy} and *r*_{xz}, the third coefficient *r*_{yz} can not vary freely between −1 and +1 but is restricted to a narrower range (Stanley and Wang, 1969):

Second, values for the correlations generating maximal facilitation are not as trivial to find as in a two-process situation. Limited by the above mentioned constraints, it is not possible to construct a correlation matrix with coefficients *r*_{xy} = *r*_{xz} = *r*_{yz} = −1. Our simulations with a multivariate gamma distribution (not presented here) suggest, for example, that setting *r*_{xy} = *r*_{xz} = *r*_{yz} = −0.5 yields a relatively large redundant signals effect when processing times *X*, *Y*, *Z* have an identical underlying distribution.

## Conclusion and Outlook

We have shown that the race model inequality extends naturally from the bimodal to the trimodal case, as long as essential assumptions about context invariance are specified. Moreover, the trimodal case permits “mixed models,” that is, models (i) where the race assumption is only valid for certain modality combinations but not for others, and (ii) where not all unimodal distributions may be available.

For an application of the generalized race model inequalities presented here, the next step is to extend the current statistical tests developed for the bimodal case to the different trimodal cases (for a recent overview, see Gondan and Minakata, 2016). This will also require extensive simulation work as in Kiesel et al. (2007) including an extension to introduce intersubject variability. Finally, it is well-known that the upper bound of the bimodal race model inequality corresponds to maximal negative dependency between the two processes (Miller, 1982; Colonius, 1990, 2016); a particularly challenging task for future study is to characterize the new race model inequalities with respect to the trivariate statistical dependencies underlying their bounds.

## Author Contributions

HC, FW, and AD conceived of the analysis. FW performed the simulations. HC and FW wrote the paper.

## Funding

Supported by DFG (German Science Foundation) SFB/TRR-31 (Project B4, HC), DFG Cluster of Excellence EXC 1077/1 Hearing4all (HC) and DFG Grant DI 506/12-1 (AD).

## Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

## Footnotes

1. ^By “sharper” we mean “strictly smaller or equal.”

## References

Colonius, H. (1990). Possibly dependent probability summation of reaction time. *J. Math. Psychol.* 34, 253–275.

Colonius, H. (2016). An invitation to coupling and copulas, with applications to multisensory modeling. *J. Math. Psychol.* 74, 2–10. doi: 10.1016/j.jmp.2016.02.004

Colonius, H., and Vorberg, D. (1994). Distribution inequalities for parallel models with unlimited capacity. *J. Math. Psychol.* 38, 35–58.

Corneil, B., van Wanrooij, M., Munoz, D., and van Opstal, A. (2002). Auditory-visual interactions subserving goal-directed saccades in a complex scene. *J. Neurophsiol.* 88, 438–2002.

Diederich, A. (1992a). *Intersensory Facilitation: Race, Superposition, and Diffusion Models for Reaction Time with Multiple Stimuli.* Frankfurt: Peter Lang.

Diederich, A. (1992b). Probability inequalities for testing separate activation models of divided attention. *Percept. Psychophys.* 52, 714–716.

Diederich, A. (1995). Intersensory facilitation of reaction time: evaluation of counter and diffusion coactivation models. *J. Math. Psychol.* 39, 197–215.

Diederich, A., and Colonius, H. (1987). Intersensory facilitation in the motor component? *Psychol. Res.* 49, 23–29.

Diederich, A., and Colonius, H. (1991). A further test of the superposition model for the redundant-signals effect in bimodal detection. *Percept. Psychophys.* 50, 83–86.

Diederich, A., and Colonius, H. (2004). Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. *Percept. Psychophys.* 66, 1388–1404. doi: 10.3758/BF03195006

Diederich, A., and Colonius, H. (2012). “Modeling multisensory processes in saccadic responses,” in *The Neural Bases of Multisensory Processes*, eds M. M. Murray and M. T. Wallace (Boca Raton, FL: CRC Press), 253–276.

Gielen, S., Schmidt, R., and van den Heuvel, P. (1983). On the nature of intersensory facilitation of reaction time. *Percept. Psychophys.* 34, 161–168.

Gondan, M., and Minakata, K. (2016). A tutorial on testing the race model inequality. *Atten. Percept. Psychophys*. 78, 723–735. doi: 10.3758/s13414-015-1018-y

Gu, Y., Angelaki, D., and DeAngelis, G. (2008). Neural correlates of multisensory cue integration in macaque MSTd. *Nat. Neurosci.* 11, 1201–1210. doi: 10.1038/nn.2191

Hagmann, C., and Russo, N. (2016). Multisensory integration of redundant trisensory stimulation. *Attent. Percept. Psychophys*. 78, 2558–2568. doi: 10.3758/s13414-016-1192-6

Hecht, D., Reiner, M., and Karni, A. (2008). Multisensory enhancement: gains in choice and in simple response times. *Exp. Brain Res.* 189, 133–143. doi: 10.1007/s00221-008-1410-0

Hoechenberger, R., Busch, N., and Ohla, K. (2015). Nonlinear response speedup in bimodal visual-olfactory object identification. *Front. Psychol.* 6:1477. doi: 10.3389/fpsyg.2015.01477

Kaliuzhna, M., Ferré, E., Herbelin, B., Blanke, O., and Haggard, P. (2016). Multisensory effects on somatosensation: a trimodal visuo-vestibular-tactile interaction. *Sci. Rep.* 6:26301. doi: 10.1038/srep26301

Kiesel, A., Miller, J., and Ulrich, R. (2007). Systematic biases and type i error accumulation in tests of the race model inequality. *Behav. Res. Methods* 39, 539–551. doi: 10.3758/BF03193024

Luce, R. (1986). *Response Times: Their Role in Inferring Elementary Mental Organization*. New York, NY: Oxford University Press.

Miller, J. (1982). Divided attention: evidence for coactivation with redundant signals. *Cogn. Psychol.* 14, 247–279.

Miller, J. O. (2016). Statistical facilitation and the redundant signals effect: what are race and coactivation models? *Atten. Percept. Psychophys.* 78, 516–519. doi: 10.3758/s13414-015-1017-z

Oskarsson, P., Eriksson, L., and Carlander, O. (2012). Enhanced perception and performance by multimodal threat cueing in simulated combat vehicle. *Hum. Factors* 54, 122–137. doi: 10.1177/0018720811424895

Pomper, U., Brincker, J., Harwood, J., Prikhodko, I., and Senkowski, D. (2014). Taking a call is facilitated by the multisensory processing of smartphone vibrations, sounds, and flashes. *PLoS ONE* 9:e103238. doi: 10.1371/journal.pone.0103238

Raab, D. (1962). Statistical facilitation of simple reaction time. *Trans. N.Y. Acad. Sci.* 24, 574–590.

Schwarz, W. (1989). A new model to explain the redundant-signals effect. *Percept. Psychophys.* 46, 498–500.

Stanley, J., and Wang, M. (1969). Restriction on the possible values of *r*_{12}, given values of *r*_{13} and *r*_{23}. *Educ. Psychol. Meas.* 29, 579–581.

Townsend, J., and Wenger, M. (2004). A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. *Psychol. Rev.* 111, 1003–1035. doi: 10.1037/0033-295X.111.4.1003

Wang, W., Hu, L., Cui, H., Xie, X., and Hu, Y. (2013). Spatio-temporal measures of electrophysiological correlates for behavioral multisensory enhancement during visual, auditory and somatosensory stimulation: a behavioral and ERP study. *Neurosci. Bull.* 29, 715–724. doi: 10.1007/s12264-013-1386-z

Wang, W., Hu, L., Valentini, E., Xie, X., Cui, H., and Hu, Y. (2012). Dynamic characteristics of multisensory facilitation and inhibition. *Cogn. Neurodyn.* 6, 409–419. doi: 10.1007/s11571-012-9197-x

Keywords: multisensory integration, race model inequality, context invariance, trimodal case, redundant signals effect, superposition model, statistical facilitation, probability summation

Citation: Colonius H, Wolff FH and Diederich A (2017) Trimodal Race Model Inequalities in Multisensory Integration: I. Basics. *Front. Psychol*. 8:1141. doi: 10.3389/fpsyg.2017.01141

Received: 23 February 2017; Accepted: 22 June 2017;

Published: 11 July 2017.

Edited by:

Martin Lages, University of Glasgow, United KingdomReviewed by:

Juergen Heller, University of Tübingen, GermanyEdgar Erdfelder, University of Mannheim, Germany

Copyright © 2017 Colonius, Wolff and Diederich. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Hans Colonius, hans.colonius@uol.de