Changes in Magnitude and Variability of Corticospinal Excitability During Rewarded Time-Sensitive Behavior

Introduction

The interaction between time estimation and reward perception is crucial to execute behaviors in everyday life. The saying “time flies when we are having fun” refers to how reward influences brain activity during time-sensitive behavior. Previous studies have shown that time estimation and reward perception act by utilizing partially overlapping processing routes (Apaydin et al., 2018). Several brain areas are specialized in temporal processing including the striatum, supplementary motor area, and prefrontal cortex (Bueti et al., 2008; Coull et al., 2011; Üstün et al., 2017; Apaydin et al., 2018), and these brain areas influence M1 activity to execute time-sensitive behavior. Recent studies have indicated that dopamine regulates corticostriatal circuits, and dopamine signaling could modulate time estimation and time-sensitive behaviors (Wiener et al., 2014; Tomasi et al., 2015; Soares et al., 2016).

In human studies, because the corticospinal tract can be activated by transcranial magnetic stimulation (TMS), it has been suggested that the changes in the magnitude and variability of motor evoked potentials (MEPs) depend on M1 activity (Rösler, 2001). Monetary rewards increase MEP amplitudes for the rewarded behavior (Gupta and Aron, 2011; Kapogiannis et al., 2011; Thabit et al., 2011; Borgomaneri et al., 2014; Pisoni et al., 2014; Suzuki et al., 2014), but deprivation of reward as a penalty also increases MEP amplitudes (Suzuki et al., 2018). These observations suggest that reward probability is functionally related to the effectiveness of a reward stimulus, and reward-related signals modulate M1 motor output and MEPs. Especially, a previous study (Nosik and Carr, 2015) indicated that reward probability could momentarily change the value of a consequential reward stimulus, and this phenomenon is termed the “establishing operation.” A previous study on the change in corticospinal excitability during reward tasks indicated that MEP amplitudes before reward stimuli were higher for low reward probability and suggested that this might be related to reward expectation (Suzuki et al., 2014). However, previous studies did not assess the variability of MEP amplitudes but only assessed the magnitude of corticospinal excitability. In addition, previous studies used observational settings without specific behavioral tasks (Kapogiannis et al., 2011; Pisoni et al., 2014) or behavioral tasks unrelated to time perception (Gupta and Aron, 2011; Thabit et al., 2011; Suzuki et al., 2014, 2018). Therefore, it is impossible to know whether expecting a reward or non-reward, based on reward probability, affects the magnitude and variability of corticospinal excitability during time-sensitive behavioral tasks and whether the observed reward-related corticospinal excitability changes are associated with time-sensitive behavioral changes. Therefore, although corticospinal excitability changes are associated with reward expectations, it remains unclear whether reward probabilities affect the magnitude and variability of expectation-related M1 excitability in the context of time-sensitive behavior. These are serious lacunae to elucidate the relationship between reward probability and MEP amplitude changes during time-sensitive behavioral tasks. In addition to expanding on previous findings, exploring how reward probabilities during time-sensitive behavioral tasks affect expectation-related corticospinal excitability may have interesting implications for behavioral science and neuroscience.

Because the temporal resolution of TMS is adequate for observing changes in corticospinal excitability during the rewarded time-sensitive behavioral tasks, we considered that changes in the magnitude and variability of MEPs would be observed using this technique during rewarded time-sensitive behavioral tasks. Therefore, we designed a paradigm involving high and low reward probabilities for time-sensitive behaviors. This paradigm facilitates the investigation of the magnitude and variability of M1 excitability in the context of reward expectation and time estimation. If corticospinal excitability and time estimation change in line with the “establishing operation,” high reward probability contains low reward stimulus value, despite the amount of rewards being large, because high reward probability momentarily decreases the value of a consequential reward stimulus (Nosik and Carr, 2015). In contrast, low reward probability contains high reward stimulus value, despite the amount of rewards being small, because low reward probability momentarily increases the value of a consequential reward stimulus. This raises the question of whether the magnitude and variability of corticospinal excitability related to reward perception reflect the value or the amount of rewards during time-sensitive behavioral tasks. We predicted that if reward amount and value differentially affect M1 excitability, then reward probability should differentially alter the magnitude and variability of MEP amplitudes from the view point of the amount and value of the reward during time-sensitive behavioral tasks. We, therefore, used TMS to investigate expectation-related corticospinal excitation during time-sensitive behavioral tasks with high and low reward probability and to clarify how the magnitude and variability of corticospinal excitations would be altered by reward probability.

Materials and Methods

Participants

We recruited 12 healthy participants [eight women and four men, aged 20–21 years, mean ± standard deviation (SD): 20.8 ± 0.4 years] for the behavioral and MEP amplitude measurements. Two participants only took part in the resting state experiments, four participants only in the behavioral experiments, and six participants in both the resting and behavioral experiments described below. No participant had risks of adverse events from TMS (Rossi et al., 2009) or used medication or had any psychiatric or neurological diseases. The Ethics Committee of the Saitama Prefectural University approved the experimental procedures, and the experiments were performed in accordance with the principles of the Declaration of Helsinki. Written informed consent was obtained from all participants.

Electromyographic (EMG) Recordings

The skin above the first dorsal interosseous (FDI) muscle was cleaned with alcohol to reduce its electrical resistance. Then, double differential surface electrodes (FAD-DEMG1, 4Assist, Tokyo, Japan) adhered on the skin for recording surface EMG activity from the FDI muscle in order to assess corticospinal excitability changes during the rewarded time-sensitive behavioral tasks. The EMG signals were amplified a hundredfold by a DL-140 amplifier (4Assist, Tokyo, Japan), bandpass filtered between 5 and 2,000 Hz and digitized at 10 kHz by a PowerLab system (ADInstruments, Dunedin, New Zealand), and stored on magnetic media.

TMS

A figure-eight coil (internal diameter of each wing: 70 mm) on the subject’s scalp and a Magstim 200² stimulator (Magstim, Whitland, UK) delivered TMS to the scalp via the coil. The coil handle was held approximately 45° to the midline and tangentially to the scalp, thereby a current was induced from the posterolateral to the anteromedial left brain. We determined the appropriate coil position to elicit MEPs in the FDI muscle, and this position was termed the “hotspot” by moving the coil on the left side of the scalp. Then, the hotspot was marked by a soft-tipped pen. The coil was fixed at the hotspot throughout this experiment. The resting motor threshold (RMT) at the hotspot of the relaxed FDI muscle was determined to elicit a MEP of at least 0.05 mV in 5 out of 10 consecutive trials.

Resting State Experiment

Following excitation of cortical neurons by TMS over the M1, multiple descending volleys are temporally and spatially summated in corticospinal neurons (Rösler, 2001). A previous study (Kiers et al., 1993) noted that MEP amplitudes, shapes, and sizes randomly fluctuated between stimuli. We, therefore, conducted a resting state experiment to confirm the fluctuation of MEP amplitudes. Each participant sat comfortably with their right hand resting on the table throughout the resting state experiment. The MEPs for the FDI muscle were evoked by 20 TMS of 120% of the RMT at the hotspot (the interstimulus interval was 5 s).

Behavioral Experiment

The behavioral experiment was carried out on a different day from the resting state experiment. Previous experiments using reward tasks (Gupta and Aron, 2011; Thabit et al., 2011; Suzuki et al., 2014, 2018) carried out 18–100 trials per condition. Therefore, the time-sensitive reward task comprised three fixed-ratio (FR) schedules of 50 trials per schedule; the 50 trials of the FR_A schedule contained a reward stimulus delivered after every response, the 50 trials of the FR_B schedule contained a reward stimulus delivered after every two responses, and the 50 trials of the FR_C schedule contained a reward stimulus delivered after every four responses. The order of the three FR schedules was randomized for counterbalancing purposes. The participants were not aware of the reward probabilities and the order of the schedules. The reward probabilities were predetermined.

Each participant sat comfortably in front of a 27.5 × 31.0 cm screen located approximately (mean ± SD) 66.9 ± 6.5 cm from the face at 11.3 ± 4.7° downward from the eye level with the right palm and forearm resting on the test equipment with two buttons located 4.0 cm apart parallel to the coronal plane (Figure 1A). The left button was pressed with the index finger as quickly as possible after a red circle cue was presented. The red circle cues were presented on the screen at random intervals of 5–6 s (Figure 1B). The participant was instructed to press the left button with the index finger as quickly as possible in response to the appearance of the red circle. Just after the button press, the word “10-yen” or “no pay” was presented for 1 s as feedback. The word “10-yen” denoted 10 Japanese yen (approximately $0.1). In the FR_A schedule, the word “10-yen” or “no pay” would be presented in 100% (50 reward stimuli in 50 presses of the left button) and 0% (zero no-reward stimuli in 50 presses of the left button) of trials, respectively. In the FR_B schedule, the word “10-yen” or “no pay” would be presented in 50% (25 reward stimuli in 50 presses of the left button) and 50% (25 no-reward stimuli in 50 presses of the left button) of trials, respectively. In the FR_C schedule, the word “10-yen” or “no pay” would be presented in 26% (13 reward stimuli in 50 presses of the left button) and 74% (37 no-reward stimuli in 50 presses of the left button) of trials, respectively. Schultz (2007) noted that dopamine concentrations were greatest at 1 s after the presentation of a reward stimulus and returned to baseline after approximately 4 s. Borgomaneri et al. (2012) noted that corticospinal excitability increased at least 300 ms after the presentation of pictures representing negative emotion. Thabit et al. (2011) noted that corticospinal excitability increased 1 s after the presentation of a reward stimulus for 3- to 4-s intervals. We set the delivery time of TMS and inter-trial interval in our protocol in consideration of the previous studies’ time courses and delivered TMS of 120% of the FDI’s RMT 1 s after pressing the left button. Then, the participant had to mentally estimate/wait for 2.5 s from pressing the left button to pressing the right button. Therefore, 50 TMSs were delivered in each FR schedule because the participants pressed the left button iteratively 50 times after the reward or no-reward stimulus. This ensured that the magnitude and variability of corticospinal excitability reflected the expectation of reward or non-reward during time-sensitive behavioral tasks. Each participant received items corresponding to a total of 870 Japanese yen (approximately $8.7) as reward accumulated at the end of the experiment.

FIGURE 1

Figure 1. Experimental design for the rewarded time-sensitive behavioral task. Each participant sat comfortably in front of a screen with the right palm and forearm resting on the test equipment with two buttons (A). The left button was pressed with the index finger as quickly as possible after a red circle cue was presented (B). Just after the left button press, the word “10-yen” or “no pay” was presented for 1 s as feedback. The word “10-yen” denoted 10 Japanese yen (approximately $0.1). Then, the participant had to mentally estimate/wait for 2.5 s from pressing the left button to pressing the right button. Single-pulse TMS of 120% of the FDI’s RMT was delivered 1 s after pressing the left button. TMS, transcranial magnetic stimulation; FDI, first dorsal interosseous; RMT, resting motor threshold.

Data Analysis

To facilitate investigations of intraindividual MEP variability during the time-sensitive reward task, the MEP data were normalized by linear transformation. The normalized MEP data are expressed as Z scores. We predicted that TMS over the M1 would naturally induce a random fluctuation of MEP amplitudes and that time-oriented reward perception would transform activity of the M1 via corticospinal excitability from a random process into an autoregressive process because the autoregressive process could indicate that the MEP amplitude was affected not by random fluctuation but by the preceding MEP amplitudes related to reward or no-reward stimuli from the previous trials. Therefore, a state-changing model was constructed, which included trend, autoregressive, and random fluctuation processes to distinguish between inherent MEP changes by the reward stimulus and MEP random fluctuation as follows:

$f(t)=\alpha +\beta t+{\displaystyle {\sum }_{i=1}^p{\varphi }_i{x}_{t-i}+{\epsilon }_t}(1)$

where α is the y-intercept of the MEP amplitude, reflecting initial corticospinal excitability; β is the MEP amplitude slope, reflecting changes in corticospinal excitability; φ and x are the coefficient and previous reference MEP amplitudes of the autoregressive model, reflecting the temporal dependance structure of a time series; ε_t is the random variation, reflecting the inherent fluctuation of MEPs; i is the order of the model, and t is the number of TMS deliveries during the time-sensitive reward task. By the least-squares method, each participant’s data were fitted to the model. If the model is applicable, the series of values of ε_t in Equation (1) should be uncorrelated to each other (i.e., independence). Therefore, we assessed the applicability of the model with the Ljung–Box test to measure the independence of ε_t as a white noise and residuals process. The following equation was used for the Ljung–Box test.

$Q(h)=n(n+2){\displaystyle {\sum }_{i=1}^h\frac{{\widehat{\rho }}_i{}^2}{n-i}}(2)$

where n is the sample size (${\widehat{\rho }}_i$) is the sample autocorrelation at lag i, and h is the number of lags being tested. Thus, the data eliminate inherent fluctuations of MEPs, permitting the evaluation of whether reward probability affects corticospinal excitability during time-sensitive behavioral tasks. Differences in the MEP amplitudes eliminating inherent fluctuations between three FR schedules and 50 trials were compared by two-way repeated measures analysis of variance (ANOVA). Post hoc testing with Bonferroni correction was performed to compare differences in MEP amplitudes among the three FR schedules. We also compared the MEP amplitudes across trials following presentation of the word “10-yen” or “no-pay” to assess the effect of the immediately preceding reward or no-reward stimulus on expectation-related corticospinal excitability by unpaired t-test. Moreover, the permutated Brunner–Munzel test was performed to carefully assess intra- and inter-individual changes for small sample data because the asymptotic permutational distribution of this test using the central limit theorem can deduce the standard normal distribution and accurate p-value (Fagerland et al., 2011). Response time was calculated as the elapsed time between the left and the right button presses. The time lag between the absolute target time (2.5 s) and subjective response time (the elapsed time between the left and right button presses) was calculated for each trial for each participant to predict change in the participant’s time estimation. To assess group changes, we compared time estimation data based on the response time across the FR schedules using one-way ANOVA. In addition, we compared the time estimation lag across trials immediately preceding a reward (“10-yen”) or no-reward (“no-pay”) stimulus by unpaired t-test and the permutated Brunner–Munzel test. We defined statistical significance as p < 0.05. All statistical analyses were performed with R 3.4.0 software (R Foundation for Statistical Computing, Vienna, Austria).

Results

No participant had adverse TMS-related effects in any experiment.

Corticospinal Excitability During the Resting State

The mean ± standard errors of MEP amplitudes of the FDI muscle during the resting state was 0.94 ± 0.06 mV. Figure 2A shows the time course of changes in FDI MEP amplitudes in the resting state. Figures 2B,C show the random fluctuation of MEP amplitudes [ε_t value in Equation (1)] and the MEP amplitudes eliminating inherent random fluctuations, respectively. Table 1 shows the α, β, p, and φ values in Equation (1) for the resting state. Two of eight (25.0%) participants’ α values were positive, and six of eight (75.0%) participants’ α values were negative. However, six of eight (75.0%) participants’ β values were positive and two of eight (25.0%) participants’ β values were negative. Figure 2D shows the time-series plots of the decomposed mean MEP amplitudes during the resting state. Figure 2A indicates that the raw MEP amplitude increases and decreases during trials, whereas Figures 2C,D indicate that the MEP amplitudes eliminating inherent fluctuations (ε_t) were generally stable. Based on the p parameter estimation of Equation (1), in seven of eight (87.5%) participants, the p-value of the model was 0, which indicates that the errors were uncorrelated across time. In one of eight (12.5%) participants, the p-value of the model was 1, indicating an autoregressive process with a 1-bin time lag and that previous corticospinal excitability affected the variability of corticospinal excitability. The Ljung–Box test showed that the series of ε_t of the model was independent in eight of eight (100%) participants, which indicates that the model was efficient.

FIGURE 2

Figure 2. The time course of changes in each participant’s MEP amplitudes (A), each participant’s MEP random fluctuation (B), each participant’s MEP amplitudes eliminating random fluctuation (C), and the mean MEP amplitudes eliminating random fluctuation (D) during the resting state. The symbols and error bars denote means and standard errors of the mean, respectively. Raw MEP amplitude changes across trials were jumbled, whereas MEP amplitudes eliminating random fluctuations were generally stable. MEP, motor-evoked potential.

TABLE 1

Table 1. Assessment of the model fit in the resting state experiment.

Corticospinal Excitability During the Time-Sensitive Behavioral Tasks

All subjects completed all experimental conditions. Erroneous button presses did not occur during the experiments. Table 2 shows the MEP amplitudes obtained from the FDI muscle during the three FR schedules. Figure 3 shows the time courses of changes in FDI MEP amplitudes during the three FR schedules. Table 3 shows the differences in α, β, p, and φ values for the three FR schedules. The α values were almost the same across the three FR schedules; 4 of 10 (40.0%) participants’ α values were positive for the FR_A schedule, five of 10 (50.0%) participants’ α values were positive for the FR_B schedule, and 4 of 10 (40.0%) participants’ α values were positive for the FR_C schedule. However, the β values were higher for the FR_B and FR_C schedules than for the FR_A schedule; five of 10 (50.0%) participants’ β values were positive for the FR_A schedule, seven of 10 (70.0%) participants’ β values were positive for the FR_B schedule, and eight of 10 (80.0%) participants’ β values were positive for the FR_C schedule. Figure 4A shows the time-series plots of the decomposed mean MEP amplitudes during the rewarded time-sensitive behavioral tasks. Two-way repeated measures ANOVA showed that there was no significant interaction effect in the three FR schedules and 50 trials (F = 0.267, p = 0.769). This allowed us to pool the MEP amplitudes measured from the FDI muscle in the three FR schedules. Post hoc Bonferroni correction showed that the MEP amplitudes obtained for the FDI muscle significantly increased for the FR_B and FR_C schedules compared to the FR_A schedule (FR_A vs. FR_B, p < 0.0001; FR_A vs. FR_C, p < 0.0001; FR_B vs. FR_C, p = 1.000; Figure 4B). In addition, the permutated Brunner–Munzel test also showed that the MEP amplitudes for the FDI muscle in the FR_B and FR_C schedules were significantly greater than those in the FR_A schedule (FR_A vs. FR_B, p < 0.0001; FR_A vs. FR_C, p < 0.0001), but no such difference was observed between the FR_B and FR_C schedules (p = 0.812; Figure 4B). However, unpaired t-tests showed that there were no significant differences in MEP amplitudes immediately preceding the reward (“10-yen”) or no-reward (“no-pay”) stimulus in any FR schedule (FR_A, p = 0.746; FR_B, p = 0.758; FR_C, p = 0.969; Figures 4C–E). The permutated Brunner–Munzel test also showed that there were no significant differences in MEP amplitudes immediately preceding the reward (“10-yen”) or no-reward (“no-pay”) stimulus in any FR schedule (FR_A, p = 0.925; FR_B, p = 0.617; FR_C, p = 0.986). Based on the p parameter estimation of Equation (1), a 0 p-value was more frequent in the FR schedules of lower reward probability; three of 10 (30.0%) participants’ p-values were 0 in the FR_A schedule, 4 of 10 (40.0%) participants’ p-values were 0 in the FR_B schedule, and seven of 10 (70.0%) participants’ p-values were 0 in the FR_C schedule. The Ljung–Box test showed that the series of ε_t values in Equation (1) was independent in 10 of 10 (100%) participants for the three FR schedules.

TABLE 2

Table 2. MEP amplitudes corresponding to the FR schedules.

FIGURE 3

Figure 3. The time course of changes in each participant’s MEP amplitudes during the FR_A (A), FR_B (B), and FR_C (C) schedules; each participant’s MEP random fluctuation during the FR_A (D), FR_B (E), and FR_C (F) schedules; and each participant’s MEP amplitudes eliminating random fluctuation during the FR_A (G), FR_B (H), and FR_C (I) schedules. The slopes (β values) were higher for the FR_B and FR_C schedules than for the FR_A schedule. MEP, motor-evoked potential; FR, fixed-ratio.

TABLE 3

Table 3. Assessment of the model fit.

FIGURE 4

Figure 4. The time course of changes in MEP amplitudes eliminating random fluctuation in the three FR schedules (A), bar graphs of MEP amplitudes to pool the data for the three FR schedules (B), and the MEP amplitudes of the immediately preceding reward or no-reward stimulus in the FR_A (C), FR_B (D) and FR_C (E) schedules. The black (FR_A), grey (FR_B) and white (FR_C) symbols and error bars (A) denote means and standard errors of the mean, respectively. The columns and error bars (B–E) denote the means and standard errors of the mean, respectively. Two-way repeated measures ANOVA showed that there was no significant interaction effect in the three FR schedules and 50 trials (F = 0.267, p = 0.769). Post hoc Bonferroni correction showed that the MEP amplitudes obtained for the FDI muscle significantly increased for the FR_B and FR_C schedules compared to the FR_A schedule (FR_A vs. FR_B, *p < 0.0001; FR_A vs. FR_C, *p < 0.0001; FR_B vs. FR_C, p = 1.000; B). However, there were no significant differences in MEP amplitudes immediately preceding the reward or no-reward stimulus in any FR schedule (unpaired t-tests, FR_A, p = 0.746; FR_B, p = 0.758; FR_C, p = 0.969; C–E). MEP, motor-evoked potential; FR, fixed-ratio; ANOVA, analysis of variance; Rew: presentation of reward (“10-yen”) in the immediately preceding trial; noRew: presentation of no-reward (“no-pay”) in the immediately preceding trial.

Time Estimation During the Time-Sensitive Behavioral Tasks

The time lag between absolute target time and subjective response time was −0.35 ± 0.02 ms for the FR_A schedule, −0.18 ± 0.03 ms for the FR_B schedule, and −0.32 ± 0.03 ms for the FR_C schedule. Although the time lag was negative in all three FR schedules, one-way ANOVA showed that there were no significant differences among the FR schedules (F = 0.458, p = 0.499; Figures 5A,B). Additionally, unpaired t-tests showed that there were no significant differences in the time estimation lag immediately preceding the reward (“10-yen”) or no-reward (“no-pay”) stimulus in any FR schedule (FR_A, p = 0.483; FR_B, p = 0.964; FR_C, p = 0.992; Figures 5C–E). The permutated Brunner–Munzel test also showed that there were no significant differences in MEP amplitudes immediately preceding the reward (“10-yen”) or no-reward (“no-pay”) stimulus in any FR schedule (FR_A, p = 0.384; FR_B, p = 0.982; FR_C, p = 0.894).

FIGURE 5

Figure 5. Time estimation lags among the three FR schedules at the individual level (A) and the group level (B), and among the immediately preceding reward or no-reward stimulus in the FR_A (C), FR_B (D) and FR_C (E) schedules. The columns and error bars denote the means and standard errors of the mean, respectively. The difference between time estimation lags observed among the three FR schedules was small and non-significant (one-way ANOVA, F = 0.458, p = 0.499; B), although the time estimation lags were negative in all three FR schedules. Additionally, there were also no significant differences in time estimation lag immediately preceding the reward (“10-yen”) or no-reward (“no-pay”) stimulus in any FR schedule (unpaired t-tests, FR_A, p = 0.483; FR_B, p = 0.964; FR_C, p = 0.992; C–E). FR, fixed-ratio; ANOVA, analysis of variance.

Discussion

To test the hypothesis that reward amount and value should differentially affect the magnitude and variability of corticospinal excitability, we measured changes in the magnitude and variability of the MEP amplitude related to reward expectation during a time-sensitive behavioral task. Our results showed that: (a) the variability of expectation-related MEP amplitudes transformed from a random process during the resting state into an autoregressive processes during the time-sensitive behavioral task; (b) the random variation of MEP amplitudes in the FR_C, FR_B, and FR_A schedules decreased in a stepwise fashion; (c) the magnitude of the MEP amplitudes increased for the FR_B and FR_C schedules compared to the FR_A schedule; and (d) the time estimation lag was negative for and similar among the three FR schedules. These observations show that reward probability modulated M1 motor output and MEPs. In fact, although the magnitude of the MEP amplitudes was higher in low reward probability (FR_C schedule) than in high reward probability (FR_A schedule), the variability of the MEP amplitudes was transformed into a time-varying autoregressive process by high reward probability (FR_A schedule) rather than by low reward probability (FR_C schedule). This implies that reward probability does not equally affect the magnitude and variability of corticospinal excitability. To our knowledge, this is the first systematic study to report that reward probabilities change the magnitude and variability of expectation-related corticospinal excitabilities during time-sensitive behavior.

Many areas including the ventral tegmental area, striatum, supplementary motor area, and prefrontal cortex influence M1 activity in terms of reward processing (Wickens et al., 2003; Haruno et al., 2004; Campos et al., 2005; Ikemoto, 2007; Hikosaka et al., 2008). In addition, similar brain areas are also specialized in temporal processing including the striatum, supplementary motor area, and prefrontal cortex (Bueti et al., 2008; Macdonald et al., 2012; Failing and Theeuwes, 2016; Apaydin et al., 2018). Dopamine neurons connect to the striatum and prefrontal cortex (Haber and Knutson, 2010; Averbeck et al., 2014; Haber, 2016). In addition, the prefrontal cortex connects to the supplementary motor area (Goldman-Rakic, 1987); thus, prefrontal input is provided from dopamine neurons to the supplementary motor area, which in turn connects to the M1. Moreover, a retrograde tracing study found that approximately 70% of dopamine neurons in the midbrain projected to the M1 (Hosp et al., 2011). Previous studies have suggested that bursts of dopaminergic activity in the midbrain serve as time perception (Soares et al., 2016). These previous findings regarding neural networks and physiological mechanisms suggested that overall coactivation of the corticostriatal circuit including the ventral tegmental area, striatum, supplementary motor area, and prefrontal cortex might reveal the time perception and reward processing through direct and indirect projections of dopaminergic and glutamatergic neurons, and these circuits may influence corticospinal excitability via the M1. In our study, TMS was delivered 1 s after the presentation of reward or no-reward stimuli in accordance with the previous studies’ time courses regarding dopamine concentration and corticospinal excitation by reward presentation (Schultz, 2007; Thabit et al., 2011). This experimental setup allowed us to investigate changes in the magnitude and variability of MEPs during rewarded time-sensitive behavioral tasks. In our study, the magnitudes of the MEP amplitudes before reward presentation increased for low reward probability. This is the first novel observation of our study. Although the exact mechanism for high MEP amplitudes for low reward probability were not identified, we predict that M1 excitability during the time-sensitive behavioral task could have been influenced by reward probability. One possibility is that the activities of many brain regions, including the ventral tegmental area, striatum, supplementary motor area, and prefrontal cortex may affect M1 activity with different gains according to reward probability. Especially, recent research findings have suggested that low reward probability, rather than high reward probability, increases the number of behaviors (Derosa et al., 2015; Fisher et al., 2018). This phenomenon termed the “establishing operation” occurs as a result of low reward probability momentarily increasing the value of a consequential reward stimulus (Derosa et al., 2015; Nosik and Carr, 2015; Fisher et al., 2018). In addition, previous studies have suggested that low reward probabilities (Suzuki et al., 2014), upsetting images (Oliveri et al., 2003; Coelho et al., 2010; Borgomaneri et al., 2012), and unexpected penalties also increase corticospinal excitability (Suzuki et al., 2018). These may imply that M1 excitation may increase in line with the “establishing operation” or with no-reward in low reward probability. However, the MEP amplitudes immediately preceding the reward (“10-yen”) or no-reward (“no-pay”) stimulus did not differ in any of the three FR schedules. Therefore, changes in M1 excitability related to reward probability might be affected by the global reward signal throughout each FR schedule. To clarify this, further research is needed on the time course of changes in M1 excitability in relation to various reward settings, including rewards and penalties, in fixed- and variable-ratio schedules.

Kiers et al. (1993) studied the variability of MEPs produced by TMS and noted that the variability in MEPs is essentially random in the resting state. In our study, the p-value of the model was 0 in most datasets during the resting state, which indicates that the variability of the MEP amplitudes was uncorrelated across time and a random process. However, TMS-evoked MEP amplitude variability was a time-varying autoregressive process during the time-sensitive behavioral task. In addition, the random variability of MEP amplitudes decreased from low reward probability (i.e., FR_C) to high reward probability (i.e., FR_A). This is the second novel observation of our study. It has been previously noted that the frontal network was engaged in time perception, reward perception, and working memory (Üstün et al., 2017; Apaydin et al., 2018). In our study, the participant waited for 5–6 s until seeing the next reward stimulus in the FR_A schedule, whereas the participant waited for 20–24 s until seeing the next reward stimulus in the FR_C schedule. This interval of reward presentation may affect the variability of corticospinal excitabilities during time-sensitive behavioral tasks from the standpoint of memory retention time. In fact, the red circle cue did not indicate reward signals and schedules but only preannounced reward appearance. Therefore, the subjects might expect the reward in reference to the history of reward appearances. Hence, our findings showed that high reward probability facilitates the variability of expectation-related M1 excitability in an autoregressive manner, which extends the results of previous studies and supports the proposition that reward probability affects the variability of expectation-related corticospinal excitability.

In this study, the time estimation lag was negative in all three FR schedules. Soares et al. (2016) found that activation or inhibition of dopamine neurons contributed to decelerate or accelerate time estimation, respectively. Our result suggests that reward may decelerate time estimation and delay response time, and consequently, the time estimation lag became negative. However, there were no differences in the time estimation lag among the three FR schedules. Additionally, the time estimation lag immediately preceding the reward (“10-yen”) or no-reward (“no-pay”) stimulus did not differ in any of the three FR schedules. In previous reward tasks (Kapogiannis et al., 2008; Gupta and Aron, 2011; Thabit et al., 2011; Suzuki et al., 2014, 2018), 10–500 Japanese yen (approximately $0.1 to $5) were used as a monetary reward. However, in previous penalty tasks (Suzuki et al., 2018), the penalty stimulus indicated that the participant lost 100 Japanese yen (approximately $1.0). In our study, the reward stimulus was the word “10-yen,” which had a rewarding value as it represented 10 actual Japanese yen. The non-reward stimulus was the word “no-pay,” which did not have rewarding value. Therefore, the stimulus gap between “10-yen” and “no-pay” may be too small to clarify the changes in the time estimation lag among the three FR schedules. In the context of the gap between reward and no-reward, a higher reward may emphasize changes in the time estimation lag during time-sensitive behavioral tasks. In our study, the participant had to mentally estimate/wait for 2.5 s after TMS with suprathreshold intensity. Although previous studies suggested that TMS delays or shortens the reaction time according to the intensity of the stimuli (Pascual-Leone et al., 1992a,b), a 2.5 s waiting time is sufficiently long to reduce the effect of TMS on reaction time. Therefore, the effect on the time estimation lag of TMS in this study was considered minimal. However, the role of changes in corticospinal excitability during time-sensitive behavioral tasks for decelerating time estimation remains unclear. Further research is needed to investigate the relationship between the time estimation process and corticospinal excitability using higher reward stimuli.

A potential limitation of our study is the small sample size, although the permutated Brunner–Munzel test can deduce the standard normal distribution and accurate p-value in small sample data (Fagerland et al., 2011). In addition, corticostriatal neuronal activities related to midbrain dopaminergic neurons could not be directly observed. A previous study (Fiorillo et al., 2003) suggested that reward and penalty outcomes are related to the firing of dopaminergic neurons. A study by Koepp et al. (1998) found evidence that dopamine was released in the human striatum during a behavioral task. Another study (Zald et al., 2004) noted that rewards increased dopamine transmission. A larger number of participants will be needed in future studies, and additional detailed examination using both TMS and brain imaging methods should be conducted to identify the neuronal effects of reward probabilities.

In conclusion, we found that reward probabilities were associated with expectation-related corticospinal excitabilities during a time-sensitive behavioral task. In fact, the magnitude of corticospinal excitability increased in low reward probability, whereas the variability of corticospinal excitability transformed into an autoregressive process in high reward probability. These results imply that the magnitude and variability of corticospinal excitabilities can be differentially altered by reward probability. These findings have implications for the characteristics of corticospinal excitation including M1 changes during rewarded time-sensitive behavior.

Ethics Statement

The experimental procedures were approved by the Ethics Committee of the Saitama Prefectural University and performed in accordance with the principles of the Declaration of Helsinki. All participants provided written informed consent.

Author Contributions

MS participated in the design of the study, carried out the experiment, performed the statistical analyses, and drafted the manuscript. TS conceived the study, participated in its design, carried out the experiment, and drafted the manuscript. Y-JW conceived the study, participated in its design, and helped with the experiment. TH conceived the study, participated in its design, and drafted the manuscript. All authors read and approved the final manuscript.

Funding

MS was supported by a Grant-in-Aid for Japan Society for the Promotion of Science (JSPS) KAKENHI 18H03133 and 18K19750.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Abbreviations

FDI, first dorsal interosseous; FR, fixed ratio; MEP, motor evoked potential; RMT, resting motor threshold; TMS, transcranial magnetic stimulation.

References

Apaydin, N., Üstün, S., Kale, E. H., Çelikağ, I., Özgüven, H. D., Baskak, B., et al. (2018). Neural mechanisms underlying time perception and reward anticipation. Front. Hum. Neurosci. 12:115. doi: 10.3389/fnhum.2018.00115

PubMed Abstract | CrossRef Full Text | Google Scholar

Averbeck, B. B., Lehman, J., Jacobson, M., and Haber, S. N. (2014). Estimates of projection overlap and zones of convergence within frontal-striatal circuits. J. Neurosci. 34, 9497–9505. doi: 10.1523/jneurosci.5806-12.2014

PubMed Abstract | CrossRef Full Text | Google Scholar

Borgomaneri, S., Gazzola, V., and Avenanti, A. (2012). Motor mapping of implied actions during perception of emotional body language. Brain Stimul. 5, 70–76. doi: 10.1016/j.brs.2012.03.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Borgomaneri, S., Gazzola, V., and Avenanti, A. (2014). Temporal dynamics of motor cortex excitability during perception of natural emotional scenes. Soc. Cogn. Affect. Neurosci. 9, 1451–1457. doi: 10.1093/scan/nst139

PubMed Abstract | CrossRef Full Text | Google Scholar

Bueti, D., Bahrami, B., and Walsh, V. (2008). Sensory and association cortex in time perception. J. Cogn. Neurosci. 20, 1054–1062. doi: 10.1162/jocn.2008.20060

PubMed Abstract | CrossRef Full Text | Google Scholar

Campos, M., Breznen, B., Bernheim, K., and Andersen, R. A. (2005). Supplementary motor area encodes reward expectancy in eye-movement tasks. J. Neurophysiol. 94, 1325–1335. doi: 10.1152/jn.00022.2005

PubMed Abstract | CrossRef Full Text | Google Scholar

Coelho, C. M., Lipp, O. V., Marinovic, W., Wallis, G., and Riek, S. (2010). Increased corticospinal excitability induced by unpleasant visual stimuli. Neurosci. Lett. 481, 135–138. doi: 10.1016/j.neulet.2010.03.027

PubMed Abstract | CrossRef Full Text | Google Scholar

Coull, J. T., Cheng, R. K., and Meck, W. H. (2011). Neuroanatomical and neurochemical substrates of timing. Neuropsychopharmacology 36, 3–25. doi: 10.1038/npp.2010.113

PubMed Abstract | CrossRef Full Text | Google Scholar

Derosa, N. M., Fisher, W. W., and Steege, M. W. (2015). An evaluation of time in establishing operation on the effectiveness of functional communication training. J. Appl. Behav. Anal. 48, 115–130. doi: 10.1002/jaba.180

PubMed Abstract | CrossRef Full Text | Google Scholar

Fagerland, M. W., Sandvik, L., and Mowinckel, P. (2011). Parametric methods outperformed non-parametric methods in comparisons of discrete numerical variables. BMC Med. Res. Methodol. 11:44. doi: 10.1186/1471-2288-11-44

PubMed Abstract | CrossRef Full Text | Google Scholar

Failing, M., and Theeuwes, J. (2016). Reward alters the perception of time. Cognition 148, 19–26. doi: 10.1016/j.cognition.2015.12.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Fiorillo, C. D., Tobler, P. N., and Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902. doi: 10.1126/science.1077349

PubMed Abstract | CrossRef Full Text | Google Scholar

Fisher, W. W., Greer, B. D., Mitteer, D. R., Fuhrman, A. M., Romani, P. W., and Zangrillo, A. N. (2018). Further evaluation of differential exposure to establishing operations during functional communication training. J. Appl. Behav. Anal. 51, 360–373. doi: 10.1002/jaba.451

PubMed Abstract | CrossRef Full Text | Google Scholar

Goldman-Rakic, P. S. (1987). Motor control function of the prefrontal cortex. Ciba Found. Symp. 132, 187–200. doi: 10.1002/9780470513545.ch12

PubMed Abstract | CrossRef Full Text | Google Scholar

Gupta, N., and Aron, A. R. (2011). Urges for food and money spill over into motor system excitability before action is taken. Eur. J. Neurosci. 33, 183–188. doi: 10.1111/j.1460-9568.2010.07510.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Haber, S. N. (2016). Corticostriatal circuitry. Dialogues Clin. Neurosci. 18, 7–21.

PubMed Abstract | Google Scholar

Haber, S. N., and Knutson, B. (2010). The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 35, 4–26. doi: 10.1038/npp.2009.129

PubMed Abstract | CrossRef Full Text | Google Scholar

Haruno, M., Kuroda, T., Doya, K., Toyama, K., Kimura, M., Samejima, K., et al. (2004). A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J. Neurosci. 24, 1660–1665. doi: 10.1523/jneurosci.3417-03.2004

PubMed Abstract | CrossRef Full Text | Google Scholar

Hikosaka, O., Bromberg-Martin, E., Hong, S., and Matsumoto, M. (2008). New insights on the subcortical representation of reward. Curr. Opin. Neurobiol. 18, 203–208. doi: 10.1016/j.conb.2008.07.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Hosp, J. A., Pekanovic, A., Rioult-Pedotti, M. S., and Luft, A. R. (2011). Dopaminergic projections from midbrain to primary motor cortex mediate motor skill learning. J. Neurosci. 31, 2481–2487. doi: 10.1523/JNEUROSCI.5411-10.2011

PubMed Abstract | CrossRef Full Text | Google Scholar

Ikemoto, S. (2007). Dopamine reward circuitry: two projection systems from the ventral midbrain to the nucleus accumbens-olfactory tubercle complex. Brain Res. Rev. 56, 27–78. doi: 10.1016/j.brainresrev.2007.05.004

PubMed Abstract | CrossRef Full Text | Google Scholar

Kapogiannis, D., Campion, P., Grafman, J., and Wassermann, E. M. (2008). Reward-related activity in the human motor cortex. Eur. J. Neurosci. 27, 1836–1842. doi: 10.1111/j.1460-9568.2008.06147.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Kapogiannis, D., Mooshagian, E., Campion, P., Grafman, J., Zimmermann, T. J., Ladt, K. C., et al. (2011). Reward processing abnormalities in Parkinson’s disease. Mov. Disord. 26, 1451–1457. doi: 10.1002/mds.23701

PubMed Abstract | CrossRef Full Text | Google Scholar

Kiers, L., Cros, D., Chiappa, K. H., and Fang, J. (1993). Variability of motor potentials evoked by transcranial magnetic stimulation. Electroencephalogr. Clin. Neurophysiol. 89, 415–423. doi: 10.1016/0168-5597(93)90115-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Koepp, M. J., Gunn, R. N., Lawrence, A. D., Cunningham, V. J., Dagher, A., Jones, T., et al. (1998). Evidence for striatal dopamine release during a video game. Nature 393, 266–268. doi: 10.1038/30498

PubMed Abstract | CrossRef Full Text | Google Scholar

Macdonald, C. J., Cheng, R. K., and Meck, W. H. (2012). Acquisition of “Start” and “Stop” response thresholds in peak-interval timing is differentially sensitive to protein synthesis inhibition in the dorsal and ventral striatum. Front. Integr. Neurosci. 6:10. doi: 10.3389/fnint.2012.00010

PubMed Abstract | CrossRef Full Text | Google Scholar

Nosik, M. R., and Carr, J. E. (2015). On the distinction between the motivating operation and setting event concepts. Behav. Anal. 38, 219–223. doi: 10.1007/s40614-015-0042-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Oliveri, M., Babiloni, C., Filippi, M. M., Caltagirone, C., Babiloni, F., Cicinelli, P., et al. (2003). Influence of the supplementary motor area on primary motor cortex excitability during movements triggered by neutral or emotionally unpleasant visual cues. Exp. Brain Res. 149, 214–221. doi: 10.1007/s00221-002-1346-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Pascual-Leone, A., Brasil-Neto, J. P., Valls-Sole, J., Cohen, L. G., and Hallett, M. (1992a). Simple reaction time to focal transcranial magnetic stimulation. Comparison with reaction time to acoustic, visual and somatosensory stimuli. Brain 115, 109–122. doi: 10.1093/brain/115.1.109

PubMed Abstract | CrossRef Full Text | Google Scholar

Pascual-Leone, A., Valls-Sole, J., Wassermann, E. M., Brasil-Neto, J., Cohen, L. G., and Hallett, M. (1992b). Effects of focal transcranial magnetic stimulation on simple reaction time to acoustic, visual and somatosensory stimuli. Brain 115, 1045–1059. doi: 10.1093/brain/115.4.1045

PubMed Abstract | CrossRef Full Text | Google Scholar

Pisoni, A., Lo Gerfo, E., Ottone, S., Ponzano, F., Zarri, L., Vergallito, A., et al. (2014). Fair play doesn’t matter: MEP modulation as a neurophysiological signature of status quo bias in economic interactions. Neuroimage 101, 150–158. doi: 10.1016/j.neuroimage.2014.06.056

PubMed Abstract | CrossRef Full Text | Google Scholar

Rösler, K. M. (2001). Transcranial magnetic brain stimulation: a tool to investigate central motor pathways. News Physiol. Sci. 16, 297–302. doi: 10.1152/physiologyonline.2001.16.6.297

PubMed Abstract | CrossRef Full Text | Google Scholar

Rossi, S., Hallett, M., Rossini, P. M., Pascual-Leone, A., and Safety of TMS Consensus Group. (2009). Safety, ethical considerations, and application guidelines for the use of transcranial magnetic stimulation in clinical practice and research. Clin. Neurophysiol. 120, 2008–2039. doi: 10.1016/j.clinph.2009.08.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Schultz, W. (2007). Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 30, 259–288. doi: 10.1146/annurev.neuro.28.061604.135722

PubMed Abstract | CrossRef Full Text | Google Scholar

Soares, S., Atallah, B. V., and Paton, J. J. (2016). Midbrain dopamine neurons control judgment of time. Science 354, 1273–1277. doi: 10.1126/science.aah5234

PubMed Abstract | CrossRef Full Text | Google Scholar

Suzuki, M., Hamaguchi, T., and Matsunaga, A. (2018). Nonequivalent modulation of corticospinal excitability by positive and negative outcomes. Brain Behav. 8:e00862. doi: 10.1002/brb3.862

PubMed Abstract | CrossRef Full Text | Google Scholar

Suzuki, M., Kirimoto, H., Sugawara, K., Oyama, M., Yamada, S., Yamamoto, J., et al. (2014). Motor cortex-evoked activity in reciprocal muscles is modulated by reward probability. PLoS One 9:e90773. doi: 10.1371/journal.pone.0090773

PubMed Abstract | CrossRef Full Text | Google Scholar

Thabit, M. N., Nakatsuka, M., Koganemaru, S., Fawi, G., Fukuyama, H., and Mima, T. (2011). Momentary reward induce changes in excitability of primary motor cortex. Clin. Neurophysiol. 122, 1764–1770. doi: 10.1016/j.clinph.2011.02.021

PubMed Abstract | CrossRef Full Text | Google Scholar

Tomasi, D., Wang, G. J., Studentsova, Y., and Volkow, N. D. (2015). Dissecting neural responses to temporal prediction, attention, and memory: effects of reward learning and interoception on time perception. Cereb. Cortex 25, 3856–3867. doi: 10.1093/cercor/bhu269

PubMed Abstract | CrossRef Full Text | Google Scholar

Üstün, S., Kale, E. H., and Çiçek, M. (2017). Neural networks for time perception and working memory. Front. Hum. Neurosci. 11:83. doi: 10.3389/fnhum.2017.00083

PubMed Abstract | CrossRef Full Text | Google Scholar

Wickens, J. R., Reynolds, J. N., and Hyland, B. I. (2003). Neural mechanisms of reward-related motor learning. Curr. Opin. Neurobiol. 13, 685–690. doi: 10.1016/j.conb.2003.10.013

PubMed Abstract | CrossRef Full Text | Google Scholar

Wiener, M., Lee, Y. S., Lohoff, F. W., and Coslett, H. B. (2014). Individual differences in the morphometry and activation of time perception networks are influenced by dopamine genotype. Neuroimage 89, 10–22. doi: 10.1016/j.neuroimage.2013.11.019

PubMed Abstract | CrossRef Full Text | Google Scholar

Zald, D. H., Boileau, I., El-Dearedy, W., Gunn, R., Mcglone, F., Dichter, G. S., et al. (2004). Dopamine transmission in the human striatum during monetary reward tasks. J. Neurosci. 24, 4105–4112. doi: 10.1523/jneurosci.4643-03.2004

PubMed Abstract | CrossRef Full Text | Google Scholar