Weber-Fechner Law in Temporal Difference Learning derived from Control as Inference

Keiichiro Takahashi¹Taisuke Kobayashi^2,3*Tomoya Yamanokuchi¹Takamitsu Matsubara¹

• ¹Nara Sentan Kagaku Gijutsu Daigakuin Daigaku, Ikoma, Japan
• ²National Institute of Informatics, Chiyoda-ku, Japan
• ³Sogo Kenkyu Daigakuin Daigaku, Miura District, Japan

This paper investigates a novel nonlinear update rule for value and policy functions based on temporal difference (TD) errors in reinforcement learning (RL). The update rule in the standard RL states that the TD error is linearly proportional to the degree of updates, treating all rewards equally without no bias. On the other hand, the recent biological studies revealed that there are nonlinearities in the TD error and the degree of updates, biasing policies optimistic or pessimistic. Such biases in learning due to nonlinearities are expected to be useful and intentionally leftover features in biological learning. Therefore, this research explores a theoretical framework that can leverage the nonlinearity between the degree of the update and TD errors. To this end, we focus on a control as inference framework utilized in the previous work, in which the uncomputable nonlinear term needed to be approximately excluded from the derivation of the standard RL. By analyzing it, Weber-Fechner law (WFL) is found, namely, perception (a.k.a. the degree of updates) in response to stimulus change (a.k.a. TD error) is attenuated by increase in the stimulus intensity (a.k.a. the value function). To numerically reveal the utilities of WFL on RL, we then propose a practical implementation using a reward-punishment framework and modifying the definition of optimality. Further analysis of this implementation reveals that two utilities can be expected i) to accelerate escaping from the situations with small rewards, and ii) to pursue the minimum punishment as much as possible. We finally investigate and discuss the expected utilities through simulations and robot experiments. As a result, the proposed RL algorithm with WFL shows the expected utilities that accelerate the reward-maximizing startup and continue to suppress punishments during learning.