AUTHOR=Sang Jianghui , Ahmad Khan Zaki , Yin Hengfu , Wang Yupeng 

TITLE=Reward shaping using directed graph convolution neural networks for reinforcement learning and games

JOURNAL=Frontiers in Physics

VOLUME=Volume 11 - 2023

YEAR=2023

URL=https://www.frontiersin.org/journals/physics/articles/10.3389/fphy.2023.1310467

DOI=10.3389/fphy.2023.1310467

ISSN=2296-424X

ABSTRACT=Game theory can employ reinforcement learning algorithms to identify the optimal policy or equilibrium solution. Potential-based Reward Shaping (PBRS) methods are prevalently used for accelerating reinforcement learning, ensuring the optimal policy remains consistent. Existing PBRS research performs message passing in Graph Convolution Neural Networks (GCNs) to propagate information from rewarding states. However, in an irreversible time-series reinforcement learning problem, undirected graphs will not only mislead message passing schemes, but also lose distinctive direction structure. In this paper, a novel approach called Directed Graph Convolution Neural Networks for reward shaping ϕ DCN has been proposed to tackle this problem.The key innovation of ϕ DCN is the extension of spectral-based undirected graph convolution to directed graphs. Messages can be efficiently propagated by leveraging a directed graph Laplacian as a substitute for the state transition matrix. As a consequence, potential-based reward shaping can then be implemented by propagated messages. The incorporation of temporal dependencies between states makes ϕ DCN more suitable for real-world scenarios compared to existing potentialbased reward shaping methods based on undirected graph convolutional networks. Preliminary experiments demonstrate that the proposed ϕ DCN exhibits a substantial improvement compared to other competing algorithms on both Atari and Mujoco benchmarks.