AUTHOR=Xing Qiang , Chen Zhong , Wang Ruisheng , Zhang Ziqi 

TITLE=Bi-level deep reinforcement learning for PEV decision-making guidance by coordinating transportation-electrification coupled systems

JOURNAL=Frontiers in Energy Research

VOLUME=Volume 10 - 2022

YEAR=2023

URL=https://www.frontiersin.org/journals/energy-research/articles/10.3389/fenrg.2022.944313

DOI=10.3389/fenrg.2022.944313

ISSN=2296-598X

ABSTRACT=The random charging and dynamic traveling behaviors of massive electric vehicles (EVs) pose challenges to the efficient and safe operation of transportation-electrification coupled systems (TECSs). To realize real-time scheduling of urban EV fleet charging demand, this paper proposes an EV decision-making guidance (EVDG) strategy based on the bi-layer deep reinforcement learning, achieving the reduction of user charging costs while ensuring the stable operation of distribution networks (DNs). The EVDG problem is first decoupled into a bi-layer finite Markov decision process, in which the upper-lower layers are used respectively for charging station (CS) recommendation and path navigation. Specifically, the upper-layer agent realizes the mapping relationship between the environment state and the optimal CS by perceiving the EV charging requirements, CS equipment resources and DN operation conditions. And the action decision output of the upper-layer is embedded into the state space of the lower-layer agent. Meanwhile, the lower-level agent determines the optimal road segment for path navigation by capturing the real-time EV state and the transportation network information. Further, two elaborate reward mechanisms are developed to motivate and penalize the decision-making learning of the dual agents. Then a modified Rainbow algorithm based on a deep Q-network is proposed as the solution to the concerned bi-level decision-making problem. Case studies are conducted within a practical urban zone with the TECS. Extensive experimental results verify the superiority and adaptability of our proposed methodology.