AUTHOR=Ma Ming , Du Wanlin , Wang Ling , Ding Cangbi , Liu Siqi TITLE=Research on the multi-timescale optimal voltage control method for distribution network based on a DQN-DDPG algorithm JOURNAL=Frontiers in Energy Research VOLUME=Volume 10 - 2022 YEAR=2023 URL=https://www.frontiersin.org/journals/energy-research/articles/10.3389/fenrg.2022.1097319 DOI=10.3389/fenrg.2022.1097319 ISSN=2296-598X ABSTRACT=

A large number of distributed generators (GDs) such as photovoltaic panels (PVs) and energy storage (ES) systems are connected to distribution networks (DNs), and these high permeability GDs can cause voltage over-limit problems. Utilizing new developments in deep reinforcement learning, this paper proposes a multi-timescale control method for maintaining optimal voltage of a DN based on a DQN-DDPG algorithm. Here, we first analyzed the output characteristics of the devices with voltage regulation function in the DN and then used the deep Q network (DQN) algorithm to optimize the voltage regulation over longer times and the deep deterministic policy gradient (DDPG) algorithm to optimize the voltage regulation mode over short time periods. Second, the design strategy of the DQN-DDPG algorithm as based on the Markov decision process transformation was presented for the stated objectives and constraints considering the state of ES charge for prolonging the energy storage capacity. Lastly, the proposed strategy was verified on a simulation platform, and the results obtained were compared to those from a particle swarm optimization algorithm, demonstrating the method’s effectiveness.