Voltage Hierarchical Control Strategy of Active Distribution Network Based on Deep Reinforcement Learning

doi:10.12096/j.2096-4528.pgt.23029

Abstract

Abstract:

Objectives The randomness and volatility of distributed power generation poses significant challenges for the voltage control in active distribution network (AND). In this context, there is an urgent need for an efficient voltage control strategy to ensure the safe operation of ADN. Methods Based on the deep reinforcement learning method, a voltage control strategy for double-layer regional distribution networks was proposed. First, based on the adjustment characteristics of voltage regulating equipment and the complexity of controllable elements, a regional coordinated control area and a local autonomous control area were designed for the radiating grid structure of the ADN, and the voltage control model of each area was constructed. Then, the model was solved by deep Q-Network (DQN) algorithm and deep deterministic policy gradient (DDPG) algorithm to achieve the purpose of tracking voltage changes in real time, and effectively solve the voltage control problem during the operation of the ADN. Finally, the method was verified by IEEE 33-bus simulation examples. Results The DQN algorithm and the DDPG algorithm were used to solve the control variables in the coordinated control region and the local autonomous region respectively, realizing real-time decision-making of voltage regulation in the ADN system, and solving the problems of bidirectional flow of ADN power flow and complex and changeable voltage. Conclusions The proposed control strategy has obvious effect on controlling voltage deviation, and has strong accuracy and practicality.

Key words: active distribution network (ADN), regional coordination control, local autonomous control, deep reinforcement learning, voltage control strategy

CLC Number:

TK 01

Wanlin DU, Ling WANG, Wei LUO, Yuanzhe ZHU, Hong LÜ, Xiaonan MA, Xia ZHOU. Voltage Hierarchical Control Strategy of Active Distribution Network Based on Deep Reinforcement Learning[J]. Power Generation Technology, 2024, 45(4): 734-743.

Figures/Tables 12

Fig. 1 Voltage distribution state diagram of distribution network

Fig. 2 Structure diagram of hierarchical control method of distribution network

Fig. 3 Overall control flow chart

Fig. 4 IEEE 33-node distribution network topology

Tab. 1 Equipment parameters

序号	布置节点	类型	额定容量
1	L17、L19、L20、L33	分布式光伏	2.2 MW
2	L11、L14、L22、L28	储能电池	800 kW⋅h
3	L18、L25	SVC	2 MV⋅A

Tab. 2 Control zone division results

协调控制区域节点编号	自治控制区域编号	所含节点编号	控制节点编号
L1—L33	自治控制区域1	L1—L5、L19—L2、L3—L25	L20、L22、L25
	自治控制区域2	L6—L18	L11、L14、L17、L18
	自治控制区域3	L26—L33	L28、L33

Fig. 5 DQN agent training process

Fig. 6 Node voltage diagram before and after regional coordination control

Fig. 7 DDPG agent training results

Fig. 8 Voltage control effect without active power reduction

Fig. 9 Voltage control effect with active power reduction

Tab. 3 Control results under different strategies

策略	优化前	强化学习算法	SSA	POS
响应时间/s	0	0.11	5.52	4.84
平均电压偏差/pu	0.024 0	0.002 3	0.002 9	0.003 1

References 29

1	江道灼，徐宁，江崇熙，等．蜂巢状有源配电网构想、关键技术与展望[J]．电力系统自动化，2019，43(17)：1-11． doi:10.7500/AEPS20180730017
	JIANG D Z， XU N， JIANG C X，et al ．Conception，key technology and prospect of honeycomb-shape active distribution network[J]．Automation of Electric Power Systems，2019，43(17)：1-11． doi:10.7500/AEPS20180730017
2	刘建伟，李学斌，刘晓鸥．有源配电网中分布式电源接入与储能配置[J]．发电技术，2022，43(3)：476-484． doi:10.12096/j.2096-4528.pgt.21068
	LIU J W， LI X B， LIU X O ．Distributed power access and energy storage configuration in active distribution network[J]．Power Generation Technology，2022，43(3)：476-484． doi:10.12096/j.2096-4528.pgt.21068
3	于淼，闫旻睿，万克厅，等．数据驱动的有源配电网运行态势智能感知方法[J]．电力建设，2024，45(7)：34-53．
	YU M， YAN M R， WAN K T，et al ．Data-driven intelligent situational awareness of active distribution networks[J]．Electric Power Construction，2024，45(7)：34-53．
4	蔡欢，袁旭峰，熊炜，等．柔性互联配电网运行调度研究综述[J]．智慧电力，2022，50(6)：92-99．
	CAI H， YUAN X F， XIONG W，et al ．Review on operation scheduling of flexible interconnected distribution network[J]．Smart Power，2022，50(6)：92-99．
5	闫群民，董新洲，穆佳豪，等．基于改进多目标粒子群算法的有源配电网储能优化配置[J]．电力系统保护与控制，2022，50(10)：11-19．
	YAN Q M， DONG X Z， MU J H，et al ．Optimal configuration of energy storage in an active distribution network based on improved multi-objective particle swarm optimization[J]．Power System Protection and Control，2022，50(10)：11-19．
6	仲泽天，李梦月，王加澍，等．一种有源配电网分布式光伏消纳能力评估方法[J]．电网与清洁能源，2023，39(2)：60-68．
	ZHONG Z T， LI M Y， WANG J S，et al ．An assessment method for distributed photovoltaic absorption capacity of active distribution networks[J]．Power System and Clean Energy，2023，39(2)：60-68．
7	LOPION P， MARKEWITZ P， ROBINIUS M，et al ．A review of current challenges and trends in energy systems modeling[J] ． Renewable and Sustainable Energy Reviews，2018，96：156-166． doi:10.1016/j.rser.2018.07.045
8	彭生江，陆军，张中丹，等．光伏接入对电网的影响研究[J]．电力系统保护与控制，2021，49(5)：157-164． doi:10.19783/j.cnki.pspc.200505
	PENG S J， LU J， ZHANG Z D，et al ．Research on the influence of photovoltaic access on a power grid[J]．Power System Protection and Control，2021，49(5)：157-164． doi:10.19783/j.cnki.pspc.200505
9	段穰达．有源配网后评价指标体系及其综合评价方法[J]．发电技术，2021，42(1)：86-93． doi:10.12096/j.2096-4528.pgt.20102
	DUAN R D ．A post-evaluation index system of active distribution network project and its comprehensive evaluation method[J]．Power Generation Technology，2021，42(1)：86-93． doi:10.12096/j.2096-4528.pgt.20102
10	张华赢，艾精文，汪伟．基于约束型深度强化学习的主动配电网电压控制策略[J/OL] ．电测与仪表，2022，．
	ZHANG H Y， AI J W， WANG W ．Active distribution network voltage control strategy based on constrained deep reinforcement learning[J/OL]．Journal of Electrical Measurement and Instrumentation，2022，
11	WECKX S， DRIESEN J ．Optimal local reactive power control by PV inverters[J]．IEEE Transactions on Sustainable Energy，2016，7(4)：1624-1633. doi:10.1109/tste.2016.2572162
12	符杨，廖剑波，李振坤，等．考虑越限风险的主动配电网日前优化调度及运行[J]．中国电机工程学报，2017，37(21)：6328-6338． doi:10.13334/j.0258-8013.pcsee.162014
	FU Y， LIAO J B， LI Z K，et al ．Day-ahead optimal scheduling and operating of active distribution network considering violation risk[J]．Proceedings of the CSEE，2017，37(21)：6328-6338． doi:10.13334/j.0258-8013.pcsee.162014
13	李鹏，姜磊，王加浩，等．基于深度强化学习的新能源配电网双时间尺度无功电压优化[J/OL]．中国电机工程报：1-15．．
	LI P， JIANG L， WANG J H，et al ．Optimization of dual-time scale reactive voltage for distribution network with renewable energy based on deep reinforcement learning[J/OL]．Proceedings of the CSEE，1-15．．
14	蒋平，梁乐．基于内点法和遗传算法相结合的交直流系统无功优化[J]．高电压技术，2015，41(3)：724-729． doi:10.13336/j.1003-6520.hve.2015.03.004
	JIANG P， LIANG L ．Reactive power optimization of hybrid AC/HVDC power system combining interior point algorithm and genetic algorithm[J]．High Voltage Engineering，2015，41(3)：724-729． doi:10.13336/j.1003-6520.hve.2015.03.004
15	李晓利，高金峰．用于配电网多目标无功优化的改进粒子群优化算法[J]．电力自动化设备，2019，39(1)：106-111． doi:10.16081/j.issn.1006-6047.2019.01.016
	LI X L， GAO J F ．Improved particle swarm optimization algorithm for multi-objective reactive power optimization of distribution network[J]．Electric Power Automation Equipment，2019，39(1)：106-111． doi:10.16081/j.issn.1006-6047.2019.01.016
16	林少华，吴杰康，莫超，等．基于二阶锥规划的含分布式电源配电网动态无功分区与优化方法[J]．电网技术，2018，42(1)：238-246． doi:10.13335/j.1000-3673.pst.2017.0718
	LIN S H， WU J K， MO C，et al ．Dynamic partition and optimization method for reactive power of distribution networks with distributed generation based on second-order cone programming[J]．Power System Technology，2018，42(1)：238-246． doi:10.13335/j.1000-3673.pst.2017.0718
17	肖遥，别朝红，黄格超，等．基于二阶锥最优潮流的机会约束分布鲁棒优化研究[J]．电网技术，2021，45(4)：1505-1517． doi:10.13335/j.1000-3673.pst.2020.1002
	XIAO Y， BIE Z H， HUANG G C，et al ．Chance-constrained distributional robust optimization based on second-order cone optimal power flow[J]．Power System Technology，2021，45(4)：1505-1517． doi:10.13335/j.1000-3673.pst.2020.1002
18	李国庆，陈蓉，沈杰，等．大规模光伏接入的配电网动态无功优化研究[J]．太阳能学报，2018，39(8)：2373-2380．
	LI G Q， CHEN R， SHEN J，et al ．Research of dynamic reactive power optimization of large-scale photovoltaic access distribution network[J]．Acta Energiae Solaris Sinica，2018，39(8)：2373-2380．
19	刘斌，程杰，肖荣，等．面向中低压一体化的配网无功优化方法[J]．电气自动化，2019，41(6)：45-47．
	LIU B， CHENG J， XIAO R，et al ．Reactive power optimization method for integrated medium and low voItage distribution networks[J]．Electrical Automation，2019，41(6)：45-47．
20	褚国伟，张友旺，葛乐，等．自储能柔性互联配电网多时间尺度电压优化[J]．电力系统自动化，2021，45(9)：71-79． doi:10.7500/AEPS20200223007
	CHU G W， ZHANG Y W， GE L，et al ．Multi-time-scale voltage optimization of flexible interconnected distribution network with self-energy storage[J]．Automation of Electric Power Systems，2021，45(9)：71-79． doi:10.7500/AEPS20200223007
21	郭清元，吴杰康，莫超，等．基于混合整数二阶锥规划的新能源配电网电压无功协同优化模型[J]．中国电机工程学报，2018，38(5)：1385-1396． doi:10.13334/j.0258-8013.pcsee.170247
	GUO Q Y， WU J K， MO C，et al ．A model for multi-objective coordination optimization of voltage and reactive power in distribution networks based on mixed integer second-order cone programming[J]．Proceedings of the CSEE，2018，38(5)：1385-1396． doi:10.13334/j.0258-8013.pcsee.170247
22	于成龙，侯俊杰，蒲洪波，等．新一代人工智能在国防科技领域发展探讨[J]．国防科技，2020，41(4)：13-18．
	YU C L， HOU J J， PU H B，et al ．Development of the new generation artificial intelligence in national defense science and technology[J]．National Defense Technology，2020，41(4)：13-18．
23	蒲天骄，乔骥，韩笑，等．人工智能技术在电力设备运维检修中的研究及应用[J]．高电压技术，2020，46(2)：369-383． doi:10.13336/j.1003-6520.hve.20200131001
	PU T J， QIAO J， HAN X，et al ．Research and application of artificial intelligence in operation and maintenance for power equipment[J]．High Voltage Engineering，2020，46(2)：369-383． doi:10.13336/j.1003-6520.hve.20200131001
24	YAP K Y， SARIMUTHU C R， LIM J M Y ．Artificial intelligence based MPPT techniques for solar power system：a review[J]．Journal of Modern Power Systems and Clean Energy，2020，8(6)：1043-1059． doi:10.35833/mpce.2020.000159
25	李琦，乔颖，张宇精．配电网持续无功优化的深度强化学习方法[J]．电网技术，2020，44(4)：1473-1480． doi:10.13335/j.1000-3673.pst.2019.2218
	LI Q， QIAO Y， ZHANG Y J ．Continuous reactive power optimization of distribution network using deep reinforcement learning[J]．Power System Technology，2020，44(4)：1473-1480． doi:10.13335/j.1000-3673.pst.2019.2218
26	WANG Y ．Grid voltage control method based on generator reactive power regulation using reinforcement learning[C]//2020 IEEE/IAS Industrial and Commercial Power System Asia (I&CPS Asia)．IEEE，2020：1060-1065． doi:10.1109/icpsasia48933.2020.9208556
27	毕聪博，唐聿劼，罗永红，等．电力系统优化控制中强化学习方法应用及挑战[J]．中国电机工程学报，2024，44(1)：1-21．
	BI C B， TANG Y J， LUO Y H，et al ．Review on critical problems in reinforcement learning methods applied in power system optimization and control scenarios[J]．Proceedings of the CSEE，2024，44(1)：1-21．
28	倪爽，崔承刚，杨宁，等．基于深度强化学习的配电网多时间尺度在线无功优化[J]．电力系统自动化，2021，45(10)：77-85．
	NI S， CUI C G， YANG N，et al ．Multi-time-scale online optimization for reactive power of distribution network based on deep reinforcement learning[J]．Automation of Electric Power Systems，2021，45(10)：77-85．
29	高一航，徐晓春，李鹏，等．面向高比例新能源并网的双时间尺度电网电压控制[J]．电网与清洁能源，2021，37(5)：115-122． doi:10.3969/j.issn.1674-3814.2021.05.014
	GAO Y H， XU X C， LI P，et al ．Dual-time-scale grid voltage control for high-proportion new energy grid-connected[J]．Power System and Clean Energy，2021，37(5)：115-122． doi:10.3969/j.issn.1674-3814.2021.05.014

[1]	Chengping HU, Ming FAN, Aiwang LIU, Yunhui SHI. Multi-Area Interconnected Integrated Energy System Planning Considering Cloud Energy Storage [J]. Power Generation Technology, 2024, 45(4): 641-650.
[2]	Zhenyu ZHAO, Geriletu BAO, Xinxin LI. Optimization and Scheduling of Integrated Energy Systems With Carbon Capture and Storage-Power to Gas Based on Information Gap Decision Theory [J]. Power Generation Technology, 2024, 45(4): 651-665.
[3]	Kang YANG, Lanqing LI, Yifeng LI, Dongkuo SONG, Bolun WANG, Jin CHEN, Xia ZHOU, Yu SHAN. A Novel Distributed Photovoltaic Output Interval Prediction Method [J]. Power Generation Technology, 2024, 45(4): 684-695.
[4]	Zirui MENG, Yawen LIU, Xing JU. Operation Analysis of a Photovoltaic-Piezoelectric Composite Independent Power Supply System [J]. Power Generation Technology, 2024, 45(4): 696-704.
[5]	Bai XIAO, Xuechun ZHAO, Guangde DONG. Summary and Prospect of Comprehensive Evaluation Methods of Power Quality [J]. Power Generation Technology, 2024, 45(4): 716-733.
[6]	Yu CHEN, Hong DING, Yong CUI, Li ZHU, Shijun CHEN, Qiuyang LING, Yongsheng XU, Jian ZHENG. Research on Temperature Situation Awareness and Auxiliary Decision-Making System Scheme of Substation Equipment [J]. Power Generation Technology, 2024, 45(4): 744-752.
[7]	Weijie WANG, Xinjie ZENG, Yuantu XU, Shuyi LI, Canhua RUAN, Ning TONG, Xiaomei WU. Renewable Energy Distribution Network Overcurrent Protection Based on Positive-Sequence Sudden-Change Component Locus Identification [J]. Power Generation Technology, 2024, 45(4): 753-764.
[8]	Guichao DUAN, Gong WANG, Shengxian CAO, Jie DUAN. Research on Operation Strategy of Virtual Power Plant and Distributed Control of DG [J]. Power Generation Technology, 2024, 45(4): 765-771.
[9]	Xin YUAN, Jun LIU, Heng CHEN, Peiyuan PAN, Gang XU, Xiuyan WANG. Effect of Carbon Capture Technology Application on Peak Shaving Capacity of Coal-Fired Units [J]. Power Generation Technology, 2024, 45(3): 373-381.
[10]	Jiahai YUAN, Yuelin HU, Jian ZHANG. The Carbon Emission Efficiency of China’s Listed Thermal Power Companies: An Improved Three-Stage Slack Based Measure-Data Envelopment Analysis Model [J]. Power Generation Technology, 2024, 45(3): 458-467.
[11]	Mingyi ZHANG, Peng HUANG, Yawu WANG. Research on Output Characteristics of a Circular Dielectric Elastomer Generator [J]. Power Generation Technology, 2024, 45(3): 517-526.
[12]	Jianqiang YE, Dunhu SUN. Research on Power Planning Based on Robust Optimization Under Carbon Trading Condition [J]. Power Generation Technology, 2024, 45(3): 566-574.
[13]	Fangfang WANG, Pengwei YANG, Guangjin ZHAO, Qi LI, Xiaona LIU, Shuangchen MA. Development and Challenge of Flexible Operation Technology of Thermal Power Units Under New Power System [J]. Power Generation Technology, 2024, 45(2): 189-198.
[14]	Zhan LI, Zhenyong YANG, Lei LIU, Zhensan CHEN, Weiming JI, Feng HONG. Analysis of the Influence of Furnace Side Heat Storage Coefficient on Primary Frequency Modulation Capacity Under Deep Modulation Condition of Thermal Power Unit [J]. Power Generation Technology, 2024, 45(2): 226-232.
[15]	Zheng YANG, Yipeng SUN, Zhiqiang WEN, Liang CHENG, Zhanguo LI. Research on Dry-Wet Conversion Strategy of Supercritical Thermal Power Units Under Deep Peaking Condition [J]. Power Generation Technology, 2024, 45(2): 233-239.