关键词: 无人机空战, 分层强化学习, 权重切换, 深度双Q网络, 分布式深度确定性策略梯度

Abstract: Reinforcement learning in unmanned aerial vehicle (UAV) air combat applications faces challenges, which are posed by the difficulty of rigid reward functions and single models in handling complex tasks within high-dimensional continuous state spaces. This severely limits the algorithm's decision generalization capability in dynamic and varied situations. Addressing the aforementioned issues, an autonomous decision-making framework that integrates the essence of hierarchical and distributed architectures has been proposed, incorporating the Deep Double Q-Network (DDQN) and Deep Deterministic Policy Gradient (DDPG) algorithms. Based on the advantage differences between the opposing forces in various situations, a series of DDPG algorithm models with different reward function weight combinations have been designed to construct a bottom-level distributed DDPG (D3PG) decision network. Introducing the DDQN algorithm, which excels in handling discrete action spaces, to construct the top-level decision-making network, allows for autonomous selection and switching to the most suitable bottom-level policy model based on real-time situation changes, thereby achieving instant adjustment and optimization of decision-making. To further enhance the realism and challenge of the combat environment, a self-play mechanism is introduced into the DDPG algorithm training to construct an enemy decision-making model with high intelligence. The effectiveness and advancement of the method are verified through the training process of competing against intelligent opponents and simulation experiments under random initial situations.

Key words: UAV aerial combat, hierarchical reinforcement learning, weight switching, double deep Q-network, distributed deep deterministic policy gradient