基于深度确定性策略梯度学习的无线反向散射数据卸载优化

doi:10.3969/j.issn.1000-1093.2021.12.013

兵工学报 ›› 2021, Vol. 42 ›› Issue (12): 2655-2663.doi: 10.3969/j.issn.1000-1093.2021.12.013

基于深度确定性策略梯度学习的无线反向散射数据卸载优化

耿天立^1，2，高昂^1，2，王琦^1，2，段渭军^1，2，胡延苏³

（1.西北工业大学电子信息学院，陕西西安 710072； 2.物联网技术及应用国家地方联合工程实验室，陕西西安 710072； 3.长安大学电子与控制学院，陕西西安 710072）

上线日期:2022-01-15
作者简介:耿天立（1996—），男，硕士研究生。E-mail: gengtianli@mail.nwpu.edu.cn
基金资助:
中国博士后基金项目（2017M623243）；陕西省博士后基金项目（2018BSHYDZZ26）；陕西省重点研发计划项目（2019ZDLGY13-02-02）；广西壮族自治区重点研发计划项目（AB19110036）；太仓市重点研发计划项目（TC2018SF03、TC2019SF03）；西安市科技计划项目(201805042YD20CG26(4)、GXYD21.2)；西北工业大学种子基金项目（CX2020159）；陕西省自然科学基金项目（2021JM-186）

A Deep Deterministic Policy Gradient Optimization Approach for Multi-users Data Offloading in Wireless PoweredCommunication Network

GENG Tianli^1，2， GAO Ang^1，2， WANG Qi^1，2， DUAN Weijun^1，2, HU Yansu³

(1.School of Electronics and Information，Northwestern Polytechnical University，Xi'an 710072，Shaanxi，China；2.State-Province Joint Engineering Laboratory of IoT Technology and Application，Xi'an 710072，Shaanxi，China; 3.School of Electronic Control, Chang'an University, Xi'an 710072, Shaanxi, China)

Online:2022-01-15

摘要/Abstract

摘要： 无线驱动通信网络中，无线设备（WD）可以通过无线反向散射和主动射频传输两种方式进行数据卸载。如何合理分配系统中WD的主动传输和反向散射传输工作模式及其对应的工作时间，从而减小传输延迟、提高传输效率就显得尤为必要。在综合考虑卸载数据量大小、信道条件和WD之间公平性情况下，提出一种基于深度确定性策略梯度（DDPG）的数据卸载方法，在连续动作空间内搜索多个WD的最优时间分配。仿真实验结果表明：DDPG可在有限时间步长内实现算法收敛；由于引入了Jain公平指数，多个WD可同时完成数据卸载；与传统的均分算法、贪心算法对比，DDPG算法可将平均传输延迟减小77.4%和24.2%，可有效提高WD的能耗效率，尤其对于卸载数据量较小的WD效果更加显著。

关键词: 反向散射, 数据卸载, 深度确定性策略梯度, 强化学习

Abstract: In the wireless powered communication network (WPCN)，the wireless devices can offload data through wireless backscattering and active radio frequency transmission.How to adjust the working mode as well as manage the time allocation of ambient backscattering and active RF transmission properly is a great challenge for reducing the system transmission delay and enhancing the transmission efficiency.A deep deterministic policy gradient(DDPG) algorithm is proposed to search the best time allocation in a continuous domain，in which the data size，the channel conditions and the fairness between wireless devices are considered.The experimental results show that DDPG algorithm achieves the algorithm convergence in finite time step， and all the wireless devices can complete the data offloading at the same time by introducing Jain fairness index.Compared with the traditional Round-Robin and Greedy algorithms，DDPG algorithm can be used to reduce the average transmission delay by 77.7% and 24.2%，respectively，and the energy efficiency is largely improved especially for wireless devices with a small amount of offloading data.

Key words: backscattering, dataoffloading, deepdeterministicpolicygradient, reinforcedlearning

中图分类号:

TN926⁺.3

耿天立，高昂，王琦，段渭军，胡延苏. 基于深度确定性策略梯度学习的无线反向散射数据卸载优化[J]. 兵工学报, 2021, 42(12): 2655-2663.

GENG Tianli， GAO Ang， WANG Qi， DUAN Weijun, HU Yansu. A Deep Deterministic Policy Gradient Optimization Approach for Multi-users Data Offloading in Wireless PoweredCommunication Network[J]. Acta Armamentarii, 2021, 42(12): 2655-2663.

参考文献

［1］ LU X，JIANG H，NIYATO D，et al. Wireless-powered device-to-device communications with ambient backscattering: performance modeling and analysis［J］. IEEE Transactions on Wireless Communications，2018，17(3):1528-1544.
［2］ YE Y H，SHI L Q，HU R Q Y，et al. Energy-effificient resource allocation for wirelessly powered backscatter communications［J］.IEEE Communications Letters，2019， 23(8):1418-1422.
［3］叶迎晖，施丽琴，卢光跃.反向散射辅助的无线供能通信网络中用户能效公平性研究［J］.通信学报，2020，41(7):84-94.
YE Y H，SHI L Q，LU G Y. User-centric energy efficiency fairness in backscatter-assisted wireless powered communication network［J］.Journal on Communications， 2020，41(7):84-94.(in Chinese)
［4］ CHEN W Y，DING H Y，WANG S L，et al. Ambient backscatter communications over NOMA downlink channels［J］. China Communications，2020，17(6):80-100.
［5］谢天怡，吕斌，杨真真.反向散射通信辅助的认知无线电能量通信网络的时间分配研究［J］.信号处理，2018，34(1):98-106.
XIE T Y，L B，YANG Z Z.Time allocation optimization in backscatter assisted cognitive wireless powered communication networks［J］.Journal of Signal Processing， 2018，34(1):98-106.(in Chinese)
［6］ HOANG D T，NIYATO D，WANG P，et al.Optimal time sharing in RF-powered backscatter cognitive radio networks［C］∥Proceedings of IEEE International Conference on Communications.Paris，France:IEEE，2017.
［7］ KISHORE R，GURUGOPINATH S，SOFOTASIOS P C，et al. Opportunistic ambient backscatter communication in RF-powered cognitive radio networks［J］.IEEE Transactions on Cognitive Communications and Networking，2019，5(2):413-426.
［8］ HOU Z W，CHEN H，LI Y H，et al.A contract-based incentive mechanism for energy harvesting-based Internet of Things［C］∥Proceedings of 2019 IEEE International Conference on Communications. Paris，France:IEEE，2017.
［9］ HOANG D T，NIYATO D，WANG P，et al.Overlay RF-powered backscatter cognitive radio networks:a game theoretic approach［C］∥Proceedings of 2019 IEEE International Conference on Communications. Paris，France:IEEE，2017.
［10］ WEN X K，BI S Z，LIN X H，et al.Throughput maximization for ambient backscatter communication: a reinforcement learning approach［C］∥Proceedings of 2019 IEEE 3rd Information Technology，Networking，Electronic and Automation Control Conference. Chengdu，China:IEEE， 2019:997-1003.
［11］ XIE Y T，XU Z Z，ZHONG Y X，et al. Backscatter-assisted computation offloading for energy harvesting IoT devices via policy-based deep reinforcement learning［C］∥Proceedings of IEEE/CIC International Conference on Communications Workshops.Changchun，China:IEEE，2019:65-70.
［12］张宏鹏，黄长强，轩永波，等.基于深度神经网络的无人作战飞机自主空战机动决策［J］.兵工学报，2020，41(8):1613-1622.
ZHANG H P，HUANG C Q，XUAN Y B，et al.Maneuver decision of autonomous air combat of unmanned combat aerial vehicle based on deep neural network［J］.Acta Armamentarii，2020，41(8): 1613-1622.(in Chinese)
［13］ MESSOUS M A，SENOUCI S M，SEDJELMACI H，et al. A game theory based efficient computation offloading in an UAV network［J］.IEEE Transactions on Vehicular Technology，2019，68(5):4964-4974.
［14］ HOANG D T，NIYATO D，WANG P，et al.Ambient backscatter: a new approach to improve network performance for RF-powered cognitive radio networks［J］.IEEE Transactions on Communications，2017，65(9):3659-3674.
［15］ LIU C H，CHEN Z Y，TANG J，et al.Energy-efficient control for effective and fair communication coverage:a deep reinforcement learning approach［J］.IEEE Journal on Selected Areas in Communications，2018，36(9):2059-2070.

[1]	李松, 麻壮壮, 张蕴霖, 邵晋梁. 基于安全强化学习的多智能体覆盖路径规划[J]. 兵工学报, 2023, 44(S2): 101-113.
[2]	曹子建, 孙泽龙, 闫国闯, 傅妍芳, 杨博, 李秦洁, 雷凯麟, 高领航. 基于强化学习的无人机集群对抗策略推演仿真[J]. 兵工学报, 2023, 44(S2): 126-134.
[3]	杨加秀, 李新凯, 张宏立, 王昊. 基于积分强化学习的四旋翼无人机鲁棒跟踪[J]. 兵工学报, 2023, 44(9): 2802-2813.
[4]	张建东, 王鼎涵, 杨啟明, 史国庆, 陆屹, 张耀中. 基于分层强化学习的无人机空战多维决策[J]. 兵工学报, 2023, 44(6): 1547-1563.
[5]	李超, 王瑞星, 黄建忠, 江飞龙, 魏雪梅, 孙延鑫. 稀疏奖励下基于强化学习的无人集群自主决策与智能协同[J]. 兵工学报, 2023, 44(6): 1537-1546.
[6]	郑泽新, 李伟, 邹鲲, 李艳福. 基于强化学习的对空雷达抗干扰波形设计[J]. 兵工学报, 2023, 44(5): 1422-1430.
[7]	蒋岩, 丁语嫣, 张兴龙, 徐昕. 基于模型预测与策略学习的智能车辆人机协同控制算法[J]. 兵工学报, 2023, 44(11): 3465-3477.
[8]	赵文飞, 陈健, 王, 滕克难. 基于强化学习的海上要地群协同防空动态火力分配[J]. 兵工学报, 2023, 44(11): 3516-3528.
[9]	李佳键, 史彦军, 杨雨, 李波, 赵熙俊. 无人集群作战任务的多智能体强化学习卸载决策[J]. 兵工学报, 2023, 44(11): 3295-3309.
[10]	卫宁，王冠. 强化学习在智能无人系统决策管理中的应用[J]. 兵工学报, 2022, 43(S2): 164-169.
[11]	李理，李旭光，郭凯杰，史超，陈昭文. 国产化环境下基于强化学习的地空协同作战仿真[J]. 兵工学报, 2022, 43(S1): 74-81.
[12]	魏连震, 龚建伟, 陈慧岩, 李子睿, 龚乘. 基于强化学习补偿的地面无人战车行进间跟瞄自适应控制[J]. 兵工学报, 2022, 43(8): 1947-1955.
[13]	马也，范文慧，常天庆. 基于智能算法的无人集群防御作战方案优化方法[J]. 兵工学报, 2022, 43(6): 1415-1425.
[14]	李庆波，李芳，董瑞星，樊瑞山，谢文龙. 利用强化学习开展比例导引律的导航比设计[J]. 兵工学报, 2022, 43(12): 3040-3047.
[15]	朱建文，赵长见，李小平，包为民. 基于强化学习的集群多目标分配与智能决策方法[J]. 兵工学报, 2021, 42(9): 2040-2048.

基于深度确定性策略梯度学习的无线反向散射数据卸载优化

A Deep Deterministic Policy Gradient Optimization Approach for Multi-users Data Offloading in Wireless PoweredCommunication Network

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价