A Deep Deterministic Policy Gradient Optimization Approach for Multi-users Data Offloading in Wireless PoweredCommunication Network

doi:10.3969/j.issn.1000-1093.2021.12.013

Abstract

Abstract: In the wireless powered communication network (WPCN)，the wireless devices can offload data through wireless backscattering and active radio frequency transmission.How to adjust the working mode as well as manage the time allocation of ambient backscattering and active RF transmission properly is a great challenge for reducing the system transmission delay and enhancing the transmission efficiency.A deep deterministic policy gradient(DDPG) algorithm is proposed to search the best time allocation in a continuous domain，in which the data size，the channel conditions and the fairness between wireless devices are considered.The experimental results show that DDPG algorithm achieves the algorithm convergence in finite time step， and all the wireless devices can complete the data offloading at the same time by introducing Jain fairness index.Compared with the traditional Round-Robin and Greedy algorithms，DDPG algorithm can be used to reduce the average transmission delay by 77.7% and 24.2%，respectively，and the energy efficiency is largely improved especially for wireless devices with a small amount of offloading data.

Key words: backscattering, dataoffloading, deepdeterministicpolicygradient, reinforcedlearning

CLC Number:

TN926⁺.3

GENG Tianli， GAO Ang， WANG Qi， DUAN Weijun, HU Yansu. A Deep Deterministic Policy Gradient Optimization Approach for Multi-users Data Offloading in Wireless PoweredCommunication Network[J]. Acta Armamentarii, 2021, 42(12): 2655-2663.

References

［1］ LU X，JIANG H，NIYATO D，et al. Wireless-powered device-to-device communications with ambient backscattering: performance modeling and analysis［J］. IEEE Transactions on Wireless Communications，2018，17(3):1528-1544.
［2］ YE Y H，SHI L Q，HU R Q Y，et al. Energy-effificient resource allocation for wirelessly powered backscatter communications［J］.IEEE Communications Letters，2019， 23(8):1418-1422.
［3］叶迎晖，施丽琴，卢光跃.反向散射辅助的无线供能通信网络中用户能效公平性研究［J］.通信学报，2020，41(7):84-94.
YE Y H，SHI L Q，LU G Y. User-centric energy efficiency fairness in backscatter-assisted wireless powered communication network［J］.Journal on Communications， 2020，41(7):84-94.(in Chinese)
［4］ CHEN W Y，DING H Y，WANG S L，et al. Ambient backscatter communications over NOMA downlink channels［J］. China Communications，2020，17(6):80-100.
［5］谢天怡，吕斌，杨真真.反向散射通信辅助的认知无线电能量通信网络的时间分配研究［J］.信号处理，2018，34(1):98-106.
XIE T Y，L B，YANG Z Z.Time allocation optimization in backscatter assisted cognitive wireless powered communication networks［J］.Journal of Signal Processing， 2018，34(1):98-106.(in Chinese)
［6］ HOANG D T，NIYATO D，WANG P，et al.Optimal time sharing in RF-powered backscatter cognitive radio networks［C］∥Proceedings of IEEE International Conference on Communications.Paris，France:IEEE，2017.
［7］ KISHORE R，GURUGOPINATH S，SOFOTASIOS P C，et al. Opportunistic ambient backscatter communication in RF-powered cognitive radio networks［J］.IEEE Transactions on Cognitive Communications and Networking，2019，5(2):413-426.
［8］ HOU Z W，CHEN H，LI Y H，et al.A contract-based incentive mechanism for energy harvesting-based Internet of Things［C］∥Proceedings of 2019 IEEE International Conference on Communications. Paris，France:IEEE，2017.
［9］ HOANG D T，NIYATO D，WANG P，et al.Overlay RF-powered backscatter cognitive radio networks:a game theoretic approach［C］∥Proceedings of 2019 IEEE International Conference on Communications. Paris，France:IEEE，2017.
［10］ WEN X K，BI S Z，LIN X H，et al.Throughput maximization for ambient backscatter communication: a reinforcement learning approach［C］∥Proceedings of 2019 IEEE 3rd Information Technology，Networking，Electronic and Automation Control Conference. Chengdu，China:IEEE， 2019:997-1003.
［11］ XIE Y T，XU Z Z，ZHONG Y X，et al. Backscatter-assisted computation offloading for energy harvesting IoT devices via policy-based deep reinforcement learning［C］∥Proceedings of IEEE/CIC International Conference on Communications Workshops.Changchun，China:IEEE，2019:65-70.
［12］张宏鹏，黄长强，轩永波，等.基于深度神经网络的无人作战飞机自主空战机动决策［J］.兵工学报，2020，41(8):1613-1622.
ZHANG H P，HUANG C Q，XUAN Y B，et al.Maneuver decision of autonomous air combat of unmanned combat aerial vehicle based on deep neural network［J］.Acta Armamentarii，2020，41(8): 1613-1622.(in Chinese)
［13］ MESSOUS M A，SENOUCI S M，SEDJELMACI H，et al. A game theory based efficient computation offloading in an UAV network［J］.IEEE Transactions on Vehicular Technology，2019，68(5):4964-4974.
［14］ HOANG D T，NIYATO D，WANG P，et al.Ambient backscatter: a new approach to improve network performance for RF-powered cognitive radio networks［J］.IEEE Transactions on Communications，2017，65(9):3659-3674.
［15］ LIU C H，CHEN Z Y，TANG J，et al.Energy-efficient control for effective and fair communication coverage:a deep reinforcement learning approach［J］.IEEE Journal on Selected Areas in Communications，2018，36(9):2059-2070.

[1]	LI Li， LI Xuguang， GUO Kaijie， SHI Chao， CHEN Zhaowen. Simulation of Ground-air Cooperative Combat Based on Reinforcement Learning in Localization Environment [J]. Acta Armamentarii, 2022, 43(S1): 74-81.
[2]	CHEN Zhongyuan, WEI Wenshu, CHEN Wanchun. Reinforcement Learning-based Intelligent Guidance Law for Cooperative Attack of Multiple Missiles [J]. Acta Armamentarii, 2021, 42(8): 1638-1647.