YSG-SLAM:a Real-time Semantic RGB-D SLAM Based on YOLACT in Dynamic Scene

doi:10.12382/bgxb.2024.0443

Abstract

Abstract:

Simultaneous localization and mapping (SLAM) algorithm has the issues such as localization drift and poor real-time performance in attitude estimation within dynamic environment.A real-time semantic RGB-D SLAM system named as YSG-SLAM is proposed.To enhance the system’s real-time performance,two parallel threads,one for semantic segmentation to obtain 2D semantic information and another for semantic mapping, are introduced.To improve the accuracy and robustness of the system in handling the dynamic objects,YSG-SLAM incorporates a fast dynamic feature removal algorithm and couples it with a missed detection compensation module to mitigate potential missed detections from YOLACT segmentation algorithm.This effectively enhances the feature point removal accuracy and overall system stability.To reduce the localization errors caused by feature point clustering and optimize the spatial distribution of feature points,an adaptive corner extraction threshold calculation method is designed to make the distribution of features more uniform.The semantic mapping thread makes full use of 2D semantic information and 3D point cloud data to optionally construct semantic maps and octree maps,thereby improving the environmental perception capability of system and the task execution ability of robots in complex environment.YSG-SLAM has been evaluated on TUM and Bonn datasets,showing a 93% reduction in various localization errors compared to the original ORB-SLAM2.Experimental results indicate that YSG-SLAM significantly improves real-time performance,has high localization accuracy,and can construct two types of maps.

Key words: dynamic environment, semantic segmentation, adaptive threshold, missed detection compensation, semantic mapping

CLC Number:

TP242.3

ZHAI Weiguang, WANG Feng, MA Xingyu, ZHAO Wei, MENG Pengshuai. YSG-SLAM:a Real-time Semantic RGB-D SLAM Based on YOLACT in Dynamic Scene[J]. Acta Armamentarii, 2025, 46(6): 240443-.

Figures/Tables 17

References 28

[1]	MUR-ARTAL R, TARDÓS J D. Orb-slam2:an open-source slam system for monocular,stereo,and rgb-d cameras[J]. IEEE Transactions on Robotics, 2017, 33(5):1255-1262.
[2]	QIN T, LI P L, SHEN S J. Vins-mono:a robust and versatile monocular visual-inertial state estimator[J]. IEEE Transactions on Robotics, 2018, 34(4):1004-1020.
[3]	DAVISON A J, REID I D, MOLTON N D, et al. MonoSLAM:real-time single camera SLAM[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6):1052-1067.
[4]	CAMPOS C, ELVIRA R, RODRÍGUEZ J J G, et al. Orb-slam3:an accurate open-source library for visual,visual-inertial,and multimap slam[J]. IEEE Transactions on Robotics, 2021, 37(6):1874-1890.
[5]	FORSTER C, ZHANG Z C, GASSNER M, et al. SVO:semidirect visual odometry for monocular and multicamera systems[J]. IEEE Transactions on Robotics, 2016, 33(2):249-265.
[6]	张福斌, 张炳烁, 杨玉帅. 基于单目/IMU/里程计融合的 SLAM 算法[J]. 兵工学报, 2022, 43(11):2810-2818.
	ZHANG F B, ZHANG B S, YANG Y S. SLAM algorithm based on monocular/IMU/odometer fusion[J]. Acta Armamentarii, 2022, 43(11):2810-2818. (in Chinese)
[7]	ZHAO X Y, LI Q H, WANG C H, et al. Robust depth-aided RGBD-inertial odometry for indoor localization[J]. Measurement, 2023, 209:112487-112498.
[8]	LIU Y, WU Y L, PAN W Z. Dynamic RGB-D SLAM based on static probability and observation number[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70:1-11.
[9]	BAHRAINI M S, BOZORG M, RAD A B. SLAM in dynamic environments via ML-RANSAC[J]. Mechatronics, 2018, 49:105-118.
[10]	ZHANG T W, ZHANG H Y, LI Y, et al. Flowfusion:dynamic dense rgb-d slam based on optical flow[C]// Proceedings of 2020 IEEE International Conference on Robotics and Automation.Paris, France: IEEE, 2020:7322-7328.
[11]	SUN Y X, LIU M, MENG M Q H. Motion removal for reliable RGB-D SLAM in dynamic environments[J]. Robotics and Autonomous Systems, 2018, 108:115-128.
[12]	ZHANG L, WEI L Q, SHEN P Y, et al. Semantic SLAM based on object detection and improved octomap[J]. IEEE Access, 2018, 6:75545-75559.
[13]	REDMON J, FARHADI A. YOLO9000:better,faster,stronger[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu,HI,US:IEEE, 2017:7263-7271.
[14]	XIAO L H, WANG J G, QIU X S, et al. Dynamic-SLAM:semantic monocular visual localization and mapping based on deep learning in dynamic environment[J]. Robotics and Autonomous Systems, 2019, 117:1-16.
[15]	LIU Y B, MIURA J. RDS-SLAM:real-time dynamic SLAM using semantic segmentation methods[J]. IEEE Access, 2021, 9:23772-23785.
[16]	BESCOS B, FÁCIL J M, CIVERA J, et al. DynaSLAM:tracking,mapping,and inpainting in dynamic scenes[J]. IEEE Robotics and Automation Letters, 2018, 3(4):4076-4083.
[17]	HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]// Proceedings of the IEEE International Conference on Computer Vision.Venice,Italy:IEEE, 2017:2961-2969.
[18]	YU C, LIU Z X, LIU X J, et al. DS-SLAM:a semantic visual SLAM towards dynamic environments[C]// Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems.Madrid,Spain:IEEE, 2018:1168-1174.
[19]	BADRINARAYANAN V, KENDALL A, CIPOLLA R. Segnet:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495.
[20]	WU W X, GUO L, GAO H L, et al. YOLO-SLAM:a semantic SLAM system towards dynamic environment with geometric constraint[J]. Neural Computing and Applications, 2022, 34(8):6011-6026.
[21]	CHANG J F, DONG N, LI D H. A real-time dynamic object segmentation framework for SLAM system in dynamic scenes[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70:1-9.
[22]	BOLYA D, ZHOU C, XIAO F Y, et al. Yolact:real-time instance segmentation[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision.Seoul,Korea:IEEE, 2019:9157-9166.
[23]	LUO Y, RAO Z R, WU R S. FD-SLAM:a semantic SLAM based on enhanced fast-SCNN dynamic region detection and DeepFillv2-Driven background inpainting[J]. IEEE Access, 2023, 11:110615-110626.
[24]	KENYE L, KALA R. Improving RGB-D SLAM in dynamic environments using semantic aided segmentation[J]. Robotica, 2022, 40(6):2065-2090.
[25]	YU X Y, ZHAN Y A, ZHU F. Improved ORB feature extraction algorithm based on quadtree encoding[J]. Theoretical Computer Science, 2018, 45(2):232-235.
[26]	孙浩, 王朋. 一种基于区域划分的改进ORB算法[J]. 北京航空航天大学学报, 2020, 46(9):1763-1769.
	SUN H, WANG P. An improved ORB algorithm based on region division[J]. Journal of Beijing University of Aeronautics and Astronautics, 2020, 46(9):1763-1769. (in Chinese)
[27]	GONG H, GONG L, MA T B, et al. AHY-SLAM:toward faster and more accurate visual SLAM in dynamic scenes using homogenized feature extraction and object detection method[J]. Sensors, 2023, 23(9):4241.
[28]	WANG X Q, ZHENG S Y, LIN X H, et al. Improving RGB-D SLAM accuracy in dynamic environments based on semantic and geometric constraints[J]. Measurement, 2023, 217:113084.

图像序列	原始图像	ORB-SLAM2	YSG-SLAM
v_bird
v_coffeehouse
v_pomegranate

图像序列	原始图像	ORB-SLAM2	YSG-SLAM
v_bird
v_coffeehouse
v_pomegranate

两种算法及提升率/%	图像序列
两种算法及提升率/%	v_bird	v_pomegranate	v_underground	v_dogman	v_coffeehouse
ORB-SLAM2	4.65424	4.27598	4.50987	3.98462	3.88937
YSG-SLAM	5.65170	5.84350	5.69493	5.31122	5.69493
提升率/%	21.0	36.7	30.6	33.3	46.4

两种算法及提升率/%	图像序列
两种算法及提升率/%	v_bird	v_pomegranate	v_underground	v_dogman	v_coffeehouse
ORB-SLAM2	4.65424	4.27598	4.50987	3.98462	3.88937
YSG-SLAM	5.65170	5.84350	5.69493	5.31122	5.69493
提升率/%	21.0	36.7	30.6	33.3	46.4

图像序列	ORB-SLAM2				YSG-SLAM				提升率/%
图像序列	均方根	平均值	中位数	标准差	均方根	平均值	中位数	标准差	均方根	平均值	中位数	标准差
fr3_s_static	0.2899	0.2606	0.2484	0.1271	0.1619	0.1376	0.1136	0.0854	44.15	47.20	54.27	32.81
fr3_w_half	7.9219	4.4695	1.2568	6.5406	0.4142	0.3507	0.3022	0.2204	94.77	92.15	75.95	96.63
fr3_w_rpy	6.4220	4.5134	2.2990	4.5685	0.5450	0.4130	0.3233	0.3556	91.51	90.85	85.94	92.22
fr3_w_xyz	7.1415	5.6403	4.6159	4.3804	0.3997	0.2816	0.2298	0.2836	94.40	95.01	95.02	93.53