A SLAM in Dynamic Environment Based on Instance Segmentation and Optical Flow

doi:10.12382/bgxb.2023.0568

Abstract

Abstract:

A visual semantic SLAM algorithm based on instance segmentation and optical flow is proposed to address the issue of excessive removal of features by traditional semantic SLAM algorithms in dynamic environments.The proposed algorithm utilizes a Mask R-CNN network to perform the instance-level segmentation of potential dynamic objects in an image, and also identifies and eliminates dynamic objects in the optical flow thread. The remaining static optical flow points and static feature points are then used to optimize the location estimation process, ensuring the optimal utilization of both semantic and optical flow information. The proposed algorithm is validated through testing on open datasets and an unmanned ground platform experiment. The experimental results indicate that the average error of the proposed algorithm is 75% and 8.5% lower than those of ORB-SLAM2 and Dyna-SLAM, respectively, on TUM dataset.

Key words: dynamic environment, optical flow method, instance segmentation, semantic simultaneous localization and mapping

CLC Number:

TP242.3

YUE Shengzhe, WANG Zhengjie. A SLAM in Dynamic Environment Based on Instance Segmentation and Optical Flow[J]. Acta Armamentarii, 2024, 45(1): 156-165.

Figures/Tables 15

Fig.1 Framework of algorithm

Fig.2 Removal process of dynamic object point

Fig.3 Examples of dynamic objects

Fig.4 Optical flow point selection

Fig.5 Factor graph

Fig.6 Map completion process

Fig.7 TUM dataset legend

Fig.8 Comparison of map completion effects

Table 1 Comparison of absolute trajectory errors in TUM datasets

编号	ORB-SLAM2		DYNA-SLAM		本文算法
编号	RMSE	STD	RMSE	STD	RMSE	STD
sitting_static	0.009	0.004	0.006	0.003	0.006	0.003
walking_static	0.328	0.115	0.037	0.043	0.034	0.039
walking_xyz	0.892	0.480	0.091	0.057	0.075	0.050

Table 2 Comparison of camera absolute trajectory errors under TUM dataset

Table 3 Comparison of absolute pose errors in KITTI dataset

序列	ORB-SLAM2				本文算法
序列	RMSE	Mean	Max	Min	RMSE	Mean	Max	Min
01	9.558	8.025	16.98	4.398	6.229	6.074	11.958	1.502
02	6.816	5.621	15.005	0.483	5.006	3.978	10.262	0.251
04	1.154	1.057	1.846	0.506	1.147	1.165	2.149	0.105
06	0.801	0.716	1.290	0.480	0.597	0.741	1.086	0.323

Table 4 Comparison of absolute pose errors of algorithms of the same type

序列	DYNA-SLAM	本文方法
00	3.505	2.853
01	9.003	8.720
02	5.219	4.723
03	1.299	1.195
04	1.591	1.292
05	1.779	1.499
06	0.824	0.721
07	2.489	2.232
08	3.291	2.965
09	2.564	2.181
10	2.743	2.276

Fig.9 Ground unmanned platform

Fig.10 Comparison of outdoor dynamic environment trajectories

Fig.11 Comparison of low and high dynamic environment images

References 27

[1]	CADENA C, CARLONE L, CARRILLO H, et al. Past, present, and future of simultaneous localization and mapping: toward the robust-perception age[J]. IEEE Transactions on Robotics, 2016, 32(6): 1309-1332. doi: 10.1109/TRO.2016.2624754 URL
[2]	TAKETOMI T, UCHIYAMA H, IKEDA S. Visual SLAM algorithms: a survey from 2010 to 2016[J]. IPSJ Transactions on Computer Vision and Applications, 2017, 9(1):16. doi: 10.1186/s41074-017-0027-2
[3]	张福斌, 张炳烁, 杨玉帅. 基于单目/IMU/里程计融合的SLAM算法[J]. 兵工学报, 2022, 43(11):2810-2818.
	ZHANG F B, ZHANG B S, YANG Y S. SLAM algorithm based on monocular/IMU/odometer fusion[J]. Acta Armamentarii, 2022, 43(11): 2810-2818. (in Chinese) doi: 10.12382/bgxb.2022.0240
[4]	MUR-ARTAL R, TARDOS D J. ORB-SLAM2:an open-source SLAM system for monocular, stereo, and RGB-D cameras[J]. IEEE Transactions on Robotics, 2017, 33(5):1255-1262. doi: 10.1109/TRO.2017.2705103 URL
[5]	刘全攀, 王正杰, 王寰. 基于双目视觉-惯性导航的轻型无人机导航算法[J]. 兵工学报, 2020, 41(增刊2):241-248.
	LIU Q P, WANG Z J, WANG H. Navigation algorithm of light UAV based on stereo visual inertial navigation odometer[J]. Acta Armamentarii, 2020, 41(S2): 241-248. (in Chinese)
[6]	ENGEL J, KOLTUN V, CREMERS D. Direct sparse odometry[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(3):611-625. doi: 10.1109/TPAMI.2017.2658577 pmid: 28422651
[7]	SUN Y X, LIU M, MENG M Q H. Improving RGB-D SLAM in dynamic environments: a motion removal approach[J]. Robotics and Autonomous Systems, 2016, 89: 110-122. doi: 10.1016/j.robot.2016.11.012 URL
[8]	LI S L, LEE D. RGB-D SLAM in dynamic environments using static point weighting[J]. IEEE Robotics and Automation Letters, 2017, 2(4):2263-2270. doi: 10.1109/LRA.2017.2724759 URL
[9]	WANG Y B, HUANG S D. Towards dense moving object segmentation based robust dense RGB-D SLAM in dynamic scenarios[C]// Proceedings of IEEE International Conference on Control, Automation, Robotics & Vision. Singapore: IEEE, 2014: 1841-1846.
[10]	TAN W, LIU H M, DONG Z L, et al. Robust monocular SLAM in dynamic environments[C]// Proceedings of IEEE International Symposium on Mixed and Augmented Reality. Adelaide, SA, Australia:IEEE, 2013:209-218.
[11]	BESCOS B, FACIL J M, CIVERA J, et al. DynaSLAM: tracking, mapping, and inpainting in dynamic scenes[J]. IEEE Robotics and Automation Letters, 2018, 3(4):4076-4083. doi: 10.1109/LSP.2016. URL
[12]	CHENG J B, WANG Z, ZHOU H Y, et al. DM-SLAM:a feature-based SLAM system for rigid dynamic scenes[J]. ISPRS International Journal of Geo-Information, 2020, 9(4):202. doi: 10.3390/ijgi9040202 URL
[13]	ZHONG F W, WANG S, ZHANG Z Q, et al. Detect-SLAM: making object detection and SLAM mutually beneficial[C]// Proceedings of IEEE Winter Conference on Applications of Computer Vision. Lake Tahoe, NV, US: IEEE, 2018:1001-1010.
[14]	YU C, LIU Z X, LIU X J, et al. DS-SLAM:a semantic visual SLAM towards dynamic environments[C]// Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems.Madrid, Spain:IEEE, 2018:1168-1174.
[15]	LIU Y B, MIURA J. RDS-SLAM: real-time dynamic SLAM using semantic segmentation methods[J]. IEEE Access, 2021, 9: 23772-23785. doi: 10.1109/Access.6287639 URL
[16]	ALCANTARILLA P F, YEBES J J, ALMAZAN J, et al. On combining visual SLAM and dense scene flow to increase the robustness of localization and mapping in dynamic environments[C]// Proceedings of IEEE International Conference on Robotics and Automation. Saint Paul, MN, US: IEEE, 2012: 1290-1297.
[17]	刘立涛, 聂亮. 合成孔径光学成像系统与图像复原技术[J]. 自动化技术与应用, 2021, 40(3): 96-101.
	LIU L T, NIE L. Synthetic aperture optical imaging system and image restoration technology[J]. Automation Technology and Applications, 2021, 40(3): 96-101. (in Chinese)
[18]	FISCHLER M, BOLLES R. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981, 24(6): 381-395. doi: 10.1145/358669.358692 URL
[19]	HE K M, GKIOXARI G, DOLLAR P, et al. Mask r-cnn[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2):386-397. doi: 10.1109/TPAMI.2018.2844175 pmid: 29994331
[20]	TSUNG-YI L, MICHAEL M, SERGE J B, et al. Microsoft coco: common objects in context[J]. Lecture Notes in Computer Science, 2014, 48: 740-755.
[21]	NISTER D, NARODITSKY O, BERGEN J. Visual odometry for ground vehicle applications[J]. Journal of Field Robotics, 2006, 23(1):3-20. doi: 10.1002/rob.v23:01 URL
[22]	CATANIA L, LUATI A. Robust estimation of a location parameter with the integrated hogg function[J]. Statistics and Probability Letters, 2020, 164: 108812. doi: 10.1016/j.spl.2020.108812 URL
[23]	DELLAERT F. Square root SAM: simultaneous localization and mapping via square root information smoothing[J]. The International Journal of Robotics Research, 2006, 25(12):1181-1203. doi: 10.1177/0278364906072768 URL
[24]	KAESS M, JOHANNSSON H, ROBERTS R, et al. iSAM2: Incremental smoothing and mapping using the Bayes tree[J]. The International Journal of Robotics Research, 2012, 31(2):216-235. doi: 10.1177/0278364911430419 URL
[25]	STURM J, ENGELHARD N, ENDRES F, et al. A benchmark for the evaluation of RGB-D SLAM systems[C]// Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems.Vilamoura, Algarve, Portugal: IEEE, 2012: 573-580.
[26]	ZHANG J, HENEIN M, MAHONY R, et al. Robust ego and object 6-DoF motion estimation and tracking[C]// Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. Las Vegas, NV, US: IEEE, 2020: 1875-1881.
[27]	NEWSON A, ALMANSA A, FRADET M, et al. Video inpainting of complex scenes[J]. SIAM Journal on Imaging Sciences, 2014, 7(4):1993-2019. doi: 10.1137/140954933 URL