欢迎访问《兵工学报》官方网站,今天是 分享到:

兵工学报 ›› 2022, Vol. 43 ›› Issue (5): 1107-1116.doi: 10.12382/bgxb.2021.0262

• 论文 • 上一篇    下一篇

结合关键帧提取的视频-文本跨模态实体分辨双重编码方法

曾志贤1,2, 曹建军1,2, 翁年凤1,2, 蒋国权1,2, 范强1,2   

  1. (1.国防科技大学 计算机学院, 湖南 长沙 410003; 2.国防科技大学 第六十三研究所, 江苏 南京 210007)
  • 上线日期:2022-03-17
  • 通讯作者: 曹建军(1975—),男,教授,博士生导师 E-mail:caojj@nudt.edu.cn
  • 作者简介:曾志贤(1996—),男,硕士研究生。E-mail: zeng_zhixian@yeah.net
  • 基金资助:
    国家自然科学基金项目(61371196);中国博士后科学基金特别资助项目(2015M582832);国家重大科技专项项目(2015ZX01040-201)

Dual Encoding Integrating Key Frame Extraction for Video-text Cross-modal Entity Resolution

ZENG Zhixian1,2, CAO Jianjun1,2, WENG Nianfeng1,2, JIANG Guoquan1,2, FAN Qiang1,2   

  1. (1. College of Computer Science and Technology,National University of Defense Technology,Changsha 410003,Hunan,China;2. The 63rd Research Institute,National University of Defense Technology,Nanjing 210007,Jiangsu,China)
  • Online:2022-03-17

摘要: 现有的视频-文本跨模态实体分辨方法在视频处理上均采用均匀取帧的方法,必然导致视频信息的丢失,增加问题的复杂度。针对这一问题,提出一种结合关键帧提取的视频-文本跨模态实体分辨双重编码方法(DEIKFE)。以充分保留视频信息表征为前提,设计关键帧提取算法提取视频中的关键帧,获得视频关键帧集合表示。对于视频关键帧集合和文本,采用多级编码的方法,分别提取表征视频和文本的全局、局部和时序的特征,将其进行拼接形成多级编码表示。将该编码表示映射至共同嵌入空间,采用强负样本跨模态三元组损失对模型参数进行优化,使得匹配的视频-文本相似度越大,而不匹配的视频-文本相似度越小。通过在MSR-VTT、VATEX两个数据集上进行实验验证,与现有方法进行对比,在总体性能R@sum上分别提升了9.22%、2.86%,证明了该方法的优越性。

关键词: 跨模态实体分辨, 关键帧提取, 共同嵌入空间, 双重编码, 强负样本

Abstract: Existing video-text cross-modal entity resolution methods all adopt a method of uniformly extracting frames in video processing,which inevitably leads to the loss of video information and increases the model complexity.A dual encoding integrating key frame extraction (DEIKFE) is proposed for video-text cross-modal entity resolution. On the premise of fully retaining the video information,a key frame extraction algorithm is designed to extract the key frames in the video,which makes up the video key frame set. For the video key frame set and the text,a multi-level encoding method is adopted to extract the global,local,and time-series features,which are spliced to form a multi-level encoding representation. And the encoding representation is mapped into a common embedding space,and the model parameters are optimized by cross-modal triplet ranking loss based on the hard negative sample to make the matched video-text similarity greater and the unmatched video-text similarity smaller. The experiments on MSR-VTT and VATEX datasets show that the overall performance of R@sum is increased by 9.22% and 2.86%,respectively,comparedwith the existing methods, which can fully demonstrate the superiority of the proposed method.

Key words: cross-modalentityresolution, keyframeextraction, commonembeddingspace, dualencoding, hardnegativesample

中图分类号: