PACE: Predictive and Contrastive Embedding for Unsupervised Action

Jiahao Wang, Jie Qin, Yunhong Wang, Annan Li, IJCAI 2022

• We are the first to exploit both predictability and similarity information for unsupervised action segmentation. By leveraging their complementarity, we overcome the limitations of existing clustering- and prediction-based methods.
• We design a unified framework to simultaneously learn predictive and contrastive embeddings, based on which accurate action boundaries are detected.
• Extensive experiments on three challenging benchmarks demonstrate the superiority of PACE, with up to 26.9% improvements in F1-score over the state of the art.

Few-shot Fine-grained Action Recognition via Bidirectional Attention and Contrastive Meta-learning

Jiahao Wang, Yunhong Wang, Sheng Liu, Annan Li, ACM Multimedia 2021

Project

• We propose the few-shot fine-grained action recognition problem, which is spawned from real-world demands. A novel framework is devised to accurately recognize unseen f ine-grained actions with few support samples.
• We combine task-driven and saliency-supervised signals to capture subtle action details, simulating the top-down and bottom-up attention mechanism of human vision.
• To generate discriminative representations for low interclass variance data, we propose contrastive meta-learning, making full use of potential contrastive pairs.

Bidirectional Maximum Entropy Training with Word Co-occurrence for Video Captioning

Sheng Liu, Annan Li, Jiahao Wang, Yunhong Wang, IEEE TMM

• We are the first to introduce the maximum entropy RL theory and probability distribution optimization into the video captioning task. The proposed BME method, which can be employed in a plug-and-play manner, enlarges the search space for RL algorithms and maximizes the relative entropy between neighboring words to increase the diversity of descriptions.
• To the best of our knowledge, this is the first work to exploit word co-occurrence in the training process of video captioning. It enables the model to pay different attention to neighboring word distributions at the phrase level, resulting in more human-like captions.
• Extensive experiments on MSVD and MSR-VTT demonstrate the effectiveness of our approach, by achieving competitive results in different metrics. Further studies show that our method significantly increases the diversity of descriptions compared with the baseline models.

EvCap: Element-Aware Video Captioning

Sheng Liu, Annan Li, Yuwei Zhao, Jiahao Wang, Yunhong Wang, IEEE TCSVT

• We propose the idea of the element-aware usage of linguistic features in video captioning models which previously rely on the only visual representation which results in the caption that is easily polluted by hallucinations.
• We design an EvCap framework that consists of a multimodal multi-branch encoder-decoder framework and a postoperation fusion strategy to flexibly insert and aggregate additional element-related features besides visual representations.
• We conduct extensive experiments with CNNs, ViT and CLIP as the encoder on four representative datasets. The results demonstrate that EvCap can surpass most state-of-theart methods without training encoders, e.g. MSVD+10.9%, MSR-VTT+1.5%, VATEX+18.9% and TVC+4.7% on CIDEr score under ViT architectures. Furthermore, we also explore the performance of the multi-modal multi-feature framework w.r.t. the types and number of features adopted, e.g. visual and linguistic features, hoping to provide some insights for future model design.

Will You Ever Become Popular? Learning to Predict Virality of Dance Clips

Jiahao Wang, Yunhong Wang, Nina Weng, Tianrui Chai, Annan Li, Faxi Zhang, Sansi Yu, ACM TOMM

• We first study virality prediction from dance challenges using visual cues, which has great commercial value. To facilitate the research, we release VDV dataset, a large-scale multi-modal dance virality prediction benchmark.
• A multi-modal framework modeling both body movements and appearance dynamics is developed. For skeleton-based prediction, we devise a pyramidal skeleton graph convolutional network (PSGCN). For appearance-based prediction, relational temporal convolutional networks (RTCN) are proposed. An attentive modality fusion approach is introduced to aggregate predictions from multiple streams.

Atrous Temporal Convolutional Network for Video Action Segmentation

Jiahao Wang, Zhengyin Du, Annan Li, Yunhong Wang, ICIP 2019

• A novel encoder-decoder fully convolutional network that utilizes atrous temporal convolutions to explicitly extract contextual and local features for accurate temporal action segmentation.
• We propose atrous temporal pyramid pooling (ATPP), an effective and efficient way to capture multiscale action dependencies. To our best knowledge, this is the first work that introduces atrous temporal convolutions to tackle the action segmentation problem in a multiscale manner.
• Experimental results demonstrate that our model achieves improvements over the state of the art on 50 Salads, JIGSAWS and GTEA datasets.