Co-training Transformer with Movies and Photos Improves Motion Recognition

Motion recognition has develop into a significant focus space for the analysis neighborhood as a result of many functions can profit from improved modeling, reminiscent of video retrieval, video captioning, video question-answering, and many others. Transformer-based approaches have lately demonstrated state-of-the-art efficiency on a number of benchmarks. Whereas Transformer fashions require information to be taught higher visible priors in comparison with ConvNets, motion recognition datasets are comparatively small in scale. Giant Transformer fashions are usually first skilled on picture datasets and later fine-tuned on a goal motion recognition dataset.

Whereas the present pre-training and fine-tuning motion recognition paradigm is simple and manifests robust empirical outcomes, it might be overly restrictive for constructing general-purpose action-recognition fashions. In comparison with a dataset like ImageNet that covers a wide range of object recognition courses, motion recognition datasets like Kinetics and One thing-One thing-v2 (SSv2) pertain to restricted subjects. For instance, Kinetics embrace object-centric actions like “cliff diving” and “mountaineering’ whereas SSv2 accommodates object-agnostic actions like ’pretending to place one thing onto one thing else.’ In consequence, we noticed poor efficiency adapting an motion recognition mannequin that has been fine-tuned on one dataset to a different disparate dataset.

Variations in objects and video backgrounds amongst datasets additional exacerbate studying a general-purpose motion recognition classification mannequin. Even though video datasets could also be growing in measurement, prior work suggests vital information augmentation and regularization is important to realize robust efficiency. This latter discovering could point out the mannequin rapidly overfits on the goal dataset, and in consequence, hinders its capability to generalize to different motion recognition duties.

In “Co-training Transformer with Movies and Photos Improves Motion Recognition”, we suggest a coaching technique, named CoVeR, that leverages each picture and video information to collectively be taught a single general-purpose motion recognition mannequin. Our method is buttressed by two essential findings. First, disparate video datasets cowl a various set of actions, and coaching them collectively in a single mannequin may result in a mannequin that excels at a variety of actions. Second, video is an ideal supply for studying movement info, whereas pictures are nice for exploiting structural look. Leveraging a various distribution of picture examples could also be useful in constructing strong spatial representations in video fashions. Concretely, CoVeR first pre-trains the mannequin on a picture dataset, and through fine-tuning, it concurrently trains a single mannequin on a number of video and picture datasets to construct strong spatial and temporal representations for a general-purpose video understanding mannequin.

Structure and Coaching Technique

We utilized the CoVeR method to the lately proposed spatial-temporal video transformer, referred to as TimeSFormer, that accommodates 24 layers of transformer blocks. Every block accommodates one temporal consideration, one spatial consideration, and one multilayer perceptron (MLP) layer. To be taught from a number of video and picture datasets, we undertake a multi-task studying paradigm and equip the motion recognition mannequin with a number of classification heads. We pre-train all non-temporal parameters on the large-scale JFT dataset. Throughout fine-tuning, a batch of movies and pictures are sampled from a number of video and picture datasets. The sampling price is proportional to the scale of the datasets. Every pattern inside the batch is processed by the TimeSFormer after which distributed to the corresponding classifier to get the predictions.

In contrast with the usual coaching technique, CoVeR has two benefits. First, because the mannequin is straight skilled on a number of datasets, the discovered video representations are extra basic and will be straight evaluated on these datasets with out further fine-tuning. Second, Transformer-based fashions could simply overfit to a smaller video distribution, thus degrading the generalization of the discovered representations. Coaching on a number of datasets mitigates this problem by lowering the chance of overfitting.

CoVeR adopts a multi-task studying technique skilled on a number of datasets, every with their very own classifier.

Benchmark Outcomes

We consider the CoVeR method to coach on Kinetics-400 (K400), Kinetics-600 (K600), Kinetics-700 (K700), SomethingSomething-V2 (SSv2), and Moments-in-Time (MiT) datasets. In contrast with different approaches — reminiscent of TimeSFormer, Video SwinTransformer, TokenLearner, ViViT, MoViNet, VATT, VidTr, and OmniSource — CoVeR established the brand new state-of-the-art on a number of datasets (proven beneath). Not like earlier approaches that prepare a devoted mannequin for one single dataset, a mannequin skilled by CoVeR will be straight utilized to a number of datasets with out additional fine-tuning.

Switch Studying

We use switch studying to additional confirm the video motion recognition efficiency and examine with co-training on a number of datasets, outcomes are summarized beneath. Particularly, we prepare on the supply datasets, then fine-tune and consider on the goal dataset.

We first think about K400 because the goal dataset. CoVeR co-trained on SSv2 and MiT improves the top-1 accuracy on K400→K400 (the place the mannequin is skilled on K400 after which fine-tuned on K400) by 1.3%, SSv2→K400 by 1.7%, and MiT→K400 by 0.4%. Equally, we observe that by transferring to SSv2, CoVeR achieves 2%, 1.8%, and 1.1% enchancment over SSv2→SSv2, K400→SSv2, and MiT→SSv2, respectively. The 1.2% and a couple of% efficiency enchancment on K400 and SSv2 signifies that CoVeR co-trained on a number of datasets may be taught higher visible representations than the usual coaching paradigm, which is helpful for downstream duties.

Comparability of switch studying the illustration discovered by CoVeR and customary coaching paradigm. A→B means the mannequin is skilled on dataset A after which fine-tuned on dataset B.

Conclusion

On this work, we current CoVeR, a coaching paradigm that collectively learns motion recognition and object recognition duties in a single mannequin for the aim of developing a general-purpose motion recognition framework. Our evaluation signifies that it might be useful to combine many video datasets into one multi-task studying paradigm. We spotlight the significance of continuous to be taught on picture information throughout fine-tuning to take care of strong spatial representations. Our empirical findings recommend CoVeR can be taught a single general-purpose video understanding mannequin which achieves spectacular efficiency throughout many motion recognition datasets with out a further stage of fine-tuning on every downstream software.

Acknowledgements

We wish to thank Christopher Fifty, Wei Han, Andrew M. Dai, Ruoming Pang, and Fei Sha for preparation of the CoVeR paper, Yue Zhao, Hexiang Hu, Zirui Wang, Zitian Chen, Qingqing Huang, Claire Cui and Yonghui Wu for useful discussions and feedbacks, and others on the Mind Group for help all through this mission.