ViMoNet：一种基于运动与视频的多模态视觉语言框架用于人类行为理解 (ViMoNet: A Multimodal Vision-Language Framework for Human Behavior Understanding from Motion and Video)

This study investigates how large language models (LLMs) can be used to understand human behavior using motion and video data. We think that mixing both types is essential to completely capture the nuanced movements and meanings of human actions, in contrast to recent models that simply concentrate on motion data or films. To address this, we provide ViMoNet, a straightforward yet effective framework for comprehending, characterizing, and deducing human action. ViMoNet employs a joint training strategy that leverages the advantages of two data types: detailed motion-text data, which is more exact, and generic video-text data, which is more comprehensive but less detailed. This aids in the model's acquisition of rich data regarding time and space in human behavior. Additionally, we provide a brand new dataset named VIMOS that contains a variety of films, motion sequences, instructions, and subtitles. We developed ViMoNet-Bench, a standardized benchmark with carefully labeled samples, to evaluate how well models understand human behavior. Our tests show that ViMoNet outperforms existing methods in caption generation, motion understanding, and behavior interpretation.

翻译：本研究探讨如何利用大语言模型（LLMs）结合运动与视频数据来理解人类行为。我们认为，与近期仅聚焦于运动数据或视频的模型不同，融合两种数据类型对于全面捕捉人类动作的细微运动及语义至关重要。为此，我们提出了ViMoNet——一个简洁而高效的框架，用于理解、描述和推断人类行为。ViMoNet采用联合训练策略，充分发挥两种数据类型的优势：精确度更高的详细运动-文本数据，以及覆盖更广但细节较少的通用视频-文本数据。这有助于模型获取人类行为在时间和空间上的丰富信息。此外，我们构建了一个全新的数据集VIMOS，其中包含多样化的视频、运动序列、指令及字幕。为评估模型对人类行为的理解能力，我们开发了ViMoNet-Bench标准化基准，该基准包含精心标注的样本。实验结果表明，ViMoNet在字幕生成、运动理解和行为解释任务上均优于现有方法。