Object manipulation actions represent an important share of the Activities of Daily Living (ADLs). In this work, we study how to enable service robots to use human multi-modal data to understand object manipulation actions, and how they can recognize such actions when humans perform them during human-robot collaboration tasks. The multi-modal data in this study consists of videos, hand motion data, applied forces as represented by the pressure patterns on the hand, and measurements of the bending of the fingers, collected as human subjects performed manipulation actions. We investigate two different approaches. In the first one, we show that multi-modal signal (motion, finger bending and hand pressure) generated by the action can be decomposed into a set of primitives that can be seen as its building blocks. These primitives are used to define 24 multi-modal primitive features. The primitive features can in turn be used as an abstract representation of the multi-modal signal and employed for action recognition. In the latter approach, the visual features are extracted from the data using a pre-trained image classification deep convolutional neural network. The visual features are subsequently used to train the classifier. We also investigate whether adding data from other modalities produces a statistically significant improvement in the classifier performance. We show that both approaches produce a comparable performance. This implies that image-based methods can successfully recognize human actions during human-robot collaboration. On the other hand, in order to provide training data for the robot so it can learn how to perform object manipulation actions, multi-modal data provides a better alternative.
翻译:日常生活活动( ADLs) 中很大一部分天体操纵动作 。 在这项工作中, 我们研究如何使服务机器人能够使用人类多模式数据来理解天体操纵动作, 当人类在人- 机器人合作任务中执行这些动作时, 如何让它们识别这些动作 。 本研究中的多模式数据由视频、 手动数据、 手压模式所代表的应用力、 手指弯曲的测量、 作为人类主体收集的操作动作 。 我们研究了两种不同的方法 。 首先, 我们发现, 多模式信号( 动作、 手指弯曲和手压力) 能够被拆解成一组原始信号( 动作、 手指弯曲和手压力), 以及当人类在人- 机器人合作任务中执行这些原始数据被用来定义24个多模式的原始特征 。 原始特征反过来可以用作多模式信号的抽象表示, 并且用于行动识别。 在后一种方法中, 我们从数据分类前的深层神经网络中提取视觉特征。 视觉特征功能功能可以被解析成为一系列的数据 。 。 我们也可以在人类的统计操作中 。 我们对其它的操作进行 进行 进行 显示 。