基于人脸表情、身体姿态和语音的多模态情感识别方法研究

项目名称： 基于人脸表情、身体姿态和语音的多模态情感识别方法研究

项目编号： No.61501249

项目类型： 青年科学基金项目

立项/批准年度： 2016

项目学科： 无线电电子学、电信技术

项目作者： 闫静杰

作者单位： 南京邮电大学

项目金额： 19万元

中文摘要： 情感识别是情感计算、模式识别、人工智能和社交信号处理等领域的重要研究课题。传统的情感识别研究主要集中于人脸表情识别、语音情感识别和身体姿态情感识别等单模态情感识别，但人类是以多模态的方式来表达内心的情感，因此，研究多模态情感识别具有重要的理论意义和应用价值。本项目旨在申请人原有单模态情感识别和双模态情感识别研究成果的基础上，进一步开展基于人脸表情、身体姿态和语音的多模态情感识别关键技术研究。本项目的主要内容包括:（1）建立基于人脸表情、身体姿态和语音的多模态情感数据库；（2）研究人脸表情、身体姿态和语音三种模态的情感特征提取，提出基于Harris空时特征的视频情感特征提取方法；（3）研究三种模态的情感特征融合，提出稀疏多类减秩回归融合方法和稀疏多类核减秩回归融合方法；（4）建立基于人脸表情、身体姿态和语音的多模态情感识别系统。

中文关键词： 多模态情感识别；特征提取；特征融合

英文摘要： Emotion recognition is the main research issue of affective computing, pattern recognition, artificial intelligence, social signal processing and so on. The traditional research of emotion recognition mainly focuses on facial expression recognition, speech emotion recognition, body gesture emotion recognition and other monomodal emotion recognitions, but humans utilize the multimodal pattern to express inner emotion. Therefore, the research of multimodal emotion recognition has the important theoretical significance and practical application. On the base of the original research results of monomodal emotion recognition and bimodal emotion recognition, this project will further conduct key technology research of multimodal emotion recognition based on facial expression, body gesture and speech. The main contents of this project include: (1) Establishing the multimodal emotion database based on facial expression, body gesture and speech; (2) Researching the emotion feature extraction of the facial expression, body gesture and speech modality, as well as proposing the method of video emotion feature extraction based on the harris spatio-temporal feature; (3) Researching the emotion feature fusion of three modalities and proposing the sparse multiset reduced rank regression fusion method and sparse multiset kernel reduced rank regression fusion method; (4) Establishing the multimodal emotion recognition system based on facial expression, body gesture and speech.

英文关键词： Multimodal emotion recognition;Feature extraction;Feature fusion