Linguistically Regularized LSTMs for Sentiment Classification

2018 年 5 月 4 日 黑龙江大学自然语言处理实验室 西土城的搬砖日常
Linguistically Regularized LSTMs for Sentiment Classification

文章来源知乎专栏:西土城的搬砖日常

原文链接:https://zhuanlan.zhihu.com/p/23906487


《Linguistically Regularized LSTMs for Sentiment Classification》阅读笔记

作者:Qiao Qian , Minlie Huang , Jinhao Lei , Xiaoyan Zhu 

出处:arXiv 2017

论文相关领域:情感分析

主要工作:通过改变损失函数的方式,把语言学规则(情感词典,否定词和程度副词)融入到现有的句子级LSTM情感分类模型中。


一、相关工作

1、情感分类的神经网络模型

近年来,出现了很多解决情感分类问题的神经网络模型,主要有:

  • 通过递归编码建立句子的语义表示,输入文本通常是树结构的,具体工作可参考:

  1. [Socher et al. 2011] Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

  2. [Socher et al. 2013] Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

  3. [Dong et al. 2014] Adaptive Multi-Compositionality for Recursive Neural Models with Applications to Sentiment Analysis

  4. [Qian et al. 2015] Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network


  • 通过CNN建立句子的语义表示,输入是文本序列,具体工作可参考:

  1. [Kim 2014] Convolutional Neural Networks for Sentence Classification

  2. [Kalchbrenner, Grefenstette, and Blunsom 2014] A Convolutional Neural Network for Modelling Sentences


  • 通过LSTM模型建立句子的语义表示,可以用在文本序列的建模上,也可以是树结构的输入,具体工作可参考:

  1. [Zhu, Sobhani, and Guo 2015] Long Short-Term Memory Over Tree Structures

  2. [Tai, Socher, and Manning 2015] Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks


2、情感分析任务的语言学知识

语言学知识对情感分析任务至关重要,主要包括情感词典、否定词(如not、nerer、neither等等)、程度副词(如very、extremely等)。


  • 情感词典(sentiment lexicon)

情感词典(英文)应用比较广泛的有Hu and Liu2004年提出的情感词典和MPQA词典,详细介绍可分别参考以下两篇文章:

  1. [Hu and Liu 2004]  Mining and Summarizing Customer Reviews

  2. [Wilson, Wiebe, and Hoffmann 2005] Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis


  • 否定词(negation)

否定词在情感分析中也是一个关键元素,它会改变文本表达的情感倾向。有很多相关研究:

  1. [Polanyi and Zaenen 2006] Contextual Valence Shifters

  2. [Taboada et al. 2011] Lexicon-Based Methods for Sentiment Analysis

  3. [Zhu et al. 2014] An Empirical Study on the Effect of Negation Words on Sentiment

  4. [Kiritchenko and Mohammad 2016] Sentiment Composition of Words with Opposing Polarities

其中,

文章1:对否定词的处理是将含有否定词的文本的情感倾向反转;

文章2:由于不同的否定表达以不同的方式不同程度影响着句子的情感倾向,文中提出否定词按照某个常量值的程度改变着文本的情感倾向;

文章3: 将否定词作为特征结合到神经网络模型中;

文章4: 将否定词和其他语言学知识与SVM结合在一起,分析文本情感倾向。


  • 程度副词 (intensity words)            

程度副词影响文本的情感强度,在细粒度情感中非常关键,相关研究可参考:

  1. [Taboada et al. 2011] Lexicon-Based Methods for Sentiment Analysis

  2. [Wei, Wu, and Lin 2011] A regression approach to affective rating of chinese words from anew

  3. [Malandrakis et al. 2013] Distributional semantic models for affective text analysis

  4. [Wang, Zhang, and Lan 2016] Ecnu at semeval-2016 task 7: An enhanced supervised learning method for lexicon sentiment intensity ranking

其中,

文章1:直接让程度副词通过一个固定值来改变情感强度;

文章2:利用线性回归模型来预测词语的情感强度值;

文章3:通过核函数结合语义信息预测情感强度得分;

文章4:提出一种learning-to-rank模型预测情感强度得分。


二、本文提出的模型(Linguistically Regularized LSTM)

Linguistically Regularized LSTM是把语言学规则(包括情感词典、否定词和程度副词)以约束的形式和LSTM结合起来。作者定义了四种规则来将语言学知识结合进来,每一个规则主要是考虑当前词和它相邻位置词的情感分布来定义的。


1、Non-Sentiment Regularizer(NSR)

NSR的基本想法是,如果相邻的两个词都是non-opinion(不表达意见)的词,那么这两个词的情感分布应该是比较相近的。


将这种想法结合到模型中,有如下定义:




其中M是边界参数是对称KL散度,是要预测的位置t处的词的分布,它的向量表示是


关于KL散度:

KL散度是用来衡量两个函数或者分布之间的差异性的一个指标,其原始定义式如下:



两个分布的差异越大,KL散度值越大;两个分布的差异越小,KL散度值越小;当两个分布相同时,KL散度值为0。


这里所用的对称KL散度定义如下:




所以我们可以看到,当相邻的两个词分布较近,KL散度小于M时,NSR的值为0;随着两个词的分布差异增大时,NSR值变大。


2、Sentiment Regularizer(SR)

SR的基本想法是,如果当前词是情感词典中的词,那么它的情感分布应该和前一个词以及后一个词有明显不同。


例如:This movie is interesting. 在位置t=4处的词“interesting”是一个表达正向情感的词,所以在t=4处的情感分布应该比t=3处要positive得多。这个叫做sentiment drift(情感漂流)。


为了将这种想法结合到模型中,作者提出一个极性漂流分布,情感词典中的每一类词,有一个漂流分布值。例如,情感词典中的词可能划分为以下几类:strong positive,weakly positive,weakly negative和strong negative,对于每一类情感词,有一个漂流分布,由模型学习得到。


SR定义如下:




所以我们可以看到,当前词t是情感词典中的词的时候,前一个位置t-1的情感分布加上漂流分布之后,如果与位置t的分布相近的话,SR值为0,随着其分布差异的增大,SR值增大。


3、Negation Regularizer(NR)

否定词通常会反转文本的情感倾向(从正向变为负向,或者是从负向变为正向),但是具体情况是跟否定词本身和它所否定的对象有关的。


例如:“not good”和“not bad”中,“not”的角色并不一样,前者是将正向情感转为负向情感,后者是将负向情感转为正向情感。


对否定词的处理,本文是这样做的:针对每一个否定词,提出一个转化矩阵,这个矩阵是模型自己学出来的。如果当前词是否定词的话,那么它的前一个词或者后一个词经过转化矩阵之后的分布与当前词的分布应该是比较近的。






我们可以看到,如果在当前词是否定词的情况下,如果它的前一个词或者后一个词与当前词的分布较近,那么NR的值比较小。


4、Intensity Regularizer(IR)

程度副词改变文本的情感强度(比如从positive变为very positive),这种表达对细粒度情感分析很重要。


程度副词对于情感倾向的改变和否定词很像,只是改变的程度不同,这里也是通过一个转化矩阵来定义IR的。如果当前词是程度副词的话,那么它的前一个词或者后一个词,经过转化矩阵之后得到的分布,应该与当前词的分布比较接近。


5、将上述规则结合到LSTM模型中

定义一个新的损失函数将上述规则结合到模型中,



其中,样本的实际分布,预测得到的样本分布,上述规则中的一个或者多个的组合,i是句子的索引,t是位置的索引。


模型训练的目标是最小化损失函数,让样本的实际分布与预测分布尽可能接近的同时,让模型符合上述四种规则。


三、实验

1、数据

本文实验用了两个数据集来验证模型性能,

  • Movie Reviews,包括negative和positive两类;

  • Stanford Sentiment Treebank,包括very negative,negative,neural,positive,very positive五个类别。


两个数据集的具体统计信息如下图所示,



2、实验结果



从实验结果可以看出,

  • LR-LSTM和BI-LSTM与标准LSTM相比都有较大提升;

  • LR-BI-LSTM在句子级标注数据上的结果与BI-LSTM在短语级标注数据上的结果基本持平,通过引入LR,可以减少标注成本,并得到差不多的结果;

  • 本文的LR-LSTM和LR-BI-LSTM和Tree-LSTM的结果基本持平,但是本文的模型更简单,效率更高,同时省去了短语级的标注工作。


3、不同规则的效果分析



从实验结果可以看出,NSR和SR对提升模型性能最重要,NR和IR对模型性能提升重要性没有那么强,可能是因为在测试数据中只有14%的句子中含有否定词,只有23%的句子含有程度副词。


为了进一步研究NR和IR的作用,作者在又分别在仅包含否定词的子数据集(Neg.Sub)和仅包含程度副词的子数据集(Int.Sub)上做了对比实验,实验结果如下图,



从实验结果可以看出,

  • 在这些子数据集上,LR-Bi-LSTM的性能优于Bi-LSTM;

  • 去掉NR或者IR的约束,在MR和SST两个数据集上模型性能都有明显下降。


四、总结

这篇文章通过损失函数将语言学规则引入现有的句子级情感分析的LSTM模型。在没有增大模型复杂度的情况下,有效的利用情感词典、否定词和程度副词的信息,在实验数据集上取得了较好效果。随着深度学习的发展,人们慢慢忽略了宝贵的经典自然语言资源,如何有效将这部分知识有效地融入到深度学习模型中是一个非常有意义的工作。




推荐阅读

基础 | TreeLSTM Sentiment Classification

基础 | 详解依存树的来龙去脉及用法

基础 | 基于注意力机制的seq2seq网络

原创 | Simple Recurrent Unit For Sentence Classification

原创 | Highway Networks For Sentence Classification


欢迎关注交流


登录查看更多
7

相关内容

情感分类是对带有感情色彩的主观性文本进行分析、推理的过程,即分析对说话人的态度,倾向正面,还是反面。它与传统的文本主题分类又不相同,传统主题分类是分析文本讨论的客观内容,而情感分类是要从文本中得到它是否支持某种观点的信息。

While the general task of textual sentiment classification has been widely studied, much less research looks specifically at sentiment between a specified source and target. To tackle this problem, we experimented with a state-of-the-art relation extraction model. Surprisingly, we found that despite reasonable performance, the model's attention was often systematically misaligned with the words that contribute to sentiment. Thus, we directly trained the model's attention with human rationales and improved our model performance by a robust 4~8 points on all tasks we defined on our data sets. We also present a rigorous analysis of the model's attention, both trained and untrained, using novel and intuitive metrics. Our results show that untrained attention does not provide faithful explanations; however, trained attention with concisely annotated human rationales not only increases performance, but also brings faithful explanations. Encouragingly, a small amount of annotated human rationales suffice to correct the attention in our task.

0
5
下载
预览

Aspect-based sentiment analysis (ABSA), which aims to identify fine-grained opinion polarity towards a specific aspect, is a challenging subtask of sentiment analysis (SA). In this paper, we construct an auxiliary sentence from the aspect and convert ABSA to a sentence-pair classification task, such as question answering (QA) and natural language inference (NLI). We fine-tune the pre-trained model from BERT and achieve new state-of-the-art results on SentiHood and SemEval-2014 Task 4 datasets.

0
7
下载
预览

The emerging technique of deep learning has been widely applied in many different areas. However, when adopted in a certain specific domain, this technique should be combined with domain knowledge to improve efficiency and accuracy. In particular, when analyzing the applications of deep learning in sentiment analysis, we found that the current approaches are suffering from the following drawbacks: (i) the existing works have not paid much attention to the importance of different types of sentiment terms, which is an important concept in this area; and (ii) the loss function currently employed does not well reflect the degree of error of sentiment misclassification. To overcome such problem, we propose to combine domain knowledge with deep learning. Our proposal includes using sentiment scores, learnt by regression, to augment training data; and introducing penalty matrix for enhancing the loss function of cross entropy. When experimented, we achieved a significant improvement in classification results.

0
3
下载
预览

Sentiment analysis is a widely studied NLP task where the goal is to determine opinions, emotions, and evaluations of users towards a product, an entity or a service that they are reviewing. One of the biggest challenges for sentiment analysis is that it is highly language dependent. Word embeddings, sentiment lexicons, and even annotated data are language specific. Further, optimizing models for each language is very time consuming and labor intensive especially for recurrent neural network models. From a resource perspective, it is very challenging to collect data for different languages. In this paper, we look for an answer to the following research question: can a sentiment analysis model trained on a language be reused for sentiment analysis in other languages, Russian, Spanish, Turkish, and Dutch, where the data is more limited? Our goal is to build a single model in the language with the largest dataset available for the task, and reuse it for languages that have limited resources. For this purpose, we train a sentiment analysis model using recurrent neural networks with reviews in English. We then translate reviews in other languages and reuse this model to evaluate the sentiments. Experimental results show that our robust approach of single model trained on English reviews statistically significantly outperforms the baselines in several different languages.

0
11
下载
预览

Most existing sentiment analysis approaches heavily rely on a large amount of labeled data that usually involve time-consuming and error-prone manual annotations. The distribution of this labeled data is significantly imbalanced among languages, e.g., more English texts are labeled than texts in other languages, which presents a major challenge to cross-lingual sentiment analysis. There have been several cross-lingual representation learning techniques that transfer the knowledge learned from a language with abundant labeled examples to another language with much fewer labels. Their performance, however, is usually limited due to the imperfect quality of machine translation and the scarce signal that bridges two languages. In this paper, we employ emojis, a ubiquitous and emotional language, as a new bridge for sentiment analysis across languages. Specifically, we propose a semi-supervised representation learning approach through the task of emoji prediction to learn cross-lingual representations of text that can capture both semantic and sentiment information. The learned representations are then utilized to facilitate cross-lingual sentiment classification. We demonstrate the effectiveness and efficiency of our approach on a representative Amazon review data set that covers three languages and three domains.

0
6
下载
预览

We propose a novel two-layered attention network based on Bidirectional Long Short-Term Memory for sentiment analysis. The novel two-layered attention network takes advantage of the external knowledge bases to improve the sentiment prediction. It uses the Knowledge Graph Embedding generated using the WordNet. We build our model by combining the two-layered attention network with the supervised model based on Support Vector Regression using a Multilayer Perceptron network for sentiment analysis. We evaluate our model on the benchmark dataset of SemEval 2017 Task 5. Experimental results show that the proposed model surpasses the top system of SemEval 2017 Task 5. The model performs significantly better by improving the state-of-the-art system at SemEval 2017 Task 5 by 1.7 and 3.7 points for sub-tracks 1 and 2 respectively.

0
21
下载
预览

Aspect based sentiment analysis (ABSA) can provide more detailed information than general sentiment analysis, because it aims to predict the sentiment polarities of the given aspects or entities in text. We summarize previous approaches into two subtasks: aspect-category sentiment analysis (ACSA) and aspect-term sentiment analysis (ATSA). Most previous approaches employ long short-term memory and attention mechanisms to predict the sentiment polarity of the concerned targets, which are often complicated and need more training time. We propose a model based on convolutional neural networks and gating mechanisms, which is more accurate and efficient. First, the novel Gated Tanh-ReLU Units can selectively output the sentiment features according to the given aspect or entity. The architecture is much simpler than attention layer used in the existing models. Second, the computations of our model could be easily parallelized during training, because convolutional layers do not have time dependency as in LSTM layers, and gating units also work independently. The experiments on SemEval datasets demonstrate the efficiency and effectiveness of our models.

0
12
下载
预览

Sentiment analysis is a key component in various text mining applications. Numerous sentiment classification techniques, including conventional and deep learning-based methods, have been proposed in the literature. In most existing methods, a high-quality training set is assumed to be given. Nevertheless, constructing a high-quality training set that consists of highly accurate labels is challenging in real applications. This difficulty stems from the fact that text samples usually contain complex sentiment representations, and their annotation is subjective. We address this challenge in this study by leveraging a new labeling strategy and utilizing a two-level long short-term memory network to construct a sentiment classifier. Lexical cues are useful for sentiment analysis, and they have been utilized in conventional studies. For example, polar and privative words play important roles in sentiment analysis. A new encoding strategy, that is, $\rho$-hot encoding, is proposed to alleviate the drawbacks of one-hot encoding and thus effectively incorporate useful lexical cues. We compile three Chinese data sets on the basis of our label strategy and proposed methodology. Experiments on the three data sets demonstrate that the proposed method outperforms state-of-the-art algorithms.

0
6
下载
预览

In this work, we present our findings and experiments for stock-market prediction using various textual sentiment analysis tools, such as mood analysis and event extraction, as well as prediction models, such as LSTMs and specific convolutional architectures.

0
6
下载
预览

This project addresses the problem of sentiment analysis in twitter; that is classifying tweets according to the sentiment expressed in them: positive, negative or neutral. Twitter is an online micro-blogging and social-networking platform which allows users to write short status updates of maximum length 140 characters. It is a rapidly expanding service with over 200 million registered users - out of which 100 million are active users and half of them log on twitter on a daily basis - generating nearly 250 million tweets per day. Due to this large amount of usage we hope to achieve a reflection of public sentiment by analysing the sentiments expressed in the tweets. Analysing the public sentiment is important for many applications such as firms trying to find out the response of their products in the market, predicting political elections and predicting socioeconomic phenomena like stock exchange. The aim of this project is to develop a functional classifier for accurate and automatic sentiment classification of an unknown tweet stream.

0
3
下载
预览
小贴士
相关资讯
一文读懂依存句法分析
AINLP
13+阅读 · 2019年4月28日
Jointly Improving Summarization and Sentiment Classification
黑龙江大学自然语言处理实验室
3+阅读 · 2018年6月12日
笔记 | Sentiment Analysis
黑龙江大学自然语言处理实验室
8+阅读 · 2018年5月6日
【推荐】用TensorFlow实现LSTM社交对话股市情感分析
机器学习研究会
10+阅读 · 2018年1月14日
论文浅尝 | Distant Supervision for Relation Extraction
开放知识图谱
4+阅读 · 2017年12月25日
Simple Recurrent Unit For Sentence Classification
哈工大SCIR
6+阅读 · 2017年11月29日
原创 | Attention Modeling for Targeted Sentiment
黑龙江大学自然语言处理实验室
22+阅读 · 2017年11月5日
Highway Networks For Sentence Classification
哈工大SCIR
4+阅读 · 2017年9月30日
相关VIP内容
专知会员服务
67+阅读 · 2020年5月31日
六篇 EMNLP 2019【图神经网络(GNN)+NLP】相关论文
专知会员服务
61+阅读 · 2019年11月3日
相关论文
Fine-grained Sentiment Analysis with Faithful Attention
Ruiqi Zhong,Steven Shao,Kathleen McKeown
5+阅读 · 2019年8月19日
Chi Sun,Luyao Huang,Xipeng Qiu
7+阅读 · 2019年3月22日
Khuong Vo,Dang Pham,Mao Nguyen,Trung Mai,Tho Quan
3+阅读 · 2018年6月22日
Ethem F. Can,Aysu Ezen-Can,Fazli Can
11+阅读 · 2018年6月8日
Zhenpeng Chen,Sheng Shen,Ziniu Hu,Xuan Lu,Qiaozhu Mei,Xuanzhe Liu
6+阅读 · 2018年6月7日
Abhishek Kumar,Daisuke Kawahara,Sadao Kurohashi
21+阅读 · 2018年5月25日
Wei Xue,Tao Li
12+阅读 · 2018年5月18日
Ou Wu,Tao Yang,Mengyang Li,Ming Li
6+阅读 · 2018年3月21日
Jordan Prosky,Xingyou Song,Andrew Tan,Michael Zhao
6+阅读 · 2018年1月18日
Afroze Ibrahim Baqapuri
3+阅读 · 2015年9月14日
Top