## Linguistically Regularized LSTMs for Sentiment Classification

2018 年 5 月 4 日 黑龙江大学自然语言处理实验室 西土城的搬砖日常

# 《Linguistically Regularized LSTMs for Sentiment Classification》阅读笔记

## 一、相关工作

1、情感分类的神经网络模型

• 通过递归编码建立句子的语义表示，输入文本通常是树结构的，具体工作可参考：

1. [Socher et al. 2011] Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

2. [Socher et al. 2013] Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

3. [Dong et al. 2014] Adaptive Multi-Compositionality for Recursive Neural Models with Applications to Sentiment Analysis

4. [Qian et al. 2015] Learning Tag Embeddings and Tag-specific Composition Functions in Recursive Neural Network

• 通过CNN建立句子的语义表示，输入是文本序列，具体工作可参考：

1. [Kim 2014] Convolutional Neural Networks for Sentence Classification

2. [Kalchbrenner, Grefenstette, and Blunsom 2014] A Convolutional Neural Network for Modelling Sentences

• 通过LSTM模型建立句子的语义表示，可以用在文本序列的建模上，也可以是树结构的输入，具体工作可参考：

1. [Zhu, Sobhani, and Guo 2015] Long Short-Term Memory Over Tree Structures

2. [Tai, Socher, and Manning 2015] Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

2、情感分析任务的语言学知识

• 情感词典（sentiment lexicon）

1. [Hu and Liu 2004]  Mining and Summarizing Customer Reviews

2. [Wilson, Wiebe, and Hoffmann 2005] Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis

• 否定词（negation）

1. [Polanyi and Zaenen 2006] Contextual Valence Shifters

2. [Taboada et al. 2011] Lexicon-Based Methods for Sentiment Analysis

3. [Zhu et al. 2014] An Empirical Study on the Effect of Negation Words on Sentiment

4. [Kiritchenko and Mohammad 2016] Sentiment Composition of Words with Opposing Polarities

• 程度副词 （intensity words）

1. [Taboada et al. 2011] Lexicon-Based Methods for Sentiment Analysis

2. [Wei, Wu, and Lin 2011] A regression approach to affective rating of chinese words from anew

3. [Malandrakis et al. 2013] Distributional semantic models for affective text analysis

4. [Wang, Zhang, and Lan 2016] Ecnu at semeval-2016 task 7: An enhanced supervised learning method for lexicon sentiment intensity ranking

## 二、本文提出的模型（Linguistically Regularized LSTM）

Linguistically Regularized LSTM是把语言学规则（包括情感词典、否定词和程度副词）以约束的形式和LSTM结合起来。作者定义了四种规则来将语言学知识结合进来，每一个规则主要是考虑当前词和它相邻位置词的情感分布来定义的。

1、Non-Sentiment Regularizer(NSR)

NSR的基本想法是，如果相邻的两个词都是non-opinion（不表达意见）的词，那么这两个词的情感分布应该是比较相近的。

KL散度是用来衡量两个函数或者分布之间的差异性的一个指标，其原始定义式如下：

2、Sentiment Regularizer(SR)

SR的基本想法是，如果当前词是情感词典中的词，那么它的情感分布应该和前一个词以及后一个词有明显不同。

SR定义如下：

3、Negation Regularizer(NR)

4、Intensity Regularizer(IR)

5、将上述规则结合到LSTM模型中

## 三、实验

1、数据

• Movie Reviews，包括negative和positive两类；

• Stanford Sentiment Treebank，包括very negative,negative,neural,positive,very positive五个类别。

2、实验结果

• LR-LSTM和BI-LSTM与标准LSTM相比都有较大提升；

• LR-BI-LSTM在句子级标注数据上的结果与BI-LSTM在短语级标注数据上的结果基本持平，通过引入LR,可以减少标注成本，并得到差不多的结果；

• 本文的LR-LSTM和LR-BI-LSTM和Tree-LSTM的结果基本持平，但是本文的模型更简单，效率更高，同时省去了短语级的标注工作。

3、不同规则的效果分析

• 在这些子数据集上，LR-Bi-LSTM的性能优于Bi-LSTM；

• 去掉NR或者IR的约束，在MR和SST两个数据集上模型性能都有明显下降。

### 相关内容

While the general task of textual sentiment classification has been widely studied, much less research looks specifically at sentiment between a specified source and target. To tackle this problem, we experimented with a state-of-the-art relation extraction model. Surprisingly, we found that despite reasonable performance, the model's attention was often systematically misaligned with the words that contribute to sentiment. Thus, we directly trained the model's attention with human rationales and improved our model performance by a robust 4~8 points on all tasks we defined on our data sets. We also present a rigorous analysis of the model's attention, both trained and untrained, using novel and intuitive metrics. Our results show that untrained attention does not provide faithful explanations; however, trained attention with concisely annotated human rationales not only increases performance, but also brings faithful explanations. Encouragingly, a small amount of annotated human rationales suffice to correct the attention in our task.

Aspect-based sentiment analysis (ABSA), which aims to identify fine-grained opinion polarity towards a specific aspect, is a challenging subtask of sentiment analysis (SA). In this paper, we construct an auxiliary sentence from the aspect and convert ABSA to a sentence-pair classification task, such as question answering (QA) and natural language inference (NLI). We fine-tune the pre-trained model from BERT and achieve new state-of-the-art results on SentiHood and SemEval-2014 Task 4 datasets.

The emerging technique of deep learning has been widely applied in many different areas. However, when adopted in a certain specific domain, this technique should be combined with domain knowledge to improve efficiency and accuracy. In particular, when analyzing the applications of deep learning in sentiment analysis, we found that the current approaches are suffering from the following drawbacks: (i) the existing works have not paid much attention to the importance of different types of sentiment terms, which is an important concept in this area; and (ii) the loss function currently employed does not well reflect the degree of error of sentiment misclassification. To overcome such problem, we propose to combine domain knowledge with deep learning. Our proposal includes using sentiment scores, learnt by regression, to augment training data; and introducing penalty matrix for enhancing the loss function of cross entropy. When experimented, we achieved a significant improvement in classification results.

Sentiment analysis is a widely studied NLP task where the goal is to determine opinions, emotions, and evaluations of users towards a product, an entity or a service that they are reviewing. One of the biggest challenges for sentiment analysis is that it is highly language dependent. Word embeddings, sentiment lexicons, and even annotated data are language specific. Further, optimizing models for each language is very time consuming and labor intensive especially for recurrent neural network models. From a resource perspective, it is very challenging to collect data for different languages. In this paper, we look for an answer to the following research question: can a sentiment analysis model trained on a language be reused for sentiment analysis in other languages, Russian, Spanish, Turkish, and Dutch, where the data is more limited? Our goal is to build a single model in the language with the largest dataset available for the task, and reuse it for languages that have limited resources. For this purpose, we train a sentiment analysis model using recurrent neural networks with reviews in English. We then translate reviews in other languages and reuse this model to evaluate the sentiments. Experimental results show that our robust approach of single model trained on English reviews statistically significantly outperforms the baselines in several different languages.

Most existing sentiment analysis approaches heavily rely on a large amount of labeled data that usually involve time-consuming and error-prone manual annotations. The distribution of this labeled data is significantly imbalanced among languages, e.g., more English texts are labeled than texts in other languages, which presents a major challenge to cross-lingual sentiment analysis. There have been several cross-lingual representation learning techniques that transfer the knowledge learned from a language with abundant labeled examples to another language with much fewer labels. Their performance, however, is usually limited due to the imperfect quality of machine translation and the scarce signal that bridges two languages. In this paper, we employ emojis, a ubiquitous and emotional language, as a new bridge for sentiment analysis across languages. Specifically, we propose a semi-supervised representation learning approach through the task of emoji prediction to learn cross-lingual representations of text that can capture both semantic and sentiment information. The learned representations are then utilized to facilitate cross-lingual sentiment classification. We demonstrate the effectiveness and efficiency of our approach on a representative Amazon review data set that covers three languages and three domains.

We propose a novel two-layered attention network based on Bidirectional Long Short-Term Memory for sentiment analysis. The novel two-layered attention network takes advantage of the external knowledge bases to improve the sentiment prediction. It uses the Knowledge Graph Embedding generated using the WordNet. We build our model by combining the two-layered attention network with the supervised model based on Support Vector Regression using a Multilayer Perceptron network for sentiment analysis. We evaluate our model on the benchmark dataset of SemEval 2017 Task 5. Experimental results show that the proposed model surpasses the top system of SemEval 2017 Task 5. The model performs significantly better by improving the state-of-the-art system at SemEval 2017 Task 5 by 1.7 and 3.7 points for sub-tracks 1 and 2 respectively.

Aspect based sentiment analysis (ABSA) can provide more detailed information than general sentiment analysis, because it aims to predict the sentiment polarities of the given aspects or entities in text. We summarize previous approaches into two subtasks: aspect-category sentiment analysis (ACSA) and aspect-term sentiment analysis (ATSA). Most previous approaches employ long short-term memory and attention mechanisms to predict the sentiment polarity of the concerned targets, which are often complicated and need more training time. We propose a model based on convolutional neural networks and gating mechanisms, which is more accurate and efficient. First, the novel Gated Tanh-ReLU Units can selectively output the sentiment features according to the given aspect or entity. The architecture is much simpler than attention layer used in the existing models. Second, the computations of our model could be easily parallelized during training, because convolutional layers do not have time dependency as in LSTM layers, and gating units also work independently. The experiments on SemEval datasets demonstrate the efficiency and effectiveness of our models.

Sentiment analysis is a key component in various text mining applications. Numerous sentiment classification techniques, including conventional and deep learning-based methods, have been proposed in the literature. In most existing methods, a high-quality training set is assumed to be given. Nevertheless, constructing a high-quality training set that consists of highly accurate labels is challenging in real applications. This difficulty stems from the fact that text samples usually contain complex sentiment representations, and their annotation is subjective. We address this challenge in this study by leveraging a new labeling strategy and utilizing a two-level long short-term memory network to construct a sentiment classifier. Lexical cues are useful for sentiment analysis, and they have been utilized in conventional studies. For example, polar and privative words play important roles in sentiment analysis. A new encoding strategy, that is, $\rho$-hot encoding, is proposed to alleviate the drawbacks of one-hot encoding and thus effectively incorporate useful lexical cues. We compile three Chinese data sets on the basis of our label strategy and proposed methodology. Experiments on the three data sets demonstrate that the proposed method outperforms state-of-the-art algorithms.

In this work, we present our findings and experiments for stock-market prediction using various textual sentiment analysis tools, such as mood analysis and event extraction, as well as prediction models, such as LSTMs and specific convolutional architectures.

This project addresses the problem of sentiment analysis in twitter; that is classifying tweets according to the sentiment expressed in them: positive, negative or neutral. Twitter is an online micro-blogging and social-networking platform which allows users to write short status updates of maximum length 140 characters. It is a rapidly expanding service with over 200 million registered users - out of which 100 million are active users and half of them log on twitter on a daily basis - generating nearly 250 million tweets per day. Due to this large amount of usage we hope to achieve a reflection of public sentiment by analysing the sentiments expressed in the tweets. Analysing the public sentiment is important for many applications such as firms trying to find out the response of their products in the market, predicting political elections and predicting socioeconomic phenomena like stock exchange. The aim of this project is to develop a functional classifier for accurate and automatic sentiment classification of an unknown tweet stream.

AINLP
13+阅读 · 2019年4月28日

3+阅读 · 2018年6月12日

8+阅读 · 2018年5月6日

10+阅读 · 2018年1月14日

4+阅读 · 2017年12月25日

6+阅读 · 2017年11月29日

22+阅读 · 2017年11月5日

4+阅读 · 2017年9月30日

Ruiqi Zhong,Steven Shao,Kathleen McKeown
5+阅读 · 2019年8月19日
Chi Sun,Luyao Huang,Xipeng Qiu
7+阅读 · 2019年3月22日
Khuong Vo,Dang Pham,Mao Nguyen,Trung Mai,Tho Quan
3+阅读 · 2018年6月22日
Ethem F. Can,Aysu Ezen-Can,Fazli Can
11+阅读 · 2018年6月8日
Zhenpeng Chen,Sheng Shen,Ziniu Hu,Xuan Lu,Qiaozhu Mei,Xuanzhe Liu
6+阅读 · 2018年6月7日
Abhishek Kumar,Daisuke Kawahara,Sadao Kurohashi
21+阅读 · 2018年5月25日
Wei Xue,Tao Li
12+阅读 · 2018年5月18日
Ou Wu,Tao Yang,Mengyang Li,Ming Li
6+阅读 · 2018年3月21日
Jordan Prosky,Xingyou Song,Andrew Tan,Michael Zhao
6+阅读 · 2018年1月18日
Afroze Ibrahim Baqapuri
3+阅读 · 2015年9月14日
Top