Morality plays an important role in culture, identity, and emotion. Recent advances in natural language processing have shown that it is possible to classify moral values expressed in text at scale. Morality classification relies on human annotators to label the moral expressions in text, which provides training data to achieve state-of-the-art performance. However, these annotations are inherently subjective and some of the instances are hard to classify, resulting in noisy annotations due to error or lack of agreement. The presence of noise in training data harms the classifier's ability to accurately recognize moral foundations from text. We propose two metrics to audit the noise of annotations. The first metric is entropy of instance labels, which is a proxy measure of annotator disagreement about how the instance should be labeled. The second metric is the silhouette coefficient of a label assigned by an annotator to an instance. This metric leverages the idea that instances with the same label should have similar latent representations, and deviations from collective judgments are indicative of errors. Our experiments on three widely used moral foundations datasets show that removing noisy annotations based on the proposed metrics improves classification performance.
翻译:在文化、身份和情感方面,道德感在文化、身份和情感方面起着重要作用。在自然语言处理方面的最近进展表明,有可能对以比例表示的道德价值进行分类。道德感的分类依靠人文批注者在文本中标注道德表现,这为达到最先进的性能提供了培训数据。然而,这些说明本质上是主观性的,有些情况很难分类,由于错误或缺乏一致意见而导致语义不清。培训数据中的噪音损害了分类者准确识别文本道德基础的能力。我们建议了两种标准来审计注释的噪音。第一种标准是实例标签的催化剂,这是说明者对应如何标注的争议的代号。第二个标准是说明者指派给某个实例的标签的静音系数。这一指标利用了一种想法,即同一标签的例子应该具有类似的潜在代表性,而偏离集体判断则是错误的标志。我们在三个广泛使用的道德基础数据集上进行的实验表明,根据拟议的指标消除了音调的注释可以改进性能。