TxMM Lecture 7: Sentiment Analysis

  • Types
  • Evaluation SA
  • SA system for Twitter
  • Demos and software

Goal that you know how to design a SA application, but also set up an experiment using it.

Literature:

  • Ch.1 & Ch.2 from Bing Liu. Sentiment Analysis and Opinion Mining, 2012
  • Knowledge clip by Martha

TODO:

  • ponyland.science.ru.nl Sign up for a cluster; deadline 30 October.

Definition SA and Affective Computing

The booming of this research field since the 2000s is strongly tied to the rise of social media as a place for expressing personal and subjective opinions.

p.6 for definition.

"Sentiment analysis, also called opinion mining, is the field of study that analyzes people's opinions, sentiments, evaluations, appraisals, attitudes, and emotions towards entities such as products, services, organizations, individuals, issues, event, topics, and their attributes." (p.7 Liu 2012)

One may specialize this towards discussing expert opinions, but that's for another lecture.

Sentiment expressions

Subjective language use expresses something of an internal private state. But when is it affective?

"I think he went home" is subjective, but this expresses no sentiment.

"He broke my heart" is also subjective, but this has a clear affective load.

Mood, emotions, sentiment

  • Emotion = short term, reaction to trigger
  • Mood = long term, less specific, not dependent on trigger
  • Sentiment = long term + target = opinion.

Typical opinion/SA search queries

  1. Find an opinion of some person or organization on some topic (focus on who holds the opinion)
  2. Find positive/negative sentiments on
    • Customer opinions on a product
  3. Detect how opinions change over time
  4. Comparative opinions

Can we use a search engine for 1?

If a person is well-known and their are documents about that person (e.g.Mark Rutte's position on tax dividend), then it may be possible to answer this question with a search machine. The answer is likely to be found in a single document (e.g. a news paper article).

Who is interested in SA?

  • Businesses and organization; product and service benchmarking
  • Market intelligence
    • Busineness spends a lot of money on finding consumer opinions etc.
  • Policy making
  • Individual interested in other's opinions, e.g. when purchasing a product.
  • Search engines e.g., Placing ads

Levels of Sentiment Analysis

  • Document level, e.g. product reviews
  • Sentence level; subjective classification (distinguish objective and subjective sentences)
  • Entity and Aspect level: relate sentiment to features of a product, event or entity

Feature based opinion mining, goal is to find quintuple (E A S H C)

  • E: Target of opinion; entity, event, topic
  • A: Aspect or feature of E
  • S: Opinion Content (sentiment score of A)
  • H: ...
  • C: ...

Detecting E A S H C

  • E: use Named Entity Recognition/Event Detection
  • A: info extraction/aspect categorization.
    • Can be explicit or implicit

This camera is expensive -> implicit aspect 'price'

E and A are highly domain dependent.

  • S: sentiment classification
  • H: opinion holder
    • In tweets and reviews, usually the author
    • If present in text: Named Entity Recognition
  • T: context; TODO, too slow.

Sentiment Scores

Common labels:

  • Negative
  • Positive
  • Neutral (can have several meanings)
    • Unknown
    • NA
    • Actually neutral

Alternatives:

  • Objective vs. subjective
  • Emotion classes: joy, fear, anger, sadness
  • Ordinal Scales 1-10
  • Likert 1-5

Q: Why can we not use a scale from 1-100? A: Because it is already difficult to asses on a five point scale what 2,3,4 means etc. People are better at assessing pairwise opinions that can be aggregrated into a five point scale. But 1000 values is too fine-grained. Human judments do not agree enough, and the difference betweeen 64 and 65 is completely unclear w.r.t. sentiments.

Q: To which degree would human annotators agree of cases like these? A: This task is actually very hard and people usually do not agree that much.

Ordinal scale

Because ordinal is ranked; we can express the strength of the mistake.

Ordinal regression can use or SVMs for ordinal regression can use this extra information.

Polarity/Subjectivity/Intensity Scale

Didn't quite get the meaning of this graph. What are we supposed to learn here exactly?

SA Challenges

  1. Sentiment words do not always express a sentiment

Entity+sentiment is not enough.

"If I can find a good camera in the shop, I will buy it."

But we cannot just ignore e.g. conditionals, cf.:

"If you are looking for a good car, get an Audio"

  1. Sentiment words are ambiguous, context- and domain dependent

This camera sucks vs. this vacuum cleaner sucks.

crazy can mean (insane) or "crazy good".

  1. Sarcasm

"I just love going to the dentist!"

  1. Objective sentences may still express sentiments

"The washing machine uses a lot of water"

  1. Negations and their scope

"I do not think he is good, and Ana neither"

"I think that he is not good"

"I do not think he is good"

A practical solution is to append a _NEG suffix to each word between an negation and a clause-level punctuation mark.

SA Evaluation

We usually only care about positive and negative sentiments; so we ignore neutral statements.

We do not compute accuracy because of that reason; because then we take into account also the unknown cases.

The F-score is now computed only using the positive and negative cases.

Experimental setup

  • Data cleaning and normalization
  • NLP: tokenization and POS-tagging
  • Feature extraction; feature vectors for each tweet
  • Train machine learning classifier
  • Try out some variations on a development set (?)
  • Evaluation: average F-score on positive and negative tweets
  • Reflection: analysis of outomces, errors, ablation test.
    • Ablation: remove features one by one to see which drop the F-score

Reflection:

  • What's the difference between features and observable properties in the text that express sentiment?
    • what type of properties are hard to capture in the form of features?
  • Who is the opinion holder? What is the target?
  • How useful is your analysis?