Event Detection | Lecture by Florian Kunneman

Detecting what events? Some examples:

  • Temporal: they happen at a specific time.
  • They have a topic: what are they about?
  • Generating networks of people being mentioned
  • Distinguishing events from temporal data; e.g. tweets/second over time of "event hashtags".
    • E.g. the tweet "ontbijt" correlates with moments Dutch people have breakfast at; and can actually show this information.
  • Recognizing twitter bots by their periodicity of their tweet events

Event detection

The definition of event detection is quite broad.

  • Can be seen in linguistic terms, as something that is being described "that participates in an opposition structure in a given text" (Pustejovsky et al., 2003)
  • In terms of topic detection and tracking: "some unique thing that happens at some point in time" (Alan et al., 1998)
  • As something that may be discussed in the media as significant (McMinn et al., 2013)

So there is a movement from events in a more linguistic sense, towards the study of significant events, i.e. they are mundane. But when is it significant? When it might be discussed in the media (above).

Twitter

Event detection in twitter gives us an idea of what is going on in the world with instantaneous access (people tweet about things happening right now), and with some extras: opinions, emotions, anticipation, reflection, "anticipointment"

A usual pipeline:

  • Streaming API
  • Pre-processing
  • Detecting Events
  • Ranking Events
  • Summarizing Selected Events
  • Reporting Events

Closed-domain vs. open-domain event detection

Examples of closed-domain:

  • Earthquake detection via Twitter
  • Flooding event detection

Open-domain event detection operates without any knowledge beforehand. The only thing we can assume is that events are often associated with a limited set of terms, and that events have some beginning and end time. You can also mine for date expressions to find out in advance when events are going to happen.

Time Series Event Detection

Underlying idea: when something is going on people pay attention and talk/tweet about it.

Events as burstiness

Words have an average frequency of use. When a word is suddenly being used a lot, this is a signal that some event is going on.

You can indicate events with clusters of tweets at a particular time point. You can also cluster on Twitter component words, or segments of tweets.

You could also argue that certain events are tied to emotions. So if you mine for large bursts of a particular sentiment on twitter, this could also indicate an event.

Kunneman discusses a 2014 paper of his as an example.

Made a binary classifier to predict if a term will be "bursty" or not, using a Hidden Markov Model; and a Poisson probability.

When you find "bursty" candidates for a particular day, you can calculate the "bursty term similarity" for pairs of candidates.

The general idea is that when bursty terms co-occur they may be a good descriptor of an event. With some extra tricks: also measure the importance of words in a particular time frame of the day as a weight; and measure the similarity in specific subwindows with tf-idf weighting.

E.g. if the terms "Montenegro" and "bus accidents" are bursty on the same day, this may indicate that a bus accident happened in Montenegro.

Using this similarity measure, you can cluster terms from bursty tweets. Kunneman used 3-nearest neighbour; discard tweets not in any cluster.

We end up with thousands of clusters. (500+ per day) Now we rank them to determine which ones are most likely events.

We also care specifically about which events are significant, as said above. Terms themselves are not indicative enough that an event is significant.

Some features that can be used in a classifier for significant events:

  • number of different users mentioning the term
  • average number of event terms per tweet
  • average number of hashtags per tweet
  • etc.

Insights:

  • Bursty terms are a useful signal for a diversity of events
  • But in Kunneman's paper this method mostly converged on news articles, less on events where people get together

Note that a lot of modelling decisions were made, and that in general it's good to reflect on the effects of those modelling decisions.

Event detection: Information extraction

People tend to tweet about what they are looking forward to.

Kunneman & Van den Bosch, 2016.

Preprocessing

  1. Extraction of future date references.

Only "tomorrow" omitted in this study, as this would introduce too many expressions and noise.

Date taggers exist but the researchers found that they are more suitable for news articles. Also this research requires special focus on forward looking date expressions.

  1. Extraction of event terms

Extracting entities. One approach was to use Wikipedia entries for entity look-up. Wikipedia is fairly flexible in the sense that new pages will be made for new events, probably. Another simpler approach was to take any hashtag as an event term. This worked quite well and was used.

Output: date-event pairs.

Event Ranking

Apply lower threshold: date-term pairs only considered when they are mentioned at least n=5 times.

Date-term pairs are ranked on their fit, in terms of G2 log likelihood ratio. This formula has nice properties in this case. Many people tweet about personal events, but they are likely to get a low score here because dates and terms that co-occur often score higher. But these are indeed expected to co-occur often when they have broader social relevance. E.g. when people are tweeting about a football match, this match is one the same day for everyone.

They made a simple adjustment to G2 called G2u that gives a higher score when more people tweet about a certain term. (To avoid high scores to be dominated by advertisement agencies or something like that).

Event Clustering

You want to remove duplicate events as well as group different event descriptions for the same event.

Approach:

  • Agglomerative Hierarchical Clustering with similarity threshold of 0.7
  • Event tweets aggregated
  • tf*df weighting
  • cosine similarity

So for example the ranking 1) 2014-09-04 flikken 2) 2014-09-04flikken maastricht 3) 2014-09-04 maastricht; are now clustered into "2014-09-04 - flikken, seizoen, begint, maastricht, tros"

Evaluation

Precision@250; mutual F-score; Cohen's Kappa.

Recall-based evaluation is tricky because you do not have access to a full list of all events. The chosen approach is to take curated event calendars as a reference point. They include football games, game and movie releases, performances, celebrations etc. This also allows you to do the evaluation separately for different event categories.

Insight

  • Again open-domain event detection
  • Tendency towards social events
  • Nice variance of event types
  • Requires other sources to evaluate recall

Practical take-away:

  • Be critical of existing modules (e.g. for extracting date expressions) and do not be afraid to move away for them if your problem would benefit from this.

Quiz

  1. Why is the performance of any approach to event detection from Twitter rarely compared to other approaches? Give at least two reasons.
  2. Why is the Duplicate Event Rate (DER) a relevant evaluation metric?
  3. Name two drawbacks to the event detection corpora that are described
  4. Why do the authors choose to evaluate precision by comparing the detected events to Google queries and the New York Times archive?