Topic modelling

Literature:

  • Ch. 17 Zhai 2016.
    • Focus on pp. 330-340
    • and section 17.6 on evaluation, pp. 383-384
  • David M Blei. Probabilistic Topic Models. Communiciations of the ACM, April 2012, Vol. 55 No. 4, Pages 77-84.

Video

Difference with clustering

Clustering is about discovering structure in terms of relationships in some unstructured data collection.

Topic modeling is instead about discovering main themes in an unstructured collection of items. We can indeed organize a collection according to the discovered themes, but finding the themes is the main point, not the grouping. These "themes" are not necessarily what we normally understand to be a topic. It may be some sub-structure that is not necessarily obvious to the human reader.

Probabilistic modeling

Topic modeling is an example of probabilistic modeling where we assume a generative process that we express in terms of hidden variables.

In topic modelling, usually a hidden topic is indicated with a $\theta$.

A topic $\theta$ is a distribution over word vectors from the same vocabulary, where different topics assign different probabilities to particular words.

In a simple Naive Bayes model, each document is "generated" by a single latent topic only. We can also make more complex models where documents can contain multiple topics (that we can weigh with weight).

In that case, we have our k topic { $\theta_1,...,\theta_k$ } and weights { $\pi_{i1},...,\pi_{ik}$ }.

The weights have two indices where $k$ indicates the topic and $i$ the document, meaning we have KxN weights with N documents. So each document will have weights for each topic, e.g. for document 1 we we may have $\theta_{11}=0.3$, $\theta_{12}=0.12$, ... , $\theta_{1k}=0.08$.

Q: How do we determine the topics?

Q: How do we determine the weights?

An article is expressed as a distribution over topics; a topic is a distribution over words.

Latent Dirichlet Analysis additionally imposes a prior on distribution of topics in the document, and a prior on distribution of words within the document (not topic?).

Links

Also have a look here: https://www.youtube.com/watch?v=NYkbqzTlW3w&feature=youtu.be