WEClustering: word embeddings based text clustering technique for large datasets | Mehta, Bawa, Singh 2021

Complex Intelligent Systems, 2021 Sep 7: 1-14. doi: https://link.springer.com/article/10.1007%2Fs40747-021-00512-9

This paper proposes a clustering technique based on word embeddings (extracted from BERT) for 1) dealing with sparse but highly dimensional textual datasets and 2) with large datasets. Traditional clustering techniques struggle or are simply unsuitable in this situation.

Introduction

Document clustering is typically done on using Bag of Words (BOW) models, typically with TF-IDF scoring. This approach has two major limitations:

  1. These models use plain statistical measures (e.g. word occurrence frequency) and hence do not take into account semantics i.e. (word) meaning
    • So no way to handle polysemy, nor synonymy
  2. Suffers from the curse of dimensionality
    • Size of the vector representation is often determined by the vocabulary size, which can increase into the millions for very large data sets
    • Additionally, this will make these representations increasingly sparse (cf. Zipfs law)

This paper instead examines how word embeddings can tackle these issues.

Word embeddings:

  • "distributed (dense) representation of words using real numbers instead of the discrete representation using 0’s and 1’s." (2)
  • dimensionality between 100s to 1000s
  • Idea behind original Word2Vec: "optimize an objective function such that the probability of a central word in a context window of a fixed size m is maximized." (2)
    • Other algorithms: Glove, FastText
    • These provided a fixed embedding for each word (local word context is used computing the embedding, but we do not have different embeddings for different use contexts!)
  • BERT instead generates contextual embeddings

The proposed method clusters based on these contextual embeddings and thus has some claim to being a "context-sensitive semantic approach" (2).

Background

Explanation of BERT and various clustering methods.

Research papers on text clustering & BERT embeddings:

22: Li Y, Cai J, Wang J (2020) A text document clustering method based on weighted Bert model. In: 2020 IEEE 4th information

35: Park J, Park C, Kim J, Cho M, Park S (2019) Adc: advanced document clustering using contextualized representations. Expert Syst Appl 137:157–166

43: Shi H, Wang C, Sakai T (2020) Self-supervised document clustering based on bert with data augment. arXiv preprint arXiv:2011.08523

WEClustering technique

  • "combines the semantic advantages of the contextual word embeddings derived from the BERT model with statistical scoring mechanisms." (4)
    • Capture the statistical importance of a word with TF-IDF scoring
    • Capture context-based semantics with BERT embeddings

Five phases:

  1. Preprocesing
    • lower case (depends on the BERT model; can also handle cased text)
    • split on sentences: a sentence is used as the relevant word context
  2. Extract & filtering of embeddings (size 1024)
    • Remove embeddings corresponding to digits, punctuation and stop words
  3. Cluster on word embeddings to form "concepts" to be used for representing documents
    • Arrange all word vectors across your corpus as a N x 1024 matrix (where N is the vocabulary size)
    • Minibatch-K means on this matrix to form clusters of words, colloquially named "concepts" in this paper.
    • Determine K using the Elbow method
    • The goal is to have document representations in the end, so the "idea is to use these concepts as the new vocabulary (or features) instead of individual words as a vocabulary to represent any document." (5)
  4. Create a concept-document matrix (CD)
    • Each concept will be scored on its relevance to a particular document
    • Relevance of a concept for a document is determined by summing the TF-IDF values in that document for the words belonging to the concept.
  5. Cluster again on the concept-document matrix to get a document clustering

So the dimensionality reduction strategy is to first extract a limited amount of concepts, represent documents in terms of those concepts, and then cluster on the lower dimensional CD matrix.

Details & Analysis

Used clustering metrics based on BoW:

  • K-means
  • Agglomerative clustering
  • Hierarhical Density-based spatial clustering of applications (HDBSCAN))
  • Genie
  • Disambiguiated Core Semantics (DCS); uses WordNet in combination with K-means
  • Stamantic Clustering (STC): TF-IDF + semantic relations from WordNet

Performance metrics:

  • Silhouette: measure of how dense and separated clusters are
    • $s = (b-a)/max(b-a))$
    • a: average distance between sample and other points in the next nearest cluster
    • range [-1, 1] (inclusive)
  • Adjusted Rand Index (ARI)
    • when true labels are available
    • Measure clustering assignments while ignoring the permutations of the same clustering
  • Purity
    • "calculates the quality of clustering by first assigning all the data points in a cluster to the class for which the max- imum number of data points are present in this cluster. This is done and summed over all the clusters and then normalized by the total number of data points " (8), cf. Manning CD, Schütze H, Raghavan P (2008) Introduction to infor- mation retrieval. Cambridge University Press, Cambridge

On average, the proposed WEClustering outperformed all other clustering methods on all metrics.

Critical remarks

However, I find the comparison with the other methods quite unfair. WEClustering solves issues with high dimensionality by creating document representations in lower dimensionality. This is essentially a word-embedding based dimensionality reduction technique. However, they compare performance with clustering methods without any dimensionality reduction. So a fairer comparison would be to also apply a common dimensionality reduction technique such as PCA in combination with the other clustering methods. I did this myself in my thesis project and this tends to increase the performance, especially for clustering metrics such as K-means that suffer very significantly from the curse of dimensionality because of using Euclidean-like distance metrics.

The reported performance increase (for larger datasets a minimum improvement of up to 90%) may thus be exaggerated.

BERT embeddings are contextual, but one of the main claims of the authors is to deal with polysemy. There are no methods and results that actually support this claim, so it's wishful thinking at this stage.