Lecture 2 Text and Multimedia Mining: the NLP Pipeline

How do we go from textual information to numerical vectors that we can use in text mining algorithms?

  • Gather
  • Raw text: Clean, filter, convert
  • Clean text: Preprocess with NLP tools
  • Linguistically annotated text: Feature creation and selection
  • Feature vector

Raw text

We either have written or spoken material.

Written:

  • Digitized documents with optical character recognition
  • Born-Digital documents

The quality of OCR is not perfect, which introduces problems.

  1. Introduce spelling
  2. Introduction of words that weren't present in the original
  3. Deletion of words that were not recognized
  4. Loss of text formatting

But 2 and 3 are most severe and hard to counter.

Digital inputs are also not necessarily clean. We want to exclude:

  • Advertisements
  • Photos
  • Links
  • etc.

Storing clean text

Clean text: actual textual content separated from other information, uniformly represented in an appropriate character encoding.

However, we do not assume yet that there's no spelling errors and so on. That's all part of the NLP process. So no text normalization is done yet.

Character encoding

You need to take into account the character encoding of your data

There are 10 types of people in the world: those who understand binary, and those who don't.

Encodings:

ASCII: American Standard Code for Information Interchange

But obviously, you can only encode English this way. Initially, computers in other had their own encoding, but from the 80s there was work on a global standard: UNICODE

Ascii uses 8-bit, 7 to encode the character plus a bit for the stop sign. (I think the stop sign is there because ASCII originally was also used as a telegraph language?)

Unicode:

  • For all writing systems
  • +100.000 characters
  • Independent of platform, software
  • Descriptive representation of characters: "a+umlaut"
  • The actual interpretation of

UTF-8 is the most common Unicode implementation that is also compatible with ASCII (UTF-16, UTF-32). 1 bit from character of ASCII-table; 2-bits for more information.

UTF-8 has great both backward and forward compatibility.

Markup

A markup language is a set of conventions for encoding texts:

  • What is allowed?
  • What is required?
  • Markup format
  • Meaning of the markup

As a convention, a mark up language does not do anything in itself. It offers structure for other programs.

Processing clean text

Tokenization

  • Using regular expression
  • Easy for Western languages that use whitespace and interpunction as separators

  • A word is a delimited string of characters (vague def)

  • A token is one instance of a word occurring in a document
  • A type is an equivalence class of tokens

But how do we handle numerical data, parsing hyphens, dealing with emoticons.

Linguistically, compound words ("emailaders"; alles in het Duits lol), or ... (went fast.)

Sentence splitting: relatively easy because sentences usually start with a capital letter and end with a dot. (But again: German may complicate things here).

Tokenizer demo: which tokenizer works best?

https://text-processing.com/demo/tokenize/

PoS Tagsets

Universal POS Tagset

  • VERB
  • NOUN
  • PRON
  • ADJ
  • ADV
  • ADP (prepositions and postpositions)
  • CONJ
  • DET
  • NUM
  • PRT (particles or function words)
  • X: other; typos, abbreviations
  • . punctiation

The WSJ (Wall Street Journal) tagset has 37-PoS-tags, so is way more detailed.

https://nlpprogress.com : state of the art is 97.24% accuracy.

Unknown accuracy is accuracy on previously unseen words.

So PoS-tagging (for English) is now very good. However, this is token accuracy. If you start looking at sentences, almost half of the sentences contains at least one mistake. On top, this performance does not transfer to all text genres and corpora. Essential disambiguations are not(?) treated correctly almost always.

Lemmas

A lemma is a canonical form of the word.

Usually either the infinitive or nominal masculine singular form.

Stem

A cruder word of representing the word.

A stem is the part of a word that remains after cutting of affixes (inflections). This is not further analyzable and is morphologically simple. It captures the principle portion of the meaning of the word.

Parsing

Analyze the syntactic constituents

  • You need a grammar
    • Grammar induction or grammar writing
  • Dependency vs. constituency
  • You can do full or partial parsing
  • Grammar theory

Types of grammar

  • Symbolic: rules for grammatical relations, using recursive rewrite rules
  • Probabilistic: rules extracted from a corpus, using statistical information to resolve ambiguity.
  • Constraint-based: describe only what is not allowed.

Note that there are many different grammars with different theoretical assumptions.

Dependency vs. constituency

These are two types of parsing trees.

Constituent phrase structure:

  • A constituent is a word group that functions as a single unit within a hierarchical structure
  • Rewrite rules have terminals and non-terminals
  • Each element is connected to at least one other element.

Dependency:

  • The verb is the center of the structure
  • Each element is connected to exactly one other element.
  • Flatter tree, where one is the head and the other element is the dependent.

See slide for example.

Comment from Martha:

"Watch out: types of grammars in this case is not exactly the same thing that we talk about when we talk about types of formal grammars in the Chomsky hierarchy (for those who have heard of the Chomsky hierarchy). It is also not completely unrelated"

Treebank

A treebank is a ... REVIEW.

Automatic parsing

A main issue for automatic parsing is dealing with ambiguity.

  1. The tourist saw museums
  2. The circular saw blades

Is saw a verb or a noun? From a syntactic viewpoint we cannot distinguish these cases that are semantically different.

So semantic context is relevant for parsing, but we cannot capture this.

Sometimes there are multiple parses possible, purely on syntactic terms:

  1. Time flies like an arrow

Does time fly, like an arrow? Or do "time flies" like arrows?

The longer the sentence, the more possibilities for parsing there will be.

Current best parsing performance is 75% LAS (label attachment score).

Referring expressions

Example with co-reference chains:

Kweli gave birth at the Bronx Zoo only after some meticulous matchmaking. Her calf, M'bura, was introduced to visitors last week. “It’s a zebra horse!” said Dagney Donaldson,7. Not exactly. Okapis are the only known relative of the giraffe, but with the silhouette of an antelope. They are notoriously skittish.

To breed such an antisocial animal in captivity requires a mix of patience, genetic know-how and romantic savoir-faire. So as M’bura, now a healthy 5-month-old, gallops around her shady habitat in the Bronx, zoo officials are taking a victory lap of their own. “There’s a lot of science in it, so you collect the data and use the data as a guide,” said Jim Breheny, director of the Bronx Zoo.

Put this in: https://demo.allennlp.org/coreference-resolution

Recommended tools

  • CoreNLP from Stanford
    • Java-based multiple languages
  • AllenNLP
    • English only
  • Frog (Pipeline for Dutch)
  • NLTK Python
  • SpaCy

Feature creation and selection

A dominant approach is bag of words; or bag-of-lemmas.

The following two documents will have the same representation

  1. d1 man bites dog
  2. d2 dog bites man

This approach will result in sparse vectors, because the vector length is the whole vocabulary (so all words occurring in all documents).

Alternative: use n-grams.

A bag of words is a bag of unigrams, but we can also take a bag of bigrams (or in the other, even in terms of characters).