Lecture 2 Text and Multimedia Mining: the NLP Pipeline
How do we go from textual information to numerical vectors that we can use in text mining algorithms?
- Gather
- Raw text: Clean, filter, convert
- Clean text: Preprocess with NLP tools
- Linguistically annotated text: Feature creation and selection
- Feature vector
Raw text
We either have written or spoken material.
Written:
- Digitized documents with optical character recognition
- Born-Digital documents
The quality of OCR is not perfect, which introduces problems.
- Introduce spelling
- Introduction of words that weren't present in the original
- Deletion of words that were not recognized
- Loss of text formatting
But 2 and 3 are most severe and hard to counter.
Digital inputs are also not necessarily clean. We want to exclude:
- Advertisements
- Photos
- Links
- etc.
Storing clean text
Clean text: actual textual content separated from other information, uniformly represented in an appropriate character encoding.
However, we do not assume yet that there's no spelling errors and so on. That's all part of the NLP process. So no text normalization is done yet.
Character encoding
You need to take into account the character encoding of your data
There are 10 types of people in the world: those who understand binary, and those who don't.
Encodings:
ASCII: American Standard Code for Information Interchange
But obviously, you can only encode English this way. Initially, computers in other had their own encoding, but from the 80s there was work on a global standard: UNICODE
Ascii uses 8-bit, 7 to encode the character plus a bit for the stop sign. (I think the stop sign is there because ASCII originally was also used as a telegraph language?)
Unicode:
- For all writing systems
- +100.000 characters
- Independent of platform, software
- Descriptive representation of characters: "a+umlaut"
- The actual interpretation of
UTF-8 is the most common Unicode implementation that is also compatible with ASCII (UTF-16, UTF-32). 1 bit from character of ASCII-table; 2-bits for more information.
UTF-8 has great both backward and forward compatibility.
Markup
A markup language is a set of conventions for encoding texts:
- What is allowed?
- What is required?
- Markup format
- Meaning of the markup
As a convention, a mark up language does not do anything in itself. It offers structure for other programs.
Processing clean text
Tokenization
- Using regular expression
-
Easy for Western languages that use whitespace and interpunction as separators
-
A word is a delimited string of characters (vague def)
- A token is one instance of a word occurring in a document
- A type is an equivalence class of tokens
But how do we handle numerical data, parsing hyphens, dealing with emoticons.
Linguistically, compound words ("emailaders"; alles in het Duits lol), or ... (went fast.)
Sentence splitting: relatively easy because sentences usually start with a capital letter and end with a dot. (But again: German may complicate things here).
Tokenizer demo: which tokenizer works best?
https://text-processing.com/demo/tokenize/
PoS Tagsets
Universal POS Tagset
- VERB
- NOUN
- PRON
- ADJ
- ADV
- ADP (prepositions and postpositions)
- CONJ
- DET
- NUM
- PRT (particles or function words)
- X: other; typos, abbreviations
- . punctiation
The WSJ (Wall Street Journal) tagset has 37-PoS-tags, so is way more detailed.
https://nlpprogress.com : state of the art is 97.24% accuracy.
Unknown accuracy is accuracy on previously unseen words.
So PoS-tagging (for English) is now very good. However, this is token accuracy. If you start looking at sentences, almost half of the sentences contains at least one mistake. On top, this performance does not transfer to all text genres and corpora. Essential disambiguations are not(?) treated correctly almost always.
Lemmas
A lemma is a canonical form of the word.
Usually either the infinitive or nominal masculine singular form.
Stem
A cruder word of representing the word.
A stem is the part of a word that remains after cutting of affixes (inflections). This is not further analyzable and is morphologically simple. It captures the principle portion of the meaning of the word.
Parsing
Analyze the syntactic constituents
- You need a grammar
- Grammar induction or grammar writing
- Dependency vs. constituency
- You can do full or partial parsing
- Grammar theory
Types of grammar
- Symbolic: rules for grammatical relations, using recursive rewrite rules
- Probabilistic: rules extracted from a corpus, using statistical information to resolve ambiguity.
- Constraint-based: describe only what is not allowed.
Note that there are many different grammars with different theoretical assumptions.
Dependency vs. constituency
These are two types of parsing trees.
Constituent phrase structure:
- A constituent is a word group that functions as a single unit within a hierarchical structure
- Rewrite rules have terminals and non-terminals
- Each element is connected to at least one other element.
Dependency:
- The verb is the center of the structure
- Each element is connected to exactly one other element.
- Flatter tree, where one is the head and the other element is the dependent.
See slide for example.
Comment from Martha:
"Watch out: types of grammars in this case is not exactly the same thing that we talk about when we talk about types of formal grammars in the Chomsky hierarchy (for those who have heard of the Chomsky hierarchy). It is also not completely unrelated"
Treebank
A treebank is a ... REVIEW.
Automatic parsing
A main issue for automatic parsing is dealing with ambiguity.
- The tourist saw museums
- The circular saw blades
Is saw a verb or a noun? From a syntactic viewpoint we cannot distinguish these cases that are semantically different.
So semantic context is relevant for parsing, but we cannot capture this.
Sometimes there are multiple parses possible, purely on syntactic terms:
- Time flies like an arrow
Does time fly, like an arrow? Or do "time flies" like arrows?
The longer the sentence, the more possibilities for parsing there will be.
Current best parsing performance is 75% LAS (label attachment score).
Referring expressions
Example with co-reference chains:
Kweli gave birth at the Bronx Zoo only after some meticulous matchmaking. Her calf, M'bura, was introduced to visitors last week. “It’s a zebra horse!” said Dagney Donaldson,7. Not exactly. Okapis are the only known relative of the giraffe, but with the silhouette of an antelope. They are notoriously skittish.
To breed such an antisocial animal in captivity requires a mix of patience, genetic know-how and romantic savoir-faire. So as M’bura, now a healthy 5-month-old, gallops around her shady habitat in the Bronx, zoo officials are taking a victory lap of their own. “There’s a lot of science in it, so you collect the data and use the data as a guide,” said Jim Breheny, director of the Bronx Zoo.
Put this in: https://demo.allennlp.org/coreference-resolution
Recommended tools
- CoreNLP from Stanford
- Java-based multiple languages
- AllenNLP
- English only
- Frog (Pipeline for Dutch)
- NLTK Python
- SpaCy
Feature creation and selection
A dominant approach is bag of words; or bag-of-lemmas.
The following two documents will have the same representation
- d1 man bites dog
- d2 dog bites man
This approach will result in sparse vectors, because the vector length is the whole vocabulary (so all words occurring in all documents).
Alternative: use n-grams.
A bag of words is a bag of unigrams, but we can also take a bag of bigrams (or in the other, even in terms of characters).