TMM Lecture 3: Word Embeddings
- What are word embeddings?
- Word embeddings can be biased.
- Describe a strategy for dealing with this problem
N.B. documents are represented using vectors, but the representation is not the document. We make decisions along the way to get there.
Introducing word embeddings
They are word representations, which can be created with some count-based method:
- Latent Semantic Analysis
- Neural language models
Each word is represented as a vector.
One hot vectors for two different words in the vocabulary will be orthogonal.
This means that we cannot meaningfully quantify the similarity of two individual word vectors: their dot product will always be zero.
So one-hot encoding is not very useful here.
The term "one hot" comes from electrical engineering where only one piece in a circuit has voltage.
Example of word embeddings on the GloVe model
Closeness using word embeddings can be determined using cosine similarity.
A word embedding vector represents a word in a lower dimensional space than one-hot vectors, in about 200-500 dimensions.
This has as a downside is that the components of the vector are no longer interpretable.
Useful for grouping similar words together in space. They can also be used as input representations for other algorithms.
One way of getting a text representation is to average all word vectors of the words it contains.
Word2Vec Skip-gram
One way of getting word embeddings is through a neural network.
You get a vector back based on the hidden layers using a softmax classifier.
What do word embeddings capture?
Can capture
- Synonyms or near synonyms
- Functionally related words
- Words from the same paradigm
Beyond word2vec:
ELMos ("Embeddings from Language Models"): captures different meanings of words corresponding to different contexts.
BERT ("Bidirectional Encoder Representations for Transformers"): takes into a account similarities beyond the scope of a single sentence.
Bias in word embeddings
In AI and machine learning, bias refers to prior information, a necessary prerequisite for intelligent action (Bishop, 2006).
The problem is when that information is known to lead to harm.
Case study: gender bias in profession words.
You may need to do bias correction.
WG
For your research project, think about:
- Knowledge needed for RQ; what is de knowledge gap
- What data
- What approach
- How to validate
Literature search:
- libguides.ru.nl/LibraryOfScienceEN/computingscience
- Google scholar
- peer reviewed
- ACM Digital Library and IEE Xplore
- NOT arXiv until paper is published in a paper that is peer reviewed
- Cast a wide net in your search
- Search forward and back: given a paper, check references in that paper; and find references that refer to that paper.
- Systematically collect and try out keyword in queries
- Select candidate papers, select few for detailed reading.
- Find a reference management tool.
Tip from Max: connected papers
Exercise:
- Take a RQ and determine whether it can be answered using text and/or multimedia mining. If not, change it so they can be.
- Think about, what knowledge is needed, data, approach, validation, references.
Q from Matthijs: The project is based on applying TxMM techniques to answer some RQ, right? Or should the research be aimed at investigating TxMM techniques?
A: to do the latter, you really need extremely solid validation. This is likely to exceed the scope of the project.