Lecture 6: Sentiment Analysis
Recording part 1 Recording part 2
Learning goals:
- Relation between text categorization and authorship attribution
- What are different types of authorship attribution?
- In which domains is it applicable?
- Describe two types of methods for authorship attribution
Definition
Usually we can attribute one author from a set of candidate authors for which we have texts with undisputed authorship.
In terms of machine learning this is:
- Multiclass ...
- Single-label text-categorization ...
- using textual features.
Question: if you have multiple authors, do we then not have a multi-label categorization problem?
A: Not necessarily; the problem would then rather be to figure out which paragraph is written by which author, which is still a single-label problem.
Tasks
- Verification
- Did author A write this?
- Identification
- Is this text written by author A, B, or C?
- Author profiling
- What is the age or gender of this author?
- Which personal traits can we detect?
- From which region is this author?
Questions and Assumptions
Authors are seen as generators of text. Then we ask if there are constant elements in texts written by the same author? The content, communicative goal and genre may differ; can we nevertheless detect the stylistic fingerprint of an author?
So... what do our words say about us? (cf. "The Secret Life of Nouns; what our words say about us" - Pennebaker, 2011)
The human stylome hypothesis: "authors can be distinguished by measuring specific properties of their writings, their stylome as it were." (Pennabaker, 2011)
Finding: common non-content words (so function words), such as pronouns, determiners, prepositions, conjunctives, are apparently the most informative about an author's writing style.
Application areas
- Literary studies
- Resolving disputes on authorship
- Linking pseudonyms or anonymous publications to authors
- Analyse collaborative writing
- Plagiarism detection
- Forensic linguistics
- E.g linking someone to incriminating text-messages
Text classification vs. Authorship Attribution
In text classification tasks we are interested in what a text is about, i.e. it is topic-based. Authorship attribution is style-based instead.
- Topic based: we care most about content words that carry semantic information
- Style-based: we care most about function words because these common words are most helpful for discriminating between authors.
Predictive Stylome Features
- Using the frequencies of functions words is already quite successful (e.g. using a Bayesian classifier).
- word use
- word n-grams
- character n-grams
- most common words
- preference for longer/shorter words
In general we can use:
- Lexical features
- Character features
- Syntactic features
- Semantic features
- Application-specific features
1 and 2 are shallow in the sense that they see a text as a mere sequence of word-tokens or characters
3 and 4 require deeper linguistic analysis
1. Lexical features
Main advantages of using lexical features:
- Language independent
- No preprocessing other than tokenizing
Tokenize the text and then either you have multiple options.
-
Use bag of word features (so very sparse vectors with word counts)
-
Extract all words from text of candidate authors
-
Take top-k frequent words as features for your classifier
-
Use n-grams instead
Is most useful for larger collections (as otherwise collocations will be extremely sparse)
Downsides:
- Significantly higher dimensionality
- Higher sparseness
-
Requires more data to perform well
-
Aggregated features
Count other things like:
- Sentence length
- Word length
- Type/token ratio as a measure of vocabulary richness/diversity.
2. Character features
Split the text into characters
Again we can use either character n-grams directly, or some aggregated feature.
Aggregated character-level features:
- Alphabetic character counts
- Digit counts
- Uppercase and lowercase counts
- Punctuation counts
- Whitespace counts
Advantages:
- Language- and domain independent
- No NLP needed
- Offers an option when tokenization is hard
- Proven to capture something of writing style
- E.g. as one can imagine, the use of interpunction is often a matter of style
3. Syntactic features
Assumption: authors unknowingly use similar syntactic patterns. Syntactic information is therefore considered a more robust authorial fingerprint.
We can do (from "deep" to "superficial"):
- Full syntactical analysis, e.g. frequency of patterns
- Chunking (e.g. only looking at particular verbphrase (VP) and nounphrase (NP) and PP combinations)
- This is shallow parsing.
- Aggregated features may be counts or lengths of VPs and NPs etc.
- Only look at POS tags (e.g. bigrams of POS tags).
But this has disadvantages:
- Relies on robust NLP
- Language dependent
- Parser will always introduce some noise
Because full syntactic analysis technique really relies on robust NLP, it is used less frequently even though it may produce very good results.
4. Semantic features
Semantic role labeling & patterns of semantic roles. E.g. does an author writes sentences with active agents, or more with beneficiaries?
We can also use word embeddings. This is more focused on topic rather than style (see above), but it may still be useful.
5. Application-specific features
One may suit the style of an application area better. E.g. literary analysis differs from e-mails or SMS messages.
Possible structural features:
- Use of greetings and farewells
- Types of signatures
- Indentation use
- Paragraph length
Learning methods
- Profile-based
- A single representation per author
- Instance-based
(or a hybrid)
Profile-based methods
The ZIP method:
- Concatenate known texts of author A
- Compress e.g. with zip, rar, bzip
- Repeat but now use all texts of author A including the unknown text.
- The difference in bit-wise size of the compressed files indicates the similarity of the unseen text with each candidate author
Then you do this for author B as well, and compare the difference in file size.
This works because compression algorithms like zip remove/summarize duplicate information in texts. So if the unseen text uses the characteristic style of author A, then we expect that including that text will hardly increase the file size; and that in turn it would show a larger increase in file size if the text turned out to be from author B.
Example application on a book of Grunberg: Dario Benedetto et al. Language trees and zipping. Physical Review Letters, 88(4), 2002.
ZIP method originally developed for language identification.
Common n-grams (CNG distance)
Create an author profile by listing the most frequent n-grams used in texts by the author. Then compute the distance between the text and profile; e.g. cosine similarity.
Parameters:
- L: profile length; how many n-grams to use
- n: which n-gram
Best results for 1000<=L<=5000 and 3<=n<=5.
This works well for a balanced training corpus.
Q: Why doesn't it work for unbalanced training data (e.g. more data for author B?)
I assume just because simple frequency will always be higher with more training data; but wouldn't normalization help here? I guess there will always be more uncertainty with less data.
Instance-based methods
Treat all samples separately (so do not combine them into an author profile).
Vector space model:
- Each text is a vector in the vector space
- Compute cosine distance between vectors
or some other machine learning algorithm that handles high-dimensional, noisy, and sparse data.
Q: How would you use an instance-based method if you only have one long text per author as training data?
A: You can split the text. Samples with 500-1000 words work decent.
Hybrid
Example:
- Represent texts independently (instance-based)
- Average text representations for one author (over features) to produce a profile vector of feature centroids (cf. profile-based)
- Compute distance (based on features) between centroid vector to vector of unseen text
When to use what?
Q: We have a set of emails and want to verify the author. Which method?
A: Profile based is better, because emails are very short for instance-based methods. When building a profile, we concatenate the emails, which mitigates the problem.
Author profiling
- Gender
- Age
- Region
- Personal traits (introversion vs. extroversion)
Example: you are what you tweet
PAN: Digital Text Forensics; benchmarking for Author identification, author profiling, author obfuscation. Nice starting point for a project!
In this field we really should consider GDPR and privacy concerns? What type of knowledge is OK to extract? Which type should remain private?
Further reading
Stamatatos, Efstathios: "A survey of modern authorship attribution methods." Journal of the American Society for information Science and Technology 60.3. (2009): 538-556.