Combining rule-based and embedding-based approaches to normalize textual entities with an ontology | Ferré, Deléger, Zweigenbaum, Nédellec

Abstract

In this paper, we propose a two-step method to normalize multi-word terms with concepts from a domain-specific ontology. Normalization is a critical step of information extraction. The method uses vector representations of terms computed with word embedding information and hierarchical information among ontology concepts. A training dataset and a first result dataset with high precision and low recall are generated by using the ToMap unsupervised normalization method. It is based on the similarities between the form of the term to normalize and the form of concept labels. Then, a projection of the space of terms towards the space of concepts is learned by globally minimizing the distances between vectors of terms and vectors of concepts. It applies multivariate linear regression using the previously generated training dataset. Finally, a distance calculation is carried out between the projections of term vectors and the concept vectors, providing a prediction of normalization by a concept for each term. (3443)

two-step method to normalize multi-word terms with concepts from a domain-specific ontology (3443)

The method uses vector representations of terms computed with word embedding information and hierarchical information among ontology concepts. (3443)

  • Assumes availability of domain-specific ontology
  • Uses word embeddings
  • The chosen normalization method, ToMap, is unsupervised
    • Because the ontology is given I assumed this would be a supervised method, but I assume this method is unsupervised because we do not presuppose/label which term vector corresponds to which concept

Introduction

The focus is on textual entities such as particular type of bacteria.

Task description:

  • Named Entity Recognition
  • Entity Linking (to an ontology), which in this paper is called "normalization"

The best-performing normalization methods are supervised, but this is not always practical because:

1) This requires manual annotation of concepts in texts 2) Many domains have a very large amount of target concepts (example is given op biomedical metathesaurus UMLS which contains >3 million concepts)

The goal of this paper is to try to automate this "labeling" step. This would allow further experimentation with "distant supervised" (3443) methods, i.e. supervised methods that use the labels that are learned in an unsupervised manner.

Related work

CONTES sounds interesting to look into; also uses word embeddings:

Another recent method, CONTES (Ferré et al., 2017) is based on an approach that does not take into account the form of terms and concept labels, but only distributional information for terms, and hierarchical ontological information for concepts. It is based on the ability of the latest word embedding methods to generate relevant semantic spaces, as well as on building "concept embeddings" that preserves hierarchical information between concepts. This kind of method aims to overcome the problem of variability of forms of associated terms/labels, but does not otherwise take into account relevant morphological information. (3443)

In this work, the CONTES method is combined with a rule-based approach called the ToMap method from Golik et al., 2011.

"GOLIK WIKTORIA, WARNIER PIERRE, NÉDELLEC CLAIRE (2011), Corpus-Based Extension of Termino-Ontology by Linguistic Analysis: A Use Case in Biomedical Event Extraction, 37–39, in: WS 2 Workshop Extended Abstracts, 9th International Conference on Terminology and Artificial Intelligence." (3447)

Material

Task description using specific entities:

The task is to assign concepts from the OntoBiotope ontology to textual entities denoting bacterial habitats (entities are provided and do not have to be detected beforehand) (3444)

Train and test corpora with manually annotations.

Word embeddings trained on titles and abstract of MEDLINE articles returned on the search query "bacteria". Generally speaking training on a larger corpus has benefits, but in this case the domain was specifically biomedical texts on bacteria, so the authors preferred a more focused corpus.

Method

ToMap

ToMap (Golik et al., 2011) uses the internal "morpho-syntactic structures of entities" (3444). Core algorithm:

  1. Find the syntactic head of the entity (assumed to be the most informative)
    • And I think the entity here is a multi-word description
  2. Look for match between syntactic head of an entity and syntactic head of ontology concept
  3. Assign entity to concept(s) with matching heads
  4. When multiple options, pick the entity with the highest Jaccard index

This core algorithm is complemented with some additional rules:

  • If syntactic head is uninformative (provide black list), match on modifiers instead
  • Various heuristics, such as disambiguation rules for words such as "plant"
  • When no match is found, the entity is assigned the root concept of the ontology

CONTES

See papers for word2vec with Skip-gram parameters. (3444) Notably: keep al words (min-count=0); vector size of 200 (worked well in similar work for biomedical domain); small window size of 2, see reasoning below.

Indeed, a common hypothesis is that larger window emphasizes the learning of domain similarity between words, while a narrow context window leads to hyponymic gathering. (3445)

Example hyponymy: "pigeon, crow, eagle, and seagull are all hyponyms of bird" (Wikipedia)

  1. You start with computing word embeddings
  2. Then create vectors for entities "by computing the barycenter of the vectors of the words that compose the entity." (3445)
    • A barycenter is the point around which two bodies (that orbit each other) orbit; center of mass.
  3. Compute (in parallel) concept vectors
    • "Each concept is associated with a vector of equivalent size to the number of concepts in the ontology" (3445)
    • I.e. the vector size is the same as the amount of concepts
    • The idea here is that each vector dimension will represent a concept and that we end up with an "ontological space"
    • "Each dimension is associated with a fixed concept." (3445) OK, but fixed how?
    • Ah, I thought these would also be word embedding vectors, but these are typical one-hot encoded vectors.
  4. Project term vectors into the ontological space
    • Learn this projection on a training corpus, "minimize the distance between the projection of terms in the ontological space and the vectors of the associated concepts" (3445)
    • Use multivariate linear regression for this
  5. Apply learned projection on term vectors of test corpus (term vectors being... the multi-word entity vectors?; or all word embeddings?) which results of term vectors in the ontological space
  6. Calculate cosine distance with term vectors in ontological space and the ontological vectors and pick the closest concept as the one that normalizes a term.

Vector encoding for hierarchy

How to encode ontological concepts from a hierarchy as a vector? Two steps:

  1. "The vector is initialized as a one-hot (i.e. all weights are set to zero except the weight associated with the current concept, which is set to one)," (3445)
  2. "then each of the weights of the dimensions associated with the ancestors of the current concept are also set to one. The method encodes the hierarchical information of the ontology: if we estimate the cosine similarity between ontology concepts, the parent/child concepts are always the closest." (3445)

N.B. this only encodes the is a relation in ontologies (which is however the most common relation).

Previous work used a simple one-hot encoding and the results show improvements of this method.

Combining the two approaches

ToMap is precise, but relies on syntactic-morphological similarity and thus does not deal well with form variability. It also cannot always provide normalizations, so tends to have a low recall score.

CONTES using word embeddings "has been designed to address the problem of term variability, because it does not rely on the similarity of form between terms and concept labels." (3445)

The proposed HONOR system (Hierarchical Ontological NORmalization) combines these methods sequentially:

  1. Apply ToMap, which results in an incomplete normalization
  2. Fill in the incomplete cases with the full normalization of CONTES

Implementation of CONTES and ToMap can be found in https://github.com/ArnaudFerre/CONTES

If I understand correctly, HONOR can work without manually labelled data.

Distant supervision

  1. Use ToMap to predict on a larger corpus
  2. Use these unsupervised normalization as the concepts with which to train CONTES instead of a gold standard

Future work

The key hypothesis behind the efficiency of the CONTES and HONOR methods is that the semantic space of terms (based on distributional semantics) and the semantic space of the ontology (based on the specific hierarchical information of the current ontology) are homologous. Even if this work seems to indicate that there are at least some similarities between these two vector spaces, this hypothesis is most likely too strong. It would certainly be interesting to alter the space of embeddings to increase this similarity and then use a non-linear algorithm to find a better projection from the embedding space to the ontological space. (3447)

Finding methods for better non-linear projections between terms and concepts is also to be done.

What's the point here? That these vectors are not that interpretable? "Like the CONTES method, a problem of high dimensional representation of the concept vectors persists" (3447)