TMM Semantic Gap

What is multimedia?

Wide definition:

  • More than one modality involved
  • Modalities are complementary; both together convey the information

Narrow definition:

  • Additionally; it has to be produced by humans for consumption by other humans.

The semantic gap

The semantic gap is the difference between the information that a machine can extract from the perceptual data and the interpretation that a user in a given situation has for the same data.

Computers deal in bits and bytes up to list arrays, documents and image. In short, data.

Humans deal in event and objects, knowledge about them, and insights (wisdom; self-realization).

The challenge is to cross this gap.

This semantic gap is obvious in image recognition, but can also be seen for example in music.

Machines can easily identify exact matches between sound fragments. But humans can identify approximate matches (e.g. covers) and overall characteristics (e.g. a similar "feel" to it; or the signature style of a composer). Here machine struggle.

One important insight is that the semantic gap is defined with respect to a specific situation, or in IR terms, a users information need.

Even the best neural network solutions (e.g. see demo on Google image search) make assumptions here and may not return what we are looking for, even if they are technically very impressive.

Challenges of visual variability

For example, are we looking for images that depict the same entity? Or are we looking for images that depict the same class of the entity?

Various variability classes:

  • Is an entity fixed?
    • E.g. a building always has the same background
    • The background helps identifying the entity
  • Does the entity move?
    • The "Gouden Koets" moves around has more visual variability
  • Does the entity change?
    • Lady Gaga wears a different extravagant outfit each time
    • Classifying Lady Gage is therefore extremely challenging.

Identical instances:

  • Sometimes only experts can differentiate entities
    • E.g. cheeta pups that look practically identical.
  • Sometimes only people familiar with the situation can differentiatie
    • Picture of beer cans: who's beer was who?

Events: triples (entities, time, place)

  • Can be seen as instance or classes
  • Can be seen as a scene: if the time is not specified and/or the entity is not that relevant.

The terminology of "semantic"

We talked about the semantic gap, and we may talk about semantic content.

There is a difference in the TMM literature and the computer vision literature. Objects, scenes and events are the semantic content of images or videos. In computer vision, the semantic content refers to the present objects, or the depicted concepts ("sky" or "living room"), and collections of those that constitute scenes and events.

When we are discussing the semantic gap, "semantic" is basically anything you can see you saw after looking at that image. Here we can also talk about style, aesthetics, composition, connotations. This goes beyond semantic content. A lot of what humans see in a picture, is orthogonal to the semantic content, but is very much relevant for the interpretation of the picture (and thus for the semantic gap).

E.g. compare pictures of weddings. They all depict similar objects and represent the same concept. But we immediately see the style of the pictures, showing a particular atmosphere.

What's interesting here is that different people see different things in pictures; but this is more subtle than saying that is purely subjective, because things like style are certainly "there", even though they are not semantic content in the limited sense of being an object.