Scene Perception | Aude Oliva

Introduction

One usually takes action on objects, but in scenes.

Remarkably, we are able to interpret the meaning of multifaceted and complex scene images—a wedding, a birthday party, or a stadium crowd—in a fraction of a second (Potter, 1975)! This is about the same time it takes a person to identify that a single object is a face, a dog, or a car (Grill-Spector & Kanwisher, 2005; Intraub, 1981; Thorpe, Fize, & Marlot, 1996).

How does the human visual system analyse real-world scenes? From three perspectives:

  • Behavioral
  • Computational
  • Cognitive neuroscience

With special attention to the principles that apply both to 2D and 3D scenes (because many experiments are done using 2D pictures).

Behavioral studies

Meaning of a scene is captured in a glance irregardless of the complexity of the scene.

Scene is considered as a whole, not as sum of parts. It is not problematic for parts to have a similar layout or similar objects, while belonging to different semantical categories.

Potter and Levy (1969) study shows that scene meaning can be captured in 100ms.

Main question: What type of representation is built so fast?

Theoretical Perspectives

Biderman (1981): three levels of quick representations

Scene understanding arises from:

1) Prominent object or surface 2) Multiple-component representation with a layout of objects and surfaces 3) Global representation of scene-emergent features potentially independent of objects and surface identification

Three levels:

1) Prominent object representation: one main region with a large and characteristic object, e.g. a bed indicating a bedroom 2) Multiple-component representation: objects and regions in a layout, segmented from the background. 3) Global scene-emergent features: general grasping of all objects over the percept.

Developed into two "complementary paths of scene perception":

  • Object-centered approach
    • Segmented components serve as scene descriptors
    • Road because we see cars
  • Scene or space-centered approach
    • Spatial layout and global image properties act as scene descriptors
    • street because urban env flanked with tall vertical structures

Time course of scene analysis

  • Early: <100ms
  • Late: 200-300ms

Global-to-local / coarse-to-fine approach, following that time-course.

You test this by only flashing a scene for < 100 sec and asking about global features and details. Above chance reporting on global features.

Building Blocks of Scene Perception

Temporal history and adaptation: how does what we have recently perceived influence what is presently perceived?

Aftereffects of spatial layout. World looks larger after spending a whole day in a small office.

Greene and Oliva (2010): "prolonged exposure to scenes with shared global properties can influence how observers perceive that property in later scenes" --> can even influence semantical categorization, e.g. bias an interpretation of an ambiguous scene.

"the visual system extracts statistics and becomes attuned to recently processed global properties"

Computational Framework of scene perception

Oliva and Torralba framework: spatial boundaries and content.

Spatial boundaries orthogonal to previous distinctions concerning content: Both boundaries and content can be built from the three aforementioned representation levels.

Neuroimaging studies of scene perception

Distinct neural pathways for the representation of scene/space-centered vs content/object-centered information.

Scene-and Space-Centered Cortical Regions

PPA - Parahippocampal place area

RSC - retrosplenial complex

OPA - occipital place area

PPA and RSC selective to spatial layout but not number of objects.

What's diff PPA and RSC?

PPA: representation that takes viewpoint/perspective on scene into account.

RSC: more general place-centered representation.

Object-Centered Cortical Regions

LOC - Lateral Occipital Complex, represents object shapes and object categories.

Object-Centered region hypothesizes to code the content of scenes, as objects make up the content.

Complementary representations

Pattern analysis on brain regions. Classification errors using particular brain response: overlapping or complementary errors?

They found a dissociation between the types of classifi- cation errors made in two brain regions. The PPA con- fused scenes with similar spatial boundaries, regardless of the type of content, whereas the LOC made the opposite errors (confusing scenes with the same content, independent of their spatial layout). In the PPA, scenes representing closed urban environments, such as streets and buildings, were most confused with closed natural scenes, such as forests and canyons (see also Kravitz, Peng, & Baker, 2011). On the other hand, the LOC confused scenes that contained similar objects and surfaces, such as mistaking fields, deserts, or ocean with forest, mountain, or canyons; LOC accuracy was insensitive to the spatial layout. Using multivoxel pattern analysis as well, MacEvoy

Altogether, the current state of the art in neuroimaging studies suggests that scene analysis in the brain recruits distinct regions that perform different and complementary computations to create a singular and unique representation of the scene in view.