STRUCTURED CONNECTIONIST MODELS OF LANGUAGE, COGNITION AND ACTION

For some sixteen years, an interdisciplinary group at ICSI and UC Berkeley has been building a theory of language that is consistent with biological, psychological and computational constraints. The intellectual base for the project is a synthesis of cognitive linguistics and structured connectionist modeling [1,2], centered on a three-part Embodiment Hypothesis: (1) many concepts are directly embodied in motor, perceptual, and other neural structures; (2) all other concepts derive their inferential structure via mappings from these embodied structures; and (3) structured connectionist models provide a suitable computational abstraction over such neurally grounded representations. We extend this with a Simulation Hypothesis, that language understanding exploits some of the same neural structures used for action, perception, imagination, memory and other cognitive processes, i.e., linguistic structures provide parameters for simulations drawing on these embodied structures. This talk focuses on the computational requirements for biologically plausible models of language understanding and acquisition. In particular, we explore the idea that the basic unit of linguistic representation is a construction, or mapping between form (sound, gestures, etc.) and meaning (embodied concepts motivated by biological structures and realizable as structured connectionist models). Constructions support a language understanding process modeled as having two distinct phases: utterances are first analyzed to determine which constructions are involved and how their corresponding meanings are related; the events and actions specified by the resulting network of related concepts (or semantic specification) are then simulated to produce inferences using embodied conceptual structures. Simulation itself relies on an active structure called an executing schema (or x-schema) [3]. X-schemas capture hierarchical structure, sequential flow, concurrency and other properties based on models of motor control but also necessary for event structure in general. Results of simulation are used to update a belief network representing the current context. We concentrate on two systems that show how a simulation-based approach to language understanding can drive disparate linguistic phenomena. The first of these is a model of metaphorical inference in news stories [3]. The basic model is extended so that simulationbased inferences in a source domain (e.g., using an x-schema for falling) are projected via metaphorical mappings (e.g., Falling Is Failure) to license inferences in more abstract target domains; the sentence “France fell into a recession” is thus understood to involve economic failure. The second model shows how partial interpretations of utterances in context can help children acquire their first multi-unit constructions, represented as structured mappings between linguistic forms and their embodied experience [4]. New mappings are hypothesized to capture bindings available in context but not licensed by an existing construction, and thus omitted from the (partial) semantic specification. Together these models show how embodied conceptual and linguistic structures can be integrated within a simulation-based framework to provide a common representational toolkit for language, cognition and action.