When push comes to shove: a computational model of the role of motor control in the acquisition of action verbs

When Push Comes to Shove: A Computational Model of the Role of Motor Control in the Acquisition of Action Verbs by David Robert Bailey Doctor of Philosophy in Computer Science University of California, Berkeley Professor Jerome A. Feldman, Chair Children learn a variety of verbs for hand actions starting in their second year of life. The semantic distinctions can be subtle, and they vary across languages, yet they are learned quickly. How is this possible? This dissertation explores the hypothesis that to explain the acquisition and use of action verbs, motor control must be taken into account. It presents a model of embodied semantics|based on the principles of neural computation in general and on the human motor system in particular|which takes a set of labelled actions and learns both to label novel actions and to obey verbal commands. A key feature of the model is the executing schema, an active controller mechanism which, by actually driving behavior, allows the model to carry out verbal commands. A hard-wired mechanism links the activity of executing schemas to a set of linguistically important features including hand posture, joint motions, force, aspect and goals. The feature set is relatively small and is xed, helping to make learning tractable. Moreover, the use of traditional feature structures facilitates the use of model merging, a Bayesian probabilistic learning algorithm which rapidly learns plausible word meanings, automatically determines an appropriate number of senses for each verb, and can plausibly be mapped to a connectionist recruitment learning architecture. The learning algorithm is demonstrated on a handful of English verbs, and also proves capable of making some interesting distinctions found crosslinguistically. Professor Jerome A. Feldman Dissertation Committee Chair

[1]  Franz Josef Radermacher,et al.  Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Judea Pearl) , 1990, SIAM Rev..

[2]  P. Kay Basic Color Terms: Their Universality and Evolution , 1969 .

[3]  J. M. Anglin,et al.  Overextension and underextension in the child's expressive and receptive speech , 1982, Journal of Child Language.

[4]  K. Nelson,et al.  Structure and strategy in learning to talk. , 1973 .

[5]  Terry Winograd,et al.  A procedural model of language understanding , 1986 .

[6]  Stephen M. Omohundro,et al.  Best-First Model Merging for Dynamic Learning and Recognition , 1991, NIPS.

[7]  N. A. Bernshteĭn The co-ordination and regulation of movements , 1967 .

[8]  J. Tanji The supplementary motor area in the cerebral cortex , 1994, Neuroscience Research.

[9]  Nigel Goddard,et al.  The Perception of Articulated Motion: Recognizing Moving Light Displays , 1992 .

[10]  M. Jeannerod The representing brain: Neural correlates of motor intention and imagery , 1994, Behavioral and Brain Sciences.

[11]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[12]  S. Carey The child as word learner , 1978 .

[13]  Jorma Rissanen,et al.  Universal coding, information, prediction, and estimation , 1984, IEEE Trans. Inf. Theory.

[14]  D. Slobin The Crosslinguistic Study of Language Acquisition , 1987 .

[15]  Kay Atkinson-King,et al.  A First Language: The Early Stages , 1975 .

[16]  E. Markman Categorization and naming in children , 1989 .

[17]  Patrick Suppes,et al.  Language and Learning for Robots , 1994 .

[18]  E. Markman,et al.  Word learning in children: an examination of fast mapping. , 1987, Child development.

[19]  P. Kay,et al.  Basic Color Terms: Their Universality and Evolution , 1973 .

[20]  Mark Rowlands,et al.  The body in mind , 1999 .

[21]  B. Habibi,et al.  Pengi : An Implementation of A Theory of Activity , 1998 .

[22]  Tadao Murata,et al.  Petri nets: Properties, analysis and applications , 1989, Proc. IEEE.

[23]  R. Howe,et al.  Human grasp choice and robotic grasp analysis , 1990 .

[24]  P. Fitts The information capacity of the human motor system in controlling the amplitude of movement. 1954. , 1992, Journal of experimental psychology. General.

[25]  Apostolos P. Georgopoulos,et al.  Cortical Representation of Intended Movements , 1993 .

[26]  Andrew J. Viterbi,et al.  Error bounds for convolutional codes and an asymptotically optimum decoding algorithm , 1967, IEEE Trans. Inf. Theory.

[27]  G. Marcus Negative evidence in language acquisition , 1993, Cognition.

[28]  G. Kane Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[29]  M. Tomasello,et al.  Early syntactic development: A Construction Grammar approach. , 1999 .

[30]  Gary L. Drescher,et al.  Made-up minds - a constructivist approach to artificial intelligence , 1991 .

[31]  L. Talmy Lexicalisation patterns: semantic structure in lexical forms , 1985 .

[32]  Jeffrey Mark Siskind,et al.  Naive physics, event perception, lexical semantics, and language acquisition , 1992 .

[33]  A. Meltzoff,et al.  Imitation of Facial and Manual Gestures by Human Neonates , 1977, Science.

[34]  T. Odlin Women, Fire, and Dangerous Things: What Categories Reveal about the Mind , 1988 .

[35]  B. L. Whorf Language, Thought, and Reality: Selected Writings of Benjamin Lee Whorf , 1956 .

[36]  Marco C. Bettoni,et al.  Made-Up Minds: A Constructivist Approach to Artificial Intelligence , 1993, IEEE Expert.

[37]  Dekai Wu,et al.  Automatic inference: a probabilistic basis for natural language interpretation , 1992 .

[38]  G. Lynch,et al.  Variations in Synaptic Plasticity and Types of Memory in Corticohippocampal Networks , 1992, Journal of Cognitive Neuroscience.

[39]  P. Fitts The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[40]  M. Jeannerod The cognitive neuroscience of action , 1997, Trends in Cognitive Sciences.

[41]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[42]  S. Pinker Learnability and Cognition: The Acquisition of Argument Structure , 1989 .

[43]  Jerome A. Feldman,et al.  Connectionist Models and Their Properties , 1982, Cogn. Sci..

[44]  M. Bowerman,et al.  Learning to express motion events in English and Korean: The influence of language-specific lexicalization patterns , 1991, Cognition.

[45]  A. Gopnik,et al.  Words and plans: early language and the development of intelligent action , 1982, Journal of Child Language.

[46]  Eve V. Clark,et al.  The principle of contrast: A constraint on language acquisition. , 1987 .

[47]  R. Klatzky,et al.  Action for Perception: Manual Exploratory Movements for Haptically Processing Objects and Their Features , 1996 .

[48]  Andreas Stolcke,et al.  Bayesian learning of probabilistic language models , 1994 .

[49]  Alison Gopnik,et al.  Names, relational words, and cognitive development in English and Korean speakers: Nouns are not always learned before verbs. , 1995 .

[50]  Wayne D. Gray,et al.  Basic objects in natural categories , 1976, Cognitive Psychology.

[51]  A. Damasio,et al.  Nouns and verbs are retrieved with differently distributed neural systems. , 1993, Proceedings of the National Academy of Sciences of the United States of America.

[52]  Eugene Charniak,et al.  Statistical language learning , 1997 .

[53]  G. Rizzolatti,et al.  Action recognition in the premotor cortex. , 1996, Brain : a journal of neurology.

[54]  J. Mandler How to build a baby: II. Conceptual primitives. , 1992, Psychological review.

[55]  L. Gleitman,et al.  Language and Experience: Evidence from the Blind Child , 1988 .

[56]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[57]  Nancy Chang A Motor-and Image-schematic Analysis of Aspectual Composition , 1997 .

[58]  K. Pearson Common principles of motor control in vertebrates and invertebrates. , 1993, Annual review of neuroscience.

[59]  Lokendra Shastri,et al.  Semantic Networks: An Evidential Formalization and Its Connectionist Realization , 1988 .

[60]  A. Meltzoff,et al.  Imitation of facial and manual gestures by human neonates , 1977, Science.

[61]  Michael A. Arbib,et al.  Schemas that integrate vision and touch for hand control , 1987 .

[62]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[63]  Jordan Zlatev A Study of Perceptually Grounded Polysemy in a Spatial Microdomain , 1992 .

[64]  Tom M. Mitchell,et al.  Generalization as Search , 2002 .

[65]  Martha Flanders,et al.  Frames of Reference in Sensorimotor Integration , 1996 .

[66]  Wu Dekai Automatic Inference: A Probabilistic Basis for Natural Language , 1992 .

[67]  Nils J. Nilsson,et al.  Teleo-Reactive Programs for Agent Control , 1993, J. Artif. Intell. Res..

[68]  L. Gleitman The Structural Sources of Verb Meanings , 2020, Sentence First, Arguments Afterward.

[69]  Daniel Jurafsky,et al.  A Probabilistic Model of Lexical and Syntactic Access and Disambiguation , 1996, Cogn. Sci..

[70]  Gilles Fauconnier,et al.  Mental Spaces: Aspects of Meaning Construction in Natural Language , 1985 .

[71]  S. Lehman The Neural and Behavioural Organization of Goal‐Directed Movements , 1990, Neurology.

[72]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[73]  H N Zelaznik,et al.  Spatial topological constraints in a bimanual task. , 1991, Acta psychologica.

[74]  Janellen Huttenlocher,et al.  Emergence of action categories in the child: Evidence from verb meanings. , 1983 .

[75]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[76]  Stephen Monsell,et al.  The Latency and Duration of Rapid Movement Sequences: Comparisons of Speech and Typewriting , 1978 .

[77]  L. Shastri,et al.  From simple associations to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings using temporal synchrony , 1993, Behavioral and Brain Sciences.

[78]  J. Flanagan,et al.  Hand and brain : the neurophysiology and psychology of hand movements , 1996 .

[79]  P. Kay,et al.  The linguistic significance of the meanings of basic color terms , 1978 .

[80]  Libby Levison,et al.  Connecting Planning and Acting: Towards an Architecture for Object-Specific Reasoning , 1995 .

[81]  Wolfgang Reisig Petri Nets: An Introduction , 1985, EATCS Monographs on Theoretical Computer Science.

[82]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[83]  D. Sofge THE ROLE OF EXPLORATION IN LEARNING CONTROL , 1992 .

[84]  J. Lammens A computational model of color perception and color naming , 1995 .

[85]  M. Latash Control of human movement , 1993 .

[86]  Rodney A. Brooks,et al.  A Robust Layered Control Syste For A Mobile Robot , 2022 .

[87]  Libby Levison The Topic is Open , 1993 .

[88]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[89]  Rajesh P. N. Rao,et al.  Embodiment is the foundation, not a level , 1996, Behavioral and Brain Sciences.

[90]  R. Brown,et al.  A First Language , 1973 .

[91]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[92]  G. Lakoff Women, fire, and dangerous things : what categories reveal about the mind , 1989 .

[93]  M. Tomasello,et al.  Beyond names for things : young children's acquisition of verbs , 1999 .

[94]  G. Lakoff,et al.  Metaphors We Live by , 1982 .

[95]  C. Pollard,et al.  Center for the Study of Language and Information , 2022 .

[96]  Earl D. Sacerdoti,et al.  The Nonlinear Nature of Plans , 1975, IJCAI.

[97]  M. L. Shik,et al.  Neurophysiology of locomotor automatism. , 1976, Physiological reviews.

[98]  M. Tomasello,et al.  Joint attention on actions: acquiring verbs in ostensive and non-ostensive contexts , 1992, Journal of Child Language.