Human-Level Psychometrics for Cognitive Architectures

As artificial intelligence approaches the human level, the task of designing cognitive tests and challenges for robots becomes critical for clarifying and reproducing elements of the “magic” of human cognition. In solving this task, the store of knowledge of experimental psychology cannot be ignored. At the same time, methods and test paradigms used for evaluation of humans and/or brain models need to be adjusted before they can be used for evaluation of computational agents. The present work introduces new test paradigms and metrics that are adequate to assess higher cognitive features in expected computational agents, while at the same time allowing for comparison with humans. Index Terms – Cognitive evaluation, psychometrics, intelligent agents, cognitive systems, human-level intelligence, machine consciousness, emotional intelligence, IQ, g factor, Turing test.

[1]  P. Veenhuis The Emperor's New Mind: Concerning Computers, Minds, and the Laws of Physics , 1995 .

[2]  G. Ascoli,et al.  A simple neural network model of the hippocampus suggesting its pathfinding role in episodic memory retrieval. , 2005, Learning & memory.

[3]  H. Simon,et al.  Perception in chess , 1973 .

[4]  E. Tulving [Episodic memory: from mind to brain]. , 2004, Revue neurologique.

[5]  C. Moore,et al.  Binding the self in time , 2001 .

[6]  M. Davies,et al.  Folk Psychology: The Theory of Mind Debate , 1995 .

[7]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[8]  E. Tulving,et al.  Episodic and semantic memory , 1972 .

[9]  A. Jensen,et al.  The g factor , 1996, Nature.

[10]  G. A. Miller The magical number seven plus or minus two: some limits on our capacity for processing information. , 1956, Psychological review.

[11]  S. Baron-Cohen,et al.  Does the autistic child have a “theory of mind” ? , 1985, Cognition.

[12]  H. Wellman,et al.  Children Talk About the Mind , 1995 .

[13]  J. Carroll Psychometrics, intelligence, and public perception. , 1997 .

[14]  Martin Davies,et al.  Mental Simulation: Evaluations and Applications - Reading in Mind and Language , 1995 .

[15]  Appa Rao Korukonda Taking stock of Turing test: a review, analysis, and appraisal of issues surrounding thinking machines , 2003, Int. J. Hum. Comput. Stud..

[16]  Giorgio A. Ascoli,et al.  Towards Virtual Brains , 2002 .

[17]  R. Gordon Folk Psychology as Simulation , 1986 .

[18]  S. Stich,et al.  Mindreading: An Integrated Account of Pretence, Self-Awareness, and Understanding Other Minds , 2003 .

[19]  Daniel J. Povinelli,et al.  The Self: Elevated in Consciousness and Extended in Time , 2001 .

[20]  Kenneth A. De Jong,et al.  A General-Purpose Computational Model of the Conscious Mind , 2004, ICCM.

[21]  Alexei V. Samsonovich,et al.  Fundamental Principles and Mechanisms of the Conscious Self , 2005, Cortex.

[22]  Anthony J Bishara,et al.  Capture by misleading information and its false acceptance in patients with traumatic brain injury. , 2006, Brain : a journal of neurology.

[23]  A. Gopnik,et al.  The Development of Categorization in the Second Year and Its Relation to Other Cognitive and Linguistic Developments. , 1987 .

[24]  G. Matthews,et al.  Emotional Intelligence: Science and Myth , 2003 .

[25]  Lee Ryan,et al.  An fMRI study of episodic memory: retrieval of object, spatial, and temporal information. , 2004, Behavioral neuroscience.

[26]  W. Scott Phoria adaptation to prisms. A cerebellar-dependent response , 1984 .

[27]  S. Gelman,et al.  Mapping the Mind: Domain Specificity In Cognition And Culture , 1994 .

[28]  J. Barresi Extending Self-Consciousness Into the Future , 2001 .

[29]  E. Tulving Elements of episodic memory , 1983 .

[30]  G. Gallup Absence of self-recognition in a monkey (Macaca fascicularis) following prolonged exposure to a mirror. , 1977, Developmental psychobiology.

[31]  James S. Albus,et al.  Engineering of Mind: An Introduction to the Science of Intelligent Systems , 2001 .

[32]  Kenneth A. De Jong,et al.  Designing A Self-Aware Neuromorphic Hybrid , 2005 .

[33]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[34]  J. Starns,et al.  Episodic generation can cause semantic forgetting: Retrieval-induced forgetting of false memories , 2004, Memory & cognition.

[35]  S. Baron-Cohen Mindblindness: An Essay on Autism and Theory of Mind , 1997 .

[36]  A. Gopnik,et al.  The theory theory. , 1994 .

[37]  David R. Shanks Representation of categories and concepts in memory. , 1997 .

[38]  J. Feldman The Simplicity Principle in Human Concept Learning , 2003 .

[39]  E. Tulving,et al.  Toward a theory of episodic memory: the frontal lobes and autonoetic consciousness. , 1997, Psychological bulletin.

[40]  Robert W. Schrauf,et al.  Stability in autobiographical memories , 2004, Memory.

[41]  H. Lefcourt,et al.  Humor: The Psychology of Living Buoyantly , 2001 .

[42]  J. Gregory Trafton,et al.  Enabling effective human-robot interaction using perspective-taking in robots , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[43]  Bernhard Hommel,et al.  Acquisition and Control of Voluntary Action — 1 Acquisition and Control of Voluntary Action , 2001 .

[44]  Jane Heal,et al.  Simulation and cognitive penetrability , 1996 .

[45]  Randolph Blake,et al.  Psychophysical magic: rendering the visible ‘invisible’ , 2005, Trends in Cognitive Sciences.

[46]  A. Goldman In Defense of the Simulation Theory , 1992 .

[47]  D. Stuss,et al.  The frontal lobes and self-awareness , 2005 .

[48]  Richard D. Roberts,et al.  The Armed Services Vocational Aptitude Battery (ASVAB) , 2000 .

[49]  J E Lisman,et al.  Storage of 7 +/- 2 short-term memories in oscillatory subcycles , 1995, Science.

[50]  Giorgio A. Ascoli,et al.  The Conscious Self: Ontology, Epistemology and the Mirror Quest , 2005, Cortex.

[51]  Joseph P. Huston,et al.  Integrated memory for objects, places, and temporal order: Evidence for episodic-like memory in mice , 2005, Neurobiology of Learning and Memory.

[52]  Chris Moore,et al.  The Self in Time: Developmental Perspectives , 2001 .