The Emergence of Organizing Structure in Conceptual Representation

Both scientists and children make important structural discoveries, yet their computational underpinnings are not well understood. Structure discovery has previously been formalized as probabilistic inference about the right structural form-where form could be a tree, ring, chain, grid, etc. (Kemp & Tenenbaum, 2008). Although this approach can learn intuitive organizations, including a tree for animals and a ring for the color circle, it assumes a strong inductive bias that considers only these particular forms, and each form is explicitly provided as initial knowledge. Here we introduce a new computational model of how organizing structure can be discovered, utilizing a broad hypothesis space with a preference for sparse connectivity. Given that the inductive bias is more general, the model's initial knowledge shows little qualitative resemblance to some of the discoveries it supports. As a consequence, the model can also learn complex structures for domains that lack intuitive description, as well as predict human property induction judgments without explicit structural forms. By allowing form to emerge from sparsity, our approach clarifies how both the richness and flexibility of human conceptual organization can coexist.

[1]  S. Carey The Origin of Concepts , 2000 .

[2]  Neil D. Lawrence,et al.  Spectral Dimensionality Reduction via Maximum Entropy , 2011, AISTATS.

[3]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[4]  R N Shepard,et al.  Multidimensional Scaling, Tree-Fitting, and Clustering , 1980, Science.

[5]  Daniel,et al.  Default Probability , 2004 .

[6]  Alexandre d'Aspremont,et al.  Model Selection Through Sparse Max Likelihood Estimation Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data , 2022 .

[7]  James L. McClelland Emergence in Cognitive Science , 2010, Top. Cogn. Sci..

[8]  Pablo A. Parrilo,et al.  Rank-Sparsity Incoherence for Matrix Decomposition , 2009, SIAM J. Optim..

[9]  Noah D. Goodman,et al.  Theory Acquisition and the Language of Thought , 2008 .

[10]  J. Tenenbaum,et al.  Word learning as Bayesian inference. , 2007, Psychological review.

[11]  A. Stepanyants,et al.  Cooperative synapse formation in the neocortex , 2009, Proceedings of the National Academy of Sciences.

[12]  E. Heit Properties of inductive reasoning , 2000, Psychonomic bulletin & review.

[13]  Geoffrey E. Hinton,et al.  Zero-shot Learning with Semantic Output Codes , 2009, NIPS.

[14]  Daniel N. Osherson,et al.  Joshua Stern, Ormond Wilkie, Michael Stob, Edward E. Smith: Default Probability , 1991, Cognitive Sciences.

[15]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[16]  C. Sumiyoshi CATEGORY BASED INDUCTION , 1997 .

[17]  Charles Kemp,et al.  How to Grow a Mind: Statistics, Structure, and Abstraction , 2011, Science.

[18]  James L. McClelland,et al.  Semantic Cognition: A Parallel Distributed Processing Approach , 2004 .

[19]  J. Tenenbaum,et al.  Probabilistic models of cognition: exploring representations and inductive biases , 2010, Trends in Cognitive Sciences.

[20]  Evan Heit,et al.  A Bayesian Analysis of Some Forms of Inductive Reasoning , 1998 .

[21]  Mark W. Schmidt,et al.  Optimizing Costly Functions with Simple Constraints: A Limited-Memory Projected Quasi-Newton Algorithm , 2009, AISTATS.

[22]  Nir Friedman,et al.  Learning Belief Networks in the Presence of Missing Values and Hidden Variables , 1997, ICML.

[23]  Joshua B. Tenenbaum,et al.  Church: a language for generative models , 2008, UAI.

[24]  H. Markram,et al.  The neocortical microcircuit as a tabula rasa. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[25]  J. Piaget,et al.  The early growth of logic in the child : classification and seriation , 1965 .

[26]  Neil D. Lawrence,et al.  The Bigraphical Lasso , 2013, ICML.

[27]  Linda C. van der Gaag,et al.  Probabilistic Graphical Models , 2014, Lecture Notes in Computer Science.

[28]  Zoubin Ghahramani,et al.  Semi-supervised learning : from Gaussian fields to Gaussian processes , 2003 .

[29]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.

[30]  L. Rips Inductive judgments about natural categories. , 1975 .

[31]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[32]  Refractor Vision , 2000, The Lancet.

[33]  R. Wilton The Psychology of Learning and Motivation: Advances in Research and Theory. Vol 4. , 1972 .

[34]  E. Markman Categorization and naming in children , 1989 .

[35]  J. Lake,et al.  The ring of life provides evidence for a genome fusion origin of eukaryotes , 2004, Nature.

[36]  Neil D. Lawrence,et al.  A Unifying Probabilistic Perspective for Spectral Dimensionality Reduction: Insights and New Models , 2010, J. Mach. Learn. Res..

[37]  Joshua B. Tenenbaum,et al.  Discovering Structure by Learning Sparse Graphs , 2010 .

[38]  T. Kuhn,et al.  The Structure of Scientific Revolutions. , 1964 .

[39]  J. Tenenbaum,et al.  Structured statistical models of inductive reasoning. , 2009, Psychological review.

[40]  Charles Kemp,et al.  The discovery of structural form , 2008, Proceedings of the National Academy of Sciences.

[41]  J. W. Hutchinson Netscal: A network scaling algorithm for nonsymmetric proximity data , 1989 .

[42]  James L. McClelland,et al.  Letting structure emerge: connectionist and dynamical systems approaches to cognition , 2010, Trends in Cognitive Sciences.

[43]  Amy Perfors,et al.  Hypothesis generation, sparse categories, and the positive test strategy. , 2011, Psychological review.

[44]  G. Ekman Dimensions of Color Vision , 1954 .

[45]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[46]  Vincent Y. F. Tan,et al.  Learning Latent Tree Graphical Models , 2010, J. Mach. Learn. Res..

[47]  Nick Chater,et al.  A rational analysis of the selection task as optimal data selection. , 1994 .

[48]  Venkat Chandrasekaran,et al.  Gaussian Multiresolution Models: Exploiting Sparse Markov and Covariance Structure , 2010, IEEE Transactions on Signal Processing.

[49]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[50]  Francis T. Durso,et al.  Network Structures in Proximity Data , 1989 .

[51]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[52]  Allan Collins,et al.  A spreading-activation theory of semantic processing , 1975 .

[53]  Joshua B. Tenenbaum,et al.  The acquisition of inductive constraints , 2008 .