Discrete symbolic optimization and Boltzmann sampling by continuous neural dynamics: Gradient Symbolic Computation

Gradient Symbolic Computation is proposed as a means of solving discrete global optimization problems using a neurally plausible continuous stochastic dynamical system. Gradient symbolic dynamics involves two free parameters that must be adjusted as a function of time to obtain the global maximizer at the end of the computation. We provide a summary of what is known about the GSC dynamics for special cases of settings of the parameters, and also establish that there is a schedule for the two parameters for which convergence to the correct answer occurs with high probability. These results put the empirical results already obtained for GSC on a sound theoretical footing.

[1]  Pierre-Jean Reissman,et al.  From simulated annealing to stochastic continuation: a new trend in combinatorial optimization , 2012, Journal of Global Optimization.

[2]  Géraldine Legendre,et al.  The Harmonic Mind: From Neural Computation to Optimality-Theoretic GrammarVolume I: Cognitive Architecture (Bradford Books) , 2006 .

[3]  Tony Plate,et al.  Holographic Recurrent Networks , 1992, NIPS.

[4]  Christopher Potts,et al.  Harmonic Grammar with linear programming: from linear systems to linguistic typology* , 2010, Phonology.

[5]  Paul Smolensky,et al.  Schema Selection and Stochastic Inference in Modular Environments , 1983, AAAI.

[6]  Paul Smolensky,et al.  Distributed Recursive Structure Processing , 1991, SCAI.

[7]  Bruce E. Hajek,et al.  Cooling Schedules for Optimal Annealing , 1988, Math. Oper. Res..

[8]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[9]  Joe Pater,et al.  Weighted Constraints in Generative Linguistics , 2009, Cogn. Sci..

[10]  Bruce Hayes,et al.  A Maximum Entropy Model of Phonotactics and Phonotactic Learning , 2008, Linguistic Inquiry.

[11]  F. Keil.  Concepts, Kinds, and Cognitive Development , 1989 .

[12]  Trevor Bekolay,et al.  Supplementary Materials for A Large-Scale Model of the Functioning Brain , 2012 .

[13]  Gökhan BakIr,et al.  Energy-Based Models , 2007 .

[14]  Matthew Goldrick,et al.  Optimization and Quantization in Gradient Symbol Systems: A Framework for Integrating the Continuous and the Discrete in Cognition , 2014, Cogn. Sci..

[15]  J. Goldsmith.  Autosegmental and Metrical Phonology , 1990 .

[16]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[17]  Pyeong Whan Cho,et al.  Bifurcation analysis of a Gradient Symbolic Computation model of incremental processing , 2016, CogSci.

[18]  Y. Miyata,et al.  Harmonic grammar: A formal multi-level connectionist theory of linguistic well-formedness: Theoretic , 1990 .

[19]  Nils J. Nilsson,et al.  Principles of Artificial Intelligence , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Geoffrey E. Hinton.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[21]  Barbara Gentz,et al.  A sample-paths approach to noise-induced synchronization: Stochastic resonance in a double-well potential , 2000, math/0012267.

[22]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[23]  Jonathan C. Mattingly,et al.  Ergodicity for SDEs and approximations: locally Lipschitz vector fields and degenerate noise , 2002 .

[24]  John J. McCarthy,et al.  Optimality Theory in Phonology: A Reader , 2003 .

[25]  R. Tweedie,et al.  Exponential convergence of Langevin distributions and their discrete approximations , 1996 .

[26]  C. Hwang,et al.  Diffusion for global optimization in R n , 1987 .

[27]  C. Gardiner.  Handbook of Stochastic Methods , 1983 .

[28]  Paul Voosen,et al.  The AI detectives. , 2017, Science.

[29]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[30]  Paul Smolensky,et al.  Harmony in Linguistic Cognition , 2006, Cogn. Sci..

[31]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[32]  A. Stuart,et al.  Extracting macroscopic dynamics: model problems and algorithms , 2004 .

[33]  P. Smolensky,et al.  Optimality Theory: Constraint Interaction in Generative Grammar , 2004 .

[34]  Paul Smolensky,et al.  Harmonic Grammars for Formal Languages , 1992, NIPS.

[35]  Stephen Grossberg,et al.  Absolute stability of global pattern formation and parallel memory storage by competitive neural networks , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[36]  A Prince,et al.  Optimality: From Neural Networks to Universal Grammar , 1997, Science.

[37]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[38]  Peter Rossmanith,et al.  Simulated Annealing , 2008, Taschenbuch der Algorithmen.

[39]  Alison L Gibbs,et al.  On Choosing and Bounding Probability Metrics , 2002, math/0209021.

[40]  Jennifer Culbertson,et al.  Cognitive Biases, Linguistic Universals, and Constraint-Based Grammar Learning , 2013, Top. Cogn. Sci..

[41]  W. Ebeling.  Stochastic Processes in Physics and Chemistry , 1995 .

[42]  Kathryn A. Dowsland,et al.  Simulated Annealing , 1989, Handbook of Natural Computing.

[43]  Christopher D. Manning,et al.  Learning Continuous Phrase Representations and Syntactic Parsing with Recursive Neural Networks , 2010 .

[44]  Mark Johnson,et al.  Learning OT constraint rankings using a maximum entropy model , 2003 .

[45]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[46]  Pyeong Whan Cho,et al.  Incremental parsing in a continuous dynamical system: sentence processing in Gradient Symbolic Computation , 2017 .