Learning and analyzing vector encoding of symbolic representations

We present a formal language with expressions denoting general symbol structures and queries which access information in those structures. A sequence-to-sequence network processing this language learns to encode symbol structures and query them. The learned representation (approximately) shares a simple linearity property with theoretical techniques for performing this task.

[1]  Allen Newell,et al.  Physical Symbol Systems , 1980, Cogn. Sci..

[2]  David S. Touretzky,et al.  BoltzCONS: Dynamic Symbol Structures in a Connectionist Network , 1990, Artif. Intell..

[3]  Joe Pater.  The harmonic mind : from neural computation to optimality-theoretic grammar , 2009 .

[4]  Tony Plate,et al.  Holographic Recurrent Networks , 1992, NIPS.

[5]  Jason Weston,et al.  Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks , 2015, ICLR.

[6]  Rabab Kreidieh Ward,et al.  Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval , 2015, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[7]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[8]  Pentti Kanerva,et al.  Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors , 2009, Cognitive Computation.

[9]  Christopher D. Manning,et al.  Learning Continuous Phrase Representations and Syntactic Parsing with Recursive Neural Networks , 2010 .

[10]  Tony A. Plate,et al.  Holographic Reduced Representation: Distributed Representation for Cognitive Structures , 2003 .

[11]  Chris Eliasmith,et al.  Biologically Plausible, Human-scale Knowledge Representation , 2016, CogSci.

[12]  Wojciech Zaremba,et al.  Learning to Execute , 2014, ArXiv.

[13]  P. Smolensky.  Representation in Connectionist Networks , 1990 .

[14]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[15]  Geoffrey Zweig,et al.  Linguistic Regularities in Continuous Space Word Representations , 2013, NAACL.

[16]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[17]  Jianfeng Gao,et al.  Reasoning in Vector Space: An Exploratory Study of Question Answering , 2016, ICLR.