On the computational power of neural nets

This paper deals with finite networks which consist of interconnections of synchronously evolving processors. Each processor updates its state by applying a “sigmoidal” scalar nonlinearity to a linear combination of the previous states of all units. We prove that one may simulate all Turing Machines by rational nets. In particular, one can do this in linear time, and there is a net made up of about 1,000 processors which computes a universal partial-recursive function. Products (high order nets) are not required, contrary to what had been stated in the literature. Furthermore, we assert a similar theorem about non-deterministic Turing Machines. Consequences for undecidability and complexity issues about nets are discussed too.

[1]  S C Kleene,et al.  Representation of Events in Nerve Nets and Finite Automata , 1951 .

[2]  Claude E. Shannon,et al.  A Universal Turing Machine with Two Internal States , 1956 .

[3]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[4]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[5]  J. R. Brown,et al.  Artificial neural network on a SIMD architecture , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[6]  Jean Berstel,et al.  Rational series and their languages , 1988, EATCS monographs on theoretical computer science.

[7]  Westervelt,et al.  Dynamics of iterated-map neural networks. , 1989, Physical review. A, General physics.

[8]  S. Smale,et al.  On a theory of computation and complexity over the real numbers; np-completeness , 1989 .

[9]  James L. McClelland,et al.  Finite State Automata and Simple Recurrent Networks , 1989, Neural Computation.

[10]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[11]  S. Franklin,et al.  Neural computability. II , 1989, International 1989 Joint Conference on Neural Networks.

[12]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[13]  Eduardo D. Sontag,et al.  Mathematical control theory: deterministic systems , 1990 .

[14]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[15]  J. Doug Tygar,et al.  The computability and complexity of optical beam tracing , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.

[16]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[17]  Eduardo D. Sontag,et al.  Mathematical Control Theory: Deterministic Finite Dimensional Systems , 1990 .

[18]  Noga Alon,et al.  Efficient simulation of finite automata by neural nets , 1991, JACM.

[19]  Omid M. Omidvar.  Progress in neural networks , 1991 .

[20]  Roy Batruni,et al.  A multilayer neural network with piecewise-linear structure and back-propagation learning , 1991, IEEE Trans. Neural Networks.

[21]  Eduardo Sontag,et al.  Algebraic theory of sign-linear systems , 1991, 1991 American Control Conference.

[22]  Y. C. Lee,et al.  Turing equivalence of neural networks with second order connection weights , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[23]  Georg Schnitger,et al.  On the computational power of sigmoid versus Boolean threshold circuits , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[24]  C. L. Giles,et al.  Second-order recurrent neural networks for grammatical inference , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[25]  Eduardo D. Sontag,et al.  Feedforward Nets for Interpolation and Classification , 1992, J. Comput. Syst. Sci..

[26]  D. Wolpert,et al.  A Computationally Universal Field Computer That is Purely Linear , 1993 .