Depth-Size Tradeoffs for Neural Computation

The tradeoffs between the depth (i.e., the time for parallel computation) and the size (i.e., the number of threshold gates) in neural networks are studied. The authors focus the study on the neural computations of symmetric Boolean functions and some arithmetic functions. It is shown that a significant reduction in the size is possible for symmetric functions and some arithmetic functions, at the expense of a small constant increase in depth. In the process, several neural networks which have the minimum size among all the known constructions have been developed. Results on implementing symmetric functions can be used to improve results about arbitrary Boolean functions. In particular, it is shown that any Boolean function can be computed in a depth-3 neural network with O(2/sup n/ /sup 2/) threshold gates; it is also proven that at least Omega (2/sup n/ /sup 3/) threshold gates are required. >

[1]  N. P. Red’kin.  Synthesis of threshold circuits for certain classes of Boolean functions , 1970 .

[2]  Pavel Pudlák,et al.  Threshold circuits of bounded depth , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[3]  Jehoshua Bruck,et al.  Polynomial threshold functions, AC functions and spectrum norms , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.

[4]  Richard J. Lipton,et al.  Unbounded fan-in circuits and associative functions , 1983, J. Comput. Syst. Sci..

[5]  Nicholas Pippenger.  The Complexity of Computations by Networks , 1987, IBM J. Res. Dev..

[6]  F. Fich.  Two problems in concrete complexity: cycle detection and parallel prefix computation , 1982 .

[7]  Uzi Vishkin,et al.  Constant Depth Reducibility , 1984, SIAM J. Comput..

[8]  Jehoshua Bruck,et al.  Neural computation of arithmetic functions , 1990 .

[9]  Michael E. Saks,et al.  On threshold circuits for parity , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.

[10]  Robert C. Minnick,et al.  Linear-Input Logic , 1961, IRE Trans. Electron. Comput..

[11]  William H. Kautz.  The Realization of Symmetric Switching Functions with Linear-Input Logical Elements , 1961, IRE Trans. Electron. Comput..

[12]  T. Kailath,et al.  Computing With Almost Optimal Size Threshold Circuits , 1991, Proceedings. 1991 IEEE International Symposium on Information Theory.

[13]  Jehoshua Bruck,et al.  On the Power of Threshold Circuits with Small Weights , 1991, SIAM J. Discret. Math..

[14]  Noam Nisan,et al.  Constant depth circuits, Fourier transform, and learnability , 1989, 30th Annual Symposium on Foundations of Computer Science.

[15]  Ingo Wegener,et al.  The complexity of Boolean functions , 1987 .

[16]  Alon Orlitsky,et al.  A geometric approach to threshold circuit complexity , 1991, COLT '91.

[17]  Alon Orlitsky,et al.  On the Circuit Complexity of Neural Networks , 1990, NIPS.

[18]  Claude E. Shannon,et al.  The synthesis of two-terminal switching circuits , 1949, Bell Syst. Tech. J..

[19]  Jehoshua Bruck,et al.  Depth efficient neural networks for division and related problems , 1993, IEEE Trans. Inf. Theory.

[20]  Thomas Hofmeister,et al.  Some Notes on Threshold Circuits, and Multiplication in Depth 4 , 1991, Inf. Process. Lett..

[21]  David E. Muller,et al.  Bounds to Complexities of Networks for Sorting and for Switching , 1975, JACM.

[22]  Michael Sipser,et al.  Parity, circuits, and the polynomial-time hierarchy , 1981, 22nd Annual Symposium on Foundations of Computer Science (sfcs 1981).

[23]  Saburo Muroga,et al.  The principle of majority decision logical elements and the complexity of their circuits , 1959, IFIP Congress.

[24]  Stephen A. Cook,et al.  Log Depth Circuits for Division and Related Problems , 1986, SIAM J. Comput..

[25]  J. Reif,et al.  On Threshold Circuits and Polynomial Computation , 1992, SIAM J. Comput..

[26]  Jehoshua Bruck,et al.  Harmonic Analysis of Polynomial Threshold Functions , 1990, SIAM J. Discret. Math..