Inhibitory synapses in neural networks with sigmoidal nonlinearities

We study the behavior of anti-Hebbian synapses at a neural node that contains the standard sigmoidal nonlinearity. The criterion generalizes the idea already discussed by one of the authors for linear networks and consists in removing the correlation between the input to the synapse, and the node output. We show how the solution, just as for the linear case, is unique and can be learned with a standard anti-Hebbian rule.We suggest how these synapses can be embedded in fully self-organizing networks to generate orthogonal nonlinear components and be used for multidimensional approximation.

[1]  Simon Haykin,et al.  Adaptive filter theory (2nd ed.) , 1991 .

[2]  J. Karhunen.  Optimization criteria and nonlinear PCA neural networks , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[3]  P. Foldiak,et al.  Adaptive network for optimal linear feature extraction , 1989, International 1989 Joint Conference on Neural Networks.

[4]  Lei Xu,et al.  Theories for unsupervised learning: PCA and its nonlinear extensions , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[5]  Gustavo Deco,et al.  Nonlinear higher-order statistical decorrelation by volume-conserving neural architectures , 1995, Neural Networks.

[6]  Sun-Yuan Kung,et al.  A neural network learning algorithm for adaptive principal component extraction (APEX) , 1990, International Conference on Acoustics, Speech, and Signal Processing.

[7]  David Zipser,et al.  Feature Discovery by Competive Learning , 1986, Cogn. Sci..

[8]  J. Rubner,et al.  A Self-Organizing Network for Principal-Component Analysis , 1989 .

[9]  Teuvo Kohonen,et al.  Self-Organization and Associative Memory , 1988 .

[10]  Mohamad H. Hassoun,et al.  Nonlinear Hebbian rule: a statistical interpretation , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[11]  F. Palmieri,et al.  Hebbian learning and self-association in nonlinear neural networks , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[12]  James M. Ortega,et al.  Iterative solution of nonlinear equations in several variables , 2014, Computer science and applied mathematics.

[13]  R. Linsker,et al.  From basic network principles to neural architecture , 1986 .

[14]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[15]  Jie Zhu,et al.  Self-association and Hebbian learning in linear neural networks , 1995, IEEE Trans. Neural Networks.

[16]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[17]  Jie Zhu,et al.  Anti-Hebbian learning in topologically constrained linear networks: a tutorial , 1993, IEEE Trans. Neural Networks.

[18]  Deco,et al.  Learning time series evolution by unsupervised extraction of correlations. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[19]  Andrew J. Viterbi,et al.  Principles of coherent communication , 1966 .

[20]  Peter Földiák,et al.  Adaptation and decorrelation in the cortex , 1989 .

[21]  Pierre Comon,et al.  Blind separation of sources, part II: Problems statement , 1991, Signal Process..

[22]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..