Online entropy manipulation: stochastic information gradient

Entropy has found significant applications in numerous signal processing problems including independent components analysis and blind deconvolution. In general, entropy estimators require O(N/sup 2/) operations, N being the number of samples. For practical online entropy manipulation, it is desirable to determine a stochastic gradient for entropy, which has O(N) complexity. In this paper, we propose a stochastic Shannon's entropy estimator. We determine the corresponding stochastic gradient and investigate its performance. The proposed stochastic gradient for Shannon's entropy can be used in online adaptation problems where the optimization of an entropy-based cost function is necessary.

[1]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[2]  A. Benveniste,et al.  Robust identification of a nonminimum phase system: Blind adjustment of a linear equalizer in data communications , 1980 .

[3]  D. Donoho.  ON MINIMUM ENTROPY DECONVOLUTION , 1981 .

[4]  E. T. Jaynes,et al.  Papers on probability, statistics and statistical physics , 1983 .

[5]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[6]  Paul A. Viola,et al.  Empirical Entropy Manipulation for Real-World Problems , 1995, NIPS.

[7]  D. W. Scott,et al.  On Locally Adaptive Density Estimation , 1996 .

[8]  Aapo Hyvärinen,et al.  New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit , 1997, NIPS.

[9]  David Horn,et al.  Probability Density Estimation Using Entropy Maximization , 1998, Neural Computation.

[10]  Luc Devroye,et al.  Combinatorial methods in density estimation , 2001, Springer series in statistics.

[11]  Inbar Fijalkow,et al.  A globally convergent approach for blind MIMO adaptive deconvolution , 2001, IEEE Trans. Signal Process..

[12]  Deniz Erdogmus,et al.  Blind source separation of time-varying instantaneous mixtures using an on-line algorithm , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[13]  Jing Zhao,et al.  Simultaneous extraction of Principal Components using givens rotations and output variances , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[14]  Deniz Erdogmus,et al.  An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems , 2002, IEEE Trans. Signal Process..

[15]  Kari Torkkola,et al.  On feature extraction by mutual information maximization , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[16]  Deniz Erdogmus,et al.  Blind source separation using Renyi's -marginal entropies , 2002, Neurocomputing.

[17]  Deniz Erdogmus,et al.  Generalized information potential criterion for adaptive system training , 2002, IEEE Trans. Neural Networks.