Fast Projection onto the $\ell_{\infty,1}$ -Mixed Norm Ball Using Steffensen Root Search

Mixed norms that promote structured sparsity have broad application in signal processing and machine learning problems. In this work we present a new algorithm for computing the projection onto the $\ell_{\infty,1}$ ball, which has found application in cognitive neuroscience and classification tasks. This algorithm is based on a Steffensen type root search technique, with a number of improvements over prior root search methods for the same problem. First, we theoretically derive an initial guess for the root search algorithm that helps to reduce the number of iterations to be performed. Second, we change the root search method, and through an analysis of the root search function, we construct a pruning strategy that significantly reduces the number of operations. Numerical simulations show that, compared to the state-of-the-art, our algorithm is between 4 and 5 times faster on average, and of up to 14 times faster for very sparse solutions.

[1]  Jieping Ye,et al.  Efficient Methods for Overlapping Group Lasso , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  B. Wohlberg,et al.  Fast Projection onto the 𝓁 ∞, 1 -Mixed Norm Ball Using Steffensen Root Search. , 2018, ICASSP 2018.

[3]  Krzysztof C. Kiwiel,et al.  Breakpoint searching algorithms for the continuous quadratic knapsack problem , 2007, Math. Program..

[4]  Han Liu,et al.  Blockwise coordinate descent procedures for the multi-task lasso, with applications to neural semantic basis discovery , 2009, ICML '09.

[5]  L. W. Johnson,et al.  On Steffensen–s Method , 1968 .

[6]  Laurent Condat Fast projection onto the simplex and the lll 1 ball , 2014 .

[7]  Philip Wolfe,et al.  Validation of subgradient optimization , 1974, Math. Program..

[8]  Suvrit Sra Fast projections onto mixed-norm balls with applications , 2012, Data Mining and Knowledge Discovery.

[9]  Alkiviadis G. Akritas On the Budan-Fourier controversy , 1981, SIGS.

[10]  Trevor Darrell,et al.  Transfer learning for image classification with sparse prototype representations , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Suvrit Sra,et al.  Fast Projections onto ℓ1, q -Norm Balls for Grouped Feature Selection , 2011, ECML/PKDD.

[12]  Changshui Zhang,et al.  Efficient Euclidean projections via Piecewise Root Finding and its application in gradient projection , 2011, Neurocomputing.

[13]  Paul Rodríguez,et al.  Accelerated Gradient Descent Method for Projections onto the $\ell_{1}$ -Ball , 2018, 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[14]  Rodriguez Paul An accelerated newton's method for projections onto the l1-ball , 2017 .

[15]  Stephen J. Wright,et al.  Simultaneous Variable Selection , 2005, Technometrics.

[16]  Jun Liu,et al.  Efficient Euclidean projections in linear time , 2009, ICML '09.

[17]  A. Fiacco A Finite Algorithm for Finding the Projection of a Point onto the Canonical Simplex of R " , 2009 .

[18]  S. Amat,et al.  An Overview on Steffensen-Type Methods , 2016 .

[19]  A. Gramfort,et al.  Mixed-norm estimates for the M/EEG inverse problem using accelerated gradient methods , 2012, Physics in medicine and biology.

[20]  Stephen P. Boyd,et al.  Proximal Algorithms , 2013, Found. Trends Optim..

[21]  Michael P. Friedlander,et al.  Probing the Pareto Frontier for Basis Pursuit Solutions , 2008, SIAM J. Sci. Comput..

[22]  M. Kowalski Sparse regression using mixed norms , 2009 .

[23]  Yoram Singer,et al.  Efficient projections onto the l1-ball for learning in high dimensions , 2008, ICML '08.