Quadratic programming over ellipsoids with applications to constrained linear regression and tensor decomposition

A novel algorithm to solve the quadratic programming (QP) problem over ellipsoids is proposed. This is achieved by splitting the QP problem into two optimisation sub-problems, (1) quadratic programming over a sphere and (2) orthogonal projection. Next, an augmented-Lagrangian algorithm is developed for this multiple constraint optimisation. Benefitting from the fact that the QP over a single sphere can be solved in a closed form by solving a secular equation, we derive a tighter bound of the minimiser of the secular equation. We also propose to generate a new positive semidefinite matrix with a low condition number from the matrices in the quadratic constraint, which is shown to improve convergence of the proposed augmented-Lagrangian algorithm. Finally, applications of the quadratically constrained QP to bounded linear regression and tensor decomposition paradigms are presented.

[1]  Danny C. Sorensen,et al.  Minimization of a Large-Scale Quadratic FunctionSubject to a Spherical Constraint , 1997, SIAM J. Optim..

[2]  Marco Locatelli.  Some results for quadratic problems with one or two quadratic constraints , 2015, Oper. Res. Lett..

[3]  Pierre Comon,et al.  Blind Multilinear Identification , 2012, IEEE Transactions on Information Theory.

[4]  David P. Williamson,et al.  Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming , 1995, JACM.

[5]  Jeff T. Linderoth.  A simplicial branch-and-bound algorithm for solving quadratically constrained quadratic programs , 2005, Math. Program..

[6]  Masakazu Kojima,et al.  Faster, but weaker, relaxations for quadratically constrained quadratic programs , 2014, Comput. Optim. Appl..

[7]  Pierre Comon,et al.  CONFAC Decomposition Approach to Blind Identification of Underdetermined Mixtures Based on Generating Function Derivatives , 2012, IEEE Transactions on Signal Processing.

[8]  Shuai Li,et al.  On Context-Dependent Clustering of Bandits , 2016, ICML.

[9]  Alexandre d'Aspremont,et al.  Phase recovery, MaxCut and complex semidefinite programming , 2012, Math. Program..

[10]  Masakazu Kojima,et al.  Exact Solutions of Some Nonconvex Quadratic Optimization Problems via SDP and SOCP Relaxations , 2003, Comput. Optim. Appl..

[11]  K. Mani Chandy,et al.  Quadratically Constrained Quadratic Programs on Acyclic Graphs With Application to Power Flow , 2012, IEEE Transactions on Control of Network Systems.

[12]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[13]  Danny C. Sorensen,et al.  Algorithm 873: LSTRS: MATLAB software for large-scale trust-region subproblems and regularization , 2008, TOMS.

[14]  William W. Hager,et al.  Minimizing a Quadratic Over a Sphere , 2001, SIAM J. Optim..

[15]  Y. Ye,et al.  Semidefinite programming relaxations of nonconvex quadratic optimization , 2000 .

[16]  Franz Rendl,et al.  A semidefinite framework for trust region subproblems with applications to large scale minimization , 1997, Math. Program..

[17]  Peng Zhang,et al.  Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation , 2016, Neural Computing and Applications.

[18]  Nikolaos V. Sahinidis,et al.  Semidefinite relaxations for quadratically constrained quadratic programming: A review and comparisons , 2011, Math. Program..

[19]  Shuai Li.  The art of clustering bandits , 2016 .

[20]  Shuai Li,et al.  Distributed Clustering of Linear Bandits in Peer to Peer Networks , 2016, ICML.

[21]  Zdenek Dostál,et al.  An optimal algorithm and superrelaxation for minimization of a quadratic function subject to separable convex constraints with applications , 2012, Math. Program..

[22]  A. Cichocki,et al.  Tensor decompositions for feature extraction and classification of high dimensional datasets , 2010 .

[23]  Nikos D. Sidiropoulos,et al.  Convex Optimization-Based Beamforming , 2010, IEEE Signal Processing Magazine.

[24]  Benjamin Friedlander,et al.  Performance analysis of blind signal copy using fourth order cumulants , 1996 .

[25]  Shuai Li,et al.  Online Optimization Methods for the Quantification Problem , 2016, KDD.

[26]  Michael K. Ng,et al.  Fast Algorithms for the Generalized Foley-Sammon Discriminant Analysis , 2010, SIAM J. Matrix Anal. Appl..

[27]  Stanley J. Reeves,et al.  Image Restoration: Fundamentals of Image Restoration , 2014 .

[28]  Zhi-Quan Luo,et al.  Semidefinite Relaxation of Quadratic Optimization Problems , 2010, IEEE Signal Processing Magazine.

[29]  Jing Xue,et al.  Face recognition based on random subspace method and tensor subspace analysis , 2017, Neural Computing and Applications.

[30]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[31]  M. Kojima,et al.  Second order cone programming relaxation of nonconvex quadratic optimization problems , 2001 .

[32]  Shuai Li,et al.  Collaborative Filtering Bandits , 2015, SIGIR.

[33]  W. Gander,et al.  A Constrained Eigenvalue Problem , 1989 .

[34]  Andrzej Cichocki,et al.  Error Preserving Correction: A Method for CP Decomposition at a Target Error Bound , 2019, IEEE Transactions on Signal Processing.

[35]  Anton van den Hengel,et al.  Semidefinite Programming , 2014, Computer Vision, A Reference Guide.

[36]  David P. Baron.  Quadratic programming with quadratic constraints , 1972 .

[37]  Andrzej Cichocki,et al.  An augmented Lagrangian algorithm for decomposition of symmetric tensors of order-4 , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[38]  Kenneth Holmstrom.  TOMLAB - An Environment for Solving Optimization Problems in MATLAB , 1997 .

[39]  Salah Bourennane,et al.  Multiway Filtering Based on Fourth-Order Cumulants , 2005, EURASIP J. Adv. Signal Process..

[40]  Nan Zhang,et al.  Twin support vector machine: theory, algorithm and applications , 2017, Neural Computing and Applications.

[41]  Masakazu Kojima,et al.  A Quadratically Constrained Quadratic Optimization Model for Completely Positive Cone Programming , 2013, SIAM J. Optim..

[42]  Jean-Francois Cardoso,et al.  Super-symmetric decomposition of the fourth-order cumulant tensor. Blind identification of more sources than sensors , 1991, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing.

[43]  E. Yaz.  Linear Matrix Inequalities In System And Control Theory , 1998, Proceedings of the IEEE.

[44]  Jiawang Nie,et al.  Generating Polynomials and Symmetric Tensor Decompositions , 2014, Foundations of Computational Mathematics.

[45]  Yi Chen,et al.  Global Solutions to Large-Scale Spherical Constrained Quadratic Minimization via Canonical Dual Approach , 2013 .

[46]  Tamir Hazan,et al.  Multi-way Clustering Using Super-Symmetric Non-negative Tensor Factorization , 2006, ECCV.

[47]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[48]  Marc Teboulle,et al.  Hidden convexity in some nonconvex quadratically constrained quadratic programming , 1996, Math. Program..

[49]  Yinyu Ye,et al.  Semidefinite programming based algorithms for sensor network localization , 2006, TOSN.

[50]  Zdenek Dostl.  Optimal Quadratic Programming Algorithms: With Applications to Variational Inequalities , 2009 .