Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization

The affine rank minimization problem consists of finding a matrix of minimum rank that satisfies a given system of linear equality constraints. Such problems have appeared in the literature of a diverse set of fields including system identification and control, Euclidean embedding, and collaborative filtering. Although specific instances can often be solved with specialized algorithms, the general affine rank minimization problem is NP-hard because it contains vector cardinality minimization as a special case. In this paper, we show that if a certain restricted isometry property holds for the linear transformation defining the constraints, the minimum-rank solution can be recovered by solving a convex optimization problem, namely, the minimization of the nuclear norm over the given affine space. We present several random ensembles of equations where the restricted isometry property holds with overwhelming probability, provided the codimension of the subspace is sufficiently large. The techniques used in our analysis have strong parallels in the compressed sensing framework. We discuss how affine rank minimization generalizes this preexisting concept and outline a dictionary relating concepts from cardinality minimization to those of rank minimization. We also discuss several algorithmic approaches to minimizing the nuclear norm and illustrate our results with numerical examples.

[1]  Gebräuchliche Fertigarzneimittel,et al.  V , 1893, Therapielexikon Neurologie.

[2]  I. J. Schoenberg Remarks to Maurice Frechet's Article ``Sur La Definition Axiomatique D'Une Classe D'Espace Distances Vectoriellement Applicable Sur L'Espace De Hilbert , 1935 .

[3]  C. Eckart,et al.  The approximation of one matrix by another of lower rank , 1936 .

[4]  L. Mirsky SYMMETRIC GAUGE FUNCTIONS AND UNITARILY INVARIANT NORMS , 1960 .

[5]  G. Marsaglia,et al.  When Does Rank(A+B)=Rank(A)+Rank(B)? , 1972, Canadian Mathematical Bulletin.

[6]  J. Claerbout,et al.  Robust Modeling With Erratic Data , 1973 .

[7]  H. Andrews,et al.  Singular Value Decomposition (SVD) Image Coding , 1976, IEEE Trans. Commun..

[8]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[9]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[10]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[11]  S. Szarek The finite dimensional basis problem with an appendix on nets of Grassmann manifolds , 1983 .

[12]  P. Strevens Iii , 1985 .

[13]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[14]  Editors , 1986, Brain Research Bulletin.

[15]  Z. Bai,et al.  On the limit of the largest eigenvalue of the large dimensional sample covariance matrix , 1988 .

[16]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[17]  Walter Gander,et al.  Algorithms for the Polar Decomposition , 1990, SIAM J. Sci. Comput..

[18]  Eduardo D. Sontag,et al.  Mathematical Control Theory: Deterministic Finite Dimensional Systems , 1990 .

[19]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[20]  G. Watson Characterization of the subdifferential of some matrix norms , 1992 .

[21]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[22]  Y. C. Pati,et al.  Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[23]  Stéphane Mallat,et al.  Matching pursuits with time-frequency dictionaries , 1993, IEEE Trans. Signal Process..

[24]  Zhifeng Zhang,et al.  Adaptive time-frequency decompositions , 1994 .

[25]  Shie Qian,et al.  Signal representation using adaptive normalized Gaussian functions , 1994, Signal Process..

[26]  Dimitri P. Bertsekas,et al.  Nonlinear Programming , 1997 .

[27]  Nathan Linial,et al.  The geometry of graphs and some of its algorithmic applications , 1994, Proceedings 35th Annual Symposium on Foundations of Computer Science.

[28]  J. Faraut,et al.  Analysis on Symmetric Cones , 1995 .

[29]  Balas K. Natarajan,et al.  Sparse Approximate Solutions to Linear Systems , 1995, SIAM J. Comput..

[30]  G. Lorentz,et al.  Constructive approximation : advanced problems , 1996 .

[31]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[32]  Stephen P. Boyd,et al.  Semidefinite Programming , 1996, SIAM Rev..

[33]  H. Upmeier ANALYSIS ON SYMMETRIC CONES (Oxford Mathematical Monographs) , 1996 .

[34]  F. A. Lootsma Distance Matrix Completion by Numerical Optimization , 1997 .

[35]  L. Vandenberghe,et al.  Optimal wire and transistor sizing for circuits with non-tree topology , 1997, ICCAD 1997.

[36]  S. Szarek Metric Entropy of Homogeneous Spaces , 1997, math/9701213.

[37]  L. Faybusovich Euclidean Jordan Algebras and Interior-point Algorithms , 1997 .

[38]  L. Ghaoui,et al.  A cone complementarity linearization algorithm for static output-feedback and related problems , 1997, IEEE Trans. Autom. Control..

[39]  G. Papavassilopoulos,et al.  On the rank minimization problem over a positive semidefinite linear matrix inequality , 1997, IEEE Trans. Autom. Control..

[40]  C. Beck,et al.  Computational study and comparisons of LFT reducibility methods , 1998, Proceedings of the 1998 American Control Conference. ACC (IEEE Cat. No.98CH36207).

[41]  Stephen P. Boyd,et al.  Low-authority controller design via convex optimization , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[42]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[43]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[44]  K. Grigoriadis,et al.  Alternating projection algorithms for linear matrix inequalities problems with rank constraints , 1999 .

[45]  Pablo A. Parrilo,et al.  On cone-invariant linear matrix inequalities , 2000, IEEE Trans. Autom. Control..

[46]  Michael W. Trosset,et al.  Distance Matrix Completion by Numerical Optimization , 2000, Comput. Optim. Appl..

[47]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[48]  Henry Wolkowicz,et al.  Handbook of Semidefinite Programming , 2000 .

[49]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[50]  M. R. Osborne,et al.  A new approach to variable selection in least squares problems , 2000 .

[51]  B. Fares,et al.  An augmented Lagrangian method for a class of LMI-constrained problems in robust control theory , 2001 .

[52]  Farid Alizadeh,et al.  Associative and Jordan Algebras, and Polynomial Time Interior-Point Algorithms for Symmetric Cones , 2001, Math. Oper. Res..

[53]  Stephen P. Boyd,et al.  A rank minimization heuristic with application to minimum order system approximation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[54]  Kim-Chuan Toh,et al.  SDPT3 — a Matlab software package for semidefinite-quadratic-linear programming, version 3.0 , 2001 .

[55]  S. Szarek,et al.  Chapter 8 - Local Operator Theory, Random Matrices and Banach Spaces , 2001 .

[56]  Renato D. C. Monteiro,et al.  A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization , 2003, Math. Program..

[57]  Takao Watanabe,et al.  A unified algebraic approach to linear control design: Robert E. Skelton, Tetsuya Iwasaki and Karolos M. Grigoriadis; Copyright Taylor & Francis, 1998, ISBN: 0-7484-0592-5 , 2003, Autom..

[58]  Marc Teboulle,et al.  Mirror descent and nonlinear projected subgradient methods for convex optimization , 2003, Oper. Res. Lett..

[59]  Stephen P. Boyd,et al.  Log-det heuristic for matrix rank minimization with applications to Hankel and Euclidean distance matrices , 2003, Proceedings of the 2003 American Control Conference, 2003..

[60]  Dimitris Achlioptas,et al.  Database-friendly random projections: Johnson-Lindenstrauss with binary coins , 2003, J. Comput. Syst. Sci..

[61]  Adrian Lewis,et al.  The mathematics of eigenvalue optimization , 2003, Math. Program..

[62]  Sanjoy Dasgupta,et al.  An elementary proof of a theorem of Johnson and Lindenstrauss , 2003, Random Struct. Algorithms.

[63]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[64]  John B. Moore,et al.  A Newton-like method for solving rank constrained linear matrix inequalities , 2006, 2004 43rd IEEE Conference on Decision and Control (CDC) (IEEE Cat. No.04CH37601).

[65]  Nathan Srebro,et al.  Learning with matrix factorizations , 2004 .

[66]  Renato D. C. Monteiro,et al.  Local Minima and Convergence in Low-Rank Semidefinite Programming , 2005, Math. Program..

[67]  Nathan Srebro,et al.  Fast maximum margin matrix factorization for collaborative prediction , 2005, ICML.

[68]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[69]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[70]  D. Donoho,et al.  Neighborliness of randomly projected simplices in high dimensions. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[71]  D. Donoho,et al.  Sparse nonnegative solution of underdetermined linear equations by linear programming. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[72]  Harald Haas,et al.  Asilomar Conference on Signals, Systems, and Computers , 2006 .

[73]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[74]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[75]  P. Koev,et al.  On the largest principal angle between random subspaces , 2006 .

[76]  E.J. Candes Compressive Sampling , 2022 .

[77]  M. Bingham Analysis on Symmetric Cones (oxford Mathematical Monographs) , 2006 .

[78]  Leonid Faybusovich,et al.  Jordan-Algebraic Approach to Convexity Theorems for Quadratic Mappings , 2006, SIAM J. Optim..

[79]  Richard G. Baraniuk,et al.  An Architecture for Compressive Imaging , 2006, 2006 International Conference on Image Processing.

[80]  Stephen P. Boyd,et al.  Enhancing Sparsity by Reweighted ℓ1 Minimization , 2007, 0711.1612.

[81]  E. Candès,et al.  Sparsity and incoherence in compressive sampling , 2006, math/0611957.

[82]  Rick Chartrand,et al.  Exact Reconstruction of Sparse Signals via Nonconvex Minimization , 2007, IEEE Signal Processing Letters.

[83]  Stephen P. Boyd,et al.  Portfolio optimization with linear and fixed transaction costs , 2007, Ann. Oper. Res..

[84]  Laurent El Ghaoui,et al.  Rank Minimization under LMI constraints: A Framework for Output Feedback Problems , 2007 .

[85]  Ivan Markovsky,et al.  Structured low-rank approximation and its applications , 2008, Autom..

[86]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[87]  Lieven Vandenberghe,et al.  Interior-Point Method for Nuclear Norm Approximation with Application to System Identification , 2009, SIAM J. Matrix Anal. Appl..