Matrix concentration inequalities via the method of exchangeable pairs

This paper derives exponential concentration inequalities and polynomial moment inequalities for the spectral norm of a random matrix. The analysis requires a matrix extension of the scalar concentration theory developed by Sourav Chatterjee using Stein’s method of exchangeable pairs. When applied to a sum of independent random matrices, this approach yields matrix generalizations of the classical inequalities due to Hoeffding, Bernstein, Khintchine and Rosenthal. The same technique delivers bounds for sums of dependent random matrices and more general matrix-valued functions of dependent random variables.

[1]  W. Hoeffding.  A Combinatorial Central Limit Theorem , 1951 .

[2]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[3]  H. L. Le Roy,et al.  Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability; Vol. IV , 1969 .

[4]  H. Rosenthal.  On the subspaces ofLp(p>2) spanned by sequences of independent random variables , 1970 .

[5]  C. Stein.  A bound for the error in the normal approximation to the distribution of a sum of dependent random variables , 1972 .

[6]  D. Burkholder.  Distribution Function Inequalities for Martingales , 1973 .

[7]  E. Lieb.  Convex trace functions and the Wigner-Yanase-Dyson conjecture , 1973 .

[8]  V. V. Petrov.  Some Inequalities for the Distributions of Sums of Independent Random Variables , 1975 .

[9]  Iosif Pinelis,et al.  Some Inequalities for the Distribution of Sums of Independent Random Variables , 1978 .

[10]  李幼升,et al.  Ph , 1989 .

[11]  Colin McDiarmid,et al.  Surveys in Combinatorics, 1989: On the method of bounded differences , 1989 .

[12]  G. Pisier,et al.  Non commutative Khintchine and Paley inequalities , 1991 .

[13]  I. Pinelis.  OPTIMUM BOUNDS FOR THE DISTRIBUTIONS OF MARTINGALES IN BANACH SPACES , 1994, 1208.2200.

[14]  D. Petz.  A survey of certain trace inequalities , 1994 .

[15]  R. Bhatia.  Matrix Analysis , 1996 .

[16]  M. Rudelson.  Random Vectors in the Isotropic Position , 1996, math/9608208.

[17]  G. Pisier,et al.  Non-Commutative Martingale Inequalities , 1997, math/9704209.

[18]  G. Lugosi,et al.  On Concentration-of-Measure Inequalities , 1998 .

[19]  A. Buchholz.  Operator Khintchine inequality in non-commutative probability , 2001 .

[20]  M. Ledoux.  The concentration of measure phenomenon , 2001 .

[21]  Rudolf Ahlswede,et al.  Strong converse for identification via quantum channels , 2000, IEEE Trans. Inf. Theory.

[22]  V. Paulsen.  Completely Bounded Maps and Operator Algebras: Completely Bounded Multilinear Maps and the Haagerup Tensor Norm , 2003 .

[23]  M. Junge,et al.  Noncommutative Burkholder/Rosenthal inequalities , 2003 .

[24]  Martin Mathieu.  COMPLETELY BOUNDED MAPS AND OPERATOR ALGEBRAS (Cambridge Studies in Advanced Mathematics 78) , 2004 .

[25]  S. Chatterjee.  Concentration inequalities with exchangeable pairs (Ph.D. thesis) , 2005, math/0507526.

[26]  S. Chatterjee.  Concentration Inequalities With Exchangeable Pairs , 2005 .

[27]  Mark Rudelson,et al.  Sampling from large matrices: An approach through geometric functional analysis , 2005, JACM.

[28]  M. Junge,et al.  Noncommutative Burkholder/Rosenthal inequalities II: Applications , 2007, 0705.1952.

[29]  Richard H. Liang.  Stein ’ s method for concentration inequalities , 2007 .

[30]  Nicholas J. Higham,et al.  Functions of matrices - theory and computation , 2008 .

[31]  Avi Wigderson,et al.  Derandomizing the Ahlswede-Winter matrix-valued Chernoff bound using pessimistic estimators, and applications , 2008, Theory Comput..

[32]  E. Carlen.  TRACE INEQUALITIES AND QUANTUM ENTROPY: An introductory course , 2009 .

[33]  Z. Bai,et al.  Corrections to LRT on large-dimensional covariance matrix by RMT , 2009, 0902.0552.

[34]  H. Rauhut.  Compressive Sensing and Structured Random Matrices , 2009 .

[35]  R. Oliveira.  Concentration of the adjacency matrix and of the Laplacian in random graphs with independent edges , 2009, 0911.0600.

[36]  Massimo Fornasier,et al.  Compressive Sensing and Structured Random Matrices , 2010 .

[37]  R. Oliveira.  Sums of random Hermitian matrices and an inequality by Rudelson , 2010, 1004.3821.

[38]  Massimo Fornasier,et al.  Theoretical Foundations and Numerical Methods for Sparse Recovery , 2010, Radon Series on Computational and Applied Mathematics.

[39]  Sham M. Kakade,et al.  Dimension-free tail inequalities for sums of random matrices , 2011, ArXiv.

[40]  Shuen Cheung,et al.  Chance – Constrained Linear Matrix Inequalities with Dependent Perturbations : A Safe Tractable Approximation Approach ∗ Sin – , 2011 .

[41]  V. Koltchinskii,et al.  Oracle inequalities in empirical risk minimization and sparse recovery problems , 2011 .

[42]  Nathan Srebro,et al.  Concentration-Based Guarantees for Low-Rank Matrix Reconstruction , 2011, COLT.

[43]  Ameet Talwalkar,et al.  Divide-and-Conquer Matrix Factorization , 2011, NIPS.

[44]  Stanislav Minsker.  On Some Extensions of Bernstein's Inequality for Self-adjoint Operators , 2011, 1112.5448.

[45]  Alex Gittens,et al.  The spectral norm error of the naive Nystrom extension , 2011, ArXiv.

[46]  J. Tropp.  FREEDMAN'S INEQUALITY FOR MATRIX MARTINGALES , 2011, 1101.3039.

[47]  David Gross,et al.  Recovering Low-Rank Matrices From Few Coefficients in Any Basis , 2009, IEEE Transactions on Information Theory.

[48]  Richard Y. Chen,et al.  The Masked Sample Covariance Estimator: An Analysis via Matrix Concentration Inequalities , 2011, 1109.1637.

[49]  Anthony Man-Cho So,et al.  Moment inequalities for sums of random matrices and their applications in optimization , 2011, Math. Program..

[50]  Alex Gittens,et al.  TAIL BOUNDS FOR ALL EIGENVALUES OF A SUM OF RANDOM MATRICES , 2011, 1104.4513.

[51]  Benjamin Recht,et al.  A Simpler Approach to Matrix Completion , 2009, J. Mach. Learn. Res..

[52]  Richard Y. Chen,et al.  The Masked Sample Covariance Estimator: An Analysis via the Matrix Laplace Transform , 2012 .

[53]  Laurent Demanet,et al.  Matrix Probing and its Conditioning , 2012, SIAM J. Numer. Anal..

[54]  Joel A. Tropp,et al.  User-Friendly Tail Bounds for Sums of Random Matrices , 2010, Found. Comput. Math..

[55]  Daniel J. Hsu,et al.  Tail inequalities for sums of random matrices that depend on the intrinsic dimension , 2012 .

[56]  Holger Rauhut,et al.  Compressive Sensing with structured random matrices , 2012 .

[57]  Martin J. Wainwright,et al.  Restricted strong convexity and weighted matrix completion: Optimal bounds with noise , 2010, J. Mach. Learn. Res..

[58]  Albert Cohen,et al.  On the Stability and Accuracy of Least Squares Approximations , 2011, Foundations of Computational Mathematics.

[59]  Laurent Demanet,et al.  Sublinear Randomized Algorithms for Skeleton Decompositions , 2011, SIAM J. Matrix Anal. Appl..

[60]  M. Junge,et al.  Noncommutative Bennett and Rosenthal inequalities , 2011, 1111.1027.