Polynomial time algorithms for dual volume sampling

We study dual volume sampling, a method for selecting k columns from an n*m short and wide matrix (n <= k <= m) such that the probability of selection is proportional to the volume spanned by the rows of the induced submatrix. This method was proposed by Avron and Boutsidis (2013), who showed it to be a promising method for column subset selection and its multiple applications. However, its wider adoption has been hampered by the lack of polynomial time sampling algorithms. We remove this hindrance by developing an exact (randomized) polynomial time sampling algorithm as well as its derandomization. Thereafter, we study dual volume sampling via the theory of real stable polynomials and prove that its distribution satisfies the “Strong Rayleigh” property. This result has numerous consequences, including a provably fast-mixing Markov chain sampler that makes dual volume sampling much more attractive to practitioners. This sampler is closely related to classical algorithms for popular experimental design methods that are to date lacking theoretical analysis but are known to empirically work well.

[1]  D. Wagner,et al.  Multivariate stable polynomials: theory and applications , 2009, 0911.3569.

[2]  Shang-Hua Teng,et al.  Lower-stretch spanning trees , 2004, STOC '05.

[3]  W. J. Studden,et al.  Theory Of Optimal Experiments , 1972 .

[4]  Christos Boutsidis,et al.  Near-Optimal Column-Based Matrix Reconstruction , 2014, SIAM J. Comput..

[5]  O. Macchi.  The coincidence approach to stochastic point processes , 1975, Advances in Applied Probability.

[6]  R. Lyons.  Determinantal probability measures , 2002, math/0204325.

[7]  Christos Boutsidis,et al.  Deterministic Feature Selection for K-Means Clustering , 2011, IEEE Transactions on Information Theory.

[8]  Christos Boutsidis,et al.  Stochastic Dimensionality Reduction for K-means Clustering , 2011, ArXiv.

[9]  Christos Boutsidis,et al.  Randomized Dimensionality Reduction for $k$ -Means Clustering , 2011, IEEE Transactions on Information Theory.

[10]  Christos Boutsidis,et al.  An improved approximation algorithm for the column subset selection problem , 2008, SODA.

[11]  Shang-Hua Teng,et al.  Nearly-linear time algorithms for graph partitioning, graph sparsification, and solving linear systems , 2003, STOC '04.

[12]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[13]  Nima Anari,et al.  Effective-Resistance-Reducing Flows and Asymmetric TSP , 2014, ArXiv.

[14]  Manfred K. Warmuth,et al.  Unbiased estimates for linear regression via volume sampling , 2017, NIPS.

[15]  Christos Boutsidis,et al.  Faster Subset Selection for Matrices and Applications , 2011, SIAM J. Matrix Anal. Appl..

[16]  Avner Magen,et al.  Near Optimal Dimensionality Reductions That Preserve Volumes , 2008, APPROX-RANDOM.

[17]  F. Pukelsheim.  Optimal Design of Experiments , 1993 .

[18]  Adams Wei Yu,et al.  On Computationally Tractable Selection of Experiments in Regression Models , 2016 .

[19]  Yuval Peres,et al.  Concentration of Lipschitz Functionals of Determinantal and Other Strong Rayleigh Measures , 2011, Combinatorics, Probability and Computing.

[20]  Malik Magdon-Ismail,et al.  On selecting a maximum volume sub-matrix of a matrix and related problems , 2009, Theor. Comput. Sci..

[21]  Luis Rademacher,et al.  Efficient Volume Sampling for Row/Column Subset Selection , 2010, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[22]  Nikhil Srivastava,et al.  Graph Sparsification by Effective Resistances , 2011, SIAM J. Comput..

[23]  Alexei Borodin,et al.  Determinantal point processes , 2009, 0911.1153.

[24]  Tomás Feder,et al.  Balanced matroids , 1992, STOC '92.

[25]  J. Borcea,et al.  Applications of stable polynomials to mixed determinants: Johnson's conjectures, unimodality, and symmetrized Fischer products , 2006, math/0607755.

[26]  Stephen P. Boyd,et al.  Sensor Selection via Convex Optimization , 2009, IEEE Transactions on Signal Processing.

[27]  Suvrit Sra,et al.  Fast Mixing Markov Chains for Strongly Rayleigh Measures, DPPs, and Constrained Sampling , 2016, NIPS.

[28]  T. Liggett,et al.  Negative dependence and the geometry of polynomials , 2007, 0707.2340.

[29]  Alan J. Miller,et al.  A Fedorov Exchange Algorithm for D-optimal Design , 1994 .

[30]  Ping Ma,et al.  A statistical perspective on algorithmic leveraging , 2013, J. Mach. Learn. Res..

[31]  Nima Anari,et al.  The Kadison-Singer Problem for Strongly Rayleigh Measures and Applications to Asymmetric TSP , 2014, ArXiv.

[32]  Alan J. Miller,et al.  A review of some exchange algorithms for constructing discrete D-optimal designs , 1992 .

[33]  Navin Goyal,et al.  Expanders via random spanning trees , 2008, SODA.

[34]  Jelena Kovacevic,et al.  Discrete Signal Processing on Graphs: Sampling Theory , 2015, IEEE Transactions on Signal Processing.

[35]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[36]  T. J. Mitchell,et al.  Exploratory designs for computational experiments , 1995 .

[37]  Is Duff,et al.  Preconditioning of linear least-squares problems by identifying basic variables , 2014 .

[38]  Ben Taskar,et al.  Determinantal Point Processes for Machine Learning , 2012, Found. Trends Mach. Learn..

[39]  Michael W. Mahoney,et al.  Optimal Subsampling Approaches for Large Sample Linear Regression , 2015, 1509.05111.

[40]  Mohit Singh,et al.  A Randomized Rounding Approach to the Traveling Salesman Problem , 2011, 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science.

[41]  R. Pemantle.  Towards a theory of negative dependence , 2000, math/0404095.

[42]  Suvrit Sra,et al.  Gaussian quadrature for matrix inverse forms with applications , 2015, ICML.

[43]  Nima Anari,et al.  Monte Carlo Markov Chain Algorithms for Sampling Strongly Rayleigh Distributions and Determinantal Point Processes , 2016, COLT.

[44]  Sergio Barbarossa,et al.  Signals on Graphs: Uncertainty Principle and Sampling , 2015, IEEE Transactions on Signal Processing.

[45]  Jorge Cortés,et al.  Scheduling of control nodes for improved network controllability , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).