Non-Negative Matrix Factorization via Low-Rank Stochastic Manifold Optimization

Several real-world applications, notably in non-negative matrix factorization, graph-based clustering, and machine learning, require solving a convex optimization problem over the set of stochastic and doubly stochastic matrices. A common feature of these problems is that the optimal solution is generally a low-rank matrix. This paper suggests reformulating the problem by taking advantage of the low-rank factorization X = UVT and develops a Riemannian optimization framework for solving optimization problems on the set of low-rank stochastic and doubly stochastic matrices. In particular, this paper introduces and studies the geometry of the low-rank stochastic multinomial and the doubly stochastic manifold in order to derive first-order optimization algorithms. Being carefully designed and of lower dimension than the original problem, the proposed Riemannian optimization framework presents a clear complexity advantage. The claim is attested through numerical experiments on real-world and synthetic data for Non-negative Matrix Factorization (NFM) applications. The proposed algorithm is shown to outperform, in terms of running time, state-of-the-art methods for NFM.

[1]  Amnon Shashua,et al.  Doubly Stochastic Normalization for Spectral Clustering , 2006, NIPS.

[2]  Nicolas Boumal,et al.  The non-convex Burer-Monteiro approach works on smooth semidefinite programs , 2016, NIPS.

[3]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[4]  Babak Hassibi,et al.  Similarity clustering in the presence of outliers: Exact recovery via convex program , 2016, 2016 IEEE International Symposium on Information Theory (ISIT).

[5]  Erkki Oja,et al.  Clustering by Low-Rank Doubly Stochastic Matrix Decomposition , 2012, ICML.

[6]  Babak Hassibi,et al.  A Riemannian Approach for Graph-Based Clustering by Doubly Stochastic Matrices , 2018, 2018 IEEE Statistical Signal Processing Workshop (SSP).

[7]  Bamdev Mishra,et al.  Fixed-rank matrix factorizations and Riemannian low-rank optimization , 2012, Comput. Stat..

[8]  Nicolas Gillis,et al.  The Why and How of Nonnegative Matrix Factorization , 2014, ArXiv.

[9]  Feiping Nie,et al.  Structured Doubly Stochastic Matrix for Graph Based Clustering: Structured Doubly Stochastic Matrix , 2016, KDD.

[10]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[11]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[12]  Babak Hassibi,et al.  Low-Rank Riemannian Optimization on Positive Semidefinite Stochastic Matrices with Applications to Graph Clustering , 2018, ICML.

[13]  Erkki Oja,et al.  Low-Rank Doubly Stochastic Matrix Decomposition for Cluster Analysis , 2016, J. Mach. Learn. Res..

[14]  Junbin Gao,et al.  Heterogeneous Tensor Decomposition for Clustering via Manifold Optimization , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Maya R. Gupta,et al.  Clustering by Left-Stochastic Matrix Factorization , 2011, ICML.

[16]  Christopher Adams,et al.  Stochastic Matrix Factorization , 2016, ArXiv.

[17]  Nicolas Gillis,et al.  Accelerated Multiplicative Updates and Hierarchical ALS Algorithms for Nonnegative Matrix Factorization , 2011, Neural Computation.

[18]  Babak Hassibi,et al.  Manifold Optimization Over the Set of Doubly Stochastic Matrices: A Second-Order Geometry , 2018, IEEE Transactions on Signal Processing.