LanczosNet: Multi-Scale Deep Graph Convolutional Networks

We propose the Lanczos network (LanczosNet), which uses the Lanczos algorithm to construct low rank approximations of the graph Laplacian for graph convolution. Relying on the tridiagonal decomposition of the Lanczos algorithm, we not only efficiently exploit multi-scale information via fast approximated computation of matrix power but also design learnable spectral filters. Being fully differentiable, LanczosNet facilitates both graph kernel learning as well as learning node embeddings. We show the connection between our LanczosNet and graph based manifold learning methods, especially the diffusion maps. We benchmark our model against several recent deep graph networks on citation networks and QM8 quantum chemistry dataset. Experimental results show that our model achieves the state-of-the-art performance in most tasks. Code is released at: \url{this https URL}.

[1]  William H. Press,et al.  Numerical Recipes 3rd Edition: The Art of Scientific Computing , 2007 .

[2]  Joan Bruna,et al.  Community Detection with Graph Neural Networks , 2017 .

[3]  Sanja Fidler,et al.  3D Graph Neural Networks for RGBD Semantic Segmentation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Yoel Shkolnisky,et al.  Diffusion Interpretation of Nonlocal Neighborhood Filters for Signal Denoising , 2009, SIAM J. Imaging Sci..

[5]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[6]  A. Singer.  From graph to manifold Laplacian: The convergence rate , 2006 .

[7]  Raquel Urtasun,et al.  Deep Parametric Continuous Convolutional Neural Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Pierre Vandergheynst,et al.  Wavelets on Graphs via Spectral Graph Theory , 2009, ArXiv.

[9]  Mathias Niepert,et al.  Learning Graph Representations with Embedding Propagation , 2017, NIPS.

[10]  Razvan Pascanu,et al.  Learning Deep Generative Models of Graphs , 2018, ICLR 2018.

[11]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[12]  R. Coifman,et al.  Non-linear independent component analysis with diffusion maps , 2008 .

[13]  O. A. von Lilienfeld,et al.  Electronic spectra from TDDFT and machine learning in chemical space. , 2015, The Journal of chemical physics.

[14]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[15]  C. Lanczos.  An iteration method for the solution of the eigenvalue problem of linear differential and integral operators , 1950 .

[16]  Le Song,et al.  Discriminative Embeddings of Latent Variable Models for Structured Data , 2016, ICML.

[17]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[18]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[19]  Mikhail Belkin,et al.  Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples , 2006, J. Mach. Learn. Res..

[20]  Mathias Niepert,et al.  Learning Convolutional Neural Networks for Graphs , 2016, ICML.

[21]  Amit Singer,et al.  Detecting intrinsic slow variables in stochastic dynamical systems by anisotropic diffusion maps , 2009, Proceedings of the National Academy of Sciences.

[22]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[23]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[24]  B. Parlett.  The Symmetric Eigenvalue Problem , 1981 .

[25]  Renjie Liao,et al.  Graph Partition Neural Networks for Semi-Supervised Classification , 2018, ICLR.

[26]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[27]  Jordan B. Pollack,et al.  Recursive Distributed Representations , 1990, Artif. Intell..

[28]  R R Coifman,et al.  Geometric diffusions as a tool for harmonic analysis and structure definition of data: multiscale methods. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[29]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[30]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[31]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[32]  Ronen Talmon,et al.  Empirical intrinsic geometry for nonlinear modeling and time series filtering , 2013, Proceedings of the National Academy of Sciences.

[33]  Le Song,et al.  Stochastic Training of Graph Convolutional Networks with Variance Reduction , 2017, ICML.

[34]  Joan Bruna,et al.  Deep Convolutional Networks on Graph-Structured Data , 2015, ArXiv.

[35]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[36]  Ronald R. Coifman,et al.  Diffusion Maps, Spectral Clustering and Eigenfunctions of Fokker-Planck Operators , 2005, NIPS.

[37]  P. Vandergheynst,et al.  Accelerated filtering on graphs using Lanczos method , 2015, 1509.04537.

[38]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[39]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[40]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[41]  Raia Hadsell,et al.  Graph networks as learnable physics engines for inference and control , 2018, ICML.

[42]  Donald F. Towsley,et al.  Diffusion-Convolutional Neural Networks , 2015, NIPS.

[43]  Regina Barzilay,et al.  Junction Tree Variational Autoencoder for Molecular Graph Generation , 2018, ICML.

[44]  Pierre Vandergheynst,et al.  Graph Signal Processing: Overview, Challenges, and Applications , 2017, Proceedings of the IEEE.

[45]  Kilian Q. Weinberger,et al.  Metric Learning for Kernel Regression , 2007, AISTATS.

[46]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[47]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[48]  Zhizhen Zhao,et al.  Analog forecasting with dynamics-adapted kernels , 2014, 1412.3831.

[49]  John D. Lafferty,et al.  Diffusion Kernels on Statistical Manifolds , 2005, J. Mach. Learn. Res..

[50]  Xiaojin Zhu,et al.  Semi-Supervised Learning Literature Survey , 2005 .

[51]  Dimitrios Giannakis,et al.  Dynamics-Adapted Cone Kernels , 2014, SIAM J. Appl. Dyn. Syst..

[52]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[53]  Hongyuan Zha,et al.  Low-Rank Matrix Approximation Using the Lanczos Bidiagonalization Process with Applications , 1999, SIAM J. Sci. Comput..

[54]  Sanja Fidler,et al.  NerveNet: Learning Structured Policy with Graph Neural Networks , 2018, ICLR.

[55]  D. Donoho,et al.  Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[56]  Pascal Frossard,et al.  The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains , 2012, IEEE Signal Processing Magazine.

[57]  Pietro Perona,et al.  Self-Tuning Spectral Clustering , 2004, NIPS.

[58]  Joseph Gomes,et al.  MoleculeNet: a benchmark for molecular machine learning† †Electronic supplementary information (ESI) available. See DOI: 10.1039/c7sc02664a , 2017, Chemical science.

[59]  Joan Bruna,et al.  Few-Shot Learning with Graph Neural Networks , 2017, ICLR.

[60]  Fan Chung,et al.  Spectral Graph Theory , 1996 .

[61]  Sanja Fidler,et al.  Situation Recognition with Graph Neural Networks , 2018 .

[62]  Andrew P. Witkin,et al.  Scale-Space Filtering , 1983, IJCAI.

[63]  Neta Rabin,et al.  Multi-scale kernels for Nyström based extension schemes , 2018, Appl. Math. Comput..

[64]  Joan Bruna,et al.  Spectral Networks and Locally Connected Networks on Graphs , 2013, ICLR.

[65]  Lisa Zhang,et al.  Inference in Probabilistic Graphical Models by Graph Neural Networks , 2018, 2019 53rd Asilomar Conference on Signals, Systems, and Computers.

[66]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.