Geometric Multimodal Learning Based on Local Signal Expansion for Joint Diagonalization

Multimodal learning, also known as multi-view learning, data integration, or data fusion, is an emerging field in signal processing, machine learning, and pattern recognition domains. It aims at building models, learned from several related and complementary modalities, in order to increase the generalization performances of a predictive learning model. Multimodal manifold learning extends spectral or diffusion geometry-aware data analysis to multiple modalities. This can be performed through the definition of undirected graph Laplacian matrices in different modalities. However, finding common eigenbasis of multiple Laplacians is not always a relevant solution for multimodal manifold learning problems. As a matter of fact, the Laplacians of all modalities are not simultaneously diagonalizable in many real-world problems due to the major differences between the different modalities. In this paper, we propose a multimodal manifold learning approach based on intrinsic local tangent spaces of underlying data manifolds in order to discover the local geometrical structure around matching and mismatching samples in different modalities in sparse diagonalization problems. This approach searches for approximate common eigenbasis of Laplacian matrices by expanding the signal of limited existing information about matching and mismatching samples of different modalities to their on-manifold neighbors. Experiments on synthetic and real-world datasets in supervised, unsupervised, and semi-supervised problems demonstrate the superiority of our proposed approach over existing state-of-the-art related methods.

[1]  Shiliang Sun,et al.  Local Tangent Space Discriminant Analysis , 2016, Neural Processing Letters.

[2]  Antoine Souloumiac,et al.  Jacobi Angles for Simultaneous Diagonalization , 1996, SIAM J. Matrix Anal. Appl..

[3]  Peyman Adibi,et al.  Two-stage multiple kernel learning for supervised dimensionality reduction , 2015, Pattern Recognit..

[4]  Ting Wang,et al.  Semi-supervised cross-modal common representation learning with vector-valued manifold regularization , 2020, Pattern Recognit. Lett..

[5]  Christian Jutten,et al.  Multimodal Soft Nonnegative Matrix Co-Factorization for Convolutive Source Separation , 2017, IEEE Transactions on Signal Processing.

[6]  Rong Wang,et al.  Parameter-Free Weighted Multi-View Projected Clustering with Structured Graph Learning , 2020, IEEE Transactions on Knowledge and Data Engineering.

[7]  Jocelyn Chanussot,et al.  Semisupervised charting for spectral multimodal manifold learning and alignment , 2021, Pattern Recognit..

[8]  Pascal Frossard,et al.  Clustering on Multi-Layer Graphs via Subspace Analysis on Grassmann Manifolds , 2013, IEEE Transactions on Signal Processing.

[9]  Alexander M. Bronstein,et al.  Coupled quasi‐harmonic bases , 2012, Comput. Graph. Forum.

[10]  Ling Chen,et al.  Multi-layer multi-view topic model for classifying advertising video , 2017, Pattern Recognit..

[11]  Xuelong Li,et al.  Auto-Weighted Multi-View Learning for Image Clustering and Semi-Supervised Classification , 2018, IEEE Transactions on Image Processing.

[12]  Elif Vural,et al.  Domain adaptation via transferring spectral properties of label functions on graphs , 2016, 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[13]  Jing Huang,et al.  Audio-visual deep learning for noise robust speech recognition , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[14]  Xianglei Xing,et al.  A Fusion Scheme of Local Manifold Learning Methods , 2017 .

[15]  Naoto Yokoya,et al.  CoSpace: Common Subspace Learning From Hyperspectral-Multispectral Correspondences , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[16]  Louis-Philippe Morency,et al.  Multimodal Machine Learning: A Survey and Taxonomy , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  C. Jacobi Über ein leichtes Verfahren die in der Theorie der Säcularstörungen vorkommenden Gleichungen numerisch aufzulösen*). , 2022 .

[18]  Junbin Gao,et al.  Multiview Subspace Clustering via Tensorial t-Product Representation , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Feiping Nie,et al.  Heterogeneous image feature integration via multi-modal spectral clustering , 2011, CVPR 2011.

[20]  Ronen Talmon,et al.  Parallel Transport on the Cone Manifold of SPD Matrices for Domain Adaptation , 2018, IEEE Transactions on Signal Processing.

[21]  Hong Qiao,et al.  An improved local tangent space alignment method for manifold learning , 2011, Pattern Recognit. Lett..

[22]  Xuelong Li,et al.  Multi-view Subspace Clustering , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[23]  Wei Zhang,et al.  Consistent and Specific Multi-View Subspace Clustering , 2018, AAAI.

[24]  Tat-Seng Chua,et al.  NUS-WIDE: a real-world web image database from National University of Singapore , 2009, CIVR '09.

[25]  Arie Yeredor,et al.  Non-orthogonal joint diagonalization in the least-squares sense with application in blind source separation , 2002, IEEE Trans. Signal Process..

[26]  Nassir Navab,et al.  Manifold Learning for Multi-Modal Image Registration , 2010, BMVC.

[27]  Pascal Frossard,et al.  Tangent space estimation for smooth embeddings of Riemannian manifolds , 2012 .

[28]  Davide Eynard,et al.  Multimodal diffusion geometry by joint diagonalization of Laplacians , 2012, ArXiv.

[29]  Jun Li,et al.  Nonparametric discriminant multi-manifold learning for dimensionality reduction , 2015, Neurocomputing.

[30]  Pietro Perona,et al.  One-shot learning of object categories , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Wei Yuan,et al.  Multi-view manifold learning with locality alignment , 2018, Pattern Recognit..

[32]  Pramod K. Varshney,et al.  Compressive Sensing-Based Detection With Multimodal Dependent Data , 2017, IEEE Transactions on Signal Processing.

[33]  Jocelyn Chanussot,et al.  Hyperspectral Anomaly Detection via Global and Local Joint Modeling of Background , 2019, IEEE Transactions on Signal Processing.

[34]  Pietro Perona,et al.  Self-Tuning Spectral Clustering , 2004, NIPS.

[35]  Junbin Gao,et al.  Shared Generative Latent Representation Learning for Multi-view Clustering , 2019, AAAI.

[36]  Feiping Nie,et al.  Large-Scale Multi-View Spectral Clustering via Bipartite Graph , 2015, AAAI.

[37]  Xuelong Li,et al.  Self-weighted Multiview Clustering with Multiple Graphs , 2017, IJCAI.

[38]  Marc Niethammer,et al.  Multi-modal registration for correlative microscopy using image analogies , 2014, Medical Image Anal..

[39]  Tong Lu,et al.  Learning discriminated and correlated patches for multi-view object detection using sparse coding , 2017, Pattern Recognit..

[40]  Naoto Yokoya,et al.  Learnable manifold alignment (LeMA): A semi-supervised cross-modality learning framework for land cover and land use classification , 2019, ISPRS journal of photogrammetry and remote sensing : official publication of the International Society for Photogrammetry and Remote Sensing.

[41]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[42]  Davide Eynard,et al.  Multimodal Manifold Analysis by Simultaneous Diagonalization of Laplacians , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Vittorio Murino,et al.  A Unifying Framework in Vector-valued Reproducing Kernel Hilbert Spaces for Manifold Regularization and Co-Regularized Multi-view Learning , 2014, J. Mach. Learn. Res..

[44]  Francesco Camastra,et al.  Data dimensionality estimation methods: a survey , 2003, Pattern Recognit..

[45]  Xiao Xiang Zhu,et al.  MIMA: MAPPER-Induced Manifold Alignment for Semi-Supervised Fusion of Optical Image and Polarimetric SAR Data , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[46]  Christian Jutten,et al.  Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects , 2015, Proceedings of the IEEE.

[47]  Gustau Camps-Valls,et al.  Kernel Manifold Alignment for Domain Adaptation , 2015, PloS one.