Attribute Prototype Learning for Interactive Face Retrieval

Interactive face retrieval aims at finding target subjects in face databases through human and machine interaction, which involves user feedback based on human perception and machine similarity measure in feature spaces. In this article, we propose an attribute prototype learning method to tackle the semantic gap between human and machine in face perception for fast interactive face retrieval. We reformulate the theoretical explanation of the interactive retrieval model and develop the algorithm of the heuristic solution of the model. Each module of the prototype model is learned with a set of identity-related facial attributes. The outputs of the prototype modules form the semantic representation. To adapt the prototype models across different databases, we propose a transfer selection algorithm based on the coherence measurements in interactive face retrieval. Coherence analysis proves that the proposed attribute prototype representation can effectively narrow down the semantic gap even in the case of cross-database transfer learning. The prototype representation can effectively reduce the feature dimension in the retrieval process. Real user retrieval with the Bayesian relevance feedback model shows that attribute prototype space is superior to low-level feature space and proves that interactive retrieval with attribute prototype representation can converge fast in large face databases.

[1]  Yan-Ying Chen,et al.  Scalable Face Image Retrieval Using Attribute-Enhanced Sparse Codewords , 2013, IEEE Transactions on Multimedia.

[2]  Jonathon S. Hare,et al.  Semantic Face Signatures: Recognizing and Retrieving Faces by Verbal Descriptions , 2018, IEEE Transactions on Information Forensics and Security.

[3]  Xiangyang Xue,et al.  Multi-task Deep Neural Network for Joint Face Recognition and Facial Attribute Prediction , 2017, ICMR.

[4]  Karsten Klein,et al.  A Visual Analytics Approach Using the Exploration of Multidimensional Feature Spaces for Content-Based Medical Image Retrieval , 2015, IEEE Journal of Biomedical and Health Informatics.

[5]  Thomas Villmann,et al.  Prototype-based models in machine learning. , 2016, Wiley interdisciplinary reviews. Cognitive science.

[6]  Jun Wu,et al.  Robust discriminative extreme learning machine for relevance feedback in image retrieval , 2017, Multidimens. Syst. Signal Process..

[7]  Wei Zhang,et al.  On the Perception Analysis of User Feedback for Interactive Face Retrieval , 2020, ACM Trans. Appl. Percept..

[8]  Fei Yin,et al.  Robust Classification with Convolutional Prototype Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Ming Shao,et al.  Prototype based feature learning for face image set classification , 2013, 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG).

[10]  Shigeo Morishima,et al.  Face Retrieval Framework Relying on User's Visual Memory , 2018, ICMR.

[11]  Jiwen Lu,et al.  Prototype-Based Discriminative Feature Learning for Kinship Verification , 2015, IEEE Transactions on Cybernetics.

[12]  Shiguang Shan,et al.  Learning Prototype Hyperplanes for Face Verification in the Wild , 2013, IEEE Transactions on Image Processing.

[13]  Trevor Darrell,et al.  Transfer learning for image classification with sparse prototype representations , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Ioannis A. Kakadiaris,et al.  Joint prototype and metric learning for set-to-set matching: Application to biometrics , 2015, 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[15]  Frank-Michael Schleif,et al.  Low-Rank Kernel Space Representations in Prototype Learning , 2016, WSOM.

[16]  Ying Tan,et al.  Coherence Analysis of Metrics in LBP Space for Interactive Face Retrieval , 2014, MMM.

[17]  Jaeyeon Lee,et al.  Facial Attribute Recognition by Recurrent Learning With Visual Fixation , 2019, IEEE Transactions on Cybernetics.

[18]  Yuchun Fang,et al.  Feature selection in interactive face retrieval , 2011, 2011 4th International Congress on Image and Signal Processing.

[19]  Marwan Mattar,et al.  Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments , 2008 .

[20]  Anil K. Jain,et al.  Face Matching and Retrieval Using Soft Biometrics , 2010, IEEE Transactions on Information Forensics and Security.

[21]  Adriana Kovashka,et al.  WhittleSearch: Image search with relative attribute feedback , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Lorenzo Bruzzone,et al.  A novel active learning method for content based remote sensing image retrieval , 2015, 2015 23nd Signal Processing and Communications Applications Conference (SIU).

[23]  Adriana Kovashka,et al.  WhittleSearch: Interactive Image Search with Relative Attribute Feedback , 2015, International Journal of Computer Vision.

[24]  Ran He,et al.  Reducing Impact of Inaccurate User Feedback in Face Retrieval , 2008, 2008 Chinese Conference on Pattern Recognition.

[25]  Shiguang Shan,et al.  Heterogeneous Face Attribute Estimation: A Deep Multi-Task Learning Approach , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Cheng-Lin Liu.  One-Vs-All Training of Prototype Classifier for Pattern Classification and Retrieval , 2010, 2010 20th International Conference on Pattern Recognition.

[27]  Xiaoou Tang,et al.  Learning Deep Representation for Face Alignment with Auxiliary Attributes , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Bart Thomee,et al.  Interactive search in image retrieval: a survey , 2012, International Journal of Multimedia Information Retrieval.

[29]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Shuicheng Yan,et al.  Recognizing Profile Faces by Imagining Frontal View , 2019, International Journal of Computer Vision.

[31]  Swami Sankaranarayanan,et al.  Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms , 2018, Proceedings of the National Academy of Sciences.

[32]  W. T. Maddox,et al.  Dissociable Prototype Learning Systems: Evidence from Brain Imaging and Behavior , 2008, The Journal of Neuroscience.

[33]  R. L. Solso,et al.  Prototype formation of faces: A case of pseudo-memory , 1981 .

[34]  Rama Chellappa,et al.  Attributes for Improved Attributes: A Multi-Task Network Utilizing Implicit and Explicit Relationships for Facial Attribute Classification , 2017, AAAI.

[35]  Dacheng Tao,et al.  Relative Attribute SVM+ Learning for Age Estimation , 2016, IEEE Transactions on Cybernetics.

[36]  Ying Tan,et al.  Prototype Generation Using Multiobjective Particle Swarm Optimization for Nearest Neighbor Classification , 2016, IEEE Transactions on Cybernetics.

[37]  Fang Zhao,et al.  Multi-Prototype Networks for Unconstrained Set-based Face Recognition , 2019, IJCAI.

[38]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[39]  Xiaogang Wang,et al.  A Deep Sum-Product Architecture for Robust Facial Attributes Analysis , 2013, 2013 IEEE International Conference on Computer Vision.

[40]  Yue Lu,et al.  An Image-Based Approach to Detection of Fake Coins , 2017, IEEE Transactions on Information Forensics and Security.

[41]  Ricardo da Silva Torres,et al.  On interactive learning-to-rank for IR: Overview, recent advances, challenges, and directions , 2016, Neurocomputing.

[42]  Yuchun Fang,et al.  Experiments in Mental Face Retrieval , 2005, AVBPA.

[43]  Tal Hassner,et al.  Face recognition in unconstrained videos with matched background similarity , 2011, CVPR 2011.

[44]  Yuchun Fang,et al.  Attribute-enhanced metric learning for face retrieval , 2018, EURASIP J. Image Video Process..

[45]  Shuicheng Yan,et al.  3D-Aided Dual-Agent GANs for Unconstrained Face Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Yidong Li,et al.  Fine-Grained Facial Expression Recognition in the Wild , 2021, IEEE Transactions on Information Forensics and Security.

[47]  Suresh Sundaram,et al.  Online Writer Identification With Sparse Coding-Based Descriptors , 2018, IEEE Transactions on Information Forensics and Security.

[48]  Richa Singh,et al.  RGB-D Face Recognition With Texture and Attribute Features , 2014, IEEE Transactions on Information Forensics and Security.

[49]  Ricardo da Silva Torres,et al.  Diversity-based interactive learning meets multimodality , 2017, Neurocomputing.

[50]  Jean-Yves Baudouin,et al.  Gender-based prototype formation in face recognition. , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[51]  Shree K. Nayar,et al.  Attribute and simile classifiers for face verification , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[52]  Mohamed R. Amer,et al.  Facial Attributes Classification Using Multi-task Representation Learning , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[53]  Adriana Kovashka,et al.  Attribute Pivots for Guiding Relevance Feedback in Image Search , 2013, 2013 IEEE International Conference on Computer Vision.

[54]  Yang Zhong,et al.  Face attribute prediction using off-the-shelf CNN features , 2016, 2016 International Conference on Biometrics (ICB).

[55]  Chi-Man Pun,et al.  Chronological Age Estimation Under the Guidance of Age-Related Facial Attributes , 2019, IEEE Transactions on Information Forensics and Security.

[56]  Andrew Zisserman,et al.  Deep Face Recognition , 2015, BMVC.

[57]  Yuchun Fang,et al.  A Bi-objective Optimization Model for Interactive Face Retrieval , 2011, MMM.

[58]  Xiaogang Wang,et al.  Deep Learning Face Attributes in the Wild , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[59]  Yu-Heng Lei,et al.  Where is who: large-scale photo retrieval by facial attributes and canvas layout , 2012, SIGIR '12.

[60]  Jie Li,et al.  Bayesian Face Sketch Synthesis , 2017, IEEE Transactions on Image Processing.