Understanding emotional impact of images using Bayesian multiple kernel learning

Affective classification and retrieval of multimedia such as audio, image, and video have become emerging research areas in recent years. The previous research focused on designing features and developing feature extraction methods. Generally, a multimedia content can be represented with different feature representations (i.e., views). However, the most suitable feature representation related to people?s emotions is usually not known a priori. We propose here a novel Bayesian multiple kernel learning algorithm for affective classification and retrieval tasks. The proposed method can make use of different representations simultaneously (i.e., multiview learning) to obtain a better prediction performance than using a single feature representation (i.e., single-view learning) or a subset of features, with the advantage of automatic feature selections. In particular, our algorithm has been implemented within a multilabel setup to capture the correlation between emotions, and the Bayesian formulation enables our method to produce probabilistic outputs for measuring a set of emotions triggered by a single image. As a case study, we perform classification and retrieval experiments with our algorithm for predicting people?s emotional states evoked by images, using generic low-level image features. The empirical results with our approach on the widely-used International Affective Picture System (IAPS) data set outperform several existing methods in terms of classification performance and results interpretability.

[1]  Nicu Sebe,et al.  Emotional valence categorization using holistic image features , 2008, 2008 15th IEEE International Conference on Image Processing.

[2]  Allan Hanbury,et al.  Affective image classification using features inspired by psychology and art theory , 2010, ACM Multimedia.

[3]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[4]  Zenglin Xu,et al.  Simple and Efficient Multiple Kernel Learning by Group Lasso , 2010, ICML.

[5]  Sam J. Maglio,et al.  Emotional category data on images from the international affective picture system , 2005, Behavior research methods.

[6]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[7]  Mehryar Mohri,et al.  Learning Non-Linear Combinations of Kernels , 2009, NIPS.

[8]  Mehmet Gönen,et al.  Bayesian Efficient Multiple Kernel Learning , 2012, ICML.

[9]  Alan Hanjalic,et al.  Extracting Moods from Pictures and Sounds , 2006 .

[10]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[11]  Jorma Laaksonen,et al.  Analyzing Emotional Semantics of Abstract Art Using Low-Level Image Features , 2011, IDA.

[12]  Yu Ying-lin,et al.  Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction , 2006, 2006 IEEE International Conference on Systems, Man and Cybernetics.

[13]  Nadia Bianchi-Berthouze,et al.  K-DIME: An Affective Image Filtering System , 2003, IEEE Multim..

[14]  R. Arnheim,et al.  Art and Visual Perception: A Psychology of the Creative Eye. , 1956 .

[15]  R. Arnheim Art and visual perception: A psychology of the creative eye, New version , 1955 .

[16]  Alberto Del Bimbo,et al.  Semantics in Visual Information Retrieval , 1999, IEEE Multim..

[17]  Irina Rish,et al.  An empirical study of the naive Bayes classifier , 2001 .

[18]  Tsuhan Chen,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < , 2022 .

[19]  J. Russell,et al.  Evidence for a three-factor theory of emotions , 1977 .

[20]  A. Hanjalic,et al.  Extracting moods from pictures and sounds: towards truly personalized TV , 2006, IEEE Signal Processing Magazine.

[21]  Claudio Gutierrez,et al.  Survey of graph database models , 2008, CSUR.

[22]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[23]  Jiebo Luo,et al.  Learning multi-label scene classification , 2004, Pattern Recognit..

[24]  C. Osgood,et al.  The Measurement of Meaning , 1958 .

[25]  Tong Zhang,et al.  An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods , 2001, AI Mag..

[26]  L. Ou,et al.  A study of colour emotion and colour preference. Part I: Colour emotions for single colours , 2004 .

[27]  Thomas Sikora,et al.  The MPEG-7 visual standard for content description-an overview , 2001, IEEE Trans. Circuits Syst. Video Technol..

[28]  Lior Shamir,et al.  Impressionism, expressionism, surrealism: Automated recognition of painters and schools of art , 2010, TAP.

[29]  Erkki Oja,et al.  PicSOM-self-organizing image retrieval with MPEG-7 content descriptors , 2002, IEEE Trans. Neural Networks.

[30]  W. Chu Studying Aesthetics in Photographic Images Using a Computational Approach , 2013 .

[31]  P. Lang International affective picture system (IAPS) : affective ratings of pictures and instruction manual , 2005 .

[32]  Manik Varma,et al.  More generality in efficient multiple kernel learning , 2009, ICML '09.

[33]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[34]  Shengming Jiang,et al.  Image Retrieval by Emotional Semantics: A Study of Emotional Space and Feature Extraction , 2006, SMC.

[35]  Matthew J. Beal Variational algorithms for approximate Bayesian inference , 2003 .

[36]  James Ze Wang,et al.  On shape and the computability of emotions , 2012, ACM Multimedia.

[37]  James Ze Wang,et al.  Studying Aesthetics in Photographic Images Using a Computational Approach , 2006, ECCV.

[38]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[39]  Ethem Alpaydin,et al.  Multiple Kernel Learning Algorithms , 2011, J. Mach. Learn. Res..

[40]  Rosalind W. Picard Affective Computing , 1997 .

[41]  Neil D. Lawrence,et al.  Semi-supervised Learning via Gaussian Processes , 2004, NIPS.

[42]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[43]  Qianhua He,et al.  A survey on emotional semantic image retrieval , 2008, 2008 15th IEEE International Conference on Image Processing.

[44]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[45]  Changle Zhou,et al.  Content-Based Affective Image Classification and Retrieval Using Support Vector Machines , 2005, ACII.

[46]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .