On Annotation and Evaluation of Multi-modal Corpora in Affective Human-Computer Interaction

In this paper, we discuss the topic of affective human-computer interaction from a data driven viewpoint. This comprises the collection of respective databases with emotional contents, feasible annotation procedures and software tools that are able to conduct a suitable labeling process. A further issue that is discussed in this paper is the evaluation of the results that are computed using statistical classifiers. Based on this we propose to use fuzzy memberships in order to model affective user state and endorse respective fuzzy performance measures.

[1]  Sascha Meudt,et al.  Multi classifier systems and forward backward feature selection algorithms to classify emotional coloured speech , 2013, ICMI '13.

[2]  Patrick Thiam,et al.  Majority-Class Aware Support Vector Domain Oversampling for Imbalanced Classification Problems , 2014, ANNPR.

[3]  Sascha Meudt,et al.  Multi-Modal Classifier-Fusion for the Recognition of Emotions , 2013 .

[4]  Günther Palm,et al.  Combination of sequential class distributions from multiple channels using Markov fusion networks , 2014, Journal on Multimodal User Interfaces.

[5]  Friedhelm Schwenker,et al.  Fuzzy-Input Fuzzy-Output One-Against-All Support Vector Machines , 2007, KES.

[6]  Sascha Meudt,et al.  A New Multi-class Fuzzy Support Vector Machine Algorithm , 2014, ANNPR.

[7]  Günther Palm,et al.  Revisiting AVEC 2011 - An Information Fusion Architecture , 2012, WIRN.

[8]  Sascha Meudt,et al.  Developing an Expressive Speech Labeling Tool Incorporating the Temporal Characteristics of Emotion , 2010, LREC.

[9]  Bruno Apolloni.  Neural Nets and Surroundings - 22nd Italian Workshop on Neural Nets, WIRN 2012, Vietri sul Mare, Salerno, Italy, May 17-19, 2012 , 2013, WIRN.

[10]  Markus Kächele,et al.  Classification of Emotional States in a Woz Scenario Exploiting Labeled and Unlabeled Bio-physiological Data , 2011, PSL.

[11]  Sascha Meudt,et al.  Prosodic, Spectral and Voice Quality Feature Selection Using a Long-Term Stopping Criterion for Audio-Based Emotion Recognition , 2014, 2014 22nd International Conference on Pattern Recognition.

[12]  Markus Kächele,et al.  Inferring Depression and Affect from Application Dependent Meta Knowledge , 2014, AVEC '14.

[13]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[14]  Ingo Siegert,et al.  ikannotate - A Tool for Labelling, Transcription, and Annotation of Emotionally Coloured Speech , 2011, ACII.

[15]  Jonghwa Kim,et al.  Transsituational Individual-Specific Biopsychological Classification of Emotions , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[16]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[17]  Friedhelm Schwenker,et al.  Partially Supervised Learning , 2011, Lecture Notes in Computer Science.

[18]  Markus Kächele,et al.  Using unlabeled data to improve classification of emotional states in human computer interaction , 2013, Journal on Multimodal User Interfaces.

[19]  J. Allwood.  A Framework for Studying Human Multimodal Communication , 2013 .

[20]  Björn W. Schuller,et al.  LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework , 2013, Image Vis. Comput..

[21]  Friedhelm Schwenker,et al.  A Multiple Classifier System Approach for Facial Expressions in Image Sequences Utilizing GMM Supervectors , 2010, 2010 20th International Conference on Pattern Recognition.

[22]  Friedhelm Schwenker,et al.  Investigating fuzzy-input fuzzy-output support vector machines for robust voice quality classification , 2013, Comput. Speech Lang..

[23]  Dietmar F. Rösner,et al.  LAST MINUTE: a Multimodal Corpus of Speech-based User-Companion Interactions , 2012, LREC.

[24]  Lakhmi C. Jain,et al.  Knowledge-Based Intelligent Information and Engineering Systems , 2004, Lecture Notes in Computer Science.

[25]  Markus Kächele,et al.  Cascaded Fusion of Dynamic, Spatial, and Textural Feature Sets for Person-Independent Facial Emotion Recognition , 2014, 2014 22nd International Conference on Pattern Recognition.

[26]  Björn W. Schuller,et al.  AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge , 2014, AVEC '14.

[27]  Sascha Meudt,et al.  Fusion of Audio-visual Features using Hierarchical Classifier Systems for the Recognition of Affective States and the State of Depression , 2014, ICPRAM.

[28]  Antonio Torralba,et al.  LabelMe: Online Image Annotation and Applications , 2010, Proceedings of the IEEE.

[29]  Didier Dubois,et al.  Fuzzy sets and systems ' . Theory and applications , 2007 .

[30]  Günther Palm,et al.  Wizard-of-Oz Data Collection for Perception and Interaction in Multi-User Environments , 2006, LREC.

[31]  Friedhelm Schwenker,et al.  Multimodal Emotion Classification in Naturalistic User Behavior , 2011, HCI.

[32]  Frank Honold,et al.  Multimodal Interaction History and its use in Error Detection and Recovery , 2014, ICMI.

[33]  Roddy Cowie,et al.  FEELTRACE: an instrument for recording perceived emotion in real time , 2000 .

[34]  Günther Palm,et al.  How Low Level Observations Can Help to Reveal the User's State in HCI , 2011, ACII.

[35]  Elisabeth André,et al.  Emotion recognition based on physiological changes in music listening , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Matej Rojc,et al.  Coverbal Synchrony in Human-Machine Interaction , 2013 .

[37]  Björn W. Schuller,et al.  AVEC 2011-The First International Audio/Visual Emotion Challenge , 2011, ACII.