How Low Level Observations Can Help to Reveal the User's State in HCI

For next generation human computer interaction (HCI), it is crucial to assess the affective state of a user. However, this respective user state is - even for human annotators - only indirectly inferable using background information and the observation of the interaction's progression as well as the social signals produced by the interlocutors. In this paper, coincidences of directly observable patterns and different user states are examined in order to relate the former to the latter. This evaluation motivates a hierarchical label system, where labels of latent user states are supported by low level observations. The dynamic patterns of occurrences of various social signals may in an integration step infer the latent user's state. Thus, we expect to advance the understanding of the recognition of affective user states as compositions of lower level observations for automatic classifiers in HCI.

[1]  C. Gobl,et al.  Voice quality and loudness in affect perception , 2008 .

[2]  J. Russell,et al.  Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. , 1999 .

[3]  Nick Campbell,et al.  On the Use of NonVerbal Speech Sounds in Human Communication , 2007, COST 2102 Workshop.

[4]  Günther Palm,et al.  Wizard-of-Oz Data Collection for Perception and Interaction in Multi-User Environments , 2006, LREC.

[5]  C. Darwin The Expression of Emotion in Man and Animals , 2020 .

[6]  A. Kendon,et al.  Nonverbal Communication, Interaction, and Gesture , 1981 .

[7]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[8]  Günther Palm,et al.  Spotting laughter in natural multiparty conversations: A comparison of automatic online and offline approaches using audiovisual data , 2012, TIIS.

[9]  Günther Palm,et al.  Multimodal Laughter Detection in Natural Discourses , 2009, Human Centered Robot Systems, Cognition, Interaction, Technology.

[10]  Anna Esposito Verbal and Nonverbal Communication Behaviours, COST Action 2102 International Workshop, Vietri sul Mare, Italy, March 29-31, 2007, Revised Selected and Invited Papers , 2007, COST 2102 Workshop.

[11]  Günther Palm,et al.  The PIT Corpus of German Multi-Party Dialogues , 2008, LREC.

[12]  P. Ekman Facial expression and emotion. , 1993, The American psychologist.

[13]  George N. Votsis,et al.  Emotion recognition in human-computer interaction , 2001, IEEE Signal Process. Mag..

[14]  Holger Hoffmann,et al.  Evaluation of the PIT Corpus Or What a Difference a Face Makes? , 2010, LREC.

[15]  Friedhelm Schwenker,et al.  Investigating fuzzy-input fuzzy-output support vector machines for robust voice quality classification , 2013, Comput. Speech Lang..

[16]  Heiko Neumann,et al.  Robust Stereoscopic Head Pose Estimation in Human-Computer Interaction and a Unified Evaluation Framework , 2011, ICIAP.