Towards Automatic Recognition of Spontaneous Facial Actions

Charles Darwin (1872/1998) was the first to fully recognize that facial expression is one of the most powerful and immediate means for human beings to communicate their emotions, intentions, and opinions to each other. In addition to providing information about affective state, facial expressions also provide information about cognitive state, such as interest, boredom, confusion, and stress, and conversational signals with information about speech emphasis and syntax. Facial expressions also contain information about whether an expression of emotion is posed or felt (Ekman, 2001; Frank, Ekman, & Friesen, 1993). In order to objectively measure the richness and complexity of facial expressions, behavioral scientists have found it necessary to develop objective coding standards. The Facial Action Coding System (FACS) from Ekman and Friesen (1978) is arguably the most comprehensive and influential of such standards. FACS is based on the anatomy of the human face, and codes expressions in terms of component movements, called " action units " (AUs). Ekman and Friesen defined 46 AUs to describe each independent movement of the face. FACS measures all visible facial muscle movements, including head and eye movements, and not just those presumed to be related to emotion. When learning FACS, a coder is trained to identify the characteristic pattern of bulges, wrinkles, and movements for each facial AU. The AUs approximate individual facial muscle movements but there is not always a 1:1 correspondence. FACS has been used to verify the physiological presence of emotion in a number of studies, with high (over 75%) agreement Because it is comprehensive, FACS also allows for the discovery of new patterns related to emotional or situational states. For example, using FACS Ekman et al (1990) and Davidson et al (1990) found that smiles which featured both orbicularis oculi (AU6), as well as zygomatic major action (AU12), were correlated with self-reports of enjoyment, as well as different patterns of brain activity, whereas smiles that featured only zygomatic major (AU12) were not. Subsequent research demonstrated that the presence of smiles that involve the orbicularis oculi (hereafter " enjoyment smiles ") on the part of a person who has survived the death of their romantic partner predicts successful coping with that traumatic loss (Bonnano & Keltner, 1997). Other work has shown a similar pattern. For example, infants show enjoyment smiles to the presence of their mothers, but not to strangers (Fox & Davidson, 1988). Mothers do not show as many enjoyment …

[1]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[2]  P. Ekman,et al.  Behavioral Markers and Recognizability of the Smile of Enjoyment , 1993 .

[3]  Matthew Brand,et al.  Flexible flow for 3D nonrigid tracking and shape recovery , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[4]  Paul A. Viola,et al.  Robust Real-time Object Detection , 2001 .

[5]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[6]  C. Darwin The Expression of the Emotions in Man and Animals , .

[7]  P. Ekman,et al.  The Duchenne smile: emotional expression and brain physiology. II. , 1990, Journal of personality and social psychology.

[8]  Daphne Blunt Bugental,et al.  Unmasking the "Polite Smile" , 1986 .

[9]  Penio S. Penev,et al.  Local feature analysis: A general statistical theory for object representation , 1996 .

[10]  Fabio Lavagetto,et al.  MPEG-4:Audio/Video and Synthetic Graphics/Audio for Real-Time , 1997 .

[11]  Gwen Littlewort,et al.  Towards Social Robots: Automatic Evaluation of Human-robot Interaction by Face Detection and Expression Classification , 2003, NIPS.

[12]  P. Ekman Telling lies: clues to deceit in the marketplace , 1985 .

[13]  P. Ekman,et al.  The ability to detect deceit generalizes across different types of high-stake lies. , 1997, Journal of personality and social psychology.

[14]  P. Ekman,et al.  Linkages between facial expressions of anger and transient myocardial ischemia in men with coronary artery disease. , 2001, Emotion.

[15]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[16]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[17]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  John G. Daugman,et al.  Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression , 1988, IEEE Trans. Acoust. Speech Signal Process..

[19]  M. Sayette,et al.  The effect of alcohol on emotional response to a social stressor. , 1992, Journal of studies on alcohol.

[20]  Thomas S. Huang,et al.  Final Report To NSF of the Planning Workshop on Facial Expression Understanding , 1992 .

[21]  Javier R. Movellan,et al.  Visual Speech Recognition with Stochastic Networks , 1994, NIPS.

[22]  Marian Stewart Bartlett,et al.  Automatic Analysis of Spontaneous Facial Behavior: A Final Project Report , 2001 .

[23]  Maja Pantic,et al.  Expert system for automatic analysis of facial expressions , 2000, Image Vis. Comput..

[24]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[25]  J. P. Foley,et al.  Gesture and Environment , 1942 .

[26]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[27]  Marian Stewart Bartlett,et al.  Classifying Facial Action , 1995, NIPS.

[28]  J. Movellan,et al.  Are Your Eyes Smiling? Detecting genuine smiles with support vector machines and Gabor wavelets , 2001 .

[29]  H. Berenbaum,et al.  Emotional experience and expression in schizophrenia and depression. , 1992, Journal of abnormal psychology.

[30]  Michael J. Lyons,et al.  Classifying facial attributes using a 2-D Gabor wavelet representation and discriminant analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[31]  L. Rothkrantz,et al.  Toward an affect-sensitive multimodal human-computer interaction , 2003, Proc. IEEE.

[32]  P. Ekman,et al.  Voluntary facial action generates emotion-specific autonomic nervous system activity. , 1990, Psychophysiology.

[33]  Joachim M. Buhmann,et al.  Distortion Invariant Object Recognition in the Dynamic Link Architecture , 1993, IEEE Trans. Computers.

[34]  T. Sejnowski,et al.  Measuring facial expressions by computer image analysis. , 1999, Psychophysiology.

[35]  P. Ekman,et al.  Facial Expression in Affective Disorders , 2005 .

[36]  T. Moffitt,et al.  Facial expressions of emotion and psychopathology in adolescent boys. , 1995, Journal of abnormal psychology.

[37]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[38]  P. Ekman,et al.  Facial expression and the affective component of cynical hostility in male coronary heart disease patients. , 1998, Health psychology : official journal of the Division of Health Psychology, American Psychological Association.

[39]  G. Bonanno,et al.  Facial expressions of emotion and the course of conjugal bereavement. , 1997, Journal of abnormal psychology.

[40]  M. Katsikitis,et al.  A Controlled Quantitative Study of Facial Expression in Parkinson's Disease and Depression , 1991, The Journal of nervous and mental disease.

[41]  Gwen Littlewort,et al.  A Prototype for Automatic Recognition of Spontaneous Facial Actions , 2002, NIPS.

[42]  C N Karson Physiology of normal and abnormal blinking. , 1988, Advances in neurology.

[43]  Roberto Brunelli,et al.  Face Recognition: Features Versus Templates , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  N. Smelser,et al.  International Encyclopedia of the Social and Behavioral Sciences , 2001 .

[45]  N. Fox,et al.  Patterns of brain electrical activity during facial signs of emotion in 10-month-old infants. , 1988 .

[46]  P. Ekman,et al.  Facial signs of emotional experience. , 1980 .

[47]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[48]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[49]  J. Gottman,et al.  Why Marriages Succeed or Fail , 1995 .

[50]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[51]  Maja Pantic,et al.  Automatic Analysis of Facial Expressions: The State of the Art , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  P. Ekman,et al.  Felt, false, and miserable smiles , 1982 .

[53]  B. Braathen,et al.  First steps towards automatic recognition of spontaneous facial action units , 2001, PUI '01.

[54]  T. Jung,et al.  Combined eye activity measures accurately estimate changes in sustained visual task performance , 2000, Biological Psychology.

[55]  G. Kitagawa Monte Carlo Filter and Smoother for Non-Gaussian Nonlinear State Space Models , 1996 .

[56]  A. Inkeles,et al.  International Encyclopedia of the Social Sciences. , 1968 .

[57]  M. Holland,et al.  Blinking and Mental Load , 1972, Psychological reports.

[58]  Terrence J. Sejnowski,et al.  The “independent components” of natural scenes are edge filters , 1997, Vision Research.

[59]  Marian Stewart Bartlett,et al.  Face image analysis by unsupervised learning , 2001 .

[60]  Tetsuo Ono,et al.  Robovie: an interactive humanoid robot , 2001 .

[61]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[62]  R. Krause,et al.  Facial expression of schizophrenic patients and their interaction partners. , 1989, Psychiatry.

[63]  A. Kring,et al.  Do schizophrenic patients show a disjunctive relationship among expressive, experiential, and psychophysiological components of emotion? , 1996, Journal of abnormal psychology.

[64]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[65]  Marian Stewart Bartlett,et al.  Image Representations for Facial Expression Coding , 1999, NIPS.

[66]  Karson Cn Physiology of normal and abnormal blinking. , 1988 .

[67]  H. Ellgring,et al.  Nonverbal expression of psychological states in psychiatric patients , 2004, European archives of psychiatry and neurological sciences.

[68]  David Salesin,et al.  Synthesizing realistic facial expressions from photographs , 1998, SIGGRAPH.

[69]  W. Rinn,et al.  The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. , 1984, Psychological bulletin.

[70]  Marian Stewart Bartlett,et al.  A comparison of Gabor filter methods for automatic detection of facial landmarks , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[71]  P. Ekman Pictures of Facial Affect , 1976 .

[72]  Takeo Kanade,et al.  A Comparative Study of Alternative FACS Coding Algorithms , 2001 .

[73]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[74]  P. Thoits,et al.  Self-Labeling Processes in Mental Illness: The Role of Emotional Deviance , 1985, American Journal of Sociology.

[75]  P. Ekman,et al.  Smiles when lying. , 1988, Journal of personality and social psychology.