Measuring facial expressions by computer image analysis.

Facial expressions provide an important behavioral measure for the study of emotion, cognitive processes, and social interaction. The Facial Action Coding System (Ekman & Friesen, 1978) is an objective method for quantifying facial movement in terms of component actions. We applied computer image analysis to the problem of automatically detecting facial actions in sequences of images. Three approaches were compared: holistic spatial analysis, explicit measurement of features such as wrinkles, and estimation of motion flow fields. The three methods were combined in a hybrid system that classified six upper facial actions with 91% accuracy. The hybrid system outperformed human nonexperts on this task and performed as well as highly trained experts. An automated system would make facial expression measurement more widely accessible as a research tool in behavioral science and investigations of the neural substrates of emotion.

[1]  A. Bruce.  Emotional Expression , 1883, The American Naturalist.

[2]  L. Camras,et al.  Facial Expressions Used by Children in a Conflict Situation , 1977 .

[3]  L. Camras,et al.  Facial expressions used by children in a conflict situation. , 1977, Child development.

[4]  P. Ekman,et al.  Facial action coding system: a technique for the measurement of facial movement , 1978 .

[5]  J. N. Bassili.  Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. , 1979, Journal of personality and social psychology.

[6]  Takeo Kanade,et al.  Computer recognition of human faces , 1980 .

[7]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[8]  P. Ekman,et al.  Handbook of methods in nonverbal behavior research , 1982 .

[9]  P. Ekman.  Emotion in the human face , 1982 .

[10]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[11]  P. Ekman.  Expression and the Nature of Emotion , 1984 .

[12]  Edmund T. Rolls,et al.  The responsiveness of neurons in the insular gustatory cortex of the macaque monkey is independent of hunger , 1988, Physiology & Behavior.

[13]  A. Damasio,et al.  Intact recognition of facial expression, gender, and age in patients with impaired recognition of face identity , 1988, Neurology.

[14]  P. Ekman.  The argument and evidence about universals in facial expressions of emotion. , 1989 .

[15]  M. Hasselmo,et al.  The role of expression and identity in the face-selective responses of neurons in the temporal visual cortex of the monkey , 1989, Behavioural Brain Research.

[16]  P. Ekman,et al.  Approach-withdrawal and cerebral asymmetry: emotional expression and brain physiology. I. , 1990, Journal of personality and social psychology.

[17]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[18]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[19]  Kenji Mase,et al.  Recognition of Facial Expression from Optical Flow , 1991 .

[20]  H. Wallbott.  Effects of distortion of spatial and temporal resolution of video stimuli on emotion attributions , 1992 .

[21]  S. Kaiser,et al.  Automated coding of facial behavior in human-computer interactions with facs , 1992 .

[22]  Thomas S. Huang,et al.  Final Report To NSF of the Planning Workshop on Facial Expression Understanding , 1992 .

[23]  Roberto Brunelli,et al.  Face Recognition: Features Versus Templates , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Tomaso Poggio,et al.  Example Based Image Analysis and Synthesis , 1993 .

[25]  Pertti Roivainen,et al.  3-D Motion Estimation in Model-Based Facial Image Coding , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Demetri Terzopoulos,et al.  Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[28]  A. O'Toole,et al.  Structural aspects of face recognition and the other-race effect , 1994, Memory & cognition.

[29]  S. Kinomura,et al.  Functional anatomy of taste perception in the human brain studied with positron emission tomography , 1994, Brain Research.

[30]  Javier R. Movellan,et al.  Visual Speech Recognition with Stochastic Networks , 1994, NIPS.

[31]  David Beymer,et al.  Face recognition under varying pose , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Alice J. O'Toole,et al.  Connectionist models of face processing: A survey , 1994, Pattern Recognit..

[33]  R. Adolphs,et al.  Fear and the human amygdala , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[34]  Marian Stewart Bartlett,et al.  Classifying Facial Action , 1995, NIPS.

[35]  Rangachar Kasturi,et al.  Machine vision , 1995 .

[36]  R. Adolphs,et al.  Cortical Systems for the Recognition of Emotion in Facial Expressions , 1996, The Journal of Neuroscience.

[37]  V. Bruce,et al.  Face processing: Human perception and principal components analysis , 1996, Memory & cognition.

[38]  D. Perrett,et al.  A differential neural response in the human amygdala to fearful and happy facial expressions , 1996, Nature.

[39]  Larry S. Davis,et al.  Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  S. Rauch,et al.  Response and Habituation of the Human Amygdala during Visual Processing of Facial Expression , 1996, Neuron.

[41]  Larry S. Davis,et al.  Human expression recognition from motion using a radial basis function network architecture , 1996, IEEE Trans. Neural Networks.

[42]  Garrison W. Cottrell,et al.  Representing Face Images for Emotion Classification , 1996, NIPS.

[43]  D. Perrett,et al.  A specific neural substrate for perceiving facial expressions of disgust , 1997, Nature.

[44]  Tomaso A. Poggio,et al.  Linear Object Classes and Image Synthesis From a Single Example Image , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Timothy F. Cootes,et al.  Automatic Interpretation and Coding of Face Images Using Flexible Models , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Alex Pentland,et al.  Coding, Analysis, Interpretation, and Recognition of Facial Expressions , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[47]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[48]  Nirbhay N. Singh,et al.  Facial Expressions of Emotion , 1998 .

[49]  V. Bruce,et al.  Human Face Perception and Identification , 1998 .

[50]  J. Lien,et al.  Automatic recognition of facial expressions using hidden markov models and estimation of expression intensity , 1998 .

[51]  J. Cohn,et al.  Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. , 1999, Psychophysiology.

[52]  Vicki Bruce,et al.  Face Recognition: From Theory to Applications , 1999 .

[53]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  Antonio R. Damasio,et al.  The interaction of affect and cognition: A neurobiological perspective. , 2000 .

[55]  Curtis Padgett,et al.  Representing F Ace Images for Emotion Classiication , 2022 .