Recognizing Communicative Facial Expressions
for Discovering Interpersonal Emotions in Group Meetings

Abstract

This paper proposes a novel facial expression recognizer and describes its application to group meeting analysis. Our goal is to automatically discover the interpersonal emotions that evolve over time in meetings, e.g. how each person feels about the others, or who affectively influences the others the most. As the emotion cue, we focus on facial expression, more specifically smile, and aim to recognize "who is smiling at whom, when, and how often", since frequently smiling carries affective messages that are strongly directed to the person being looked at; this point of view is our novelty. To detect such communicative smiles, we propose a new algorithm that jointly estimates facial pose and expression in the framework of the particle filter. The main feature is its automatic selection of interest points that can robustly capture small changes in expression even in the presence of large head rotations. Based on the recognized facial expressions and their directions to others, which are indicated by the estimated head poses, we visualize interpersonal smile events as a graph structure, we call it the interpersonal emotional network; it is intended to indicate the emotional relationships among meeting participants. A four-person meeting captured by an omnidirectional video system is used to confirm the effectiveness of the proposed method and the potential of our approach for deep understanding of human relationships developed through communications.

References

Conferences

[1] S. Kumano, K. Otsuka, D. Mikami and Y. Junji, "Recognizing Communicative Facial Expressions for Discovering Interpersonal Emotions in Group Meetings", International Conference on Multimodal Interfaces (ICMI), Sept. 2009. pdf, poster