School of Computer Science › Language Technologies Institute › Research › research-articles › CAREER: Learning Nonverbal Signatures

Stock photo of a person with long dark hair, visible only from the neck down, sitting in a chair and holding an open book with one hand and gesturing toward another an unseen person with the other.

CAREER: Learning Nonverbal Signatures

By LP Morency

Nonverbal communication is an essential component of our daily life. A quick eyebrow frown from a student will inform the teacher of possible confusion. A tilt of the head may show doubt during negotiation. A patient's smile dynamics can even predict their future recovery. This project will build innovative technologies to allow computers to understand subtle nonverbal behaviors of human users. The main novelty of this project will be in its capacity to learn the inherent variability between individuals on how nonverbal behaviors are expressed. The same emotion can be expressed very differently by different people. A nonverbal signature is how a specific person gestures visually when talking with other people. By recognizing these nonverbal communicative behaviors such as facial expressions, head gestures and eye contact, computers and mobile devices will be able to better understand the user's emotions and affective states. In mental health treatment, the developed technologies have the potential to change how doctors assess disorders with new behavioral objective measures and possibility to predict eventual relapse for the patient. For researchers and animators of virtual characters (e.g., movie studies), the nonverbal signatures will enable an "a la carte" selection of the virtual human nonverbal signature that best aligns with the character's profile and personality. In education and online learning, the student's nonverbal signature will help to assess engagement and perceive affective states related to success in learning.

This project will advocate a novel paradigm for learning visual representations of human nonverbal behaviors: a major focus on intra-personal variability followed by analysis of group structures and eventual learning of visual representations generalizable across the population. The research methodology will be motivated by well-studied concepts of idiosyncrasy in nonverbal behavior expressions. The project will introduce the concept of nonverbal signatures which are low-dimensional computational representations of an individual's nonverbal behaviors contextualized by the verbal content and affective context. This project will address four fundamental research challenges: (1) personalized nonverbal embedding -- learning computational representations that summarize the individual variations in nonverbal appearance, (2) learning nonverbal signatures -- contextualizing the nonverbal behaviors with verbal and affective cues to learn more effective representations, (3) signature portfolio analysis -- discovering structure, prototypes and idiosyncrasy from a collection of nonverbal signatures, and (4) generalized nonverbal representations -- learning generalizable nonverbal representations able to adapt to new individuals. These four research aims will be complemented by a comprehensive evaluation plan to include four intermediate evaluations and a continuous overarching evaluation. This research effort will open the door to new sources of human-centric data where accurate interpretation of nonverbal behaviors is essential.