Emotion recognition goes beyond facial expressions, study finds

A person’s facial expression provides crucial information for us to recognize their emotions. But there’s much more to this process than that. This is according to research conducted by Dr. Leda Berio and Professor Albert Newen from the Institute of Philosophy II at Ruhr University Bochum, Germany.

The team describes emotion recognition not as a separate module, but as part of a comprehensive process that helps us form a general impression of another person. This process of person impression formation also includes physical and cultural characteristics as well as background information. The paper was published on September 24, 2024 in the journal Philosophy and Phenomenological Research.

Understanding the situation affects how we recognize emotions

In the 1970s, the theory was put forward that the face is the window to our emotions. Researcher Paul Ekman described basic emotions such as fear, anger, disgust, joy and sadness using typical facial expressions, which were found to be similar across all cultures.

“However, in recent years it’s become increasingly obvious that there are many situations in life where a typical facial expression is not necessarily the key piece of information that guides our assessment of other people’s feelings,” points out Newen.

“People almost universally rate a typical facial expression of fear as anger when they have the background knowledge that the assessed person’s been turned away by a waiter even though they’d demonstrably reserved a table.”

In such a situation, people expect the person to be angry, and this expectation determines the perception of their emotion, even if their facial expression would typically be attributed to a different emotion.

“In addition, we can sometimes recognize emotions even without seeing the face; for example, the fear experienced by a person who’s being attacked by a snarling dog, even though we only see them from behind in a stance of freeze or fright,” illustrates Berio.

Recognizing an emotion is part of our overall impression of a person

Berio and Newen propose that recognizing emotions is a sub-process of our ability to form an overall impression of a person. In doing so, people are guided by certain characteristics of the other person. For example, physical appearance characteristics such as skin color, age and gender, cultural characteristics such as clothing and attractiveness, as well as situational characteristics such as facial expressions, gestures and posture.

Based on such characteristics, people tend to quickly assess others and immediately associate social status and even certain personality traits with them. These associations dictate how we perceive other people’s emotions.

“If we perceive a person as a woman and they show a negative emotion, we’re more likely to attribute the emotion to fear, whereas with a man it’s more likely to be read as anger,” as Berio points out.

Background information is included in the assessment

In addition to the perception of characteristics and initial associations, we also hold detailed person images that we use as background information for individuals in our social circle—family members, friends and colleagues.

“If a family member suffers from Parkinson’s, we learn to assess the typical facial expression of this person, which seems to indicate anger, as neutral, because we are aware that a rigid facial expression is part of the disease,” says Berio.

The background information also includes person models of typical occupational groups. “We hold stereotypical assumptions about the social roles and responsibilities of for example doctors, students and workmen,” says Newen. “We generally perceive doctors as less emotional, for example, which changes the way we assess their emotions.”

In other words, people make use of the wealth of characteristics and background knowledge to assess the emotion of another person. Only in rare cases do they read the emotion from a person’s facial expression alone.

“All this has implications for emotion recognition using artificial intelligence (AI): It will only be a reliable option when AI doesn’t rely solely on facial expressions, which is what most systems currently do,” says Newen.