Advancing Face-to-Face Emotion Communication: A Multimodal Dataset (AFFEC)

Emotion recognition plays a pivotal role in advancing human-computer interaction by enabling systems to interpret and adapt to human affect. Yet, capturing emotions in face-to-face contexts remains challenging due to subtle nonverbal cues, stable individual differences, and the dynamic unfolding of interactions. Existing datasets often rely on limited modalities, actor-driven expressions, or highly controlled conditions, thereby missing the ecological richness and theoretical nuance required for adaptive models. In this work, we introduce Advancing Face-to-Face Emotion Communication (AFFEC), a multimodal dataset designed to address these gaps. AFFEC comprises 84 context-primed emotional dialogues across six emotions, recorded from 72 participants over 5,000+ trials and annotated with more than 20,000 labels. It integrates electroencephalography (EEG), eye tracking, galvanic skin response (GSR), facial videos, and Big Five personality assessments. Crucially, AFFEC explicitly differentiates between felt emotions (participants' internal affect) and perceived emotions (their interpretation of the stimulus), bridging Basic Emotion Theory's focus on universal signals with the Theory of Constructed Emotion's emphasis on subjective experience and context. Baseline analyses spanning unimodal features and simple multimodal fusion confirm that AFFEC contains robust and complementary signals: performance consistently surpasses the 0.33 chance level across all modalities and targets, reaching macro-F1 of 0.50 for felt arousal (EEG CNN) and 0.46 for perceived valence with Eye+AU+GSR+Personality fusion. With conservative RandomForest baselines on peripheral signals, perceived valence is more tractable than arousal for most modalities, while EEG shows the expected arousal advantage; incorporating personality traits yields consistent gains, most notably for valence prediction. These results, while intended as reference baselines rather than optimized benchmarks, validate the dataset's richness, and key limitations, including sample homogeneity, acted stimuli, and conservative preprocessing, are explicitly discussed. The participant sample (N = 72, 72% male, no racial or ethnic background recorded) constrains cross-demographic and cross-cultural generalisation, and this is discussed together with mitigation strategies. By combining theoretical grounding with methodological breadth, AFFEC offers a reproducible and extensible resource for affective computing. It provides a foundation for developing context-sensitive, adaptive, and empathetic emotion-aware technologies across domains such as human-agent interaction, social robotics, and digital health.