Facial Expression and Behavior Modeling from Digital Avatars to Expressive Humanoid Robots: A Review
Xu Zhang, Longbing Cao
ACM Computing Surveys (2026)
Quick facts
| Authors | Xu Zhang, Longbing Cao |
|---|---|
| Published in | ACM Computing Surveys (2026) |
| DOI | 10.1145/3856804 |
| Article type | Survey (structured literature review) in ACM Computing Surveys, published online 15 October 2026 |
| Scope | Facial expression and behavior modeling for digital avatars and expressive humanoid robots, from a deployment-oriented perspective |
| Framework | Modeling paradigms → expression control strategies → deployment, assessed by evaluation frameworks; analysed along expressiveness, controllability and deployability |
| Modeling paradigms | Parametric (3DMMs, blendshapes, FLAME), generative (GANs, diffusion models) and volumetric / neural implicit (NeRF, 3D Gaussian splatting, SDF) |
| Control strategies | Direct mapping and retargeting; abstract generation and synthesis; learning-based optimization and adaptation |
| Deployment dimensions | Parameter-to-actuator mapping and DoF adaptation; real-time performance and onboard computation; calibration, physical constraints and mechanical realism; evaluation on hardware |
| Datasets reviewed | CK+, JAFFE, AffectNet, RAF-DB, BP4D, DISFA, D3DFACS, VOCASET, X2C, ExFace, REFLEX, HRI-SENSE |
| Metric families | Geometry and kinematic (e.g. facial landmark distance, MAE, servo RMSE, skin strain similarity); semantic and affective (AU and emotion classification accuracy, user ratings); temporal and runtime (latency, FPS, smoothness) |
| Robot platforms discussed | Ameca, Sophia, EMYS, Nikola, Alter3, Eva, Emo, NAO, Furhat, Zeno, Baxter, iCub, Nico, Einstein |
| Review protocol | Peer-reviewed works from 2020 to 2025 plus earlier seminal work; IEEE Xplore, ACM Digital Library, SpringerLink and Google Scholar; an initial pool of about 850 papers, about 70% removed at title/abstract screening, then full-text review and snowballing |
| Future directions | Multimodal pipelines (including large language models), adaptive personalization, humanity modeling, uncanny-valley and ethical safeguards, standardized evaluation |
| ACM CCS concepts | Robotics; Human computer interaction (HCI); Animation; Computer vision |
| BibTeX key | zhang2026expressive |
Key points
- Only the parametric paradigm (3DMMs, blendshapes) and its intersection with the generative paradigm have been demonstrated on a physical robot face; volumetric and neural implicit models (NeRF, 3D Gaussian splatting, SDF) remain confined to the screen because they lack actuator-aligned control.
- Expression control strategies are grouped into direct mapping and retargeting, abstract generation from semantic or multimodal inputs (emotion labels, text, audio), and learning-based optimization and adaptation (reinforcement learning, imitation, user feedback).
- Deployment on robotic hardware is analysed through parameter-to-actuator mapping and degree-of-freedom (DoF) adaptation, real-time onboard computation, calibration and mechanical realism, and evaluation on hardware.
- Most public facial expression datasets are built for modeling and analysis rather than robot control; X2C is highlighted for providing 100,000 image-control pairs with actuator-level annotations from the Ameca humanoid robot.
- Evaluation metrics are classified as geometry and kinematic, semantic and affective, and temporal and runtime; benchmarks are categorized as recognition-centric, synthesis-centric, retargeting-focused, and robot-centered, and the field still lacks standardized, platform-agnostic evaluation protocols.
Abstract
Endowing humanoid robots with expressive, controllable, and robust facial expressions and behaviors is essential for advancing human-like humanoid-human interaction across social, healthcare, and service domains. However, transforming the semantic richness and flexibility of virtual avatars into the mechanical, real-time, and calibration constraints of physical humanoids remains a fundamental challenge. This survey reviews facial expression and behavior modeling for both and analyzes their virtual-to-physical gaps from a deployment-oriented perspective. Our review is organized along three axes: (i) modeling paradigms, with emphasis on actuator compatibility and transferability; (ii) expression control strategies, assessed for their potential to enable robust robotic deployment; and (iii) deployment dimensions, which examine the challenges of parameter-to-actuator mapping, calibration, and real-time operation. This survey provides a critical synthesis of the trade-offs between expressive fidelity and hardware feasibility. We analyze persistent gaps between algorithmic advances and embodied realization, and review the datasets, metrics, and evaluation frameworks necessary for advancing expressive humanoids and their facial expression and behavior modeling. Finally, we identify open challenges and future research directions, including multimodal coordination, adaptive personalization, and standardized evaluation, providing guidance for next-generation expressive human-like humanoids, and systematically integrating advances in humanoid modeling with actionable deployment of humanoid facial expressions and behaviors.
Frequently asked questions
What does this survey cover?
It reviews facial expression and behavior modeling for both digital avatars and physical humanoid robots, and analyses the gap between them along three axes: modeling paradigms, expression control strategies, and deployment dimensions. It also reviews the datasets, metrics and evaluation frameworks for expressive humanoids.
What is the virtual-to-physical gap for robot faces?
Virtual avatars offer rich, flexible facial semantics, but physical humanoids must realise expressions through mechanical actuators under real-time and calibration constraints. The survey analyses how modeling and control methods from avatars can be translated into actionable actuator commands.
Which facial modeling paradigms have been used on physical robot faces?
According to the survey, only the parametric paradigm (3DMMs and blendshapes) and its intersection with the generative paradigm have been demonstrated on a physical robot face. Volumetric and neural implicit models such as NeRF and 3D Gaussian splatting remain screen-based.
Which datasets support robot facial expression modeling?
The survey reviews CK+, JAFFE, AffectNet, RAF-DB, BP4D, DISFA, D3DFACS, VOCASET, X2C, ExFace, REFLEX and HRI-SENSE, and notes that most are designed for modeling and analysis; X2C provides 100,000 image-control pairs with actuator-level annotations from the Ameca robot.
How should robotic facial expression systems be evaluated?
The survey groups metrics into geometry and kinematic, semantic and affective, and temporal and runtime families, and categorizes benchmarks as recognition-centric, synthesis-centric, retargeting-focused and robot-centered. It concludes that standardized, platform-agnostic protocols are still missing.
What open challenges does the survey identify?
Multimodal coordination, adaptive personalization and standardized evaluation, together with humanity modeling, avoiding the uncanny valley, and safeguards against manipulation and bias.
Is there a list of the papers covered by this survey?
Yes. The cited papers are organized by the survey's sections, with DOI or arXiv links, at https://github.com/CodeZx6/Awesome-Expressive-Humanoid-Faces.
How do I cite this paper?
Cite it as: Zhang, X., & Cao, L. (2026). Facial Expression and Behavior Modeling from Digital Avatars to Expressive Humanoid Robots: A Review. ACM Computing Surveys. https://doi.org/10.1145/3856804 BibTeX is available at https://codezx6.github.io/papers/expressive-humanoid-faces-survey.html#cite and in https://codezx6.github.io/publications.bib.
中文摘要
从数字化身到表情人形机器人的面部表情与行为建模综述
本文（ACM Computing Surveys 综述）面向数字化身与实体人形机器人，系统回顾面部表情与行为建模，并从部署角度分析虚拟到实体的差距。综述沿三条主线展开：建模范式（参数化、生成式、体渲染与神经隐式）、表情控制策略（直接映射与重定向、抽象生成、基于学习的优化），以及部署维度（参数到执行器映射、标定与实时运行）；同时梳理数据集、评测指标与基准，并提出多模态协同、自适应个性化与标准化评测等未来方向。
关键词：人形机器人；面部表情建模；面部行为建模；数字化身；表情控制；人机交互
Keywords
digital avatar humanoid robots facial expression modeling facial behavior modeling expression control real-time deployment benchmarking and evaluation human-robot interaction
Also referred to as: survey of facial expression modeling for humanoid robots; review of robotic facial expression and behavior modeling; from digital avatars to expressive humanoid robots; virtual-to-physical gap in robot facial expression; expressive robot face survey; humanoid facial expression survey.
Research area map and related search terms
Field path (broad to narrow): artificial intelligence › robotics › computer vision › computer graphics › human-computer interaction › human-robot interaction › social robotics › humanoid robots › affective robotics › robot facial expression › facial expression and behavior modeling for humanoid robots › this paper
Task
robot facial expression generation facial expression synthesis facial behavior modeling facial animation expression retargeting facial motion retargeting to robots avatar-to-robot expression transfer facial expression imitation talking-head generation speech-driven facial animation facial expression recognition (FER) action unit (AU) detection 3D face reconstruction face reenactment emotion-driven expression generation actuator control for robot faces real-time robot expression control
Method
3D Morphable Models (3DMM) blendshapes FLAME Basel Face Model (BFM) LSFM DECA EMOCA Facial Action Coding System (FACS) action units facial landmarks GANs StyleGAN diffusion models DiffusionRig DiffusionAvatars NeRF HeadNeRF RigNeRF 3D Gaussian splatting (3DGS) GaussianAvatars HeadStudio signed distance functions (SDF) neural implicit representations direct mapping AU-to-actuator mapping landmark-to-servo mapping inverse kinematics for faces motion capture (MoCap) reinforcement learning imitation learning RLHF self-supervised robot learning large language models for robot expression multimodal generation (text, audio, emotion labels) sim-to-real transfer calibration degrees of freedom (DoF) adaptation
Datasets
CK+ JAFFE AffectNet RAF-DB BP4D DISFA D3DFACS VOCASET X2C ExFace REFLEX HRI-SENSE FER2013
Robots and platforms
Ameca Sophia EMYS Nikola Alter3 Eva Emo NAO Furhat Zeno Baxter iCub Nico Einstein robot
Toolkits and metrics
OpenFace MediaPipe ARKit VICON facial landmark distance (FLD) MAE RMSE FID LPIPS PSNR SSIM average expression distance (AED) average pose distance (APD) average keypoint distance (AKD) latency FPS concordance correlation coefficient (CCC) user studies
Related areas
affective computing embodied AI digital humans virtual avatars android robots animatronics soft robotics silicone robot skin uncanny valley nonverbal communication gaze and head motion co-speech gesture emotion recognition speech emotion recognition emotional speech synthesis multimodal interaction AI ethics trustworthy robotics
Applications
healthcare robots therapy and autism support eldercare companions education robots customer service robots receptionist robots entertainment robots telepresence
中文
人形机器人 仿人机器人 表情机器人 机器人面部表情 面部表情建模 面部行为建模 表情生成 表情迁移 表情重定向 数字人 数字化身 虚拟形象 面部动画 说话人脸生成 人脸重建 三维可变形模型 混合形状 面部动作编码系统 动作单元 神经辐射场 三维高斯泼溅 扩散模型 生成对抗网络 执行器映射 舵机控制 实时控制 标定 强化学习 模仿学习 人机交互 社交机器人 情感计算 具身智能 恐怖谷 综述
Terms are grouped by role. "Related areas", "Applications" and "Related benchmarks (not used in this paper)" describe the surrounding field, not results of this paper. See the site-wide research area map.
Related papers
Other topics
- DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech (2026)
- PhysioSER: Learning Physiology-Informed Vocal Spectrotemporal Representations for Speech Emotion Recognition (2026)
- DSTCN: Exploiting dynamic spatio-temporal correlations for origin-destination demand prediction (2026)
- URMDet-SimFire: Trustworthy multimodal detection with uncertainty and reliability modeling for fire and smoke analytics (2026)
- S2CMEN: A Mutually Enhancement Network for Superpixel Segmentation and Classification of Hyperspectral Image (2025)
- MR-UFP: Enhancing urban flow prediction via mutual reinforcement with multi-scale regional information (2025)
- BiST-IF: Enhancing origin–destination flow prediction via bi-directional spatio-temporal inference and interconnected feature evolution (2025)
- Automatic visual recognition for leaf disease based on enhanced attention mechanism (2024)
- S3CFSL: Spatial-Spectral–Semantic Cross-Domain Few-Shot Learning for Hyperspectral Image Classification (2024)
- ST-FCL: Spatio-temporal fusion and contrastive learning for urban flow prediction (2023)
- MC-STL: Mask- and Contrast-Enhanced Spatio-Temporal Learning for Urban Flow Prediction (2023)
Cite this paper
@article{zhang2026expressive,
title = {Facial Expression and Behavior Modeling from Digital Avatars to Expressive Humanoid Robots: A Review},
author = {Zhang, Xu and Cao, Longbing},
journal = {ACM Computing Surveys},
year = {2026},
doi = {10.1145/3856804},
issn = {0360-0300},
url = {https://doi.org/10.1145/3856804}
}Zhang, X., & Cao, L. (2026). Facial Expression and Behavior Modeling from Digital Avatars to Expressive Humanoid Robots: A Review. ACM Computing Surveys. https://doi.org/10.1145/3856804