Facial expression and behavior modeling for digital avatars and expressive humanoid robots

Expressive humanoid robots need facial expressions and behaviors that are controllable, robust and real-time on mechanical actuators, while digital avatars already offer rich, flexible facial semantics. The review by Zhang and Cao (ACM Computing Surveys, 2026) analyses this virtual-to-physical gap from a deployment-oriented perspective.

It organizes the field along modeling paradigms (parametric, generative, volumetric and neural implicit), expression control strategies (direct mapping and retargeting, abstract generation, learning-based adaptation) and deployment dimensions (parameter-to-actuator mapping, calibration, real-time operation), and reviews datasets, metrics and benchmarks. Related work on this site also runs on the Ameca humanoid robot: speech emotion recognition (PhysioSER) and emotion-controllable speech synthesis (DUET).

Papers compared

Facial Expression and Behavior Modeling from Digital Avatars to Expressive Humanoid Robots: A Review

Xu Zhang, Longbing Cao · ACM Computing Surveys (2026)

This ACM Computing Surveys review covers facial expression and behavior modeling for digital avatars and physical humanoid robots. It analyses the virtual-to-physical gap along three axes: modeling paradigms, expression control strategies, and deployment (parameter-to-actuator mapping, calibration, real-time operation), and reviews the datasets, metrics and benchmarks for expressive humanoids.

• Only the parametric paradigm (3DMMs, blendshapes) and its intersection with the generative paradigm have been demonstrated on a physical robot face; volumetric and neural implicit models (NeRF, 3D Gaussian splatting, SDF) remain confined to the screen because they lack actuator-aligned control.
• Expression control strategies are grouped into direct mapping and retargeting, abstract generation from semantic or multimodal inputs (emotion labels, text, audio), and learning-based optimization and adaptation (reinforcement learning, imitation, user feedback).
• Deployment on robotic hardware is analysed through parameter-to-actuator mapping and degree-of-freedom (DoF) adaptation, real-time onboard computation, calibration and mechanical realism, and evaluation on hardware.
• Most public facial expression datasets are built for modeling and analysis rather than robot control; X2C is highlighted for providing 100,000 image-control pairs with actuator-level annotations from the Ameca humanoid robot.

Key concepts

Expressive humanoid robot

A human-like robot whose face uses mechanical actuators under a skin to display facial expressions, such as Ameca or Sophia. Papers: Expressive humanoid faces survey

Virtual-to-physical gap

The difference between what digital avatars can express on screen and what a physical robot face can realise under actuator, calibration and real-time constraints. Papers: Expressive humanoid faces survey

3D Morphable Model and blendshapes

Parametric face models that describe identity and expression with a small set of interpretable coefficients. Papers: Expressive humanoid faces survey

Facial Action Coding System (FACS) and action units

A standard description of facial movements as action units, often used to map expressions to robot actuators. Papers: Expressive humanoid faces survey

Expression retargeting

Transferring a facial expression from a person or avatar to a different face, here to a robot's actuators. Papers: Expressive humanoid faces survey

NeRF and 3D Gaussian splatting

Volumetric and neural implicit face representations with photorealistic rendering but no actuator-aligned control handles. Papers: Expressive humanoid faces survey

Frequently asked questions

Is there a survey of facial expression modeling for humanoid robots?

Yes. "Facial Expression and Behavior Modeling from Digital Avatars to Expressive Humanoid Robots: A Review" (Zhang and Cao, ACM Computing Surveys, 2026, DOI 10.1145/3856804) reviews modeling paradigms, expression control strategies, deployment on robotic hardware, and evaluation frameworks.

Can NeRF or 3D Gaussian splatting drive a physical robot face?

The review reports that no such system has yet been described: volumetric and neural implicit models lack actuator-aligned control handles, so they remain screen-based, while parametric models (3DMMs, blendshapes) and their combination with generative models have been demonstrated on physical robot faces.

Which datasets are used for robot facial expression modeling?

The review covers CK+, JAFFE, AffectNet, RAF-DB, BP4D, DISFA, D3DFACS, VOCASET, X2C, ExFace, REFLEX and HRI-SENSE; X2C provides 100,000 image-control pairs with actuator-level annotations from the Ameca robot.

Which robots are discussed?

Platforms discussed include Ameca, Sophia, EMYS, Nikola, Alter3, Eva, Emo, NAO, Furhat, Zeno, Baxter, iCub, Nico and the Einstein robot.

Fields, tasks, datasets and related search terms

Fields (broad to narrow)

artificial intelligence robotics computer vision computer graphics human-computer interaction human-robot interaction social robotics humanoid robots affective robotics robot facial expression facial expression and behavior modeling for humanoid robots

Task

robot facial expression generation facial expression synthesis facial behavior modeling facial animation expression retargeting facial motion retargeting to robots avatar-to-robot expression transfer facial expression imitation talking-head generation speech-driven facial animation facial expression recognition (FER) action unit (AU) detection 3D face reconstruction face reenactment emotion-driven expression generation actuator control for robot faces real-time robot expression control

Method

3D Morphable Models (3DMM) blendshapes FLAME Basel Face Model (BFM) LSFM DECA EMOCA Facial Action Coding System (FACS) action units facial landmarks GANs StyleGAN diffusion models DiffusionRig DiffusionAvatars NeRF HeadNeRF RigNeRF 3D Gaussian splatting (3DGS) GaussianAvatars HeadStudio signed distance functions (SDF) neural implicit representations direct mapping AU-to-actuator mapping landmark-to-servo mapping inverse kinematics for faces motion capture (MoCap) reinforcement learning imitation learning RLHF self-supervised robot learning large language models for robot expression multimodal generation (text, audio, emotion labels) sim-to-real transfer calibration degrees of freedom (DoF) adaptation

Datasets

CK+ JAFFE AffectNet RAF-DB BP4D DISFA D3DFACS VOCASET X2C ExFace REFLEX HRI-SENSE FER2013

Robots and platforms

Ameca Sophia EMYS Nikola Alter3 Eva Emo NAO Furhat Zeno Baxter iCub Nico Einstein robot

Toolkits and metrics

OpenFace MediaPipe ARKit VICON facial landmark distance (FLD) MAE RMSE FID LPIPS PSNR SSIM average expression distance (AED) average pose distance (APD) average keypoint distance (AKD) latency FPS concordance correlation coefficient (CCC) user studies

Related areas

affective computing embodied AI digital humans virtual avatars android robots animatronics soft robotics silicone robot skin uncanny valley nonverbal communication gaze and head motion co-speech gesture emotion recognition speech emotion recognition emotional speech synthesis multimodal interaction AI ethics trustworthy robotics

Applications

healthcare robots therapy and autism support eldercare companions education robots customer service robots receptionist robots entertainment robots telepresence

中文

人形机器人 仿人机器人 表情机器人 机器人面部表情 面部表情建模 面部行为建模 表情生成 表情迁移 表情重定向 数字人 数字化身 虚拟形象 面部动画 说话人脸生成 人脸重建 三维可变形模型 混合形状 面部动作编码系统 动作单元 神经辐射场 三维高斯泼溅 扩散模型 生成对抗网络 执行器映射 舵机控制 实时控制 标定 强化学习 模仿学习 人机交互 社交机器人 情感计算 具身智能 恐怖谷 综述

Cite these papers

@article{zhang2026expressive,
  title        = {Facial Expression and Behavior Modeling from Digital Avatars to Expressive Humanoid Robots: A Review},
  author       = {Zhang, Xu and Cao, Longbing},
  journal      = {ACM Computing Surveys},
  year         = {2026},
  doi          = {10.1145/3856804},
  issn         = {0360-0300},
  url          = {https://doi.org/10.1145/3856804}
}