“Have you got our keys handy?”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 14.7 dB · + RGI 17.0 dB (+2.3 dB)
Abstract
Diffusion-based audio-visual speech enhancement (AVSE) iteratively recovers speech from noisy audio conditioned on temporally aligned visual cues. Its real-world deployment may confront distribution shifts, such as visual degradation and audio-visual mismatch, which misguide the reverse diffusion process. Our analysis discloses that visual quality alone does not necessarily determine whether visual conditioning remains useful or misguides enhancement since its effect is coupled with both the pretrained model and the acoustic input. Motivated by these observations, we propose Reliability-Guided Inference (RGI), which takes the departure of both internal visual representations and model responses from those of matched audio-video pairs as evidence of distribution shift inside the pretrained model and converts this evidence into a reliability score that regulates visual conditioning during reverse diffusion. Specifically, at each reverse step, RGI weights the difference between audio-visual and audio-only predictions according to the reliability score, without updating the pretrained models. To retain visual cues that can remain useful at low input signal-to-noise ratio, RGI imposes a lower bound on the conditioning coefficient, the conditioning floor. A diffusion-step schedule further modulates the retained visual contribution along the sampling trajectory. Experiments across four corpora and three frozen diffusion AVSE backbones show that RGI improves enhancement under a range of distribution shifts on the corpora with clean references, with gains that depend on the backbone and corpus. Recordings collected from the Ameca humanoid robot further show that RGI can be deployed for robot interaction.
Backbone = frozen audio-visual diffusion model (AV-UDiffSE) · RGI = Reliability-Guided Inference (ours) · SNR = signal-to-noise ratio · SI-SDR = scale-invariant signal-to-distortion ratio
RGI across distribution shifts
Occlusion
“Reading in poor light gives you eyestrain.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 12.4 dB · + RGI 16.2 dB (+3.8 dB)
“bin green by e four again”
InputAMECA
SI-SDR: noisy 4.9 dB · Backbone 13.2 dB · + RGI 13.6 dB (+0.4 dB)
Blackout
“Draw each graph on a new axis.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 15.3 dB · + RGI 17.7 dB (+2.5 dB)
“Norwegian sweaters are made of lamb's wool.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 15.5 dB · + RGI 16.0 dB (+0.5 dB)
“lay blue with q five again”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 12.7 dB · + RGI 12.8 dB (+0.0 dB)
Wrong speaker
“Ralph controlled the stopwatch from the bleachers.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 17.7 dB · + RGI 18.1 dB (+0.4 dB)
“Do you hear the sleigh bells ringing?”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 16.5 dB · + RGI 17.2 dB (+0.7 dB)
“place white in s seven now”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 8.6 dB · + RGI 13.7 dB (+5.0 dB)
Frozen frames
“Have you got our keys handy?”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 16.7 dB · + RGI 17.2 dB (+0.6 dB)
“The small boy put the worm on the hook.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 17.5 dB · + RGI 17.7 dB (+0.1 dB)
“lay white in d two again”
InputAMECA
SI-SDR: noisy 5.2 dB · Backbone 11.1 dB · + RGI 11.2 dB (+0.1 dB)
Audio-video offset
“Have you got our keys handy?”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 16.6 dB · + RGI 17.2 dB (+0.6 dB)
“The small boy put the worm on the hook.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 17.6 dB · + RGI 17.7 dB (+0.1 dB)
“bin green by e four again”
InputAMECA
SI-SDR: noisy 4.9 dB · Backbone 13.3 dB · + RGI 13.7 dB (+0.3 dB)
Low resolution
“Norwegian sweaters are made of lamb's wool.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 16.5 dB · + RGI 16.6 dB (+0.1 dB)
“The small boy put the worm on the hook.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 17.4 dB · + RGI 17.6 dB (+0.1 dB)
“lay blue with q five again”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 13.1 dB · + RGI 13.0 dB (−0.2 dB)
Original video
“Draw each graph on a new axis.”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 18.0 dB · + RGI 18.3 dB (+0.3 dB)
“Have you got our keys handy?”
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 16.6 dB · + RGI 17.2 dB (+0.6 dB)
“bin green by e four again”
InputAMECA
SI-SDR: noisy 4.9 dB · Backbone 13.3 dB · + RGI 13.6 dB (+0.3 dB)
Occlusion
“I just saw Jim near the new archeological museum.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.2 dB · Backbone 0.1 dB · + RGI 4.9 dB (+4.8 dB)
“He stole a dime from a beggar.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 12.8 dB · + RGI 16.6 dB (+3.8 dB)
“Do they allow atheists in church?”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 9.4 dB · + RGI 13.9 dB (+4.5 dB)
“Are your grades higher or lower than Nancy's?”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 7.2 dB · + RGI 10.8 dB (+3.6 dB)
“Should giraffes be kept in small zoos?”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 11.1 dB · + RGI 13.5 dB (+2.4 dB)
“From it, spokes of order and degree led to the outward rim of the common man.”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 6.8 dB · + RGI 9.9 dB (+3.1 dB)
“Should giraffes be kept in small zoos?”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 7.4 dB · + RGI 9.9 dB (+2.5 dB)
“The overweight charmer could slip poison into anyone's tea.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 11.0 dB · + RGI 13.8 dB (+2.8 dB)
“Why else would Danny allow others to go?”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 7.1 dB · + RGI 9.3 dB (+2.2 dB)
“Try to recall the events in chronological order.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 6.3 dB · + RGI 8.7 dB (+2.4 dB)
“The small boy put the worm on the hook.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 7.9 dB · + RGI 6.5 dB (−1.4 dB)
“I just saw Jim near the new archeological museum.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 2.6 dB · + RGI 4.5 dB (+1.9 dB)
“I know I didn't meet her early enough.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 5.8 dB · + RGI 2.3 dB (−3.5 dB)
“Why else would Danny allow others to go?”
InputAMECA
InputAMECA
SI-SDR: noisy −5.1 dB · Backbone 4.8 dB · + RGI 6.7 dB (+1.9 dB)
“Would you allow acts of violence?”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 4.3 dB · + RGI 1.4 dB (−2.9 dB)
Blackout
“Our campus, unfortunately, owns no films.”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 10.5 dB · + RGI 13.8 dB (+3.4 dB)
“Should giraffes be kept in small zoos?”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 8.2 dB · + RGI 13.1 dB (+4.9 dB)
“All that time rifle barrels were pointing unwaveringly at his head and body.”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 7.5 dB · + RGI 12.2 dB (+4.6 dB)
“All that time rifle barrels were pointing unwaveringly at his head and body.”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 7.3 dB · + RGI 12.2 dB (+4.9 dB)
“Should giraffes be kept in small zoos?”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 4.3 dB · + RGI 9.6 dB (+5.3 dB)
“All that time rifle barrels were pointing unwaveringly at his head and body.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 7.2 dB · + RGI 10.4 dB (+3.2 dB)
“The easygoing zoologist relaxed throughout the voyage.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 9.1 dB · + RGI 12.2 dB (+3.1 dB)
“At twilight on the twelfth day we'll have Chablis.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 6.9 dB · + RGI 9.6 dB (+2.7 dB)
Wrong speaker
“At twilight on the twelfth day we'll have Chablis.”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 8.4 dB · + RGI 13.3 dB (+4.9 dB)
“At twilight on the twelfth day we'll have Chablis.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 4.4 dB · + RGI 9.2 dB (+4.8 dB)
“The emblem depicts the Acropolis all aglow.”
InputAMECA
InputAMECA
SI-SDR: noisy 5.0 dB · Backbone 7.9 dB · + RGI 12.7 dB (+4.8 dB)
“Try to recall the events in chronological order.”
InputAMECA
InputAMECA
SI-SDR: noisy −5.0 dB · Backbone 5.1 dB · + RGI 7.0 dB (+1.8 dB)