Speech¶
The part that forms the mouth. Which source it follows, how the shapes run into each other, and the rules that keep speech readable when blending alone would smooth it away.
This is the driver most projects spend their time with, and the one the rest of the performance is layered over.
Drive Modes¶
Where the mouth's instructions come from. One character can only be in one mode at a time, and the mode is a setting on the Speech driver.
| Mode | What drives the mouth |
|---|---|
| Cue Drive | The cues from a bake. The most accurate, and the one to use for dialogue you have |
| Viseme Stream | Live recognition, from a microphone or from a clip with no bake behind it |
| Amplitude | The loudness of the audio, with no recognition at all |
| Disabled | Nothing. The mouth is left to whatever else is driving it |
Cue Drive is the one to build around. It knows which sound is happening and when, so everything downstream, the rules, the emphasis, the timing, has something exact to work from.
Amplitude is the cheapest and the least specific. It opens the mouth in proportion to how loud the audio is. On a distant character or a crowd it is entirely convincing, and it needs no bake, no model and almost no work at runtime.
Viseme Stream is for speech that does not exist ahead of time: a player's microphone, voice chat, a line generated while the game runs. Covered in Live Speech.
The Fallback Chain¶
A mode you asked for is not always a mode that can run. A clip might have no bake, or a character might be speaking audio that arrived a second ago.
Rather than stopping, FaceCue falls through: Cue Drive looks for cues, and finding none, drops to Amplitude so the mouth still moves. The mouth being roughly right is better than a mouth that does nothing while the audio plays.
A Silent Mouth Is Usually a Registry Problem
If a line plays and the mouth does nothing at all, the fallback did not happen either, which usually means the audio never reached FaceCue and not that the bake was missing. Start at When a Clip Will Not Load.
Coarticulation¶
Every sound is shaped by the sounds around it. Your mouth is already rounding for the vowel in boot while it is still making the b, and the b looks different because of it.
FaceCue reproduces that instead of stepping between poses. Shapes overlap and run into each other, and the mouth is almost never sitting on one clean shape. That is the difference between speech and a puppet cycling through positions.
You can see it directly in the Cue Clip Composer, where the rounding mark on a cue trails back over the cues in front of the rounded vowel.
Anticipation¶
The mouth runs slightly ahead of the voice.
Real speakers set a shape up before the sound arrives, because the tongue and lips have to be in place for the sound to be made at all. A mouth that moves exactly on the audio reads as slightly late, which is one of the things that makes lip sync feel mechanical without anyone being able to say why.
How far ahead it runs is tunable, and it is one of the few settings worth touching early if a character reads as lagging.
Delivery¶
Two stylistic choices about how a character speaks, and they are the ones that give two characters different mouths on the same line.
How far each shape runs into its neighbours in time. More overlap is a looser, more casual mouth. Less is more deliberate and more precisely articulated.
How strongly each shape holds its own form while blended. A shape that yields easily gives a softer read. One that holds its ground gives a crisper one.
Neither is a correctness setting. They are the difference between a mumbling character and a precise one, and they are the first place to go when a character sounds right and looks wrong.
The Rules Underneath¶
Blending alone would smooth speech into mush, so a layer of rules sits over the top.
Some sounds have to land. The lips must actually meet for p, b and m, and if they do not, the word stops reading no matter how good the rest is. Those closures are protected, and they take the mouth even when an expression is competing for it.
Some shapes ease back so they do not compete. An unstressed vowel in ordinary speech does not reach its full shape, it relaxes toward a neutral one. Forcing every vowel to its full form is what makes a mouth look like it is reciting.
These rules are tuned per language, because languages do not behave the same way. See Language Support.
Loudness and Emphasis¶
Two separate readings, both of which open the mouth.
How loud a sound was is the obvious one. How much it stood out from the line around it is not the same thing: a word can be quiet and still be the one being leant on, because it was longer, or higher, or landed after a pause.
Both feed the mouth, and the second one also drives the head and the brows. See Emphasis.
Organic Variation¶
The same phoneme twice in a row should not be identical twice in a row, because a real mouth never is.
FaceCue varies each one slightly, drawing from the character's own randomness so two copies of the same character do not vary in unison. It is a small effect and it is the sort of thing nobody notices until it is switched off.
After the Line Ends¶
A mouth does not go slack the instant audio stops.
FaceCue winds it down, easing back to rest over a short interval so the line ends on a settle. The clip ending is not the same event as the speaker finishing, and the face should read as the second.