Meta shared a clip displaying the brand new Audio To Expression function launched in Quest v71 in motion.
Audio To Expression is an on-device AI mannequin that generates believable facial muscle actions from solely microphone audio enter, offering estimated facial expressions with none face monitoring {hardware}.
The sooner Oculus Lipsync SDK from 2015 supplied this functionality for lip motion solely, and Meta’s Audio To Expression is its official substitute, providing cheeks, eyelid, and eyebrow motion too. And remarkably, Meta claims Audio To Expression really makes use of much less CPU than Oculus Lipsync.
Audio To Expression helps Quest 2, Quest 3 and Quest 3S. It additionally technically helps Quest Professional, although that headset has face monitoring sensors that builders can leverage to signify the true facial expressions of the wearer, not simply an estimate.
Within the embedded clip, you may see the stark distinction between Audio To Expression and the previous Oculus Lipsync SDK, given the identical enter.
Meta Audio To Expression (left) vs the previous Oculus Lipsync SDK (proper), with the identical enter.
In addition to bettering the realism of avatars pushed by non-Professional Quest house owners in social VR and multiplayer video games, Audio To Expression can be used for NPC faces, which may very well be helpful for smaller studios and impartial builders that may’t afford to make use of facial seize know-how.
Meta’s personal Meta Avatars do not but help Audio To Expression (they nonetheless use the Oculus Lipsync SDK), although they do have simulated eye motion, whereby the developer tags every digital object within the scene by its stage of visible saliency, in addition to occasional blinking for added realism, so they are not simply restricted to lip motion.
Builders can discover the Audio To Expression documentation right here: Unity / Unreal / Customized Engines.
