Alexander Sychov: All products 50% off

Ynnk Voice Lipsync

Yuri N Kalinin - Code Plugins - Jun 23, 2021

Create lip-sync animation from audio

  • Supported Platforms
  • Supported Engine Versions
    4.26 - 4.27, 5.0 - 5.4
  • Download Type
    Engine Plugin
    This product contains a code plugin, complete with pre-built binaries and all its source code that integrates with Unreal Engine, which can be installed to an engine version of your choice then enabled on a per-project basis.

This plugin uses voice recognition engine to generate lip-sync animation from SoundWave assets or PCM audio data. Animation is saved as curves in data assets and can be played in runtime together with audio. This approach allows to achieve well-looking lip-sync animation easily without subtitles.


Additional feature: recognize input from microphone (speech-to-text) in runtime.


Video #1

Video #2: Lip-sync Curves to Anim Seqiences

Video #3: Generate Pose Asset with visemes (CC3)

Video #4: MetaHuman Setup (tutorial)

Video #5: Pose asset for MetaHuman from default visemes


New tutorial for MetaHuman.

New tutorial for CC3/CC4.


Feb. 24 update note: New language setup and packaging pipeline


Unlike text-to-lipsync solution, this is true voice-to-lipsync plugin. You don't need subtitles to get lips animatied, and resulted animation is much closer to speech then in case of subtitles-based solution.


Lip-sync can be generated in runtime, but not in real-time. I. e. it doesn't work with microphone or other streamed audio.


Fully supported languages: English, Chinese. Also supported: Russian, Italian, German, French, Spanish, Portuguese, Polish.


Whisper add-on (to use Whisper instead of Vosk): YnnkWhisperRecognizer

Technical Details

Features:

  • subtitles aren't needed;
  • can generate lip-sync in runtime for loaded/TTS audio;
  • can generate Anim Sequence assets with lip-sync in curves;
  • lip-sync for animation curves (universal) or morph targets (when possible);
  • asynchronous audio recognition and building lip-sync;
  • (beta!) create lip-sync in runtime on PC and Android using remote server;
  • additional feature: recognize microphone input (Speech-to-text) in runtime (Windows only).

Code Modules:

  • YnnkVoiceLipsync (Runtime)
  • YnnkVoiceLipsyncUncooked (UncookedOnly)

Number of Blueprints: 0

Number of C++ Classes: 8

Network Replicated: No

Supported Development Platforms: Windows x64

Supported Target Build Platforms: Windows x64, Android

Documentation: [Doc]

Example Project: 5.3 | 5.2 | MetaHuman 5.1 with Enhancer Plugin

Executable Demo: [ZIP]

Enhancer Plugin: [5.1 - 5.3]