Skip to content

Lip Sync using Sonic

Video Lecture

Section Video Links
Lip Sync using Sonic Lip Sync using Sonic Lip Sync using Sonic

Description

We will use the Sonic custom node to add lip synced audio to a still image.

This will produce a short video the same length as the audio.

Install Custom Node

Install the custom node ComfyUI Sonic using the manager, or you can use your command/terminal prompt.

  1. Navigate to your ComfyUI/custom_nodes folder.
  2. Run, 
    git clone https://github.com/smthemex/ComfyUI_Sonic.git
  3. Restart ComfyUI

Install Models

We need to create this folder structure and download each model into its place.

--  ComfyUI/models/sonic/
    |-- audio2bucket.pth
    |-- audio2token.pth
    |-- unet.pth
    |-- yoloface_v5m.pt
    |-- whisper-tiny/
        |--config.json
        |--model.safetensors
        |--preprocessor_config.json
    |-- RIFE/
        |--flownet.pkl

Download audio2bucket.pth,audio2token.pth, unet.pth, yoloface_v5m.pt and flownet.pkl from https://drive.google.com/drive/folders/1oe8VTPUy0-MHHW2a_NJ1F8xL-0VN5G7W

Download config.json, model.safetensors and preprocessor_config.json from https://huggingface.co/openai/whisper-tiny/tree/main

Sample Workflows

Download this Example Audio and save into you ComfyUI/input folder.

Initial Image Workflow

ComfyUI Sonic (GitHub)