Skip to content

Lip Sync using Sonic

Video Lecture

Section Video Links
Lip Sync using Sonic Lip Sync using Sonic

Note

The install information below contains newer information than what is shown in the video. Be sure to follow the instructions below when installing.

Description

We will use the Sonic custom node to add lip synced audio to a still image.

This will produce a short video the same length as the audio.

Install ComfyUI Sonic Custom Node

Install the custom node ComfyUI Sonic using the manager,

ComfyUI Manager Sonic

Or, you can use your command/terminal prompt.

  1. Navigate to your ./ComfyUI_windows_portable/ComfyUI/custom_nodes folder.
  2. Run, 
    git clone https://github.com/Sean-Bradley/ComfyUI-Sonic
  3. Navigate to your ./ComfyUI_windows_portable folder.
  4. Run 
    python_embeded\python -m pip install -r ComfyUI/custom_nodes/ComfyUI-Sonic/requirements.txt
  5. Restart ComfyUI

Install Models

We need to create this folder structure and download each model into its place.

📂 ComfyUI/
├── 📂 models/
│   └── 📂 sonic/
│       ├── audio2bucket.pth
│       ├── audio2token.pth
│       ├── unet.pth
│       ├── face_yolov8m.pt
│       ├── 📂 whisper-tiny/
│       │   ├── config.json
│       │   ├── model.safetensors
│       │   └── preprocessor_config.json
│       └── 📂 RIFE/
│           └── flownet.pkl

Download each file and place into your ComfyUI/models/sonic/ folder

Download each file and place into your ComfyUI/models/sonic/whisper-tiny/ folder

Download this file and place into your ComfyUI/models/sonic/RIFE/ folder

Sample Workflows

Download this Example Audio and save into your ComfyUI/input folder.

Initial Image Workflow

WGET Commands

If you are using Runpod, or a similar hosted GPU service, then you can access your running pod/instance using a terminal.

#
#
#
#
#
# CD into ./ComfyUI/models/sonic/ folder
wget https://huggingface.co/Sean-Bradley/ComfyUI/resolve/main/models/sonic/audio2bucket.pth
wget https://huggingface.co/Sean-Bradley/ComfyUI/resolve/main/models/sonic/audio2token.pth
wget https://huggingface.co/Sean-Bradley/ComfyUI/resolve/main/models/sonic/face_yolov8m.pt
wget https://huggingface.co/Sean-Bradley/ComfyUI/resolve/main/models/sonic/unet.pth

#
#
#
#
# CD into ./ComfyUI/models/sonic/whisper-tiny/ folder
wget https://huggingface.co/Sean-Bradley/ComfyUI/resolve/main/models/sonic/whisper-tiny/config.json
wget https://huggingface.co/Sean-Bradley/ComfyUI/resolve/main/models/sonic/whisper-tiny/model.safetensors
wget https://huggingface.co/Sean-Bradley/ComfyUI/resolve/main/models/sonic/whisper-tiny/preprocessor_config.json

#
#
# CD into ./ComfyUI/models/sonic/RIFE/ folder
wget https://huggingface.co/Sean-Bradley/ComfyUI/resolve/main/models/sonic/RIFE/flownet.pkl

Wait for files to download fully before running your workflows.

ComfyUI Sonic (GitHub)