LTXV In Context LoRA Pose Control

Video Lecture

Section	Video Links
LTXV ICLora Pose

Description

We will use the LTXV In Context LoRA Pose Control model to add pose context while generating a video.

Install the ComfyUI-LTXVideo Custom Nodes

Install the custom node ComfyUI-LTXVideo using the manager, or you can use your command/terminal prompt.

1. Navigate to your ComfyUI/custom_nodes folder.
2. Run,

   git clone https://github.com/Lightricks/ComfyUI-LTXVideo.git
   

3. Restart ComfyUI

Install the ComfyUI-KJNodes Custom Nodes

Install the custom node ComfyUI-KJNodes using the manager, or you can use your command/terminal prompt.

1. Navigate to your ComfyUI/custom_nodes folder.
2. Run,

   git clone https://github.com/kijai/ComfyUI-KJNodes.git
   

3. Restart ComfyUI

Install the IC LoRA Pose Control Model

Download ltxv-097-ic-lora-pose-control-comfyui.safetensors and save it into your ComfyUI/models/lora/ folder.

📂 ComfyUI/
├── 📂 models/
│   └── 📂 loras/
│       └── ltxv-097-ic-lora-pose-control-comfyui.safetensors

Pose Videos

Download	Pose Video	Guiding Image	Workflow
Dance 1
Dance 2
Macarena
Turning Walk
Karate Kick
Walk left to right
Jog Right to Left
Jog Left to Right Close

WGET Commands

If you are using Runpod, or a similar hosted GPU service, then you can access your running pod/instance using a terminal.

# CD into ./ComfyUI/models/loras/ folder
wget https://huggingface.co/Lightricks/LTX-Video-ICLoRA-pose-13b-0.9.7/resolve/main/ltxv-097-ic-lora-pose-control-comfyui.safetensors

Wait for files to download fully before running your workflows.

Troubleshooting

Error : ImportError: cannot import name 'apply_rotary_emb' from 'comfy.ldm.lightricks.model'

Open .\ComfyUI\custom_nodes\ComfyUI-LTXVideo\tricks\modules\ltx_model.py

Delete/Add lines as shown below, save and restart ComfyUI.

import math

import comfy.ldm.common_dit
import comfy.ldm.modules.attention
import torch
from comfy.ldm.lightricks.model import (
    BasicTransformerBlock,
    LTXVModel,
-   apply_rotary_emb,
    precompute_freqs_cis,
)
+ from comfy.ldm.flux.math import apply_rope1
from comfy.ldm.lightricks.symmetric_patchifier import latent_to_pixel_coords
from torch import nn

from ..utils.feta_enhance_utils import get_feta_scores


class LTXModifiedCrossAttention(nn.Module):
    def forward(self, x, context=None, mask=None, pe=None, transformer_options={}):
        context = x if context is None else context
        context_v = x if context is None else context

        step = transformer_options.get("step", -1)
        total_steps = transformer_options.get("total_steps", 0)
        attn_bank = transformer_options.get("attn_bank", None)
        sample_mode = transformer_options.get("sample_mode", None)
        if attn_bank is not None and self.idx in attn_bank["block_map"]:
            len_conds = len(transformer_options["cond_or_uncond"])
            pred_order = transformer_options["pred_order"]
            if (
                sample_mode == "forward"
                and total_steps - step - 1 < attn_bank["save_steps"]
            ):
                step_idx = f"{pred_order}_{total_steps-step-1}"
                attn_bank["block_map"][self.idx][step_idx] = x.cpu()
            elif sample_mode == "reverse" and step < attn_bank["inject_steps"]:
                step_idx = f"{pred_order}_{step}"
                inject_settings = attn_bank.get("inject_settings", {})
                if len(inject_settings) > 0:
                    inj = (
                        attn_bank["block_map"][self.idx][step_idx]
                        .to(x.device)
                        .repeat(len_conds, 1, 1)
                    )
                if "q" in inject_settings:
                    x = inj
                if "k" in inject_settings:
                    context = inj
                if "v" in inject_settings:
                    context_v = inj

        q = self.to_q(x)
        k = self.to_k(context)
        v = self.to_v(context_v)

        q = self.q_norm(q)
        k = self.k_norm(k)

-       if pe is not None:
-           q = apply_rotary_emb(q, pe)
-           k = apply_rotary_emb(k, pe)
+       q, k = apply_rope1(q, k)

        feta_score = None

Useful Links

ComfyUI-LTXVideo (GitHub)

ComfyUI-KJNodes

ltxv-097-ic-lora-pose-control-comfyui.safetensors (huggingface)

Dancing Videos (pexels)