metadata

license: unknown

Stable Video Diffusion Temporal Controlnet

Overview

Introducing the Stable Video Diffusion Temporal Controlnet! This tool uses a controlnet style encoder with the svd base. It's designed to enhance your video diffusion projects by providing precise temporal control.

Setup

Controlnet Model: First, obtain a svd temporal controlnet model folder from this repo and use it with the github inference code here: https://github.com/CiaraStrawberry/sdv_controlnet
Installation: run pip install -r requirements.txt
Execution: Run "run_inference.py".

Demo

Notes

Focus on Central Object: The system tends to extract motion features primarily from a central object and, occasionally, from the background. It's best to avoid overly complex motion or obscure objects.
Simplicity in Motion: Stick to motions that svd can handle well without the controlnet. This ensures it will be able to apply the motion.

Acknowledgements

Diffusers Team: For the svd implementation.
Pixeli99: For providing a practical svd training script: SVD_Xtend