Update readme

README.md

@@ -11,8 +11,6 @@ inference: false
 
 <video src='https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/animatediff_lightning_samples_t2v.mp4' width="100%" autoplay muted loop></video>
 
-<video src='https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/animatediff_lightning_samples_v2v.mp4' width="100%" autoplay muted loop></video>
-
 AnimateDiff-Lightning is a lightning-fast text-to-video generation model. It can generate 16-frame 512px videos in a few steps. For more information, please refer to our research paper: [AnimateDiff-Lightning: Cross-Model Diffusion Distillation](https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/animatediff_lightning_report.pdf). We release the model as part of the research.
 
 Our models are distilled from [AnimateDiff SD1.5 v2](https://huggingface.co/guoyww/animatediff). This repository contains checkpoints for 1-step, 2-step, 4-step, and 8-step distilled models. The generation quality of our 2-step, 4-step, and 8-step model is great. Our 1-step model is only provided for research purposes.
@@ -67,15 +65,39 @@ export_to_gif(output.frames[0], "animation.gif")
 
 ## ComfyUI Usage
 
-1. Download [AnimateDiff-Lightning ComfyUI Workflow](https://huggingface.co/ByteDance/AnimateDiff-Lightning/raw/main/comfyui/animatediff_lightning_workflow.json)
-
-2. Install nodes. You can install them manually or use [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager).
+1. Download [animatediff_lightning_workflow.json](https://huggingface.co/ByteDance/AnimateDiff-Lightning/raw/main/comfyui/animatediff_lightning_workflow.json) and import it in ComfyUI.
+1. Install nodes. You can install them manually or use [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager).
     * [ComfyUI-AnimateDiff-Evolved](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved)
     * [ComfyUI-VideoHelperSuite](https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite)
+1. Download your favorite base model checkpoint and put them under `/models/checkpoints/`
+1. Download AnimateDiff-Lightning checkpoint `animatediff_lightning_Nstep_comfyui.safetensors` and put them under `/custom_nodes/ComfyUI-AnimateDiff-Evolved/models/`
+
+
+![ComfyUI Workflow](https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/comfyui/animatediff_lightning_workflow.jpg)
+
 
-3. Download your favorite base model checkpoint and put them under `/models/checkpoints/`
+## Video-to-Video Generation
+
+<video src='https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/animatediff_lightning_samples_v2v.mp4' width="100%" autoplay muted loop></video>
+
+AnimateDiff-Lightning is great for video-to-video generation. We provide the simplist comfyui workflow using ControlNet.
+
+1. Download [animatediff_lightning_v2v_openpose_workflow.json](https://huggingface.co/ByteDance/AnimateDiff-Lightning/raw/main/comfyui/animatediff_lightning_v2v_openpose_workflow.json) and import it in ComfyUI.
+1. Install nodes. You can install them manually or use [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager).
+    * [ComfyUI-AnimateDiff-Evolved](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved)
+    * [ComfyUI-VideoHelperSuite](https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite)
+    * [ComfyUI-Advanced-ControlNet](https://github.com/Kosinkadink/ComfyUI-Advanced-ControlNet)
+    * [comfyui_controlnet_aux](https://github.com/Fannovel16/comfyui_controlnet_aux)
+1. Download your favorite base model checkpoint and put them under `/models/checkpoints/`
+1. Download AnimateDiff-Lightning checkpoint `animatediff_lightning_Nstep_comfyui.safetensors` and put them under `/custom_nodes/ComfyUI-AnimateDiff-Evolved/models/`
+1. Download [ControlNet OpenPose](https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main) `control_v11p_sd15_openpose.pth` checkpoint to `/models/controlnet/`
+1. Upload your video and run the pipeline.
 
-4. Download AnimateDiff-Lightning checkpoint `animatediff_lightning_Nstep_comfyui.safetensors` and put them under `/custom_nodes/ComfyUI-AnimateDiff-Evolved/models/`
+Additional notes:
 
+1. Video shouldn't be too long or too high resolution. We used 576x1024 8 second 30fps videos for testing.
+1. Set the frame rate to match your input video. This allows audio to match with the output video.
+1. DWPose will download checkpoint itself on its first run.
+1. DWPose may get stuck in UI, but the pipeline is actually still running in the background. Check ComfyUI log and your output folder.
 
-![ComfyUI Workflow](https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/comfyui/animatediff_lightning_workflow.jpg)
+![ComfyUI OpenPose Workflow](https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/comfyui/animatediff_lightning_v2v_openpose_workflow.jpg)