Update README.md
Browse files
README.md
CHANGED
@@ -7,16 +7,31 @@ license_link: LICENSE
|
|
7 |
<!-- ## **HunyuanVideo** -->
|
8 |
|
9 |
<p align="center">
|
10 |
-
<img src="
|
11 |
</p>
|
12 |
|
13 |
# HunyuanVideo: A Systematic Framework For Large Video Generation Model Training
|
14 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
15 |
-----
|
16 |
|
17 |
This repo contains PyTorch model definitions, pre-trained weights and inference/sampling code for our paper exploring HunyuanVideo. You can find more visualizations on our [project page](https://aivideo.hunyuan.tencent.com).
|
18 |
|
19 |
-
> [**HunyuanVideo: A Systematic Framework For Large Video Generation Model Training**](https://
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
20 |
|
21 |
## π₯π₯π₯ News!!
|
22 |
* Dec 3, 2024: π€ We release the inference code and model weights of HunyuanVideo.
|
@@ -36,6 +51,7 @@ This repo contains PyTorch model definitions, pre-trained weights and inference/
|
|
36 |
|
37 |
## Contents
|
38 |
- [HunyuanVideo: A Systematic Framework For Large Video Generation Model Training](#hunyuanvideo--a-systematic-framework-for-large-video-generation-model-training)
|
|
|
39 |
- [π₯π₯π₯ News!!](#-news!!)
|
40 |
- [π Open-source Plan](#-open-source-plan)
|
41 |
- [Contents](#contents)
|
@@ -71,7 +87,7 @@ using a large language model, and used as the condition. Gaussian noise and cond
|
|
71 |
input, our generate model generates an output latent, which is decoded to images or videos through
|
72 |
the 3D VAE decoder.
|
73 |
<p align="center">
|
74 |
-
<img src="
|
75 |
</p>
|
76 |
|
77 |
## π **HunyuanVideo Key Features**
|
@@ -83,7 +99,7 @@ tokens and feed them into subsequent Transformer blocks for effective multimodal
|
|
83 |
This design captures complex interactions between visual and semantic information, enhancing
|
84 |
overall model performance.
|
85 |
<p align="center">
|
86 |
-
<img src="
|
87 |
</p>
|
88 |
|
89 |
### **MLLM Text Encoder**
|
@@ -91,13 +107,13 @@ Some previous text-to-video model typically use pretrainednCLIP and T5-XXL as te
|
|
91 |
Compared with CLIP, MLLM has been demonstrated superior ability in image detail description
|
92 |
and complex reasoning; (iii) MLLM can play as a zero-shot learner by following system instructions prepended to user prompts, helping text features pay more attention to key information. In addition, MLLM is based on causal attention while T5-XXL utilizes bidirectional attention that produces better text guidance for diffusion models. Therefore, we introduce an extra bidirectional token refiner for enhacing text features.
|
93 |
<p align="center">
|
94 |
-
<img src="
|
95 |
</p>
|
96 |
|
97 |
### **3D VAE**
|
98 |
HunyuanVideo trains a 3D VAE with CausalConv3D to compress pixel-space videos and images into a compact latent space. We set the compression ratios of video length, space and channel to 4, 8 and 16 respectively. This can significantly reduce the number of tokens for the subsequent diffusion transformer model, allowing us to train videos at the original resolution and frame rate.
|
99 |
<p align="center">
|
100 |
-
<img src="
|
101 |
</p>
|
102 |
|
103 |
### **Prompt Rewrite**
|
@@ -253,14 +269,13 @@ We list some more useful configurations for easy usage:
|
|
253 |
|
254 |
|
255 |
## π BibTeX
|
256 |
-
If you find [HunyuanVideo](https://
|
257 |
|
258 |
```BibTeX
|
259 |
-
@misc{
|
260 |
-
title={HunyuanVideo: A Systematic Framework For Large Video
|
261 |
-
author={
|
262 |
-
year={
|
263 |
-
eprint={XXX},
|
264 |
archivePrefix={arXiv},
|
265 |
primaryClass={cs.CV}
|
266 |
}
|
|
|
7 |
<!-- ## **HunyuanVideo** -->
|
8 |
|
9 |
<p align="center">
|
10 |
+
<img src="https://aivideo.hunyuan.tencent.com/download/HunyuanVideo/material/logo.png" height=100>
|
11 |
</p>
|
12 |
|
13 |
# HunyuanVideo: A Systematic Framework For Large Video Generation Model Training
|
14 |
|
15 |
+
<div align="center">
|
16 |
+
<a href="https://github.com/Tencent/HunyuanVideo"><img src="https://img.shields.io/static/v1?label=HunyuanVideo Code&message=Github&color=blue&logo=github-pages"></a>  
|
17 |
+
<a href="https://aivideo.hunyuan.tencent.com"><img src="https://img.shields.io/static/v1?label=Project%20Page&message=Github&color=blue&logo=github-pages"></a>  
|
18 |
+
<a href="https://github.com/Tencent/HunyuanVideo/blob/main/assets/hunyuanvideo.pdf"><img src="https://img.shields.io/static/v1?label=Tech Report&message=Arxiv:HunyuanVideo&color=red&logo=arxiv"></a>  
|
19 |
+
<a href="https://huggingface.co/tencent/HunyuanVideo"><img src="https://img.shields.io/static/v1?label=HunyuanVideo&message=HuggingFace&color=yellow"></a>    
|
20 |
+
<a href="https://huggingface.co/tencent/HunyuanVideo-PromptRewrite"><img src="https://img.shields.io/static/v1?label=HunyuanVideo-PromptRewrite&message=HuggingFace&color=yellow"></a>    
|
21 |
+
</div>
|
22 |
+
|
23 |
-----
|
24 |
|
25 |
This repo contains PyTorch model definitions, pre-trained weights and inference/sampling code for our paper exploring HunyuanVideo. You can find more visualizations on our [project page](https://aivideo.hunyuan.tencent.com).
|
26 |
|
27 |
+
> [**HunyuanVideo: A Systematic Framework For Large Video Generation Model Training**](https://github.com/Tencent/HunyuanVideo/blob/main/assets/hunyuanvideo.pdf) <br>
|
28 |
+
|
29 |
+
## π₯ Demo
|
30 |
+
<div align="center">
|
31 |
+
<video src="https://github.com/user-attachments/assets/f37925a3-7d42-40c9-8a9b-5a010c7198e2" width="50%">
|
32 |
+
</div>
|
33 |
+
|
34 |
+
Due to the limitation of github page, the video is compressed. The original video can be downloaded from [here](https://aivideo.hunyuan.tencent.com/download/HunyuanVideo/material/demo.mov).
|
35 |
|
36 |
## π₯π₯π₯ News!!
|
37 |
* Dec 3, 2024: π€ We release the inference code and model weights of HunyuanVideo.
|
|
|
51 |
|
52 |
## Contents
|
53 |
- [HunyuanVideo: A Systematic Framework For Large Video Generation Model Training](#hunyuanvideo--a-systematic-framework-for-large-video-generation-model-training)
|
54 |
+
- [π₯ Demo](#-demo)
|
55 |
- [π₯π₯π₯ News!!](#-news!!)
|
56 |
- [π Open-source Plan](#-open-source-plan)
|
57 |
- [Contents](#contents)
|
|
|
87 |
input, our generate model generates an output latent, which is decoded to images or videos through
|
88 |
the 3D VAE decoder.
|
89 |
<p align="center">
|
90 |
+
<img src="https://aivideo.hunyuan.tencent.com/download/HunyuanVideo/material/overall.png" height=300>
|
91 |
</p>
|
92 |
|
93 |
## π **HunyuanVideo Key Features**
|
|
|
99 |
This design captures complex interactions between visual and semantic information, enhancing
|
100 |
overall model performance.
|
101 |
<p align="center">
|
102 |
+
<img src="https://aivideo.hunyuan.tencent.com/download/HunyuanVideo/material/backbone.png" height=350>
|
103 |
</p>
|
104 |
|
105 |
### **MLLM Text Encoder**
|
|
|
107 |
Compared with CLIP, MLLM has been demonstrated superior ability in image detail description
|
108 |
and complex reasoning; (iii) MLLM can play as a zero-shot learner by following system instructions prepended to user prompts, helping text features pay more attention to key information. In addition, MLLM is based on causal attention while T5-XXL utilizes bidirectional attention that produces better text guidance for diffusion models. Therefore, we introduce an extra bidirectional token refiner for enhacing text features.
|
109 |
<p align="center">
|
110 |
+
<img src="https://aivideo.hunyuan.tencent.com/download/HunyuanVideo/material/text_encoder.png" height=275>
|
111 |
</p>
|
112 |
|
113 |
### **3D VAE**
|
114 |
HunyuanVideo trains a 3D VAE with CausalConv3D to compress pixel-space videos and images into a compact latent space. We set the compression ratios of video length, space and channel to 4, 8 and 16 respectively. This can significantly reduce the number of tokens for the subsequent diffusion transformer model, allowing us to train videos at the original resolution and frame rate.
|
115 |
<p align="center">
|
116 |
+
<img src="https://aivideo.hunyuan.tencent.com/download/HunyuanVideo/material/3dvae.png" height=150>
|
117 |
</p>
|
118 |
|
119 |
### **Prompt Rewrite**
|
|
|
269 |
|
270 |
|
271 |
## π BibTeX
|
272 |
+
If you find [HunyuanVideo](https://github.com/Tencent/HunyuanVideo/blob/main/assets/hunyuanvideo.pdf) useful for your research and applications, please cite using this BibTeX:
|
273 |
|
274 |
```BibTeX
|
275 |
+
@misc{kong2024hunyuanvideo,
|
276 |
+
title={HunyuanVideo: A Systematic Framework For Large Video Generative Models},
|
277 |
+
author={Weijie Kong, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Qi Tian, Jianwei Zhang, Kathrina Wu, Qin Lin, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, Caesar Zhong},
|
278 |
+
year={2024},
|
|
|
279 |
archivePrefix={arXiv},
|
280 |
primaryClass={cs.CV}
|
281 |
}
|