license: mit
Latte: Latent Diffusion Transformer for Video Generation
This repo contains pre-trained weights for our paper exploring latent diffusion models with transformers (Latte). You can find more visualizations on our project page.
News
(π₯ New) May. 23, 2024. π₯ The updated LatteT2V model is released at here. If you want to use the updated model to generate images directly, please make sure
video_length=1,enable_temporal_attentions=Trueandenable_vae_temporal_decoder=Falsein t2v_sample.yaml.(π₯ New) Mar. 20, 2024. π₯ An updated LatteT2V model is coming soon, stay tuned!
(π₯ New) Feb. 24, 2024. π₯ We are very grateful that researchers and developers like our work. We will continue to update our LatteT2V model, hoping that our efforts can help the community develop. Our Latte discord channel is created for discussions. Coders are welcome to contribute.
(π₯ New) Jan. 9, 2024. π₯ An updated LatteT2V model initialized with the PixArt-Ξ± is released, the checkpoint can be found here.
(π₯ New) Oct. 31, 2023. π₯ The training and inference code is released. All checkpoints (including FaceForensics, SkyTimelapse, UCF101, and Taichi-HD) can be found here. In addition, the LatteT2V inference code is provided.
Contact Us
Yaohui Wang: wangyaohui@pjlab.org.cn Xin Ma: xin.ma1@monash.edu
Citation
If you find this work useful for your research, please consider citing it.
@article{ma2024latte,
title={Latte: Latent Diffusion Transformer for Video Generation},
author={Ma, Xin and Wang, Yaohui and Jia, Gengyun and Chen, Xinyuan and Liu, Ziwei and Li, Yuan-Fang and Chen, Cunjian and Qiao, Yu},
journal={arXiv preprint arXiv:2401.03048},
year={2024}
}
Acknowledgments
Latte has been greatly inspired by the following amazing works and teams: DiT and PixArt-Ξ±, we thank all the contributors for open-sourcing.