Spaces:
Running
on
Zero
Apply for community grant: Academic project (gpu)
Dear HuggingFace, I am writing for applying the GPU Grant for this space: MotionInversion.
This is a video generation project that produces video highly aligned with the motion in reference video. In this project, we propose a novel method to capture and reproduce ref-video motion within ten minutes of training time. In both qualitative and quantitative experiments, our method demonstrate practical advantages and effectiveness. We implement several checkpoints in this space, allowing everyone to experience it conveniently and efficiently.
arxiv: https://arxiv.org/abs/2403.20193
website: coming soon
code: coming soon
abs:
In this work, we present a novel approach for motion customization in video generation, addressing the widespread gap in the exploration of motion representation within video generative models. Recognizing the unique challenges posed by the spatiotemporal nature of video, our method introduces Motion Embeddings, a set of explicit, temporally coherent embeddings derived from a given video. These embeddings are designed to integrate seamlessly with the temporal transformer modules of video diffusion models, modulating self-attention computations across frames without compromising spatial integrity. Our approach provides a compact and efficient solution to motion representation, utilizing two types of embeddings: a Motion Query-Key Embedding to modulate the temporal attention map and a Motion Value Embedding to modulate the attention values. Additionally, we introduce an inference strategy that excludes spatial dimensions from the Motion Query-Key Embedding and applies a differential operation to the Motion Value Embedding, both designed to debias appearance and ensure the embeddings focus solely on motion. Our contributions include the introduction of a tailored motion embedding for customization tasks and a demonstration of the practical advantages and effectiveness of our method through extensive experiments.
Thank you for reviewing our request.
Best,
Motion Inversion authors
Hi @ziyangmai , we've assigned ZeroGPU to this Space. Please check the compatibility and usage sections of this page so your Space can run on ZeroGPU.