MotiF: Making Text Count in Image Animation with Motion Focal Loss

Recent works (e.g., Motion-prompting) use motion input to control image-to-video generation. While motion-conditioned models are effective, it is equally critical for models to directly translate text conditioning into motion rendered in videos. In this work, we focus on text-image-to-video (TI2V) task.

A common limitation of existing TI2V methods is their tendency to generate videos with limited and identical motions when given an image and multiple prompts. We hypothesize that this limitation arises from insufficient emphasis on motion patterns when all regions are optimized equally in the L2 loss. This can result in "Condition Leakage," where the loss becomes low simply by copying the condition frames. To address this, we propose Motion Focal loss (MotiF) to guide TI2V training to focus on regions with more motion via motion heatmap re-weighting.

We thus propose TI2V-Bench, a benchmark that comprises text-image pairs from 22 diverse scenarios featuring a variety of objects and scenes. Each scenario includes 3 to 5 images with similar content presented in different styles, alongside 3 to 5 distinct prompts designed to animate these images and generate varied outputs. The benchmark includes a total of 320 image-text pairs, consisting of 88 unique images and 133 unique prompts.
Check TI2V-Bench in https://huggingface.co/datasets/wang-sj16/TI2V-Bench!

We conduct human evaluation to compare MotiF to nine open-sourced models on TI2V-Bench. We achieved considerable improvements across the board with an average preference of 72%. Analysis of justification choices reveals that MotiF excels particularly in enhancing text alignment and generating accurate object motion.

Check more details and comparison results from our project page (https://wang-sj16.github.io/motif/) and the paper (https://arxiv.org/abs/2412.16153)!