|
# InstructPix2Pix for AnimateDiff |
|
|
|
This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model. |
|
|
|
## Inference |
|
|
|
You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is [here](https://github.com/viddle-app/viddle-pix2pix-animatediff) |
|
|
|
Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully. |
|
|
|
### Init Image |
|
|
|
![Init Image](images/mona-lisa.jpg) |
|
|
|
_Make her smile_ |
|
|
|
![Smiling](images/mona-lisa-smiles.mp4) |
|
|
|
_Make her on fire_ |
|
|
|
![On Fire](images/mona-lisa-on-fire.mp4) |
|
|
|
## Training |
|
The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1. |
|
|
|
## LICENSE |
|
Creative OpenRAIL License |