# InstructPix2Pix for AnimateDiff

This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model. 

## Inference 

You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is [here](https://github.com/viddle-app/viddle-pix2pix-animatediff)

Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully.

### Init Image

![Init Image](images/mona-lisa.jpg)

_Make her smile_

![Smiling](images/mona-lisa-smiles.gif)

_Make her on fire_

![On Fire](images/mona-lisa-on-fire.gif)

## Training
The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1. 

## LICENSE 
Creative OpenRAIL License