File size: 912 Bytes
506646f c533bf6 506646f f2a93bb 506646f f2a93bb 506646f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# InstructPix2Pix for AnimateDiff
This model is a fine-tuned version of the AnimateDiff V2 model using the InstructPix2Pix as the T2I model.
## Inference
You need use a pipeline built for InstructPix2Pix and AnimateDiff inference. And example is [here](https://github.com/viddle-app/viddle-pix2pix-animatediff)
Using the pipeline, InstructPix2Pix T2I model and this motion module, you can generate animations that use an init image faithfully.
### Init Image
_Make her smile_
_Make her on fire_
## Training
The training procedure was modified to pass the first frame of all videos to the extra channels of the InstructPix2Pix model. The model was trained 4500 steps with a learning rate of 5e-7 and 5500 steps at 1e-7, with a batch size of 1.
## LICENSE
Creative OpenRAIL License |