license: other
license_name: tencent-hunyuan-community
license_link: https://huggingface.co/Tencent-Hunyuan/HunyuanDiT/blob/main/LICENSE.txt
language:
- en
Using HunyuanDiT IP-Adapter
Instructions
The dependencies and installation are basically the same as the base model, and we use the module weights for training. Download the model using the following commands:
cd HunyuanDiT
# Use the huggingface-cli tool to download the model.
# We recommend using module weights as the base model for IP-Adapter inference, as our provided pretrained weights are trained on them.
huggingface-cli download Tencent-Hunyuan/IP-Adapter/ipa.pt --local-dir ./ckpts/t2i/model
huggingface-cli download Tencent-Hunyuan/IP-Adapter/clip_img_encoder.pt --local-dir ./ckpts/t2i/model/clip_img_encoder
# Quick start
python3 sample_ipadapter.py --infer-mode fa --ref-image-path ipadapter/input/tiger.png --i-scale 1.0 --prompt 一只老虎在海洋中游泳,背景是海洋。构图方式是居中构图,呈现了动漫风格和文化,营造了平静的氛围。 --infer-steps 100 --is-ipa True --load-key module
Examples of ref input and IP-Adapter results are as follows:
Ref Input | ||
IP-Adapter Output | ||
一只老虎在奔跑。 (A tiger running.) |
一个卡通美女,抱着一只小猪。 (A cartoon beauty holding a little pig.) |
一片紫色薰衣草地。 (A purple lavender field.) |
一只老虎在看书。 (A tiger is reading a book.) |
一个卡通美女,穿着绿色衣服。 (A cartoon beauty wearing green clothes.) |
一片紫色薰衣草地,有一只可爱的小狗。 (A purple lavender field with a cute puppy.) |
一只老虎在咆哮。 (A tiger is roaring.) |
一个卡通美女,戴着墨镜。 (A cartoon beauty wearing sunglasses.) |
水墨风格,一片紫色薰衣草地。 (Ink style. A purple lavender field.) |
Training
We provide base model weights for IP-Adapter training, you can use module
weights for IP-Adapter training.
Here is an example, we load the module
weights into the main model and conduct IP-Adapter training.
If apply multiple resolution training, you need to add the --multireso
and --reso-step 64
parameter.
task_flag="IP_Adapter" # the task flag is used to identify folders. # checkpoint root for resume
index_file=path/to/your/index_file
results_dir=./log_EXP # save root for results
batch_size=1 # training batch size
image_size=1024 # training image resolution
grad_accu_steps=1 # gradient accumulation
warmup_num_steps=0 # warm-up steps
lr=0.0001 # learning rate
ckpt_every=10 # create a ckpt every a few steps.
ckpt_latest_every=10000 # create a ckpt named `latest.pt` every a few steps.
ckpt_every_n_epoch=2 # create a ckpt every a few epochs.
epochs=8 # total training epochs
PYTHONPATH=. \
sh $(dirname "$0")/run_g_ipadapter.sh \
--task-flag ${task_flag} \
--noise-schedule scaled_linear --beta-start 0.00085 --beta-end 0.018 \
--predict-type v_prediction \
--multireso \
--reso-step 64 \
--uncond-p 0.22 \
--uncond-p-t5 0.22\
--uncond-p-img 0.05\
--index-file ${index_file} \
--random-flip \
--lr ${lr} \
--batch-size ${batch_size} \
--image-size ${image_size} \
--global-seed 999 \
--grad-accu-steps ${grad_accu_steps} \
--warmup-num-steps ${warmup_num_steps} \
--use-flash-attn \
--use-fp16 \
--extra-fp16 \
--results-dir ${results_dir} \
--resume\
--resume-module-root ckpts/t2i/model/pytorch_model_module.pt \
--epochs ${epochs} \
--ckpt-every ${ckpt_every} \
--ckpt-latest-every ${ckpt_latest_every} \
--ckpt-every-n-epoch ${ckpt_every_n_epoch} \
--log-every 10 \
--deepspeed \
--use-zero-stage 2 \
--gradient-checkpointing \
--no-strict \
--training-parts ipadapter \
--is-ipa True \
--resume-ipa True \
--resume-ipa-root ckpts/t2i/model/ipa.pt \
"$@"
Recommended parameter settings
Parameter | Description | Recommended Parameter Value | Note |
---|---|---|---|
--batch-size |
Training batch size | 1 | Depends on GPU memory |
--grad-accu-steps |
Size of gradient accumulation | 2 | - |
--lr |
Learning rate | 0.0001 | - |
--training-parts |
be trained parameters when training IP-Adapter | ipadapter | - |
--is-ipa |
training IP-Adapter or not | True | - |
--resume-ipa-root |
resume ipa model or not when training | ipa model path | - |
Inference
Use the following command line for inference.
a. Use the parameter float i-scale to specify the weight of IP-Adapter reference image. The bigger parameter indicates more relativity to reference image.
python3 sample_ipadapter.py --infer-mode fa --ref-image-path ipadapter/input/beach.png --i-scale 1.0 --prompt 一只老虎在海洋中游泳,背景是海洋。构图方式是居中构图,呈现了动漫风格和文化,营造了平静的氛围。 --infer-steps 100 --is-ipa True --load-key module