|
--- |
|
tags: |
|
- text-to-image |
|
- stable-diffusion |
|
|
|
language: |
|
- en |
|
library_name: diffusers |
|
--- |
|
|
|
# IP-Adapter-FaceID Model Card |
|
|
|
|
|
<div align="center"> |
|
|
|
[**Project Page**](https://ip-adapter.github.io) **|** [**Paper (ArXiv)**](https://arxiv.org/abs/2308.06721) **|** [**Code**](https://github.com/tencent-ailab/IP-Adapter) |
|
</div> |
|
|
|
--- |
|
|
|
|
|
|
|
## Introduction |
|
|
|
An experimental version of IP-Adapter-FaceID: we use face ID embedding from a face recognition model instead of CLIP image embedding, additionally, we use LoRA to improve ID consistency. IP-Adapter-FaceID can generate various style images conditioned on a face with only text prompts. |
|
|
|
 |
|
|
|
## Usage |
|
|
|
Firstly, you should use [insightface](https://github.com/deepinsight/insightface) to extract face ID embedding: |
|
|
|
```python |
|
|
|
import cv2 |
|
from insightface.app import FaceAnalysis |
|
|
|
|
|
app = FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider']) |
|
app.prepare(ctx_id=0, det_size=(640, 640)) |
|
|
|
image = cv2.imread("person.jpg") |
|
faces = app.get(image) |
|
|
|
faceid_embeds = torch.from_numpy(faces[0].normed_embedding).unsqueeze(0) |
|
``` |
|
|
|
Then, you can generate images conditioned on the face embeddings: |
|
|
|
```python |
|
|
|
import torch |
|
from diffusers import StableDiffusionPipeline, DDIMScheduler, AutoencoderKL |
|
from PIL import Image |
|
|
|
from ip_adapter.ip_adapter_faceid import IPAdapterFaceID |
|
|
|
base_model_path = "SG161222/Realistic_Vision_V4.0_noVAE" |
|
vae_model_path = "stabilityai/sd-vae-ft-mse" |
|
ip_ckpt = "ip-adapter-faceid_sd15.bin" |
|
device = "cuda" |
|
|
|
noise_scheduler = DDIMScheduler( |
|
num_train_timesteps=1000, |
|
beta_start=0.00085, |
|
beta_end=0.012, |
|
beta_schedule="scaled_linear", |
|
clip_sample=False, |
|
set_alpha_to_one=False, |
|
steps_offset=1, |
|
) |
|
vae = AutoencoderKL.from_pretrained(vae_model_path).to(dtype=torch.float16) |
|
pipe = StableDiffusionPipeline.from_pretrained( |
|
base_model_path, |
|
torch_dtype=torch.float16, |
|
scheduler=noise_scheduler, |
|
vae=vae, |
|
feature_extractor=None, |
|
safety_checker=None |
|
) |
|
|
|
# load ip-adapter |
|
ip_model = IPAdapterFaceID(pipe, ip_ckpt, device) |
|
|
|
# generate image |
|
prompt = "photo of a woman in red dress in a garden" |
|
negative_prompt = "monochrome, lowres, bad anatomy, worst quality, low quality, blurry" |
|
|
|
images = ip_model.generate( |
|
prompt=prompt, negative_prompt=negative_prompt, faceid_embeds=faceid_embeds, num_samples=4, width=512, height=768, num_inference_steps=30, seed=2023 |
|
) |
|
|
|
``` |
|
|
|
|
|
## Limitations and Bias |
|
- The model does not achieve perfect photorealism and ID consistency. |
|
- The generalization of the model is limited due to limitations of the training data, base model and face recognition model. |
|
|
|
|
|
|
|
## Non-commercial use |
|
**This model is released exclusively for research purposes and is not intended for commercial use.** |
|
|
|
|
