|
--- |
|
language: |
|
- en |
|
pipeline_tag: image-to-image |
|
tags: |
|
- Diffusion Models |
|
- Stable Diffusion |
|
- ControlNet |
|
- Perturbed-Attention Guidance |
|
- PAG |
|
--- |
|
|
|
# Super-Resolution with Perturbed-Attention Guidance |
|
|
|
[Project](https://ku-cvlab.github.io/Perturbed-Attention-Guidance/) / [arXiv](https://arxiv.org/abs/2403.17377) / [GitHub](https://github.com/KU-CVLAB/Perturbed-Attention-Guidance) |
|
|
|
This repository is based on [Diffusers](https://huggingface.co/docs/diffusers/index). |
|
|
|
[ControlNet](https://arxiv.org/abs/2302.05543) is a neural network structure to control diffusion models by adding extra conditions. The pipeline is a modification of StableDiffusionControlNetPipeline to support image generation with ControlNet and Perturbed-Attention Guidance (PAG). |
|
|
|
In addition to the examples provided with Openpose, you can also generate using various conditions. Please refer to "ControlNet" section of an [official document](https://huggingface.co/docs/diffusers/using-diffusers/controlnet) for details. |
|
|
|
## Quickstart |
|
|
|
Loading ControlNet and Custom Piepline: |
|
|
|
``` |
|
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline |
|
|
|
controlnet = ControlNetModel.from_pretrained( |
|
"lllyasviel/sd-controlnet-openpose", |
|
torch_dtype=torch.float16 |
|
) |
|
|
|
pipe = StableDiffusionControlNetPipeline.from_pretrained( |
|
"runwayml/stable-diffusion-v1-5", |
|
custom_pipeline="hyoungwoncho/sd_perturbed_attention_guidance_controlnet", |
|
controlnet=controlnet, |
|
torch_dtype=torch.float16 |
|
) |
|
|
|
device="cuda" |
|
pipe = pipe.to(device) |
|
``` |
|
|
|
Prepare Conditional Images: |
|
|
|
``` |
|
from controlnet_aux import OpenposeDetector |
|
|
|
openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet") |
|
original_image = load_image( |
|
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/person.png" |
|
) |
|
openpose_image = openpose(original_image) |
|
|
|
prompts="" |
|
``` |
|
|
|
Conditional Generation with ControlNet and PAG: |
|
|
|
``` |
|
output = pipe( |
|
prompts, |
|
image=openpose_image, |
|
num_inference_steps=50, |
|
guidance_scale=0.0, |
|
pag_scale=4.0, |
|
pag_applied_layers_index=["m0"] |
|
).images[0] |
|
``` |
|
|
|
## Parameters |
|
|
|
guidance_scale : gudiance scale of CFG (ex: 7.5) |
|
|
|
pag_scale : gudiance scale of PAG (ex: 4.0) |
|
|
|
pag_applied_layers_index : index of the layer to apply perturbation (ex: ['m0']) |
