|
--- |
|
base_model: stabilityai/stable-diffusion-xl-base-1.0 |
|
library_name: diffusers |
|
license: openrail++ |
|
tags: |
|
- text-to-image |
|
- text-to-image |
|
- diffusers-training |
|
- diffusers |
|
- lora |
|
- template:sd-lora |
|
- stable-diffusion-xl |
|
- stable-diffusion-xl-diffusers |
|
datasets: |
|
- data-is-better-together/open-image-preferences-v1-binarized |
|
language: |
|
- en |
|
pipeline_tag: text-to-image |
|
--- |
|
|
|
|
|
# Low Rank Adapted Supervised Fine Tuned Stable Diffusion XL |
|
|
|
## Comparison |
|
|
|
| Prompt | SDXL | Fine Tuned | |
|
| :--: | :--: | :--: | |
|
| a boat in the canals of Venice, painted in gouache with soft, flowing brushstrokes and vibrant, translucent colors, capturing the serene reflection on the water under a misty ambiance, with rich textures and a dynamic perspective |  |  | |
|
| Grainy shot of a robot cooking in the kitchen, with soft shadows and nostalgic film texture. |  |  | |
|
|
|
## Model description |
|
|
|
These are ariG23498/open-image-preferences-v1-sdxl-lora LoRA adaption weights for stabilityai/stable-diffusion-xl-base-1.0. |
|
|
|
The weights were trained using [DreamBooth](https://github.com/ariG23498/diffusers/blob/aritra/sdxl-lora/examples/dreambooth/train_dreambooth_lora_sdxl.py) using the |
|
[open-image-preferences-v1-binarized](https://huggingface.co/datasets/data-is-better-together/open-image-preferences-v1-binarized) dataset. |
|
|
|
|
|
## Use with `diffusers` |
|
|
|
```py |
|
from diffusers import AutoPipelineForText2Image |
|
import torch |
|
|
|
pipeline = AutoPipelineForText2Image.from_pretrained( |
|
"stabilityai/stable-diffusion-xl-base-1.0", |
|
torch_dtype=torch.bfloat16 |
|
).to('cuda') |
|
pipeline.load_lora_weights('ariG23498/open-image-preferences-v1-sdxl-lora', weight_name='pytorch_lora_weights.safetensors') |
|
prompt = "ENTER PROMPT" |
|
image = pipeline(prompt).images[0] |
|
``` |
|
|
|
## Command to train the model |
|
|
|
```shell |
|
!accelerate launch examples/dreambooth/train_dreambooth_lora_sdxl.py \ |
|
--pretrained_model_name_or_path "stabilityai/stable-diffusion-xl-base-1.0" \ |
|
--dataset_name "data-is-better-together/open-image-preferences-v1-binarized" \ |
|
--hub_model_id "ariG23498/open-image-preferences-v1-sdxl-lora" \ |
|
--push_to_hub \ |
|
--output_dir "open-image-preferences-v1-sdxl-lora" \ |
|
--image_column "chosen" \ |
|
--caption_column "prompt" \ |
|
--mixed_precision="bf16" \ |
|
--resolution=1024 \ |
|
--train_batch_size=1 \ |
|
--repeats=1 \ |
|
--report_to="wandb"\ |
|
--gradient_accumulation_steps=1 \ |
|
--gradient_checkpointing \ |
|
--learning_rate=1.0 \ |
|
--text_encoder_lr=1.0 \ |
|
--optimizer="prodigy"\ |
|
--lr_scheduler="constant" \ |
|
--lr_warmup_steps=0 \ |
|
--rank=8 \ |
|
--checkpointing_steps=2000 \ |
|
--seed="0" |
|
``` |
