|
--- |
|
license: apache-2.0 |
|
--- |
|
# Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference |
|
|
|
<a href="https://arxiv.org/abs/2406.04314"><img src="https://img.shields.io/badge/Paper-arXiv-red?style=for-the-badge" height=22.5></a> |
|
<a href="https://github.com/RockeyCoss/SPO"><img src="https://img.shields.io/badge/Gihub-Code-succees?style=for-the-badge&logo=GitHub" height=22.5></a> |
|
<a href="https://rockeycoss.github.io/spo.github.io/"><img src="https://img.shields.io/badge/Project-Page-blue?style=for-the-badge" height=22.5></a> |
|
|
|
## Abstract |
|
<p> |
|
Generating visually appealing images is fundamental to modern text-to-image generation models. |
|
A potential solution to better aesthetics is direct preference optimization (DPO), |
|
which has been applied to diffusion models to improve general image quality including prompt alignment and aesthetics. |
|
Popular DPO methods propagate preference labels from clean image pairs to all the intermediate steps along the two generation trajectories. |
|
However, preference labels provided in existing datasets are blended with layout and aesthetic opinions, which would disagree with aesthetic preference. |
|
Even if aesthetic labels were provided (at substantial cost), it would be hard for the two-trajectory methods to capture nuanced visual differences at different steps. |
|
</p> |
|
<p> |
|
To improve aesthetics economically, this paper uses existing generic preference data and introduces step-by-step preference optimization |
|
(SPO) that discards the propagation strategy and allows fine-grained image details to be assessed. Specifically, |
|
at each denoising step, we 1) sample a pool of candidates by denoising from a shared noise latent, |
|
2) use a step-aware preference model to find a suitable win-lose pair to supervise the diffusion model, and |
|
3) randomly select one from the pool to initialize the next denoising step. |
|
This strategy ensures that diffusion models focus on the subtle, fine-grained visual differences |
|
instead of layout aspect. We find that aesthetic can be significantly enhanced by accumulating these |
|
improved minor differences. |
|
</p> |
|
<p> |
|
When fine-tuning Stable Diffusion v1.5 and SDXL, SPO yields significant |
|
improvements in aesthetics compared with existing DPO methods while not sacrificing image-text alignment |
|
compared with vanilla models. Moreover, SPO converges much faster than DPO methods due to the step-by-step |
|
alignment of fine-grained visual details. |
|
</p> |
|
|
|
## Model Description |
|
The models in this repository are step-aware preference models used for fine-tuning SD v1.5 and SDXL. For more details, please visit our [GitHub repository](https://github.com/RockeyCoss/SPO). |
|
|
|
## Citation |
|
If you find our work or codebase useful, please consider giving us a star and citing our work. |
|
``` |
|
@article{liang2024step, |
|
title={Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization}, |
|
author={Liang, Zhanhao and Yuan, Yuhui and Gu, Shuyang and Chen, Bohan and Hang, Tiankai and Cheng, Mingxi and Li, Ji and Zheng, Liang}, |
|
journal={arXiv preprint arXiv:2406.04314}, |
|
year={2024} |
|
} |
|
``` |
