OnePlus

The OneDiffusion finetune, style changer, one step closer to other sdxl style(s).

Proof of Concept. As for now, I only upload the patch for the transformer.

Inference

The current inference script doesn't support reassigning the transformer model, you have to replace the transformer directory manually. This will be the case until a built-in diffusers implementation becomes available.

(I use a custom pipeline for training, that allows reassigning the pipeline components.)

Given that the OneDiffusion model is already downloaded, rename the "transformer" to "transformer_old". Copy the content of this repo to the OneDiffusion folder.

from safetensors.torch import load_file, save_file

if __name__ == '__main__':
    sd = load_file('./OneDiffusion/transformer_old/diffusion_pytorch_model.safetensors')
    adapter = load_file('./OneDiffusion/transformer/one_adapter.safetensors')
    for key, tensor in adapter.items():
        assert key in sd
        sd[key] = tensor

    save_file(sd, './OneDiffusion/transformer/diffusion_pytorch_model.safetensors', {'format': 'pt'})

With this, the "transformer_old" has the original weight, the "transformer" has the patched model. You're ready to run inference on the finetune.

pipeline = OneDiffusionPipeline.from_pretrained('./OneDiffusion').to(device=device, dtype=torch.bfloat16)

Prompt

[[text2image]] pny_style, ...

The text encoder understands handwritten prompts, comma separated keywords, it works without Molmo.

Training

The training script is similar to the SD3 dreambooth script. I reinvented what PEFT does otherwise, so it doesn't cost me a fortune to train a large model.

Based on previous work: Kolors, VAE, SD3 training script; Flux merge, text encoder and latent manipulation.

It's possible to train the model in int8.

The base model is similar to Kolors, but with more options, and newer architecture. I don't modify the T5 XL and VAE, they don't have to be smaller.