recoilme/sdxs

Stable Diffusion XS

Model Details

Stable Diffusion XS (SDXS) is a modified version stable diffusion for fast inference.

Usage

from diffusers import DiffusionPipeline
import torch

MODEL_PATH = "sdxs"
base = DiffusionPipeline.from_pretrained(
    MODEL_PATH,
    trust_remote_code=True,
    torch_dtype=torch.float16,
    variant="fp16",
    use_safetensors=True
).to("cuda")

prompt = "柴犬、カラフルアート"
negative_prompt = ""

def tokenize_prompt(tokenizer, prompt):
    text_inputs = tokenizer(
        prompt,
        padding="max_length",
        max_length=tokenizer.model_max_length,
        truncation=True,
        return_tensors="pt",
    )
    text_input_ids = text_inputs.input_ids
    return text_input_ids

def encode_prompt(text_encoders, tokenizers, prompt, hidden_size, model_max_length=77 ):
    prompt_embeds_list = []

    for i, text_encoder in enumerate(text_encoders):
        if text_encoder is not None:
            tokenizer = tokenizers[i]

            text_input_ids = tokenize_prompt(tokenizer, prompt)
            prompt_embeds = text_encoder(
                    text_input_ids.to(text_encoders[i].device), output_hidden_states=True, return_dict=False
                )
            pooled_prompt_embeds = prompt_embeds[0]
            prompt_embeds = prompt_embeds[-1][-2]
        else:
            prompt_embeds = torch.zeros((1, model_max_length, hidden_size))
            pooled_prompt_embeds = torch.zeros((1, hidden_size)) 

        # We are only ALWAYS interested in the pooled output of the final text encoder
        prompt_embeds = prompt_embeds.to("cuda")
        bs_embed, seq_len, _ = prompt_embeds.shape
        prompt_embeds = prompt_embeds.view(bs_embed, seq_len, -1)
        prompt_embeds_list.append(prompt_embeds)

    prompt_embeds = torch.concat(prompt_embeds_list, dim=-1)
    pooled_prompt_embeds = pooled_prompt_embeds.view(bs_embed, -1)
    return prompt_embeds, pooled_prompt_embeds


prompt_embeds, pooled_prompt_embeds = encode_prompt([None, base.text_encoder],[None, base.tokenizer], prompt, 768)
negative_prompt_embeds, negative_pooled_prompt_embeds = encode_prompt([None, base.text_encoder],[None, base.tokenizer], negative_prompt, 768)

#generator = [torch.Generator(device="cuda").manual_seed(i) for i in range(4)]

image = base(
    prompt_embeds=prompt_embeds,
    pooled_prompt_embeds=pooled_prompt_embeds,
    negative_prompt_embeds=negative_prompt_embeds,
    negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,
    num_inference_steps=20,
).images[0]

display(image)

Model Details

• Developed by: AiArtLab
• Model type: Diffusion-based text-to-image generative model
• Model Description: This model is a fine-tuned model based on colorfulxl_v27.
• License:

Uses

Direct Use

Research: possible research areas/tasks include:

• Generation of artworks and use in design and other artistic processes.
• Applications in educational or creative tools.
• Research on generative models.
• Safe deployment of models which have the potential to generate harmful content.
• Probing and understanding the limitations and biases of generative models.

Excluded uses are described below.

Out-of-Scope Use

The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

Limitations and Bias

Limitations

• The model does not achieve perfect photorealism
• The model cannot render legible text
• The model struggles with more difficult tasks which involve compositionality, such as rendering an image corresponding to “A red cube on top of a blue sphere”
• Faces and people in general may not be generated properly.
• The autoencoding part of the model is lossy.

Bias

While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases.

How to cite

@misc{SDXS, 
    url    = {[https://huggingface.co/recoilme/sdxs](https://huggingface.co/recoilme/sdxs)}, 
    title  = {Stable Diffusion XS}, 
    author = {recoilme}
}

Contact

• For questions and comments about the model, please join https://aiartlab.org/.
• For future announcements / information about AiArtLab AI models, research, and events, please follow Discord.
• For business and partnership inquiries, please contact https://t.me/recoilme