Hitokomoru Diffusion V2

A latent diffusion model that has been trained on Japanese Artist artwork, ヒトこもる/Hitokomoru. The current model is fine-tuned from waifu-diffusion-1-4 (wd-1-4-anime_e2.ckpt) with a learning rate of 2.0e-6, 15000 training steps and 4 batch sizes on the 257 artworks collected from Danbooru. This model supposed to be a continuation of hitokomoru-diffusion fine-tuned from Anything V3.0. Dataset has been preprocessed using Aspect Ratio Bucketing Tool so that it can be converted to latents and trained at non-square resolutions. Like other anime-style Stable Diffusion models, it also supports Danbooru tags to generate images.

e.g. 1girl, white hair, golden eyes, beautiful eyes, detail, flower meadow, cumulonimbus clouds, lighting, detailed sky, garden

Use it with the Automatic1111's Stable Diffusion Webui see: how-to-use
Use it with 🧨 diffusers

Model Details

Developed by: Linaqruf
Model type: Diffusion-based text-to-image generation model
Model type: This is a model that can be used to generate and modify images based on text prompts.
License: CreativeML Open RAIL++-M License
Finetuned from model: waifu-diffusion-v1-4-epoch-2

How to Use

Download the hitokomoru-v2.ckpt here, or download the safetensors version here.
This model is fine-tuned from waifu-diffusion-v1-4-epoch-2, which is also fine-tuned from stable-diffusion-2-1-base. So in order to run this model in Automatic1111's Stable Diffusion Webui, you need to put inference config .YAML file next to the model, you can find it here
You need to adjust your prompt using aesthetic tags, Based Official Waifu Diffusion 1.4 release notes, an ideal negative prompt to guide the model towards high aesthetic generations would look like:

worst quality, low quality, medium quality, deleted, lowres, comic, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, jpeg artifacts, signature, watermark, username, blurry

And, the following should also be prepended to prompts to get high aesthetic results:

masterpiece, best quality, high quality, absurdres

🧨 Diffusers

This model can be used just like any other Stable Diffusion model. For more information, please have a look at the Stable Diffusion. You can also export the model to ONNX, MPS and/or FLAX/JAX.

You should install dependencies below in order to running the pipeline

pip install diffusers transformers accelerate scipy safetensors

Running the pipeline (if you don't swap the scheduler it will run with the default DDIM, in this example we are swapping it to DPMSolverMultistepScheduler):

import torch
from torch import autocast
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler

model_id = "Linaqruf/hitokomoru-diffusion-v2"

# Use the DPMSolverMultistepScheduler (DPM-Solver++) scheduler here instead
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")

prompt = "masterpiece, best quality, high quality, 1girl, solo, sitting, confident expression, long blonde hair, blue eyes, formal dress"
negative_prompt = "worst quality, low quality, medium quality, deleted, lowres, comic, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, jpeg artifacts, signature, watermark, username, blurry"

with autocast("cuda"):
    image = pipe(prompt, 
                 negative_prompt=negative_prompt, 
                 width=512,
                 height=728,
                 guidance_scale=12,
                 num_inference_steps=50).images[0]
    
image.save("anime_girl.png")

Example

Here is some cherrypicked samples:

Prompt and settings for Example Images

masterpiece, best quality, high quality, 1girl, solo, sitting, confident expression, long blonde hair, blue eyes, formal dress, jewelry, make-up, luxury, close-up, face, upper body.

Negative prompt: worst quality, low quality, medium quality, deleted, lowres, comic, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, jpeg artifacts, signature, watermark, username, blurry

Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 994051800, Size: 512x768, Model hash: ea61e913a0, Model: hitokomoru-v2, Batch size: 2, Batch pos: 0, Denoising strength: 0.6, Clip skip: 2, ENSD: 31337, Hires upscale: 1.5, Hires steps: 20, Hires upscaler: Latent (nearest-exact)

License

This model is open access and available to all, with a CreativeML OpenRAIL-M license further specifying rights and usage. The CreativeML OpenRAIL License specifies:

You can't use the model to deliberately produce nor share illegal or harmful outputs or content
The authors claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license
You may re-distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL-M to all your users (please read the license entirely and carefully) Please read the full license here

Credit

ヒトこもる/Hitokomoru for Datasets