ghoskno/Color-Canny-Controlnet-model

Color-Canny CantrolNet

These are ControlNet checkpoints trained on runwayml/stable-diffusion-v1-5, using fused color and canny edge as conditioning.

You can find some example images in the following.

Examples

Color examples

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

negative prompt: text, bad anatomy, blurry, (low quality, blurry)

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

negative prompt: text, bad anatomy, blurry, (low quality, blurry)

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the grass

negative prompt: text, bad anatomy, blurry, (low quality, blurry)

Brightness examples

This model also can be used to control image brightness. The following images are generated with different brightness conditioning image and controlnet strength(0.5 ~ 0.7).

Limitations and Bias

• No strict control by input color
• Sometimes generate image with confusion When color description in prompt

Training

Dataset We train this model on laion-art dataset with 2.6m images, the processed dataset can be found in ghoskno/laion-art-en-colorcanny.

Training Details

• Hardware: Google Cloud TPUv4-8 VM

• Optimizer: AdamW

• Train Batch Size: 4 x 4 = 16

• Learning rate: 0.00001 constant

• Gradient Accumulation Steps: 4

• Resolution: 512

• Train Steps: 36000