Aptronym commited on
Commit
65c2d07
1 Parent(s): 659d2cd

Upload 3 files

Browse files
Controlnet-SDXL/README.md ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - text_to_image
5
+ - diffusers
6
+ - controlnet
7
+ - controlnet-canny-sdxl-1.0
8
+ ---
9
+
10
+ # ***Drawing like Midjourney! Come on!***
11
+ ![images](./masonry.webp)
12
+
13
+ # Controlnet-Canny-Sdxl-1.0
14
+
15
+ <!-- Provide a quick summary of what the model is/does. -->
16
+
17
+ Hello, I am very happy to announce the controlnet-canny-sdxl-1.0 model, **a very powerful controlnet that can generate high resolution images visually comparable with midjourney**.
18
+ The model was trained with large amount of high quality data(over 10000000 images), with carefully filtered and captioned(powerful vllm model). Besides, useful tricks are applied
19
+ during the training, including date augmentation, mutiple loss and multi resolution. With only 1 stage training, the performance outperforms the other opensource canny models
20
+ ([diffusers/controlnet-canny-sdxl-1.0], [TheMistoAI/MistoLine]). I release it and hope to advance the application of stable diffusion models. Canny is one of the most important
21
+ ControlNet series models and can be applied to many jobs associated with drawing and designing.
22
+
23
+ ## Model Details
24
+
25
+
26
+ ### Model Description
27
+
28
+ <!-- Provide a longer summary of what this model is. -->
29
+
30
+ - **Developed by:** xinsir
31
+ - **Model type:** ControlNet_SDXL
32
+ - **License:** apache-2.0
33
+ - **Finetuned from model [optional]:** stabilityai/stable-diffusion-xl-base-1.0
34
+
35
+ ### Model Sources [optional]
36
+
37
+ <!-- Provide the basic links for the model. -->
38
+
39
+ - **Paper [optional]:** https://arxiv.org/abs/2302.05543
40
+
41
+ ## Uses
42
+
43
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
44
+
45
+ ### Examples
46
+
47
+ prompt: A closeup of two day of the dead models, looking to the side, large flowered headdress, full dia de Los muertoe make up, lush red lips, butterflies,
48
+ flowers, pastel colors, looking to the side, jungle, birds, color harmony , extremely detailed, intricate, ornate, motion, stunning, beautiful, unique, soft lighting
49
+
50
+ ![images_0)](./000031_scribble_concat.webp)
51
+
52
+ prompt: ghost with a plague doctor mask in a venice carnaval hyper realistic
53
+ ![images_1)](./000028_scribble_concat.webp)
54
+
55
+ prompt: A picture surrounded by blue stars and gold stars, glowing, dark navy blue and gray tones, distributed in light silver and gold, playful, festive atmosphere, pure fabric, chalk, FHD 8K
56
+ ![images_2)](./000016_scribble_concat.webp)
57
+
58
+ prompt: Delicious vegetarian pizza with champignon mushrooms, tomatoes, mozzarella, peppers and black olives, isolated on white background , transparent isolated white background , top down view, studio photo, transparent png, Clean sharp focus. High end retouching. Food magazine photography. Award winning photography. Advertising photography. Commercial photography
59
+ ![images_3)](./000010_scribble_concat.webp)
60
+
61
+ prompt: a blonde woman in a wedding dress in a maple forest in summer with a flower crown laurel. Watercolor painting in the style of John William Waterhouse. Romanticism. Ethereal light.
62
+ ![images_4)](./000006_scribble_concat.webp)
63
+
64
+ ### Examples Anime(Note that you need to change the base model to CounterfeitXL, others remains the same)
65
+
66
+ ![images_5)](./000013_scribble_concat.webp)
67
+
68
+ ![images_6)](./000034_scribble_concat.webp)
69
+
70
+ ![images_7)](./000059_scribble_concat.webp)
71
+
72
+ ![images_8)](./000078_scribble_concat.webp)
73
+
74
+ ![images_9)](./000097_scribble_concat.webp)
75
+
76
+
77
+ ## How to Get Started with the Model
78
+
79
+ Use the code below to get started with the model.
80
+
81
+ ```python
82
+ from diffusers import ControlNetModel, StableDiffusionXLControlNetPipeline, AutoencoderKL
83
+ from diffusers import DDIMScheduler, EulerAncestralDiscreteScheduler
84
+ from PIL import Image
85
+ import torch
86
+ import numpy as np
87
+ import cv2
88
+
89
+ def HWC3(x):
90
+ assert x.dtype == np.uint8
91
+ if x.ndim == 2:
92
+ x = x[:, :, None]
93
+ assert x.ndim == 3
94
+ H, W, C = x.shape
95
+ assert C == 1 or C == 3 or C == 4
96
+ if C == 3:
97
+ return x
98
+ if C == 1:
99
+ return np.concatenate([x, x, x], axis=2)
100
+ if C == 4:
101
+ color = x[:, :, 0:3].astype(np.float32)
102
+ alpha = x[:, :, 3:4].astype(np.float32) / 255.0
103
+ y = color * alpha + 255.0 * (1.0 - alpha)
104
+ y = y.clip(0, 255).astype(np.uint8)
105
+ return y
106
+
107
+ controlnet_conditioning_scale = 1.0
108
+ prompt = "your prompt, the longer the better, you can describe it as detail as possible"
109
+ negative_prompt = 'longbody, lowres, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality'
110
+
111
+
112
+
113
+ eulera_scheduler = EulerAncestralDiscreteScheduler.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", subfolder="scheduler")
114
+
115
+
116
+ controlnet = ControlNetModel.from_pretrained(
117
+ "xinsir/controlnet-canny-sdxl-1.0",
118
+ torch_dtype=torch.float16
119
+ )
120
+
121
+ # when test with other base model, you need to change the vae also.
122
+ vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
123
+
124
+ pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
125
+ "stabilityai/stable-diffusion-xl-base-1.0",
126
+ controlnet=controlnet,
127
+ vae=vae,
128
+ safety_checker=None,
129
+ torch_dtype=torch.float16,
130
+ scheduler=eulera_scheduler,
131
+ )
132
+
133
+ # need to resize the image resolution to 1024 * 1024 or same bucket resolution to get the best performance
134
+
135
+ controlnet_img = cv2.imread("your image path")
136
+ height, width, _ = controlnet_img.shape
137
+ ratio = np.sqrt(1024. * 1024. / (width * height))
138
+ new_width, new_height = int(width * ratio), int(height * ratio)
139
+ controlnet_img = cv2.resize(controlnet_img, (new_width, new_height))
140
+
141
+ controlnet_img = cv2.Canny(controlnet_img, 100, 200)
142
+ controlnet_img = HWC3(controlnet_img)
143
+ controlnet_img = Image.fromarray(controlnet_img)
144
+
145
+ images = pipe(
146
+ prompt,
147
+ negative_prompt=negative_prompt,
148
+ image=controlnet_img,
149
+ controlnet_conditioning_scale=controlnet_conditioning_scale,
150
+ width=new_width,
151
+ height=new_height,
152
+ num_inference_steps=30,
153
+ ).images
154
+
155
+ images[0].save(f"your image save path, png format is usually better than jpg or webp in terms of image quality but got much bigger")
156
+ ```
157
+
158
+
159
+ ## Evaluation Metric
160
+ 1 Laion Aesthetic Score [https://laion.ai/blog/laion-aesthetics/]
161
+ 2 PerceptualSimilarity [https://github.com/richzhang/PerceptualSimilarity]
162
+
163
+
164
+ ## Evaluation Data
165
+ The test data is randomly sample from midjourney upscale images with prompts, as the purpose of the project is to letting people draw images like midjourney. midjourney’s users include a large number of professional designers,
166
+ and the upscale image tend to have more beauty score and prompt consistency, it is suitable to use it as the test set to judge the ability of controlnet. We select 300 prompt-image pairs randomly and generate 4 images per prompt,
167
+ totally 1200 images generated. We caculate the Laion Aesthetic Score to measure the beauty and the PerceptualSimilarity to measure the control ability, we find the quality of images have a good consistency with the meric values.
168
+ We compare our methods with other SOTA huggingface models and list the result below. We are the models that have highest aesthectic score, and can generate visually appealing images if you prompt it properly.
169
+
170
+ ## Quantitative Result
171
+ | metric | xinsir/controlnet-canny-sdxl-1.0 | diffusers/controlnet-canny-sdxl-1.0 | TheMistoAI/MistoLine |
172
+ |-------|-------|-------|-------|
173
+ | laion_aesthetic | **6.03** | 5.93 | 5.82 |
174
+ | perceptual similarity | **0.4200** | 0.5053 | 0.5387 |
175
+
176
+ laion_aesthetic(the higher the better)
177
+ perceptual similarity(the lower the better)
178
+ Note: The values are calculate when saved in webp format, if you save in png format the aesthetic values will increase 0.1-0.3 but the relative relation remains unchanged.
179
+
180
+
181
+ ## Training Details
182
+
183
+ The model is trained using high quality data, only 1 stage training, the resolution setting is the same with sdxl-base, 1024*1024. We use random threshold to generate canny images like lvming zhang, It is essential to find proper hyerparameters
184
+ to realize data augmentation, too easy or too hard will hurt the model performance. Besides, we use random mask to random mask out a random percentage of canny images to force the model to learn more semantic meaning between the prompt and the line.
185
+ We use over 10000000 images, which are annotated carefully, cogvlm is proved to be a powerful image caption model[https://github.com/THUDM/CogVLM?tab=readme-ov-file]. For comic images, it is recommened to use waifu tagger to generate special tags
186
+ [https://huggingface.co/spaces/SmilingWolf/wd-tagger]. More than 64 A100s are used to train the model and the real batch size is 2560 when used accumulate_grad_batches.
187
+
188
+
189
+ ### Training Data
190
+
191
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
192
+
193
+ The data consists of many sources, including midjourney, laion 5B, danbooru, and so on. The data is carefully filtered and annotated.
194
+
195
+
196
+ ### Conclusion
197
+
198
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
199
+
200
+ In our evaluation, the model got better aesthetic score in real images compared with stabilityai/stable-diffusion-xl-base-1.0, and comparable performance in cartoon sytle images.
201
+ The model is better in control ability when test with perception similarity due to more strong data augmentation and more training steps.
202
+ Besides, the model has lower rate to generate abnormal images which tend to include some abnormal human structure.
Controlnet-SDXL/config.json ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "ControlNetModel",
3
+ "_diffusers_version": "0.20.0.dev0",
4
+ "act_fn": "silu",
5
+ "addition_embed_type": "text_time",
6
+ "addition_embed_type_num_heads": 64,
7
+ "addition_time_embed_dim": 256,
8
+ "attention_head_dim": [
9
+ 5,
10
+ 10,
11
+ 20
12
+ ],
13
+ "block_out_channels": [
14
+ 320,
15
+ 640,
16
+ 1280
17
+ ],
18
+ "class_embed_type": null,
19
+ "conditioning_channels": 3,
20
+ "conditioning_embedding_out_channels": [
21
+ 16,
22
+ 32,
23
+ 96,
24
+ 256
25
+ ],
26
+ "controlnet_conditioning_channel_order": "rgb",
27
+ "cross_attention_dim": 2048,
28
+ "down_block_types": [
29
+ "DownBlock2D",
30
+ "CrossAttnDownBlock2D",
31
+ "CrossAttnDownBlock2D"
32
+ ],
33
+ "downsample_padding": 1,
34
+ "encoder_hid_dim": null,
35
+ "encoder_hid_dim_type": null,
36
+ "flip_sin_to_cos": true,
37
+ "freq_shift": 0,
38
+ "global_pool_conditions": false,
39
+ "in_channels": 4,
40
+ "layers_per_block": 2,
41
+ "mid_block_scale_factor": 1,
42
+ "norm_eps": 1e-05,
43
+ "norm_num_groups": 32,
44
+ "num_attention_heads": null,
45
+ "num_class_embeds": null,
46
+ "only_cross_attention": false,
47
+ "projection_class_embeddings_input_dim": 2816,
48
+ "resnet_time_scale_shift": "default",
49
+ "transformer_layers_per_block": [
50
+ 1,
51
+ 2,
52
+ 10
53
+ ],
54
+ "upcast_attention": null,
55
+ "use_linear_projection": true
56
+ }
Controlnet-SDXL/controlnet-canny-sdxl-1.0.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b3e4ac47bc814019d50dc842f579301440deb6d8f09ee1b91a30f527ace1b852
3
+ size 2502139104