README.md

metadata

license: apache-2.0
datasets:
  - KBlueLeaf/danbooru2023-webp-4Mpixel
  - KBlueLeaf/danbooru2023-metadata-database
base_model:
  - black-forest-labs/FLUX.1-schnell
  - FA770/Sumeshi_Flux.1_S_v002E
pipeline_tag: text-to-image
tags:
  - anime
  - girls

Model Information

Note: This model is a Schnell-based model, but it requires CFG scale of 3 or higher (not guidance scale) with 20 steps or more. It needs to be used with clip_l_nigemono_flux1C. At this time, this model cannot be generated in WebUI Forge. Please use it with ComfyUI.

My English is terrible, so I use translation tools.

Description

Nigemono_Flux.1_Compact7.5B is an experimental anime model to verify the reduction of parameters in the Flux model. The number of parameters has been reduced from 12B (double blocks 19/ single blocks 38) to 7.5B (double blocks 12/ single blocks 24). You can use a negative prompt which works to some extent. The output is blurred and the style varies depending on the prompt, perhaps because the model has not been fully trained.

Usage

• Resolution: Like other Flux models
• (Distilled) Guidance Scale: (Distilled) Guidance Scale: 0 (Does not work due to Schnell-based model)
• CFG Scale: 6 ~ 9 (recommend 7; scale 1 does not generate decent outputs)
• Steps: 20 ~ 100 or and more (recommend 40)
• sampler: Euler
• scheduler: Simple, Beta

Prompt Format (from Kohaku-XL-Epsilon)

<1girl/1boy/1other/...>, <character>, <series>, <artists>, <general tags>, <quality tags>, <year tags>, <meta tags>, <rating tags>

Due to the small amount of training, the <character><series><artists> tags are almost non-functional. As training is focused on girl characters, it may not generate boy or other non-persons well. Since the dataset was created using hakubooru, the prompt format will be the same as the KohakuXL format. However, based on experiments, it is not strictly necessary to follow this format, as it interprets meaning to some extent even in natural language.

Special Tags

• Quality Tags: masterpiece, best quality, great quality, good quality, normal quality, low quality, worst quality
• Rating Tags: safe, sensitive, nsfw, explicit
• Date Tags: newest, recent, mid, early, old

Training

Dataset Preparation

I used hakubooru-based custom scripts.

• Exclude Tags: traditional_media, photo_(medium), scan, animated, animated_gif, lowres, non-web_source, variant_set, tall image, duplicate, pixel-perfect_duplicate
• Minimum Post ID: 1,000,000

Blocks Reduction

I reduced the parameters by gradually removing blocks and retraining, so as not to completely disrupt the model's generation capability. Removing too many blocks at once results in completely noisy output, requiring retraining from scratch.

• Training Hardware: A single RTX 4090
• Method: LoRA training and merging the results
• Training Script: sd-scripts
• Basic Settings:

accelerate launch --num_cpu_threads_per_process 4 flux_train_network.py --network_module networks.lora_flux --sdpa --gradient_checkpointing --cache_latents --cache_latents_to_disk --cache_text_encoder_outputs  --cache_text_encoder_outputs_to_disk --max_data_loader_n_workers 1 --save_model_as "safetensors" --mixed_precision "bf16" --fp8_base --save_precision "bf16" --full_bf16 --min_bucket_reso 384 --max_bucket_reso 1536 --seed 1 --max_train_epochs 1 --keep_tokens_separator "|||" --network_dim 32 --network_alpha 32 --unet_lr 1e-4 --train_batch_size 2 --gradient_accumulation_steps 3 --optimizer_type adamw8bit --lr_scheduler="constant_with_warmup" --lr_warmup_steps 500 --vae_batch_size 4 --cache_info --guidance_scale 1 --timestep_sampling shift --model_prediction_type raw --discrete_flow_shift 1.8 --loss_type l2 --highvram --network_args "in_dims=[16,16,16,0,16]" --network_train_unet_only --bucket_no_upscale

--Continued training from sumeshi flux.1s v002E--

1. remove double blocks 18 / single blocks 34,35,36,37 / guidance_in (sumeshi flux.1s has zero tensor)

2. 3,893images (res1024 bs2 /res512 bs8 acc2 warmup50 --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles 1 --discrete_flow_shift 1.8) 4epochs

3. merged into model

4. remove double blocks 16,17 / single blocks 32,33

5. 3,893images (res1024 bs2 /res512 bs8 acc1 warmup50 --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles 2 --discrete_flow_shift 1.8) 2epochs

6. merged into model

7. remove double blocks 14,15 / single blocks 28,29,30,31

8. 3,893images (res1024 bs2 /res512 bs8 acc2 warmup50 --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles 2 --discrete_flow_shift 1.8) 2epochs

9. merged into model

10. remove double blocks 12,13 / single blocks 24,25,26,27

11. 3,893images (res1024 bs2 /res512 bs8 acc2 warmup50 --lr_scheduler="cosine_with_restarts" --lr_scheduler_num_cycles 2 --discrete_flow_shift 1.8) 2epochs

12. merged into model

13. 3,893images (Full-finetuned res1024 bs1 acc1 afafactor --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False" lr5e-6 warmup100 max_grad_norm 0.0 discrete_flow_shift 2) 2epochs

14. 12,000images (res1024 bs2 acc5 warmup100 --discrete_flow_shift 2) 1epochs

15. 12,000images (res1024 bs2 acc3 warmup100 --discrete_flow_shift 2) 2epochs

16. merged into model

17. 12,000images (res1024 bs2 acc3 warmup100 --discrete_flow_shift 2.5) 3epochs

18. 12,000images (res1024 bs2 acc3 warmup100 --discrete_flow_shift 2.5) 3epochs

19. 12,000images (res512 bs8 acc1 warmup100 timestep_sampling flux_shift(fixed ver.) ) 12epochs

20. merged into model

21. 12,000images (res512 bs4 acc1 warmup100 timesteps_sampling sigmoid sigmoid_scale 0.75) 7epochs

22. 3,893images (res512 bs4 acc1 warmup100 timesteps_sampling sigmoid sigmoid_scale 0.6 --caption_dropout_rate 0.1) 16epochs

23. merged into model

24. 3,893images (res1024 bs2 / res512 bs4 acc2 warmup100 unet_lr1e-4 te_lr5e-5 timesteps_sampling sigmoid sigmoid_scale 0.6 --caption_dropout_rate 0.1) 6epochs

25. merged into model and CLIP_L

Resources (License)

• FLUX.1-schnell (Apache2.0)
• danbooru2023-webp-4Mpixel (MIT)
• danbooru2023-metadata-database (MIT)

Acknowledgements

• black-forest-labs: Thanks for publishing a great open source model.
• kohya-ss: Thanks for publishing the essential training scripts and for the quick updates.
• Kohaku-Blueleaf: Thanks for the extensive publication of the scripts for the dataset and the various training conditions.