k4d3 commited on
Commit
abf5666
β€’
1 Parent(s): 7a97152

fix license add readme

Browse files
Files changed (2) hide show
  1. LICENSE +10 -17
  2. README.md +200 -0
LICENSE CHANGED
@@ -1,21 +1,14 @@
1
- MIT License
2
 
3
- Copyright (c) 2024 Balazs Horvath
4
 
5
- Permission is hereby granted, free of charge, to any person obtaining a copy
6
- of this software and associated documentation files (the "Software"), to deal
7
- in the Software without restriction, including without limitation the rights
8
- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
- copies of the Software, and to permit persons to whom the Software is
10
- furnished to do so, subject to the following conditions:
11
 
12
- The above copyright notice and this permission notice shall be included in all
13
- copies or substantial portions of the Software.
 
14
 
15
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
- SOFTWARE.
 
1
+ # DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
2
 
3
+ Version 2, December 2024
4
 
5
+ Copyright (C) 2024 Balazs Horvath
 
 
 
 
 
6
 
7
+ Everyone is permitted to copy and distribute verbatim or modified
8
+ copies of this license document, and changing it is allowed as long
9
+ as the name is changed.
10
 
11
+ DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
12
+ TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
13
+
14
+ 1. You just DO WHAT THE FUCK YOU WANT TO.
 
 
 
README.md CHANGED
@@ -1,3 +1,203 @@
1
  ---
2
  license: wtfpl
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: wtfpl
3
  ---
4
+
5
+ <!-- markdownlint-disable MD029 -->
6
+
7
+ # AI Image Processing Toolkit
8
+
9
+ ---
10
+
11
+ A collection of specialized scripts for AI image processing, dataset preparation, and model training workflows.
12
+
13
+ ## πŸ› οΈ Scripts Overview
14
+
15
+ ---
16
+
17
+ ### WDV3 (Waifu Diffusion V3 Tagger)
18
+
19
+ An image tagging script using the WD V3 tagger models. Supports multiple model architectures (ViT, SwinV2, ConvNext) and can process both single images and directories recursively.
20
+
21
+ #### Features
22
+
23
+ - Multiple model architecture support
24
+ - Batch processing capabilities
25
+ - Adjustable confidence thresholds
26
+ - CUDA acceleration with FP16 support
27
+ - JXL image format support
28
+
29
+ ### Training Functions (train_functions.zsh)
30
+
31
+ A set of ZSH functions for managing AI model training workflows:
32
+
33
+ - Script execution management
34
+ - Training variable setup
35
+ - Git repository state tracking
36
+ - Output directory management
37
+ - Automatic cleanup of empty outputs
38
+
39
+ ### Git Wrapper (git-wrapper.zsh)
40
+
41
+ Enhanced Git functionality for dataset management:
42
+
43
+ - Automatic submodule handling
44
+ - LFS integration for JXL files
45
+ - Dataset-specific Git attributes management
46
+
47
+ ### Check4sig (check4sig.zsh)
48
+
49
+ Dataset caption file watermark detection utility:
50
+
51
+ - Scans .caption files for watermark-related text
52
+ - Batch processing support
53
+ - Interactive editing with nvim
54
+ - Recursive directory scanning
55
+
56
+ ### Gallery-dl Wrapper (gallery-dl.zsh)
57
+
58
+ Directory-aware wrapper for gallery-dl:
59
+
60
+ - Automatically changes to ~/datasets directory
61
+ - Maintains consistent download locations
62
+ - Preserves original command functionality
63
+
64
+ ### JoyCaption (joy)
65
+
66
+ Advanced image captioning system using CLIP and LLM:
67
+
68
+ - Multiple caption styles (descriptive, training prompts, art critic, etc.)
69
+ - Custom image adapters
70
+ - Tag-based caption generation
71
+ - Batch processing support
72
+
73
+ ### PNG to MP4 Converter (png2mp4)
74
+
75
+ Training progress visualization tool:
76
+
77
+ - Converts PNG sequences to MP4
78
+ - Customizable frame rates and durations
79
+ - Step counter overlay support
80
+ - Multiple sample handling
81
+
82
+ ### XY Plot Generator (xyplot)
83
+
84
+ Image comparison grid generator:
85
+
86
+ - Supports multiple image formats
87
+ - Customizable grid layouts
88
+ - Optional row/column labels
89
+ - Automatic image padding and alignment
90
+
91
+ ### Caption Concatenator (concat_captions)
92
+
93
+ Utility for combining multiple caption files:
94
+
95
+ - Merges .caption and .tags files
96
+ - Maintains original image associations
97
+ - Batch processing support
98
+ - Error handling for missing files
99
+
100
+ <!-- ⚠️ TODO: add more scripts -->
101
+
102
+ ## πŸš€ Installation
103
+
104
+ ---
105
+
106
+ 1. Clone the repository: (optional)
107
+
108
+ ```bash
109
+ git clone https://huggingface.co/k4d3/toolkit
110
+ ```
111
+
112
+ 2. Add the repository to your PATH: (optional)
113
+
114
+ ```bash
115
+ export PATH="$PATH:~/path/to/toolkit"
116
+ ```
117
+
118
+ 3. Add the `.zshrc` to your shell: (optional and you will need to make changes to it)
119
+
120
+ ```bash
121
+ source ~/path/to/toolkit/.zshrc
122
+ nano ~/.zshrc
123
+ ```
124
+
125
+ ## πŸ“ Requirements
126
+
127
+ ---
128
+
129
+ - miniconda with the environment set up for training with sd-scripts, timm, etc
130
+ - ZSH shell (optional)
131
+ - CUDA-capable GPU (recommended)
132
+ - Required Python packages:
133
+ - torch
134
+ - transformers
135
+ - pillow
136
+ - pillow-jxl
137
+ - opencv-python
138
+ - numpy
139
+ - and a lot more
140
+
141
+ ## πŸ”§ Usage
142
+
143
+ ---
144
+
145
+ Each script can be used independently or as part of a workflow. Here are some common usage examples:
146
+
147
+ <!-- ⚠️ TODO: add more usage examples -->
148
+
149
+ ### JoyCaption
150
+
151
+ ```bash
152
+ joy --feed-from-tags=10 --custom_prompt="Write a very long descriptive caption for this image in a formal tone. Do not mention feelings and emotions evoked by the image." .
153
+ ```
154
+
155
+ ### png2mp4
156
+
157
+ ```bash
158
+ png2mp4 --repeat 16
159
+ ```
160
+
161
+ ### inject_to_txt
162
+
163
+ ```bash
164
+ inject_to_txt 1_honovy "honovy"
165
+ ```
166
+
167
+ ### replace_comma_with_keep_tags_txt
168
+
169
+ ```bash
170
+ replace_comma_with_keep_tags_txt 1 1_honovy
171
+ ```
172
+
173
+ ## πŸ“¦ Directory Structure
174
+
175
+ ---
176
+
177
+ ```bash
178
+ ~/
179
+ β”œβ”€β”€ datasets/
180
+ β”œβ”€β”€ output_dir/
181
+ β”œβ”€β”€ models/
182
+ β”œβ”€β”€ toolkit/
183
+ ```
184
+
185
+ ## πŸ“„ License
186
+
187
+ ---
188
+
189
+ [WTFPL](http://www.wtfpl.net/) - Do what the fuck you want with it.
190
+
191
+ The included data and models are copyrighted by their respective owners with their own licenses.
192
+
193
+ ## 🀝 Contributing
194
+
195
+ ---
196
+
197
+ Contributions are welcome! For major changes, please open an issue first to discuss what you would like to change.
198
+
199
+ ## πŸ“š Documentation
200
+
201
+ ---
202
+
203
+ If the documentation of a script is missing, ask a language model about it.