fix license add readme
Browse files
LICENSE
CHANGED
@@ -1,21 +1,14 @@
|
|
1 |
-
|
2 |
|
3 |
-
|
4 |
|
5 |
-
|
6 |
-
of this software and associated documentation files (the "Software"), to deal
|
7 |
-
in the Software without restriction, including without limitation the rights
|
8 |
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9 |
-
copies of the Software, and to permit persons to whom the Software is
|
10 |
-
furnished to do so, subject to the following conditions:
|
11 |
|
12 |
-
|
13 |
-
copies
|
|
|
14 |
|
15 |
-
|
16 |
-
|
17 |
-
|
18 |
-
|
19 |
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20 |
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21 |
-
SOFTWARE.
|
|
|
1 |
+
# DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
|
2 |
|
3 |
+
Version 2, December 2024
|
4 |
|
5 |
+
Copyright (C) 2024 Balazs Horvath
|
|
|
|
|
|
|
|
|
|
|
6 |
|
7 |
+
Everyone is permitted to copy and distribute verbatim or modified
|
8 |
+
copies of this license document, and changing it is allowed as long
|
9 |
+
as the name is changed.
|
10 |
|
11 |
+
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
|
12 |
+
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
|
13 |
+
|
14 |
+
1. You just DO WHAT THE FUCK YOU WANT TO.
|
|
|
|
|
|
README.md
CHANGED
@@ -1,3 +1,203 @@
|
|
1 |
---
|
2 |
license: wtfpl
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
license: wtfpl
|
3 |
---
|
4 |
+
|
5 |
+
<!-- markdownlint-disable MD029 -->
|
6 |
+
|
7 |
+
# AI Image Processing Toolkit
|
8 |
+
|
9 |
+
---
|
10 |
+
|
11 |
+
A collection of specialized scripts for AI image processing, dataset preparation, and model training workflows.
|
12 |
+
|
13 |
+
## π οΈ Scripts Overview
|
14 |
+
|
15 |
+
---
|
16 |
+
|
17 |
+
### WDV3 (Waifu Diffusion V3 Tagger)
|
18 |
+
|
19 |
+
An image tagging script using the WD V3 tagger models. Supports multiple model architectures (ViT, SwinV2, ConvNext) and can process both single images and directories recursively.
|
20 |
+
|
21 |
+
#### Features
|
22 |
+
|
23 |
+
- Multiple model architecture support
|
24 |
+
- Batch processing capabilities
|
25 |
+
- Adjustable confidence thresholds
|
26 |
+
- CUDA acceleration with FP16 support
|
27 |
+
- JXL image format support
|
28 |
+
|
29 |
+
### Training Functions (train_functions.zsh)
|
30 |
+
|
31 |
+
A set of ZSH functions for managing AI model training workflows:
|
32 |
+
|
33 |
+
- Script execution management
|
34 |
+
- Training variable setup
|
35 |
+
- Git repository state tracking
|
36 |
+
- Output directory management
|
37 |
+
- Automatic cleanup of empty outputs
|
38 |
+
|
39 |
+
### Git Wrapper (git-wrapper.zsh)
|
40 |
+
|
41 |
+
Enhanced Git functionality for dataset management:
|
42 |
+
|
43 |
+
- Automatic submodule handling
|
44 |
+
- LFS integration for JXL files
|
45 |
+
- Dataset-specific Git attributes management
|
46 |
+
|
47 |
+
### Check4sig (check4sig.zsh)
|
48 |
+
|
49 |
+
Dataset caption file watermark detection utility:
|
50 |
+
|
51 |
+
- Scans .caption files for watermark-related text
|
52 |
+
- Batch processing support
|
53 |
+
- Interactive editing with nvim
|
54 |
+
- Recursive directory scanning
|
55 |
+
|
56 |
+
### Gallery-dl Wrapper (gallery-dl.zsh)
|
57 |
+
|
58 |
+
Directory-aware wrapper for gallery-dl:
|
59 |
+
|
60 |
+
- Automatically changes to ~/datasets directory
|
61 |
+
- Maintains consistent download locations
|
62 |
+
- Preserves original command functionality
|
63 |
+
|
64 |
+
### JoyCaption (joy)
|
65 |
+
|
66 |
+
Advanced image captioning system using CLIP and LLM:
|
67 |
+
|
68 |
+
- Multiple caption styles (descriptive, training prompts, art critic, etc.)
|
69 |
+
- Custom image adapters
|
70 |
+
- Tag-based caption generation
|
71 |
+
- Batch processing support
|
72 |
+
|
73 |
+
### PNG to MP4 Converter (png2mp4)
|
74 |
+
|
75 |
+
Training progress visualization tool:
|
76 |
+
|
77 |
+
- Converts PNG sequences to MP4
|
78 |
+
- Customizable frame rates and durations
|
79 |
+
- Step counter overlay support
|
80 |
+
- Multiple sample handling
|
81 |
+
|
82 |
+
### XY Plot Generator (xyplot)
|
83 |
+
|
84 |
+
Image comparison grid generator:
|
85 |
+
|
86 |
+
- Supports multiple image formats
|
87 |
+
- Customizable grid layouts
|
88 |
+
- Optional row/column labels
|
89 |
+
- Automatic image padding and alignment
|
90 |
+
|
91 |
+
### Caption Concatenator (concat_captions)
|
92 |
+
|
93 |
+
Utility for combining multiple caption files:
|
94 |
+
|
95 |
+
- Merges .caption and .tags files
|
96 |
+
- Maintains original image associations
|
97 |
+
- Batch processing support
|
98 |
+
- Error handling for missing files
|
99 |
+
|
100 |
+
<!-- β οΈ TODO: add more scripts -->
|
101 |
+
|
102 |
+
## π Installation
|
103 |
+
|
104 |
+
---
|
105 |
+
|
106 |
+
1. Clone the repository: (optional)
|
107 |
+
|
108 |
+
```bash
|
109 |
+
git clone https://huggingface.co/k4d3/toolkit
|
110 |
+
```
|
111 |
+
|
112 |
+
2. Add the repository to your PATH: (optional)
|
113 |
+
|
114 |
+
```bash
|
115 |
+
export PATH="$PATH:~/path/to/toolkit"
|
116 |
+
```
|
117 |
+
|
118 |
+
3. Add the `.zshrc` to your shell: (optional and you will need to make changes to it)
|
119 |
+
|
120 |
+
```bash
|
121 |
+
source ~/path/to/toolkit/.zshrc
|
122 |
+
nano ~/.zshrc
|
123 |
+
```
|
124 |
+
|
125 |
+
## π Requirements
|
126 |
+
|
127 |
+
---
|
128 |
+
|
129 |
+
- miniconda with the environment set up for training with sd-scripts, timm, etc
|
130 |
+
- ZSH shell (optional)
|
131 |
+
- CUDA-capable GPU (recommended)
|
132 |
+
- Required Python packages:
|
133 |
+
- torch
|
134 |
+
- transformers
|
135 |
+
- pillow
|
136 |
+
- pillow-jxl
|
137 |
+
- opencv-python
|
138 |
+
- numpy
|
139 |
+
- and a lot more
|
140 |
+
|
141 |
+
## π§ Usage
|
142 |
+
|
143 |
+
---
|
144 |
+
|
145 |
+
Each script can be used independently or as part of a workflow. Here are some common usage examples:
|
146 |
+
|
147 |
+
<!-- β οΈ TODO: add more usage examples -->
|
148 |
+
|
149 |
+
### JoyCaption
|
150 |
+
|
151 |
+
```bash
|
152 |
+
joy --feed-from-tags=10 --custom_prompt="Write a very long descriptive caption for this image in a formal tone. Do not mention feelings and emotions evoked by the image." .
|
153 |
+
```
|
154 |
+
|
155 |
+
### png2mp4
|
156 |
+
|
157 |
+
```bash
|
158 |
+
png2mp4 --repeat 16
|
159 |
+
```
|
160 |
+
|
161 |
+
### inject_to_txt
|
162 |
+
|
163 |
+
```bash
|
164 |
+
inject_to_txt 1_honovy "honovy"
|
165 |
+
```
|
166 |
+
|
167 |
+
### replace_comma_with_keep_tags_txt
|
168 |
+
|
169 |
+
```bash
|
170 |
+
replace_comma_with_keep_tags_txt 1 1_honovy
|
171 |
+
```
|
172 |
+
|
173 |
+
## π¦ Directory Structure
|
174 |
+
|
175 |
+
---
|
176 |
+
|
177 |
+
```bash
|
178 |
+
~/
|
179 |
+
βββ datasets/
|
180 |
+
βββ output_dir/
|
181 |
+
βββ models/
|
182 |
+
βββ toolkit/
|
183 |
+
```
|
184 |
+
|
185 |
+
## π License
|
186 |
+
|
187 |
+
---
|
188 |
+
|
189 |
+
[WTFPL](http://www.wtfpl.net/) - Do what the fuck you want with it.
|
190 |
+
|
191 |
+
The included data and models are copyrighted by their respective owners with their own licenses.
|
192 |
+
|
193 |
+
## π€ Contributing
|
194 |
+
|
195 |
+
---
|
196 |
+
|
197 |
+
Contributions are welcome! For major changes, please open an issue first to discuss what you would like to change.
|
198 |
+
|
199 |
+
## π Documentation
|
200 |
+
|
201 |
+
---
|
202 |
+
|
203 |
+
If the documentation of a script is missing, ask a language model about it.
|