Spaces:
Runtime error
Runtime error
Vision-CAIR
commited on
Commit
β’
db145b1
1
Parent(s):
db67f95
Delete MiniGPT4_Train.md
Browse files- MiniGPT4_Train.md +0 -41
MiniGPT4_Train.md
DELETED
@@ -1,41 +0,0 @@
|
|
1 |
-
## Training of MiniGPT-4
|
2 |
-
|
3 |
-
The training of MiniGPT-4 contains two alignment stages.
|
4 |
-
|
5 |
-
**1. First pretraining stage**
|
6 |
-
|
7 |
-
In the first pretrained stage, the model is trained using image-text pairs from Laion and CC datasets
|
8 |
-
to align the vision and language model. To download and prepare the datasets, please check
|
9 |
-
our [first stage dataset preparation instruction](dataset/README_1_STAGE.md).
|
10 |
-
After the first stage, the visual features are mapped and can be understood by the language
|
11 |
-
model.
|
12 |
-
To launch the first stage training, run the following command. In our experiments, we use 4 A100.
|
13 |
-
You can change the save path in the config file
|
14 |
-
[train_configs/minigpt4_stage1_pretrain.yaml](train_configs/minigpt4_stage1_pretrain.yaml)
|
15 |
-
|
16 |
-
```bash
|
17 |
-
torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/minigpt4_stage1_pretrain.yaml
|
18 |
-
```
|
19 |
-
|
20 |
-
A MiniGPT-4 checkpoint with only stage one training can be downloaded
|
21 |
-
[here (13B)](https://drive.google.com/file/d/1u9FRRBB3VovP1HxCAlpD9Lw4t4P6-Yq8/view?usp=share_link) or [here (7B)](https://drive.google.com/file/d/1HihQtCEXUyBM1i9DQbaK934wW3TZi-h5/view?usp=share_link).
|
22 |
-
Compared to the model after stage two, this checkpoint generate incomplete and repeated sentences frequently.
|
23 |
-
|
24 |
-
|
25 |
-
**2. Second finetuning stage**
|
26 |
-
|
27 |
-
In the second stage, we use a small high quality image-text pair dataset created by ourselves
|
28 |
-
and convert it to a conversation format to further align MiniGPT-4.
|
29 |
-
To download and prepare our second stage dataset, please check our
|
30 |
-
[second stage dataset preparation instruction](dataset/README_2_STAGE.md).
|
31 |
-
To launch the second stage alignment,
|
32 |
-
first specify the path to the checkpoint file trained in stage 1 in
|
33 |
-
[train_configs/minigpt4_stage1_pretrain.yaml](train_configs/minigpt4_stage2_finetune.yaml).
|
34 |
-
You can also specify the output path there.
|
35 |
-
Then, run the following command. In our experiments, we use 1 A100.
|
36 |
-
|
37 |
-
```bash
|
38 |
-
torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/minigpt4_stage2_finetune.yaml
|
39 |
-
```
|
40 |
-
|
41 |
-
After the second stage alignment, MiniGPT-4 is able to talk about the image coherently and user-friendly.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|