File size: 11,252 Bytes
0d72233 0f39c5e 1a68347 0f39c5e 3fcef88 0f39c5e 384f77f 0f39c5e 384f77f 0f39c5e 609c6e3 0f39c5e 609c6e3 0f39c5e 384f77f 0f39c5e 384f77f 0f39c5e 384f77f 0f39c5e 384f77f 0f39c5e 3004e7b 0f39c5e 52fc5f6 0f39c5e 384f77f 0f39c5e 384f77f 0f39c5e 52fc5f6 384f77f 52fc5f6 384f77f 52fc5f6 384f77f 52fc5f6 384f77f 52fc5f6 384f77f 52fc5f6 0f39c5e 609c6e3 0f39c5e 3fcef88 0f39c5e 0d72233 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 |
---
language:
- en
- zh
---
# AnomalyGPT: Detecting Industrial Anomalies using Large Vision-Language Models
![License](https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-red.svg)
<p align="left">
🌐 <a href="https://anomalygpt.github.io" target="_blank">Project Page</a> • 🤗 <a href="https://huggingface.co/spaces/FantasticGNU/AnomalyGPT" target="_blank">Online Demo</a> • 📃 <a href="https://arxiv.org/abs/2308.15366" target="_blank">Paper</a> • 🤖 <a href="https://huggingface.co/FantasticGNU/AnomalyGPT" target="_blank">Model</a> • 📹 <a href="https://www.youtube.com/watch?v=lcxBfy0YnNA" target="_blank">Video</a>
</p>
Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, Jinqiao Wang
****
<span id='all_catelogue'/>
## Catalogue:
* <a href='#introduction'>1. Introduction</a>
* <a href='#environment'>2. Running AnomalyGPT Demo</a>
* <a href='#install_environment'>2.1 Environment Installation</a>
* <a href='#download_imagebind_model'>2.2 Prepare ImageBind Checkpoint</a>
* <a href='#download_vicuna_model'>2.3 Prepare Vicuna Checkpoint</a>
* <a href='#download_anomalygpt'>2.4 Prepare Delta Weights of AnomalyGPT</a>
* <a href='#running_demo'>2.5 Deploying Demo</a>
* <a href='#train_anomalygpt'>3. Train Your Own AnomalyGPT</a>
* <a href='#data_preparation'>3.1 Data Preparation</a>
* <a href='#training_configurations'>3.2 Training Configurations</a>
* <a href='#model_training'>3.3 Training AnoamlyGPT</a>
* <a href='#examples'>4. Examples</a>
<!-- * <a href='#results'>5. Results</a> -->
* <a href='#license'>License</a>
* <a href='#citation'>Citation</a>
* <a href='#acknowledgments'>Acknowledgments</a>
****
<span id='introduction'/>
### 1. Introduction: <a href='#all_catelogue'>[Back to Top]</a>
!()[./images/compare.png]
**AnomalyGPT** is the first Large Vision-Language Model (LVLM) based Industrial Anomaly Detection (IAD) method that can detect anomalies in industrial images without the need for manually specified thresholds. Existing IAD methods can only provide anomaly scores and need manually threshold setting, while existing LVLMs cannot detect anomalies in the image. AnomalyGPT can not only indicate the presence and location of anomaly but also provide information about the image.
!()[./images/AnomalyGPT.png]
We leverage a pre-trained image encoder and a Large Language Model (LLM) to align IAD images and their corresponding textual descriptions via simulated anomaly data. We employ a lightweight, visual-textual feature-matching-based image decoder to obtain localization result, and design a prompt learner to provide fine-grained semantic to LLM and fine-tune the LVLM using prompt embeddings. Our method can also detect anomalies for previously unseen items with few normal sample provided.
****
<span id='environment'/>
### 2. Running AnomalyGPT Demo <a href='#all_catelogue'>[Back to Top]</a>
<span id='install_environment'/>
#### 2.1 Environment Installation
Clone the repository locally:
```
git clone https://github.com/CASIA-IVA-Lab/AnomalyGPT.git
```
Install the required packages:
```
pip install -r requirements.txt
```
<span id='download_imagebind_model'/>
#### 2.2 Prepare ImageBind Checkpoint:
You can download the pre-trained ImageBind model using [this link](https://dl.fbaipublicfiles.com/imagebind/imagebind_huge.pth). After downloading, put the downloaded file (imagebind_huge.pth) in [[./pretrained_ckpt/imagebind_ckpt/]](./pretrained_ckpt/imagebind_ckpt/) directory.
<span id='download_vicuna_model'/>
#### 2.3 Prepare Vicuna Checkpoint:
To prepare the pre-trained Vicuna model, please follow the instructions provided [[here]](./pretrained_ckpt#1-prepare-vicuna-checkpoint).
<span id='download_anomalygpt'/>
#### 2.4 Prepare Delta Weights of AnomalyGPT:
We use the pre-trained parameters from [PandaGPT](https://github.com/yxuansu/PandaGPT) to initialize our model. You can get the weights of PandaGPT trained with different strategies in the table below. In our experiments and online demo, we use the Vicuna-7B and `openllmplayground/pandagpt_7b_max_len_1024` due to the limitation of computation resource. Better results are expected if switching to Vicuna-13B.
| **Base Language Model** | **Maximum Sequence Length** | **Huggingface Delta Weights Address** |
| :---------------------: | :-------------------------: | :----------------------------------------------------------: |
| Vicuna-7B (version 0) | 512 | [openllmplayground/pandagpt_7b_max_len_512](https://huggingface.co/openllmplayground/pandagpt_7b_max_len_512) |
| Vicuna-7B (version 0) | 1024 | [openllmplayground/pandagpt_7b_max_len_1024](https://huggingface.co/openllmplayground/pandagpt_7b_max_len_1024) |
| Vicuna-13B (version 0) | 256 | [openllmplayground/pandagpt_13b_max_len_256](https://huggingface.co/openllmplayground/pandagpt_13b_max_len_256) |
| Vicuna-13B (version 0) | 400 | [openllmplayground/pandagpt_13b_max_len_400](https://huggingface.co/openllmplayground/pandagpt_13b_max_len_400) |
Please put the downloaded 7B/13B delta weights file (pytorch_model.pt) in the [./pretrained_ckpt/pandagpt_ckpt/7b/](./pretrained_ckpt/pandagpt_ckpt/7b/) or [./pretrained_ckpt/pandagpt_ckpt/13b/](./pretrained_ckpt/pandagpt_ckpt/13b/) directory.
After that, you can download AnomalyGPT weights from the table below.
| Setup and Datasets | Weights Address |
| :---------------------------------------------------------: | :-------------------------------: |
| Unsupervised on MVTec-AD | [AnomalyGPT/train_mvtec](https://huggingface.co/FantasticGNU/AnomalyGPT/blob/main/train_mvtec/pytorch_model.pt) |
| Unsupervised on VisA | [AnomalyGPT/train_visa](https://huggingface.co/FantasticGNU/AnomalyGPT/blob/main/train_visa/pytorch_model.pt) |
| Supervised on MVTec-AD, VisA, MVTec-LOCO-AD and CrackForest | [AnomalyGPT/train_supervised](https://huggingface.co/FantasticGNU/AnomalyGPT/blob/main/train_supervised/pytorch_model.pt) |
After downloading, put the AnomalyGPT weights in the [./code/ckpt/](./code/ckpt/) directory.
In our [online demo](https://huggingface.co/spaces/FantasticGNU/AnomalyGPT), we use the supervised setting as our default model to attain an enhanced user experience. You can also try other weights locally.
<span id='running_demo'/>
#### 2.5. Deploying Demo
Upon completion of previous steps, you can run the demo locally as
```bash
cd ./code/
python web_demo.py
```
****
<span id='train_anomalygpt'/>
### 3. Train Your Own AnomalyGPT <a href='#all_catelogue'>[Back to Top]</a>
**Prerequisites:** Before training the model, making sure the environment is properly installed and the checkpoints of ImageBind, Vicuna and PandaGPT are downloaded.
<span id='data_preparation'/>
#### 3.1 Data Preparation:
You can download MVTec-AD dataset from [[this link]](https://www.mvtec.com/company/research/datasets/mvtec-ad/downloads) and VisA from [[this link]](https://github.com/amazon-science/spot-diff). You can also download pre-training data of PandaGPT from [[here]](https://huggingface.co/datasets/openllmplayground/pandagpt_visual_instruction_dataset/tree/main). After downloading, put the data in the [[./data]](./data/) directory.
The directory of [[./data]](./data/) should look like:
```
data
|---pandagpt4_visual_instruction_data.json
|---images
|-----|-- ...
|---mvtec_anomaly_detection
|-----|-- bottle
|-----|-----|----- ground_truth
|-----|-----|----- test
|-----|-----|----- train
|-----|-- capsules
|-----|-- ...
|----VisA
|-----|-- split_csv
|-----|-----|--- 1cls.csv
|-----|-----|--- ...
|-----|-- candle
|-----|-----|--- Data
|-----|-----|-----|----- Images
|-----|-----|-----|--------|------ Anomaly
|-----|-----|-----|--------|------ Normal
|-----|-----|-----|----- Masks
|-----|-----|-----|--------|------ Anomaly
|-----|-----|--- image_anno.csv
|-----|-- capsules
|-----|-----|----- ...
```
<span id='training_configurations'/>
#### 3.2 Training Configurations
The table below show the training hyperparameters used in our experiments. The hyperparameters are selected based on the constrain of our computational resources, i.e. 2 x RTX3090 GPUs.
| **Base Language Model** | **Epoch Number** | **Batch Size** | **Learning Rate** | **Maximum Length** |
| :---------------------: | :--------------: | :------------: | :---------------: | :----------------: |
| Vicuna-7B | 50 | 16 | 1e-3 | 1024 |
<span id='model_training'/>
#### 3.3 Training AnomalyGPT
To train AnomalyGPT on MVTec-AD dataset, please run the following commands:
```yaml
cd ./code
bash ./scripts/train_mvtec.sh
```
The key arguments of the training script are as follows:
* `--data_path`: The data path for the json file `pandagpt4_visual_instruction_data.json`.
* `--image_root_path`: The root path for training images of PandaGPT.
* `--imagebind_ckpt_path`: The path of ImageBind checkpoint.
* `--vicuna_ckpt_path`: The directory that saves the pre-trained Vicuna checkpoints.
* `--max_tgt_len`: The maximum sequence length of training instances.
* `--save_path`: The directory which saves the trained delta weights. This directory will be automatically created.
* `--log_path`: The directory which saves the log. This directory will be automatically created.
Note that the epoch number can be set in the `epochs` argument at [./code/config/openllama_peft.yaml](./code/config/openllama_peft.yaml) file and the learning rate can be set in [./code/dsconfig/openllama_peft_stage_1.json](./code/dsconfig/openllama_peft_stage_1.json)
****
<span id='examples'/>
### 4. Examples
![](./images/demo_1.png)
<h4 align='center'>An image of concrete with crack. </h4>
****
![](./images/demo_5.png)
<h4 align='center'>A crack capsule. </h4>
****
![](./images/demo_8.png)
<h4 align='center'>An image of a cut hazelnut. </h4>
****
![](./images/demo_7.png)
<h4 align='center'>A damaged bottle. </h4>
****
![](./images/demo_2.png)
<h4 align='center'>A photo of normal carpet. </h4>
****
![](./images/demo_4.png)
<h4 align='center'>A photo of a piece of wood with defect. </h4>
****
![](./images/demo_3.png)
<h4 align='center'>A piece of normal fabric. </h4>
****
<span id='license'/>
### License
AnomalyGPT is licensed under the [CC BY-NC-SA 4.0 license](./LICENSE).
****
<span id='citation'/>
### Citation:
If you found AnomalyGPT useful in your research or applications, please kindly cite using the following BibTeX:
```
@article{gu2023anomalyagpt,
title={AnomalyGPT: Detecting Industrial Anomalies using Large Vision-Language Models},
author={Gu, Zhaopeng and Zhu, Bingke and Zhu, Guibo and Chen, Yingying and Tang, Ming and Wang, Jinqiao},
journal={arXiv preprint arXiv:2308.15366},
year={2023}
}
```
****
<span id='acknowledgments'/>
### Acknowledgments:
We borrow some codes and the pre-trained weights from [PandaGPT](https://github.com/yxuansu/PandaGPT). Thanks for their wonderful work! |