wiki/OpenVINO.md

# OpenVINO  
OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.  
* Compiles models for your hardware.  
* Supports **Linux and Windows**  
* Supports *CPU* / *iGPU* / *GPU* / *NPU*  
* Supports **AMD** GPUs on **Windows** with **FP16** support.  
* Supports **INTEL** dGPUs and iGPUs.  
* Supports **NVIDIA** GPUs.  
* Supports **CPUs** with **BF16** and **INT8** support.  
* Supports **Quantization** and **Model Compression**. 
* Supports multiple devices at the same time using **Hetero Device**.  

It is basically a TensorRT / Olive competitor that works with any hardware.  


# Installation
## Preparations
- Install the drivers for your device.
- Install `git` and `python`.
- Open CMD in a folder you want to install SD.Next.

Note: Do not mix OpenVINO with your old install. Treat OpenVINO as a seperate backend.  

## Using SD.Next with OpenVINO
Install SD.Next from Github:
```
git clone https://github.com/vladmandic/automatic
```

Then enter into the automatic folder:
```
cd automatic
```

Then start WebUI with this command:    

Windows:
```
.\webui.bat --use-openvino
```

Linux:

```
./webui.sh --use-openvino
```

# More Info

## Limitations
Same limitations with TensorRT / Olive applies here too.    
Compilation takes a few minutes and any change to Resolution / Batch Size / LoRa will trigger recompilation.    
Attention Slicing and HyperTile will not work.    
OpenVINO will lock you in the Diffusers backend.    

## Quantization  
Quantization enables 8 bit support without autocast.  
Enable `OpenVINO Quantize Models with NNCF` option in Compute Settings to use it.  

## Model Compression  
Enable `Compress Model weights with NNCF` option in Compute Settings to use it.  
Select a 4 bit mode from `OpenVINO compress mode for NNCF` to use 4 bit.  
For GPUs; select both CPU and GPU from the device selection if you want to use GPU with Model Compression.  

Note: VAE will be compressed to INT8 if you use a 4 bit mode.  

## Custom Devices
Use the `OpenVINO devices to use` option in `Compute Settings` if you want to specify a device.  
Selecting multiple devices will use multiple devices as a single `HETERO` device.  

Using `--device-id` cli argument with the WebUI will use a **GPU** with the specified **Device ID**.    
Using `--use-cpu openvino` cli argument with the WebUI will use the **CPU**.     

## Model Caching
OpenVINO will save compiled models to cache folder so you won't have to compile them again.    
`OpenVINO disable model caching` option in **Compute Settings** will disable caching.    
`Directory for OpenVINO cache` option in **System Paths** will set a new location for saving OpenVINO caches.