README_EN.md

<p align="left">
    中文</a>&nbsp ｜ &nbsp<a href="README_EN.md">English</a>&nbsp
</p>
<br>

<div align="center">
<h1>
  360智脑
</h1>
</div>
<div align="center">
    🤗 <a href="">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp
    🤖 <a href="">ModelScope</a>&nbsp&nbsp ｜ &nbsp&nbsp
    💬 <a href="">WeChat (微信)</a>&nbsp&nbsp
</div>
<br>
<p align="center">
 Welcome to visit 360Zhinao official website<a href="https://ai.360.com"> https://ai.360.com</a> to experience more powerful functions.
</p>

# Models Introduction
 🎉🎉🎉We opensource our 360Zhinao series，The following models are open sourced：
 - **360Zhinao-7B-Base**
 - **360Zhinao-7B-Chat-4K**
 - **360Zhinao-7B-Chat-32K**
 - **360Zhinao-7B-Chat-360K**

The characteristics of the 360Zhinao open-source project are：
- **Base Model:** Leveraging a high-quality corpus of 3.4 trillion Tokens, primarily in Chinese, English, and code, we achieved competitive performance in relevant benchmark evaluations of the same scale.
- **Chat Model:** Powerful chat capabilities and three different sequence lengths of 4k, 32k, and 360k. It is understood that 360k (about 500,000 words) is the longest sequcence length among the current chinese open source Large language model. 

# News and Updates
- 2024.04.10 We release **360Zhinao-7B** 1.0 version,  include the base model and three chat model with sequence length of 4k, 32k, 360k. 

# Table of contents
- [Download URL](#Download-URL)
- [Model Evaluation](#Model-Evaluation)
- [Quickstart](#Quickstart)
- [Model Inference](#Model-Inference)
- [Model Finetune](#Model-Finetune)
- [License](#License)


# Download URL
See the following table for this release and download links:
| | Zhinao-Base | Zhinao-Chat | Zhinao-Chat(Int8) | Zhinao-Chat(Int4) |
|-|-|-|-|-|
| 1.8B  |  <a href="">🤖</a>  <a href="">🤗</a> |  <a href="">🤖</a>  <a href="">🤗</a>  |  <a href="">🤖</a>  <a href="">🤗</a> |  <a href="">🤖</a>  <a href="">🤗</a>  |
| 7B  |  <a href="">🤖</a>  <a href="">🤗</a>  | <a href="">🤖</a>  <a href="">🤗</a>  |  <a href="">🤖</a>  <a href="">🤗</a> |  <a href="">🤖</a>  <a href="">🤗</a>  |

# Model Evaluation
We validate the performance of our model on the mainstream OpenCompass evaluation datasets, including C-Eval, AGIEval, MMLU, CMMLU, HellaSwag, MATH, GSM8K, HumanEval, MBPP, BBH, LAMBADA. The competencies examined include natural language understanding, knowledge, mathematical computation and reasoning, code generation, logical reasoning, etc.

## Base Models
| Model | C-Eval | AGIEval | MMLU | CMMLU | HellaSwag | MATH | GSM8K | HumanEval | MBPP | BBH | LAMBADA |
| - | - | - | - | - | - | - | - | - | - | - | - |
| Phi-1.5-1.3B | 27.8 | 23.4 | 44.3 | 26 | 57.1 | 2.6 | 32.5 | 25 | 33 | 29.6 | 54.6 |
| Qwen-1.8B | 53.3 | 36.5 | 46.4 | 51.9 | 58.7 | 2.4 | 10.2 | 7.3 | 14 | 22.6 | 54.3 |
| Qwen-1.5-1.8B | 59.48 | 38.76 | 47.14 | 57.08 | 56.02 | 9.66 | 34.87 | 23.17 | 17.6 | 27.02 | 56.49 |
| Baichuan2-7B-Base | 56.3 | 34.6 | 54.7 | 57 | 67 | 5.4 | 24.6 | 17.7 | 24 | 41.8 | 73.3 |
| ChatGLM3-6B-Base | 67 | 47.4 | 62.8 | 66.5 | 76.5 | 19.2 | 61 | 44.5 | 57.2 | 66.2 | 77.1 |
| DeepSeek-7B-Base | 45 | 24 | 49.3 | 46.8 | 73.4 | 4.2 | 18.3 | 25 | 36.4 | 42.8 | 72.6 |
| InternLM2-7B | 65.7 | 50.2 | 65.5 | 66.2 | 79.6 | 19.9 | 70.6 | 41.5 | 42.4 | 64.4 | 72.1 |
| InternLM-7B | 53.4 | 36.9 | 51 | 51.8 | 70.6 | 6.3 | 31.2 | 13.4 | 14 | 37 | 67 |
| LLaMA-2-7B | 32.5 | 21.8 | 46.8 | 31.8 | 74 | 3.3 | 16.7 | 12.8 | 14.8 | 38.2 | 73.3 |
| LLaMA-7B | 27.3 | 20.6 | 35.6 | 26.8 | 74.3 | 2.9 | 10 | 12.8 | 16.8 | 33.5 | 73.3 |
| Mistral-7B-v0.1 | 47.4 | 32.8 | 64.1 | 44.7 | 78.9 | 11.3 | 47.5 | 27.4 | 38.6 | 56.7 | 75 |
| MPT-7B | 23.5 | 21.3 | 27.5 | 25.9 | 75 | 2.9 | 9.1 | 17.1 | 22.8 | 35.6 | 70 |
| Qwen-7B | 63.4 | 45.3 | 59.7 | 62.5 | 75 | 13.3 | 54.1 | 27.4 | 31.4 | 45.2 | 67.5 |
| XVERSE-7B | 61.1 | 39 | 58.4 | 60.8 | 73.7 | 2.2 | 11.7 | 4.9 | 10.2 | 31 | 24 |
| Yi-6B | 73 | 44.3 | 64 | 73.5 | 73.1 | 6.3 | 39.9 | 15.2 | 23.6 | 44.9 | 68 |
| Zhinao-1.8B-Base | 49.78 | 31.87 | 50.05 | 52.58 | 57.31 | 4.82 | 15.01 | 14.02 | 19.4 | 29.76 | 69.77 |
| 360Zhinao-7B-Base | 74.11 | 49.49 | 67.44 | 72.38 | 83.05 | 16.38 | 53.83 | 35.98 | 42.4 | 43.95 | 78.59 |

The above results, the official [Opencompass](https://rank.opencompass.org.cn/leaderboard-llm) can query or can emersion.

# Quickstart
Simple examples to illustrate how to use 360Zhinao-7B-Base and 360Zhinao-7B-Chat quickly using 🤖 ModelScope and 🤗 Transformers

## Dependency Installation
- python 3.8 and above
- pytorch 2.0 and above
- transformers 4.37.2 and above
- CUDA 11.4 and above are recommended.

```shell
pip install -r requirements.txt 
```
We recommend installing Flash-Attention (which currently supports flash attention 2) to increase your performance and reduce your memory footprint. (flash-attention is optional and will work without installation)

>flash-attn >= 2.3.6
```shell
FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn==2.3.6
```

## 🤗 Transformers
### Demonstration of Base Model Inference

This code demonstrates fast inference with 360Zhinao-7B-Base models using transformers.
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt')
inputs = inputs.to(model.device)

pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config)
print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
```
### Demonstration of Chat Model Inference

This code demo uses transformers to quickly use the 360Zhinao-7B-Chat-4K model for inference.
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Chat-4K"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

messages = []
#round-1
messages.append({"role": "user", "content": "介绍一下刘德华"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

#round-2
messages.append({"role": "user", "content": "他有什么代表作？"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)
```

## 🤖 ModelScope
### Demonstration of Base Model Inference

This code demonstrates using ModelScope to quickly use the 360Zhinao-7B-Base model for inference.

```python
from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt')
inputs = inputs.to(model.device)

pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config)
print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
```

### Demonstration of Base Model Inference

This code demonstrates using ModelScope to quickly use the 360Zhinao-7B-Chat-4K model for inference.

```python
from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Chat-4K"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

messages = []
#round-1
messages.append({"role": "user", "content": "介绍一下刘德华"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

#round-2
messages.append({"role": "user", "content": "他有什么代表作？"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)
```

## CLI Demo
Use terminal interaction for a fast experience
```shell
python cli_demo.py
```
<p align="center">
    <img src="assets/cli_demo.gif" width="600" />
<p>

## Web Demo
You can also use web interaction for a quick experience
```shell
streamlit run web_demo.py
```
<p align="center">
    <img src="assets/web_demo.gif" width="600" />
<p>

## API Demo
Start command
```shell
python openai_api.py
```

Request parameter
```shell
curl --location --request POST 'http://localhost:8360/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data-raw '{
    "max_new_tokens": 200,
    "do_sample": true,
    "top_k": 0,
    "top_p": 0.8,
    "temperature": 1.0,
    "repetition_penalty": 1.0,
    "messages": [
        {
            "role": "user",
            "content": "你叫什么名字？"
        }
    ]
}'
```

# Model Inference
## Quantization
We provide quantization schemes based on AutoGPTQ and open source the Int4 quantization models. The quantization model has little effect loss, but it can significantly reduce the video memory occupation and improve the inference speed.

The BF16, Int8, and Int4 models are tested on the benchmarks, and the results are as follows:

| Quantization | MMLU | CEval (val) | GSM8K | Humaneval |
|-|-|-|-|-|
| 360Zhinao-7B-Chat-4K (BF16) |-|-|-|-|
| 360Zhinao-7B-Chat-4K (Int8) |-|-|-|-|
| 360Zhinao-7B-Chat-4K (Int4) |-|-|-|-|

## Deployment
### vLLM Installation
If you want to deploy and accelerate inference, we recommend using `vLLM==0.3.3`。

If you are using **CUDA 12.1 and PyTorch 2.1**, you can install vLLM directly with the following command.
```shell
pip install vllm==0.3.3
```

Otherwise, please refer to the official vLLM [Installation Instructions](https://docs.vllm.ai/en/latest/getting_started/installation.html)。

>Once the installation is complete, you will need to do the following
1. Copy the vllm/zhinao.py file to the vllm/model_executor/models directory corresponding to your env environment.
2. Then add a line to vllm/model_executor/models/\_\_init\_\_.py

    ```shell
    "ZhinaoForCausalLM": ("zhinao", "ZhinaoForCausalLM"),
    ```

### vLLM Service Start

Starting the service
```shell
python -m vllm.entrypoints.openai.api_server \
    --served-model-name 360Zhinao-7B-Chat-4K \
    --model qihoo360/360Zhinao-7B-Chat-4k \
    --trust-remote-code \
    --tensor-parallel-size 1
    --max-model-len 18000 \
    --host 0.0.0.0 \
    --port 8360
```

Use curl to request the service
```shell
curl http://localhost:8360/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "360Zhinao-7B-Chat-4K",
    "max_tokens": 200,
    "top_k": 0,
    "top_p": 0.8,
    "temperature": 1.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"}
    ],
    "stop": [
        "<eod>",
        "<|im_end|>",
        "<|im_start|>"
    ]
}'
```
Use python to request the service
```python
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="360Zhinao-7B-Chat-4K",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"},
    ],
    stop=[
        "<eod>",
        "<|im_end|>",
        "<|im_start|>"
    ],
    presence_penalty=0.0,
    frequency_penalty=0.0
)
print("Chat response:", chat_response)
```

> Notice: If you need to enable repetition penalty, recommended to use *presence_penalty* and *frequency_penalty* parameters.

> 

# Model Finetune
## Training data

Training Data: data/training_data_sample.json. The sample data is 10,000 pieces sampled from [multiturn_chat_0.8M](https://huggingface.co/datasets/BelleGroup/multiturn_chat_0.8M) and format converted.

Data Format:
```json
[
  {
    "id": 1,
    "conversations": [
        {
            "from": "system",
            "value": "You are a helpful assistant."
        },
        {
            "from": "user",
            "value": "您好啊"
        },
        {
            "from": "assistant",
            "value": "你好！我今天能为您做些什么？有什么问题或需要帮助吗? 我在这里为您提供服务。"
        }
    ]
  }
]
```
## Fine-tuning scripts
```shell
set -x

HOSTFILE=hostfile
DS_CONFIG=./finetune/ds_config_zero2.json

# PARAMS
LR=5e-6
EPOCHS=3
MAX_LEN=4096
BATCH_SIZE=4
NUM_NODES=1
NUM_GPUS=8
MASTER_PORT=29500

IS_CONCAT=False # Whether to concatenate to maximum length (MAX_LEN)

DATA_PATH="./data/training_data_sample.json"
MODEL_PATH="qihoo360/360Zhinao-7B-Base"
OUTPUT_DIR="./outputs/"

deepspeed --hostfile ${HOSTFILE} \
        --master_port ${MASTER_PORT} \
        --num_nodes ${NUM_NODES} \
        --num_gpus ${NUM_GPUS} \
        finetune.py \
        --report_to "tensorboard" \
        --data_path ${DATA_PATH} \
        --model_name_or_path ${MODEL_PATH} \
        --output_dir ${OUTPUT_DIR} \
        --model_max_length ${MAX_LEN} \
        --num_train_epochs ${EPOCHS} \
        --per_device_train_batch_size ${BATCH_SIZE} \
        --gradient_accumulation_steps 1 \
        --save_strategy steps \
        --save_steps 200 \
        --learning_rate ${LR} \
        --lr_scheduler_type cosine \
        --adam_beta1 0.9 \
        --adam_beta2 0.95 \
        --adam_epsilon 1e-8 \
        --max_grad_norm 1.0 \
        --weight_decay 0.1 \
        --warmup_ratio 0.01 \
        --gradient_checkpointing True \
        --bf16 True \
        --tf32 True \
        --deepspeed ${DS_CONFIG} \
        --is_concat ${IS_CONCAT} \
        --logging_steps 1 \
        --log_on_each_node False
```
```shell
bash finetune/ds_finetune.sh
```
- By configuring the **hostfile**, single-machine and multi-machine training can be realized.
- By configuring **ds_config**, realize zero2 and zero3 training
- By configuring the **fp16**、**bf16** realize mixed precision training, bf16 is recommended to be consistent with the pre-trained model.
- By configuring **is_concat**, Whether the training data is concatenated or not is controlled. When the magnitude of the training data is large, the training efficiency can be improved by concatenation.

# License

The source code of this warehouse follows the open source license Apache 2.0.

The 360 ​Zhinao open source model supports commercial use. If you need to use this model and its derivative models for commercial purposes, please contact us via email (g-zhinao-opensource@360.cn) to apply. For the specific license agreement, please see [《360 Zhinao Open Source Model License》](./360智脑开源模型许可证.txt).