qihoo360/360Zhinao2-7B-Chat-32K-Int4

360Zhinao2 (360智脑)

🤗 HuggingFace   |    🤖 ModelScope   ｜    💬 WeChat (微信)   ｜    GitHub  

Feel free to visit 360Zhinao's official website https://ai.360.com for more experience.

Introduction

🎉🎉🎉 We released the 360Zhinao2 model series:

• 360Zhinao2-7B-Base
• 360Zhinao2-7B-Chat-4K
• 360Zhinao2-7B-Chat-32K
• 360Zhinao2-7B-Chat-360K

Notable features of our 360Zhinao models are:

• Base Model: Using popular two-stage training method, In the first stage we totally train 10T tokens with a cosine learning rate schedule. In the second stage we increase the proportion of high-quality data and totally train 100B tokens, with the learning rate decaying directly to 0. The total training data for 360Zhinao2-7B amounts to 10.1T tokens.
• Chat Models: Powerful chat capabilities and three context lengths of 4K, 32K and 360K.

News and Updates

• [2024.11.18] 🔥🔥🔥We release 360Zhinao2-7B, providing access to both the Base model and Chat models with text lengths of 4K, 32K, and 360K.
• [2024.05.23] We released two models, 360Zhinao-search and 360Zhinao-1.8B-Reranking, which ranked first respectively in the Retrieval and Reranking tasks of C-MTEB Leaderboard .
• [2024.05.20] We extended llama3 and released llama3-8B-360Zhinao-360k-Instruct🤗
• [2024.04.12] We released 360Zhinao-7B v1.0, including the base model and three chat models with context lengths 4K, 32K and 360K. Technical report is on arXiv.

Table of contents

• Download URL
• Model Evaluation
• Quickstart
• Model Inference
• Model Finetune
• License

Download URL

Size	Model	BF16	Int4
7B	360Zhinao2-7B-Base	🤖 🤗
7B	360Zhinao2-7B-Chat-4K	🤖 🤗	🤖 🤗
7B	360Zhinao2-7B-Chat-32K	🤖 🤗	🤖 🤗
7B	360Zhinao2-7B-Chat-360K	🤖 🤗	🤖 🤗

Model Evaluation

Base Model

We used the open-source tool OpenCompass to evaluate the model and compared it with open-source models under 10B from the past six months. The 360Zhinao2-7B model is competive. The 360Zhinao2-7B model performs well on Chinese benchmarks such as CEval, C3 and LCSTS. The average socres of Chinese benchmarks is No 1. It also ranks No 1 on Math which is a challenging competition math dataset. The 360Zhinao2-7B model has advantages in Chinese benchmark and challenging competition math.

Type	Datasets	language	glm4-9b	Qwen2.5-7B	internlm2.5-7b	Yi1.5-9B	gemma2-9b	Llama3.1-8B	360Zhinao2-7B
Exam	ceval	zh	75.83	81.41	77.71	73.51	56.36	51.67	83.04
	mmlu	en	75.5	75.5	71.55	71.43	72.22	66.75	67.84
	cmmlu	zh	74.24	81.79	78.77	74.2	58.89	52.49	73.8
	ARC-c	en	94.92	80	85.08	87.46	77.63	80.68	87.12
	ARC-e	en	98.41	84.83	95.24	94.53	78.84	89.77	92.77
Language	WiC	en	51.57	52.82	50.78	50.63	50.47	50	49.84
	WSC	en	68.27	68.27	69.23	66.35	68.27	67.31	65.38
Knowledge	BoolQ	en	81.8	83.88	89.51	84.46	85.6	82.2	88.29
	commonsense_qa	en	71.17	73.22	68.55	71.58	68.47	71.25	69.78
Understanding	C3	zh	91.51	92	93.04	85.86	81.64	83.51	93.26
	race-middle	en	91.99	91.02	92.06	91.16	88.09	81.69	90.46
	race-high	en	90.71	87.91	90.08	88.34	82.08	78.73	86.74
	lcsts	zh	18.29	15.82	15.96	16.49	10.62	17.29	18.61
	eprstmt-dev	zh	91.88	86.88	91.25	91.88	48.12	83.12	90
	lambada	en	71.67	71.14	69.98	70.64	75.43	74.23	72.56
Reasoning	hellaswag	en	70.25	72.76	70.38	71.55	66.83	74.65	71.49
	siqa	en	81.73	72.52	78.97	76.2	58.96	64.18	77.12
	bbh	en	73.68	54.63	59.43	67.86	68.45	59.9	46.54
Code	humaneval	en	69.51	75	60.37	26.22	5.49	27.44	60.98
	mbpp	en	60	60	43.6	56.8	51.2	42.6	54
Math	math	en	26.86	38	27.14	27.06	28.52	15.32	38.34
	gsm8k	en	78.54	79.76	52.54	71.11	73.09	56.25	75.51
Overall	avg_zh		70.35	71.58	71.35	68.39	51.13	57.62	71.74
	avg_all		73.11	71.78	69.60	68.88	61.60	62.32	70.61

Chat Model

Post-Training Data

360's proprietary general fine-tuning dataset consists of 500,000 samples. This dataset considers various skills and 360's vertical business data, with the following generation methods:

1. Data Diversity: Layered sampling based on 360's proprietary tagging system, considering domain, intent, difficulty, and length to ensure instruction diversity.
2. Data Quality: Using open-source data and proprietary preference-ordered data to train 360gpt-pro-rm (reward benchmark score of 92.59). This model is used for sample screening to filter out low-quality responses.
3. Complex Instruction Evolution: Optimizing complex instructions through evolutionary methods to enhance instruction-following capabilities.

Training Methods

1. Full Parameter Fine-Tuning

   Based on the general post-training data, full parameter fine-tuning is performed, and the optimal checkpoint is selected as sft-base.

2. LoRA Off-policy DPO

   Using human-labeled preference pairs, LoRA fine-tuning is applied to the sft-base model, followed by LoRA DPO training.

3. Iterative On-Policy DPO

   The sft-base model samples multiple answers on training prompts, and 360gpt-pro-rm scores them. The highest and lowest scoring answers form each pair, and such pairs are used for DPO training. This on-policy DPO method is iteratively used to improve model performance.

4. Model Merging

   Automatic evaluations on 360's white-box evaluation set v4 revealed that different models excel in different skills. A model merging scheme was considered to result in the final chat model.

Model Performance

We evaluated the 360Zhinao2-7B-Chat-4k model on IFEval, MT-bench, CF-bench which are all popular benchmark to evaluate chat model ability. The evaluation results show that 360Zhinao2-7B is highly competitive. Especially the IFEval (prompt strict) score get the highest score among open-source 7B models. only a little worse than GLM4-9B. The detailed results are shown in the table below:

Model	MT-bench	IFEval(strict prompt)	CFBench(CSR,ISR,PSR)
Qwen2.5-7B-Instruct	8.07	0.556	0.81	0.46	0.57
Yi-9B-16k-Chat	7.44	0.455	0.75	0.4	0.52
GLM4-9B-Chat	8.08	0.634	0.82	0.48	0.61
InternLM2.5-7B-Chat	7.39	0.540	0.78	0.4	0.54
360Zhinao2-7B-Chat-4k	7.86	0.577	0.8	0.44	0.57

Long Context Fine-Tuning

Similar to the method used during the open-sourcing of 360Zhinao1, we expanded the RoPE base to 1,000,000 and 50,000,000, sequentially concatenated SFT data of mixed long and short texts to 32k and 360k. By combining techniques like gradient checkpointing, ZeRO3 offload, and ring attention, we fine-tuned models to achieve 32k and 360k long context capabilities. These models ranked in the top tier across various 32k benchmarks.

Model	LooGLE-Long Dependency QA	Loong-Set 1 (32k)	LongBench-Chat (32k cutoff)	LEval-96 question subset	LEval-closed ended
GLM4-9B-Chat	0.36	55.24	6.60	0.49	63.96
InternLM2.5-7B-Chat	0.39	42.76	5.70	0.44	61.64
360Zhinao2-7B-Chat-32k	0.33	39.37	5.44	0.44	60.48
360Zhinao2-7B-Chat-360k	0.34	32.16	5.08	0.38	53.00
Yi-1.5-9B-Chat	0.25	32.77	4.70	0.37	56.22

Quickstart

We provide simple examples illustrating the use of 360Zhinao2-7B-Base and 360Zhinao2-7B-Chat on 🤖ModelScope and 🤗Transformers.

Dependency Installation

• python >= 3.8
• pytorch >= 2.0
• transformers >= 4.37.2
• CUDA >= 11.4

pip install -r requirements.txt 

Optionally, we recommend installing Flash-Attention 2 to improve performance and reduce memory footprint.

flash-attn >= 2.3.6

FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn==2.3.6

🤗 Transformers

Demonstration of Base Model Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao2-7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt')
inputs = inputs.to(model.device)

pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config)
print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Demonstration of Chat Model Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao2-7B-Chat-4K"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

messages = []
#round-1
messages.append({"role": "user", "content": "介绍一下刘德华"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

#round-2
messages.append({"role": "user", "content": "他有什么代表作？"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

🤖 ModelScope

Demonstration of Base Model Inference

from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao2-7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt')
inputs = inputs.to(model.device)

pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config)
print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Demonstration of Chat Model Inference

from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao2-7B-Chat-4K"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

messages = []
#round-1
messages.append({"role": "user", "content": "介绍一下刘德华"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

#round-2
messages.append({"role": "user", "content": "他有什么代表作？"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

CLI Demo

Use terminal for command-line interface:

python cli_demo.py

Note: for Mac users, device = 'mps' is not supported yet.

Web Demo

streamlit run web_demo.py

API Demo

Launch api:

python openai_api.py

Then request with parameters:

curl 'http://localhost:8360/v1/chat/completions' \
-H 'Content-Type: application/json' \
-d '{
    "max_new_tokens": 200,
    "do_sample": true,
    "top_k": 0,
    "top_p": 0.8,
    "temperature": 1.0,
    "repetition_penalty": 1.0,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"}
    ]
}'

Model Inference

Quantization

We provide quantization schemes based on AutoGPTQ and release the Int4 quantization models.

Deployment

vLLM Installation

We recommend using vLLM==0.3.3.

If you are using CUDA 12.1 and PyTorch 2.1, you can install vLLM directly with:

pip install vllm==0.3.3

Otherwise, please refer to the official vLLM Installation Instructions.

After installation, perform the following steps:

1. Copy vllm/zhinao.py into vllm/model_executor/models in your vllm installation directory (in python/conda env).

2. Copy vllm/serving_chat.py into vllm/entrypoints/openai in your vllm installation directory.

3. Then add a line in vllm/model_executor/models/__init__.py

   "ZhinaoForCausalLM": ("zhinao", "ZhinaoForCausalLM"),
   

vLLM Service Start

Start the service:

python -m vllm.entrypoints.openai.api_server \
    --served-model-name 360Zhinao2-7B-Chat-4K \
    --model qihoo360/360Zhinao2-7B-Chat-4K \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --host 0.0.0.0 \
    --port 8360

Use curl to request the service:

curl http://localhost:8360/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "360Zhinao2-7B-Chat-4K",
    "max_tokens": 200,
    "top_k": -1,
    "top_p": 0.8,
    "temperature": 1.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"}
    ],
    "stop": [
        "<eod>",
        "<|im_end|>",
        "<|im_start|>"
    ]
}'

Use python to request the service:

from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8360/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="360Zhinao2-7B-Chat-4K",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"},
    ],
    stop=[
        "<eod>",
        "<|im_end|>",
        "<|im_start|>"
    ],
    presence_penalty=0.0,
    frequency_penalty=0.0
)
print("Chat response:", chat_response)

If you need to enable repetition penalty, we recommend setting presence_penalty and frequency_penalty instead of repetition_penalty.

Model Finetune

Training data

Training Data: data/training_data_sample.json. This example data has 10,000 rows sampled from multiturn_chat_0.8M with converted format.

Data Format:

[
  {
    "id": 1,
    "conversations": [
        {
            "from": "system",
            "value": "You are a helpful assistant."
        },
        {
            "from": "user",
            "value": "您好啊"
        },
        {
            "from": "assistant",
            "value": "你好！我今天能为您做些什么？有什么问题或需要帮助吗? 我在这里为您提供服务。"
        }
    ]
  }
]

Finetuning scripts

set -x

HOSTFILE=hostfile
DS_CONFIG=./finetune/ds_config_zero2.json

# PARAMS
LR=5e-6
EPOCHS=3
MAX_LEN=4096
BATCH_SIZE=4
NUM_NODES=1
NUM_GPUS=8
MASTER_PORT=29500

IS_CONCAT=False # Whether to concatenate to maximum length (MAX_LEN)

DATA_PATH="./data/training_data_sample.json"
MODEL_PATH="qihoo360/360Zhinao2-7B-Base"
OUTPUT_DIR="./outputs/"

deepspeed --hostfile ${HOSTFILE} \
        --master_port ${MASTER_PORT} \
        --num_nodes ${NUM_NODES} \
        --num_gpus ${NUM_GPUS} \
        finetune.py \
        --report_to "tensorboard" \
        --data_path ${DATA_PATH} \
        --model_name_or_path ${MODEL_PATH} \
        --output_dir ${OUTPUT_DIR} \
        --model_max_length ${MAX_LEN} \
        --num_train_epochs ${EPOCHS} \
        --per_device_train_batch_size ${BATCH_SIZE} \
        --gradient_accumulation_steps 1 \
        --save_strategy steps \
        --save_steps 200 \
        --learning_rate ${LR} \
        --lr_scheduler_type cosine \
        --adam_beta1 0.9 \
        --adam_beta2 0.95 \
        --adam_epsilon 1e-8 \
        --max_grad_norm 1.0 \
        --weight_decay 0.1 \
        --warmup_ratio 0.01 \
        --gradient_checkpointing True \
        --bf16 True \
        --tf32 True \
        --deepspeed ${DS_CONFIG} \
        --is_concat ${IS_CONCAT} \
        --logging_steps 1 \
        --log_on_each_node False

bash finetune/ds_finetune.sh

• Configuring HOSTFILE switches between single-machine and multi-machine training.
• configuring ds_config switches between zero1, zero2 and zero3.
• fp16, bf16 could configure mixed precision training. bf16 is recommended to be consistent with the pretrained model.
• is_concat configures whether the training data is concatenated or not.

License

The source code of this repository follows the open-source license Apache 2.0.

360​Zhinao open-source models support free commercial use. It is not necessary for you to submit a request for commercial usage.