中文  ｜  English 

Welcome to visit 360Zhinao official website https://ai.360.com to experience more powerful functions.

## Web Demo You can also use web interaction for a quick experience ```shell streamlit run web_demo.py ```

## API Demo Start command ```shell python openai_api.py ``` Request parameter ```shell curl --location --request POST 'http://localhost:8360/v1/chat/completions' \ --header 'Content-Type: application/json' \ --data-raw '{ "max_new_tokens": 200, "do_sample": true, "top_k": 0, "top_p": 0.8, "temperature": 1.0, "repetition_penalty": 1.0, "messages": [ { "role": "user", "content": "你叫什么名字？" } ] }' ``` # Model Inference ## Quantization We provide quantization schemes based on AutoGPTQ and open source the Int4 quantization models. The quantization model has little effect loss, but it can significantly reduce the video memory occupation and improve the inference speed. The BF16, Int8, and Int4 models are tested on the benchmarks, and the results are as follows: | Quantization | MMLU | CEval (val) | GSM8K | Humaneval | |-|-|-|-|-| | 360Zhinao-7B-Chat-4K (BF16) |-|-|-|-| | 360Zhinao-7B-Chat-4K (Int8) |-|-|-|-| | 360Zhinao-7B-Chat-4K (Int4) |-|-|-|-| ## Deployment ### vLLM Installation If you want to deploy and accelerate inference, we recommend using `vLLM==0.3.3`。 If you are using **CUDA 12.1 and PyTorch 2.1**, you can install vLLM directly with the following command. ```shell pip install vllm==0.3.3 ``` Otherwise, please refer to the official vLLM [Installation Instructions](https://docs.vllm.ai/en/latest/getting_started/installation.html)。 >Once the installation is complete, you will need to do the following 1. Copy the vllm/zhinao.py file to the vllm/model_executor/models directory corresponding to your env environment. 2. Then add a line to vllm/model_executor/models/\_\_init\_\_.py ```shell "ZhinaoForCausalLM": ("zhinao", "ZhinaoForCausalLM"), ``` ### vLLM Service Start Starting the service ```shell python -m vllm.entrypoints.openai.api_server \ --served-model-name 360Zhinao-7B-Chat-4K \ --model qihoo360/360Zhinao-7B-Chat-4k \ --trust-remote-code \ --tensor-parallel-size 1 --max-model-len 18000 \ --host 0.0.0.0 \ --port 8360 ``` Use curl to request the service ```shell curl http://localhost:8360/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "360Zhinao-7B-Chat-4K", "max_tokens": 200, "top_k": 0, "top_p": 0.8, "temperature": 1.0, "presence_penalty": 0.0, "frequency_penalty": 0.0, "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "你好"} ], "stop": [ "", "<|im_end|>", "<|im_start|>" ] }' ``` Use python to request the service ```python from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client.chat.completions.create( model="360Zhinao-7B-Chat-4K", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "你好"}, ], stop=[ "", "<|im_end|>", "<|im_start|>" ], presence_penalty=0.0, frequency_penalty=0.0 ) print("Chat response:", chat_response) ``` > Notice: If you need to enable repetition penalty, recommended to use *presence_penalty* and *frequency_penalty* parameters. > # Model Finetune ## Training data Training Data: data/training_data_sample.json. The sample data is 10,000 pieces sampled from [multiturn_chat_0.8M](https://huggingface.co/datasets/BelleGroup/multiturn_chat_0.8M) and format converted. Data Format: ```json [ { "id": 1, "conversations": [ { "from": "system", "value": "You are a helpful assistant." }, { "from": "user", "value": "您好啊" }, { "from": "assistant", "value": "你好！我今天能为您做些什么？有什么问题或需要帮助吗? 我在这里为您提供服务。" } ] } ] ``` ## Fine-tuning scripts ```shell set -x HOSTFILE=hostfile DS_CONFIG=./finetune/ds_config_zero2.json # PARAMS LR=5e-6 EPOCHS=3 MAX_LEN=4096 BATCH_SIZE=4 NUM_NODES=1 NUM_GPUS=8 MASTER_PORT=29500 IS_CONCAT=False # Whether to concatenate to maximum length (MAX_LEN) DATA_PATH="./data/training_data_sample.json" MODEL_PATH="qihoo360/360Zhinao-7B-Base" OUTPUT_DIR="./outputs/" deepspeed --hostfile ${HOSTFILE} \ --master_port ${MASTER_PORT} \ --num_nodes ${NUM_NODES} \ --num_gpus ${NUM_GPUS} \ finetune.py \ --report_to "tensorboard" \ --data_path ${DATA_PATH} \ --model_name_or_path ${MODEL_PATH} \ --output_dir ${OUTPUT_DIR} \ --model_max_length ${MAX_LEN} \ --num_train_epochs ${EPOCHS} \ --per_device_train_batch_size ${BATCH_SIZE} \ --gradient_accumulation_steps 1 \ --save_strategy steps \ --save_steps 200 \ --learning_rate ${LR} \ --lr_scheduler_type cosine \ --adam_beta1 0.9 \ --adam_beta2 0.95 \ --adam_epsilon 1e-8 \ --max_grad_norm 1.0 \ --weight_decay 0.1 \ --warmup_ratio 0.01 \ --gradient_checkpointing True \ --bf16 True \ --tf32 True \ --deepspeed ${DS_CONFIG} \ --is_concat ${IS_CONCAT} \ --logging_steps 1 \ --log_on_each_node False ``` ```shell bash finetune/ds_finetune.sh ``` - By configuring the **hostfile**, single-machine and multi-machine training can be realized. - By configuring **ds_config**, realize zero2 and zero3 training - By configuring the **fp16**、**bf16** realize mixed precision training, bf16 is recommended to be consistent with the pre-trained model. - By configuring **is_concat**, Whether the training data is concatenated or not is controlled. When the magnitude of the training data is large, the training efficiency can be improved by concatenation. # License The source code of this warehouse follows the open source license Apache 2.0. The 360 ​Zhinao open source model supports commercial use. If you need to use this model and its derivative models for commercial purposes, please contact us via email (g-zhinao-opensource@360.cn) to apply. For the specific license agreement, please see [《360 Zhinao Open Source Model License》](./360智脑开源模型许可证.txt).