jeffra commited on
Commit
556a2fa
1 Parent(s): 05b987a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +38 -1
README.md CHANGED
@@ -59,7 +59,44 @@ pip install "deepspeed>=0.14.2"
59
 
60
  ### Inference
61
 
62
- The Arctic github page has several code snippets and examples around running inference:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
 
64
  * Example with pure-HF: https://github.com/Snowflake-Labs/snowflake-arctic/blob/main/inference
65
  * Tutorial using vLLM: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main/inference/vllm
 
59
 
60
  ### Inference
61
 
62
+ Due to the model size we recommend using a single 8xH100 instance from your
63
+ favorite cloud provider such as: AWS [p5.48xlarge](https://aws.amazon.com/ec2/instance-types/p5/),
64
+ Azure [ND96isr_H100_v5](https://learn.microsoft.com/en-us/azure/virtual-machines/nd-h100-v5-series), etc.
65
+
66
+ In this example we are using FP8 quantization provided by DeepSpeed in the backend, we can also use FP6
67
+ quantization by specifying `q_bits=6` in the `ArcticQuantizationConfig` config. The `"150GiB"` setting
68
+ for max_memory is required until we can get DeepSpeed's FP quantization supported natively as a [HFQuantizer](https://huggingface.co/docs/transformers/main/en/hf_quantizer#build-a-new-hfquantizer-class) which we
69
+ are actively working on.
70
+
71
+ ```python
72
+ import os
73
+ # enable hf_transfer for faster ckpt download
74
+ os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
75
+
76
+ import torch
77
+ from transformers import AutoModelForCausalLM, AutoTokenizer
78
+ from transformers.models.arctic.configuration_arctic import ArcticQuantizationConfig
79
+
80
+ tokenizer = AutoTokenizer.from_pretrained("Snowflake/snowflake-arctic-instruct")
81
+
82
+ quant_config = ArcticQuantizationConfig(q_bits=8)
83
+
84
+ model = AutoModelForCausalLM.from_pretrained(
85
+ "Snowflake/snowflake-arctic-instruct",
86
+ low_cpu_mem_usage=True,
87
+ device_map="auto",
88
+ ds_quantization_config=quant_config,
89
+ max_memory={i: "150GiB" for i in range(8)},
90
+ torch_dtype=torch.bfloat16)
91
+
92
+ messages = [{"role": "user", "content": "What is 1 + 1 "}]
93
+ input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
94
+
95
+ outputs = model.generate(input_ids=input_ids, max_new_tokens=20)
96
+ print(tokenizer.decode(outputs[0]))
97
+ ```
98
+
99
+ The Arctic github page has additional code snippets and examples around running inference:
100
 
101
  * Example with pure-HF: https://github.com/Snowflake-Labs/snowflake-arctic/blob/main/inference
102
  * Tutorial using vLLM: https://github.com/Snowflake-Labs/snowflake-arctic/tree/main/inference/vllm