zephyr-7b-alpha for DeepSparse
Usage
pip install deepsparse-nightly[llm]
from deepsparse import TextGeneration
model = TextGeneration(model="hf:mgoin/zephyr-7b-alpha-ds")
out = model("Once upon a time,", max_new_tokens=100)
print(out.generations[0].text)
### there was a young woman named Lily. She was a kind and gentle soul, with a heart full of love and compassion. Lily had always been fascinated by the natural world, and she spent most of her free time exploring the forests and fields around her home.\n\nOne day, as she was wandering through the woods, Lily stumbled upon a small clearing. In the center of the clearing, she saw a beautiful butterfly fluttering its wings. The butterfly was unlike any she had
How to export from zephyr-7b-alpha
Install SparseML with this PR
git clone https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha
sparseml.transformers.export_onnx --model_path ./zephyr-7b-alpha --task text-generation --sequence_length 512 --trust_remote_code
cp deployment/model.onnx deployment/model-orig.onnx
python ~/onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx
- Downloads last month
- 3
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.