hardware requirements
Please provide the recommended capacity of CPU, GPU, and RAM, or at least a description of the hardware which you use to run it.
I used
16 GB of RAM
GPU 11gig (1080TI)
CPU i7 6800K
I started model from scratch in 37 minutes( 100 mb/s internet)
each query execute very long ( even with GPU)
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained("Salesforce/xgen-7b-8k-base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Salesforce/xgen-7b-8k-base", torch_dtype=torch.bfloat16)
model = model.to(device)
inputs = tokenizer("The world is", return_tensors="pt").to(device)
sample = model.generate(**inputs, max_length=128)
decoded_output = tokenizer.decode(sample[0])
print(decoded_output)
Please provide someone speed result which you achieve with your machine.