Benchmarks on performance
#38
by
ptrrrr
- opened
Folks: Any benchmarks on performance of Falcon-7B.
We have a fine-tuned model that does about 25 tokens per second on a A10 GPU + 24GB VRAM + 4CPU + 16 GB RAM.
This seems quite expensive and bloody slow to run. Any others have insights on what kind of performance to expect?
We are upgrading it to 16cpus+ 64GB RAM and testing on A100 also. Will share those results
