Saving Memory Using Padding-Free Transformer Layers during Finetuning
•
6
Mayank Mishra
mayank-mishra
AI & ML interests
Large Language Models, Distributed Training and Inference
Articles
Organizations
Posts
4
Post
1426
New preprint out with colleagues from MIT and IBM Research
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (2405.12981)
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (2405.12981)
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
Post
2193
Thrilled to unveil DS-MoE: a dense training and sparse inference scheme for enhanced computational and memory efficiency in your MoE models! 🚀🚀🚀
Discover more in our blog: https://huggingface.co/blog/bpan/ds-moe and dive into the details with our paper: Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models (2404.05567)
Discover more in our blog: https://huggingface.co/blog/bpan/ds-moe and dive into the details with our paper: Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models (2404.05567)
Collections
1
models
8
mayank-mishra/granite-3b-code-glaive-20k
Text Generation
•
Updated
•
4
mayank-mishra/granite-20b-code-instruct-Q4_K_M-GGUF
Text Generation
•
Updated
•
41
mayank-mishra/starcoder-GPTQ-8bit-128g
Updated
•
11
mayank-mishra/starcoder-GPTQ-4bit-128g
Updated
•
16
mayank-mishra/starcoderbase-GPTQ-4bit-128g
Updated
•
21
mayank-mishra/starcoderbase-GPTQ-8bit-128g
Updated
•
3
mayank-mishra/santacoder-GPTQ-4bit-128g
Updated
•
2
mayank-mishra/santacoder-GPTQ-8bit-128g
Updated
•
1