michaelfeil
commited on
Commit
•
1910834
1
Parent(s):
25c484c
Upload Phind/Phind-CodeLlama-34B-v2 ctranslate2 weights
Browse files- .gitattributes +8 -0
- README.md +162 -0
- config.json +30 -0
- generation_config.json +6 -0
- model.bin +3 -0
- special_tokens_map.json +24 -0
- tokenizer_config.json +37 -0
- vocabulary.json +0 -0
- vocabulary.txt +0 -0
.gitattributes
CHANGED
@@ -33,3 +33,11 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
36 |
+
pytorch_model-00006-of-00007.bin filter=lfs diff=lfs merge=lfs -text
|
37 |
+
pytorch_model-00007-of-00007.bin filter=lfs diff=lfs merge=lfs -text
|
38 |
+
pytorch_model.bin.index.json filter=lfs diff=lfs merge=lfs -text
|
39 |
+
pytorch_model-00001-of-00007.bin filter=lfs diff=lfs merge=lfs -text
|
40 |
+
pytorch_model-00002-of-00007.bin filter=lfs diff=lfs merge=lfs -text
|
41 |
+
pytorch_model-00003-of-00007.bin filter=lfs diff=lfs merge=lfs -text
|
42 |
+
pytorch_model-00004-of-00007.bin filter=lfs diff=lfs merge=lfs -text
|
43 |
+
pytorch_model-00005-of-00007.bin filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: llama2
|
3 |
+
model-index:
|
4 |
+
- name: Phind-CodeLlama-34B-v1
|
5 |
+
results:
|
6 |
+
- task:
|
7 |
+
type: text-generation
|
8 |
+
dataset:
|
9 |
+
type: openai_humaneval
|
10 |
+
name: HumanEval
|
11 |
+
metrics:
|
12 |
+
- name: pass@1
|
13 |
+
type: pass@1
|
14 |
+
value: 73.8%
|
15 |
+
verified: false
|
16 |
+
tags:
|
17 |
+
- ctranslate2
|
18 |
+
- int8
|
19 |
+
- float16
|
20 |
+
- code llama
|
21 |
+
---
|
22 |
+
# # Fast-Inference with Ctranslate2
|
23 |
+
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU.
|
24 |
+
|
25 |
+
quantized version of [Phind/Phind-CodeLlama-34B-v2](https://huggingface.co/Phind/Phind-CodeLlama-34B-v2)
|
26 |
+
```bash
|
27 |
+
pip install hf-hub-ctranslate2>=2.12.0 ctranslate2>=3.17.1
|
28 |
+
```
|
29 |
+
|
30 |
+
```python
|
31 |
+
# from transformers import AutoTokenizer
|
32 |
+
model_name = "michaelfeil/ct2fast-Phind-CodeLlama-34B-v2"
|
33 |
+
|
34 |
+
|
35 |
+
from hf_hub_ctranslate2 import GeneratorCT2fromHfHub
|
36 |
+
model = GeneratorCT2fromHfHub(
|
37 |
+
# load in int8 on CUDA
|
38 |
+
model_name_or_path=model_name,
|
39 |
+
device="cuda",
|
40 |
+
compute_type="int8_float16",
|
41 |
+
# tokenizer=AutoTokenizer.from_pretrained("{ORG}/{NAME}")
|
42 |
+
)
|
43 |
+
outputs = model.generate(
|
44 |
+
text=["def fibonnaci(", "User: How are you doing? Bot:"],
|
45 |
+
max_length=64,
|
46 |
+
include_prompt_in_result=False
|
47 |
+
)
|
48 |
+
print(outputs)
|
49 |
+
```
|
50 |
+
|
51 |
+
Checkpoint compatible to [ctranslate2>=3.17.1](https://github.com/OpenNMT/CTranslate2)
|
52 |
+
and [hf-hub-ctranslate2>=2.12.0](https://github.com/michaelfeil/hf-hub-ctranslate2)
|
53 |
+
- `compute_type=int8_float16` for `device="cuda"`
|
54 |
+
- `compute_type=int8` for `device="cpu"`
|
55 |
+
|
56 |
+
Converted on 2023-10-08 using
|
57 |
+
```
|
58 |
+
LLama-2 -> removed <pad> token.
|
59 |
+
```
|
60 |
+
|
61 |
+
# Licence and other remarks:
|
62 |
+
This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.
|
63 |
+
|
64 |
+
# Original description
|
65 |
+
|
66 |
+
|
67 |
+
# **Phind-CodeLlama-34B-v2**
|
68 |
+
We've fine-tuned Phind-CodeLlama-34B-v1 on an additional 1.5B tokens high-quality programming-related data, achieving **73.8% pass@1** on HumanEval. It's the current state-of-the-art amongst open-source models.
|
69 |
+
|
70 |
+
Furthermore, this model is **instruction-tuned** on the Alpaca/Vicuna format to be steerable and easy-to-use.
|
71 |
+
|
72 |
+
More details can be found on our [blog post](https://www.phind.com/blog/code-llama-beats-gpt4).
|
73 |
+
|
74 |
+
## Model Details
|
75 |
+
This model is fine-tuned from Phind-CodeLlama-34B-v1 and achieves **73.8% pass@1** on HumanEval.
|
76 |
+
|
77 |
+
Phind-CodeLlama-34B-v2 is **multi-lingual** and is proficient in Python, C/C++, TypeScript, Java, and more.
|
78 |
+
|
79 |
+
## Dataset Details
|
80 |
+
We fined-tuned on a proprietary dataset of 1.5B tokens of high quality programming problems and solutions. This dataset consists of instruction-answer pairs instead of code completion examples, making it structurally different from HumanEval. LoRA was not used -- both models are a native finetune. We used DeepSpeed ZeRO 3 and Flash Attention 2 to train these models in 15 hours on 32 A100-80GB GPUs. We used a sequence length of 4096 tokens.
|
81 |
+
|
82 |
+
## How to Get Started with the Model
|
83 |
+
|
84 |
+
Make sure to install Transformers from the main git branch:
|
85 |
+
|
86 |
+
```bash
|
87 |
+
pip install git+https://github.com/huggingface/transformers.git
|
88 |
+
```
|
89 |
+
|
90 |
+
## How to Prompt the Model
|
91 |
+
This model accepts the Alpaca/Vicuna instruction format.
|
92 |
+
|
93 |
+
For example:
|
94 |
+
|
95 |
+
```
|
96 |
+
### System Prompt
|
97 |
+
You are an intelligent programming assistant.
|
98 |
+
|
99 |
+
### User Message
|
100 |
+
Implement a linked list in C++
|
101 |
+
|
102 |
+
### Assistant
|
103 |
+
...
|
104 |
+
```
|
105 |
+
|
106 |
+
## How to reproduce HumanEval Results
|
107 |
+
|
108 |
+
To reproduce our results:
|
109 |
+
|
110 |
+
```python
|
111 |
+
|
112 |
+
from transformers import AutoTokenizer, LlamaForCausalLM
|
113 |
+
from human_eval.data import write_jsonl, read_problems
|
114 |
+
from tqdm import tqdm
|
115 |
+
|
116 |
+
# initialize the model
|
117 |
+
|
118 |
+
model_path = "Phind/Phind-CodeLlama-34B-v2"
|
119 |
+
model = LlamaForCausalLM.from_pretrained(model_path, device_map="auto")
|
120 |
+
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
121 |
+
|
122 |
+
# HumanEval helper
|
123 |
+
|
124 |
+
def generate_one_completion(prompt: str):
|
125 |
+
tokenizer.pad_token = tokenizer.eos_token
|
126 |
+
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=4096)
|
127 |
+
|
128 |
+
# Generate
|
129 |
+
generate_ids = model.generate(inputs.input_ids.to("cuda"), max_new_tokens=384, do_sample=True, top_p=0.75, top_k=40, temperature=0.1)
|
130 |
+
completion = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
|
131 |
+
completion = completion.replace(prompt, "").split("\n\n\n")[0]
|
132 |
+
|
133 |
+
return completion
|
134 |
+
|
135 |
+
# perform HumanEval
|
136 |
+
problems = read_problems()
|
137 |
+
|
138 |
+
num_samples_per_task = 1
|
139 |
+
samples = [
|
140 |
+
dict(task_id=task_id, completion=generate_one_completion(problems[task_id]["prompt"]))
|
141 |
+
for task_id in tqdm(problems)
|
142 |
+
for _ in range(num_samples_per_task)
|
143 |
+
]
|
144 |
+
write_jsonl("samples.jsonl", samples)
|
145 |
+
|
146 |
+
# run `evaluate_functional_correctness samples.jsonl` in your HumanEval code sandbox
|
147 |
+
```
|
148 |
+
|
149 |
+
## Bias, Risks, and Limitations
|
150 |
+
|
151 |
+
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
152 |
+
This model has undergone very limited testing. Additional safety testing should be performed before any real-world deployments.
|
153 |
+
|
154 |
+
|
155 |
+
## Training details
|
156 |
+
|
157 |
+
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
|
158 |
+
|
159 |
+
- **Hardware Type:** 32x A100-80GB
|
160 |
+
- **Hours used:** 480 GPU-hours
|
161 |
+
- **Cloud Provider:** AWS
|
162 |
+
- **Compute Region:** us-east-1
|
config.json
ADDED
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"_name_or_path": "/fsx/Phind-CodeLlama-34B-v1",
|
3 |
+
"architectures": [
|
4 |
+
"LlamaForCausalLM"
|
5 |
+
],
|
6 |
+
"bos_token_id": 1,
|
7 |
+
"eos_token_id": 2,
|
8 |
+
"hidden_act": "silu",
|
9 |
+
"hidden_size": 8192,
|
10 |
+
"initializer_range": 0.02,
|
11 |
+
"intermediate_size": 22016,
|
12 |
+
"max_position_embeddings": 16384,
|
13 |
+
"model_type": "llama",
|
14 |
+
"num_attention_heads": 64,
|
15 |
+
"num_hidden_layers": 48,
|
16 |
+
"num_key_value_heads": 8,
|
17 |
+
"pretraining_tp": 1,
|
18 |
+
"rms_norm_eps": 1e-05,
|
19 |
+
"rope_scaling": null,
|
20 |
+
"rope_theta": 1000000,
|
21 |
+
"tie_word_embeddings": false,
|
22 |
+
"torch_dtype": "bfloat16",
|
23 |
+
"transformers_version": "4.33.0.dev0",
|
24 |
+
"use_cache": true,
|
25 |
+
"vocab_size": 32000,
|
26 |
+
"bos_token": "<s>",
|
27 |
+
"eos_token": "</s>",
|
28 |
+
"layer_norm_epsilon": 1e-05,
|
29 |
+
"unk_token": "<unk>"
|
30 |
+
}
|
generation_config.json
ADDED
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"_from_model_config": true,
|
3 |
+
"bos_token_id": 1,
|
4 |
+
"eos_token_id": 2,
|
5 |
+
"transformers_version": "4.33.0.dev0"
|
6 |
+
}
|
model.bin
ADDED
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
1 |
+
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:48749279396a6bcf43d64cd78889928cff7aedcbfeff7e8147af1105728609a7
|
3 |
+
size 33758632127
|
special_tokens_map.json
ADDED
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"bos_token": {
|
3 |
+
"content": "<s>",
|
4 |
+
"lstrip": false,
|
5 |
+
"normalized": true,
|
6 |
+
"rstrip": false,
|
7 |
+
"single_word": false
|
8 |
+
},
|
9 |
+
"eos_token": {
|
10 |
+
"content": "</s>",
|
11 |
+
"lstrip": false,
|
12 |
+
"normalized": true,
|
13 |
+
"rstrip": false,
|
14 |
+
"single_word": false
|
15 |
+
},
|
16 |
+
"pad_token": "</s>",
|
17 |
+
"unk_token": {
|
18 |
+
"content": "<unk>",
|
19 |
+
"lstrip": false,
|
20 |
+
"normalized": true,
|
21 |
+
"rstrip": false,
|
22 |
+
"single_word": false
|
23 |
+
}
|
24 |
+
}
|
tokenizer_config.json
ADDED
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
{
|
2 |
+
"add_bos_token": true,
|
3 |
+
"add_eos_token": false,
|
4 |
+
"bos_token": {
|
5 |
+
"__type": "AddedToken",
|
6 |
+
"content": "<s>",
|
7 |
+
"lstrip": false,
|
8 |
+
"normalized": true,
|
9 |
+
"rstrip": false,
|
10 |
+
"single_word": false
|
11 |
+
},
|
12 |
+
"clean_up_tokenization_spaces": false,
|
13 |
+
"eos_token": {
|
14 |
+
"__type": "AddedToken",
|
15 |
+
"content": "</s>",
|
16 |
+
"lstrip": false,
|
17 |
+
"normalized": true,
|
18 |
+
"rstrip": false,
|
19 |
+
"single_word": false
|
20 |
+
},
|
21 |
+
"legacy": null,
|
22 |
+
"model_max_length": 4096,
|
23 |
+
"pad_token": null,
|
24 |
+
"padding_side": "right",
|
25 |
+
"sp_model_kwargs": {},
|
26 |
+
"spaces_between_special_tokens": false,
|
27 |
+
"tokenizer_class": "LlamaTokenizer",
|
28 |
+
"unk_token": {
|
29 |
+
"__type": "AddedToken",
|
30 |
+
"content": "<unk>",
|
31 |
+
"lstrip": false,
|
32 |
+
"normalized": true,
|
33 |
+
"rstrip": false,
|
34 |
+
"single_word": false
|
35 |
+
},
|
36 |
+
"use_default_system_prompt": true
|
37 |
+
}
|
vocabulary.json
ADDED
The diff for this file is too large to render.
See raw diff
|
|
vocabulary.txt
ADDED
The diff for this file is too large to render.
See raw diff
|
|