michaelfeil commited on
Commit
1910834
1 Parent(s): 25c484c

Upload Phind/Phind-CodeLlama-34B-v2 ctranslate2 weights

Browse files
.gitattributes CHANGED
@@ -33,3 +33,11 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ pytorch_model-00006-of-00007.bin filter=lfs diff=lfs merge=lfs -text
37
+ pytorch_model-00007-of-00007.bin filter=lfs diff=lfs merge=lfs -text
38
+ pytorch_model.bin.index.json filter=lfs diff=lfs merge=lfs -text
39
+ pytorch_model-00001-of-00007.bin filter=lfs diff=lfs merge=lfs -text
40
+ pytorch_model-00002-of-00007.bin filter=lfs diff=lfs merge=lfs -text
41
+ pytorch_model-00003-of-00007.bin filter=lfs diff=lfs merge=lfs -text
42
+ pytorch_model-00004-of-00007.bin filter=lfs diff=lfs merge=lfs -text
43
+ pytorch_model-00005-of-00007.bin filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama2
3
+ model-index:
4
+ - name: Phind-CodeLlama-34B-v1
5
+ results:
6
+ - task:
7
+ type: text-generation
8
+ dataset:
9
+ type: openai_humaneval
10
+ name: HumanEval
11
+ metrics:
12
+ - name: pass@1
13
+ type: pass@1
14
+ value: 73.8%
15
+ verified: false
16
+ tags:
17
+ - ctranslate2
18
+ - int8
19
+ - float16
20
+ - code llama
21
+ ---
22
+ # # Fast-Inference with Ctranslate2
23
+ Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU.
24
+
25
+ quantized version of [Phind/Phind-CodeLlama-34B-v2](https://huggingface.co/Phind/Phind-CodeLlama-34B-v2)
26
+ ```bash
27
+ pip install hf-hub-ctranslate2>=2.12.0 ctranslate2>=3.17.1
28
+ ```
29
+
30
+ ```python
31
+ # from transformers import AutoTokenizer
32
+ model_name = "michaelfeil/ct2fast-Phind-CodeLlama-34B-v2"
33
+
34
+
35
+ from hf_hub_ctranslate2 import GeneratorCT2fromHfHub
36
+ model = GeneratorCT2fromHfHub(
37
+ # load in int8 on CUDA
38
+ model_name_or_path=model_name,
39
+ device="cuda",
40
+ compute_type="int8_float16",
41
+ # tokenizer=AutoTokenizer.from_pretrained("{ORG}/{NAME}")
42
+ )
43
+ outputs = model.generate(
44
+ text=["def fibonnaci(", "User: How are you doing? Bot:"],
45
+ max_length=64,
46
+ include_prompt_in_result=False
47
+ )
48
+ print(outputs)
49
+ ```
50
+
51
+ Checkpoint compatible to [ctranslate2>=3.17.1](https://github.com/OpenNMT/CTranslate2)
52
+ and [hf-hub-ctranslate2>=2.12.0](https://github.com/michaelfeil/hf-hub-ctranslate2)
53
+ - `compute_type=int8_float16` for `device="cuda"`
54
+ - `compute_type=int8` for `device="cpu"`
55
+
56
+ Converted on 2023-10-08 using
57
+ ```
58
+ LLama-2 -> removed <pad> token.
59
+ ```
60
+
61
+ # Licence and other remarks:
62
+ This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo.
63
+
64
+ # Original description
65
+
66
+
67
+ # **Phind-CodeLlama-34B-v2**
68
+ We've fine-tuned Phind-CodeLlama-34B-v1 on an additional 1.5B tokens high-quality programming-related data, achieving **73.8% pass@1** on HumanEval. It's the current state-of-the-art amongst open-source models.
69
+
70
+ Furthermore, this model is **instruction-tuned** on the Alpaca/Vicuna format to be steerable and easy-to-use.
71
+
72
+ More details can be found on our [blog post](https://www.phind.com/blog/code-llama-beats-gpt4).
73
+
74
+ ## Model Details
75
+ This model is fine-tuned from Phind-CodeLlama-34B-v1 and achieves **73.8% pass@1** on HumanEval.
76
+
77
+ Phind-CodeLlama-34B-v2 is **multi-lingual** and is proficient in Python, C/C++, TypeScript, Java, and more.
78
+
79
+ ## Dataset Details
80
+ We fined-tuned on a proprietary dataset of 1.5B tokens of high quality programming problems and solutions. This dataset consists of instruction-answer pairs instead of code completion examples, making it structurally different from HumanEval. LoRA was not used -- both models are a native finetune. We used DeepSpeed ZeRO 3 and Flash Attention 2 to train these models in 15 hours on 32 A100-80GB GPUs. We used a sequence length of 4096 tokens.
81
+
82
+ ## How to Get Started with the Model
83
+
84
+ Make sure to install Transformers from the main git branch:
85
+
86
+ ```bash
87
+ pip install git+https://github.com/huggingface/transformers.git
88
+ ```
89
+
90
+ ## How to Prompt the Model
91
+ This model accepts the Alpaca/Vicuna instruction format.
92
+
93
+ For example:
94
+
95
+ ```
96
+ ### System Prompt
97
+ You are an intelligent programming assistant.
98
+
99
+ ### User Message
100
+ Implement a linked list in C++
101
+
102
+ ### Assistant
103
+ ...
104
+ ```
105
+
106
+ ## How to reproduce HumanEval Results
107
+
108
+ To reproduce our results:
109
+
110
+ ```python
111
+
112
+ from transformers import AutoTokenizer, LlamaForCausalLM
113
+ from human_eval.data import write_jsonl, read_problems
114
+ from tqdm import tqdm
115
+
116
+ # initialize the model
117
+
118
+ model_path = "Phind/Phind-CodeLlama-34B-v2"
119
+ model = LlamaForCausalLM.from_pretrained(model_path, device_map="auto")
120
+ tokenizer = AutoTokenizer.from_pretrained(model_path)
121
+
122
+ # HumanEval helper
123
+
124
+ def generate_one_completion(prompt: str):
125
+ tokenizer.pad_token = tokenizer.eos_token
126
+ inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=4096)
127
+
128
+ # Generate
129
+ generate_ids = model.generate(inputs.input_ids.to("cuda"), max_new_tokens=384, do_sample=True, top_p=0.75, top_k=40, temperature=0.1)
130
+ completion = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
131
+ completion = completion.replace(prompt, "").split("\n\n\n")[0]
132
+
133
+ return completion
134
+
135
+ # perform HumanEval
136
+ problems = read_problems()
137
+
138
+ num_samples_per_task = 1
139
+ samples = [
140
+ dict(task_id=task_id, completion=generate_one_completion(problems[task_id]["prompt"]))
141
+ for task_id in tqdm(problems)
142
+ for _ in range(num_samples_per_task)
143
+ ]
144
+ write_jsonl("samples.jsonl", samples)
145
+
146
+ # run `evaluate_functional_correctness samples.jsonl` in your HumanEval code sandbox
147
+ ```
148
+
149
+ ## Bias, Risks, and Limitations
150
+
151
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
152
+ This model has undergone very limited testing. Additional safety testing should be performed before any real-world deployments.
153
+
154
+
155
+ ## Training details
156
+
157
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
158
+
159
+ - **Hardware Type:** 32x A100-80GB
160
+ - **Hours used:** 480 GPU-hours
161
+ - **Cloud Provider:** AWS
162
+ - **Compute Region:** us-east-1
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/fsx/Phind-CodeLlama-34B-v1",
3
+ "architectures": [
4
+ "LlamaForCausalLM"
5
+ ],
6
+ "bos_token_id": 1,
7
+ "eos_token_id": 2,
8
+ "hidden_act": "silu",
9
+ "hidden_size": 8192,
10
+ "initializer_range": 0.02,
11
+ "intermediate_size": 22016,
12
+ "max_position_embeddings": 16384,
13
+ "model_type": "llama",
14
+ "num_attention_heads": 64,
15
+ "num_hidden_layers": 48,
16
+ "num_key_value_heads": 8,
17
+ "pretraining_tp": 1,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_scaling": null,
20
+ "rope_theta": 1000000,
21
+ "tie_word_embeddings": false,
22
+ "torch_dtype": "bfloat16",
23
+ "transformers_version": "4.33.0.dev0",
24
+ "use_cache": true,
25
+ "vocab_size": 32000,
26
+ "bos_token": "<s>",
27
+ "eos_token": "</s>",
28
+ "layer_norm_epsilon": 1e-05,
29
+ "unk_token": "<unk>"
30
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.33.0.dev0"
6
+ }
model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:48749279396a6bcf43d64cd78889928cff7aedcbfeff7e8147af1105728609a7
3
+ size 33758632127
special_tokens_map.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": true,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "</s>",
11
+ "lstrip": false,
12
+ "normalized": true,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": "</s>",
17
+ "unk_token": {
18
+ "content": "<unk>",
19
+ "lstrip": false,
20
+ "normalized": true,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ }
24
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "add_eos_token": false,
4
+ "bos_token": {
5
+ "__type": "AddedToken",
6
+ "content": "<s>",
7
+ "lstrip": false,
8
+ "normalized": true,
9
+ "rstrip": false,
10
+ "single_word": false
11
+ },
12
+ "clean_up_tokenization_spaces": false,
13
+ "eos_token": {
14
+ "__type": "AddedToken",
15
+ "content": "</s>",
16
+ "lstrip": false,
17
+ "normalized": true,
18
+ "rstrip": false,
19
+ "single_word": false
20
+ },
21
+ "legacy": null,
22
+ "model_max_length": 4096,
23
+ "pad_token": null,
24
+ "padding_side": "right",
25
+ "sp_model_kwargs": {},
26
+ "spaces_between_special_tokens": false,
27
+ "tokenizer_class": "LlamaTokenizer",
28
+ "unk_token": {
29
+ "__type": "AddedToken",
30
+ "content": "<unk>",
31
+ "lstrip": false,
32
+ "normalized": true,
33
+ "rstrip": false,
34
+ "single_word": false
35
+ },
36
+ "use_default_system_prompt": true
37
+ }
vocabulary.json ADDED
The diff for this file is too large to render. See raw diff
 
vocabulary.txt ADDED
The diff for this file is too large to render. See raw diff