learning

.gitattributes

@@ -1,34 +1,35 @@
 *.7z filter=lfs diff=lfs merge=lfs -text
 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text
-*.bin.* filter=lfs diff=lfs merge=lfs -text
 *.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
 *.ftz filter=lfs diff=lfs merge=lfs -text
 *.gz filter=lfs diff=lfs merge=lfs -text
 *.h5 filter=lfs diff=lfs merge=lfs -text
 *.joblib filter=lfs diff=lfs merge=lfs -text
 *.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
 *.model filter=lfs diff=lfs merge=lfs -text
 *.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
 *.onnx filter=lfs diff=lfs merge=lfs -text
 *.ot filter=lfs diff=lfs merge=lfs -text
 *.parquet filter=lfs diff=lfs merge=lfs -text
 *.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
 *.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
 saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
 *.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
 *.xz filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
-*.zstandard filter=lfs diff=lfs merge=lfs -text
-*.tfevents* filter=lfs diff=lfs merge=lfs -text
-*.db* filter=lfs diff=lfs merge=lfs -text
-*.ark* filter=lfs diff=lfs merge=lfs -text
-**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
-**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
-**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,20 +1,8 @@
 ---
 license: other
-tasks:
-- code-generation
-datasets:
-- codefuse-ai/CodeExercise-Python-27k
-- codefuse-ai/Evol-instruction-66k
-language:
-- en
-- zh
-pipeline_tag: text-generation
-library_name: transformers
-tags:
-- code
 ---
 # Model Card for CodeFuse-CodeLlama-34B
-![logo](LOGO.png)
+![Creation Approach](LOGO.png)
 
 [[中文]](#chinese)    [[English]](#english)
 
@@ -30,38 +18,17 @@ The context length of finetuning is 4K while it is able to be finetuned by 16k c
 
 ## News and Updates
 
-🔥🔥🔥 2023-09-26 We are pleased to announce the release of the [4-bit quantized version](https://huggingface.co/codefuse-ai/CodeFuse-CodeLlama-34B-4bits) of CodeFuse-CodeLlama-34B. Despite the quantization process, the model still achieves a remarkable 73.8% accuracy (greedy decoding) on the HumanEval pass@1 metric.
-
-🔥🔥🔥 2023-09-11 CodeFuse-CodeLlama34B has achieved 74.4% of pass@1 (greedy decoding) on HumanEval, which is SOTA results for openspurced LLMs at present.
+🔥🔥🔥 CodeFuse-CodeLlama34B-MFT has achived 74.4% of pass@1 on HumanEval, which is SOTA at present.
 
 <br>
 
-## Code Community
-
-**Homepage**: 🏡 https://github.com/codefuse-ai (**Please give us your support with a Star🌟 + Fork🚀 + Watch👀**)
-
-+ If you wish to fine-tune the model yourself, you can visit ✨[MFTCoder](https://github.com/codefuse-ai/MFTCoder)✨✨
-
-+ If you wish to deploy the model yourself, you can visit ✨[FasterTransformer4CodeFuse](https://github.com/codefuse-ai/FasterTransformer4CodeFuse)✨✨
-
-+ If you wish to see a demo of the model, you can visit ✨[CodeFuse Demo](https://github.com/codefuse-ai/codefuse)✨✨
-
-
 ## Performance
 
-
-| Model                       | HumanEval(pass@1) |  Date   |
-|:----------------------------|:-----------------:|:-------:|
-| **CodeFuse-CodeLlama-34B**  |     **74.4%**      | 2023.9  |
-| WizardCoder-Python-34B-V1.0 |       73.2%       | 2023.8  |
-| GPT-4(zero-shot)            |       67.0%       | 2023.3  |
-| PanGu-Coder2 15B            |       61.6%       | 2023.8  |
-| CodeLlama-34b-Python        |       53.7%       | 2023.8  |
-| CodeLlama-34b               |       48.8%       | 2023.8  |
-| GPT-3.5(zero-shot)          |       48.1%       | 2022.11 |
-| OctoCoder                   |       46.2%       | 2023.8  |
-| StarCoder-15B               |       33.6%       | 2023.5  |
-| LLaMA 2 70B(zero-shot)      |       29.9%       | 2023.7  |
+| Model                         | HumanEval(pass@1) |
+| :---------------------------- | :---------------: |
+| CodeLlama-34b                 |   48.8%(greedy decoding)   |
+| CodeLlama-34b-Python          |   53.7%(greedy decoding)   |
+| **CodeFuse-CodeLlama-34B** | **74.4%**(greedy decoding) |
 
 <br>
 
@@ -103,7 +70,6 @@ pip install -r requirements.txt
 ```
 
 ```python
-import torch
 from transformers import (
     AutoTokenizer, 
     AutoModelForCausalLM,
@@ -112,13 +78,7 @@ tokenizer = AutoTokenizer.from_pretrained(mode_name_or_path, trust_remote_code=T
 tokenizer.padding_side = "left"
 tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids("<unk>")
 tokenizer.eos_token_id = tokenizer.convert_tokens_to_ids("</s>")
-# try 4bit loading if cuda memory not enough
-model = AutoModelForCausalLM.from_pretrained(mode_name_or_path,
-                                             trust_remote_code=True,
-                                             load_in_4bit=False,
-                                             device_map="auto",
-                                             torch_dtype=torch.bfloat16)
-model.eval()
+model = AutoModelForCausalLM.from_pretrained(mode_name_or_path, trust_remote_code=True)
 
 HUMAN_ROLE_START_TAG = "<|role_start|>human<|role_end|>"
 BOT_ROLE_START_TAG = "<|role_start|>bot<|role_end|>"
@@ -139,32 +99,17 @@ gen_text = tokenizer.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], ski
 print(gen_text)
 ```
 
-## MD5
-We notice that the file may be corrupted during transfer process. Please check MD5 value before use.
 
-| Model File                       | MD5 Value                        |
-|:---------------------------------|:--------------------------------:|
-| pytorch_model-00001-of-00007.bin | 8d544b1bcb3449934184d4141137329c |
-| pytorch_model-00002-of-00007.bin | 9d5dbb30911e48a42fb6d0fcabb322a4 |
-| pytorch_model-00003-of-00007.bin | b0d4aecee0457d9332005a187e1fffed |
-| pytorch_model-00004-of-00007.bin | 5c7e002de5eab77d0194a2b0f6de0c24 |
-| pytorch_model-00005-of-00007.bin | d22a511aa26b5b17117b665a877490ab |
-| pytorch_model-00006-of-00007.bin | a5c28ac277fac07d16dd66537e54d109 |
-| pytorch_model-00007-of-00007.bin | a967e2c6195477b7407089c0bffa2d53 |
 
 
-## Citation
-If you find our [work](https://arxiv.org/abs/2311.02303) useful or helpful for your R&D works, please feel free to cite our paper as below.
-```
-@article{mftcoder2023,
-      title={MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning}, 
-      author={Bingchang Liu and Chaoyu Chen and Cong Liao and Zi Gong and Huan Wang and Zhichao Lei and Ming Liang and Dajun Chen and Min Shen and Hailian Zhou and Hang Yu and Jianguo Li},
-      year={2023},
-      journal={arXiv preprint arXiv},
-      archivePrefix={arXiv},
-      eprint={2311.02303}
-}
-```
+
+
+
+
+
+
+
+
 
 <a id="chinese"></a>
 
@@ -179,31 +124,13 @@ CodeFuse-CodeLlama34B-MFT 是一个通过QLoRA对基座模型CodeLlama-34b-Pytho
 
 <br>
 
-## 代码社区
-**大本营**： 🏡 https://github.com/codefuse-ai （**欢迎为我们的项目一键三连 Star🌟 + Fork🚀 + Watch👀**）
-
-+ 如果您想自己微调该模型，可以访问 ✨[MFTCoder](https://github.com/codefuse-ai/MFTCoder)✨✨
-
-+ 如果您想自己部署该模型，可以访问 ✨[FasterTransformer4CodeFuse](https://github.com/codefuse-ai/FasterTransformer4CodeFuse)✨✨
-
-+ 如果您想观看该模型示例，可以访问 ✨[CodeFuse Demo](https://github.com/codefuse-ai/codefuse)✨✨
-
-
 ## 评测表现(代码)
 
-| 模型                          | HumanEval(pass@1) |   日期    |
-|:----------------------------|:-----------------:|:-------:|
-| **CodeFuse-CodeLlama-34B**  |     **74.4%**      | 2023.9  |
-| WizardCoder-Python-34B-V1.0 |       73.2%       | 2023.8  |
-| GPT-4(zero-shot)            |       67.0%       | 2023.3  |
-| PanGu-Coder2 15B            |       61.6%       | 2023.8  |
-| CodeLlama-34b-Python        |       53.7%       | 2023.8  |
-| CodeLlama-34b               |       48.8%       | 2023.8  |
-| GPT-3.5(zero-shot)          |       48.1%       | 2022.11 |
-| OctoCoder                   |       46.2%       | 2023.8  |
-| StarCoder-15B               |       33.6%       | 2023.5  |
-| LLaMA 2 70B(zero-shot)      |       29.9%       | 2023.7  |
-
+| 模型                         | HumanEval(pass@1) |
+| :---------------------------- | :---------------: |
+| CodeLlama-34b                 |   48.8%(greedy decoding)   |
+| CodeLlama-34b-Python          |   53.7%(greedy decoding)   |
+| **CodeFuse-CodeLlama-34B** | **74.4%**(greedy decoding) |
 <br>
 
 ## Requirements
@@ -246,18 +173,12 @@ tokenizer = AutoTokenizer.from_pretrained(mode_name_or_path, trust_remote_code=T
 tokenizer.padding_side = "left"
 tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids("<unk>")
 tokenizer.eos_token_id = tokenizer.convert_tokens_to_ids("</s>")
-# 如果显存不够，可以考虑量化加载
-model = AutoModelForCausalLM.from_pretrained(mode_name_or_path,
-                                             trust_remote_code=True,
-                                             load_in_4bit=False,
-                                             device_map="auto",
-                                             torch_dtype=torch.bfloat16)
-model.eval()
+model = AutoModelForCausalLM.from_pretrained(mode_name_or_path, trust_remote_code=True)
 
 HUMAN_ROLE_START_TAG = "<|role_start|>human<|role_end|>"
 BOT_ROLE_START_TAG = "<|role_start|>bot<|role_end|>"
 
-text = f"{HUMAN_ROLE_START_TAG}请用C++实现求解第n个斐波那契数{BOT_ROLE_START_TAG}" 
+text = f"{HUMAN_ROLE_START_TAG}write a python function of quick sort.{BOT_ROLE_START_TAG}" 
 inputs = tokenizer(text, return_tensors='pt', padding=True, add_special_tokens=False).to("cuda")
 outputs = model.generate(
         inputs=inputs["input_ids"],
@@ -271,18 +192,4 @@ outputs = model.generate(
     )
 gen_text = tokenizer.batch_decode(outputs[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
 print(gen_text)
-```
-
-
-## MD5
-我们发现模型文件可能会在传输过程中损坏，使用前请检查文件MD5值。
-
-| 模型文件                           | MD5值                            |
-|:---------------------------------|:--------------------------------:|
-| pytorch_model-00001-of-00007.bin | 8d544b1bcb3449934184d4141137329c |
-| pytorch_model-00002-of-00007.bin | 9d5dbb30911e48a42fb6d0fcabb322a4 |
-| pytorch_model-00003-of-00007.bin | b0d4aecee0457d9332005a187e1fffed |
-| pytorch_model-00004-of-00007.bin | 5c7e002de5eab77d0194a2b0f6de0c24 |
-| pytorch_model-00005-of-00007.bin | d22a511aa26b5b17117b665a877490ab |
-| pytorch_model-00006-of-00007.bin | a5c28ac277fac07d16dd66537e54d109 |
-| pytorch_model-00007-of-00007.bin | a967e2c6195477b7407089c0bffa2d53 |
+```

config.json

@@ -1,29 +0,0 @@
-{
-  "_name_or_path": "/mnt/user/qumu/download_models/codellama/CodeLlama-34b-Python-hf",
-  "architectures": [
-    "LlamaForCausalLM"
-  ],
-  "bos_token_id": 1,
-  "eos_token": "</s>",
-  "eos_token_id": 2,
-  "hidden_act": "silu",
-  "hidden_size": 8192,
-  "initializer_range": 0.02,
-  "intermediate_size": 22016,
-  "max_position_embeddings": 16384,
-  "model_type": "llama",
-  "num_attention_heads": 64,
-  "num_hidden_layers": 48,
-  "num_key_value_heads": 8,
-  "pad_token": "<unk>",
-  "pad_token_id": 0,
-  "pretraining_tp": 1,
-  "rms_norm_eps": 1e-05,
-  "rope_scaling": null,
-  "rope_theta": 1000000,
-  "tie_word_embeddings": false,
-  "torch_dtype": "bfloat16",
-  "transformers_version": "4.32.0",
-  "use_cache": true,
-  "vocab_size": 32000
-}

generation_config.json

@@ -1,6 +0,0 @@
-{
-  "_from_model_config": true,
-  "bos_token_id": 1,
-  "eos_token_id": 2,
-  "transformers_version": "4.32.0"
-}

pytorch_model-00001-of-00007.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:fb1cd3c944b6dd2ece21c09d7378a77ccc279867dfe4b8b8b7cbfa58e4ffe4a3
-size 9852638393

pytorch_model-00002-of-00007.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1641f834b6d3938353873d95b642d0083ca806b7fd32dc8aaea0b3c64752b358
-size 9689094033

pytorch_model-00003-of-00007.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:38ce1d749b110332ffbfc237c3271c10229f0913ed3e424aa61c4261af033357
-size 9689094033

pytorch_model-00004-of-00007.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a5c7a36d9f48fdd4b629f44d6e238085fa727f393c94e6cdf17d4f5da19be613
-size 9689094033

pytorch_model-00005-of-00007.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:7ccdaa07cce1d16ed6d8ca1ca773b075f61c36d04430f8683b6e2bf458be4a72
-size 9689094033

pytorch_model-00006-of-00007.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:a257280910a701168406cc2114a669a975176ec08a373f7c2b7ba94d22c05215
-size 9689094033

pytorch_model-00007-of-00007.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:aaef8ce33ccbc255d78026458d81fc9a55513a0d1ac3ef4f4f74967300af49e7
-size 9189986713

pytorch_model.bin.index.json

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:9a714170172282cfbcaa120af13c0df08b06d040ff24dab30229d8a010821d3d
-size 35838

requirements.txt

@@ -1,14 +0,0 @@
-numpy
-pandas
-einops
-sentencepiece
-deepspeed==0.9.3
-transformers==4.32.0
-accelerate==0.21.0
-peft==0.4.0
-BitsAndBytes==0.40.2
-xformers==0.0.21
-ujson
-jsonlines
-tiktoken
-transformers_stream_generator

special_tokens_map.json

@@ -1,23 +0,0 @@
-{
-  "bos_token": {
-    "content": "<s>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "eos_token": {
-    "content": "</s>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "unk_token": {
-    "content": "<unk>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  }
-}

tokenizer.model

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
-size 499723

tokenizer_config.json

@@ -1,33 +0,0 @@
-{
-  "bos_token": {
-    "__type": "AddedToken",
-    "content": "<s>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "clean_up_tokenization_spaces": false,
-  "eos_token": {
-    "__type": "AddedToken",
-    "content": "</s>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "legacy": null,
-  "model_max_length": 1000000000000000019884624838656,
-  "pad_token": null,
-  "sp_model_kwargs": {},
-  "tokenizer_class": "LlamaTokenizer",
-  "unk_token": {
-    "__type": "AddedToken",
-    "content": "<unk>",
-    "lstrip": false,
-    "normalized": true,
-    "rstrip": false,
-    "single_word": false
-  },
-  "use_default_system_prompt": true
-}