polm commited on
Commit
3bc40d1
1 Parent(s): 6b26795

Add README etc.

Browse files
README.md CHANGED
@@ -1,3 +1,132 @@
1
  ---
2
- license: llama2
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - ja
4
+ tags:
5
+ - japanese-stablelm
6
+ - causal-lm
7
+ pipeline_tag: text-generation
8
+ datasets:
9
+ - kunishou/hh-rlhf-49k-ja
10
+ - kunishou/databricks-dolly-15k-ja
11
+ - kunishou/oasst1-89k-ja
12
+ license:
13
+ - llama2
14
+ extra_gated_fields:
15
+ Name: text
16
+ Email: text
17
+ Country: text
18
+ Organization or Affiliation: text
19
+ I allow Stability AI to contact me about information related to its models and research: checkbox
20
  ---
21
+
22
+ # Japanese-StableLM-Instruct-JAVocab-Beta-7B
23
+
24
+ ![A cute robot wearing a kimono writes calligraphy with one single brush](./japanese-stablelm-robot.jpg)
25
+
26
+ > A cute robot wearing a kimono writes calligraphy with one single brush — [Stable Diffusion XL](https://clipdrop.co/stable-diffusion)
27
+
28
+ ## Model Description
29
+
30
+ `japanese-stablelm-instruct-ja_vocab-beta-7b` is a 7B-parameter decoder-only language model based on [japanese-stablelm-ja_vocab-beta-7b](https://huggingface.co/stabilityai/japanese-stablelm-instruct-beta-7b) and further fine tuned on Databricks Dolly-15k, Anthropic HH, and other public data.
31
+
32
+ Compared to the [standard base model](https://huggingface.co/stabilityai/japanese-stablelm-base-beta-7b), this model uses a tokenizer with an expanded vocabulary derived from Japanese data. This allows it to represent the same amount of text with fewer tokens, which speeds up inference significantly.
33
+
34
+ ## Usage
35
+
36
+ First install additional dependencies in [requirements.txt](./requirements.txt):
37
+
38
+ ```sh
39
+ pip install -r requirements.txt
40
+ ```
41
+
42
+ Then start generating text with `japanese-stablelm-instruct-ja_vocab-beta-7b` by using the following code snippet:
43
+
44
+ ```python
45
+ import torch
46
+ from transformers import AutoTokenizer, AutoModelForCausalLM
47
+
48
+ model_name = "stabilityai/japanese-stablelm-instruct-ja_vocab-beta-7b"
49
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
50
+
51
+ # The next line may need to be modified depending on the environment
52
+ model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map="auto")
53
+
54
+ def build_prompt(user_query, inputs):
55
+ sys_msg = "<s>[INST] <<SYS>>\nあなたは役立つアシスタントです。\n<<SYS>>\n\n"
56
+ p = sys_msg + user_query + "\n\n" + inputs + " [/INST] "
57
+ return p
58
+
59
+ user_inputs = {
60
+ "user_query": "与えられたことわざの意味を小学生でも分かるように教えてください。",
61
+ "inputs": "情けは人のためならず"
62
+ }
63
+ prompt = build_prompt(**user_inputs)
64
+
65
+ input_ids = tokenizer.encode(
66
+ prompt,
67
+ add_special_tokens=False,
68
+ return_tensors="pt"
69
+ )
70
+
71
+ # this is for reproducibility.
72
+ # feel free to change to get different result
73
+ seed = 23
74
+ torch.manual_seed(seed)
75
+
76
+ tokens = model.generate(
77
+ input_ids.to(device=model.device),
78
+ max_new_tokens=128,
79
+ temperature=0.99,
80
+ top_p=0.95,
81
+ do_sample=True,
82
+ )
83
+
84
+ out = tokenizer.decode(tokens[0], skip_special_tokens=True)
85
+ print(out)
86
+ ```
87
+
88
+ We suggest playing with different generation config (`top_p`, `repetition_penalty` etc) to find the best setup for your tasks. For example, use higher temperature for roleplay task, lower temperature for reasoning.
89
+
90
+ ## Model Details
91
+
92
+ * **Model type**: `japanese-stablelm-instruct-ja_vocab-beta-7b` model is an auto-regressive language model based on the Llama2 transformer architecture.
93
+ * **Language(s)**: Japanese
94
+ * **License**: [Llama2 Community License](https://ai.meta.com/llama/license/).
95
+ * **Contact**: For questions and comments about the model, please join [Stable Community Japan](https://discord.gg/StableJP). For future announcements / information about Stability AI models, research, and events, please follow https://twitter.com/StabilityAI_JP.
96
+
97
+ ## Training Dataset
98
+
99
+ The following datasets were used for the instruction training. Note these are Japanese translated versions of the original datasets, shared by [kunishou](https://huggingface.co/kunishou).
100
+
101
+ - [Anthropic HH-RLHF](https://huggingface.co/datasets/kunishou/hh-rlhf-49k-ja)
102
+ - [Databricks Dolly 15-k](https://huggingface.co/datasets/kunishou/databricks-dolly-15k-ja)
103
+ - [OpenAssistant Conversations Dataset](https://huggingface.co/datasets/kunishou/oasst1-89k-ja)
104
+
105
+ ## Use and Limitations
106
+
107
+ ### Intended Use
108
+
109
+ The model is intended to be used by all individuals as a foundation for application-specific fine-tuning without strict limitations on commercial use.
110
+
111
+ ### Limitations and bias
112
+
113
+ The pre-training dataset may have contained offensive or inappropriate content even after applying data cleansing filters which can be reflected in the model generated text. We recommend users exercise reasonable caution when using these models in production systems. Do not use the model for any applications that may cause harm or distress to individuals or groups.
114
+
115
+ ## Authors
116
+ This model was developed by the Research & Development team at Stability AI Japan, and the development was co-led by [Takuya Akiba](https://huggingface.co/iwiwi) and [Meng Lee](https://huggingface.co/leemeng). The members of the team are as follows:
117
+
118
+ - [Meng Lee](https://huggingface.co/leemeng)
119
+ - [Fujiki Nakamura](https://huggingface.co/fujiki)
120
+ - [Makoto Shing](https://huggingface.co/mkshing)
121
+ - [Paul McCann](https://huggingface.co/polm-stability)
122
+ - [Takuya Akiba](https://huggingface.co/iwiwi)
123
+ - [Naoki Orii](https://huggingface.co/mrorii)
124
+
125
+ ## Acknowledgements
126
+
127
+ We thank Meta Research for releasing Llama 2 under an open license for others to build on.
128
+
129
+ We are grateful for the contributions of the EleutherAI Polyglot-JA team in helping us to collect a large amount of pre-training data in Japanese. Polyglot-JA members includes Hyunwoong Ko (Project Lead), Fujiki Nakamura (originally started this project when he commited to the Polyglot team), Yunho Mo, Minji Jung, KeunSeok Im, and Su-Kyeong Jang.
130
+
131
+ We are also appreciative of [AI Novelist/Sta (Bit192, Inc.)](https://ai-novel.com/index.php) and the numerous contributors from [Stable Community Japan](https://discord.gg/VPrcE475HB) for assisting us in gathering a large amount of high-quality Japanese textual data for model training.
132
+
config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "bos_token_id": 1,
6
+ "eos_token_id": 2,
7
+ "hidden_act": "silu",
8
+ "hidden_size": 4096,
9
+ "initializer_range": 0.02,
10
+ "intermediate_size": 11008,
11
+ "max_position_embeddings": 4096,
12
+ "model_type": "llama",
13
+ "num_attention_heads": 32,
14
+ "num_hidden_layers": 32,
15
+ "num_key_value_heads": 32,
16
+ "pad_token_id": 0,
17
+ "pretraining_tp": 1,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_scaling": null,
20
+ "tie_word_embeddings": false,
21
+ "torch_dtype": "float16",
22
+ "transformers_version": "4.31.0",
23
+ "use_cache": true,
24
+ "vocab_size": 49408
25
+ }
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 0,
6
+ "transformers_version": "4.31.0"
7
+ }
japanese-stablelm-robot.jpg ADDED
requirements.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ sentencepiece
2
+ protobuf
3
+ accelerate