KaleiNeely commited on
Commit
c223906
1 Parent(s): 0bd3c4b

Upload 10 files

Browse files
README.md ADDED
@@ -0,0 +1,173 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ### Run Huggingface RWKV6 World Model
2
+
3
+
4
+ #### CPU
5
+
6
+ ```python
7
+ import torch
8
+ from transformers import AutoModelForCausalLM, AutoTokenizer
9
+
10
+ def generate_prompt(instruction, input=""):
11
+ instruction = instruction.strip().replace('\r\n','\n').replace('\n\n','\n')
12
+ input = input.strip().replace('\r\n','\n').replace('\n\n','\n')
13
+ if input:
14
+ return f"""Instruction: {instruction}
15
+
16
+ Input: {input}
17
+
18
+ Response:"""
19
+ else:
20
+ return f"""User: hi
21
+
22
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
23
+
24
+ User: {instruction}
25
+
26
+ Assistant:"""
27
+
28
+
29
+ model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-5-world-3b", trust_remote_code=True).to(torch.float32)
30
+ tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-5-world-3b", trust_remote_code=True, padding_side='left', pad_token="<s>")
31
+
32
+ text = "请介绍北京的旅游景点"
33
+ prompt = generate_prompt(text)
34
+
35
+ inputs = tokenizer(prompt, return_tensors="pt")
36
+ output = model.generate(inputs["input_ids"], max_new_tokens=333, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )
37
+ print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))
38
+ ```
39
+
40
+ output:
41
+
42
+ ```shell
43
+ User: hi
44
+
45
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
46
+
47
+ User: 请介绍北京的旅游景点
48
+
49
+ Assistant: 北京是中国的首都,拥有众多的旅游景点,以下是其中一些著名的景点:
50
+ 1. 故宫:位于北京市中心,是明清两代的皇宫,内有大量的文物和艺术品。
51
+ 2. 天安门广场:是中国最著名的广场之一,是中国人民政治协商会议的旧址,也是中国人民政治协商会议的中心。
52
+ 3. 颐和园:是中国古代皇家园林之一,有着悠久的历史和丰富的文化内涵。
53
+ 4. 长城:是中国古代的一道长城,全长约万里,是中国最著名的旅游景点之一。
54
+ 5. 北京大学:是中国著名的高等教育机构之一,有着悠久的历史和丰富的文化内涵。
55
+ 6. 北京动物园:是中国最大的动物园之一,有着丰富的动物资源和丰富的文化内涵。
56
+ 7. 故宫博物院:是中国最著名的博物馆之一,收藏了大量的文物和艺术品,是中国最重要的文化遗产之一。
57
+ 8. 天坛:是中国古代皇家
58
+ ```
59
+
60
+ #### GPU
61
+
62
+ ```python
63
+ import torch
64
+ from transformers import AutoModelForCausalLM, AutoTokenizer
65
+
66
+ def generate_prompt(instruction, input=""):
67
+ instruction = instruction.strip().replace('\r\n','\n').replace('\n\n','\n')
68
+ input = input.strip().replace('\r\n','\n').replace('\n\n','\n')
69
+ if input:
70
+ return f"""Instruction: {instruction}
71
+
72
+ Input: {input}
73
+
74
+ Response:"""
75
+ else:
76
+ return f"""User: hi
77
+
78
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
79
+
80
+ User: {instruction}
81
+
82
+ Assistant:"""
83
+
84
+
85
+ model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-5-world-3b", trust_remote_code=True, torch_dtype=torch.float16).to(0)
86
+ tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-5-world-3b", trust_remote_code=True, padding_side='left', pad_token="<s>")
87
+
88
+ text = "介绍一下大熊猫"
89
+ prompt = generate_prompt(text)
90
+
91
+ inputs = tokenizer(prompt, return_tensors="pt").to(0)
92
+ output = model.generate(inputs["input_ids"], max_new_tokens=128, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )
93
+ print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))
94
+ ```
95
+
96
+ output:
97
+
98
+ ```shell
99
+ User: hi
100
+
101
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
102
+
103
+ User: 介绍一下大熊猫
104
+
105
+ Assistant: 大熊猫是一种中国特有的哺乳动物,也是中国的国宝之一。它们的外貌特征是圆形的黑白相间的身体,有着黑色的毛发和白色的耳朵。大熊猫的食物主要是竹子,它们会在竹林中寻找竹子,并且会将竹子放在竹笼中进行储存。大熊猫的寿命约为20至30年,但由于栖息地的丧失和人类活动的
106
+ ```
107
+
108
+ #### Batch Inference
109
+
110
+ ```python
111
+ import torch
112
+ from transformers import AutoModelForCausalLM, AutoTokenizer
113
+
114
+ def generate_prompt(instruction, input=""):
115
+ instruction = instruction.strip().replace('\r\n', '\n').replace('\n\n', '\n')
116
+ input = input.strip().replace('\r\n', '\n').replace('\n\n', '\n')
117
+ if input:
118
+ return f"""Instruction: {instruction}
119
+
120
+ Input: {input}
121
+
122
+ Response:"""
123
+ else:
124
+ return f"""User: hi
125
+
126
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
127
+
128
+ User: {instruction}
129
+
130
+ Assistant:"""
131
+
132
+ model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-5-world-3b", trust_remote_code=True).to(torch.float32)
133
+ tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-5-world-3b", trust_remote_code=True, padding_side='left', pad_token="<s>")
134
+
135
+ texts = ["请介绍北京的旅游景点", "介绍一下大熊猫", "乌兰察布"]
136
+ prompts = [generate_prompt(text) for text in texts]
137
+
138
+ inputs = tokenizer(prompts, return_tensors="pt", padding=True)
139
+ outputs = model.generate(inputs["input_ids"], max_new_tokens=128, do_sample=True, temperature=1.0, top_p=0.3, top_k=0, )
140
+
141
+ for output in outputs:
142
+ print(tokenizer.decode(output.tolist(), skip_special_tokens=True))
143
+
144
+ ```
145
+
146
+ output:
147
+
148
+ ```shell
149
+ User: hi
150
+
151
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
152
+
153
+ User: 请介绍北京的旅游景点
154
+
155
+ Assistant: 北京是中国的首都,拥有丰富的旅游资源和历史文化遗产。以下是一些北京的旅游景点:
156
+ 1. 故宫:位于北京市中心,是明清两代的皇宫,是中国最大的古代宫殿建筑群之一。
157
+ 2. 天安门广场:位于北京市中心,是中国最著名的城市广场之一,也是中国最大的城市广场。
158
+ 3. 颐和
159
+ User: hi
160
+
161
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
162
+
163
+ User: 介绍一下大熊猫
164
+
165
+ Assistant: 大熊猫是一种生活在中国中部地区的哺乳动物,也是中国的国宝之一。它们的外貌特征是圆形的黑白相间的身体,有着黑色的毛发和圆圆的眼睛。大熊猫是一种濒危物种,目前只有在野外的几个保护区才能看到它们的身影。大熊猫的食物主要是竹子,它们会在竹子上寻找食物,并且可以通
166
+ User: hi
167
+
168
+ Assistant: Hi. I am your assistant and I will provide expert full response in full details. Please feel free to ask any question and I will always answer it.
169
+
170
+ User: 乌兰察布
171
+
172
+ Assistant: 乌兰察布是中国新疆维吾尔自治区的一个县级市,位于新疆维吾尔自治区中部,是新疆的第二大城市。乌兰察布市是新疆的第一大城市,也是新疆的重要城市之一。乌兰察布市是新疆的经济中心,也是新疆的重要交通枢纽之一。乌兰察布市的人口约为2.5万人,其中汉族占绝大多数。乌
173
+ ```
added_tokens.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {
2
+ "<s>": 0
3
+ }
config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Rwkv6ForCausalLM"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration_rwkv6.Rwkv6Config",
7
+ "AutoModelForCausalLM": "modeling_rwkv6.Rwkv6ForCausalLM"
8
+ },
9
+ "attention_hidden_size": 2560,
10
+ "bos_token_id": 0,
11
+ "eos_token_id": 0,
12
+ "head_size": 64,
13
+ "head_size_divisor": 8,
14
+ "hidden_size": 2560,
15
+ "intermediate_size": null,
16
+ "layer_norm_epsilon": 1e-05,
17
+ "model_type": "rwkv6",
18
+ "num_attention_heads": 64,
19
+ "num_hidden_layers": 32,
20
+ "rescale_every": 6,
21
+ "tie_word_embeddings": false,
22
+ "transformers_version": "4.34.0",
23
+ "use_cache": true,
24
+ "vocab_size": 65536
25
+ }
configuration_rwkv6.py ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2023 The OpenAI Team Authors and HuggingFace Inc. team.
3
+ # Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+ """ RWKV configuration"""
17
+
18
+ from transformers.configuration_utils import PretrainedConfig
19
+ from transformers.utils import logging
20
+
21
+
22
+ logger = logging.get_logger(__name__)
23
+
24
+ RWKV6_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
25
+
26
+
27
+ class Rwkv6Config(PretrainedConfig):
28
+ """
29
+ This is the configuration class to store the configuration of a [`Rwkv6Model`]. It is used to instantiate a RWKV6
30
+ model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
31
+ defaults will yield a similar configuration to that of the RWVK-4
32
+ [RWKV/rwkv-5-world-1b5](https://huggingface.co/RWKV/rwkv-5-world-1b5) architecture.
33
+
34
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
35
+ documentation from [`PretrainedConfig`] for more information.
36
+
37
+
38
+ Args:
39
+ vocab_size (`int`, *optional*, defaults to 65536):
40
+ Vocabulary size of the RWKV6 model. Defines the number of different tokens that can be represented by the
41
+ `inputs_ids` passed when calling [`Rwkv6Model`].
42
+ hidden_size (`int`, *optional*, defaults to 768):
43
+ Dimensionality of the embeddings and hidden states.
44
+ num_hidden_layers (`int`, *optional*, defaults to 24):
45
+ Number of hidden layers in the model.
46
+ attention_hidden_size (`int`, *optional*):
47
+ Dimensionality of the attention hidden states. Will default to `hidden_size` if unset.
48
+ num_attention_heads (`int`, *optional*, defaults to 64):
49
+ The attention heads to use in rwkv6 self_attention module.
50
+ head_size (`int`, *optional*, defaults to 64): head_size of rwkv6 self_attention module.
51
+ intermediate_size (`int`, *optional*):
52
+ Dimensionality of the inner feed-forward layers. Will default to 4 times `hidden_size` if unset.
53
+ layer_norm_epsilon (`float`, *optional*, defaults to 1e-05):
54
+ The epsilon to use in the layer normalization layers.
55
+ bos_token_id (`int`, *optional*, defaults to 0):
56
+ The id of the beginning of sentence token in the vocabulary. Defaults to 0.
57
+ eos_token_id (`int`, *optional*, defaults to 0):
58
+ The id of the end of sentence token in the vocabulary. Defaults to 0.
59
+ rescale_every (`int`, *optional*, defaults to 6):
60
+ At inference, the hidden states (and weights of the correponding output layers) are divided by 2 every
61
+ `rescale_every` layer. If set to 0 or a negative number, no rescale is done.
62
+ tie_word_embeddings (`bool`, *optional*, defaults to `False`):
63
+ Whether or not to tie the word embeddings with the input token embeddings.
64
+ use_cache (`bool`, *optional*, defaults to `True`):
65
+ Whether or not the model should return the last state.
66
+
67
+
68
+ Example:
69
+
70
+ ```python
71
+ >>> from transformers import Rwkv6Config, Rwkv6Model
72
+
73
+ >>> # Initializing a Rwkv6 configuration
74
+ >>> configuration = Rwkv6Config()
75
+
76
+ >>> # Initializing a model (with random weights) from the configuration
77
+ >>> model = Rwkv6Model(configuration)
78
+
79
+ >>> # Accessing the model configuration
80
+ >>> configuration = model.config
81
+ ```"""
82
+
83
+ model_type = "rwkv6"
84
+
85
+ def __init__(
86
+ self,
87
+ vocab_size=65536,
88
+ hidden_size=768,
89
+ num_hidden_layers=24,
90
+ attention_hidden_size=None,
91
+ head_size=64,
92
+ head_size_divisor=8,
93
+ intermediate_size=None,
94
+ layer_norm_epsilon=1e-5,
95
+ bos_token_id=0,
96
+ eos_token_id=0,
97
+ rescale_every=6,
98
+ tie_word_embeddings=False,
99
+ use_cache=True,
100
+ **kwargs,
101
+ ):
102
+ self.vocab_size = vocab_size
103
+ self.hidden_size = hidden_size
104
+ self.num_hidden_layers = num_hidden_layers
105
+ self.attention_hidden_size = attention_hidden_size if attention_hidden_size is not None else hidden_size
106
+ self.head_size = head_size
107
+ self.head_size_divisor = head_size_divisor
108
+ self.intermediate_size = None
109
+ self.layer_norm_epsilon = layer_norm_epsilon
110
+ self.rescale_every = rescale_every
111
+ self.use_cache = use_cache
112
+
113
+ self.bos_token_id = bos_token_id
114
+ self.eos_token_id = eos_token_id
115
+
116
+ super().__init__(
117
+ tie_word_embeddings=tie_word_embeddings, bos_token_id=bos_token_id, eos_token_id=eos_token_id, **kwargs
118
+ )
generation_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "chat_format": "chatml",
3
+ "eos_token_id": 0,
4
+ "pad_token_id": 0,
5
+ "max_window_size": 4096,
6
+ "max_new_tokens": 4096,
7
+ "do_sample": true,
8
+ "top_k": 0,
9
+ "top_p": 0.1,
10
+ "repetition_penalty": 1.0,
11
+ "transformers_version": "4.31.1"
12
+ }
modeling_rwkv6.py ADDED
@@ -0,0 +1,824 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2024 The RWKV team and HuggingFace Inc. team.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """PyTorch RWKV6 World model."""
16
+
17
+ from dataclasses import dataclass
18
+ from typing import List, Optional, Tuple, Union
19
+
20
+ from pathlib import Path
21
+
22
+ import torch
23
+ import torch.nn.functional as F
24
+ import torch.utils.checkpoint
25
+ from torch import nn
26
+ from torch.nn import CrossEntropyLoss
27
+
28
+ from transformers.modeling_utils import PreTrainedModel
29
+ from transformers.utils import (
30
+ ModelOutput,
31
+ add_code_sample_docstrings,
32
+ add_start_docstrings,
33
+ add_start_docstrings_to_model_forward,
34
+ is_ninja_available,
35
+ is_torch_cuda_available,
36
+ logging,
37
+ )
38
+
39
+ from .configuration_rwkv6 import Rwkv6Config
40
+
41
+
42
+ logger = logging.get_logger(__name__)
43
+
44
+ _CHECKPOINT_FOR_DOC = "RWKV/rwkv-6-world-1b6"
45
+ _CONFIG_FOR_DOC = "Rwkv6Config"
46
+
47
+ rwkv6_cuda_kernel = None
48
+
49
+ def load_wkv6_cuda_kernel(head_size, ctx_len):
50
+ from torch.utils.cpp_extension import load as load_kernel
51
+
52
+ global rwkv6_cuda_kernel
53
+
54
+ kernel_folder = Path(__file__).parent.resolve()
55
+ cuda_kernel_files = [kernel_folder / f for f in ["wkv6_op.cpp", "wkv6_cuda.cu"]]
56
+
57
+ # Only load the kernel if it's not been loaded yet or if we changed the context length
58
+ if rwkv6_cuda_kernel is not None and rwkv6_cuda_kernel.head_size == head_size:
59
+ return
60
+
61
+ logger.info(f"Loading CUDA kernel for RWKV at head size of {head_size}.")
62
+
63
+ flags = [
64
+ "-res-usage",
65
+ # "--maxrregcount 60", # not sure, should we add this? its not in RWKV-LM
66
+ "--use_fast_math",
67
+ "-O3",
68
+ "-Xptxas -O3",
69
+ "--extra-device-vectorization",
70
+ f"-D_N_={head_size}",
71
+ f"-D_T_={ctx_len}"
72
+ ]
73
+ rwkv6_cuda_kernel = load_kernel(
74
+ name=f"wkv_{head_size}_{ctx_len}",
75
+ sources=cuda_kernel_files,
76
+ verbose=(logging.get_verbosity() == logging.DEBUG),
77
+ extra_cuda_cflags=flags,
78
+ )
79
+ rwkv6_cuda_kernel.head_size = head_size
80
+ rwkv6_cuda_kernel.ctx_len = ctx_len
81
+
82
+
83
+ class Rwkv6LinearAttention(torch.autograd.Function):
84
+ @staticmethod
85
+ def forward(ctx, receptance, key, value, time_decay, time_first, state):
86
+ with torch.no_grad():
87
+ assert receptance.dtype == torch.bfloat16
88
+ assert key.dtype == torch.bfloat16
89
+ assert value.dtype == torch.bfloat16
90
+ assert time_decay.dtype == torch.bfloat16
91
+ assert time_first.dtype == torch.bfloat16
92
+ assert state.dtype == torch.float32
93
+ #assert HEAD_SIZE == C // H
94
+ Batch, SequenceLength, HiddenSize = key.shape
95
+ NumHeads, HeadSize = time_decay.shape
96
+ ctx.Batch = Batch
97
+ ctx.SequenceLength = SequenceLength
98
+ ctx.HiddenSize = HiddenSize
99
+ ctx.NumHeads = NumHeads
100
+ assert receptance.is_contiguous()
101
+ assert key.is_contiguous()
102
+ assert value.is_contiguous()
103
+ assert time_decay.is_contiguous()
104
+ assert time_first.is_contiguous()
105
+ e_time_decay = (-torch.exp(time_decay.float())).contiguous()
106
+ ctx.save_for_backward(receptance, key, value, e_time_decay, time_first)
107
+ out = torch.empty((Batch, SequenceLength, HiddenSize), device=receptance.device, dtype=torch.bfloat16, memory_format=torch.contiguous_format)#.uniform_(-100, 100)
108
+ # FIXME - current kernel does not handle nor update state
109
+ rwkv6_cuda_kernel.forward(Batch, SequenceLength, HiddenSize, NumHeads, receptance, key, value, e_time_decay, time_first, out)
110
+ return out, state
111
+
112
+ @staticmethod
113
+ def backward(ctx, g_out, g_state):
114
+ with torch.no_grad():
115
+ assert g_out.dtype == torch.bfloat16
116
+ Batch = ctx.Batch
117
+ SequenceLength = ctx.SequenceLength
118
+ HiddenSize = ctx.HiddenSize
119
+ NumHeads = ctx.NumHeads
120
+ HeadSize = HiddenSize // NumHeads
121
+ assert g_out.is_contiguous()
122
+ receptance, key, value, e_time_decay, time_first = ctx.saved_tensors
123
+ g_receptance = torch.empty((B, T, C), device=gy.device, requires_grad=False, dtype=torch.bfloat16, memory_format=torch.contiguous_format)#.uniform_(-100, 100)
124
+ g_key = torch.empty((B, T, C), device=gy.device, requires_grad=False, dtype=torch.bfloat16, memory_format=torch.contiguous_format)#.uniform_(-100, 100)
125
+ g_value = torch.empty((B, T, C), device=gy.device, requires_grad=False, dtype=torch.bfloat16, memory_format=torch.contiguous_format)#.uniform_(-100, 100)
126
+ g_time_decay = torch.empty((B, T, C), device=gy.device, requires_grad=False, dtype=torch.bfloat16, memory_format=torch.contiguous_format)#.uniform_(-100, 100)
127
+ g_time_first = torch.empty((B, C), device=gy.device, requires_grad=False, dtype=torch.bfloat16, memory_format=torch.contiguous_format)#.uniform_(-100, 100)
128
+ #gs = torch.empty((B, C//H, H, H), device=gy.device, requires_grad=False, dtype=torch.float, memory_format=torch.contiguous_format)#.uniform_(-100, 100)
129
+ rwkv6_cuda_kernel.backward(B, T, C, H, receptance, key, value, e_time_decay, time_first, g_out, g_receptance, g_key, g_value, g_time_decay, g_time_first)
130
+ g_time_first = torch.sum(g_time_first, 0).view(NumHeads, HeadSize)
131
+ return (None, None, None, None, g_receptance, g_key, g_value, g_time_decay, g_time_first, None)
132
+
133
+ def rwkv6_linear_attention_cpu(receptance, key, value, time_decay, time_first, state):
134
+ input_dtype = receptance.dtype
135
+ # For CPU fallback. Will be slower and probably take more memory than the custom CUDA kernel if not executed
136
+ # within a torch.no_grad.
137
+ batch, seq_length, hidden_size = receptance.shape
138
+ num_heads, head_size = time_first.shape
139
+ key = key.float().view(batch, seq_length, num_heads, head_size).transpose(1, 2).transpose(-2, -1)
140
+ value = value.float().view(batch, seq_length, num_heads, head_size).transpose(1, 2)
141
+ receptance = receptance.float().view(batch, seq_length, num_heads, head_size).transpose(1, 2)
142
+ time_decay = torch.exp(-torch.exp(time_decay.float())).view(batch, seq_length, num_heads, head_size).permute(0, 2, 3, 1) # B, H, S, T
143
+ time_first = time_first.float().reshape(-1, 1, 1).reshape(num_heads, -1, 1)
144
+ out = torch.zeros_like(key).reshape(batch, seq_length, num_heads, head_size)
145
+
146
+ for current_index in range(seq_length):
147
+ current_receptance = receptance[:, :, current_index:current_index+1, :]
148
+ current_key = key[:, :, :, current_index:current_index+1]
149
+ current_value = value[:, :, current_index:current_index+1, :]
150
+ current_time_decay = time_decay[:, :, :, current_index:current_index+1]
151
+ attention_output = current_key @ current_value
152
+ out[:, current_index] = (current_receptance @ (time_first * attention_output + state)).squeeze(2)
153
+ with torch.no_grad():
154
+ state = attention_output + current_time_decay * state
155
+
156
+ return out, state
157
+
158
+ def rwkv6_linear_attention(
159
+ training,
160
+ receptance,
161
+ key,
162
+ value,
163
+ time_decay,
164
+ time_first,
165
+ state,
166
+ ):
167
+ no_cuda = any(t.device.type != "cuda" for t in [time_decay, time_first, receptance, key, value])
168
+ # Launching the CUDA kernel for just one token will actually be slower (there is no for loop in the CPU version
169
+ # in this case).
170
+ one_token = key.size(1) == 1
171
+ if not training or rwkv6_cuda_kernel is None or no_cuda or one_token:
172
+ return rwkv6_linear_attention_cpu(
173
+ receptance, key, value, time_decay, time_first, state
174
+ )
175
+ else:
176
+ return Rwkv6LinearAttention.apply(receptance, key, value, time_decay, time_first, state)
177
+
178
+
179
+ class Rwkv6SelfAttention(nn.Module):
180
+ def __init__(self, config, layer_id=0):
181
+ super().__init__()
182
+ self.config = config
183
+ kernel_loaded = rwkv6_cuda_kernel is not None and rwkv6_cuda_kernel.head_size == config.head_size
184
+ if is_ninja_available() and is_torch_cuda_available() and not kernel_loaded:
185
+ try:
186
+ load_wkv6_cuda_kernel(config.head_size, config.max_context_length) # FIXME - context_length is not a configured attribute
187
+ except Exception:
188
+ logger.info("Could not load the custom CUDA kernel for RWKV6 attention.")
189
+ self.layer_id = layer_id
190
+ hidden_size = config.hidden_size
191
+ attention_hidden_size = config.attention_hidden_size
192
+ self.attention_hidden_size = attention_hidden_size
193
+ head_size = config.head_size
194
+ num_heads = attention_hidden_size // head_size
195
+
196
+ self.time_maa_x = nn.Parameter(torch.empty(1, 1, hidden_size))
197
+ self.time_maa_w = nn.Parameter(torch.empty(1, 1, hidden_size))
198
+ self.time_maa_k = nn.Parameter(torch.empty(1, 1, hidden_size))
199
+ self.time_maa_v = nn.Parameter(torch.empty(1, 1, hidden_size))
200
+ self.time_maa_r = nn.Parameter(torch.empty(1, 1, hidden_size))
201
+ self.time_maa_g = nn.Parameter(torch.empty(1, 1, hidden_size))
202
+
203
+ TIME_MIX_EXTRA_DIM = 32 # generate TIME_MIX for w,k,v,r,g
204
+ self.time_maa_w1 = nn.Parameter(torch.empty(hidden_size, TIME_MIX_EXTRA_DIM*5))
205
+ self.time_maa_w2 = nn.Parameter(torch.empty(5, TIME_MIX_EXTRA_DIM, hidden_size))
206
+
207
+ self.time_decay = nn.Parameter(torch.empty(1, 1, attention_hidden_size))
208
+
209
+ TIME_DECAY_EXTRA_DIM = 64
210
+ self.time_decay_w1 = nn.Parameter(torch.empty(hidden_size, TIME_DECAY_EXTRA_DIM))
211
+ self.time_decay_w2 = nn.Parameter(torch.empty(TIME_DECAY_EXTRA_DIM, attention_hidden_size))
212
+
213
+ self.time_faaaa = nn.Parameter(torch.empty(num_heads, config.head_size))
214
+
215
+
216
+ self.time_shift = nn.ZeroPad2d((0, 0, 1, -1))
217
+ self.receptance = nn.Linear(hidden_size, attention_hidden_size, bias=False)
218
+ self.key = nn.Linear(hidden_size, attention_hidden_size, bias=False)
219
+ self.value = nn.Linear(hidden_size, attention_hidden_size, bias=False)
220
+ self.gate = nn.Linear(hidden_size, attention_hidden_size, bias=False)
221
+ self.output = nn.Linear(attention_hidden_size, hidden_size, bias=False)
222
+ self.ln_x = nn.GroupNorm(num_heads, hidden_size, eps=(1e-5)*(config.head_size_divisor**2))
223
+
224
+ def extract_key_value(self, hidden, state=None):
225
+ # Mix hidden with the previous timestep to produce key, value, receptance
226
+ if hidden.size(1) == 1 and state is not None:
227
+ shifted = state[0][:, :, self.layer_id]
228
+ else:
229
+ shifted = self.time_shift(hidden)
230
+ if state is not None:
231
+ shifted[:, 0] = state[0][:, :, self.layer_id]
232
+ if len(shifted.size()) == 2:
233
+ shifted = shifted.unsqueeze(1)
234
+
235
+ x = hidden
236
+
237
+ B, T, C = hidden.shape
238
+
239
+ xx = shifted - x
240
+
241
+ xxx = x + xx * self.time_maa_x
242
+ xxx = torch.tanh(xxx @ self.time_maa_w1).view(B*T, 5, -1).transpose(0, 1)
243
+ xxx = torch.bmm(xxx, self.time_maa_w2).view(5, B, T, -1)
244
+ mw, mk, mv, mr, mg = xxx.unbind(dim=0)
245
+
246
+ time_decay = x + xx * (self.time_maa_w + mw)
247
+ key = x + xx * (self.time_maa_k + mk)
248
+ value = x + xx * (self.time_maa_v + mv)
249
+ receptance = x + xx * (self.time_maa_r + mr)
250
+ gate = x + xx * (self.time_maa_g + mg)
251
+
252
+ receptance = self.receptance(receptance)
253
+ key = self.key(key)
254
+ value = self.value(value)
255
+ gate = F.silu(self.gate(gate))
256
+
257
+ time_decay = torch.tanh(time_decay @ self.time_decay_w1) @ self.time_decay_w2
258
+ time_decay = self.time_decay + time_decay
259
+
260
+ if state is not None:
261
+ state[0][:, :, self.layer_id] = hidden[:, -1]
262
+
263
+ return receptance, key, value, gate, time_decay, state
264
+
265
+ def forward(self, hidden, state=None, use_cache=False, seq_mode=True):
266
+ receptance, key, value, gate, time_decay, state = self.extract_key_value(hidden, state=state)
267
+
268
+ B,T,C = receptance.shape
269
+ H, S = self.time_faaaa.shape
270
+
271
+ layer_state = state[1][:, :, :, :, self.layer_id] if state is not None else None
272
+ out, layer_state = rwkv6_linear_attention(
273
+ self.training, receptance, key, value, time_decay, self.time_faaaa, layer_state,
274
+ )
275
+
276
+ if layer_state is not None:
277
+ state[1][:, :, :, :, self.layer_id] = layer_state
278
+
279
+ out = out.reshape(B * T, H * S)
280
+ out = F.group_norm(out, num_groups=H, weight=self.ln_x.weight.to(out.dtype), bias=self.ln_x.bias.to(out.dtype), eps=self.ln_x.eps).reshape(B, T, H * S)
281
+ out = out.to(dtype=hidden.dtype) * gate
282
+ out = self.output(out)
283
+ return out, state
284
+
285
+
286
+ class Rwkv6FeedForward(nn.Module):
287
+ def __init__(self, config, layer_id=0):
288
+ super().__init__()
289
+ self.config = config
290
+ self.layer_id = layer_id
291
+ hidden_size = config.hidden_size
292
+ # https://github.com/BlinkDL/RWKV-LM/blob/3db37a72356b736966ddd377268f02b80963af3f/RWKV-v4neo/train.py#L168
293
+ intermediate_size = (
294
+ config.intermediate_size
295
+ if config.intermediate_size is not None
296
+ else int((config.hidden_size * 3.5) // 32 * 32)
297
+ )
298
+
299
+ self.time_shift = nn.ZeroPad2d((0, 0, 1, -1))
300
+ self.time_maa_k = nn.Parameter(torch.empty(1, 1, hidden_size))
301
+ self.time_maa_r = nn.Parameter(torch.empty(1, 1, hidden_size))
302
+
303
+ self.key = nn.Linear(hidden_size, intermediate_size, bias=False)
304
+ self.receptance = nn.Linear(hidden_size, hidden_size, bias=False)
305
+ self.value = nn.Linear(intermediate_size, hidden_size, bias=False)
306
+
307
+ def forward(self, hidden, state=None):
308
+ if hidden.size(1) == 1 and state is not None:
309
+ shifted = state[2][:, :, self.layer_id]
310
+ else:
311
+ shifted = self.time_shift(hidden)
312
+ if state is not None:
313
+ shifted[:, 0] = state[2][:, :, self.layer_id]
314
+ if len(shifted.size()) == 2:
315
+ shifted = shifted.unsqueeze(1)
316
+
317
+ delta_hidden_to_shifted = shifted - hidden
318
+ key = hidden + delta_hidden_to_shifted * self.time_maa_k
319
+ receptance = hidden + delta_hidden_to_shifted * self.time_maa_r
320
+
321
+ key = torch.square(torch.relu(self.key(key)))
322
+ value = self.value(key)
323
+ receptance = torch.sigmoid(self.receptance(receptance))
324
+
325
+ if state is not None:
326
+ state[2][:, :, self.layer_id] = hidden[:, -1]
327
+
328
+ return receptance * value, state
329
+
330
+
331
+ class Rwkv6Block(nn.Module):
332
+ def __init__(self, config, layer_id):
333
+ super().__init__()
334
+ self.config = config
335
+ self.layer_id = layer_id
336
+
337
+ if layer_id == 0:
338
+ self.pre_ln = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_epsilon)
339
+
340
+ self.ln1 = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_epsilon)
341
+ self.ln2 = nn.LayerNorm(config.hidden_size, eps=config.layer_norm_epsilon)
342
+
343
+ self.attention = Rwkv6SelfAttention(config, layer_id)
344
+ self.feed_forward = Rwkv6FeedForward(config, layer_id)
345
+
346
+ def forward(self, hidden, state=None, use_cache=False, output_attentions=False, seq_mode=True):
347
+ if self.layer_id == 0:
348
+ hidden = self.pre_ln(hidden)
349
+ attention, state = self.attention(self.ln1(hidden), state=state, use_cache=use_cache, seq_mode=seq_mode)
350
+ hidden = hidden + attention
351
+
352
+ feed_forward, state = self.feed_forward(self.ln2(hidden), state=state)
353
+ hidden = hidden + feed_forward
354
+
355
+ outputs = (hidden, state)
356
+ if output_attentions:
357
+ outputs += (attention,)
358
+ else:
359
+ outputs += (None,)
360
+
361
+ return outputs
362
+
363
+
364
+ class Rwkv6PreTrainedModel(PreTrainedModel):
365
+ """
366
+ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained
367
+ models.
368
+ """
369
+
370
+ config_class = Rwkv6Config
371
+ base_model_prefix = "rwkv6"
372
+ _no_split_modules = ["Rwkv6Block"]
373
+ _keep_in_fp32_modules = ["time_decay", "time_first"]
374
+ supports_gradient_checkpointing = True
375
+
376
+ def _init_weights(self, module):
377
+ """Initialize the weights."""
378
+ if isinstance(module, Rwkv6SelfAttention):
379
+ layer_id = module.layer_id
380
+ num_hidden_layers = module.config.num_hidden_layers
381
+ hidden_size = module.config.hidden_size
382
+ attention_hidden_size = module.attention_hidden_size
383
+ head_size = module.config.head_size
384
+ num_heads = attention_hidden_size // head_size
385
+
386
+ ratio_0_to_1 = layer_id / (num_hidden_layers - 1) # 0 to 1
387
+ ratio_1_to_almost0 = 1.0 - (layer_id / num_hidden_layers) # 1 to ~0
388
+
389
+ time_weight = torch.tensor(
390
+ [i / hidden_size for i in range(hidden_size)],
391
+ dtype=module.time_maa_k.dtype,
392
+ device=module.time_maa_k.device,
393
+ )
394
+ time_weight = time_weight[None, None, :]
395
+
396
+ decay_speed = [
397
+ -6.0 + 5.0 * (h / (attention_hidden_size - 1)) ** (0.7 + 1.3 * ratio_0_to_1)
398
+ for h in range(attention_hidden_size)
399
+ ]
400
+ decay_speed = torch.tensor(decay_speed, dtype=module.time_decay.dtype, device=module.time_decay.device)
401
+ tmp = torch.tensor(
402
+ [
403
+ (1.0 - (i / (attention_hidden_size - 1.0))) * ratio_0_to_1 + 0.1 * ((i + 1) % 3 - 1)
404
+ for i in range(attention_hidden_size)
405
+ ],
406
+ dtype=module.time_faaaa.dtype,
407
+ device=module.time_faaaa.device,
408
+ )
409
+
410
+ with torch.no_grad():
411
+ module.time_maa_x.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
412
+ module.time_maa_w.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
413
+ module.time_maa_k.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
414
+ module.time_maa_v.data = 1.0 - (torch.pow(time_weight, ratio_1_to_almost0) + 0.3 * ratio_0_to_1)
415
+ module.time_maa_r.data = 1.0 - torch.pow(time_weight, 0.5 * ratio_1_to_almost0)
416
+ module.time_maa_g.data = 1.0 - torch.pow(time_weight, 0.5 * ratio_1_to_almost0)
417
+
418
+ TIME_MIX_EXTRA_DIM = 32 # generate TIME_MIX for w,k,v,r,g
419
+ module.time_maa_w1.data = torch.zeros(hidden_size, TIME_MIX_EXTRA_DIM*5, dtype=module.time_maa_w1.dtype, device=module.time_maa_w1.device).uniform_(-1e-4, 1e-4)
420
+ module.time_maa_w2.data = torch.zeros(5, TIME_MIX_EXTRA_DIM, hidden_size, dtype=module.time_maa_w2.dtype, device=module.time_maa_w2.device).uniform_(-1e-4, 1e-4)
421
+
422
+ TIME_DECAY_EXTRA_DIM = 64
423
+ module.time_decay_w1.data = torch.zeros(hidden_size, TIME_DECAY_EXTRA_DIM, dtype=module.time_decay_w1.dtype, device=module.time_decay_w1.device).uniform_(-1e-4, 1e-4)
424
+ module.time_decay_w2.data = torch.zeros(TIME_DECAY_EXTRA_DIM, attention_hidden_size, dtype=module.time_decay_w2.dtype, device=module.time_decay_w2.device).uniform_(-1e-4, 1e-4)
425
+
426
+ module.time_decay.data = decay_speed.reshape(num_heads, head_size)
427
+ module.time_faaaa.data = tmp.reshape(num_heads, head_size)
428
+
429
+ elif isinstance(module, Rwkv6FeedForward):
430
+ layer_id = module.layer_id
431
+ num_hidden_layers = module.config.num_hidden_layers
432
+ hidden_size = module.config.hidden_size
433
+
434
+ ratio_1_to_almost0 = 1.0 - (layer_id / num_hidden_layers) # 1 to ~0
435
+
436
+ time_weight = torch.tensor(
437
+ [i / hidden_size for i in range(hidden_size)],
438
+ dtype=module.time_maa_k.dtype,
439
+ device=module.time_maa_k.device,
440
+ )
441
+ time_weight = time_weight[None, None, :]
442
+
443
+ with torch.no_grad():
444
+ module.time_maa_k.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
445
+ module.time_maa_r.data = 1.0 - torch.pow(time_weight, ratio_1_to_almost0)
446
+
447
+
448
+ @dataclass
449
+ class Rwkv6Output(ModelOutput):
450
+ """
451
+ Class for the RWKV model outputs.
452
+
453
+ Args:
454
+ last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
455
+ Sequence of hidden-states at the output of the last layer of the model.
456
+ state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`):
457
+ The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to
458
+ avoid providing the old `input_ids`.
459
+ hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
460
+ Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
461
+ one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of
462
+ the model at the output of each layer plus the optional initial embedding outputs.
463
+ attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):
464
+ Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
465
+ sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in
466
+ the self-attention heads.
467
+ """
468
+
469
+ last_hidden_state: torch.FloatTensor = None
470
+ state: Optional[List[torch.FloatTensor]] = None
471
+ hidden_states: Optional[Tuple[torch.FloatTensor]] = None
472
+ attentions: Optional[Tuple[torch.FloatTensor]] = None
473
+
474
+
475
+ @dataclass
476
+ class Rwkv6CausalLMOutput(ModelOutput):
477
+ """
478
+ Base class for causal language model (or autoregressive) outputs.
479
+
480
+ Args:
481
+ loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided):
482
+ Language modeling loss (for next-token prediction).
483
+ logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`):
484
+ Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
485
+ state (list of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`):
486
+ The state of the model at the last time step. Can be used in a forward method with the next `input_ids` to
487
+ avoid providing the old `input_ids`.
488
+ hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`):
489
+ Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, +
490
+ one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of
491
+ the model at the output of each layer plus the optional initial embedding outputs.
492
+ attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`):
493
+ Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length,
494
+ sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in
495
+ the self-attention heads.
496
+ """
497
+
498
+ loss: Optional[torch.FloatTensor] = None
499
+ logits: torch.FloatTensor = None
500
+ state: Optional[List[torch.FloatTensor]] = None
501
+ hidden_states: Optional[Tuple[torch.FloatTensor]] = None
502
+ attentions: Optional[Tuple[torch.FloatTensor]] = None
503
+
504
+
505
+ RWKV6_START_DOCSTRING = r"""
506
+ This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the
507
+ library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads
508
+ etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module)
509
+ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to
510
+ general usage and behavior.
511
+
512
+ Parameters:
513
+ config ([`Rwkv6Config`]): Model configuration class with all the parameters of the model.
514
+ Initializing with a config file does not load the weights associated with the model, only the
515
+ configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights.
516
+ """
517
+
518
+ RWKV6_INPUTS_DOCSTRING = r"""
519
+ Args:
520
+ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`):
521
+ `input_ids_length` = `sequence_length` if `past_key_values` is `None` else
522
+ `past_key_values[0][0].shape[-2]` (`sequence_length` of input past key value states). Indices of input
523
+ sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their
524
+ past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See
525
+ [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input
526
+ IDs?](../glossary#input-ids)
527
+ inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):
528
+ Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
529
+ is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
530
+ model's internal embedding lookup matrix.
531
+ state (tuple of five `torch.FloatTensor` of shape `(batch_size, hidden_size, num_hidden_layers)`, *optional*):
532
+ If passed along, the model uses the previous state in all the blocks (which will give the output for the
533
+ `input_ids` provided as if the model add `state_input_ids + input_ids` as context).
534
+ use_cache (`bool`, *optional*):
535
+ If set to `True`, the last state is returned and can be used to quickly generate the next logits.
536
+ output_attentions (`bool`, *optional*):
537
+ Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned
538
+ tensors for more detail.
539
+ output_hidden_states (`bool`, *optional*):
540
+ Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for
541
+ more detail.
542
+ return_dict (`bool`, *optional*):
543
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
544
+ """
545
+
546
+
547
+ @add_start_docstrings(
548
+ "The bare RWKV6 Model transformer outputting raw hidden-states without any specific head on top.",
549
+ RWKV6_START_DOCSTRING,
550
+ )
551
+ class Rwkv6Model(Rwkv6PreTrainedModel):
552
+ def __init__(self, config):
553
+ super().__init__(config)
554
+
555
+ self.embeddings = nn.Embedding(config.vocab_size, config.hidden_size)
556
+ self.blocks = nn.ModuleList([Rwkv6Block(config, layer_id=idx) for idx in range(config.num_hidden_layers)])
557
+ self.ln_out = nn.LayerNorm(config.hidden_size)
558
+
559
+ self.layers_are_rescaled = False
560
+ self.gradient_checkpointing = False
561
+
562
+ # Initialize weights and apply final processing
563
+ self.post_init()
564
+
565
+ def get_input_embeddings(self):
566
+ return self.embeddings
567
+
568
+ def set_input_embeddings(self, new_embeddings):
569
+ self.embeddings = new_embeddings
570
+
571
+ @add_start_docstrings_to_model_forward(RWKV6_INPUTS_DOCSTRING)
572
+ @add_code_sample_docstrings(
573
+ checkpoint=_CHECKPOINT_FOR_DOC,
574
+ output_type=Rwkv6Output,
575
+ config_class=_CONFIG_FOR_DOC,
576
+ )
577
+ def forward(
578
+ self,
579
+ input_ids: Optional[torch.LongTensor] = None,
580
+ attention_mask: Optional[torch.LongTensor] = None, # noqa
581
+ inputs_embeds: Optional[torch.FloatTensor] = None,
582
+ state: Optional[List[torch.FloatTensor]] = None,
583
+ use_cache: Optional[bool] = None,
584
+ output_attentions: Optional[bool] = None,
585
+ output_hidden_states: Optional[bool] = None,
586
+ return_dict: Optional[bool] = None,
587
+ ) -> Union[Tuple, Rwkv6Output]:
588
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
589
+ output_hidden_states = (
590
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
591
+ )
592
+ # FIXME - training is supportable with the CUDA code
593
+ # rwkv6 only support inference in huggingface.
594
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
595
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
596
+
597
+ if self.training == self.layers_are_rescaled and (
598
+ self.embeddings.weight.dtype == torch.float16 or self.embeddings.weight.dtype == torch.bfloat16
599
+ ):
600
+ self._rescale_layers()
601
+
602
+ if input_ids is not None and inputs_embeds is not None:
603
+ raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
604
+ elif input_ids is None and inputs_embeds is None:
605
+ raise ValueError("You have to specify either input_ids or inputs_embeds")
606
+
607
+ if inputs_embeds is None:
608
+ inputs_embeds = self.embeddings(input_ids)
609
+
610
+ if state is None:
611
+ state = []
612
+ head_size = self.config.head_size
613
+ num_heads = self.config.attention_hidden_size // head_size
614
+ state_attn_x = torch.zeros(
615
+ (inputs_embeds.size(0), self.config.hidden_size, self.config.num_hidden_layers),
616
+ dtype=inputs_embeds.dtype,
617
+ requires_grad=False,
618
+ device=inputs_embeds.device,
619
+ ).contiguous()
620
+ state_attn_kv = torch.zeros(
621
+ (
622
+ inputs_embeds.size(0),
623
+ num_heads,
624
+ head_size,
625
+ head_size,
626
+ self.config.num_hidden_layers,
627
+ ),
628
+ dtype=torch.float32,
629
+ requires_grad=False,
630
+ device=inputs_embeds.device,
631
+ ).contiguous()
632
+ state_ffn_x = torch.zeros(
633
+ (inputs_embeds.size(0), self.config.hidden_size, self.config.num_hidden_layers),
634
+ dtype=inputs_embeds.dtype,
635
+ requires_grad=False,
636
+ device=inputs_embeds.device,
637
+ ).contiguous()
638
+ state.append(state_attn_x)
639
+ state.append(state_attn_kv)
640
+ state.append(state_ffn_x)
641
+
642
+ seq_mode = inputs_embeds.shape[1] > 1
643
+ hidden_states = inputs_embeds
644
+
645
+ all_self_attentions = () if output_attentions else None
646
+ all_hidden_states = () if output_hidden_states else None
647
+ for idx, block in enumerate(self.blocks):
648
+ hidden_states, state, attentions = block(
649
+ hidden_states, state=state, use_cache=use_cache, output_attentions=output_attentions, seq_mode=seq_mode
650
+ )
651
+ if (
652
+ self.layers_are_rescaled
653
+ and self.config.rescale_every > 0
654
+ and (idx + 1) % self.config.rescale_every == 0
655
+ ):
656
+ hidden_states = hidden_states / 2
657
+
658
+ if output_hidden_states:
659
+ all_hidden_states = all_hidden_states + (hidden_states,)
660
+
661
+ if output_attentions:
662
+ all_self_attentions = all_self_attentions + (attentions,)
663
+
664
+ hidden_states = self.ln_out(hidden_states)
665
+
666
+ if output_hidden_states:
667
+ all_hidden_states = all_hidden_states + (hidden_states,)
668
+
669
+ if not return_dict:
670
+ return (hidden_states, state, all_hidden_states, all_self_attentions)
671
+
672
+ return Rwkv6Output(
673
+ last_hidden_state=hidden_states,
674
+ state=state,
675
+ hidden_states=all_hidden_states, # None
676
+ attentions=all_self_attentions, # None
677
+ )
678
+
679
+ def _rescale_layers(self):
680
+ # Layers should be rescaled for inference only.
681
+ if self.layers_are_rescaled == (not self.training):
682
+ return
683
+ if self.config.rescale_every > 0:
684
+ with torch.no_grad():
685
+ for block_id, block in enumerate(self.blocks):
686
+ if self.training:
687
+ block.attention.output.weight.mul_(2 ** int(block_id // self.config.rescale_every))
688
+ block.feed_forward.value.weight.mul_(2 ** int(block_id // self.config.rescale_every))
689
+ else:
690
+ # Deal with quantization statistics
691
+ if hasattr(block.attention.output.weight, "SCB"):
692
+ block.attention.output.weight.SCB.div_(2 ** int(block_id // self.config.rescale_every))
693
+ block.feed_forward.value.weight.SCB.div_(2 ** int(block_id // self.config.rescale_every))
694
+ elif hasattr(block.attention.output.weight, "quant_state"):
695
+ self._bnb_4bit_dequantize_and_rescale(block.attention.output, block_id)
696
+ self._bnb_4bit_dequantize_and_rescale(block.feed_forward.value, block_id)
697
+ else:
698
+ block.attention.output.weight.div_(2 ** int(block_id // self.config.rescale_every))
699
+ block.feed_forward.value.weight.div_(2 ** int(block_id // self.config.rescale_every))
700
+
701
+ self.layers_are_rescaled = not self.training
702
+
703
+ def _bnb_4bit_dequantize_and_rescale(self, target_layer, block_id):
704
+ r"""
705
+ Perform the dequantization and rescaling of the weights of a given layer. After that operation the layer will
706
+ be quantized again.
707
+ """
708
+ if not is_bitsandbytes_available():
709
+ raise ImportError("Please install bitsandbytes to use this method.")
710
+ import bitsandbytes as bnb
711
+
712
+ dequant_weights = bnb.functional.dequantize_4bit(target_layer.weight.data, target_layer.weight.quant_state)
713
+
714
+ dequant_weights.div_(2 ** int(block_id // self.config.rescale_every))
715
+
716
+ # re-quantize the model:
717
+ # we need to put it first on CPU then back to the device
718
+ # this will create an overhead :/
719
+ # We set requires_grad=False as we cannot compute gradients on top of 4bit parameters anyway and to avoid
720
+ # bugs with bnb
721
+ quant_weight = bnb.nn.Params4bit(dequant_weights.to("cpu"), requires_grad=False).to(dequant_weights.device)
722
+ setattr(target_layer, "weight", quant_weight)
723
+
724
+
725
+ # copied from HuggingFace https://github.com/huggingface/transformers/blob/main/src/transformers/models/rwkv/modeling_rwkv.py
726
+ @add_start_docstrings(
727
+ """
728
+ The RWKV6 Model transformer with a language modeling head on top (linear layer with weights tied to the input
729
+ embeddings).
730
+ """,
731
+ RWKV6_START_DOCSTRING,
732
+ )
733
+ class Rwkv6ForCausalLM(Rwkv6PreTrainedModel):
734
+ _tied_weights_keys = ["head.weight"]
735
+
736
+ def __init__(self, config):
737
+ super().__init__(config)
738
+ self.rwkv = Rwkv6Model(config)
739
+ self.head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
740
+
741
+ # Initialize weights and apply final processing
742
+ self.post_init()
743
+
744
+ def get_output_embeddings(self):
745
+ return self.head
746
+
747
+ def set_output_embeddings(self, new_embeddings):
748
+ self.head = new_embeddings
749
+
750
+ def prepare_inputs_for_generation(self, input_ids, state=None, inputs_embeds=None, **kwargs):
751
+ # only last token for inputs_ids if the state is passed along.
752
+ if state is not None:
753
+ input_ids = input_ids[:, -1].unsqueeze(-1)
754
+
755
+ # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
756
+ if inputs_embeds is not None and state is None:
757
+ model_inputs = {"inputs_embeds": inputs_embeds}
758
+ else:
759
+ model_inputs = {"input_ids": input_ids}
760
+
761
+ model_inputs["state"] = state
762
+ return model_inputs
763
+
764
+ @add_start_docstrings_to_model_forward(RWKV6_INPUTS_DOCSTRING)
765
+ @add_code_sample_docstrings(
766
+ checkpoint=_CHECKPOINT_FOR_DOC,
767
+ output_type=Rwkv6CausalLMOutput,
768
+ config_class=_CONFIG_FOR_DOC,
769
+ )
770
+ def forward(
771
+ self,
772
+ input_ids: Optional[torch.LongTensor] = None,
773
+ attention_mask: Optional[torch.LongTensor] = None,
774
+ inputs_embeds: Optional[torch.FloatTensor] = None,
775
+ state: Optional[List[torch.FloatTensor]] = None,
776
+ labels: Optional[torch.LongTensor] = None,
777
+ use_cache: Optional[bool] = None,
778
+ output_attentions: Optional[bool] = None,
779
+ output_hidden_states: Optional[bool] = None,
780
+ return_dict: Optional[bool] = None,
781
+ ) -> Union[Tuple, Rwkv6CausalLMOutput]:
782
+ r"""
783
+ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
784
+ Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set
785
+ `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100`
786
+ are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]`
787
+ """
788
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
789
+
790
+ outputs = self.rwkv(
791
+ input_ids,
792
+ inputs_embeds=inputs_embeds,
793
+ state=state,
794
+ use_cache=use_cache,
795
+ output_attentions=output_attentions,
796
+ output_hidden_states=output_hidden_states,
797
+ return_dict=return_dict,
798
+ )
799
+ hidden_states = outputs[0]
800
+
801
+ logits = self.head(hidden_states)
802
+
803
+ loss = None
804
+ if labels is not None:
805
+ # move labels to correct device to enable model parallelism
806
+ labels = labels.to(logits.device)
807
+ # Shift so that tokens < n predict n
808
+ shift_logits = logits[..., :-1, :].contiguous()
809
+ shift_labels = labels[..., 1:].contiguous()
810
+ # Flatten the tokens
811
+ loss_fct = CrossEntropyLoss()
812
+ loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
813
+
814
+ if not return_dict:
815
+ output = (logits,) + outputs[1:]
816
+ return ((loss,) + output) if loss is not None else output
817
+
818
+ return Rwkv6CausalLMOutput(
819
+ loss=loss,
820
+ logits=logits,
821
+ state=outputs.state,
822
+ hidden_states=outputs.hidden_states,
823
+ attentions=outputs.attentions,
824
+ )
special_tokens_map.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "<s>",
4
+ "unk_token": "<s>"
5
+ }
tokenization_rwkv5.py ADDED
@@ -0,0 +1,229 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2024 The HuggingFace Inc. team.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """Tokenization classes for RWKV5."""
16
+
17
+ import os
18
+ import re
19
+ from typing import TYPE_CHECKING, List, Optional, Tuple
20
+
21
+ from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer
22
+ from transformers.utils import logging
23
+
24
+
25
+ if TYPE_CHECKING:
26
+ pass
27
+
28
+ logger = logging.get_logger(__name__)
29
+
30
+ VOCAB_FILES_NAMES = {
31
+ "vocab_file": "vocab.txt",
32
+ }
33
+ PRETRAINED_VOCAB_FILES_MAP = {
34
+ "vocab_file": {
35
+ "ArthurZ/rwkv-5-utf": "https://huggingface.co/ArthurZ/rwkv-5-utf/blob/main/vocab.txt",
36
+ },
37
+ }
38
+
39
+
40
+ def whitespace_tokenize(text):
41
+ """Runs basic whitespace cleaning and splitting on a piece of text.
42
+ The separators are kept
43
+ """
44
+ text = text.strip()
45
+ if not text:
46
+ return []
47
+ tokens = re.split(b"(?= )", text)
48
+ return tokens
49
+
50
+
51
+ class WordpieceTokenizer(object):
52
+ """Runs WordPiece tokenization."""
53
+
54
+ def __init__(self, vocab, unk_token):
55
+ self.vocab = vocab
56
+ self.unk_token = unk_token
57
+
58
+ def tokenize(self, text):
59
+ """
60
+ Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform
61
+ tokenization using the given vocabulary.
62
+
63
+ For example, `input = "unaffable"` wil return as output `["un", "##aff", "##able"]`.
64
+
65
+ Args:
66
+ text: A single token or whitespace separated tokens. This should have
67
+ already been passed through *BasicTokenizer*.
68
+
69
+ Returns:
70
+ A list of wordpiece tokens.
71
+ """
72
+
73
+ output_tokens = []
74
+ for token in whitespace_tokenize(text):
75
+ chars = list(token)
76
+ is_bad = False
77
+ start = 0
78
+ sub_tokens = []
79
+ while start < len(chars):
80
+ end = len(chars)
81
+ cur_substr = None
82
+ while start < end:
83
+ substr = bytes(chars[start:end])
84
+ if substr in self.vocab:
85
+ cur_substr = substr
86
+ break
87
+ end -= 1
88
+ if cur_substr is None:
89
+ is_bad = True
90
+ break
91
+ try:
92
+ cur_substr = cur_substr.decode()
93
+ except UnicodeDecodeError:
94
+ cur_substr = str(cur_substr)
95
+ sub_tokens.append(cur_substr)
96
+ start = end
97
+ if is_bad:
98
+ output_tokens.append(self.unk_token)
99
+ else:
100
+ output_tokens.extend(sub_tokens)
101
+ return output_tokens
102
+
103
+
104
+ class Rwkv5Tokenizer(PreTrainedTokenizer):
105
+ vocab_files_names = VOCAB_FILES_NAMES
106
+ pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
107
+ max_model_input_sizes = {"ArthurZ/rwkv-5-utf": 2048}
108
+
109
+ model_input_names = ["input_ids", "attention_mask"]
110
+
111
+ def __init__(self, vocab_file, bos_token="<s>", eos_token="<s>", unk_token="<s>", **kwargs):
112
+ if not os.path.isfile(vocab_file):
113
+ raise ValueError(
114
+ f"Can't find a vocabulary file at path '{vocab_file}'. To load the vocabulary from a Google pretrained"
115
+ " model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`"
116
+ )
117
+
118
+ with open(vocab_file, "r") as reader:
119
+ tokens = reader.readlines()
120
+ vocab = {}
121
+ for index, token in enumerate(tokens):
122
+ token = eval(token.rstrip("\n"))
123
+ vocab[token] = index
124
+
125
+ self.add_bos_token = True
126
+ self.encoder = vocab
127
+ self.decoder = {v: k for k, v in vocab.items()}
128
+ self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.encoder, unk_token=str(unk_token))
129
+ self._added_tokens_decoder = {0: AddedToken(str(bos_token))}
130
+ super().__init__(bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, **kwargs)
131
+
132
+ @property
133
+ def vocab_size(self):
134
+ return len(self.encoder)
135
+
136
+ def get_vocab(self):
137
+ vocab = {str(self.convert_ids_to_tokens(i)): i for i in range(self.vocab_size)}
138
+ vocab.update(self.added_tokens_encoder)
139
+ return vocab
140
+
141
+ def _tokenize(self, text, split_special_tokens=False):
142
+ return self.wordpiece_tokenizer.tokenize(text.encode("utf-8"))
143
+
144
+ def _convert_token_to_id(self, token):
145
+ """Converts a token (byte) to an id using the vocab."""
146
+ if token.startswith("b'\\"):
147
+ token = eval(token)
148
+ elif not isinstance(token, bytes):
149
+ token = token.encode("utf-8", errors="replace")
150
+ return self.encoder.get(token, self.unk_token_id)
151
+
152
+ def _convert_id_to_token(self, index):
153
+ """Converts an index (integer) in a token (byte) using the vocab."""
154
+ token = self.decoder.get(index, self.unk_token)
155
+ if isinstance(token, (bytes)):
156
+ token = token.decode("utf-8", errors="replace")
157
+ return token
158
+
159
+ def convert_tokens_to_string(self, tokens):
160
+ """Converts a sequence of tokens (bytes) in a single string. Additional tokens are encoded to bytes"""
161
+ out_string = b"".join([k.encode(errors="replace") if isinstance(k, str) else k for k in tokens]).decode(
162
+ "utf-8"
163
+ )
164
+ return out_string
165
+
166
+ def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]:
167
+ index = 0
168
+ if os.path.isdir(save_directory):
169
+ vocab_file = os.path.join(
170
+ save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]
171
+ )
172
+ else:
173
+ vocab_file = (filename_prefix + "-" if filename_prefix else "") + save_directory
174
+ with open(vocab_file, "w") as writer:
175
+ for token, token_index in sorted(self.encoder.items(), key=lambda kv: kv[1]):
176
+ if index != token_index:
177
+ logger.warning(
178
+ f"Saving vocabulary to {vocab_file}: vocabulary indices are not consecutive."
179
+ " Please check that the vocabulary is not corrupted!"
180
+ )
181
+ index = token_index
182
+ writer.write(str(token) + "\n")
183
+ index += 1
184
+ return (vocab_file,)
185
+
186
+ def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
187
+ if self.add_bos_token:
188
+ bos_token_ids = [self.bos_token_id]
189
+ else:
190
+ bos_token_ids = []
191
+
192
+ output = bos_token_ids + token_ids_0
193
+
194
+ if token_ids_1 is None:
195
+ return output
196
+
197
+ return output + bos_token_ids + token_ids_1
198
+
199
+ def get_special_tokens_mask(
200
+ self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
201
+ ) -> List[int]:
202
+ """
203
+ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
204
+ special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.
205
+
206
+ Args:
207
+ token_ids_0 (`List[int]`):
208
+ List of IDs.
209
+ token_ids_1 (`List[int]`, *optional*):
210
+ Optional second list of IDs for sequence pairs.
211
+ already_has_special_tokens (`bool`, *optional*, defaults to `False`):
212
+ Whether or not the token list is already formatted with special tokens for the model.
213
+
214
+ Returns:
215
+ `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
216
+ """
217
+ if already_has_special_tokens:
218
+ return super().get_special_tokens_mask(
219
+ token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True
220
+ )
221
+
222
+ if not self.add_bos_token:
223
+ return super().get_special_tokens_mask(
224
+ token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=False
225
+ )
226
+
227
+ if token_ids_1 is None:
228
+ return [1] + ([0] * len(token_ids_0))
229
+ return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1))
tokenizer_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name_or_path": "rwkv-5-tokenizer",
3
+ "add_prefix_space": false,
4
+ "tokenizer_class": "Rwkv5Tokenizer",
5
+ "use_fast": false,
6
+ "auto_map": {
7
+ "AutoTokenizer": [
8
+ "tokenization_rwkv5.Rwkv5Tokenizer",
9
+ null
10
+ ]
11
+ }
12
+ }
vocab.txt ADDED
The diff for this file is too large to render. See raw diff