sergeyzh commited on
Commit
914c8c8
·
verified ·
1 Parent(s): c71eb7c

Upload 11 files

Browse files
1_Pooling/config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 768,
3
+ "pooling_mode_cls_token": false,
4
+ "pooling_mode_mean_tokens": true,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false
7
+ }
README.md CHANGED
@@ -1,3 +1,170 @@
1
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: mit
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - ru
4
+ - en
5
+
6
+ pipeline_tag: sentence-similarity
7
+
8
+ tags:
9
+ - russian
10
+ - pretraining
11
+ - embeddings
12
+ - feature-extraction
13
+ - sentence-similarity
14
+ - sentence-transformers
15
+ - transformers
16
+
17
+ datasets:
18
+ - IlyaGusev/gazeta
19
+ - zloelias/lenta-ru
20
+ - HuggingFaceFW/fineweb-2
21
+ - HuggingFaceFW/fineweb
22
+
23
  license: mit
24
+ base_model: sergeyzh/LaBSE-ru-turbo
25
+
26
  ---
27
+
28
+ ## BERTA
29
+
30
+ Модель для расчетов эмбеддингов предложений на русском и английском языках получена методом дистилляции эмбеддингов [ai-forever/FRIDA](https://huggingface.co/ai-forever/FRIDA) (размер эмбеддингов - 1536, слоёв - 24) в [sergeyzh/LaBSE-ru-turbo](https://huggingface.co/sergeyzh/LaBSE-ru-turbo) (размер эмбеддингов - 768, слоёв - 12). Основной режим использования FRIDA - CLS pooling заменен на mean pooling. Каких-либо других изменений поведения модели не производилось. Дистиляция выполнена в максимально возможном объеме - эмбеддинги русских и английских предложений, работа префиксов.
31
+
32
+ Размер контекста модели соответствует FRIDA - 512 токенов.
33
+
34
+ ## Префиксы
35
+ Все префиксы унаследованы от FRIDA.
36
+ Оптимальный (обеспечивающий средние результаты) префикс для большинства задач - "categorize_entailment: " прописан по умолчанию в [config_sentence_transformers.json](https://huggingface.co/sergeyzh/BERTA/blob/main/config_sentence_transformers.json)
37
+
38
+ Перечень используемых префиксов и их влияние на оценки модели в [encodechka](https://github.com/avidale/encodechka):
39
+
40
+ | Префикс | STS | PI | NLI | SA | TI |
41
+ |:-----------------------|:---------:|:---------:|:---------:|:---------:|:---------:|
42
+ | - | 0,842 | 0,757 | 0,463 | **0,830** | 0,985 |
43
+ | search_query: | 0,853 | 0,767 | 0,479 | 0,825 | 0,987 |
44
+ | search_document: | 0,831 | 0,749 | 0,463 | 0,817 | 0,986 |
45
+ | paraphrase: | 0,847 | **0,778** | 0,446 | 0,825 | 0,986 |
46
+ | categorize: | **0,857** | 0,765 | 0,501 | 0,829 | **0,988** |
47
+ | categorize_sentiment: | 0,589 | 0,535 | 0,417 | 0,805 | 0,982 |
48
+ | categorize_topic: | 0,740 | 0,521 | 0,396 | 0,770 | 0,982 |
49
+ | categorize_entailment: | 0,841 | 0,762 | **0,571** | 0,827 | 0,986 |
50
+
51
+
52
+ **Задачи:**
53
+
54
+ - Semantic text similarity (**STS**);
55
+ - Paraphrase identification (**PI**);
56
+ - Natural language inference (**NLI**);
57
+ - Sentiment analysis (**SA**);
58
+ - Toxicity identification (**TI**).
59
+
60
+
61
+
62
+ # Метрики
63
+ Оценки модели на бенчмарке [ruMTEB](https://habr.com/ru/companies/sberdevices/articles/831150/):
64
+
65
+ |Model Name | Metric | FRIDA | BERTA | [rubert-mini-frida](https://huggingface.co/sergeyzh/rubert-mini-frida) | multilingual-e5-large-instruct | multilingual-e5-large |
66
+ |:-------------------------------|:--------------------|----------:|----------:|--------------------:|---------------------:|----------------------:|
67
+ |CEDRClassification | Accuracy | **0.646** | 0.622 | 0.552 | 0.500 | 0.448 |
68
+ |GeoreviewClassification | Accuracy | **0.577** | 0.548 | 0.464 | 0.559 | 0.497 |
69
+ |GeoreviewClusteringP2P | V-measure | **0.783** | 0.738 | 0.698 | 0.743 | 0.605 |
70
+ |HeadlineClassification | Accuracy | 0.890 | **0.891** | 0.880 | 0.862 | 0.758 |
71
+ |InappropriatenessClassification | Accuracy | **0.783** | 0.748 | 0.698 | 0.655 | 0.616 |
72
+ |KinopoiskClassification | Accuracy | **0.705** | 0.678 | 0.595 | 0.661 | 0.566 |
73
+ |RiaNewsRetrieval | NDCG@10 | **0.868** | 0.816 | 0.721 | 0.824 | 0.807 |
74
+ |RuBQReranking | MAP@10 | **0.771** | 0.752 | 0.711 | 0.717 | 0.756 |
75
+ |RuBQRetrieval | NDCG@10 | 0.724 | 0.710 | 0.654 | 0.692 | **0.741** |
76
+ |RuReviewsClassification | Accuracy | **0.751** | 0.723 | 0.658 | 0.686 | 0.653 |
77
+ |RuSTSBenchmarkSTS | Pearson correlation | 0.814 | 0.822 | 0.803 | **0.840** | 0.831 |
78
+ |RuSciBenchGRNTIClassification | Accuracy | **0.699** | 0.690 | 0.625 | 0.651 | 0.582 |
79
+ |RuSciBenchGRNTIClusteringP2P | V-measure | **0.670** | 0.650 | 0.586 | 0.622 | 0.520 |
80
+ |RuSciBenchOECDClassification | Accuracy | 0.546 | **0.555** | 0.493 | 0.502 | 0.445 |
81
+ |RuSciBenchOECDClusteringP2P | V-measure | **0.566** | 0.556 | 0.507 | 0.528 | 0.450 |
82
+ |SensitiveTopicsClassification | Accuracy | 0.398 | **0.399** | 0.373 | 0.323 | 0.257 |
83
+ |TERRaClassification | Average Precision | **0.665** | 0.657 | 0.606 | 0.639 | 0.584 |
84
+
85
+ |Model Name | Metric | FRIDA | BERTA | rubert-mini-frida | multilingual-e5-large-instruct | multilingual-e5-large |
86
+ |:-------------------------------|:--------------------|----------:|----------:|--------------------:|----------------------:|---------------------:|
87
+ |Classification | Accuracy | **0.707** | 0.698 | 0.631 | 0.654 | 0.588 |
88
+ |Clustering | V-measure | **0.673** | 0.648 | 0.597 | 0.631 | 0.525 |
89
+ |MultiLabelClassification | Accuracy | **0.522** | 0.510 | 0.463 | 0.412 | 0.353 |
90
+ |PairClassification | Average Precision | **0.665** | 0.657 | 0.606 | 0.639 | 0.584 |
91
+ |Reranking | MAP@10 | **0.771** | 0.752 | 0.711 | 0.717 | 0.756 |
92
+ |Retrieval | NDCG@10 | **0.796** | 0.763 | 0.687 | 0.758 | 0.774 |
93
+ |STS | Pearson correlation | 0.814 | 0.822 | 0.803 | **0.840** | 0.831 |
94
+ |Average | Average | **0.707** | 0.693 | 0.643 | 0.664 | 0.630 |
95
+
96
+
97
+
98
+ ## Использование модели с библиотекой `transformers`:
99
+
100
+ ```python
101
+ import torch
102
+ import torch.nn.functional as F
103
+ from transformers import AutoTokenizer, AutoModel
104
+
105
+
106
+ def pool(hidden_state, mask, pooling_method="mean"):
107
+ if pooling_method == "mean":
108
+ s = torch.sum(hidden_state * mask.unsqueeze(-1).float(), dim=1)
109
+ d = mask.sum(axis=1, keepdim=True).float()
110
+ return s / d
111
+ elif pooling_method == "cls":
112
+ return hidden_state[:, 0]
113
+
114
+ inputs = [
115
+ #
116
+ "paraphrase: В Ярославской области разрешили работу бань, но без посетителей",
117
+ "categorize_entailment: Женщину доставили в больницу, за ее жизнь сейчас борются врачи.",
118
+ "search_query: Сколько программистов нужно, чтобы вкрутить лампочку?",
119
+ #
120
+ "paraphrase: Ярославским баням разрешили работать без посетителей",
121
+ "categorize_entailment: Женщину спасают врачи.",
122
+ "search_document: Чтобы вкрутить лампочку, требуется три программиста: один напишет программу извлечения лампочки, другой — вкручивания лампочки, а третий проведет тестирование."
123
+ ]
124
+
125
+ tokenizer = AutoTokenizer.from_pretrained("sergeyzh/BERTA")
126
+ model = AutoModel.from_pretrained("sergeyzh/BERTA")
127
+
128
+ tokenized_inputs = tokenizer(inputs, max_length=512, padding=True, truncation=True, return_tensors="pt")
129
+
130
+ with torch.no_grad():
131
+ outputs = model(**tokenized_inputs)
132
+
133
+ embeddings = pool(
134
+ outputs.last_hidden_state,
135
+ tokenized_inputs["attention_mask"],
136
+ pooling_method="mean"
137
+ )
138
+
139
+ embeddings = F.normalize(embeddings, p=2, dim=1)
140
+ sim_scores = embeddings[:3] @ embeddings[3:].T
141
+ print(sim_scores.diag().tolist())
142
+ # [0.9530372023582458, 0.866746723651886, 0.7839133143424988]
143
+ # [0.9360030293464661, 0.8591322302818298, 0.728583037853241] - FRIDA
144
+ ```
145
+
146
+ ## Использование с `sentence_transformers` (sentence-transformers>=2.4.0):
147
+
148
+ ```python
149
+ from sentence_transformers import SentenceTransformer
150
+
151
+ # loads model with mean pooling
152
+ model = SentenceTransformer("sergeyzh/BERTA")
153
+
154
+ paraphrase = model.encode(["В Ярославской области разрешили работу бань, но без посетителей", "Ярославским баням разрешили работать без посетителей"], prompt="paraphrase: ")
155
+ print(paraphrase[0] @ paraphrase[1].T)
156
+ # 0.9530372
157
+ # 0.9360032 - FRIDA
158
+
159
+ categorize_entailment = model.encode(["Женщину доставили в больницу, за ее жизнь сейчас борются врачи.", "Женщину спасают врачи."], prompt="categorize_entailment: ")
160
+ print(categorize_entailment[0] @ categorize_entailment[1].T)
161
+ # 0.8667469
162
+ # 0.8591322 - FRIDA
163
+
164
+ query_embedding = model.encode("Сколько программистов нужно, чтобы вкрутить лампочку?", prompt="search_query: ")
165
+ document_embedding = model.encode("Чтобы вкрутить лампочку, требуется три программиста: один напишет программу извлечения лампочки, другой — вкручивания лампочки, а третий проведет тестирование.", prompt="search_document: ")
166
+ print(query_embedding @ document_embedding.T)
167
+ # 0.7839136
168
+ # 0.7285831 - FRIDA
169
+ ```
170
+
config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "sergeyzh/BERTA",
3
+ "architectures": [
4
+ "BertModel"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "classifier_dropout": null,
8
+ "directionality": "bidi",
9
+ "gradient_checkpointing": false,
10
+ "hidden_act": "gelu",
11
+ "hidden_dropout_prob": 0.1,
12
+ "hidden_size": 768,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 3072,
15
+ "layer_norm_eps": 1e-12,
16
+ "max_position_embeddings": 512,
17
+ "model_type": "bert",
18
+ "num_attention_heads": 12,
19
+ "num_hidden_layers": 12,
20
+ "pad_token_id": 0,
21
+ "pooler_fc_size": 768,
22
+ "pooler_num_attention_heads": 12,
23
+ "pooler_num_fc_layers": 3,
24
+ "pooler_size_per_head": 128,
25
+ "pooler_type": "first_token_transform",
26
+ "position_embedding_type": "absolute",
27
+ "torch_dtype": "float32",
28
+ "transformers_version": "4.48.2",
29
+ "type_vocab_size": 2,
30
+ "use_cache": true,
31
+ "vocab_size": 55083
32
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "sentence_transformers": "2.7.0",
4
+ "transformers": "4.40.1",
5
+ "pytorch": "2.2.1+cu118"
6
+ },
7
+ "prompts": {
8
+ "query": "search_query: ",
9
+ "passage": "search_document: ",
10
+ "CEDRClassification": "categorize_sentiment: ",
11
+ "GeoreviewClassification": "categorize_sentiment: ",
12
+ "GeoreviewClusteringP2P": "categorize_topic: ",
13
+ "HeadlineClassification": "categorize_topic: ",
14
+ "InappropriatenessClassification": "categorize_topic: ",
15
+ "KinopoiskClassification": "categorize_sentiment: ",
16
+ "MassiveIntentClassification": "categorize: ",
17
+ "MassiveScenarioClassification": "paraphrase: ",
18
+ "RuReviewsClassification": "categorize_sentiment: ",
19
+ "RUParaPhraserSTS": "paraphrase: ",
20
+ "RuSTSBenchmarkSTS": "categorize: ",
21
+ "STS22": "paraphrase: ",
22
+ "RuSciBenchGRNTIClassification": "categorize_topic: ",
23
+ "RuSciBenchGRNTIClusteringP2P": "categorize_topic: ",
24
+ "RuSciBenchOECDClassification": "categorize_topic: ",
25
+ "RuSciBenchOECDClusteringP2P": "categorize_topic: ",
26
+ "SensitiveTopicsClassification": "categorize_topic: ",
27
+ "TERRa": "categorize_entailment: ",
28
+ "Classification": "categorize_entailment: ",
29
+ "MultilabelClassification": "categorize: ",
30
+ "Clustering": "categorize_topic: ",
31
+ "PairClassification": "categorize_entailment: ",
32
+ "STS": "paraphrase: "
33
+ },
34
+ "default_prompt_name": "Classification",
35
+ "similarity_fn_name": null
36
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:96bfcb9b24d99f7501528b8d122610ad25f0f87f212e5debb3fb5afe87aa8888
3
+ size 513402728
modules.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ },
14
+ {
15
+ "idx": 2,
16
+ "name": "2",
17
+ "path": "2_Normalize",
18
+ "type": "sentence_transformers.models.Normalize"
19
+ }
20
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "max_seq_length": 512,
3
+ "do_lower_case": false
4
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": {
3
+ "content": "[CLS]",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "mask_token": {
10
+ "content": "[MASK]",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "[PAD]",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "sep_token": {
24
+ "content": "[SEP]",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "unk_token": {
31
+ "content": "[UNK]",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ }
37
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "[PAD]",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "[UNK]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "[CLS]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "[SEP]",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "4": {
36
+ "content": "[MASK]",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "clean_up_tokenization_spaces": true,
45
+ "cls_token": "[CLS]",
46
+ "do_basic_tokenize": true,
47
+ "do_lower_case": false,
48
+ "extra_special_tokens": {},
49
+ "mask_token": "[MASK]",
50
+ "max_length": 512,
51
+ "model_max_length": 512,
52
+ "never_split": null,
53
+ "pad_to_multiple_of": null,
54
+ "pad_token": "[PAD]",
55
+ "pad_token_type_id": 0,
56
+ "padding_side": "right",
57
+ "sep_token": "[SEP]",
58
+ "stride": 0,
59
+ "strip_accents": null,
60
+ "tokenize_chinese_chars": true,
61
+ "tokenizer_class": "BertTokenizer",
62
+ "truncation_side": "right",
63
+ "truncation_strategy": "longest_first",
64
+ "unk_token": "[UNK]"
65
+ }
vocab.txt ADDED
The diff for this file is too large to render. See raw diff