Upload 11 files

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -1,3 +1,170 @@
 ---
+language:
+- ru
+- en
+
+pipeline_tag: sentence-similarity
+
+tags:
+- russian
+- pretraining
+- embeddings
+- feature-extraction
+- sentence-similarity
+- sentence-transformers
+- transformers
+
+datasets:
+- IlyaGusev/gazeta
+- zloelias/lenta-ru
+- HuggingFaceFW/fineweb-2
+- HuggingFaceFW/fineweb
+
 license: mit
+base_model: sergeyzh/LaBSE-ru-turbo
+
 ---
+
+## BERTA
+
+Модель для расчетов эмбеддингов предложений на русском и английском языках получена методом дистилляции эмбеддингов [ai-forever/FRIDA](https://huggingface.co/ai-forever/FRIDA) (размер эмбеддингов - 1536, слоёв - 24) в [sergeyzh/LaBSE-ru-turbo](https://huggingface.co/sergeyzh/LaBSE-ru-turbo) (размер эмбеддингов - 768, слоёв - 12). Основной режим использования FRIDA - CLS pooling заменен на mean pooling. Каких-либо других  изменений поведения модели не производилось. Дистиляция выполнена в максимально возможном объеме - эмбеддинги русских и английских предложений, работа префиксов. 
+
+Размер контекста модели соответствует FRIDA - 512 токенов.
+
+## Префиксы
+Все префиксы унаследованы от FRIDA. 
+Оптимальный (обеспечивающий средние результаты) префикс для большинства задач - "categorize_entailment: " прописан по умолчанию в [config_sentence_transformers.json](https://huggingface.co/sergeyzh/BERTA/blob/main/config_sentence_transformers.json)
+
+Перечень используемых префиксов и их влияние на оценки модели в [encodechka](https://github.com/avidale/encodechka):
+
+| Префикс                | STS       | PI        | NLI       | SA        | TI        |
+|:-----------------------|:---------:|:---------:|:---------:|:---------:|:---------:|
+| -                      |   0,842   |   0,757   |   0,463   | **0,830** |   0,985   |
+| search_query:          |   0,853   |   0,767   |   0,479   |   0,825   |   0,987   |
+| search_document:       |   0,831   |   0,749   |   0,463   |   0,817   |   0,986   |
+| paraphrase:            |   0,847   | **0,778** |   0,446   |   0,825   |   0,986   |
+| categorize:            | **0,857** |   0,765   |   0,501   |   0,829   | **0,988** |
+| categorize_sentiment:  |   0,589   |   0,535   |   0,417   |   0,805   |   0,982   |
+| categorize_topic:      |   0,740   |   0,521   |   0,396   |   0,770   |   0,982   |
+| categorize_entailment: |   0,841   |   0,762   | **0,571** |   0,827   |   0,986   |
+
+
+**Задачи:**
+
+- Semantic text similarity (**STS**);
+- Paraphrase identification (**PI**);
+- Natural language inference (**NLI**);
+- Sentiment analysis (**SA**);
+- Toxicity identification (**TI**).
+
+
+
+# Метрики
+Оценки модели на бенчмарке [ruMTEB](https://habr.com/ru/companies/sberdevices/articles/831150/):
+
+|Model Name                      | Metric              | FRIDA     | BERTA     | [rubert-mini-frida](https://huggingface.co/sergeyzh/rubert-mini-frida)   | multilingual-e5-large-instruct | multilingual-e5-large |
+|:-------------------------------|:--------------------|----------:|----------:|--------------------:|---------------------:|----------------------:|
+|CEDRClassification              | Accuracy            | **0.646** |   0.622   |        0.552        |        0.500         |         0.448         |
+|GeoreviewClassification         | Accuracy            | **0.577** |   0.548   |        0.464        |        0.559         |         0.497         |
+|GeoreviewClusteringP2P          | V-measure           | **0.783** |   0.738   |        0.698        |        0.743         |         0.605         |
+|HeadlineClassification          | Accuracy            |   0.890   | **0.891** |        0.880        |        0.862         |         0.758         |
+|InappropriatenessClassification | Accuracy            | **0.783** |   0.748   |        0.698        |        0.655         |         0.616         |
+|KinopoiskClassification         | Accuracy            | **0.705** |   0.678   |        0.595        |        0.661         |         0.566         |
+|RiaNewsRetrieval                | NDCG@10             | **0.868** |   0.816   |        0.721        |        0.824         |         0.807         |
+|RuBQReranking                   | MAP@10              | **0.771** |   0.752   |        0.711        |        0.717         |         0.756         |
+|RuBQRetrieval                   | NDCG@10             |   0.724   |   0.710   |        0.654        |        0.692         |       **0.741**       |
+|RuReviewsClassification         | Accuracy            | **0.751** |   0.723   |        0.658        |        0.686         |         0.653         |
+|RuSTSBenchmarkSTS               | Pearson correlation |   0.814   |   0.822   |        0.803        |      **0.840**       |         0.831         |
+|RuSciBenchGRNTIClassification   | Accuracy            | **0.699** |   0.690   |        0.625        |        0.651         |         0.582         |
+|RuSciBenchGRNTIClusteringP2P    | V-measure           | **0.670** |   0.650   |        0.586        |        0.622         |         0.520         |
+|RuSciBenchOECDClassification    | Accuracy            |   0.546   | **0.555** |        0.493        |        0.502         |         0.445         |
+|RuSciBenchOECDClusteringP2P     | V-measure           | **0.566** |   0.556   |        0.507        |        0.528         |         0.450         |
+|SensitiveTopicsClassification   | Accuracy            |   0.398   | **0.399** |        0.373        |        0.323         |         0.257         |
+|TERRaClassification             | Average Precision   | **0.665** |   0.657   |        0.606        |        0.639         |         0.584         |
+								 
+|Model Name                      | Metric              | FRIDA     | BERTA     | rubert-mini-frida   | multilingual-e5-large-instruct | multilingual-e5-large |
+|:-------------------------------|:--------------------|----------:|----------:|--------------------:|----------------------:|---------------------:|
+|Classification                  | Accuracy            | **0.707** |   0.698   |        0.631        |        0.654          |        0.588         |
+|Clustering                      | V-measure           | **0.673** |   0.648   |        0.597        |        0.631          |        0.525         |
+|MultiLabelClassification        | Accuracy            | **0.522** |   0.510   |        0.463        |        0.412          |        0.353         |
+|PairClassification              | Average Precision   | **0.665** |   0.657   |        0.606        |        0.639          |        0.584         |
+|Reranking                       | MAP@10              | **0.771** |   0.752   |        0.711        |        0.717          |        0.756         |
+|Retrieval                       | NDCG@10             | **0.796** |   0.763   |        0.687        |        0.758          |        0.774         |
+|STS                             | Pearson correlation |   0.814   |   0.822   |        0.803        |      **0.840**        |        0.831         |
+|Average                         | Average             | **0.707** |   0.693   |        0.643        |        0.664          |        0.630         |
+
+
+
+## Использование модели с библиотекой `transformers`:
+
+```python
+import torch
+import torch.nn.functional as F
+from transformers import AutoTokenizer, AutoModel
+
+
+def pool(hidden_state, mask, pooling_method="mean"):
+    if pooling_method == "mean":
+        s = torch.sum(hidden_state * mask.unsqueeze(-1).float(), dim=1)
+        d = mask.sum(axis=1, keepdim=True).float()
+        return s / d
+    elif pooling_method == "cls":
+        return hidden_state[:, 0]
+
+inputs = [
+    # 
+    "paraphrase: В Ярославской области разрешили работу бань, но без посетителей",
+    "categorize_entailment: Женщину доставили в больницу, за ее жизнь сейчас борются врачи.",
+    "search_query: Сколько программистов нужно, чтобы вкрутить лампочку?",
+    # 
+    "paraphrase: Ярославским баням разрешили работать без посетителей",
+    "categorize_entailment: Женщину спасают врачи.",
+    "search_document: Чтобы вкрутить лампочку, требуется три программиста: один напишет программу извлечения лампочки, другой — вкручивания лампочки, а третий проведет тестирование."
+]
+
+tokenizer = AutoTokenizer.from_pretrained("sergeyzh/BERTA")
+model = AutoModel.from_pretrained("sergeyzh/BERTA")
+
+tokenized_inputs = tokenizer(inputs, max_length=512, padding=True, truncation=True, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model(**tokenized_inputs)
+    
+embeddings = pool(
+    outputs.last_hidden_state, 
+    tokenized_inputs["attention_mask"],
+    pooling_method="mean"
+)
+
+embeddings = F.normalize(embeddings, p=2, dim=1)
+sim_scores = embeddings[:3] @ embeddings[3:].T
+print(sim_scores.diag().tolist())
+# [0.9530372023582458, 0.866746723651886,  0.7839133143424988]
+# [0.9360030293464661, 0.8591322302818298, 0.728583037853241] - FRIDA
+```
+
+## Использование с `sentence_transformers` (sentence-transformers>=2.4.0):
+
+```python
+from sentence_transformers import SentenceTransformer
+
+# loads model with mean pooling
+model = SentenceTransformer("sergeyzh/BERTA")
+
+paraphrase = model.encode(["В Ярославской области разрешили работу бань, но без посетителей", "Ярославским баням разрешили работать без посетителей"], prompt="paraphrase: ")
+print(paraphrase[0] @ paraphrase[1].T) 
+# 0.9530372
+# 0.9360032 - FRIDA
+
+categorize_entailment = model.encode(["Женщину доставили в больницу, за ее жизнь сейчас борются врачи.", "Женщину спасают врачи."], prompt="categorize_entailment: ")
+print(categorize_entailment[0] @ categorize_entailment[1].T) 
+# 0.8667469
+# 0.8591322 - FRIDA
+
+query_embedding = model.encode("Сколько программистов нужно, чтобы вкрутить лампочку?", prompt="search_query: ")
+document_embedding = model.encode("Чтобы вкрутить лампочку, требуется три программиста: один напишет программу извлечения лампочки, другой — вкручивания лампочки, а третий проведет тестирование.", prompt="search_document: ")
+print(query_embedding @ document_embedding.T) 
+# 0.7839136
+# 0.7285831 - FRIDA
+```
+

config.json

@@ -0,0 +1,32 @@
+{
+  "_name_or_path": "sergeyzh/BERTA",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.2",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 55083
+}

config_sentence_transformers.json

@@ -0,0 +1,36 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.7.0",
+    "transformers": "4.40.1",
+    "pytorch": "2.2.1+cu118"
+  },
+  "prompts": {
+    "query": "search_query: ",
+    "passage": "search_document: ",
+    "CEDRClassification": "categorize_sentiment: ",
+    "GeoreviewClassification": "categorize_sentiment: ",
+    "GeoreviewClusteringP2P": "categorize_topic: ",
+    "HeadlineClassification": "categorize_topic: ",
+    "InappropriatenessClassification": "categorize_topic: ",
+    "KinopoiskClassification": "categorize_sentiment: ",
+    "MassiveIntentClassification": "categorize: ",
+    "MassiveScenarioClassification": "paraphrase: ",
+    "RuReviewsClassification": "categorize_sentiment: ",
+    "RUParaPhraserSTS": "paraphrase: ",
+    "RuSTSBenchmarkSTS": "categorize: ",
+    "STS22": "paraphrase: ",
+    "RuSciBenchGRNTIClassification": "categorize_topic: ",
+    "RuSciBenchGRNTIClusteringP2P": "categorize_topic: ",
+    "RuSciBenchOECDClassification": "categorize_topic: ",
+    "RuSciBenchOECDClusteringP2P": "categorize_topic: ",
+    "SensitiveTopicsClassification": "categorize_topic: ",
+    "TERRa": "categorize_entailment: ",
+    "Classification": "categorize_entailment: ",
+    "MultilabelClassification": "categorize: ",
+    "Clustering": "categorize_topic: ",
+    "PairClassification": "categorize_entailment: ",
+    "STS": "paraphrase: "
+  },
+  "default_prompt_name": "Classification",
+  "similarity_fn_name": null
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96bfcb9b24d99f7501528b8d122610ad25f0f87f212e5debb3fb5afe87aa8888
+size 513402728

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,65 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": false,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "max_length": 512,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}