Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 1024,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -0,0 +1,432 @@
+---
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- generated_from_trainer
+- dataset_size:25743
+- loss:MultipleNegativesRankingLoss
+base_model: Lajavaness/bilingual-embedding-large
+widget:
+- source_sentence: 'Greta Thunberg leads People With Money''s annual list of the "100
+    highest-paid activists" released on Monday. It''s been a rough year for the activist,
+    but at least she has her millions of dollars to ease the pain. Greta Thunberg,
+    16, has been ranked No. 1 on People With Money''s 10 highest-paid activists for
+    2019 with an estimated $46 million in combined earnings. UPDATE 09/24/2019: This
+    story appears to be false. (Read more) Greta Thunberg tops annual list of highest-paid
+    activists In 2017 it seemed that the activist''s spectacular career was coming
+    to an end. Suddenly she was back on top. People With Money reports on Monday (September
+    23) that Thunberg is the highest-paid activist in the world, with an astonishing
+    $46 million between August 2018 and August 2019, a nearly $20 million lead over
+    her closest competition. . Factors of people with money In compiling this annual
+    list, the magazine considers factors such as down payment, profit sharing, residuals,
+    endorsements and advertising work. The Swedish activist has an estimated net worth
+    of $145 million. She owes her fortune to savvy stock investments, substantial
+    property holdings, lucrative endorsement deals with CoverGirl cosmetics. He also
+    owns several restaurants (the "Fat Thunberg Burger" chain) in Stockholm, a football
+    team (the "Stockholm Angels"), has launched his own brand of vodka (Pure Wonderthunberg
+    - Sweden), and is tackling the youth market with best-selling perfume (With Love
+    from Greta) and a fashion line called "Greta Thunberg Seduction." The ranking
+    is significant for many Greta fans, who have been waiting for her triumphant return
+    to the glory days for what seems like a lifetime.'
+  sentences:
+  - Donald Trump next to a stack of declassified files
+  - Greta Thunberg leads People With Money's annual list of the "100 highest-paid
+    activists"
+  - Women should not to use shampoo during their periods “because the pores of the
+    head are open during menstruation and it can cause headache”.
+- source_sentence: Dr. Bonnie Henry, British Columbia's Provincial Health Officer
+    made a rare Saturday appearance in the media to issue a warning to people who
+    plan on using old pieces of gypsum boards to make their own glory holes, saying
+    that there is a risk of exposure to asbestos and recommends airing on the side
+    of caution and using newer sheets of drywall only. "We realize that many British
+    Columbians have been waiting for a long time for the opportunity to get intimate
+    and a few of those who will choose to use glory holes need to be aware that older
+    gypsum sheet that may be sitting in the basement or backyard will often contain
+    toxic levels of asbestos." Henry told reporters. "The safest method yet to ensure
+    zero risk of exposure to Covid 19 is getting off alone, using your own imagination,
+    toys and electronic devices." The health officer added. "I know this is a topic
+    that makes some people uncomfortable but I need to add that I've come across a
+    few hospital reports of young men getting injured getting their genitals stuck
+    in a small hole they made in a wall with a kitchen knife." Dr, Henry said with
+    a softer tone. "Just to be clear, glory holes are small openings in a wall where
+    a person will offer their intimate parts to a partner who is on the opposite side
+    who will be pleasuring the exposed parts, ideally, until full satisfaction. This
+    is all done in the spirit of doing your share to flatten the curve while having
+    a bit of fun this summer."
+  sentences:
+  - Petai can treat all types of cancer and muscle pain
+  - Terrible scene of dead is prepared to generate fear by covid-19 in the population
+  - BC Minister of Health warned against asbestos poisoning when using glory holes
+- source_sentence: THIS YOUNG CHADIAN WHO HAS A GUN POINTED ON THE CHEST BY A FRENCH
+    SOLDIER IN HIS COUNTRY BECAUSE HE DARED TO DEMONSTRATE AGAINST THE DESPOTIC REGIME
+    IN PAYMENT TO FRANCE IS THE NEW SYMBOL OF RESISTANCE TO FRENCH NEO-COLONIALISM.
+    Qatar Foundati
+  sentences:
+  - This young Chadian who has a gun pointed at his chest by a French soldier in his
+    country because he dared to demonstrate against the despotic regime in the pay
+    of France
+  - CNN uses footage of an explosion in Ukraine from 2015 to illustrate the current
+    conflict.
+  - Woman gives birth despite having no womb
+- source_sentence: The Prime Minister of Israel is infected with the Corona virus,
+    pray for him, may God have mercy on us from him
+  sentences:
+  - Photo of Netanyahu in hospital with coronavirus
+  - Documents about Covid-19 written by Dr. Thanin Kongsuk?
+  - Police warn burglars are going "door to door" with contaminated face masks in
+    a new scam
+- source_sentence: Jose Mujica Rafael Correa The United Nations appoints them like
+    the best presidents in the world AND THEY ARE LEFT
+  sentences:
+  - José Mujica and Rafael Correa were chosen by the United Nations as the best presidents
+    in the world
+  - Vaccination made the number of deaths from Covid-19 this year surpass that of
+    2020
+  - China defeated the new coronavirus
+pipeline_tag: sentence-similarity
+library_name: sentence-transformers
+---
+
+# SentenceTransformer based on Lajavaness/bilingual-embedding-large
+
+This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Lajavaness/bilingual-embedding-large](https://huggingface.co/Lajavaness/bilingual-embedding-large). It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+
+## Model Details
+
+### Model Description
+- **Model Type:** Sentence Transformer
+- **Base model:** [Lajavaness/bilingual-embedding-large](https://huggingface.co/Lajavaness/bilingual-embedding-large) <!-- at revision e83179d7a66e8aed1b3015e98bb5ae234ed89598 -->
+- **Maximum Sequence Length:** 512 tokens
+- **Output Dimensionality:** 1024 dimensions
+- **Similarity Function:** Cosine Similarity
+<!-- - **Training Dataset:** Unknown -->
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
+
+### Model Sources
+
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
+
+### Full Model Architecture
+
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BilingualModel 
+  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+  (2): Normalize()
+)
+```
+
+## Usage
+
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
+
+```bash
+pip install -U sentence-transformers
+```
+
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SentenceTransformer
+
+# Download from the 🤗 Hub
+model = SentenceTransformer("sentence_transformers_model_id")
+# Run inference
+sentences = [
+    'Jose Mujica Rafael Correa The United Nations appoints them like the best presidents in the world AND THEY ARE LEFT',
+    'José Mujica and Rafael Correa were chosen by the United Nations as the best presidents in the world',
+    'Vaccination made the number of deaths from Covid-19 this year surpass that of 2020',
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 1024]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities.shape)
+# [3, 3]
+```
+
+<!--
+### Direct Usage (Transformers)
+
+<details><summary>Click to see the direct usage in Transformers</summary>
+
+</details>
+-->
+
+<!--
+### Downstream Usage (Sentence Transformers)
+
+You can finetune this model on your own dataset.
+
+<details><summary>Click to expand</summary>
+
+</details>
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Dataset
+
+#### Unnamed Dataset
+
+* Size: 25,743 training samples
+* Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>label</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | sentence_0                                                                          | sentence_1                                                                         | label                                                         |
+  |:--------|:------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:--------------------------------------------------------------|
+  | type    | string                                                                              | string                                                                             | float                                                         |
+  | details | <ul><li>min: 2 tokens</li><li>mean: 115.14 tokens</li><li>max: 512 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 19.63 tokens</li><li>max: 137 tokens</li></ul> | <ul><li>min: 1.0</li><li>mean: 1.0</li><li>max: 1.0</li></ul> |
+* Samples:
+  | sentence_0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | sentence_1                                                                                                  | label            |
+  |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------|:-----------------|
+  | <code>Even though India is locked down today, but all the cities of China are open and from April 8, China has announced to open Wuhan as well, not a single leader in China, a single military leader, even a big man. Corona is not done Corona virus has ruined the economy all over the world, thousands of lives have been lost, lakhs have got this disease and countless people have been locked in homes, lock down has happened in many countries in which India is also one. Is The corona virus originated from the city of Wuhan in China, and now it has reached every corner of the world, but the virus did not reach the Chinese capital Beijing and the economic capital Shanghai near Wuhan. Today Paris is closed, New York is closed, Berlin is closed, Rome is closed, Delhi is closed, Mumbai is closed, Tokyo is closed, major economic and political centers of the world are closed but Beijing and Shanghai are open, there Corona has Showed no effect, only a few cases were reported, but in a way, there was no ...</code> | <code>Shanghai and Beijing, China's major cities, are untouched by COVID-19</code>                          | <code>1.0</code> |
+  | <code>This virus is kind of weak.... it hasn't killed any politicians either here or in China!</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | <code>"How strange, a virus so deadly that it didn't kill any politician... Is the "virus" corrupt?"</code> | <code>1.0</code> |
+  | <code>Korean environmental group dies 443 Korea is mixed. This is a mountain in a certain area of Korea. All the trees were cut down and the whole mountain was filled with solar panels. It covered me, but disaster will soon come and I will be heartbroken All. This disaster is no genius. Talent.</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | <code>This is a mountain in South Korea covered with solar panels</code>                                    | <code>1.0</code> |
+* Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters:
+  ```json
+  {
+      "scale": 20.0,
+      "similarity_fct": "cos_sim"
+  }
+  ```
+
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+
+- `per_device_train_batch_size`: 2
+- `per_device_eval_batch_size`: 2
+- `num_train_epochs`: 1
+- `fp16`: True
+- `multi_dataset_batch_sampler`: round_robin
+
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `eval_strategy`: no
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 2
+- `per_device_eval_batch_size`: 2
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `torch_empty_cache_steps`: None
+- `learning_rate`: 5e-05
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1
+- `num_train_epochs`: 1
+- `max_steps`: -1
+- `lr_scheduler_type`: linear
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.0
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: True
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `restore_callback_states_from_checkpoint`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: False
+- `fp16`: True
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: False
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: True
+- `label_names`: None
+- `load_best_model_at_end`: False
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: None
+- `hub_always_push`: False
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `include_for_metrics`: []
+- `eval_do_concat_batches`: True
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`: 
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `dispatch_batches`: None
+- `split_batches`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_eval_metrics`: False
+- `eval_on_start`: False
+- `use_liger_kernel`: False
+- `eval_use_gather_object`: False
+- `average_tokens_across_devices`: False
+- `prompts`: None
+- `batch_sampler`: batch_sampler
+- `multi_dataset_batch_sampler`: round_robin
+
+</details>
+
+### Training Logs
+| Epoch  | Step  | Training Loss |
+|:------:|:-----:|:-------------:|
+| 0.0388 | 500   | 0.0334        |
+| 0.0777 | 1000  | 0.0595        |
+| 0.1165 | 1500  | 0.0597        |
+| 0.1554 | 2000  | 0.046         |
+| 0.1942 | 2500  | 0.0238        |
+| 0.2331 | 3000  | 0.0667        |
+| 0.2719 | 3500  | 0.0283        |
+| 0.3108 | 4000  | 0.0429        |
+| 0.3496 | 4500  | 0.0414        |
+| 0.3884 | 5000  | 0.0295        |
+| 0.4273 | 5500  | 0.0323        |
+| 0.4661 | 6000  | 0.0288        |
+| 0.5050 | 6500  | 0.0389        |
+| 0.5438 | 7000  | 0.0399        |
+| 0.5827 | 7500  | 0.0245        |
+| 0.6215 | 8000  | 0.0334        |
+| 0.6603 | 8500  | 0.0212        |
+| 0.6992 | 9000  | 0.0207        |
+| 0.7380 | 9500  | 0.0206        |
+| 0.7769 | 10000 | 0.0163        |
+| 0.8157 | 10500 | 0.0318        |
+| 0.8546 | 11000 | 0.0256        |
+| 0.8934 | 11500 | 0.0277        |
+| 0.9323 | 12000 | 0.027         |
+| 0.9711 | 12500 | 0.0179        |
+
+
+### Framework Versions
+- Python: 3.11.11
+- Sentence Transformers: 3.4.1
+- Transformers: 4.48.3
+- PyTorch: 2.5.1+cu124
+- Accelerate: 1.3.0
+- Datasets: 3.3.1
+- Tokenizers: 0.21.0
+
+## Citation
+
+### BibTeX
+
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
+
+#### MultipleNegativesRankingLoss
+```bibtex
+@misc{henderson2017efficient,
+    title={Efficient Natural Language Response Suggestion for Smart Reply},
+    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
+    year={2017},
+    eprint={1705.00652},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

config.json

@@ -0,0 +1,37 @@
+{
+  "_name_or_path": "Lajavaness/bilingual-embedding-large",
+  "architectures": [
+    "BilingualModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "auto_map": {
+    "AutoConfig": "dangvantuan/bilingual_impl--config.BilingualConfig",
+    "AutoModel": "dangvantuan/bilingual_impl--modeling.BilingualModel",
+    "AutoModelForMaskedLM": "dangvantuan/bilingual_impl--modeling.BilingualForMaskedLM",
+    "AutoModelForMultipleChoice": "dangvantuan/bilingual_impl--modeling.BilingualForMultipleChoice",
+    "AutoModelForQuestionAnswering": "dangvantuan/bilingual_impl--modeling.BilingualForQuestionAnswering",
+    "AutoModelForSequenceClassification": "dangvantuan/bilingual_impl--modeling.BilingualForSequenceClassification",
+    "AutoModelForTokenClassification": "dangvantuan/bilingual_impl--modeling.BilingualForTokenClassification"
+  },
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "bilingual",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.3",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

config_sentence_transformers.json

@@ -0,0 +1,10 @@
+{
+  "__version__": {
+    "sentence_transformers": "3.4.1",
+    "transformers": "4.48.3",
+    "pytorch": "2.5.1+cu124"
+  },
+  "prompts": {},
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ebb5653a532fa5854e33ef29925a25e19ebefc9d13f2895fc80f9262308491cb
+size 2239607176

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:883b037111086fd4dfebbbc9b7cee11e1517b5e0c0514879478661440f137085
+size 17082987

tokenizer_config.json

@@ -0,0 +1,63 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "max_length": 512,
+  "model_max_length": 512,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "</s>",
+  "stride": 0,
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "<unk>"
+}