Upload folder using huggingface_hub

Readme.md

@@ -0,0 +1,60 @@
+---
+license: mit
+language: fr
+datasets:
+  - uonlp/CulturaX
+  - oscar
+  - almanach/HALvest
+  - wikimedia/wikipedia
+tags:
+  - roberta
+  - camembert
+---
+# CamemBERTv2: A Smarter French Language Model Aged to Perfection
+
+[CamemBERTv2](PAPER_LINK_SOON) is a French language model pretrained on a large corpus of 275B tokens of French text. It is the second version of the CamemBERT model, which is based on the RoBERTa architecture. CamemBERTv2 is trained using the Masked Language Modeling (MLM) objective with 40% mask rate for 3 epochs on 32 H100 GPUs. The dataset used for training is a combination of French [OSCAR](https://oscar-project.org/) dumps from the [CulturaX Project](https://huggingface.co/datasets/uonlp/CulturaX), French scientific documents from [HALvest](https://huggingface.co/datasets/almanach/HALvest), and the French Wikipedia.
+
+The model is a drop-in replacement for the original CamemBERT model. Note that the new tokenizer is different from the original CamemBERT tokenizer, so you will need to use Fast Tokenizers to use the model. It will work with `CamemBERTTokenizerFast` from `transformers` library even if the original `CamemBERTTokenizer` was sentencepiece-based.
+
+**Check the CamemBERTav2 model, a much stronger French language model, based on DeBERTaV3, [here](https://huggingface.co/almanach/camembertav2-base).**
+
+## Model update details
+The new update includes:
+- Much larger pretraining dataset: 275B unique tokens (previously ~32B)
+- A newly built tokenizer based on WordPiece with 32,768 tokens, addition of the newline and tab characters, support emojis, and better handling of numbers (numbers are split into two digits tokens)
+- Extended context window of 1024 tokens
+
+More details are available in the [CamemBERTv2 paper](PAPER_LINK_SOON).
+
+## How to use
+```python
+from transformers import AutoTokenizer, AutoModel, AutoModelForMaskedLM
+
+CamemBERTa = AutoModel.from_pretrained("almanach/camembertv2-base")
+tokenizer = AutoTokenizer.from_pretrained("almanach/camembertv2-base")
+```
+
+## Fine-tuning Results:
+
+Datasets: POS tagging and Dependency Parsing (GSD, Rhapsodie, Sequoia, FSMB), NER (FTB), the FLUE benchmark (XNLI, CLS, PAWS-X), the French Question Answering Dataset (FQuAD), Social Media NER (Counter-NER), and Medical NER (CAS1, CAS2, E3C, EMEA, MEDLINE).
+
+| Model             | UPOS      | LAS       | FTB-NER   | CLS       | PAWS-X    | XNLI      | F1 (FQuAD) | EM (FQuAD) | Counter-NER | Medical-NER |
+|-------------------|-----------|-----------|-----------|-----------|-----------|-----------|------------|------------|-------------|-------------|
+| CamemBERT         | 97.59     | 88.69     | 89.97     | 94.62     | 91.36     | 81.95     | 80.98      | 62.51      | 84.18       | 70.96       |
+| CamemBERTa        | 97.57     | 88.55     | 90.33     | 94.92     | 91.67     | 82.00     | 81.15      | 62.01      | 87.37       | 71.86       |
+| CamemBERT-bio     | -         | -         | -         | -         | -         | -         | -          | -          | -           | 73.96       |
+| **CamemBERTv2**   | 97.66     | 88.64     | 81.99     | 95.07     | 92.00     | 81.75     | 80.98      | 61.35      | 87.46       | 72.77       |
+| CamemBERTav2      | 97.71     | 88.65     | 93.40     | 95.63     | 93.06     | 84.82     | 83.04      | 64.29      | 89.53       | 73.98       |
+
+Finetuned models are available in the following collection: [CamemBERTv2 Finetuned Models](FINE_TUNE_COLLECTION_SOON)
+
+## Pretraining Codebase
+
+We use the pretraining codebase from the [CamemBERTa repository](https://github.com/WissamAntoun/camemberta) for all v2 models.
+
+
+## Citation
+
+```bibtex
+CITATION_SOON
+```

config.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "/scratch/camembertv2/runs/models/camembertv2-base-bf16/post/ckpt-p2-17000/tf/",
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 1,
+  "classifier_dropout": null,
+  "embedding_size": 768,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-07,
+  "max_position_embeddings": 1025,
+  "model_name": "camembertv2-base-bf16",
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_biased_input": true,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.18.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 32768
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3801269fc733109182cf30dc7930be290b654d22a0a3017937175d4223fef8c
+size 446629831

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "[CLS]",
+    "single_word": false,
+    "lstrip": false,
+    "rstrip": false,
+    "normalized": true
+  },
+  "eos_token": {
+    "content": "[SEP]",
+    "single_word": false,
+    "lstrip": false,
+    "rstrip": false,
+    "normalized": true
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "single_word": false,
+    "lstrip": false,
+    "rstrip": false,
+    "normalized": true
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "single_word": false,
+    "lstrip": false,
+    "rstrip": false,
+    "normalized": true
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "single_word": false,
+    "lstrip": false,
+    "rstrip": false,
+    "normalized": true
+  },
+  "cls_token": {
+    "content": "[CLS]",
+    "single_word": false,
+    "lstrip": false,
+    "rstrip": false,
+    "normalized": true
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "single_word": false,
+    "lstrip": true,
+    "rstrip": false,
+    "normalized": true
+  }
+}

summaries/eval/p1/20240813-214605/events.out.tfevents.1723578365.jzxh279.203800.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e787867d79e784bdce1ea8348be02896d0986e9f36f2497dd47af39aa257f13
+size 78

summaries/eval/p1/20240814-120817/events.out.tfevents.1723630097.jzxh033.1616687.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f112eb40f6046dc19a677609c57218e6849834a9c2944020d44eb680ca37674e
+size 78

summaries/eval/p1/20240814-121838/events.out.tfevents.1723630718.jzxh033.1662204.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cccc1cbc2a1d76853e85747506d9e77d9a467dd09027508f4f69cd9feb9bc953
+size 78

summaries/eval/p1/20240814-122900/events.out.tfevents.1723631340.jzxh033.1707553.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39f263eb32545c4216460174d2fd2c17d40863e28f1bca63ce8e671fbcfa6f57
+size 78

summaries/eval/p1/20240814-123924/events.out.tfevents.1723631964.jzxh033.1752949.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f354b168efd12598eb7c147a152bda6dcd3a0478cf680a8ff4512dd65bddbb7e
+size 78

summaries/eval/p1/20240814-153452/events.out.tfevents.1723642492.jzxh033.1801652.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69c1376c429d5e93dafb3f026076b1c4375cd2bf3985305077299f1992dd3f55
+size 78

summaries/eval/p1/20240814-154514/events.out.tfevents.1723643114.jzxh033.1847078.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:afe17c303e31b0449267a4af6f2b63d53b56c50f0c820469044b43518b79ca29
+size 78

summaries/eval/p1/20240814-155444/events.out.tfevents.1723643684.jzxh033.1893110.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d31d355583a2bb7543a39f4530d2bc736b2d1920bafa661f1234fccf5fd9d3d
+size 78

summaries/eval/p1/20240814-155535/events.out.tfevents.1723643735.jzxh033.1897761.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:888718eeee0553b69263a885f4ba6e83425353258c84480559e78e2801b3e5f5
+size 78

summaries/eval/p1/20240814-155636/events.out.tfevents.1723643796.jzxh033.1902441.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:027799563a0ad57cf7f63bde3dcbd766ebc2cf9f339f6fe05acf6bb3aa74160b
+size 78

summaries/eval/p1/20240814-155727/events.out.tfevents.1723643847.jzxh033.1907080.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:555b6aa356dff05daca338f683add9b83d488e6ea13600f3d1f86c4d5351ef7d
+size 78

summaries/eval/p1/20240814-160557/events.out.tfevents.1723644357.jzxh033.1911827.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:add393b0fdc44f3236e77d297bffe33c37a6714f8e792def599c01c61dafa12a
+size 78

summaries/eval/p1/20240814-182951/events.out.tfevents.1723652991.jzxh033.1959967.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a25c4b2500a312e8fa8b7fa96d376eb45f5fbc150989f7b3b172a2b919a05c84
+size 78

summaries/eval/p1/20240814-183337/events.out.tfevents.1723653217.jzxh033.1965339.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f3fedcc67cff2f2e6531254a5d6c4f95435d957871124b8a9c1c0deb0c76fbf
+size 78

summaries/eval/p1/20240814-183610/events.out.tfevents.1723653370.jzxh033.1971378.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a7fd3cd033592a9e93f2c3daf8d22307d5ff753409192806284b6ea2ff6777c9
+size 78

summaries/eval/p1/20240815-015947/events.out.tfevents.1723679987.jzxh054.1360830.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97f3df8f1b00f5e8042aa03dcaa378760db1f8d0dfcf8acc1b04a1aebf5e094e
+size 78

summaries/eval/p1/20240815-112305/events.out.tfevents.1723713785.jzxh053.1486628.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95108d9148d692f10b755bbeacde48dd0f5d9514810f8d743e6d2bbcfd632e1c
+size 78

summaries/eval/p1/20240815-112515/events.out.tfevents.1723713915.jzxh053.1491809.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a4125819c32ef305d33b364ab2938376e2f579d0f97325987e83d96b86c7ead
+size 78

summaries/eval/p1/20240815-112726/events.out.tfevents.1723714046.jzxh053.1497069.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aad5ade1d2bde2910a86cb67a8ec0f5b41439d8c2a23229ce53002b95ce5dabe
+size 78

summaries/eval/p1/20240815-112937/events.out.tfevents.1723714177.jzxh053.1502193.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8861cd047177d4e88e67c1299b126771311e92038b3c601bd9ef9437d34993a1
+size 78

summaries/eval/p1/20240815-113155/events.out.tfevents.1723714315.jzxh070.1778126.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e105aedd9f67e7a2e4d568c0be993afca5f1300252d198b20e31b23aeef5d0de
+size 78

summaries/eval/p1/20240815-115956/events.out.tfevents.1723715996.jzxh053.1507913.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59518ef40071379d09997d4e4b62f9a45504873ba046b7cce8ea62208e2af5e7
+size 78

summaries/eval/p1/20240815-121223/events.out.tfevents.1723716743.jzxh006.1191320.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8951d9f96ab566666b06b2f30fabb46c7bc403a7191d7fd0d64400b4b4fd6af7
+size 78

summaries/eval/p1/20240816-080728/events.out.tfevents.1723788448.jzxh006.1260241.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b0446cc99b23a0ccb7dc173e2cf6970ab3cddf4513f3f8cbeec21c48e82b1f0
+size 78

summaries/eval/p1/20240816-171845/events.out.tfevents.1723821525.jzxh006.1316689.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a7adbf122e1909fdb1e768c9aac44cc4ff46c9eafb08df722af7a4db157472d
+size 78

summaries/eval/p1/20240816-172056/events.out.tfevents.1723821656.jzxh006.1321853.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a72760da721c6af43a8004c2b9842ce4c10211db12b374d5aba1ad18e2550dd7
+size 78

summaries/eval/p1/20240816-172307/events.out.tfevents.1723821787.jzxh006.1326976.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4940d6330601330f2d2dd0f2fe1c3077637955df21d088b3ff9534389d57f1a0
+size 78

summaries/eval/p1/20240816-172518/events.out.tfevents.1723821918.jzxh006.1332108.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9269df6ecc7fed5099ee8014e631577b6e70af3a51f42525f7e06e2871b4285b
+size 78

summaries/eval/p1/20240816-214034/events.out.tfevents.1723837234.jzxh025.1638477.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cbf20ad3dbf3a217a1b87f106257669228d7f737901e5bb5cfe832363d47e8eb
+size 78

summaries/eval/p1/20240817-174040/events.out.tfevents.1723909240.jzxh029.1674212.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f3e34720aed81d44e54044f1b0985f858b6c1b323eb584e8e8c99ea4af3c264
+size 78

summaries/eval/p1/20240817-215427/events.out.tfevents.1723924467.jzxh005.1407255.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:339d14c971756336ffd00bf30231e06d77641384d55c42f36a5f123436cf3b7c
+size 78

summaries/eval/p1/20240818-174928/events.out.tfevents.1723996168.jzxh005.1476071.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50569ff87d88dac17a51a2de86f6e18cc0635cbf23d8de8145664e0054d1649d
+size 78

summaries/eval/p1/20240819-005008/events.out.tfevents.1724021408.jzxh005.1531858.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dae6882d23f4ab5ae07677b0e69e43522be034f79a61a3f9d32a9013650577a5
+size 78

summaries/eval/p1/20240819-005059/events.out.tfevents.1724021459.jzxh005.1536586.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d64a0801b351c2080261672fc875549386cd75c0e0f7ada281c1da0bce37bbdb
+size 78

summaries/eval/p1/20240819-005150/events.out.tfevents.1724021510.jzxh005.1541347.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be9ac15e7c5c50327908b554651a0c09767311f24b147bbc02d9bab9ba7fa666
+size 78

summaries/eval/p1/20240819-005241/events.out.tfevents.1724021561.jzxh005.1546037.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc5a44a5c8dd66436b0434b8f097fa39fc21ac79bd865542943519f4d8aac4a6
+size 78

summaries/eval/p1/20240819-011527/events.out.tfevents.1724022927.jzxh005.1551124.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef41178c2e561b442e9bba55b7b0f386ba51632452290f425496f6fb994c6625
+size 78

summaries/eval/p1/20240819-011946/events.out.tfevents.1724023186.jzxh005.1556523.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77db1680dd574960eb3a9b7287eb564ac7b455c86f36c80f4ab263f1e25c0c44
+size 78

summaries/eval/p2/20240819-014155/events.out.tfevents.1724024515.jzxh005.1602141.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cfd32f1c54fe01c1cd6921dffdda0f165c05f6efb825d70b53e5a987a140428
+size 78

summaries/eval/p2/20240819-110757/events.out.tfevents.1724058477.jzxh005.1660044.0.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b12fc8a87df3bddc32a2435ceff860cf9df2fbbd242cda6eb82ced42c3c0a62f
+size 78

summaries/train/p1/20240813-214605/events.out.tfevents.1723578365.jzxh279.203800.1.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac08e8a028c0cac6b31cd2a256d37bbc3be2fd94fed4364aa6895fc8ea34d471
+size 2057370

summaries/train/p1/20240814-120817/events.out.tfevents.1723630097.jzxh033.1616687.1.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2248538c9ebd45b7127a6f26b29a719767cfa05ff5e8fac290a04920119b302
+size 23674

summaries/train/p1/20240814-121838/events.out.tfevents.1723630718.jzxh033.1662204.1.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:624da326a0f05418e1ec56ccb9ca002d5d05471e7dd95b87d5c3c005d9e3f6c1
+size 23674

summaries/train/p1/20240814-122900/events.out.tfevents.1723631340.jzxh033.1707553.1.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d7d5e08e613dd1e8ac788bf3d9357e14cdadbff54d417e9b6ede52392692804
+size 23674

summaries/train/p1/20240814-123924/events.out.tfevents.1723631964.jzxh033.1752949.1.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e83fc4e39e4215c16305725e6236d9036d909e984a21a14b5c7bf0ed26cf9bd0
+size 23674

summaries/train/p1/20240814-153452/events.out.tfevents.1723642492.jzxh033.1801652.1.v2

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2699d1955162c3b05747d1de00da2aa926595ca3f01d31ca0e9a9b26378664e6
+size 23674