Hugging Face's logo

almanach
/
camembertav2-base-sequoia

Token Classification
French
deberta-v2
Eval Results
Model card Files Community
camembertav2-base-sequoia
File size: 4,067 Bytes 
2cf4cca
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
 
---
language: fr
license: mit
tags:
- deberta-v2
- token-classification
base_model: almanach/camembertav2-base
datasets:
- Sequoia
metrics:
- las
- upos
model-index:
- name: almanach/camembertav2-base-sequoia
  results:
  - task:
      type: token-classification
      name: Part-of-Speech Tagging
    dataset:
      type: Sequoia
      name: Sequoia
    metrics:
    - name: upos
      type: upos
      value: 0.99423
      verified: false
  - task:
      type: token-classification
      name: Dependency Parsing
    dataset:
      type: Sequoia
      name: Sequoia
    metrics:
    - name: las
      type: las
      value: 0.94883
      verified: false
---

# Model Card for almanach/camembertav2-base-sequoia

almanach/camembertav2-base-sequoia is a deberta-v2 model for token classification. It is trained on the Sequoia dataset for the task of Part-of-Speech Tagging and Dependency Parsing.
 The model achieves an f1 score of  on the Sequoia dataset.

The model is part of the almanach/camembertav2-base family of model finetunes.

## Model Details

### Model Description

- **Developed by:** Wissam Antoun (Phd Student at Almanach, Inria-Paris)
- **Model type:** deberta-v2
- **Language(s) (NLP):** French
- **License:** MIT
- **Finetuned from model :** almanach/camembertav2-base

### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** https://github.com/WissamAntoun/camemberta
- **Paper:** https://arxiv.org/abs/2411.08868

## Uses

The model can be used for token classification tasks in French for Part-of-Speech Tagging and Dependency Parsing.

## Bias, Risks, and Limitations

The model may exhibit biases based on the training data. The model may not generalize well to other datasets or tasks. The model may also have limitations in terms of the data it was trained on.


## How to Get Started with the Model

You can use the models directly with the hopsparser library in server mode https://github.com/hopsparser/hopsparser/blob/main/docs/server.md


## Training Details

### Training Procedure

Model trained with the [hopsparser](https://github.com/hopsparser/hopsparser) library on the Sequoia dataset.


#### Training Hyperparameters

```yml
# Layer dimensions
mlp_input: 1024
mlp_tag_hidden: 16
mlp_arc_hidden: 512
mlp_lab_hidden: 128
# Lexers
lexers:
  - name: word_embeddings
    type: words
    embedding_size: 256
    word_dropout: 0.5
  - name: char_level_embeddings
    type: chars_rnn
    embedding_size: 64
    lstm_output_size: 128
  - name: fasttext
    type: fasttext
  - name: camembertav2_base_p2_17k_last_layer
    type: bert
    model: /scratch/camembertv2/runs/models/camembertav2-base-bf16/post/ckpt-p2-17000/pt/discriminator/
    layers: [11]
    subwords_reduction: "mean"
# Training hyperparameters
encoder_dropout: 0.5
mlp_dropout: 0.5
batch_size: 8
epochs: 64
lr:
  base: 0.00003
  schedule:
    shape: linear
    warmup_steps: 100

```

#### Results

**UPOS:** 0.99423
**LAS:** 0.94883

## Technical Specifications

### Model Architecture and Objective

deberta-v2 custom model for token classification.

## Citation

**BibTeX:**

```bibtex
@misc{antoun2024camembert20smarterfrench,
      title={CamemBERT 2.0: A Smarter French Language Model Aged to Perfection},
      author={Wissam Antoun and Francis Kulumba and Rian Touchent and Éric de la Clergerie and Benoît Sagot and Djamé Seddah},
      year={2024},
      eprint={2411.08868},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.08868},
}

@inproceedings{grobol:hal-03223424,
    title = {Analyse en dépendances du français avec des plongements contextualisés},
    author = {Grobol, Loïc and Crabbé, Benoît},
    url = {https://hal.archives-ouvertes.fr/hal-03223424},
    booktitle = {Actes de la 28ème Conférence sur le Traitement Automatique des Langues Naturelles},
    eventtitle = {TALN-RÉCITAL 2021},
    venue = {Lille, France},
    pdf = {https://hal.archives-ouvertes.fr/hal-03223424/file/HOPS_final.pdf},
    hal_id = {hal-03223424},
    hal_version = {v1},
}

```