File size: 3,039 Bytes
8505fc4 724a8f0 8505fc4 724a8f0 8505fc4 724a8f0 8505fc4 d514e21 8505fc4 d514e21 8505fc4 d514e21 8505fc4 d514e21 8505fc4 d5762b8 8505fc4 d514e21 8505fc4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
---
license: mit
language:
- fr
library_name: transformers
inference: false
pipeline_tag: feature-extraction
tags:
- feature-extraction
---
# CamemBERT-L6
This model is a pruned version of the pre-trained [CamemBERT](https://huggingface.co/camembert-base) checkpoint, obtained by [dropping the top-layers](https://doi.org/10.48550/arXiv.2004.03844) from the original model.
## Usage
You can use the raw model for masked language modeling (MLM), but it's mostly intended to be fine-tuned on a downstream task, especially one that uses the whole sentence to make decisions such as text classification, extractive question answering, or semantic search. For tasks such as text generation, you should look at autoregressive models like [BelGPT-2](https://huggingface.co/antoinelouis/belgpt2).
You can use this model directly with a pipeline for [masked language modeling](https://huggingface.co/tasks/fill-mask):
```python
from transformers import pipeline
unmasker = pipeline('fill-mask', model='antoinelouis/camembert-L6')
unmasker("Bonjour, je suis un [MASK] modèle.")
```
You can also use this model to [extract the features](https://huggingface.co/tasks/feature-extraction) of a given text:
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('antoinelouis/camembert-L6')
model = AutoModel.from_pretrained('antoinelouis/camembert-L6')
text = "Remplacez-moi par le texte de votre choix."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```
## Variations
CamemBERT has originally been released in base (110M) and large (335M) variations. The following checkpoints prune the base variation by dropping the top 2, 4, 6, 8, and 10 pretrained encoding layers, respectively.
| Model | #Params | Size | Pruning |
|--------------------------------------------------------------------|:-------:|:-----:|:-------:|
| [CamemBERT-base](https://huggingface.co/camembert-base) | 110.6M | 445MB | - |
| | | | |
| [CamemBERT-L10](https://huggingface.co/antoinelouis/camembert-L10) | 96.4M | 386MB | -13% |
| [CamemBERT-L8](https://huggingface.co/antoinelouis/camembert-L8) | 82.3M | 329MB | -26% |
| **CamemBERT-L6** | 68.1M | 272MB | -38% |
| [CamemBERT-L4](https://huggingface.co/antoinelouis/camembert-L4) | 53.9M | 216MB | -51% |
| [CamemBERT-L2](https://huggingface.co/antoinelouis/camembert-L2) | 39.7M | 159MB | -64% |
## Citation
For attribution in academic contexts, please cite this work as:
```bibtex
@online{louis2023,
author = 'Antoine Louis',
title = 'CamemBERT-L6: A Pruned Version of CamemBERT',
publisher = 'Hugging Face',
month = 'october',
year = '2023',
url = 'https://huggingface.co/antoinelouis/camembert-L6',
}
``` |