---
license: apache-2.0
language:
- fr
library_name: transformers
inference: false
---
# CamemBERT-L4

This model is a pruned version of the pre-trained [CamemBERT](https://huggingface.co/camembert-base) checkpoint, obtained by [dropping the top-layers](https://arxiv.org/abs/2004.03844) from the original model.

![](illustration.jpeg)

## Usage

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('antoinelouis/camembert-L4')
model = AutoModel.from_pretrained('antoinelouis/camembert-L4')
```

## Comparison

| Model                                                              | #Params |  Size | Pruning |
|--------------------------------------------------------------------|:-------:|:-----:|:-------:|
| [CamemBERT](https://huggingface.co/camembert-base)                 |  110.6M | 445MB |    -    |
|                                                                    |         |       |         |
| [CamemBERT-L10](https://huggingface.co/antoinelouis/camembert-L10) |  96.4M  | 368MB |   -13%  |
| [CamemBERT-L8](https://huggingface.co/antoinelouis/camembert-L8)   |  82.3M  | 314MB |   -26%  |
| [CamemBERT-L6](https://huggingface.co/antoinelouis/camembert-L6)   |  68.1M  | 260MB |   -38%  |
| **CamemBERT-L4**                                                   |  53.9M  | 206MB |   -51%  |
| [CamemBERT-L4](https://huggingface.co/antoinelouis/camembert-L4)   |  39.7M  | 159MB |   -64%  |

## Citation

```bibtex
@online{louis2023,
   author    = 'Antoine Louis',
   title     = 'CamemBERT-L4: A Pruned Version of CamemBERT',
   publisher = 'Hugging Face',
   month     = 'october',
   year      = '2023',
   url       = 'https://huggingface.co/antoinelouis/camembert-L4',
}
```