File size: 1,801 Bytes
7d6c601 2d9dbaa 7d6c601 2d9dbaa 457a689 2d9dbaa |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
---
license: apache-2.0
language:
- en
pipeline_tag: fill-mask
inference: false
---
# Monarch Mixer-BERT
The 341M checkpoint for M2-BERT-large from the paper [Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture](https://arxiv.org/abs/2310.12109).
Check out our [GitHub](https://github.com/HazyResearch/m2/tree/main) for instructions on how to download and fine-tune it!
## How to use
You can load this model using Hugging Face `AutoModel`:
```python
from transformers import AutoModelForMaskedLM
mlm = AutoModelForMaskedLM.from_pretrained('alycialee/m2-bert-341m', trust_remote_code=True)
```
This model uses the Hugging Face `bert-base-uncased tokenizer`:
```
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
```
You can use this model with a pipeline for masked language modeling:
```python
from transformers import AutoModelForMaskedLM, BertTokenizer, pipeline
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
mlm = AutoModelForMaskedLM.from_pretrained('alycialee/m2-bert-341m', trust_remote_code=True)
unmasker = pipeline('fill-mask', model=mlm, tokenizer=tokenizer)
unmasker('Every morning, I enjoy a cup of [MASK] to start my day.')
```
### Remote Code
This model requires `trust_remote_code=True` to be passed to the `from_pretrained` method. This is because we use custom PyTorch code (see our GitHub). You should consider passing a `revision` argument that specifies the exact git commit of the code, for example:
```python
mlm = AutoModelForMaskedLM.from_pretrained(
'alycialee/m2-bert-341m',
trust_remote_code=True,
revision='2d9dbaa',
)
```
### Configuration
Note `use_flash_mm` is false by default. Using FlashMM is currently not supported.
Using `hyena_training_additions` is turned off. |