|
--- |
|
license: mit |
|
datasets: |
|
- wikimedia/wikipedia |
|
language: |
|
- es |
|
base_model: |
|
- openai-community/gpt2 |
|
pipeline_tag: text-generation |
|
--- |
|
|
|
|
|
|
|
# ST3: Simple Transformer 3 |
|
|
|
## Model description |
|
ST3 (Simple Transformer 3) is a lightweight transformer-based model derived from OpenAI's GPT-2 architecture. It was specifically designed to enable quick fine-tuning and experimentation, making it a great choice for researchers and developers seeking an efficient model for downstream tasks. |
|
|
|
### Key features: |
|
- **Architecture:** GPT-2-based model with 3 attention heads and 3 layers. |
|
- **Embedding size:** 288 parameters. |
|
- **Context size:** 2048 tokens, allowing for extended input/output sequences. |
|
- **Pretrained on:** Wikimedia/Wikipedia subset "20231101.es" (Spanish text corpus). |
|
- **Parameters:** 4 million FP32 parameters. |
|
- **Batch size:** 32. |
|
- **Training environment:** 1 epoch on a Kaggle P100 GPU. |
|
- **Tokenizer:** Custom WordPiece tokenizer "ST3" that generates tokens with "##" as a prefix for subword units. |
|
|
|
## Intended use |
|
ST3 is not a highly powerful or fully functional model compared to larger transformer models but can be used for: |
|
- Quick fine-tuning on small datasets. |
|
- Research purposes to test new ideas. |
|
- Educational and experimentation purposes. |
|
|
|
This model has not been fine-tuned or evaluated with performance metrics as it’s not designed for state-of-the-art tasks. |
|
|
|
### Usage |
|
To use the ST3 model, you can follow this example: |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("BueormLLC/ST3") |
|
model = AutoModelForCausalLM.from_pretrained("BueormLLC/ST3") |
|
|
|
def clean_wordpiece_tokens(text): |
|
return text.replace(" ##", "").replace("##", "") |
|
|
|
input_text = "Esto es un ejemplo" |
|
inputs = tokenizer(input_text, return_tensors="pt") |
|
|
|
outputs = model.generate(inputs.input_ids, max_length=2048, num_return_sequences=1) |
|
|
|
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
cleaned_text = clean_wordpiece_tokens(generated_text) |
|
|
|
print(cleaned_text) |
|
``` |
|
|
|
### Explanation |
|
The ST3 tokenizer uses the WordPiece algorithm, which generates tokens prefixed with "##" to indicate subword units. The provided `clean_wordpiece_tokens` function removes these prefixes, allowing for cleaner output text. |
|
|
|
## Limitations |
|
- **Performance:** ST3 lacks the power of larger models and may not perform well on complex language tasks. |
|
- **No evaluation:** The model hasn’t been benchmarked with metrics. |
|
- **Not suitable for production use** without further fine-tuning. |
|
|
|
## Training details |
|
- **Dataset:** Wikimedia/Wikipedia subset "20231101.es". |
|
- **Number of layers:** 3. |
|
- **Number of attention heads:** 3. |
|
- **Embedding size:** 288. |
|
- **Parameters:** 4 million. |
|
- **Training:** The model was trained for one epoch with a batch size of 32 on a P100 GPU provided by Kaggle. |
|
|
|
## Developer and publisher |
|
- **Developed by:** BueormAI. |
|
- **Published by:** BueormLLC. |
|
|
|
## Acknowledgments |
|
Thank you for using ST3! Your feedback and support are appreciated as we continue to develop and improve our models. |
|
|
|
If you find this model useful and would like to support further development, please consider making a donation to: |
|
|
|
- [Patreon](https://patreon.com/bueom) |
|
- [PayPal](https://paypal.me/bueorm) |
|
|
|
--- |
|
|
|
*Contributions to this project are always welcome!* |
|
|
