|
--- |
|
library_name: transformers |
|
tags: |
|
- text-generation |
|
- conversational |
|
datasets: |
|
- TIGER-Lab/WebInstructSub |
|
language: |
|
- en |
|
base_model: |
|
- HuggingFaceTB/SmolLM-360M-Instruct |
|
--- |
|
|
|
# Model Card for TrelisSmolLM-instruct |
|
|
|
This model is a fine-tuned version of TrelisSmolLM-base, optimized for instruction following and conversational tasks using the WebInstructSub dataset. |
|
|
|
To purchase the training scripts used for this model, visit: https://trelis.com/advanced-fine-tuning-scripts/ |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
TrelisLM-80M-SFT is an 80 million parameter language model derived from SmolLM-360M through pruning and distillation, and then fine-tuned on the WebInstructSub dataset for improved instruction following capabilities. |
|
|
|
- **Developed by:** Trelis AI |
|
- **Model type:** Causal Language Model |
|
- **Language(s):** English |
|
- **License:** [More Information Needed] |
|
- **Finetuned from model:** Trelis/80M-0.0090-cosmopedia |
|
|
|
### Model Sources |
|
|
|
- **Repository:** https://huggingface.co/Trelis/80M-2percent-corpus-SFT |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
|
|
This model is designed for instruction following and conversational tasks. It can be used for: |
|
|
|
- Generating responses to user prompts or questions |
|
- Engaging in task-oriented dialogues |
|
- Assisting with general language understanding and generation tasks |
|
|
|
### Out-of-Scope Use |
|
|
|
This model should not be used for: |
|
|
|
- Production systems without thorough testing and evaluation |
|
- Tasks requiring domain-specific expertise without additional fine-tuning |
|
- Any applications where errors could lead to harmful consequences |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
The model was fine-tuned on the TIGER-Lab/WebInstructSub dataset, which consists of instruction-response pairs. The training process used: |
|
|
|
- 50,000 initial rows for the main training phase |
|
- 10,000 additional rows for an annealing phase |
|
- 10,000 randomly selected rows for evaluation |
|
|
|
### Training Procedure |
|
|
|
- **Preprocessing:** The dataset was formatted into a conversational structure with user and assistant messages. |
|
- **Training type:** Supervised Fine-Tuning (SFT) |
|
- **Training regime:** BFloat16 mixed precision |
|
|
|
#### Training Hyperparameters |
|
|
|
- Batch size: 8 |
|
- Gradient Accumulation steps: 4 |
|
- Learning rate: 1e-3 |
|
- Number of epochs: 1 |
|
- Max sequence length: 2048 |
|
- Warmup steps: 20 |
|
|
|
The training used a custom learning rate scheduler with an initial constant phase followed by cosine annealing. |
|
|
|
### Software and Hardware |
|
|
|
- **Software:** Transformers, TRL (Transformer Reinforcement Learning), Accelerate |
|
- **Hardware:** [More Information Needed] |
|
|
|
## Evaluation |
|
|
|
Evaluation was performed on a randomly selected subset of 10,000 rows from the WebInstructSub dataset. |
|
|
|
### Metrics |
|
|
|
[More Information Needed] |
|
|
|
## Limitations and Bias |
|
|
|
As this model is fine-tuned on the WebInstructSub dataset, it may inherit biases present in that dataset. Additionally, as a smaller language model, it may have limitations in handling complex or highly specialized tasks compared to larger models. |
|
|
|
### Recommendations |
|
|
|
- Thoroughly test the model's outputs before using it in any sensitive applications. |
|
- Be aware that the model's knowledge is limited to its training data and it may produce incorrect or biased information. |
|
- For critical applications, consider using this model in conjunction with other sources of information or larger, more comprehensive models. |
|
|
|
## How to Get Started with the Model |
|
|
|
You can use this model with the Transformers library: |
|
|
|
```python |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
model = AutoModelForCausalLM.from_pretrained("Trelis/80M-2percent-corpus-SFT") |
|
tokenizer = AutoTokenizer.from_pretrained("Trelis/80M-2percent-corpus-SFT") |
|
|
|
# Example usage |
|
input_text = "What is the capital of France?" |
|
input_ids = tokenizer.encode(input_text, return_tensors="pt") |
|
output = model.generate(input_ids, max_length=50) |
|
response = tokenizer.decode(output[0], skip_special_tokens=True) |
|
print(response) |
