|
---
|
|
license: apache-2.0
|
|
---
|
|
|
|
# Micro Llama v0 (Development) |
|
|
|
Micro Llama v0 is a lightweight and experimental version of the LlamaForCausalLM model designed for development and testing purposes. This repository contains the necessary model configuration, tokenizer, and generation settings to run a minimal Llama architecture. |
|
|
|
## Model Overview |
|
|
|
Micro Llama v0 is based on the LlamaForCausalLM architecture. It is tailored to fit resource-constrained environments for testing the foundational components of a transformer-based language model. This version features: |
|
|
|
- **1 hidden layer** |
|
- **2048 hidden size** |
|
- **32 attention heads** |
|
- **5632 intermediate size** |
|
- **Max position embeddings** of 2048 |
|
- **Vocabulary size** of 32,000 |
|
|
|
These parameters make the model compact and suitable for development, while still maintaining key characteristics of the Llama architecture. |
|
|
|
## Files and Configuration |
|
|
|
- **`config.json`**: Contains the model architecture configuration, such as hidden size, number of attention heads, hidden layers, and activation functions. |
|
- **`generation_config.json`**: Specifies generation parameters, including max length and token behavior. |
|
- **`model.safetensors`**: Stores the model weights in a safe and efficient format. |
|
- **`special_tokens_map.json`**: Maps the special tokens used by the model, including `<s>`, `</s>`, `<unk>`, and `</s>` (for padding). |
|
- **`tokenizer.json`**: Defines the tokenizer configuration, including vocabulary size and token mapping. |
|
- **`tokenizer_config.json`**: Further configures the tokenizer, specifying token types, maximum sequence length, and other tokenizer options. |
|
|
|
## Requirements |
|
|
|
- [Transformers](https://github.com/huggingface/transformers) version 4.44.0 or above |
|
- PyTorch version compatible with the model's `float32` tensor type |
|
- `safetensors` package for loading model weights |
|
|
|
## Usage |
|
|
|
1. Clone the repository: |
|
```bash |
|
git clone https://github.com/your-repo/micro-llama.git |
|
cd micro-llama |
|
``` |
|
2. Install the required dependencies: |
|
```bash |
|
pip install transformers safetensors torch |
|
``` |
|
3. Load the model in your code: |
|
```python |
|
from transformers import LlamaForCausalLM, LlamaTokenizer |
|
|
|
tokenizer = LlamaTokenizer.from_pretrained("UnieAI-Wilson/micro-llama-0-dev") |
|
model = LlamaForCausalLM.from_pretrained("UnieAI-Wilson/micro-llama-0-dev", torch_dtype="float16") |
|
|
|
inputs = tokenizer("Your text here", return_tensors="pt") |
|
outputs = model.generate(**inputs) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
``` |
|
## License |
|
Micro Llama v0 is licensed under the Apache 2.0 License. See the LICENSE file for details. |
|
|
|
## Contribution |
|
|
|
This is an experimental and evolving project. Contributions are welcome, and feel free to submit issues or pull requests. |
|
|
|
## Disclaimer |
|
|
|
This is an early-stage development version, and the model may undergo significant changes. It is not intended for production use. |
|
|
