---
language:
- ja
- en
license: other
license_link: LICENSE
---

# Sarashina2.1-1B

This repository provides large language models trained by [SB Intuitions](https://www.sbintuitions.co.jp/).

## How to use

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
 
model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2.1-1b", torch_dtype=torch.bfloat16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2.1-1b")
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
set_seed(123)
 
text = generator(
    "おはようございます、今日の天気は",
    max_length=30,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id,
    num_return_sequences=3,
)

for t in text:
    print(t)


```

## Model Description

We constructed this Sarashina2.1-1B model, which consists of 1 billion parameters, using a two-phase training process.
First, we trained the model on 10 trillion tokens, including Japanese and English data extracted from web corpora.
Then, we trained the model using 1 trillion tokens, predominantly consisting of Japanese data, to enhance its performance in Japanese.
The following tables show the model's performance on Japanese and English tasks.
We also show the performance of small models.

#### Evaluation in Japanese tasks

| Model | Avg. | AIO | abc | JEMHopQA | NIILC | JComQA | JSQuAD |
| ----- | ---- | --- | --- | --------- | ---- | ------ | ------ |
| Qwen2.5-0.5B                 | 25.40 | 0.80  | 27.38 | 28.21 | 0.79 | 45.13 | 50.07 |
| Qwen2.5-1.5B                 | 39.61 | 7.00  | 38.14 | 27.35 | 11.81 | **79.18** | 74.18 |
| llm-jp-3-1.8b                | 43.46 | 44.50 | 46.45 | 32.48 | 30.71 | 44.06 | 62.58 |
| llm-jp-3-3.7b                | 54.24 | 54.10 | 49.63 | 36.75 | **49.61** | 58.36 | 77.01 |
| Sarashina2.1-1B (this model) | **58.31** | **54.70** | **58.44** | **41.88** | 48.82 | 64.70 | **81.34** |


### Evaluation in English tasks

| Model                        | Avg.  | PIQA  | OpenBookQA | HellaSwag | Winogrande | ARC-easy | ARC-challenge |
| ---------------------------- | ----- | ----- | ---------- | --------- | ---------- | -------- | ------------- |
| Qwen2.5-0.5B                 | 50.71 | 69.59 |   35.40    |   52.17   |   56.43    | 58.42    | 32.25         |
| Qwen2.5-1.5B                 | 60.84 | 76.17 |   40.40    |  67.83    |  63.85     | 72.01    | 44.80         |
| llm-jp-3-1.8b                | 53.01 | 72.85 |    32.60   | 61.78     | 62.27      | 57.24    |  31.31        |
| llm-jp-3-3.7b                | 56.70 | 74.92 |  36.60     | 67.75     | 62.90      | 61.91    | 36.09         |
| Sarashina2.1-1B (this model) | 56.01 | 74.10 |  37.20     |  63.16    |  61.01     | 63.64    | 36.95         |



## Ethical Considerations and Limitations
Sarashina2.1 has not been tuned to follow an instruction yet.
Therefore, sarashina2.1 might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs.
Before using sarashina2.1, we would like developers to tune models based on human preferences and safety considerations.

## License

[Sarashina Model NonCommercial License Agreement](https://huggingface.co/sbintuitions/sarashina2.1-1B/blob/main/LICENSE)