File size: 5,181 Bytes
79cc64b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 |
---
base_model: ce-lery/japanese-mistral-300m-base
inference: false
model-index:
- name: checkpoints-mistral-300M-FA2
results: []
model_creator: ce-lery
model_name: japanese-mistral-300m-base
pipeline_tag: text-generation
quantized_by: afrideva
tags:
- generated_from_trainer
- gguf
- ggml
- quantized
- q2_k
- q3_k_m
- q4_k_m
- q5_k_m
- q6_k
- q8_0
---
# ce-lery/japanese-mistral-300m-base-GGUF
Quantized GGUF model files for [japanese-mistral-300m-base](https://huggingface.co/ce-lery/japanese-mistral-300m-base) from [ce-lery](https://huggingface.co/ce-lery)
| Name | Quant method | Size |
| ---- | ---- | ---- |
| [japanese-mistral-300m-base.fp16.gguf](https://huggingface.co/afrideva/japanese-mistral-300m-base-GGUF/resolve/main/japanese-mistral-300m-base.fp16.gguf) | fp16 | 712.33 MB |
| [japanese-mistral-300m-base.q2_k.gguf](https://huggingface.co/afrideva/japanese-mistral-300m-base-GGUF/resolve/main/japanese-mistral-300m-base.q2_k.gguf) | q2_k | 176.84 MB |
| [japanese-mistral-300m-base.q3_k_m.gguf](https://huggingface.co/afrideva/japanese-mistral-300m-base-GGUF/resolve/main/japanese-mistral-300m-base.q3_k_m.gguf) | q3_k_m | 195.04 MB |
| [japanese-mistral-300m-base.q4_k_m.gguf](https://huggingface.co/afrideva/japanese-mistral-300m-base-GGUF/resolve/main/japanese-mistral-300m-base.q4_k_m.gguf) | q4_k_m | 234.80 MB |
| [japanese-mistral-300m-base.q5_k_m.gguf](https://huggingface.co/afrideva/japanese-mistral-300m-base-GGUF/resolve/main/japanese-mistral-300m-base.q5_k_m.gguf) | q5_k_m | 266.47 MB |
| [japanese-mistral-300m-base.q6_k.gguf](https://huggingface.co/afrideva/japanese-mistral-300m-base-GGUF/resolve/main/japanese-mistral-300m-base.q6_k.gguf) | q6_k | 307.38 MB |
| [japanese-mistral-300m-base.q8_0.gguf](https://huggingface.co/afrideva/japanese-mistral-300m-base-GGUF/resolve/main/japanese-mistral-300m-base.q8_0.gguf) | q8_0 | 379.17 MB |
## Original Model Card:
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# japanese-mistral-300m-base
## Overview
Welcome to my model card!
This Model feature is ...
- Suppression of unknown word generation by using byte fallback in SentencePiece tokenizer and conversion to huggingface Tokenizers format
- Pretrained by wikipedia dataset and cc100 dataset
- Use of [Mistral 300M](https://huggingface.co/ce-lery/japanese-mistral-300m-base/blob/main/config.json)
Yukkuri shite ittene!
## How to use the model
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
import torch
MODEL_NAME = "ce-lery/japanese-mistral-300m-base"
torch.set_float32_matmul_precision('high')
DEVICE = "cuda"
if torch.cuda.is_available():
print("cuda")
DEVICE = "cuda"
else:
print("cpu")
DEVICE = "cpu"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME,use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
trust_remote_code=True,
).to(DEVICE)
# streamer = TextStreamer(tokenizer)
prompt = "大規模言語モデルとは、"
inputs = tokenizer(prompt, add_special_tokens=False,return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
inputs["input_ids"],
max_new_tokens=256,
do_sample=True,
early_stopping=False,
top_p=0.95,
top_k=50,
temperature=0.9,
# streamer=streamer,
no_repeat_ngram_size=2,
num_beams=3
)
print(outputs.tolist()[0])
outputs_txt = tokenizer.decode(outputs[0])
print(outputs_txt)
```
## Receipe
If you want to restruct this model, you can refer [this Github repository](https://github.com/ce-lery/japanese-mistral-300m-recipe).
I wrote the receipe for struction this model. For example,
- Preprocess with sentencepiece
- Pretraining with flash attention2 and torch.compile and DeepSpeed
- Fine-tuning with databricks-dolly-15k-ja
If you find my mistake,error,...etc, please create issue.
If you create pulreqest, I'm very happy!
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0006
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 64
- total_train_batch_size: 256
- optimizer: Adam with betas=(0.9,0.95) and epsilon=0.0001
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 1000
- num_epochs: 1
- mixed_precision_training: Native AMP
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 4.2911 | 0.12 | 5000 | 4.2914 |
| 3.9709 | 0.24 | 10000 | 3.9900 |
| 3.8229 | 0.36 | 15000 | 3.8388 |
| 3.7197 | 0.47 | 20000 | 3.7454 |
| 3.652 | 0.59 | 25000 | 3.6739 |
| 3.597 | 0.71 | 30000 | 3.6177 |
| 3.5554 | 0.83 | 35000 | 3.5770 |
| 3.536 | 0.95 | 40000 | 3.5582 |
### Framework versions
- Transformers 4.35.2
- Pytorch 2.1.1+cu121
- Datasets 2.14.5
- Tokenizers 0.14.1 |