Hugging Face's logo

ping98k
/
th-7b-20gb-base

Text Generation
Transformers
Safetensors
Thai
llama
Generated from Trainer
text-generation-inference
Inference Endpoints
Model card Files Community
th-7b-20gb-base / README.md
ping98k's picture
ping98k
Update README.md
6fb080e verified
preview code
|
raw
history blame contribute delete
3.19 kB
metadata
license: unknown
base_model: openthaigpt/openthaigpt-1.0.0-beta-7b-chat-ckpt-hf
tags:
  - generated_from_trainer
model-index:
  - name: out
    results: []
language:
  - th
pipeline_tag: text-generation
datasets:
  - allenai/MADLAD-400

Built with Axolotl

ping98k/th-7b-20gb-base

This model is a continue pre-training version of openthaigpt/openthaigpt-1.0.0-beta-7b-chat-ckpt-hf on the 20GB Thai dataset. It achieves the following results on the evaluation set:

  • Loss: 1.5721

Inference with Pipeline 

import torch
from transformers import pipeline
text_generator = pipeline("text-generation", model="ping98k/th-7b-20gb-base", torch_dtype=torch.bfloat16, device_map="auto")
print(text_generator("แบบจำลองทางวิทยาศาสตร์ (scientific modeling) คือ", max_length=50))

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.00015
  • train_batch_size: 4
  • eval_batch_size: 4
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 2
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 100
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss
4.0347 0.0 1 4.0530
2.2753 0.05 1179 2.2083
2.1613 0.1 2358 2.0422
2.0696 0.15 3537 1.9526
1.945 0.2 4716 1.8886
1.6807 0.25 5895 1.8340
1.5838 0.3 7074 1.7961
1.7497 0.35 8253 1.7548
1.535 0.4 9432 1.7237
1.9632 0.45 10611 1.6878
1.9091 0.5 11790 1.6631
1.6837 0.55 12969 1.6344
1.7054 0.6 14148 1.6131
1.463 0.65 15327 1.5980
1.5538 0.7 16506 1.5853
1.5095 0.75 17685 1.5780
1.7322 0.8 18864 1.5742
1.5645 0.85 20043 1.5727
1.72 0.9 21222 1.5722
1.5882 0.95 22401 1.5721

Framework versions

  • Transformers 4.35.1
  • Pytorch 2.0.1+cu118
  • Datasets 2.14.7
  • Tokenizers 0.14.1