|
--- |
|
base_model: inceptionai/jais-adapted-7b-chat |
|
language: |
|
- en |
|
- ar |
|
license: apache-2.0 |
|
tags: |
|
- text-generation-inference |
|
- transformers |
|
- unsloth |
|
- llama |
|
- agriculture |
|
- farming |
|
- jais |
|
- lora |
|
--- |
|
|
|
# Uploaded model |
|
|
|
- **Developed by:** Solshine (Caleb DeLeeuw) |
|
- **License:** apache-2.0 |
|
- **Finetuned from model :** inceptionai/jais-adapted-7b-chat |
|
- **Dataset:** CopyleftCultivars/Natural-Farming-Real-QandA-Conversations-Q1-2024-Update (Real world Natural Farming advise, from over 12 countries and a multitude of real-world farm operations, using semi-synthetic data curated by domain experts) |
|
|
|
V4 (best training loss curve of unsloth configs tested) of LORA adapter trained, merged into this full merged model: Solshine/jais-adapted-7b-chat-Natural-Farmer-lora-only-V4 |
|
|
|
Training Logs: |
|
==((====))== Unsloth - 2x faster free finetuning | Num GPUs = 1 |
|
\\ /| Num examples = 169 | Num Epochs = 2 |
|
O^O/ \_/ \ Batch size per device = 2 | Gradient Accumulation steps = 4 |
|
\ / Total batch size = 8 | Total steps = 38 |
|
"-____-" Number of trainable parameters = 39,976,960 |
|
[38/38 03:29, Epoch 1/2] |
|
Step Training Loss |
|
1 2.286800 |
|
2 2.205600 |
|
3 2.201700 |
|
4 2.158100 |
|
5 2.021100 |
|
6 1.820200 |
|
7 1.822500 |
|
8 1.565700 |
|
9 1.335700 |
|
10 1.225900 |
|
11 1.081000 |
|
12 0.947700 |
|
13 0.828600 |
|
14 0.830200 |
|
15 0.796300 |
|
16 0.781200 |
|
17 0.781600 |
|
18 0.815000 |
|
19 0.741400 |
|
20 0.847600 |
|
21 0.736600 |
|
22 0.714300 |
|
23 0.706400 |
|
24 0.752800 |
|
25 0.684600 |
|
26 0.647800 |
|
27 0.775300 |
|
28 0.613800 |
|
29 0.679500 |
|
30 0.752900 |
|
31 0.589800 |
|
32 0.729400 |
|
33 0.549500 |
|
34 0.638500 |
|
35 0.609500 |
|
36 0.632200 |
|
37 0.686400 |
|
38 0.724200 |
|
|
|
|
|
**Merged model (printing trainable parameters; pytorch) readout:** |
|
''' |
|
|
|
LlamaForCausalLM( |
|
|
|
(model): LlamaModel( |
|
|
|
(embed_tokens): Embedding(64000, 4096) |
|
|
|
(layers): ModuleList( |
|
|
|
(0-31): 32 x LlamaDecoderLayer( |
|
|
|
(self_attn): LlamaSdpaAttention( |
|
|
|
(q_proj): Linear4bit(in_features=4096, out_features=4096, bias=False) |
|
|
|
(k_proj): Linear4bit(in_features=4096, out_features=4096, bias=False) |
|
|
|
(v_proj): Linear4bit(in_features=4096, out_features=4096, bias=False) |
|
|
|
(o_proj): Linear4bit(in_features=4096, out_features=4096, bias=False) |
|
|
|
(rotary_emb): LlamaRotaryEmbedding() |
|
|
|
) |
|
|
|
(mlp): LlamaMLP( |
|
|
|
(gate_proj): Linear4bit(in_features=4096, out_features=11008, bias=False) |
|
|
|
(up_proj): Linear4bit(in_features=4096, out_features=11008, bias=False) |
|
|
|
(down_proj): Linear4bit(in_features=11008, out_features=4096, bias=False) |
|
|
|
(act_fn): SiLU() |
|
|
|
) |
|
|
|
(input_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
|
|
|
(post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05) |
|
|
|
) |
|
|
|
) |
|
|
|
(norm): LlamaRMSNorm((4096,), eps=1e-05) |
|
|
|
(rotary_emb): LlamaRotaryEmbedding() |
|
|
|
) |
|
|
|
(lm_head): Linear(in_features=4096, out_features=64000, bias=False) |
|
|
|
) |
|
|
|
''' |
|
|
|
This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library. |
|
|
|
# **JAIS Adapted 7B Chat Merged with V4 LORA adapters on Google Colab via: ** |
|
''' |
|
# Install required libraries |
|
!pip install transformers peft huggingface_hub bitsandbytes accelerate |
|
|
|
import os |
|
import torch |
|
from huggingface_hub import login |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
from peft import PeftModel |
|
from google.colab import userdata |
|
|
|
# Hugging Face credentials |
|
HF_TOKEN = userdata.get('HF_TOKEN') |
|
if HF_TOKEN is None: |
|
raise ValueError("HF_TOKEN not found in Colab secrets. Please set it up.") |
|
|
|
login(token=HF_TOKEN) |
|
|
|
# Define model and LORA adapter paths |
|
BASE_MODEL_NAME = "inceptionai/jais-adapted-7b-chat" |
|
LORA_MODEL_NAME = "Solshine/jais-adapted-7b-chat-Natural-Farmer-lora-only-V4" |
|
MERGED_MODEL_NAME = "Solshine/jais-adapted-7b-chat-Natural-Farmer-lora-merged-full" |
|
|
|
|
|
|
|
# Function to get the optimal device map |
|
def get_device_map(model_size): |
|
if torch.cuda.is_available(): |
|
gpu_memory = torch.cuda.get_device_properties(0).total_memory / 1024**3 # Convert to GB |
|
if gpu_memory > model_size: |
|
return "auto" |
|
else: |
|
return { |
|
"": torch.device("cpu"), |
|
"lm_head": torch.device("cuda:0"), |
|
"model.embed_tokens": torch.device("cuda:0"), |
|
"model.norm": torch.device("cuda:0"), |
|
"model.layers.0": torch.device("cuda:0"), |
|
} |
|
return "cpu" |
|
|
|
|
|
|
|
from transformers import BitsAndBytesConfig |
|
|
|
# Configure quantization |
|
bnb_config = BitsAndBytesConfig( |
|
load_in_4bit=True, |
|
bnb_4bit_use_double_quant=True, |
|
bnb_4bit_quant_type="nf4", |
|
bnb_4bit_compute_dtype=torch.bfloat16 |
|
) |
|
|
|
# Download and load the base model |
|
print("Downloading and loading the base model...") |
|
base_model = AutoModelForCausalLM.from_pretrained( |
|
BASE_MODEL_NAME, |
|
quantization_config=bnb_config, |
|
device_map="auto", |
|
torch_dtype=torch.float16, |
|
low_cpu_mem_usage=True, |
|
offload_folder="offload", |
|
offload_state_dict=True, |
|
) |
|
|
|
# Enable gradient checkpointing if available |
|
if hasattr(base_model, 'gradient_checkpointing_enable'): |
|
base_model.gradient_checkpointing_enable() |
|
elif hasattr(base_model, 'model') and hasattr(base_model.model, 'gradient_checkpointing_enable'): |
|
base_model.model.gradient_checkpointing_enable() |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_NAME) |
|
|
|
|
|
|
|
# Download and load the LORA adapter |
|
print("Downloading and loading the LORA adapter...") |
|
lora_model = PeftModel.from_pretrained(base_model, LORA_MODEL_NAME, device_map=device_map) |
|
|
|
# Merge the base model with the LORA adapter |
|
print("Merging the base model with the LORA adapter...") |
|
merged_model = lora_model.merge_and_unload() |
|
|
|
|
|
|
|
# Save the merged model locally |
|
print("Saving the merged model locally...") |
|
merged_model.save_pretrained("./merged_model", safe_serialization=True) |
|
tokenizer.save_pretrained("./merged_model") |
|
|
|
|
|
# Push the merged model to Hugging Face Hub |
|
print("Pushing the merged model to Hugging Face Hub...") |
|
merged_model.push_to_hub(MERGED_MODEL_NAME, use_auth_token=HF_TOKEN) |
|
tokenizer.push_to_hub(MERGED_MODEL_NAME, use_auth_token=HF_TOKEN) |
|
|
|
print("Process completed successfully!") |
|
|
|
''' |
|
|
|
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth) |
