See axolotl config
axolotl version: 0.4.1
base_model: mistralai/Mistral-7B-v0.1
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
is_mistral_derived_model: true
load_in_8bit: false
load_in_4bit: true
strict: false
lora_fan_in_fan_out: false
data_seed: 49
seed: 49
datasets:
- path: NimaZahedinameghi/source_injury
type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./qlora-alpaca-out
hub_model_id: NimaZahedinameghi/source_of_injury
adapter: qlora
lora_model_dir:
sequence_len: 896
sample_packing: false
pad_to_sequence_len: true
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
wandb_project: source_of_injury
wandb_entity: uqam
gradient_accumulation_steps: 4
micro_batch_size: 16
eval_batch_size: 16
num_epochs: 3
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002
max_grad_norm: 1.0
adam_beta2: 0.95
adam_epsilon: 0.00001
save_total_limit: 12
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
warmup_steps: 20
evals_per_epoch: 4
eval_table_size:
eval_table_max_new_tokens: 128
saves_per_epoch: 6
debug:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
bos_token: "<s>"
eos_token: "</s>"
unk_token: "<unk>"
save_safetensors: true
source_of_injury
This model is built by Nima Zahedinameghi. It's a fine-tuned version of mistralai/Mistral-7B-v0.1 on the source_injury dataset. It achieves the following results on the evaluation set:
- Loss: 0.5867, after 3 epochs.
Important: The model is still underdeveloped. It has been trained on a tiny synthetic data with limited medical labels. It's capabilities are limited. I will release the new version of the model soon.
How to use the model
- Pip install the required dependencies
transformers==4.36.2
datasets==2.15.0
peft==0.6.0
accelerate==0.24.1
bitsandbytes==0.41.3.post2
safetensors==0.4.1
scipy==1.11.4
sentencepiece==0.1.99
protobuf==4.23.4 --upgrade
Load the Model and Tokenizer:
from peft import AutoPeftModelForCausalLM from transformers import AutoTokenizer model_id = 'NimaZahedinameghi/source_of_injury' model = AutoPeftModelForCausalLM.from_pretrained(model_id).cuda() tokenizer = AutoTokenizer.from_pretrained(model_id) tokenizer.pad_token = tokenizer.eos_token
Define the Prompt Function: Create a function to structure your prompt correctly:
def prompt(incident_description): return f"""[INST] <<SYS>> The claims are managed by reviewing incident descriptions submitted by workers. Claim coders review the incident description and populate a database with reasoning towards determining the source of injury (InjurySource). <</SYS>> IncidentDescription: {incident_description} [/INST] """ def prompt_tok(incident_description): _p = prompt(incident_description) input_ids = tokenizer(_p, return_tensors="pt", truncation=True).input_ids.cuda() out_ids = model.generate(input_ids=input_ids, max_new_tokens=500, do_sample=False) return tokenizer.batch_decode(out_ids.detach().cpu().numpy(), skip_special_tokens=True)[0][len(_p):]
Make Predictions: Use the function to get predictions from your model:
incident_description = "While working on a vehicle repair, I had to contort my body to access hard-to-reach areas. This position caused severe discomfort and pain in my neck and shoulders." output = prompt_tok(incident_description) print(output)
This function will take an incident description and return the reasoning and injury source as determined by your fine-tuned model. Ensure you follow the specific prompt format that matches your training setup.
Model description
the model is fine tuned on a small synthetic dataset with 4bit precision. The model is still underdeveloped and covers very limited number of medical labels.
Intended uses & limitations
Further testing is required to evaluate the model performance on custome evaluations
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 16
- eval_batch_size: 16
- seed: 49
- gradient_accumulation_steps: 4
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 20
- num_epochs: 3
Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
2.0189 | 0.1481 | 1 | 2.0511 |
2.0285 | 0.2963 | 2 | 2.0442 |
1.9559 | 0.5926 | 4 | 1.9259 |
1.802 | 0.8889 | 6 | 1.6212 |
1.4115 | 1.1852 | 8 | 1.2261 |
1.1171 | 1.4815 | 10 | 1.0004 |
0.9691 | 1.7778 | 12 | 0.8657 |
0.747 | 2.0741 | 14 | 0.7082 |
0.6407 | 2.3704 | 16 | 0.6205 |
0.6101 | 2.6667 | 18 | 0.5867 |
Framework versions
- PEFT 0.11.1
- Transformers 4.42.3
- Pytorch 2.1.2+cu118
- Datasets 2.19.1
- Tokenizers 0.19.1
- Downloads last month
- 0
Model tree for NimaZahedinameghi/source_of_injury
Base model
mistralai/Mistral-7B-v0.1