Hugging Face's logo

Na0s
/
Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT

Text Generation
Transformers
Safetensors
llama
text-generation-inference
Inference Endpoints
Model card Files Community
Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT / README.md
Na0s's picture
Na0s
Update README.md
346fd7d verified
preview code
|
raw
history blame
1.67 kB
metadata
library_name: transformers
license: apache-2.0
datasets:
  - Open-Orca/SlimOrca
pipeline_tag: text-generation
base_model: Na0s/Llama-3.1-8b-Pruned-4-Layers

Model-card-peft-lora

Model Card for Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT

Model Details

Model Description

  • Finetuned from model:[Na0s/Llama-3.1-8b-Pruned-4-Layers]

Training Details 

    LoRA BF16, 
    batch_size=2, 
    steps=10000, gradient_accumulation_steps = 4,
    warmup_steps = 5,
    max_steps = 10000
    learning_rate = 2e-4,
    fp16 = not is_bfloat16_supported(),
    bf16 = is_bfloat16_supported(),
    logging_steps = 1,
    optim = "adamw_8bit",
    weight_decay = 0.01,
    lr_scheduler_type = "linear",
    seed = 3407

Training Data

[Open-Orca/SlimOrca]

Evaluation

MMLU Pro 0-shot: 0.2937

Evaluation Data

[TIGER-AI-Lab/MMLU-Pro]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).