|
--- |
|
license: mit |
|
--- |
|
|
|
A Llama3 based model fine-tuned for cybersecurity domain. |
|
|
|
The model was finetuned over https://huggingface.co/datasets/unibuc-cs/CyberGuardianDataset. |
|
While the Llama3 original perplexity on the dataset started ~21.9, we managed to fine-tune to ~7.8 on this dataset, still preserving the general language abilities. |
|
|
|
You can load the model and see its LoRA config as below: |
|
``` |
|
from peft import get_peft_model, AutoPeftModelForCausalLM, PeftModel, PeftConfig |
|
from transformers import AutoModel |
|
config = PeftConfig.from_pretrained("unibuc-cs/CyberGuardian") |
|
model = AutoModel.from_pretrained(config.base_model_name_or_path) |
|
``` |
|
|
|
|
|
Refer to our github page for details: https://github.com/unibuc-cs/CyberGuardian |
