MomoD2R/LLAMA3_Vuln_Detection_V2

Model Card for LLAMA3_Vuln_Detection

This model is designed to detect and provide information about cybersecurity vulnerabilities, specifically focusing on Common Vulnerabilities and Exposures (CVEs). It is built using the 🤗 transformers library and fine-tuned for this specific task.

Model Details

Model Description

The LLAMA3_Vuln_Detection model leverages state-of-the-art language modeling techniques to analyze and detect cybersecurity vulnerabilities. It provides detailed descriptions of CVEs and assists in identifying potential security risks in software systems.

• Developed by: [MomoD2R]
• Funded by [optional]: [DOAC]
• Shared by [optional]: [MomoD2R]
• Model type: Causal Language Model (Fine-tuned with LoRA)
• Language(s) (NLP): English - French
• License: [Apache-2.0]
• Finetuned from model [optional]: LLAMA-3-8b-bnb-4bit

Uses

Education and personal use cases, but mostly for project with UI's or Shells.

Direct Use

The model can be used directly to query information about specific CVEs, aiding cybersecurity professionals in understanding and mitigating vulnerabilities.

Downstream Use [optional]

The model can be integrated into larger cybersecurity systems for real-time vulnerability detection and monitoring.

Out-of-Scope Use

The model should not be used for making critical security decisions without human oversight. It is also not intended for detecting non-cybersecurity-related issues.

Bias, Risks, and Limitations

The model may have biases based on the data it was trained on. It might not be comprehensive in detecting all types of vulnerabilities and should be used as a supplementary tool.

Recommendations

Users should cross-verify the model's outputs with other sources and use it in conjunction with other security tools.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer
from unsloth import FastLanguageModel

# Charger le tokenizer et le modèle avec Unsloth
tokenizer = AutoTokenizer.from_pretrained("MomoD2R/LLAMA3_Vuln_Detection_V2")
model, _ = FastLanguageModel.from_pretrained("MomoD2R/LLAMA3_Vuln_Detection_V2")

# Préparer l'entrée
prompt = "Do you know what is a CVE ?"
inputs = tokenizer(prompt, return_tensors="pt").to('cuda')

# Générer le texte
outputs = model.generate(inputs.input_ids, max_length=500, num_return_sequences=1)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(generated_text)

Training Details

Training Data

The model was fine-tuned on a dataset of CVE descriptions and cybersecurity reports to enhance its understanding of vulnerabilities.

Training Procedure

Preprocessing [optional]

The data was preprocessed to include tokenization and normalization to ensure consistency in the input.

Training Hyperparameters

• Training regime: Mixed precision (fp16)
• Batch size: 16
• Learning rate: 3e-5
• Number of epochs: 3

Speeds, Sizes, Times [optional]

The training was performed over 48 hours on a Tesla T4 GPU with a final model size of approximately 7.5 GB.

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on a separate validation set of CVE descriptions.

Factors

The evaluation focused on the accuracy and relevance of the generated CVE descriptions.

Metrics

The primary metrics used for evaluation were BLEU score and human judgment on the accuracy and relevance of the generated texts.

Results

User >> Do you know what is a CVE ?

Assistant >> Do you know what is a CVE? If not, then you should definitely know what is a CVE and how it can affect your system. CVE stands for Common Vulnerabilities and Exposures. It is a dictionary of publicly known information about common vulnerabilities and exposures. It is maintained by the MITRE Corporation, a non-profit organization that provides technical services to the United States government. The MITRE Corporation is also responsible for the creation of the CVE system.
What is a CVE?
A CVE is a publicly known information about a vulnerability or exposure that can be exploited by an attacker. It is a dictionary of common vulnerabilities and exposures. It is maintained by the MITRE Corporation, a non-profit organization that provides technical services to the United States government.
The CVE system was created in 1999 by the MITRE Corporation. It is a system that provides a standard way of identifying and reporting vulnerabilities and exposures.

Summary

The model demonstrated strong performance in generating accurate and relevant descriptions of CVEs, making it a valuable tool for cybersecurity analysis.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: Tesla T4 GPU
• Hours used: 48
• Cloud Provider: Google Cloud Platform
• Compute Region: us-central1
• Carbon Emitted: Approximately 30 kg CO2

Technical Specifications

Model Architecture and Objective

The model is based on the LLAMA architecture, fine-tuned with Low-Rank Adaptation (LoRA) techniques to specialize in cybersecurity vulnerability detection.

Compute Infrastructure

Hardware

Training was conducted on a Tesla T4 GPU with 16 GB of memory.

Software

• Framework: PyTorch
• Transformers version: 4.6.1
• Accelerate version: 0.31.0
• Bitsandbytes version: 0.43.1

Glossary

• CVE: Common Vulnerabilities and Exposures
• LoRA: Low-Rank Adaptation
• BLEU: Bilingual Evaluation Understudy Score