vantaa32
/

llama-2-7b-fourierft-alpaca

Safetensors

peftf

Model card Files Files and versions Community

vantaa32 commited on 17 days ago

Commit

79da18c

verified ·

1 Parent(s): 66d192a

Update README.md

Browse files

Files changed (1) hide show

README.md +103 -23

README.md CHANGED Viewed

@@ -3,29 +3,10 @@ base_model: meta-llama/Llama-2-7b-hf
 library_name: peft
 ---
-# Model Card for Model ID
-This is a LLaMA-2-7B model fine-tuned using FourierFT on alpaca dataset.
-Hyperparameters are set as follows:
-```
-python fourierft-alpaca.py \
-    --warmup_ratio 0.06  \
-    --num_train_epochs 2 \
-    --seed 0  \
-    --per_device_train_batch_size 2  \
-    --gradient_accumulation_steps 32  \
-    --output_dir './results'  \
-    --eval_strategy "epoch"  \
-    --mixed_precision "bf16"  \
-    --lr_scheduler_type "linear"  \
-    --learning_rate 3e-4 \
-    --logging_steps 10  \
-    --report_to "none"  \
-    --fourier_scale 512 \
-    --fourier_n_frequency     10000
-```
 <!-- Provide a quick summary of what the model is/does. -->
@@ -100,6 +81,7 @@ Use the code below to get started with the model.
 ### Training Data
 <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 [More Information Needed]
@@ -115,8 +97,25 @@ Use the code below to get started with the model.
 #### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 #### Speeds, Sizes, Times [optional]
 <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
@@ -124,7 +123,88 @@ Use the code below to get started with the model.
 [More Information Needed]
 ## Evaluation
 <!-- This section describes the evaluation protocols and provides the results. -->
 ### Testing Data, Factors & Metrics

 library_name: peft
 ---
+# Model Card for vantaa32/llama-2-7b-fourierft-alpaca
+This is a LLaMA-2-7B model fine-tuned using FourierFT on alpaca dataset. Only K and V projections are set to be trainable.
 <!-- Provide a quick summary of what the model is/does. -->
 ### Training Data
+Alpaca
 <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 [More Information Needed]
 #### Training Hyperparameters
+- **Training regime:** bf16 mixed precision <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+```
+python fourierft-alpaca.py \
+    --warmup_ratio 0.06  \
+    --num_train_epochs 2 \
+    --seed 0  \
+    --per_device_train_batch_size 2  \
+    --gradient_accumulation_steps 32  \
+    --output_dir './results'  \
+    --eval_strategy "epoch"  \
+    --mixed_precision "bf16"  \
+    --lr_scheduler_type "linear"  \
+    --learning_rate 3e-4 \
+    --logging_steps 10  \
+    --report_to "none"  \
+    --fourier_scale 512 \
+    --fourier_n_frequency     10000
+```
 #### Speeds, Sizes, Times [optional]
 <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 [More Information Needed]
 ## Evaluation
+MMLU Benchmark: 0.455
+```
+Average accuracy 0.280 - abstract_algebra
+Average accuracy 0.474 - anatomy
+Average accuracy 0.434 - astronomy
+Average accuracy 0.490 - business_ethics
+Average accuracy 0.491 - clinical_knowledge
+Average accuracy 0.438 - college_biology
+Average accuracy 0.330 - college_chemistry
+Average accuracy 0.400 - college_computer_science
+Average accuracy 0.350 - college_mathematics
+Average accuracy 0.445 - college_medicine
+Average accuracy 0.157 - college_physics
+Average accuracy 0.550 - computer_security
+Average accuracy 0.426 - conceptual_physics
+Average accuracy 0.254 - econometrics
+Average accuracy 0.503 - electrical_engineering
+Average accuracy 0.312 - elementary_mathematics
+Average accuracy 0.262 - formal_logic
+Average accuracy 0.320 - global_facts
+Average accuracy 0.500 - high_school_biology
+Average accuracy 0.330 - high_school_chemistry
+Average accuracy 0.420 - high_school_computer_science
+Average accuracy 0.588 - high_school_european_history
+Average accuracy 0.540 - high_school_geography
+Average accuracy 0.663 - high_school_government_and_politics
+Average accuracy 0.441 - high_school_macroeconomics
+Average accuracy 0.326 - high_school_mathematics
+Average accuracy 0.429 - high_school_microeconomics
+Average accuracy 0.258 - high_school_physics
+Average accuracy 0.622 - high_school_psychology
+Average accuracy 0.306 - high_school_statistics
+Average accuracy 0.588 - high_school_us_history
+Average accuracy 0.624 - high_school_world_history
+Average accuracy 0.570 - human_aging
+Average accuracy 0.481 - human_sexuality
+Average accuracy 0.628 - international_law
+Average accuracy 0.528 - jurisprudence
+Average accuracy 0.479 - logical_fallacies
+Average accuracy 0.402 - machine_learning
+Average accuracy 0.592 - management
+Average accuracy 0.641 - marketing
+Average accuracy 0.520 - medical_genetics
+Average accuracy 0.621 - miscellaneous
+Average accuracy 0.474 - moral_disputes
+Average accuracy 0.241 - moral_scenarios
+Average accuracy 0.484 - nutrition
+Average accuracy 0.579 - philosophy
+Average accuracy 0.485 - prehistory
+Average accuracy 0.372 - professional_accounting
+Average accuracy 0.345 - professional_law
+Average accuracy 0.537 - professional_medicine
+Average accuracy 0.428 - professional_psychology
+Average accuracy 0.545 - public_relations
+Average accuracy 0.514 - security_studies
+Average accuracy 0.632 - sociology
+Average accuracy 0.710 - us_foreign_policy
+Average accuracy 0.470 - virology
+Average accuracy 0.673 - world_religions
+Average accuracy 0.315 - math
+Average accuracy 0.501 - health
+Average accuracy 0.345 - physics
+Average accuracy 0.595 - business
+Average accuracy 0.480 - biology
+Average accuracy 0.330 - chemistry
+Average accuracy 0.442 - computer science
+Average accuracy 0.408 - economics
+Average accuracy 0.503 - engineering
+Average accuracy 0.391 - philosophy
+Average accuracy 0.535 - other
+Average accuracy 0.561 - history
+Average accuracy 0.540 - geography
+Average accuracy 0.594 - politics
+Average accuracy 0.519 - psychology
+Average accuracy 0.572 - culture
+Average accuracy 0.375 - law
+Average accuracy 0.374 - STEM
+Average accuracy 0.419 - humanities
+Average accuracy 0.515 - social sciences
+Average accuracy 0.526 - other (business, health, misc.)
+Average accuracy: 0.455
+```
 <!-- This section describes the evaluation protocols and provides the results. -->
 ### Testing Data, Factors & Metrics