Edit model card

code-llama-instruct-7b-text-to-sparql-axiom

This model is a fine-tuned version of codellama/CodeLlama-7b-Instruct-hf on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.1333

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0003
  • train_batch_size: 8
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 100
  • training_steps: 800
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
1.3919 0.0710 20 1.3906
0.8712 0.1421 40 0.4591
0.2672 0.2131 60 0.2378
0.213 0.2842 80 0.2065
0.1697 0.3552 100 0.2208
0.2068 0.4263 120 0.1886
0.1808 0.4973 140 0.1843
0.2073 0.5684 160 0.1812
0.1833 0.6394 180 0.1735
0.1556 0.7105 200 0.1836
0.1813 0.7815 220 0.1688
0.166 0.8526 240 0.1642
0.1773 0.9236 260 0.1609
0.1514 0.9947 280 0.1597
0.1592 1.0657 300 0.1581
0.1632 1.1368 320 0.1552
0.1601 1.2078 340 0.1554
0.1529 1.2789 360 0.1523
0.1352 1.3499 380 0.1528
0.1601 1.4210 400 0.1496
0.1523 1.4920 420 0.1482
0.1568 1.5631 440 0.1482
0.1598 1.6341 460 0.1461
0.1432 1.7052 480 0.1471
0.158 1.7762 500 0.1430
0.1479 1.8472 520 0.1422
0.1488 1.9183 540 0.1429
0.1422 1.9893 560 0.1397
0.149 2.0604 580 0.1391
0.1352 2.1314 600 0.1381
0.1357 2.2025 620 0.1389
0.1519 2.2735 640 0.1369
0.1321 2.3446 660 0.1367
0.1381 2.4156 680 0.1361
0.1362 2.4867 700 0.1349
0.1329 2.5577 720 0.1351
0.1457 2.6288 740 0.1340
0.1267 2.6998 760 0.1336
0.1433 2.7709 780 0.1335
0.1343 2.8419 800 0.1333

Framework versions

  • Transformers 4.41.2
  • Pytorch 2.3.0+cu121
  • Datasets 2.10.1
  • Tokenizers 0.19.1
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Unable to determine this model's library. Check the docs .

Model tree for alexandrualexandru/code-llama-instruct-7b-text-to-sparql-axiom

Finetuned
(37)
this model