File size: 2,230 Bytes
574c97f 298c26a 574c97f 298c26a bff9db4 76d7a33 298c26a 574c97f a6f080d 574c97f 298c26a 574c97f 298c26a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
---
license: apache-2.0
---
# Kexer models
Kexer models is a collection of fine-tuned open-source generative text models fine-tuned on Kotlin Exercices dataset.
This is a repository for fine-tuned CodeLlama-7b model in the Hugging Face Transformers format.
# Model use
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load pre-trained model and tokenizer
model_name = 'JetBrains/CodeLlama-7B-Kexer'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to('cuda')
# Create and encode input
input_text = """\
This function takes an integer n and returns factorial of a number:
fun factorial(n: Int): Int {\
"""
input_ids = tokenizer.encode(
input_text, return_tensors='pt'
).to('cuda')
# Generate
output = model.generate(
input_ids, max_length=150, num_return_sequences=1,
no_repeat_ngram_size=2, early_stopping=True
)
# Decode output
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)
```
# Training setup
The model was trained on one A100 GPU with following hyperparameters:
| **Hyperparameter** | **Value** |
|:---------------------------:|:----------------------------------------:|
| `warmup` | 10% |
| `max_lr` | 1e-4 |
| `scheduler` | linear |
| `total_batch_size` | 256 (~130K tokens per step) |
# Fine-tuning data
For this model we used 15K exmaples of Kotlin Exercices dataset {TODO: link!}. For more information about the dataset follow th link.
# Evaluation
To evaluate we used Kotlin Humaneval (more infromation here)
Fine-tuned model:
| **Model name** | **Kotlin HumanEval Pass Rate** | **Kotlin Completion** |
|:---------------------------:|:----------------------------------------:|:----------------------------------------:|
| `base model` | 26.89 | 0.388 |
| `fine-tuned model` | 42.24 | 0.344 | |