|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
# Kexer models |
|
|
|
Kexer models is a collection of fine-tuned open-source generative text models fine-tuned on Kotlin Exercices dataset. |
|
This is a repository for fine-tuned CodeLlama-7b model in the Hugging Face Transformers format. |
|
|
|
# Model use |
|
|
|
``` |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
# Load pre-trained model and tokenizer |
|
model_name = 'JetBrains/CodeLlama-7B-Kexer' # Replace with the desired model name |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModelForCausalLM.from_pretrained(model_name).cuda() |
|
|
|
# Encode input text |
|
input_text = """This function takes an integer n and returns factorial of a number: |
|
fun factorial(n: Int): Int {""" |
|
input_ids = tokenizer.encode(input_text, return_tensors='pt').to('cuda') |
|
|
|
# Generate text |
|
output = model.generate(input_ids, max_length=150, num_return_sequences=1, no_repeat_ngram_size=2, early_stopping=True) |
|
|
|
# Decode and print the generated text |
|
generated_text = tokenizer.decode(output[0], skip_special_tokens=True) |
|
print(generated_text) |
|
``` |
|
|
|
# Training setup |
|
|
|
The model was trained on one A100 GPU with following hyperparameters: |
|
|
|
| **Hyperparameter** | **Value** | |
|
|:---------------------------:|:----------------------------------------:| |
|
| `warmup` | 10% | |
|
| `max_lr` | 1e-4 | |
|
| `scheduler` | linear | |
|
| `total_batch_size` | 256 (~130K tokens per step) | |
|
|
|
|
|
# Fine-tuning data |
|
|
|
For this model we used 15K exmaples of Kotlin Exercices dataset {TODO: link!}. For more information about the dataset follow th link. |
|
|
|
# Evaluation |
|
|
|
To evaluate we used Kotlin Humaneval (more infromation here) |
|
|
|
Fine-tuned model: |
|
|
|
| **Model name** | **Kotlin HumanEval Pass Rate** | **Kotlin Completion** | |
|
|:---------------------------:|:----------------------------------------:|:----------------------------------------:| |
|
| `base model` | 26.89 | 0.388 | |
|
| `fine-tuned model` | 42.24 | 0.344 | |
