|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- jtatman/python-code-dataset-500k |
|
- jtatman/python-github-code-instruct-filtered-5k |
|
- jtatman/pile_python_instruct_format |
|
library_name: transformers |
|
tags: |
|
- code |
|
--- |
|
# Model Card for tinymistral-v2-pycoder-instruct-248m |
|
|
|
This modelcard is for tinymistral-v2-pycoder-instruct, a python-specific code generation model on top of [Locutusque/TinyMistral-248M-v2-Instruct](https://huggingface.co/Locutusque/TinyMistral-248M-v2-Instruct). |
|
|
|
## Model Details |
|
|
|
This instruct model follows the original in using ChatML format. |
|
|
|
An empty prompt will return various information from the base model, but using the instruct format will deliver python code of varying quality. |
|
|
|
### Model Description |
|
|
|
Model is in active development, base model is in active development, and all should be treated with caution. |
|
|
|
- **Developed by:** [Locutusque and M4ai] |
|
- **Funded by:** [Lint from a corner pocket] |
|
- **Shared by:** [jtatman](https://huggingface.co/jtatman) |
|
- **Model type:** [MistralForCausalLM](Locutusque/TinyMistral-248M-v2) |
|
- **License:** [MIT] |
|
- **Finetuned from model [Locutusque/TinyMistral-248M-v2](https://huggingface.co/Locutusque/TinyMistral-248M-v2-Instruct) |
|
|
|
## Uses |
|
|
|
Generate python code. |
|
|
|
### Direct Use |
|
|
|
Probably could be fine tuned with a more comprehensive dataset. Experiments are in progress. |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the prompt format below to get started with the model. |
|
|
|
<|im_start|>user |
|
Write a function for multiplying two numbers, from variables 'a' and 'b'.<|im_end|> |
|
<|im_start|>assistant |
|
|
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
Custom formatted existing python data from: |
|
- [jtatman/python-code-dataset-500k](https://huggingface.co/datasets/jtatman/python-code-dataset-500k) |
|
- [jtatman/python-github-code-instruct-filtered-5k](https://huggingface.co/datasets/jtatman/python-github-code-instruct-filtered-5k) |
|
- [jtatman/pile_python_instruct_format](https://huggingface.co/datasets/jtatman/pile_python_instruct_format) |
|
|
|
### Training Procedure |
|
|
|
Repeat training depending on compute budget. |
|
|
|
#### Preprocessing |
|
|
|
Conversion to alpaca/instruct format. |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** fp16, merge of parameter fine-tune adapters when necessary and helpful. |
|
|
|
## Evaluation |
|
|
|
#### Metrics |
|
|
|
Latest metrics: |
|
|
|
- epoch: 4.87 |
|
- global_step: 220 |
|
- learning_rate: 0.00006713780918727916 |
|
- loss: 2.3736 |
