|
--- |
|
library_name: peft |
|
base_model: Open-Orca/Mistral-7B-SlimOrca |
|
license: mit |
|
datasets: |
|
- noxneural/kashaloti |
|
language: |
|
- sq |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
|
|
|
|
## Model Details |
|
|
|
Version 1 of 71, qLora finetune of the Mistral-7B model using 1/71 of the GPT4 based part of the Orca Dataset, using approx ~14k records from a total of 1 million records. |
|
|
|
|
|
### Model Description |
|
|
|
- **Developed by:** Marlind Maksuti @ StochastX |
|
- **Model type:** Mistral-7B |
|
- **Language(s) (NLP):** Albanian, Shqip |
|
- **Finetuned from model:** Mistral-7B-SlimOrca |
|
|
|
### Model Sources |
|
|
|
|
|
- **Repository:** https://huggingface.co/Open-Orca/Mistral-7B-SlimOrca |
|
|
|
|
|
## Uses |
|
|
|
Text generation in Albanian. |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
Model is just version v0.1, outputs are still not optimal. |
|
|
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> |
|
|
|
|
|
|
|
## Training procedure |
|
|
|
|
|
The following `bitsandbytes` quantization config was used during training: |
|
- quant_method: QuantizationMethod.BITS_AND_BYTES |
|
- load_in_8bit: True |
|
- load_in_4bit: False |
|
- llm_int8_threshold: 6.0 |
|
- llm_int8_skip_modules: None |
|
- llm_int8_enable_fp32_cpu_offload: False |
|
- llm_int8_has_fp16_weight: False |
|
- bnb_4bit_quant_type: nf4 |
|
- bnb_4bit_use_double_quant: True |
|
- bnb_4bit_compute_dtype: float16 |
|
|
|
### Framework versions |
|
|
|
|
|
- PEFT 0.6.0.dev0 |
|
## Training procedure |
|
|
|
|
|
The following `bitsandbytes` quantization config was used during training: |
|
- quant_method: QuantizationMethod.BITS_AND_BYTES |
|
- load_in_8bit: True |
|
- load_in_4bit: False |
|
- llm_int8_threshold: 6.0 |
|
- llm_int8_skip_modules: None |
|
- llm_int8_enable_fp32_cpu_offload: False |
|
- llm_int8_has_fp16_weight: False |
|
- bnb_4bit_quant_type: nf4 |
|
- bnb_4bit_use_double_quant: True |
|
- bnb_4bit_compute_dtype: float16 |
|
|
|
### Framework versions |
|
|
|
|
|
- PEFT 0.6.0.dev0 |
