|
--- |
|
|
|
|
|
{} |
|
--- |
|
|
|
# Model Card for Kimiko_13B |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
This is my new Kimiko models, trained with LLaMA2-13B for...purpose |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
|
|
|
|
- **Developed by:** nRuaif |
|
- **Model type:** Decoder only |
|
- **License:** CC BY-NC-SA |
|
- **Finetuned from model [optional]:** LLaMA 2 |
|
|
|
### Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** https://github.com/OpenAccess-AI-Collective/axolotl |
|
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl) |
|
## Uses |
|
|
|
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. --> |
|
|
|
|
|
### Direct Use |
|
|
|
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. --> |
|
|
|
This model is trained on 3k examples of instructions dataset, high quality roleplay, for best result follow this format |
|
``` |
|
<<HUMAN>> |
|
How to do abc |
|
|
|
<<AIBOT>> |
|
Here is how |
|
|
|
Or with system prompting for roleplay |
|
|
|
<<SYSTEM>> |
|
A's Persona: |
|
B's Persona: |
|
Scenario: |
|
Add some instruction here on how you want your RP to go. |
|
``` |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
<!-- This section is meant to convey both technical and sociotechnical limitations. --> |
|
|
|
All bias of this model come from LlaMA2 with an exception of NSFW bias..... |
|
|
|
|
|
|
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
|
3000 examples from LIMAERP, LIMA and I sample 1000 good instruction from Airboro |
|
|
|
### Training Procedure |
|
|
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
|
|
Model is trained with 1 L4 from GCP costing a whooping 2.5USD |
|
|
|
|
|
|
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> |
|
|
|
3 epochs with 0.0002 lr, full 4096 ctx token, QLoRA |
|
|
|
#### Speeds, Sizes, Times [optional] |
|
|
|
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. --> |
|
|
|
It takes 18 hours to train this model with xformers enable |
|
|
|
[More Information Needed] |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[More Information Needed] |
|
|
|
## Environmental Impact |
|
|
|
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> |
|
|
|
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). |
|
|
|
- **Hardware Type:** L4 with 12CPUs 48gb ram |
|
- **Hours used:** 5 |
|
- **Cloud Provider:** GCP |
|
- **Compute Region:** US |
|
- **Carbon Emitted:** 0.5KG |
|
|
|
|
|
|
