README.md

---
datasets:
- danielpark/gorani-100k-llama2-13b-instruct
language:
- en
library_name: bitsandbytes, transformers, peft, accelerate, bitsandbytes, datasets, deepspeed, trl
pipeline_tag: text-generation
---

# The development of the multimodal LLM project for commercial purposes is prioritized, so the results and weights of GORANI project will no longer be updated. 


# GORANI 100k
- LFM: [llama2-13b-chat](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf)
- Model: [danielpark/gorani-100k-llama2-13b-instruct](https://huggingface.co/danielpark/gorani-100k-llama2-13b-instruct)
- Dataset: [danielpark/gorani-100k](https://huggingface.co/danielpark/gorani-100k)
- **License**: This model is licensed under the Meta's [LLaMA2 license](https://github.com/facebookresearch/llama/blob/main/LICENSE). You may not use it commercially, and you must adhere to the licenses of the included datasets. Therefore, I currently adopt the strictest and most restrictive license. Please refrain from using it for commercial purposes under any circumstances until an official license is issued. 

<br>

KORANI is derived from GORANI, a project within llama2 that experiments with the distribution of appropriate datasets to transfer or distill knowledge based on English datasets. Officially, it's called Grid Of Ranvier Node In llama2 (GORANI), based on the biological term Ranvier Node, and aims to explore the optimal dataset for transferring knowledge in various languages and specific domains. Due to strict licensing issues with English datasets, GORANI is primarily for research purposes. Therefore, we are refining and training a commercially usable Korean dataset on top of llama2, based on the experimental results of the GORANI project, and this project is named KORANI (Korean GORANI).
- I have conducted preliminary experiments using various techniques such as RoPE scaling, Attention Sinks, and Flash Attention 1 and 2, SWA(Sliding Window Attention), GQA(Grouped Query Attention).
- Please do not use the current model weights as they are not official model weight.
- The most stringent non-commercial use license (CC-BY-NC-4.0) among the licenses of the datasets used for training is also applied to the model weights.
- On 2023-11-12, it was decided that all projects would be kept private. It may be released in a non-public model format on cloud platforms by 2024.

<br>

## Template
For safety, I used the default system message from Llama-2. But if a system message is specified in any datasets, I use that content.
```python
### System:
{System}

### User:
{New user input}

### Input:
{Optional additional user input}

### Response:
{New assistant answer}
```
## Update
- Since we cannot control resources, we will record the schedule retrospectively.

| Update Schedule | Task Description           | Status |
|-----------------|----------------------------|--------|
| 23-10-05           | Completed training - 19.7k 13b weight (specific data)|  Done       |
| 23-10-06           | Submitted hf model weights (REV 01) |    Done    |
| 23-10-20           | Q.C                         |   Done     |
| 23-11-12           | Changed to a private project.                         |   Kept private     |



## Caution
The model weights and dataset have not been properly curated yet and are strictly prohibited for use under any license. In relation to this, the developers do not assume any responsibility, either implicitly or explicitly.


## Revisions
| Revision       | Commit Hash                                                 | Updated   | Train Process   | Status        |
| ---------------|------------------------------------------------------------|------------|------------------|---------------|
| Revision 01     | [6d30494fa8da84128499d55075eef57094336d03](https://huggingface.co/danielpark/gorani-100k-llama2-13b-instruct/commit/6d30494fa8da84128499d55075eef57094336d03) | 23.10.04  | 19,740/100,000     | On Training   |