Create README.md

README.md

@@ -0,0 +1,43 @@
+
+---
+language:
+- ko
+tags:
+- llama-2
+- instruct
+- instruction
+pipeline_tag: text-generation
+license: llama2
+---
+
+# Llama-2-ko-OpenOrca-gugugo-13B
+
+This model was trained for PoC purposes. This is part of an experiment to check whether model performance improves when fine-tuned with large data of about 1 million samples.
+
+### Model Details
+- Base Model: [beomi/llama-2-koen-13b](https://huggingface.co/beomi/llama-2-koen-13b)
+
+### Datasets
+Trained on 1 million samples from the dataset. The training infrastructure used AWS g5.12xlarge x 2ea (total of NVIDIA A10G 8 GPUs).
+
+- [OpenOrca-gugugo-ko](https://huggingface.co/datasets/squarelike/OpenOrca-gugugo-ko)
+
+### Hyperparameters
+The hyperparameters are simply heuristic values. For reference only:
+
+```python
+learning_rate = 3e-5
+lr_scheduler = "constant_with_warmup"
+batch_size = 1
+gradient_accumulation_steps = 8
+lora_alpha = 16
+lora_r = 16
+lora_dropout = 0.1
+lora_target_modules = "[gate_proj, down_proj, up_proj, q_proj, k_proj, o_proj, v_proj]"
+use_flash_attention_2 = True
+```
+
+### License
+- Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License, under LLAMA 2 COMMUNITY LICENSE AGREEMENT
+
+This model was created as a personal experiment, unrelated to the organization I work for.