Update README.md
Browse files
README.md
CHANGED
|
@@ -3,6 +3,10 @@ license: other
|
|
| 3 |
license_name: yi-license
|
| 4 |
license_link: LICENSE
|
| 5 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
|
| 7 |
## Model description
|
| 8 |
|
|
|
|
| 3 |
license_name: yi-license
|
| 4 |
license_link: LICENSE
|
| 5 |
---
|
| 6 |
+
## This is LoRA adapter for AEZAKMI v2 based on Yi-34B-200K
|
| 7 |
+
|
| 8 |
+
I had to change max_positional_embeddings in config.json and model_max_length to 4096 for training to start, otherwise I was OOMing straight away.
|
| 9 |
+
My first attempt had max_positional_embeddings set to 16384 and model_max_length set to 200000. This allowed fine-tuning to finish, but model was broken after applying LoRA and merging it.
|
| 10 |
|
| 11 |
## Model description
|
| 12 |
|