acalatrava
commited on
Commit
•
e4ad697
1
Parent(s):
f31cecf
v2
Browse files- README.md +52 -0
- pytorch_model.bin +1 -1
README.md
ADDED
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
datasets:
|
4 |
+
- cerebras/SlimPajama-627B
|
5 |
+
- bigcode/starcoderdata
|
6 |
+
- sam-mosaic/orca-gpt4-chatml
|
7 |
+
- alvations/globalvoices-en-es
|
8 |
+
language:
|
9 |
+
- en
|
10 |
+
- es
|
11 |
+
---
|
12 |
+
<div align="center">
|
13 |
+
|
14 |
+
# TinyLlama-1.1B-translate-en-es
|
15 |
+
|
16 |
+
</div>
|
17 |
+
|
18 |
+
This is a finetuned version with a partial dataset from alvations/globalvoices-en-es to test performance on translation task. It has been trained to translate english to spanish and viceversa with only 20k rows from the dataset.
|
19 |
+
|
20 |
+
The translation is not very accurate but it shows a lot of potential.
|
21 |
+
|
22 |
+
In order to use it you have to follow the chatml standard like so:
|
23 |
+
---
|
24 |
+
english to spanish:
|
25 |
+
```
|
26 |
+
<|im_start|>user Translate this to spanish: ```A father and son, who have been living off grid for 20 years, encounter an outsider who threatens to destroy the utopia they've built.```
|
27 |
+
<|im_start|>assistant
|
28 |
+
```
|
29 |
+
This will provide the following result:
|
30 |
+
```
|
31 |
+
Un padre y hijo, que han vivido sin comida desde hace 20 años, encuentran un invitado quien amenaza con destruir la utopía que ellos han creado.
|
32 |
+
```
|
33 |
+
---
|
34 |
+
spanish to english:
|
35 |
+
```
|
36 |
+
<|im_start|>user Traduce esto al ingles: ```España se queda sin Copilot para Windows 11: la regulación de la UE frena su despliegue en Europa.```
|
37 |
+
<|im_start|>assistant
|
38 |
+
```
|
39 |
+
Which will be completed as:
|
40 |
+
```
|
41 |
+
Spain is left without Copilot for Windows 11: the control of the UE has halted its deployment in Europe.
|
42 |
+
```
|
43 |
+
|
44 |
+
---
|
45 |
+
The results are far from perfect but there are A LOT of room to improvement since it was finetuned with only 20k rows from the dataset (which has 355k rows) for 2 epoch. This training took only about 5 hours on a "M1 Pro" processor.
|
46 |
+
|
47 |
+
The base model used is a fine-tuned model with orca dataset [acalatrava/TinyLlama-1.1B-orca-gpt4](https://huggingface.co/acalatrava/TinyLlama-1.1B-orca-gpt4)
|
48 |
+
|
49 |
+
### Training
|
50 |
+
- **Method**: QLORA
|
51 |
+
- **Time**: 10h on a M1 Pro 32GB
|
52 |
+
- **Based on**: [https://colab.research.google.com/drive/1Zmaceu65d7w4Tcd-cfnZRb6k_Tcv2b8g](https://colab.research.google.com/drive/1Zmaceu65d7w4Tcd-cfnZRb6k_Tcv2b8g) removing quantization since it's not supported on MPS
|
pytorch_model.bin
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
size 2200171846
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:45b41b0806b99333a580819febf0320f7313e20b0301a22d926aff95693c3a9e
|
3 |
size 2200171846
|