File size: 1,060 Bytes
42f4cfa 608fa9b d5fac40 172f473 00da718 c54ee96 17d615e c54ee96 172f473 71a3e59 172f473 c54ee96 172f473 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
---
license: apache-2.0
datasets:
- grammarly/coedit
language:
- en
metrics:
- accuracy
tags:
- torchtune
- grammar-correction
---
### Llama3 CoEdit
This is a Llama3 8B based model trained using [torchtune](https://pytorch.org/torchtune) on the `grammarly/coedit` dataset.
### Training details
The exact training script ([lora_finetune_distributed](./lora_finetune_distributed.py)) and config ([8B_lora.yaml](./8B_lora.yaml)) are both included in this repository.
**Training command**: ``tune run --nproc_per_node 8 lora_finetune_distributed --config 8B_lora.yaml``
> Yes I used 8 GPUs :)
In order to add the dataset, I added the following lines to the config:
```
dataset:
_component_: torchtune.datasets.instruct_dataset
source: grammarly/coedit
template: GrammarErrorCorrectionTemplate
column_map: {"sentence": "src", "output": "tgt"}
train_on_input: False
split: train
```
**Loss curve**
### Evaluation results |