|
--- |
|
license: apache-2.0 |
|
base_model: Qwen/Qwen2-1.5B |
|
metrics: |
|
- accuracy |
|
datasets: |
|
- BEE-spoke-data/stepbasin-books |
|
language: |
|
- en |
|
--- |
|
|
|
[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="200" height="32"/>](https://wandb.ai/pszemraj/long-generation-tests/runs/ethp25f9) |
|
# Qwen2-1.5B-stepbasin-books |
|
|
|
> [!IMPORTANT] |
|
> this was finetuned at 16384 context length |
|
|
|
This is an experiment on long context text **generation** (i.e. 6k+ tokens generated) to evaluate if/when generation breaks down, etc. As such, all the data on which this model has been fine-tuned are full-length books. |
|
## Details |
|
|
|
This model is a fine-tuned version of [Qwen/Qwen2-1.5B](https://huggingface.co/Qwen/Qwen2-1.5B) on https://github.com/stepbasin/books/tree/master/books |
|
|
|
It achieves the following results on the evaluation set: |
|
- Loss: 2.8110 |
|
- Accuracy: 0.4298 |
|
- Num Input Tokens Seen: 44040192 |
