This is karpathy's model from the llm.c project converted to HF format to investigate bfloat16 performance.
The training run was 150B tokens, 1.5 epochs over the 100B FineWeb sample dataset.
There's active work underway at https://github.com/karpathy/llm.c so I'd suggest following the developments there!
- Downloads last month
- 156