Update README.md
Browse files
README.md
CHANGED
|
@@ -5,6 +5,8 @@ datasets:
|
|
| 5 |
base_model:
|
| 6 |
- deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
|
| 7 |
---
|
|
|
|
|
|
|
| 8 |
These transcoders were trained on the outputs of the first 15 MLPs in deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B. We used 10 billion tokens from FineWeb edu deduped at a context length of 2048. The number of latents is 65,536 and a linear skip connection is included.
|
| 9 |
|
| 10 |
Fraction of variance unexplained ranges from 0.01 to 0.37.
|
|
|
|
| 5 |
base_model:
|
| 6 |
- deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
|
| 7 |
---
|
| 8 |
+
SAEs and transcoders can be loaded using https://github.com/EleutherAI/sae.
|
| 9 |
+
|
| 10 |
These transcoders were trained on the outputs of the first 15 MLPs in deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B. We used 10 billion tokens from FineWeb edu deduped at a context length of 2048. The number of latents is 65,536 and a linear skip connection is included.
|
| 11 |
|
| 12 |
Fraction of variance unexplained ranges from 0.01 to 0.37.
|