README.md

metadata

language: en
library_name: mlsae
license: mit
tags:
  - model_hub_mixin
  - pytorch_model_hub_mixin
datasets:
  - monology/pile-uncopyrighted

mlsae-pythia-70m-deduped-x64-k64-tfm

A Multi-Layer Sparse Autoencoder (MLSAE) trained on the residual stream activation vectors from every layer of EleutherAI/pythia-70m-deduped with an expansion factor of 64 and k = 64, over 1 billion tokens from monology/pile-uncopyrighted. This model includes the underlying transformer.

For more details, see:

• Paper: https://arxiv.org/abs/2409.04185
• GitHub repository: https://github.com/tim-lawson/mlsae
• Weights & Biases project: https://wandb.ai/timlawson-/mlsae