TristanBehrens/bach-garland-transformer

Bach Garland Transformer - A Transformer model trained on Johann Sebastian Bach Style music

Say Hello on LinkedIn and X.

This is a Transformer model trained on music by Johann Sebastian Bach. It includes all pieces of Bach's music that can be played on church organ. The samples come in the prototypical Garland notation.

The dataset contains 100K samples and comes with a total token count of 144M.

How to use

1. Clone this repository and follow the installation instructions: https://github.com/AI-Guru/helibrunna/
2. Open and run the notebook examples/music.ipynb. Do not forget to add the id of this model.
3. Enjoy!

Training

Trained with Helibrunna by Dr. Tristan Behrens.

Configuration

training:
  model_name: bach_garland_transformer
  batch_size: 40
  lr: 0.001
  lr_warmup_steps: 1000
  lr_decay_until_steps: 10000
  lr_decay_factor: 0.001
  weight_decay: 0.1
  amp_precision: bfloat16
  weight_precision: float32
  enable_mixed_precision: true
  num_epochs: 8
  output_dir: output/bach_garland_transformer
  save_every_step: 500
  log_every_step: 10
  wandb_project: bach_garland
  torch_compile: false
model:
  type: transformer
  dim: 64
  n_layers: 4
  n_heads: 4
  fc_scale: 2
  context_length: 2048
  vocab_size: 178
dataset:
  hugging_face_id: TristanBehrens/bach_garland_2024-100K
tokenizer:
  type: whitespace
  fill_token: '[EOS]'