README.md

---
license: mit
language:
- en
library_name: peft
---

# Master Thesis: High-Fidelity Video Background Music Generation using Transformers
This is the corresponding GitLab Repository of my Master Thesis. The goal this thisis is to generate video background 
music by the adaptation of MusicGen (https://arxiv.org/pdf/2306.05284.pdf) to video input as another input modality.
This should be accomplished by mapping video information into the T5 text embedding space on which MusicGen usually
works on. To this end, a Transformer Encoder network to accomplish this task, called Video Encoder. Two options are 
foreseen within the training loop for the Video Encoder: 

- freezing the weights within the MusicGen Audio Decoder
- adjusting the weights of the MusicGen Audio Decoder with Parameter Efficient Fine-Tuning (PEFT) using LoRA (https://arxiv.org/abs/2106.09685)


# Installation
- create a Python virtual environment with `Python 3.11`
- check https://pytorch.org/get-started/previous-versions/ to install `PyTorch 2.1.0` with `CUDA` on your machine
- install the local fork of audiocraft: `cd audiocraft; pip install -e .`
- install the other requirements: `pip install -r requirements.txt`


# Folder Structure
- `audiocraft` contains a local fork of the audiocraft library (https://github.com/facebookresearch/audiocraft) with 
little changes to the generation method, further information can be seen in `code/code_adaptations_audiocraft`.
- `code` contains the code for model `training` and `inference` of video background music
- `datasets` contains the code to create the datasets used for training within `data_preparation` and video examples 
used for the evaluation in `example_videos` 
- `evaluation` contains the code used to evaluate the datasets and created video embeddings
- `gradio_app` contains the code for interface to generate video background music

# Training
To train the models set the training parameters under `training/training_conf.yml` and start training with 
`python training/training.py`. The models weights will be stored under `training/models_audiocraft` or 
`training/models_peft` respectively.

# Inference
- start the user interface by running `python gradio_app/app.py`
- inside the interface select a video, parameters
- click on "submit" to start the generation

# Contact
For any questions contact me at [niklas.schulte@rwth-aachen.de](mailto:niklas.schulte@rwth-aachen.de)