Voiceblock demo: Attempt 8

app.py

@@ -0,0 +1,67 @@
+import torch
+import torchaudio
+import voicebox.src.attacks.offline.perturbation.voicebox.voicebox as vb #To access VoiceBox class
+#import voicebox.src.attacks.online.voicebox_streamer as streamer #To access VoiceBoxStreamer class
+import numpy as np
+from voicebox.src.constants import PPG_PRETRAINED_PATH
+
+#Set voicebox default parameters
+LOOKAHEAD = 5
+voicebox_kwargs={'win_length': 256,
+    'ppg_encoder_hidden_size': 256,
+    'use_phoneme_encoder': True,
+    'use_pitch_encoder': True,
+    'use_loudness_encoder': True,
+    'spec_encoder_lookahead_frames': 0,
+    'spec_encoder_type': 'mel',
+    'spec_encoder_mlp_depth': 2,
+    'bottleneck_lookahead_frames': LOOKAHEAD,
+    'ppg_encoder_path': PPG_PRETRAINED_PATH,
+    'n_bands': 128,
+    'spec_encoder_hidden_size': 512,
+    'bottleneck_skip': True,
+    'bottleneck_hidden_size': 512,
+    'bottleneck_feedforward_size': 512,
+    'bottleneck_type': 'lstm',
+    'bottleneck_depth': 2,
+    'control_eps': 0.5,
+    'projection_norm': float('inf'),
+    'conditioning_dim': 512}
+
+#Load pretrained model:
+model = vb.VoiceBox(**voicebox_kwargs)
+model.load_state_dict(torch.load('voicebox/pretrained/voicebox/voicebox_final.pt', map_location=torch.device('cpu')), strict=True)
+model.eval()
+
+#Define function to convert final audio format:
+def float32_to_int16(waveform):
+    waveform = waveform / np.abs(waveform).max()
+    waveform = waveform * 32767
+    waveform = waveform.astype(np.int16)
+    waveform = waveform.ravel()
+    return waveform
+
+#Define predict function:
+def predict(inp):
+    #How to transform audio from string to tensor
+    waveform, sample_rate = torchaudio.load(inp)
+
+    #Run model without changing weights
+    with torch.no_grad():
+        waveform = model(waveform)
+
+    #Transform output audio into gradio-readable format
+    waveform = waveform.numpy()
+    waveform = float32_to_int16(waveform)
+    return sample_rate, waveform
+
+#Set up gradio interface
+import gradio as gr
+
+interface = gr.Interface(
+    fn=predict,
+    inputs=gr.Audio(type="filepath"),
+    outputs=gr.Audio()
+)
+
+interface.launch()

requirements.txt

@@ -0,0 +1,28 @@
+torch==1.10.0
+torchaudio==0.10.0
+torchvision
+torchcrepe
+tensorboard
+textgrid
+Pillow
+numpy
+tqdm
+jiwer
+librosa
+pandas
+protobuf==3.20.0
+git+https://github.com/ludlows/python-pesq#egg=pesq
+psutil
+pystoi
+pytest
+pyworld
+pyyaml
+matplotlib
+seaborn
+ipython
+scipy
+scikit-learn
+ipywebrtc
+argbind
+sounddevice
+keyboard

voicebox/README.md

@@ -0,0 +1,136 @@
+<h1 align="center">VoiceBlock</h1>
+<h4 align="center"> Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models</h4>
+<div align="center">
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/???/???.ipynb)
+[![Demo](https://img.shields.io/badge/Web-Demo-blue)](https://master.d3hvhbnf7qxjtf.amplifyapp.com/)
+[![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](/LICENSE)
+
+</div>
+<p align="center"><img src="./figures/vb_color_logo.png" width="200"/></p>
+
+
+## Contents
+
+* <a href="#install">Installation</a>
+* <a href="#reproduce">Reproducing Results</a>
+* <a href="#streamer">Streaming Implementation</a>
+* <a href="#citation">Citation</a>
+
+<h2 id="install">Installation</h2>
+
+1. Clone the repository:
+
+        git clone https://github.com/voiceboxneurips/voicebox.git
+
+2. We recommend working from a clean environment, e.g. using `conda`:
+
+        conda create --name voicebox python=3.9
+        source activate voicebox 
+
+3. Install dependencies:
+
+        cd voicebox
+        pip install -r requirements.txt
+        pip install -e .
+
+4. Grant permissions:
+
+        chmod -R u+x scripts/
+
+<h2 id="reproduce">Reproducing Results</h2>
+
+To reproduce our results, first download the corresponding data. Note that to download the [VoxCeleb1 dataset](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html), you must register and obtain a username and password.
+
+| Task | Dataset (Size) | Command |
+|---|---|---|
+| Objective evaluation | VoxCeleb1  (39G) | `python scripts/downloads/download_voxceleb.py --subset=1 --username=<VGG_USERNAME> --password=<VGG_PASSWORD>` |
+| WER / supplemental evaluations | LibriSpeech `train-clean-360` (23G) | `./scripts/downloads/download_librispeech_eval.sh` |
+| Train attacks | LibriSpeech `train-clean-100` (11G) | `./scripts/downloads/download_librispeech_train.sh` |
+
+
+We provide scripts to reproduce our experiments and save results, including generated audio, to named and time-stamped subdirectories within `runs/`. To reproduce our objective evaluation experiments using pre-trained attacks, run:
+
+```
+python scripts/experiments/evaluate.py
+```
+
+To reproduce our training, run:
+
+```
+python scripts/experiments/train.py
+```
+
+<h2 id="streamer">Streaming Implementation</h2>
+
+As a proof of concept, we provide a streaming implementation of VoiceBox capable of modifying user audio in real-time. Here, we provide installation instructions for MacOS and Ubuntu 20.04.
+
+<h3 id="streamer-mac">MacOS</h3>
+
+See video below:
+
+<a href="https://youtu.be/LcNjO5E7F3E">
+<p align="center"><img src="./figures/demo_thumbnail.png" width="500"/></p>
+</a>
+
+<h3 id="streamer-ubuntu">Ubuntu 20.04</h3>
+
+
+1. Open a terminal and follow the [installation instructions](#install) above. Change directory to the root of this repository.
+
+2. Run the following command:
+   
+        pacmd load-module module-null-sink sink_name=voicebox sink_properties=device.description=voicebox
+
+   If you are using PipeWire instead of PulseAudio:
+        
+        pactl load-module module-null-sink media.class=Audio/Sink sink_name=voicebox sink_properties=device.description=voicebox
+        
+   PulseAudio is the default on Ubuntu. If you haven't changed your system defaults, you are probably using PulseAudio. This will add "voicebox" as an output device. Select it as the input to your chosen audio software.
+
+3. Find which audio device to read and write from. In your conda environment, run:
+  
+        python -m sounddevice
+
+   You will get output similar to this:
+  
+          0 HDA Intel HDMI: 0 (hw:0,3), ALSA (0 in, 8 out)
+          1 HDA Intel HDMI: 1 (hw:0,7), ALSA (0 in, 8 out)
+          2 HDA Intel HDMI: 2 (hw:0,8), ALSA (0 in, 8 out)
+          3 HDA Intel HDMI: 3 (hw:0,9), ALSA (0 in, 8 out)
+          4 HDA Intel HDMI: 4 (hw:0,10), ALSA (0 in, 8 out)
+          5 hdmi, ALSA (0 in, 8 out)
+          6 jack, ALSA (2 in, 2 out)
+          7 pipewire, ALSA (64 in, 64 out)
+          8 pulse, ALSA (32 in, 32 out)
+        * 9 default, ALSA (32 in, 32 out)
+  
+   In this example, we are going to route the audio through PipeWire (channel 7). This will be our INPUT_NUM and OUTPUT_NUM
+
+4. First, we need to create a conditioning embedding. To do this, run the enrollment script and follow its on-screen instructions:
+  
+        python scripts/streamer/enroll.py --input INPUT_NUM
+
+5. We can now use the streamer. Run:
+  
+        python scripts/stream.py --input INPUT_NUM --output OUTPUT_NUM
+
+6. Once the streamer is running, open `pavucontrol`.
+
+   a. In `pavucontrol`, go to the "Playback" tab and find "ALSA pug-in [python3.9]: ALSA Playback on". Set the output to "voicebox".
+
+   b. Then, go to "Recording" and find "ALSA pug-in [python3.9]: ALSA Playback from", and set the input to your desired microphone device.
+
+<h2 id="citation">Citation</h2>
+
+If you use this your academic research, please cite the following:
+
+```
+@inproceedings{authors2022voicelock,
+title={VoiceBlock: Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models},
+author={Patrick O'Reilly, Andreas Bugler, Keshav Bhandari, Max Morrison, Bryan Pardo},
+booktitle={Neural Information Processing Systems},
+month={November},
+year={2022}
+}
+```

voicebox/pretrained/denoiser/demucs/dns_48.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cfd4151600ed611d4af05083f4633d4fc31b53761cff8a185293346df745988
+size 75486933

voicebox/pretrained/phoneme/causal_ppg_128_hidden_128_hop.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be4c7a60c9af77e50af86924df8b73eb0c861a46f461e3bfe825c523a0a1a969
+size 1175695

voicebox/pretrained/phoneme/causal_ppg_256_hidden.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e8f20e4973a6b91002c97605f993cf6e16a24ca9d0d39e183438a8c16d85c87
+size 4556495

voicebox/pretrained/phoneme/causal_ppg_256_hidden_256_hop.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0836df2f8465b53d4e0b5b14f1d1ef954b3570d6f95f1af22c3ac19b3e10099
+size 4573903

voicebox/pretrained/phoneme/causal_ppg_256_hidden_512_hop.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a860d6f01058dc14b984845d27e681b5fe7c3bfffe41350e2e6e0f92e72778ad
+size 4608719

voicebox/pretrained/phoneme/ppg_causal_small.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4627bc2b63798df3391fe5c9ccbd72b929dc146b84f0fe61d1aa22848d107973
+size 18002639

voicebox/pretrained/speaker/resemblyzer/resemblyzer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:afb2230a894f5a8f91263ff0b4811bde1ea5981bedda45a579c225e5a602ada3
+size 5697307

voicebox/pretrained/speaker/resnetse34v2/resnetse34v2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d96a4dad0118e9945bc7e676d8e5ff34d493ca2209fe188b3f982005132369bc
+size 32311667

voicebox/pretrained/speaker/yvector/yvector.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2b4228cc772e689f800f1f9dc91d4ef4ee289e7e62f2822805edfc5b7faf399
+size 57703939

voicebox/pretrained/universal/universal_final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f435535934f6c8c24fda42c251e65f41627b0660d3420ba1c694e25a82be033e
+size 128811

voicebox/pretrained/voicebox/voicebox_final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eb26234cc493182545dbfcc74501f6df7e90347ca3e2a94a7966978325a34ccd
+size 30232012

voicebox/pretrained/voicebox/voicebox_final.yaml

@@ -0,0 +1,20 @@
+win_length:  256
+ppg_encoder_hidden_size:  256
+use_phoneme_encoder:  True
+use_pitch_encoder:  True
+use_loudness_encoder:  True
+spec_encoder_lookahead_frames: 0
+spec_encoder_type:  'mel'
+spec_encoder_mlp_depth:  2
+bottleneck_lookahead_frames: 5
+ppg_encoder_path:  'pretrained/phoneme/causal_ppg_256_hidden.pt'
+n_bands: 128
+spec_encoder_hidden_size: 512
+bottleneck_skip:  True
+bottleneck_hidden_size: 512
+bottleneck_feedforward_size: 512
+bottleneck_type:  'lstm'
+bottleneck_depth:  2
+control_eps:  0.5
+projection_norm: 'inf'
+conditioning_dim:  512

voicebox/requirements.txt

@@ -0,0 +1,28 @@
+torch==1.10.0
+torchaudio==0.10.0
+torchvision
+torchcrepe
+tensorboard
+textgrid
+Pillow
+numpy
+tqdm
+jiwer
+librosa
+pandas
+protobuf==3.20.0
+git+https://github.com/ludlows/python-pesq#egg=pesq
+psutil
+pystoi
+pytest
+pyworld
+pyyaml
+matplotlib
+seaborn
+ipython
+scipy
+scikit-learn
+ipywebrtc
+argbind
+sounddevice
+keyboard

voicebox/scripts/downloads/download_librispeech_eval.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+set -e
+
+DOWNLOADS_SCRIPTS_DIR=$(eval dirname "$(readlink -f "$0")")
+SCRIPTS_DIR="$(dirname "$DOWNLOADS_SCRIPTS_DIR")"
+PROJECT_DIR="$(dirname "$SCRIPTS_DIR")"
+
+DATA_DIR="${PROJECT_DIR}/data/"
+CACHE_DIR="${PROJECT_DIR}/cache/"
+
+mkdir -p "${DATA_DIR}"
+mkdir -p "${CACHE_DIR}"
+
+# download train-clean-360 subset
+echo "downloading LibriSpeech train-clean-360..."
+wget http://www.openslr.org/resources/12/train-clean-360.tar.gz
+
+# extract train-clean-360 subset
+echo "extracting LibriSpeech train-clean-360..."
+tar -xf train-clean-360.tar.gz \
+    -C "${DATA_DIR}"
+
+# delete archive
+rm -f "train-clean-360.tar.gz"

voicebox/scripts/downloads/download_librispeech_train.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+set -e
+
+DOWNLOADS_SCRIPTS_DIR=$(eval dirname "$(readlink -f "$0")")
+SCRIPTS_DIR="$(dirname "$DOWNLOADS_SCRIPTS_DIR")"
+PROJECT_DIR="$(dirname "$SCRIPTS_DIR")"
+
+DATA_DIR="${PROJECT_DIR}/data/"
+CACHE_DIR="${PROJECT_DIR}/cache/"
+
+mkdir -p "${DATA_DIR}"
+mkdir -p "${CACHE_DIR}"
+
+# download test-clean subset
+echo "downloading LibriSpeech test-clean..."
+wget http://www.openslr.org/resources/12/test-clean.tar.gz
+
+# extract test-clean subset
+echo "extracting LibriSpeech test-clean..."
+tar -xf test-clean.tar.gz \
+    -C "${DATA_DIR}"
+
+# delete archive
+rm -f "test-clean.tar.gz"
+
+# download test-other subset
+echo "downloading LibriSpeech test-other..."
+wget http://www.openslr.org/resources/12/test-other.tar.gz
+
+# extract test-other subset
+echo "extracting LibriSpeech test-other..."
+tar -xf test-other.tar.gz \
+    -C "${DATA_DIR}"
+
+# delete archive
+rm -f "test-other.tar.gz"
+
+# download train-clean-100 subset
+echo "downloading LibriSpeech train-clean-100..."
+wget http://www.openslr.org/resources/12/train-clean-100.tar.gz
+
+# extract train-clean-100 subset
+echo "extracting LibriSpeech train-clean-100..."
+tar -xf train-clean-100.tar.gz \
+    -C "${DATA_DIR}"
+
+# delete archive
+rm -f "train-clean-100.tar.gz"
+
+# download LibriSpeech alignments dataset
+wget -O alignments.zip https://zenodo.org/record/2619474/files/librispeech_alignments.zip?download=1
+unzip -d "${DATA_DIR}/LibriSpeech/" alignments.zip
+rm  -f alignments.zip

voicebox/scripts/downloads/download_rir_noise.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+
+set -e
+
+DOWNLOADS_SCRIPTS_DIR=$(eval dirname "$(readlink -f "$0")")
+SCRIPTS_DIR="$(dirname "$DOWNLOADS_SCRIPTS_DIR")"
+PROJECT_DIR="$(dirname "$SCRIPTS_DIR")"
+
+DATA_DIR="${PROJECT_DIR}/data/"
+CACHE_DIR="${PROJECT_DIR}/cache/"
+
+REAL_RIR_DIR="${DATA_DIR}/rir/real/"
+SYNTHETIC_RIR_DIR="${DATA_DIR}/rir/synthetic/"
+ROOM_NOISE_DIR="${DATA_DIR}/noise/room/"
+PS_NOISE_DIR="${DATA_DIR}/noise/pointsource/"
+
+mkdir -p "${REAL_RIR_DIR}"
+mkdir -p "${SYNTHETIC_RIR_DIR}"
+mkdir -p "${ROOM_NOISE_DIR}"
+mkdir -p "${PS_NOISE_DIR}"
+
+# download RIR/noise composite dataset
+echo "downloading RIR/noise dataset..."
+wget -O "${DATA_DIR}/rirs_noises.zip" https://www.openslr.org/resources/28/rirs_noises.zip
+
+# extract RIR/noise composite dataset
+echo "unzipping RIR/noise dataset..."
+unzip "${DATA_DIR}/rirs_noises.zip" -d "${DATA_DIR}/"
+
+# delete archive
+rm -f "${DATA_DIR}/rirs_noises.zip"
+
+# organize pointsource noise data
+echo "extracting point-source noise data"
+cp -a "${DATA_DIR}/RIRS_NOISES/pointsource_noises"/. "${PS_NOISE_DIR}"
+
+# organize room noise data
+echo "extracting room noise data"
+room_noises=($(find "${DATA_DIR}/RIRS_NOISES/real_rirs_isotropic_noises/" -maxdepth 1 -name '*noise*' -type f))
+cp -- "${room_noises[@]}" "${ROOM_NOISE_DIR}"
+
+# organize real RIR data
+echo "extracting recorded RIR data"
+rirs=($(find "${DATA_DIR}/RIRS_NOISES/real_rirs_isotropic_noises/" ! -name '*noise*' ))
+cp -- "${rirs[@]}" "${REAL_RIR_DIR}"
+
+# organize synthetic RIR data
+echo "extracting synthetic RIR data"
+cp -a "${DATA_DIR}/RIRS_NOISES/simulated_rirs"/. "${SYNTHETIC_RIR_DIR}"
+
+# delete redundant data
+rm -rf "${DATA_DIR}/RIRS_NOISES/"
+
+# separate near-field and far-field RIRs
+NEARFIELD_RIR_DIR="${REAL_RIR_DIR}/nearfield/"
+FARFIELD_RIR_DIR="${REAL_RIR_DIR}/farfield/"
+
+mkdir -p "${NEARFIELD_RIR_DIR}"
+mkdir -p "${FARFIELD_RIR_DIR}"
+
+# read list of far-field RIRs
+readarray -t FF_RIR_LIST < "${DOWNLOADS_SCRIPTS_DIR}/ff_rir.txt"
+
+# move far-field RIRs
+for name in "${FF_RIR_LIST[@]}"; do
+    mv "$name" "${FARFIELD_RIR_DIR}/$(basename "$name")"
+done
+
+# move remaining near-field RIRs
+for name in "${REAL_RIR_DIR}"/*.wav; do
+    mv "$name" "${NEARFIELD_RIR_DIR}/$(basename "$name")"
+done
+

voicebox/scripts/downloads/download_voxceleb.py

@@ -0,0 +1,189 @@
+import argparse
+from pathlib import Path
+import subprocess
+import hashlib
+import tarfile
+from zipfile import ZipFile
+
+from src.constants import VOXCELEB1_DATA_DIR, VOXCELEB2_DATA_DIR
+from src.utils import ensure_dir
+
+################################################################################
+# Download VoxCeleb1 dataset using valid credentials
+################################################################################
+
+
+def parse_args():
+
+    """Parse command-line arguments"""
+    parser = argparse.ArgumentParser(add_help=False)
+
+    parser.add_argument(
+        '--subset',
+        type=int,
+        default=1,
+        help='Specify which VoxCeleb subset to download: 1 or 2'
+    )
+
+    parser.add_argument(
+        '--username',
+        type=str,
+        default=None,
+        help='User name provided by VGG to access VoxCeleb dataset'
+    )
+
+    parser.add_argument(
+        '--password',
+        type=str,
+        default=None,
+        help='Password provided by VGG to access VoxCeleb dataset'
+    )
+
+    return parser.parse_args()
+
+
+def md5(f: str):
+    """
+    Return MD5 checksum for file. Code adapted from voxceleb_trainer repository:
+    https://github.com/clovaai/voxceleb_trainer/blob/master/dataprep.py
+    """
+
+    hash_md5 = hashlib.md5()
+    with open(f, "rb") as f:
+        for chunk in iter(lambda: f.read(4096), b""):
+            hash_md5.update(chunk)
+    return hash_md5.hexdigest()
+
+
+def download(username: str,
+             password: str,
+             save_path: str,
+             lines: list):
+    """
+    Given a list of dataset shards formatted as <URL, MD5>, download
+    each using `wget` and verify checksums. Code adapted from voxceleb_trainer
+    repository:
+    https://github.com/clovaai/voxceleb_trainer/blob/master/dataprep.py
+    """
+
+    for line in lines:
+        url = line.split()[0]
+        md5gt = line.split()[1]
+        outfile = url.split('/')[-1]
+
+        # download files
+        out = subprocess.call(
+            f'wget {url} --user {username} --password {password} -O {save_path}'
+            f'/{outfile}', shell=True)
+        if out != 0:
+            raise ValueError(f'Download failed for {url}')
+
+        # verify checksum
+        md5ck = md5(f'{save_path}/{outfile}')
+        if md5ck == md5gt:
+            print(f'Checksum successful for {outfile}')
+        else:
+            raise Warning(f'Checksum failed for {outfile}')
+
+
+def concatenate(save_path: str, lines: list):
+    """
+    Given a specification in the format <FMT, FILENAME, MD5>, concatenate all
+    downloaded data shards matching FMT into the file FILENAME and verify
+    checksums. Code adapted from voxceleb_trainer repository:
+    https://github.com/clovaai/voxceleb_trainer/blob/master/dataprep.py
+    """
+
+    for line in lines:
+        infile = line.split()[0]
+        outfile = line.split()[1]
+        md5gt = line.split()[2]
+
+        # concatenate shards
+        out = subprocess.call(
+            f'cat {save_path}/{infile} > {save_path}/{outfile}', shell=True)
+
+        # verify checksum
+        md5ck = md5(f'{save_path}/{outfile}')
+        if md5ck == md5gt:
+            print(f'Checksum successful for {outfile}')
+        else:
+            raise Warning(f'Checksum failed for {outfile}')
+
+        # delete shards
+        out = subprocess.call(
+            f'rm {save_path}/{infile}', shell=True)
+
+
+def full_extract(save_path: str, f: str):
+    """
+    Extract contents of compressed archive to data directory
+    """
+
+    save_path = str(save_path)
+    f = str(f)
+
+    print(f'Extracting {f}')
+
+    if f.endswith(".tar.gz"):
+        with tarfile.open(f, "r:gz") as tar:
+            tar.extractall(save_path)
+
+    elif f.endswith(".zip"):
+        with ZipFile(f, 'r') as zf:
+            zf.extractall(save_path)
+
+
+def main():
+
+    args = parse_args()
+
+    # prepare to load dataset file paths
+    downloads_dir = Path(__file__).parent
+
+    if args.subset == 1:
+        data_dir = VOXCELEB1_DATA_DIR
+    elif args.subset == 2:
+        data_dir = VOXCELEB2_DATA_DIR
+    else:
+        raise ValueError(f'Invalid VoxCeleb subset {args.subset}')
+
+    ensure_dir(data_dir)
+
+    # load dataset file paths
+    with open(downloads_dir / f'voxceleb{args.subset}_file_parts.txt', 'r') as f:
+        file_parts_list = f.readlines()
+
+    # load output file paths
+    with open(downloads_dir / f'voxceleb{args.subset}_files.txt', 'r') as f:
+        files_list = f.readlines()
+
+    # download subset
+    download(
+        username=args.username,
+        password=args.password,
+        save_path=data_dir,
+        lines=file_parts_list
+    )
+
+    # merge shards
+    concatenate(save_path=data_dir, lines=files_list)
+
+    # account for test data
+    archives = [file.split()[1] for file in files_list]
+    test = f"vox{args.subset}_test_{'wav' if args.subset == 1 else 'aac'}.zip"
+    archives.append(test)
+
+    # extract all compressed data
+    for file in archives:
+        full_extract(data_dir, data_dir / file)
+
+    # organize extracted data
+    out = subprocess.call(f'mv {data_dir}/dev/aac/* {data_dir}/aac/ && rm -r '
+                          f'{data_dir}/dev', shell=True)
+    out = subprocess.call(f'mv -v {data_dir}/{"wav" if args.subset == 1 else "aac"}/*'
+                          f' {data_dir}/voxceleb{args.subset}', shell=True)
+
+
+if __name__ == "__main__":
+    main()

voicebox/scripts/downloads/ff_rir.txt

@@ -0,0 +1,132 @@
+data/rir/real/air_type1_air_binaural_lecture_0_1.wav
+data/rir/real/RWCP_type3_rir_cirline_ofc_imp_rev.wav
+data/rir/real/RWCP_type1_rir_cirline_jr2_imp110.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_4_90_3.wav
+data/rir/real/RVB2014_type1_rir_largeroom2_far_anglb.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_60.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_3_90_3.wav
+data/rir/real/air_type1_air_binaural_lecture_0_5.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_30.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_30.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_15.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_165.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_75.wav
+data/rir/real/air_type1_air_binaural_lecture_0_3.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_0.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_0.wav
+data/rir/real/RWCP_type2_rir_cirline_jr1_imp110.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_5_90_3.wav
+data/rir/real/RVB2014_type1_rir_largeroom1_far_anglb.wav
+data/rir/real/air_type1_air_binaural_lecture_1_1.wav
+data/rir/real/RVB2014_type1_rir_largeroom1_far_angla.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_7_90_3.wav
+data/rir/real/RWCP_type2_rir_cirline_ofc_imp070.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp070.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_150.wav
+data/rir/real/air_type1_air_binaural_lecture_1_5.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp100.wav
+data/rir/real/RWCP_type1_rir_cirline_jr2_imp100.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp130.wav
+data/rir/real/air_type1_air_phone_corridor_hfrp.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp130.wav
+data/rir/real/RVB2014_type1_rir_largeroom1_near_angla.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_75.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp150.wav
+data/rir/real/air_type1_air_phone_lecture_hhp.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_105.wav
+data/rir/real/air_type1_air_phone_stairway_hfrp.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_105.wav
+data/rir/real/RWCP_type2_rir_cirline_jr1_imp090.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp050.wav
+data/rir/real/air_type1_air_phone_stairway2_hfrp.wav
+data/rir/real/air_type1_air_phone_stairway2_hhp.wav
+data/rir/real/RWCP_type1_rir_cirline_jr2_imp060.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_90.wav
+data/rir/real/RWCP_type2_rir_cirline_jr1_imp130.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp030.wav
+data/rir/real/RVB2014_type1_rir_largeroom2_near_angla.wav
+data/rir/real/air_type1_air_binaural_lecture_0_6.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp070.wav
+data/rir/real/air_type1_air_phone_stairway1_hhp.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_45.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp090.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_135.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_180.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp100.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp080.wav
+data/rir/real/RWCP_type2_rir_cirline_ofc_imp090.wav
+data/rir/real/RWCP_type1_rir_cirline_jr2_imp080.wav
+data/rir/real/air_type1_air_binaural_lecture_1_2.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp070.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_150.wav
+data/rir/real/air_type1_air_binaural_lecture_1_4.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_3_0_3.wav
+data/rir/real/RVB2014_type1_rir_largeroom1_near_anglb.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_15.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_120.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp050.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_1_90_3.wav
+data/rir/real/air_type1_air_phone_stairway_hhp.wav
+data/rir/real/RWCP_type1_rir_cirline_jr2_imp120.wav
+data/rir/real/RWCP_type2_rir_cirline_e2b_imp110.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp010.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_15.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_135.wav
+data/rir/real/air_type1_air_phone_bt_stairway_hhp.wav
+data/rir/real/RWCP_type2_rir_cirline_e2b_imp070.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp120.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp110.wav
+data/rir/real/air_type1_air_binaural_lecture_0_4.wav
+data/rir/real/RWCP_type2_rir_cirline_ofc_imp050.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_90.wav
+data/rir/real/RWCP_type1_rir_cirline_jr2_imp090.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_0.wav
+data/rir/real/air_type1_air_phone_stairway1_hfrp.wav
+data/rir/real/air_type1_air_binaural_lecture_1_3.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp050.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp080.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_165.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_45.wav
+data/rir/real/air_type1_air_phone_bt_corridor_hhp.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_2_90_3.wav
+data/rir/real/RWCP_type2_rir_cirline_ofc_imp110.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_120.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_3_180_3.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp110.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp060.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_45.wav
+data/rir/real/RVB2014_type1_rir_largeroom2_far_angla.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_60.wav
+data/rir/real/RWCP_type2_rir_cirline_jr1_imp070.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp130.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_3_135_3.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_75.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_180.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp120.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_60.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_105.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_135.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_3_45_3.wav
+data/rir/real/air_type1_air_binaural_lecture_1_6.wav
+data/rir/real/RWCP_type2_rir_cirline_e2b_imp090.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp170.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_90.wav
+data/rir/real/RWCP_type1_rir_cirline_jr2_imp070.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp110.wav
+data/rir/real/air_type1_air_phone_lecture_hfrp.wav
+data/rir/real/RVB2014_type1_rir_largeroom2_near_anglb.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_165.wav
+data/rir/real/RWCP_type2_rir_cirline_ofc_imp130.wav
+data/rir/real/air_type1_air_binaural_stairway_1_1_150.wav
+data/rir/real/RWCP_type1_rir_cirline_jr1_imp090.wav
+data/rir/real/RWCP_type2_rir_cirline_e2b_imp130.wav
+data/rir/real/RWCP_type1_rir_cirline_ofc_imp060.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_180.wav
+data/rir/real/RWCP_type2_rir_cirline_jr1_imp050.wav
+data/rir/real/air_type1_air_binaural_stairway_1_3_30.wav
+data/rir/real/air_type1_air_binaural_lecture_0_2.wav
+data/rir/real/air_type1_air_binaural_aula_carolina_1_6_90_3.wav
+data/rir/real/RWCP_type2_rir_cirline_e2b_imp050.wav
+data/rir/real/RWCP_type1_rir_cirline_e2b_imp090.wav
+data/rir/real/air_type1_air_phone_corridor_hhp.wav
+data/rir/real/air_type1_air_binaural_stairway_1_2_120.wav

voicebox/scripts/downloads/voxceleb1_file_parts.txt

@@ -0,0 +1,5 @@
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox1_dev_wav_partaa e395d020928bc15670b570a21695ed96
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox1_dev_wav_partab bbfaaccefab65d82b21903e81a8a8020
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox1_dev_wav_partac 017d579a2a96a077f40042ec33e51512
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox1_dev_wav_partad 7bb1e9f70fddc7a678fa998ea8b3ba19
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox1_test_wav.zip 185fdc63c3c739954633d50379a3d102

voicebox/scripts/downloads/voxceleb1_files.txt

@@ -0,0 +1 @@
+vox1_dev_wav_parta* vox1_dev_wav.zip ae63e55b951748cc486645f532ba230b

voicebox/scripts/downloads/voxceleb2_file_parts.txt

@@ -0,0 +1,9 @@
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_dev_aac_partaa da070494c573e5c0564b1d11c3b20577
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_dev_aac_partab 17fe6dab2b32b48abaf1676429cdd06f
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_dev_aac_partac 1de58e086c5edf63625af1cb6d831528
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_dev_aac_partad 5a043eb03e15c5a918ee6a52aad477f9
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_dev_aac_partae cea401b624983e2d0b2a87fb5d59aa60
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_dev_aac_partaf fc886d9ba90ab88e7880ee98effd6ae9
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_dev_aac_partag d160ecc3f6ee3eed54d55349531cb42e
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_dev_aac_partah 6b84a81b9af72a9d9eecbb3b1f602e65
+http://cnode01.mm.kaist.ac.kr/voxceleb/vox1a/vox2_test_aac.zip 0d2b3ea430a821c33263b5ea37ede312

voicebox/scripts/downloads/voxceleb2_files.txt

@@ -0,0 +1 @@
+vox2_dev_aac_parta* vox2_dev_aac.zip bbc063c46078a602ca71605645c2a402

voicebox/scripts/experiments/evaluate.py

@@ -0,0 +1,915 @@
+import os.path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torchaudio
+import psutil
+import pickle
+
+import random
+import argparse
+
+import librosa as li
+from sklearn.utils import shuffle
+from sklearn.neighbors import NearestNeighbors
+
+from pesq import pesq, NoUtterancesError
+
+from tqdm import tqdm
+from sklearn.preprocessing import LabelEncoder
+import numpy as np
+from pathlib import Path
+from tqdm import tqdm
+import builtins
+import math
+import jiwer
+from jiwer import wer, cer
+
+from typing import Iterable
+from copy import deepcopy
+
+from distutils.util import strtobool
+
+from src.data import *
+from src.constants import *
+from src.models import *
+from src.simulation import *
+from src.preprocess import *
+from src.attacks.offline import *
+from src.loss import *
+from src.pipelines import *
+from src.utils import *
+
+################################################################################
+# Evaluate attacks on speaker recognition systems
+################################################################################
+
+EVAL_DATASET = "voxceleb"  # "librispeech"
+LOOKAHEAD = 5
+VOICEBOX_PATH = VOICEBOX_PRETRAINED_PATH
+UNIVERSAL_PATH = UNIVERSAL_PRETRAINED_PATH
+BATCH_SIZE = 20     # evaluation batch size
+N_QUERY = 15        # number of query utterances per speaker
+N_CONDITION = 10    # number of conditioning utterances per speaker
+N_ENROLL = 20       # number of enrolled utterances per speaker
+ADV_ENROLL = False  # evaluate under assumption adversarial audio is enrolled
+TARGETS_TRAIN = 'centroid'  # 'random', 'same', 'single', 'median'
+TARGETS_TEST = 'centroid'  # 'random', 'same', 'single', 'median'
+TRANSFER = True  # evaluate attacks on unseen model
+DENOISER = False  # evaluate with unseen denoiser defense applied to queries
+SIMULATION = False  # apply noisy channel simulation to all queries in evaluation
+COMPUTE_OBJECTIVE_METRICS = True  # PESQ, STOI
+
+
+def set_random_seed(seed: int = 123):
+    """Set random seed to allow for reproducibility"""
+    random.seed(seed)
+    torch.manual_seed(seed)
+
+    if torch.backends.cudnn.is_available():
+        # torch.backends.cudnn.benchmark = True
+        torch.backends.cudnn.deterministic = True
+
+
+def param_count(m: nn.Module, trainable: bool = False):
+    """Count the number of trainable parameters (weights) in a model"""
+    if trainable:
+        return builtins.sum(
+            [p.shape.numel() for p in m.parameters() if p.requires_grad])
+    else:
+        return builtins.sum([p.shape.numel() for p in m.parameters()])
+
+
+def pad_sequence(sequences: list):
+
+    max_len = max([s.shape[-1] for s in sequences])
+
+    padded = torch.zeros(
+        (len(sequences), 1, max_len),
+        dtype=sequences[0].dtype,
+        device=sequences[0].device)
+
+    for i, s in enumerate(sequences):
+        padded[i, :, :s.shape[-1]] = s
+
+    return padded
+
+
+@torch.no_grad()
+def compute_embeddings_batch(audio: list,
+                             p: Pipeline,
+                             defense: nn.Module = nn.Identity()):
+    """Compute batched speaker embeddings"""
+
+    assert isinstance(p.model, SpeakerVerificationModel)
+    emb = [p(defense(audio[i].to(p.device))).to('cpu') for i in range(len(audio))]
+    emb = torch.cat(emb, dim=0)
+    return emb
+
+
+@torch.no_grad()
+def compute_transcripts_batch(audio: list, p: Pipeline):
+    """Compute batched transcripts"""
+
+    assert isinstance(p.model, SpeechRecognitionModel)
+    transcripts = []
+    for i in range(len(audio)):
+        t = p.model.transcribe(audio[i].to(p.device))
+        if isinstance(t, str):
+            transcripts.append(t)
+        elif isinstance(t, list):
+            transcripts.extend(t)
+
+    assert len(transcripts) == len(audio), f'Transcript format error'
+
+    return transcripts
+
+
+@torch.no_grad()
+def compute_attack_batch(audio: list,
+                         a: TrainableAttack,
+                         c: torch.Tensor):
+
+    if len(c) < len(audio):
+        c = c.repeat(len(audio), 1, 1)
+    adv = [a.perturbation(audio[i].to(a.pipeline.device),
+                          y=c[i:i+1].to(a.pipeline.device)).to('cpu').reshape(1, 1, -1)
+           for i in range(len(audio))]
+    return adv
+
+
+@torch.no_grad()
+def compute_pesq(audio1: list, audio2: list, mode: str = 'wb'):
+
+    assert len(audio1) == len(audio2)
+    scores = []
+
+    for i in range(len(audio1)):
+        try:
+            scores.append(
+                pesq(DataProperties.get('sample_rate'),
+                     tensor_to_np(audio1[i]).flatten(),
+                     tensor_to_np(audio2[i]).flatten(),
+                     mode)
+            )
+        except NoUtterancesError:
+            print("PESQ error, skipping audio file...")
+    return scores
+
+
+@torch.no_grad()
+def compute_stoi(audio1: list, audio2: list, extended: bool = False):
+
+    assert len(audio1) == len(audio2)
+    scores = []
+    for i in range(len(audio1)):
+        scores.append(
+            stoi(tensor_to_np(audio1[i]).flatten(),
+                 tensor_to_np(audio2[i]).flatten(),
+                 DataProperties.get('sample_rate'),
+                 extended=extended)
+        )
+    return scores
+
+
+@torch.no_grad()
+def build_ls_dataset(pipelines: dict):
+    """
+    Build LibriSpeech evaluation dataset on disk holding:
+      * query audio
+      * query embeddings
+      * conditioning embeddings
+      * enrolled embeddings
+      * ground-truth query transcripts
+    """
+
+    # locate dataset
+    data_dir = LIBRISPEECH_DATA_DIR / 'train-clean-360'
+    cache_dir = CACHE_DIR / 'ls_wer_eval'
+    ensure_dir(cache_dir)
+
+    assert os.path.isdir(data_dir), \
+        f'LibriSpeech `train-clean-360` subset required for evaluation'
+
+    spkr_dirs = list(data_dir.glob("*/"))
+    spkr_dirs = [s_d for s_d in spkr_dirs if os.path.isdir(s_d)]
+
+    # catalog audio and load transcripts
+    for spkr_dir in tqdm(spkr_dirs, total=len(spkr_dirs), desc='Building dataset'):
+
+        # identify speaker
+        spkr_id = spkr_dir.parts[-1]
+
+        # check whether cached data exists for speaker
+        spkr_cache_dir = cache_dir / spkr_id
+        if os.path.isdir(spkr_cache_dir):
+            continue
+
+        # each recording session has a separate subdirectory
+        rec_dirs = list(spkr_dir.glob("*/"))
+        rec_dirs = [r_d for r_d in rec_dirs if os.path.isdir(r_d)]
+
+        # for each speaker, process & store necessary (non-adversarial) data
+        all_audio = []
+        all_transcripts = []
+
+        # for each recording session, extract all audio files and transcripts
+        for rec_dir in rec_dirs:
+
+            rec_id = rec_dir.parts[-1]
+            trans_fn = rec_dir / f"{spkr_id}-{rec_id}.trans.txt"
+
+            # open transcript file
+            with open(trans_fn, "r") as f:
+                trans_idx = f.readlines()
+
+            if len(trans_idx) == 0:
+                print(f"Error: empty transcript {trans_fn}")
+                continue
+
+            for line in trans_idx:
+
+                split_line = line.strip().split(" ")
+                audio_fn = rec_dir / f'{split_line[0]}.{LIBRISPEECH_EXT}'
+                transcript = " ".join(split_line[1:]).replace(" ", "|")
+
+                x, _ = li.load(audio_fn, mono=True, sr=16000)
+                all_audio.append(torch.as_tensor(x).reshape(1, 1, -1).float())
+                all_transcripts.append(transcript)
+
+        # shuffle audio and transcripts in same random order
+        all_audio, all_transcripts = shuffle(all_audio, all_transcripts)
+
+        # divide audio and transcripts
+        query_audio = all_audio[:N_QUERY]
+        query_transcripts = all_transcripts[:N_QUERY]
+        condition_audio = all_audio[N_QUERY:N_QUERY+N_CONDITION]
+        enroll_audio = all_audio[N_QUERY+N_CONDITION:][:N_ENROLL]
+
+        # check for sufficient audio in each category
+        if len(query_audio) < N_QUERY:
+            print(f"Error: insufficient query audio for speaker {spkr_id}")
+            continue
+        elif len(condition_audio) < N_CONDITION:
+            print(f"Error: insufficient conditioning audio for speaker {spkr_id}")
+            continue
+        elif len(enroll_audio) < N_ENROLL:
+            print(f"Error: insufficient enrollment audio for speaker {spkr_id}")
+            continue
+
+        # compute and save embeddings
+        for p_name, p in pipelines.items():
+
+            # compute and save query embeddings
+            query_emb = compute_embeddings_batch(query_audio, p)
+            f_query = spkr_cache_dir / p_name / 'query_emb.pt'
+            ensure_dir_for_filename(f_query)
+
+            # compute and save conditioning embeddings
+            condition_emb = compute_embeddings_batch(condition_audio, p)
+            f_condition = spkr_cache_dir / p_name / 'condition_emb.pt'
+            ensure_dir_for_filename(f_condition)
+
+            # compute and save enrolled embeddings
+            enroll_emb = compute_embeddings_batch(enroll_audio, p)
+            f_enroll = spkr_cache_dir / p_name / 'enroll_emb.pt'
+            ensure_dir_for_filename(f_enroll)
+
+            torch.save(query_emb, f_query)
+            torch.save(condition_emb, f_condition)
+            torch.save(enroll_emb, f_enroll)
+
+        # save query audio
+        f_audio = spkr_cache_dir / 'query_audio.pt'
+        torch.save(query_audio, f_audio)
+
+        # save query transcripts
+        f_transcript = spkr_cache_dir / 'query_trans.pt'
+        torch.save(query_transcripts, f_transcript)
+
+@torch.no_grad()
+def build_vc_dataset(pipelines: dict):
+    """
+    Build VoxCeleb evaluation dataset on disk holding:
+      * query audio
+      * query embeddings
+      * conditioning embeddings
+      * enrolled embeddings
+    """
+
+    # locate dataset
+    data_dir = VOXCELEB1_DATA_DIR / 'voxceleb1'
+    cache_dir = CACHE_DIR / 'vc_wer_eval'
+    ensure_dir(cache_dir)
+
+    assert os.path.isdir(data_dir), \
+        f'VoxCeleb1 dataset required for evaluation'
+
+    spkr_dirs = list(data_dir.glob("*/"))
+    spkr_dirs = [s_d for s_d in spkr_dirs if os.path.isdir(s_d)]
+
+    # catalog audio
+    for spkr_dir in tqdm(spkr_dirs, total=len(spkr_dirs), desc='Building dataset'):
+
+        # identify speaker
+        spkr_id = spkr_dir.parts[-1]
+
+        # check whether cached data exists for speaker
+        spkr_cache_dir = cache_dir / spkr_id
+        if os.path.isdir(spkr_cache_dir):
+            continue
+
+        # each recording session has a separate subdirectory
+        rec_dirs = list(spkr_dir.glob("*/"))
+        rec_dirs = [r_d for r_d in rec_dirs if os.path.isdir(r_d)]
+
+        # for each speaker, process & store necessary (non-adversarial) data
+        all_audio = []
+
+        # for each recording session, extract all audio files and transcripts
+        for rec_dir in rec_dirs:
+            for audio_fn in rec_dir.glob(f"*.{VOXCELEB1_EXT}"):
+                x, _ = li.load(audio_fn, mono=True, sr=16000)
+                all_audio.append(torch.as_tensor(x).reshape(1, 1, -1).float())
+
+        # shuffle audio in random order
+        all_audio = shuffle(all_audio)
+
+        # divide audio and transcripts
+        query_audio = all_audio[:N_QUERY]
+        condition_audio = all_audio[N_QUERY:N_QUERY+N_CONDITION]
+        enroll_audio = all_audio[N_QUERY+N_CONDITION:][:N_ENROLL]
+
+        # check for sufficient audio in each category
+        if len(query_audio) < N_QUERY:
+            print(f"Error: insufficient query audio for speaker {spkr_id}")
+            continue
+        elif len(condition_audio) < N_CONDITION:
+            print(f"Error: insufficient conditioning audio for speaker {spkr_id}")
+            continue
+        elif len(enroll_audio) < N_ENROLL:
+            print(f"Error: insufficient enrollment audio for speaker {spkr_id}")
+            continue
+
+        # compute and save embeddings
+        for p_name, p in pipelines.items():
+
+            # compute and save query embeddings
+            query_emb = compute_embeddings_batch(query_audio, p)
+            f_query = spkr_cache_dir / p_name / 'query_emb.pt'
+            ensure_dir_for_filename(f_query)
+
+            # compute and save conditioning embeddings
+            condition_emb = compute_embeddings_batch(condition_audio, p)
+            f_condition = spkr_cache_dir / p_name / 'condition_emb.pt'
+            ensure_dir_for_filename(f_condition)
+
+            # compute and save enrolled embeddings
+            enroll_emb = compute_embeddings_batch(enroll_audio, p)
+            f_enroll = spkr_cache_dir / p_name / 'enroll_emb.pt'
+            ensure_dir_for_filename(f_enroll)
+
+            torch.save(query_emb, f_query)
+            torch.save(condition_emb, f_condition)
+            torch.save(enroll_emb, f_enroll)
+
+        # save query audio
+        f_audio = spkr_cache_dir / 'query_audio.pt'
+        torch.save(query_audio, f_audio)
+
+@torch.no_grad()
+def asr_metrics(true: list, hypothesis: list, batch_size: int = 5):
+    """
+    Compute word and character error rates between two lists of corresponding
+    transcripts
+    """
+
+    assert len(true) == len(hypothesis)
+
+    n_batches = math.ceil(len(true) / batch_size)
+
+    transform_wer = jiwer.Compose([
+        jiwer.ToLowerCase(),
+        jiwer.RemoveWhiteSpace(replace_by_space=True),
+        jiwer.RemoveMultipleSpaces(),
+        jiwer.ReduceToSingleSentence(word_delimiter="|"),
+        jiwer.ReduceToListOfListOfWords(word_delimiter="|"),
+    ])
+
+    wer_score = 0.0
+    cer_score = 0.0
+
+    wer_n = 0
+    cer_n = 0
+
+    for i in range(n_batches):
+
+        batch_true = true[i*batch_size:(i+1)*batch_size]
+        batch_hypothesis = hypothesis[i*batch_size:(i+1)*batch_size]
+
+        wer_n_batch = builtins.sum([len(s.split('|')) for s in batch_true])
+        cer_n_batch = builtins.sum([len(s) for s in batch_true])
+
+        attack_cer = cer(batch_true, batch_hypothesis)
+        attack_wer = wer(batch_true, batch_hypothesis,
+                         truth_transform=transform_wer,
+                         hypothesis_transform=transform_wer)
+
+        wer_score += wer_n_batch*attack_wer
+        cer_score += cer_n_batch*attack_cer
+
+        wer_n += wer_n_batch
+        cer_n += cer_n_batch
+
+    wer_score /= wer_n
+    cer_score /= cer_n
+
+    return wer_score, cer_score
+
+
+@torch.no_grad()
+def top_k(query: dict, enrolled: dict, k: int):
+    """
+    Compute portion of queries for which 'correct' ID appears in k-closest
+    enrolled entries
+    """
+
+    # concatenate query embeddings into single tensor
+    query_array = []
+    query_ids = []
+
+    for s_l in query.keys():
+        query_array.append(query[s_l])
+        query_ids.extend([s_l] * len(query[s_l]))
+
+    query_array = torch.cat(query_array, dim=0).squeeze().cpu().numpy()
+    query_ids = torch.as_tensor(query_ids).cpu().numpy()
+
+    # concatenate enrolled embeddings into single tensor
+    enrolled_array = []
+    enrolled_ids = []
+
+    for s_l in enrolled.keys():
+        enrolled_array.append(enrolled[s_l])
+        enrolled_ids.extend([s_l] * len(enrolled[s_l]))
+
+    enrolled_array = torch.cat(enrolled_array, dim=0).squeeze().cpu().numpy()
+    enrolled_ids = torch.as_tensor(enrolled_ids).cpu().numpy()
+
+    # embedding dimension
+    assert query_array.shape[-1] == enrolled_array.shape[-1]
+    d = query_array.shape[-1]
+
+    # index enrolled embeddings
+    knn = NearestNeighbors(n_neighbors=k, metric="cosine").fit(enrolled_array)
+
+    # `I` is a (n_queries, k) array holding the indices of the k-closest enrolled
+    # embeddings for each query; `D` is a (n_queries, k) array holding the corresponding
+    # embedding-space distances
+    D, I = knn.kneighbors(query_array, k, return_distance=True)
+
+    # for each row, see if at least one of the k nearest enrolled indices maps
+    # to a speaker ID that matches the query index's speaker id
+    targets = np.tile(query_ids.reshape(-1, 1), (1, k))
+
+    predictions = enrolled_ids[I]
+    matches = (targets == predictions).sum(axis=-1) > 0
+
+    return np.mean(matches)
+
+
+def init_attacks():
+    """
+    Initialize pre-trained speaker recognition pipelines and de-identification
+    attacks
+    """
+
+    # channel simulation
+    if SIMULATION:
+        sim = [
+            Offset(length=[-.15, .15]),
+            Noise(type='gaussian', snr=[30.0, 50.0]),
+            Bandpass(low=[300, 500], high=[3400, 7400]),
+            Dropout(rate=0.001)
+        ]
+    else:
+        sim = None
+
+    pipelines = {}
+
+    model_resnet = SpeakerVerificationModel(
+        model=ResNetSE34V2(nOut=512, encoder_type='ASP'),
+        n_segments=1,
+        segment_select='lin',
+        distance_fn='cosine',
+        threshold=0.0
+    )
+    model_resnet.load_weights(
+        MODELS_DIR / 'speaker' / 'resnetse34v2' / 'resnetse34v2.pt')
+
+    model_yvector = SpeakerVerificationModel(
+        model=YVector(),
+        n_segments=1,
+        segment_select='lin',
+        distance_fn='cosine',
+        threshold=0.0
+    )
+    model_yvector.load_weights(
+        MODELS_DIR / 'speaker' / 'yvector' / 'yvector.pt')
+
+    pipelines['resnet'] = Pipeline(
+        simulation=sim,
+        preprocessor=Preprocessor(Normalize(method='peak')),
+        model=model_resnet,
+        device='cuda' if torch.cuda.is_available() else 'cpu'
+    )
+
+    if TRANSFER:
+        pipelines['yvector'] = Pipeline(
+            simulation=sim,
+            preprocessor=Preprocessor(Normalize(method='peak')),
+            model=model_yvector,
+            device='cuda' if torch.cuda.is_available() else 'cpu'
+        )
+    else:
+        del model_yvector
+
+    # prepare to log attack progress
+    writer = Writer(
+        root_dir=RUNS_DIR,
+        name='evaluate-attacks',
+        use_timestamp=True,
+        log_iter=300,
+        use_tb=True
+    )
+
+    attacks = {}
+
+    # use consistent adversarial loss
+    adv_loss = SpeakerEmbeddingLoss(
+        targeted=False,
+        confidence=0.1,
+        threshold=0.0
+    )
+
+    # use consistent auxiliary loss across attacks
+    aux_loss = SumLoss().add_loss_function(
+        DemucsMRSTFTLoss(), 1.0
+    ).add_loss_function(L1Loss(), 1.0).to('cuda')
+
+    attacks['voicebox'] = VoiceBoxAttack(
+        pipeline=pipelines['resnet'],
+        adv_loss=adv_loss,
+        aux_loss=aux_loss,
+        lr=1e-4,
+        epochs=1,
+        batch_size=BATCH_SIZE,
+        voicebox_kwargs={
+            'win_length': 256,
+            'ppg_encoder_hidden_size': 256,
+            'use_phoneme_encoder': True,
+            'use_pitch_encoder': True,
+            'use_loudness_encoder': True,
+            'spec_encoder_lookahead_frames': 0,
+            'spec_encoder_type': 'mel',
+            'spec_encoder_mlp_depth': 2,
+            'bottleneck_lookahead_frames': LOOKAHEAD,
+            'ppg_encoder_path': PPG_PRETRAINED_PATH,
+            'n_bands': 128,
+            'spec_encoder_hidden_size': 512,
+            'bottleneck_skip': True,
+            'bottleneck_hidden_size': 512,
+            'bottleneck_feedforward_size': 512,
+            'bottleneck_type': 'lstm',
+            'bottleneck_depth': 2,
+            'control_eps': 0.5,
+            'projection_norm': float('inf'),
+            'conditioning_dim': 512
+        },
+        writer=writer,
+        checkpoint_name='voicebox-attack'
+    )
+    attacks['voicebox'].load(VOICEBOX_PATH)
+
+    attacks['universal'] = AdvPulseAttack(
+        pipeline=pipelines['resnet'],
+        adv_loss=adv_loss,
+        pgd_norm=float('inf'),
+        pgd_variant=None,
+        scale_grad=None,
+        eps=0.08,
+        length=2.0,
+        align='start',
+        lr=1e-4,
+        normalize=True,
+        loop=True,
+        aux_loss=aux_loss,
+        epochs=1,
+        batch_size=BATCH_SIZE,
+        writer=writer,
+        checkpoint_name='universal-attack'
+    )
+    attacks['universal'].load(UNIVERSAL_PATH)
+
+    attacks['kenansville'] = KenansvilleAttack(
+        pipeline=pipelines['resnet'],
+        batch_size=BATCH_SIZE,
+        adv_loss=adv_loss,
+        threshold_db_low=4.0,  # fix threshold
+        threshold_db_high=4.0,
+        win_length=512,
+        writer=writer,
+        step_size=1.0,
+        search='bisection',
+        min_success_rate=0.2,
+        checkpoint_name='kenansville-attack'
+    )
+
+    attacks['noise'] = WhiteNoiseAttack(
+        pipeline=pipelines['resnet'],
+        adv_loss=adv_loss,
+        aux_loss=aux_loss,
+        snr_low=-10.0,  # fix threshold
+        snr_high=-10.0,
+        writer=writer,
+        step_size=1,
+        search='bisection',
+        min_success_rate=0.2,
+        checkpoint_name='noise-perturbation'
+    )
+
+    return attacks, pipelines, writer
+
+
+@torch.no_grad()
+def evaluate_attack(attack: TrainableAttack,
+                    speaker_pipeline: Pipeline,
+                    asr_pipeline: Pipeline):
+
+    if DENOISER:
+        from src.models.denoiser.demucs import load_demucs
+        defense = load_demucs('dns_48').to(
+            'cuda' if torch.cuda.is_available() else 'cpu')
+        defense.eval()
+    else:
+        defense = nn.Identity()
+
+    # prepare for GPU inference
+    if torch.cuda.is_available():
+
+        attack.pipeline.set_device('cuda')
+        speaker_pipeline.set_device('cuda')
+        asr_pipeline.set_device('cuda')
+        attack.perturbation.to('cuda')
+
+    # locate dataset
+    if EVAL_DATASET == "librispeech":
+        cache_dir = CACHE_DIR / 'ls_wer_eval'
+    else:
+        cache_dir = CACHE_DIR / 'vc_wer_eval'
+    assert os.path.isdir(cache_dir), \
+        f'Dataset must be built/cached before evaluation'
+
+    # prepare for PESQ/STOI calculations
+    all_pesq_scores = []
+    all_stoi_scores = []
+
+    # prepare for WER/CER computations
+    all_query_transcripts = []
+    all_pred_query_transcripts = []
+    all_adv_query_transcripts = []
+
+    # prepare for accuracy computations
+    all_query_emb = {}
+    all_adv_query_emb = {}
+    all_enroll_emb = {}
+    all_enroll_emb_centroid = {}
+
+    spkr_dirs = list(cache_dir.glob("*/"))
+    spkr_dirs = [s_d for s_d in spkr_dirs if os.path.isdir(s_d)]
+    for spkr_dir in tqdm(spkr_dirs, total=len(spkr_dirs), desc='Running evaluation'):
+
+        # identify speaker
+        spkr_id = spkr_dir.parts[-1]
+
+        # use integer IDs
+        if EVAL_DATASET != "librispeech":
+            spkr_id = spkr_id.split("id")[-1]
+
+        # identify speaker recognition model
+        if isinstance(speaker_pipeline.model.model, ResNetSE34V2):
+            model_name = 'resnet'
+        elif isinstance(speaker_pipeline.model.model, YVector):
+            model_name = 'yvector'
+        else:
+            raise ValueError(f'Invalid speaker recognition model')
+
+        # load clean embeddings
+        query_emb = torch.load(spkr_dir / model_name / 'query_emb.pt')
+        condition_emb = torch.load(spkr_dir / 'resnet' / 'condition_emb.pt')
+        enroll_emb = torch.load(spkr_dir / model_name / 'enroll_emb.pt')
+
+        # load clean audio
+        query_audio = torch.load(spkr_dir / 'query_audio.pt')
+
+        # if defense in use, re-compute query audio
+        if DENOISER:
+            query_emb = compute_embeddings_batch(
+                query_audio, speaker_pipeline, defense=defense
+            )
+
+        # load clean transcript
+        if EVAL_DATASET == "librispeech":
+            query_transcripts = torch.load(spkr_dir / 'query_trans.pt')
+        else:
+            query_transcripts = None
+
+        # compute conditioning embedding centroid
+        condition_centroid = condition_emb.mean(dim=(0, 1), keepdim=True)
+
+        # compute enrolled embedding centroid
+        enroll_centroid = enroll_emb.mean(dim=(0, 1), keepdim=True)
+
+        # compute adversarial query audio
+        adv_query_audio = compute_attack_batch(
+            query_audio, attack, condition_centroid)
+
+        # compute adversarial query embeddings; optionally, pass through
+        # unseen denoiser defense
+        adv_query_emb = compute_embeddings_batch(
+            adv_query_audio, speaker_pipeline, defense=defense
+        )
+
+        if EVAL_DATASET == "librispeech":
+
+            # compute clean predicted transcripts
+            pred_query_transcripts = compute_transcripts_batch(
+                query_audio, asr_pipeline
+            )
+
+            # compute adversarial transcripts
+            adv_query_transcripts = compute_transcripts_batch(
+                adv_query_audio, asr_pipeline
+            )
+
+        # compute objective quality metric scores
+        if COMPUTE_OBJECTIVE_METRICS:
+            pesq_scores = compute_pesq(query_audio, adv_query_audio)
+            stoi_scores = compute_stoi(query_audio, adv_query_audio)
+        else:
+            pesq_scores = np.zeros(len(query_audio))
+            stoi_scores = np.zeros(len(query_audio))
+
+        # store all objective quality metric scores
+        all_pesq_scores.extend(pesq_scores)
+        all_stoi_scores.extend(stoi_scores)
+
+        # store all unit-normalized clean, adversarial, and enrolled centroid
+        # embeddings
+        all_query_emb[int(spkr_id)] = F.normalize(query_emb.clone(), dim=-1)
+        all_adv_query_emb[int(spkr_id)] = F.normalize(adv_query_emb.clone(), dim=-1)
+        all_enroll_emb[int(spkr_id)] = F.normalize(enroll_emb.clone(), dim=-1)
+        all_enroll_emb_centroid[int(spkr_id)] = F.normalize(enroll_centroid.clone(), dim=-1)
+
+        # store all transcripts
+        if EVAL_DATASET == "librispeech":
+            all_query_transcripts.extend(query_transcripts)
+            all_pred_query_transcripts.extend(pred_query_transcripts)
+            all_adv_query_transcripts.extend(adv_query_transcripts)
+
+    # free GPU memory for similarity search
+    attack.pipeline.set_device('cpu')
+    speaker_pipeline.set_device('cpu')
+    asr_pipeline.set_device('cpu')
+    attack.perturbation.to('cpu')
+    torch.cuda.empty_cache()
+
+    # compute and display final objective quality metrics
+    print(f"PESQ (mean/std): {np.mean(all_pesq_scores)}/{np.std(all_pesq_scores)}")
+    print(f"STOI (mean/std): {np.mean(all_stoi_scores)}/{np.std(all_stoi_scores)}")
+
+    if EVAL_DATASET == "librispeech":
+
+        # compute and display final WER/CER metrics
+        wer, cer = asr_metrics(all_query_transcripts, all_adv_query_transcripts)
+        print(f"Adversarial WER / CER: {wer} / {cer}")
+
+        wer, cer = asr_metrics(all_query_transcripts, all_pred_query_transcripts)
+        print(f"Clean WER / CER: {wer} / {cer}")
+
+    else:
+        wer, cer = None, None
+
+    del (wer, cer, all_pesq_scores, all_stoi_scores,
+         all_query_transcripts, all_adv_query_transcripts, all_pred_query_transcripts)
+
+    # embedding-space cosine distance calculations
+    cos_dist_fn = EmbeddingDistance(distance_fn='cosine')
+
+    # mean clean-to-adversarial query embedding distance
+    total_query_dist = 0.0
+    n = 0
+    for spkr_id in all_query_emb.keys():
+        dist = cos_dist_fn(all_query_emb[spkr_id],
+                           all_adv_query_emb[spkr_id]).mean()
+        total_query_dist += len(all_query_emb[spkr_id]) * dist.item()
+        n += len(all_query_emb[spkr_id])
+    mean_query_dist = total_query_dist / n
+    print(f"\n\t\tMean cosine distance between clean and adversarial query "
+          f"embeddings: {mean_query_dist :0.4f}")
+
+    # mean adversarial-query-to-enrolled-centroid embedding distance
+    total_centroid_dist = 0.0
+    n = 0
+    for spkr_id in all_query_emb.keys():
+        n_queries = len(all_adv_query_emb[spkr_id])
+        dist = 0.0
+        for i in range(n_queries):
+            dist += cos_dist_fn(all_enroll_emb_centroid[spkr_id],
+                                all_adv_query_emb[spkr_id][i:i+1]).item()
+        total_centroid_dist += dist
+        n += n_queries
+    mean_centroid_dist = total_centroid_dist / n
+    print(f"\t\tMean cosine distance between clean enrolled centroids and "
+          f"adversarial query embeddings: {mean_centroid_dist :0.4f}")
+
+    # top-1 accuracy for clean queries (closest embedding)
+    top_1_clean_single = top_k(all_query_emb, all_enroll_emb, k=1)
+
+    # top-1 accuracy for clean queries (centroid embedding)
+    top_1_clean_centroid = top_k(all_query_emb, all_enroll_emb_centroid, k=1)
+
+    # top-10 accuracy for clean queries (closest embedding)
+    top_10_clean_single = top_k(all_query_emb, all_enroll_emb, k=10)
+
+    # top-10 accuracy for clean queries (centroid embedding)
+    top_10_clean_centroid = top_k(all_query_emb, all_enroll_emb_centroid, k=10)
+
+    # top-1 accuracy for adversarial queries (closest embedding)
+    top_1_adv_single = top_k(all_adv_query_emb, all_enroll_emb, k=1)
+
+    # top-1 accuracy for adversarial queries (centroid embedding)
+    top_1_adv_centroid = top_k(all_adv_query_emb, all_enroll_emb_centroid, k=1)
+
+    # top-10 accuracy for adversarial queries (closest embedding)
+    top_10_adv_single = top_k(all_adv_query_emb, all_enroll_emb, k=10)
+
+    # top-10 accuracy for adversarial queries (centroid embedding)
+    top_10_adv_centroid = top_k(all_adv_query_emb, all_enroll_emb_centroid, k=10)
+
+    print(f"\n\t\tTop-1 accuracy (clean embedding / nearest enrolled embedding) {top_1_clean_single :0.4f}",
+          f"\n\t\tTop-1 accuracy (clean embedding / nearest enrolled centroid) {top_1_clean_centroid :0.4f}",
+          f"\n\t\tTop-10 accuracy (clean embedding / nearest enrolled embedding) {top_10_clean_single :0.4f}"
+          f"\n\t\tTop-10 accuracy (clean embedding / nearest enrolled centroid) {top_10_clean_centroid :0.4f}",
+          f"\n\t\tTop-1 accuracy (adversarial embedding / nearest enrolled embedding {top_1_adv_single :0.4f}",
+          f"\n\t\tTop-1 accuracy (adversarial embedding / nearest enrolled centroid) {top_1_adv_centroid :0.4f}",
+          f"\n\t\tTop-10 accuracy (adversarial embedding / nearest enrolled embedding {top_10_adv_single :0.4f}",
+          f"\n\t\tTop-10 accuracy (adversarial embedding / nearest enrolled centroid) {top_10_adv_centroid :0.4f}"
+          )
+
+
+@torch.no_grad()
+def evaluate_attacks(attacks: dict,
+                     speaker_pipelines: dict,
+                     asr_pipeline: Pipeline):
+
+    for attack_name, attack in attacks.items():
+        for sp_name, sp in speaker_pipelines.items():
+            print(f'Evaluating {attack_name} against model {sp_name} '
+                  f'{"with" if DENOISER else "without"} denoiser defense')
+            evaluate_attack(attack, sp, asr_pipeline)
+
+
+def main():
+
+    # initial random seed (keep dataset order consistent)
+    set_random_seed(0)
+
+    # initialize pipelines
+    attacks, pipelines, writer = init_attacks()
+
+    # ensure that necessary data is cached
+    if EVAL_DATASET == "librispeech":
+        build_ls_dataset(pipelines)
+    else:
+        build_vc_dataset(pipelines)
+
+    # initialize ASR model
+    asr_model = SpeechRecognitionModel(
+        model=Wav2Vec2(),
+    )
+    asr_pipeline = Pipeline(
+        model=asr_model,
+        preprocessor=Preprocessor(Normalize(method='peak')),
+        device='cuda' if torch.cuda.is_available() else 'cpu'
+    )
+
+    writer.log_cuda_memory()
+
+    evaluate_attacks(attacks, pipelines, asr_pipeline)
+
+
+if __name__ == "__main__":
+    main()
+

voicebox/scripts/experiments/train.py

@@ -0,0 +1,282 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import torchaudio
+import psutil
+import pickle
+import librosa as li
+
+from torch.utils.data import TensorDataset
+
+import time
+import random
+import argparse
+from datetime import datetime
+
+import numpy as np
+import pandas as pd
+from typing import Dict
+from pathlib import Path
+from tqdm import tqdm
+import builtins
+
+from typing import Iterable
+from copy import deepcopy
+
+from distutils.util import strtobool
+
+from src.data import *
+from src.constants import *
+from src.models import *
+from src.simulation import *
+from src.preprocess import *
+from src.attacks.offline import *
+from src.loss import *
+from src.pipelines import *
+from src.utils import *
+
+################################################################################
+# Train VoiceBox attack
+################################################################################
+
+BATCH_SIZE = 20     # training batch size
+EPOCHS = 10         # training epochs
+TARGET_PCTL = 25    # de-identification strength; in [1,5,10,15,20,25,50,90,100]
+N_EMBEDDINGS_TRAIN = 15
+TARGETED = False
+TARGETS_TRAIN = 'centroid'  # 'random', 'same', 'single', 'median'
+TARGETS_TEST = 'centroid'  # 'random', 'same', 'single', 'median'
+
+# distributions of inter- ('targeted') and intra- ('untargeted') speaker
+# distances in each pre-trained model's embedding spaces, as measured between
+# individual utterances and their speaker centroid ('single-centroid') or
+# between all pairs of individual utterances ('single-single') over the
+# LibriSpeech test-clean dataset. This allows specification of attack strength
+# during the training process
+percentiles = {
+    'resnet': {
+        'targeted': {
+            'single-centroid': {1:.495, 5:.572, 10:.617, 15:.648, 20:.673, 25:.695, 50:.773, 90:.892, 100:1.127},
+            'single-single': {1:.560, 5:.630, 10:.672, 15:.700, 20:.722, 25:.742, 50:.813, 90:.924, 100:1.194}
+        },
+        'untargeted': {
+            'single-centroid': {1:.099, 5:.117, 10:.126, 15:.133, 20:.139, 25:.145, 50:.170, 90:.253, 100:.587},
+            'single-single': {1:.181, 5:.215, 10:.235, 15:.249, 20:.262, 25:.272, 50:.323, 90:.464, 100:.817}
+        },
+    },
+    'yvector': {
+        'targeted': {
+            'single-centroid': {1:.665, 5:.757, 10:.801, 15:.830, 20:.851, 25:.868, 50:.936, 90:1.056, 100:1.312},
+            'single-single': {1:.695, 5:.779, 10:.821, 15:.847, 20:.868, 25:.885, 50:.952, 90:1.072, 100:1.428}
+        },
+        'untargeted': {
+            'single-single': {1:.218, 5:.268, 10:.301, 15:.325, 20:.345, 25:.365, 50:.455, 90:.684, 100:1.156},
+            'single-centroid': {1:.114, 5:.143, 10:.159, 15:.170, 20:.180, 25:.190, 50:.242, 90:.413, 100:.874}
+        }
+    },
+}
+
+
+def set_random_seed(seed: int = 123):
+    """Set random seed to allow for reproducibility"""
+    random.seed(seed)
+    torch.manual_seed(seed)
+
+    if torch.backends.cudnn.is_available():
+        # torch.backends.cudnn.benchmark = True
+        torch.backends.cudnn.deterministic = True
+
+
+def param_count(m: nn.Module, trainable: bool = False):
+    """Count the number of trainable parameters (weights) in a model"""
+    if trainable:
+        return builtins.sum(
+            [p.shape.numel() for p in m.parameters() if p.requires_grad])
+    else:
+        return builtins.sum([p.shape.numel() for p in m.parameters()])
+
+
+def main():
+
+    set_random_seed(0)
+
+    model = SpeakerVerificationModel(
+        model=ResNetSE34V2(nOut=512, encoder_type='ASP'),
+        n_segments=1,
+        segment_select='lin',
+        distance_fn='cosine',
+        threshold=percentiles['resnet']['targeted']['single-centroid' if
+        TARGETS_TRAIN == 'centroid' else 'single-single'][TARGET_PCTL]
+    )
+    model.load_weights(MODELS_DIR / 'speaker' / 'resnetse34v2' / 'resnetse34v2.pt')
+
+    # instantiate training pipeline
+    pipeline = Pipeline(
+        simulation=None,
+        preprocessor=Preprocessor(Normalize(method='peak')),
+        model=model,
+        device='cuda' if torch.cuda.is_available() else 'cpu'
+    )
+
+    attacks = {}
+
+    # log training progress
+    writer = Writer(
+        root_dir=RUNS_DIR,
+        name='train-attacks',
+        use_timestamp=True,
+        log_iter=300,
+        use_tb=True
+    )
+
+    # adversarial training loss
+    adv_loss = SpeakerEmbeddingLoss(
+        targeted=TARGETED,
+        confidence=0.1,
+        threshold=pipeline.model.threshold
+    )
+
+    # auxiliary loss
+    aux_loss = SumLoss().add_loss_function(
+        DemucsMRSTFTLoss(), 1.0
+    ).add_loss_function(L1Loss(), 1.0).to('cuda')
+
+    # speech features loss actually seems to do better...
+    # aux_loss = SumLoss().add_loss_function(SpeechFeatureLoss(), 1e-6).to('cuda')
+
+    attacks['voicebox'] = VoiceBoxAttack(
+        pipeline=pipeline,
+        adv_loss=adv_loss,
+        aux_loss=aux_loss,
+        lr=1e-4,
+        epochs=EPOCHS,
+        batch_size=BATCH_SIZE,
+        voicebox_kwargs={
+            'win_length': 256,
+            'ppg_encoder_hidden_size': 256,
+            'use_phoneme_encoder': True,
+            'use_pitch_encoder': True,
+            'use_loudness_encoder': True,
+            'spec_encoder_lookahead_frames': 0,
+            'spec_encoder_type': 'mel',
+            'spec_encoder_mlp_depth': 2,
+            'bottleneck_lookahead_frames': 5,
+            'ppg_encoder_path': PPG_PRETRAINED_PATH,
+            'n_bands': 128,
+            'spec_encoder_hidden_size': 512,
+            'bottleneck_skip': True,
+            'bottleneck_hidden_size': 512,
+            'bottleneck_feedforward_size': 512,
+            'bottleneck_type': 'lstm',
+            'bottleneck_depth': 2,
+            'control_eps': 0.5,
+            'projection_norm': float('inf'),
+            'conditioning_dim': 512
+        },
+        writer=writer,
+        checkpoint_name='voicebox-attack'
+    )
+
+    attacks['universal'] = AdvPulseAttack(
+        pipeline=pipeline,
+        adv_loss=adv_loss,
+        pgd_norm=float('inf'),
+        pgd_variant=None,
+        scale_grad=None,
+        eps=0.08,
+        length=2.0,
+        align='random',  # 'start',
+        lr=1e-4,
+        normalize=True,
+        loop=True,
+        aux_loss=aux_loss,
+        epochs=EPOCHS,
+        batch_size=BATCH_SIZE,
+        writer=writer,
+        checkpoint_name='universal-attack'
+    )
+
+    if torch.cuda.is_available():
+
+        # prepare for multi-GPU training
+        device_ids = get_cuda_device_ids()
+
+        # wrap pipeline for multi-GPU training
+        pipeline = wrap_pipeline_multi_gpu(pipeline, device_ids)
+
+    # load training and validation datasets. Features will be computed and
+    # cached to disk, which may take some time
+    data_train = LibriSpeechDataset(
+        split='train-clean-100', features=['pitch', 'periodicity', 'loudness'])
+    data_test = LibriSpeechDataset(
+        split='test-clean', features=['pitch', 'periodicity', 'loudness'])
+
+    # reassign targets if necessary
+    compiled_train, compiled_test = create_embedding_dataset(
+        pipeline=pipeline,
+        select_train=TARGETS_TRAIN,
+        select_test=TARGETS_TEST,
+        data_train=data_train,
+        data_test=data_test,
+        targeted=TARGETED,
+        target_class=None,
+        num_embeddings_train=N_EMBEDDINGS_TRAIN,
+        batch_size=20
+    )
+
+    # extract embedding datasets
+    data_train = compiled_train['dataset']
+    data_test = compiled_test['dataset']
+
+    # log memory use prior to training
+    writer.log_info(f'Training data ready; memory use: '
+                    f'{psutil.virtual_memory().percent :0.3f}%')
+    writer.log_cuda_memory()
+
+    for attack_name, attack in attacks.items():
+
+        writer.log_info(f'Preparing {attack_name}...')
+
+        if torch.cuda.is_available():
+
+            attack.perturbation.to('cuda')
+            attack.pipeline.to('cuda')
+
+            # wrap attack for multi-GPU training
+            attack = wrap_attack_multi_gpu(attack, device_ids)
+
+        # evaluate performance
+        with torch.no_grad():
+            x_example = next(iter(data_train))['x'].to(pipeline.device)
+            st = time.time()
+            outs = attack.perturbation(x_example)
+            dur = time.time() - st
+
+        writer.log_info(
+            f'Processing time per input (device: '
+            f'{pipeline.device}): {dur/x_example.shape[0] :0.4f} (s)'
+        )
+        writer.log_info(f'Trainable parameters: '
+                        f'{param_count(attack.perturbation, trainable=True)}')
+        writer.log_info(f'Total parameters: {param_count(attack.perturbation, trainable=False)}')
+
+        # train
+        writer.log_info('Training attack...')
+        attack.train(data_train=data_train, data_val=data_test)
+
+        # evaluate
+        writer.log_info(f'Evaluating attack...')
+        x_adv, success, detection = attack.evaluate(
+            dataset=data_test
+        )
+
+        # log results summary: success rate in achieving target threshold
+        writer.log_info(
+            f'Success rate in meeting embedding distance threshold {pipeline.model.threshold}'
+            f' ({TARGET_PCTL}%): '
+            f'{success.flatten().mean().item()}'
+        )
+
+
+if __name__ == "__main__":
+    main()

voicebox/scripts/experiments/train_phoneme_predictor.py

@@ -0,0 +1,205 @@
+import torch
+import torch.nn as nn
+from torch.utils.data import DataLoader
+
+from src.models.phoneme import PPGEncoder
+from src.constants import LIBRISPEECH_NUM_PHONEMES, LIBRISPEECH_PHONEME_DICT
+from src.data import LibriSpeechDataset
+from src.utils.writer import Writer
+
+import numpy as np
+from sklearn.metrics import confusion_matrix, classification_report
+import seaborn as sn
+import pandas as pd
+import matplotlib.pyplot as plt
+
+################################################################################
+# Train a simple model to produce phonetic posteriorgrams (PPGs)
+################################################################################
+
+
+def main():
+
+    # training hyperparameters
+    lr = .001
+    epochs = 60
+    batch_size = 250
+    device = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+    # phoneme encoder hyperparameters
+    lstm_depth = 2
+    hidden_size = 128  # 512
+    win_length = 256
+    hop_length = 128
+    n_mels = 32
+    n_mfcc = 19
+    lookahead_frames = 0  # 1
+
+    # datasets and loaders
+    train_data = LibriSpeechDataset(
+        split='train-clean-100',
+        target='phoneme',
+        features=None,
+        hop_length=hop_length
+    )
+    val_data = LibriSpeechDataset(
+        split='test-clean',
+        target='phoneme',
+        features=None,
+        hop_length=hop_length
+    )
+    train_loader = DataLoader(
+        train_data,
+        batch_size=batch_size,
+        shuffle=True)
+    val_loader = DataLoader(
+        val_data,
+        batch_size=batch_size)
+
+    # initialize phoneme encoder
+    encoder = PPGEncoder(
+        win_length=win_length,
+        hop_length=hop_length,
+        win_func=torch.hann_window,
+        n_mels=n_mels,
+        n_mfcc=n_mfcc,
+        lstm_depth=lstm_depth,
+        hidden_size=hidden_size,
+    )
+
+    # initialize classification layer and wrap as single module
+    classifier = nn.Sequential(
+        encoder,
+        nn.Linear(hidden_size, LIBRISPEECH_NUM_PHONEMES)
+    ).to(device)
+
+    # log training progress
+    writer = Writer(
+        name=f"phoneme_lookahead_{lookahead_frames}",
+        use_tb=True,
+        log_iter=len(train_loader)
+    )
+
+    import builtins
+    parameter_count = builtins.sum([
+        p.shape.numel()
+        for p in classifier[0].parameters()
+        if p.requires_grad
+    ])
+
+    writer.log_info(f'Training PPG model with lookahead {lookahead_frames}'
+                    f' ({parameter_count} parameters)')
+
+    # initialize optimizer and loss function
+    optimizer = torch.optim.Adam(classifier.parameters(), lr=lr)
+    loss_fn = nn.CrossEntropyLoss()
+
+    iter_id = 0
+    min_val_loss = float('inf')
+
+    for epoch in range(epochs):
+
+        print(f'beginning epoch {epoch}')
+
+        classifier.train()
+        for batch in train_loader:
+
+            optimizer.zero_grad(set_to_none=True)
+
+            x, y = batch['x'].to(device), batch['y'].to(device)
+
+            preds = classifier(x)
+
+            # offset labels to incorporate lookahead
+            y = y[:, :-lookahead_frames if lookahead_frames else None]
+
+            # offset predictions correspondingly
+            preds = preds[:, lookahead_frames:]
+
+            # compute cross-entropy loss
+            loss = loss_fn(
+                preds.reshape(-1, LIBRISPEECH_NUM_PHONEMES), y.reshape(-1)
+            )
+
+            loss.backward()
+            optimizer.step()
+
+            writer.log_scalar(loss, tag="CrossEntropyLoss-Train", global_step=iter_id)
+            iter_id += 1
+
+        val_loss, val_acc, n = 0.0, 0.0, 0
+        classifier.eval()
+        with torch.no_grad():
+            for batch in val_loader:
+
+                x, y = batch['x'].to(device), batch['y'].to(device)
+
+                preds = classifier(x)
+
+                # offset labels to incorporate lookahead
+                y = y[:, :-lookahead_frames if lookahead_frames else None]
+
+                # offset predictions correspondingly
+                preds = preds[:, lookahead_frames:]
+
+                n += len(x)
+                val_loss += loss_fn(
+                    preds.reshape(-1, LIBRISPEECH_NUM_PHONEMES), y.reshape(-1)
+                ) * len(x)
+                val_acc += len(x) * (torch.argmax(preds, dim=2) == y).flatten().float().mean()
+
+        val_loss /= n
+        val_acc /= n
+        writer.log_scalar(val_loss, tag="CrossEntropyLoss-Val", global_step=iter_id)
+        writer.log_scalar(val_acc, tag="Accuracy-Val")
+
+        # save weights
+        if val_loss < min_val_loss:
+            min_val_loss = val_loss
+            print(f'new best val loss {val_loss}; saving weights')
+            writer.checkpoint(classifier[0].state_dict(), 'phoneme_classifier')
+
+    # generate confusion matrix
+    classifier.eval()
+
+    # compute accuracy on validation data
+    all_preds = []
+    all_true = []
+    with torch.no_grad():
+        for batch in val_loader:
+
+            x, y = batch['x'].to(device), batch['y'].to(device)
+
+            preds = classifier(x)
+
+            # offset labels to incorporate lookahead
+            y = y[:, :-lookahead_frames if lookahead_frames else None]
+
+            # offset predictions correspondingly
+            preds = preds[:, lookahead_frames:]
+
+            all_preds.append(preds.argmax(dim=2).reshape(-1))
+            all_true.append(y.reshape(-1))
+
+    # compile predictions and targets
+    all_preds = torch.cat(all_preds, dim=0).cpu().numpy()
+    all_true = torch.cat(all_true, dim=0).cpu().numpy()
+
+    reverse_dict = {v: k for (k, v) in LIBRISPEECH_PHONEME_DICT.items() if v != 0}
+    reverse_dict[0] = 'sil'
+
+    class_report = classification_report(all_true, all_preds)
+    writer.log_info(class_report)
+
+    cm = confusion_matrix(all_true, all_preds, labels=list(range(len(reverse_dict))))
+    df_cm = pd.DataFrame(cm, index=[i for i in sorted(list(reverse_dict.keys()))],
+                         columns=[i for i in sorted(list(reverse_dict.keys()))])
+    plt.figure(figsize=(40, 28))
+    sn.set(font_scale=1.0)  # for label size
+    sn.heatmap(df_cm, annot=True, annot_kws={"size": 35 / np.sqrt(len(cm))}, fmt='g')
+
+    plt.savefig("phoneme_cm.png", dpi=200)
+
+
+if __name__ == '__main__':
+    main()

voicebox/scripts/streamer/benchmark_streamer.py

@@ -0,0 +1,97 @@
+import torch
+import librosa
+import soundfile as sf
+
+from tqdm import tqdm
+from src.attacks.offline.perturbation.voicebox import projection
+from src.attacks.online import Streamer, VoiceBoxStreamer
+from src.models import ResNetSE34V2, SpeakerVerificationModel
+from src.constants import MODELS_DIR, TEST_DIR, PPG_PRETRAINED_PATH
+
+import warnings
+warnings.filterwarnings("ignore")
+
+torch.set_num_threads(1)
+
+device = 'cpu'
+
+lookahead = 5
+
+signal_length = 64_000
+chunk_size = 640
+
+test_audio = torch.Tensor(
+    librosa.load(TEST_DIR / 'data' / 'test.wav', sr=16_000, mono=True)[0]
+).unsqueeze(0).unsqueeze(0)
+
+tests = [
+    (512, 512, 512)
+]
+resnet_model = SpeakerVerificationModel(model=ResNetSE34V2())
+condition_vector = resnet_model(test_audio)
+for (bottleneck_hidden_size,
+     bottleneck_feedforward_size,
+     spec_encoder_hidden_size) in tests:
+    print(
+f"""
+====================================
+bottleneck_hidden_size: {bottleneck_hidden_size}
+bottleneck_feedforward_size: {bottleneck_feedforward_size}
+spec_encoder_hidden_size: {spec_encoder_hidden_size}
+"""
+    )
+
+    streamer = Streamer(
+        VoiceBoxStreamer(
+            win_length=256,
+            bottleneck_type='lstm',
+            bottleneck_skip=True,
+            bottleneck_depth=2,
+            bottleneck_lookahead_frames=5,
+            bottleneck_hidden_size=bottleneck_hidden_size,
+            bottleneck_feedforward_size=bottleneck_feedforward_size,
+        
+            conditioning_dim=512,
+
+            spec_encoder_mlp_depth=2,
+            spec_encoder_hidden_size=spec_encoder_hidden_size,
+            spec_encoder_lookahead_frames=0,
+            ppg_encoder_path=PPG_PRETRAINED_PATH,
+            
+            ppg_encoder_depth=2,
+            ppg_encoder_hidden_size=256,
+            projection_norm='inf',
+            control_eps=0.5,
+            n_bands=128
+        ),
+        device,
+        hop_length=128,
+        window_length=256,
+        win_type='hann',
+        lookahead_frames=lookahead,
+        recurrent=True
+    )
+    streamer.model.load_state_dict(torch.load(MODELS_DIR / 'voicebox' / 'voicebox_final.pt'))
+    streamer.condition_vector = condition_vector
+
+    output_chunks = []
+    for i in tqdm(range(0, signal_length, chunk_size)):
+        signal_chunk = test_audio[..., i:i+chunk_size]
+        out = streamer.feed(signal_chunk)
+        output_chunks.append(out)
+    output_chunks.append(streamer.flush())
+    output_audio = torch.cat(output_chunks, dim=-1)
+    output_embedding = resnet_model(output_audio)
+
+    print(
+f"""
+RTF: {streamer.real_time_factor}
+Embedding Distance: {resnet_model.distance_fn(output_embedding, condition_vector)}
+====================================
+"""
+        )
+    sf.write(
+        'output.wav',
+        output_audio.numpy().squeeze(),
+        16_000,
+    )

voicebox/scripts/streamer/enroll.py

@@ -0,0 +1,105 @@
+"""
+Pipeline for enrolling:
+1. Provide Recording
+2. Convert to 16 kHz
+3. Divide into recordings
+4. Get embeddings for each recording
+5. Find centroid
+6. Save conditioning as some value.
+"""
+import os
+import argbind
+import sounddevice as sd
+import soundfile
+import torch
+import numpy as np
+
+import sys
+
+sys.path.append('.')
+
+from src.constants import CONDITIONING_FILENAME, CONDITIONING_FOLDER
+from src.data import DataProperties
+from src.models import ResNetSE34V2
+
+
+MIN_WINDOWS = 10
+WINDOW_SIZE = 64_000
+BLOCK_SIZE = 256
+
+RECORDING_TEXT = """
+This script will record you speaking, and will create an embedding
+to be used for conditioning Voicebox. This will overwrite any previous
+embeddings. We recommend at least 10 seconds of non-stop voice recording.
+Press enter to begin recording. To stop recording, press ctrl-C.
+"""
+
+
+def get_streams(input_name: str, block_size: int) -> sd.InputStream:
+    """
+    Gets Input stream object
+    """
+    try:
+        input_name = int(input_name)
+    except ValueError:
+        pass
+    return (
+        sd.InputStream(device=input_name,
+                       samplerate=DataProperties.get('sample_rate'),
+                       channels=1,
+                       blocksize=block_size)
+    )
+
+
+def record_from_user(input_name: str) -> torch.Tensor:
+    input_stream = get_streams(input_name, BLOCK_SIZE)
+    input(RECORDING_TEXT)
+    input_stream.start()
+    all_frames = []
+    try:
+        print("Recording...")
+        while True:
+            frames, _ = input_stream.read(BLOCK_SIZE)
+            all_frames.append(frames)
+    except KeyboardInterrupt:
+        print("Stopped Recording.")
+        pass
+    all_frames = torch.Tensor(np.array(all_frames))
+    recording = all_frames.reshape(-1)
+    return recording
+
+
+def get_embedding(recording) -> torch.Tensor:
+    model = ResNetSE34V2(nOut=512, encoder_type='ASP')
+    recording = recording.view(1, -1)
+    embedding = model(recording)
+    return embedding
+
+
+def save(embedding, audio) -> None:
+    os.makedirs(CONDITIONING_FOLDER, exist_ok=True)
+    torch.save(embedding, CONDITIONING_FILENAME)
+    soundfile.write(
+        CONDITIONING_FOLDER / 'conditioning_audio.wav',
+        audio.detach().cpu(),
+        DataProperties.get('sample_rate')
+    )
+
+
+@argbind.bind(positional=True, without_prefix=True)
+def main(input: str = None):
+    """
+    Creating a conditioning vector for VoiceBox from your voice
+
+    :param input: Index or name of input audio interface. Defaults to current device
+    :type input: str, optional
+    """
+    recording = record_from_user(input)
+    embedding = get_embedding(recording)
+    save(embedding, recording)
+
+
+if __name__ == "__main__":
+    args = argbind.parse_args()
+    with argbind.scope(args):
+        main()

voicebox/scripts/streamer/stream.py

@@ -0,0 +1,135 @@
+import argbind
+import sounddevice as sd
+import numpy as np
+import yaml
+import torch
+import os
+from typing import Union
+
+import sys
+import warnings
+
+sys.path.append('.')
+warnings.filterwarnings('ignore', category=UserWarning)
+
+from src.data.dataproperties import DataProperties
+from src.attacks.online import Streamer, VoiceBoxStreamer
+from src.constants import MODELS_DIR, CONDITIONING_FILENAME
+
+
+def get_streams(input_name: str, output_name: str, block_size: int) -> tuple[sd.InputStream, sd.OutputStream]:
+    """
+    Gets Input and Output stream objects
+    """
+    try:
+        input_name = int(input_name)
+    except ValueError:
+        pass
+    try:
+        output_name = int(output_name)
+    except ValueError:
+        pass
+    return (
+        sd.InputStream(device=input_name,
+                       samplerate=DataProperties.get('sample_rate'),
+                       channels=1,
+                       blocksize=block_size),
+        sd.OutputStream(device=output_name,
+                        samplerate=DataProperties.get('sample_rate'),
+                        channels=1,
+                        blocksize=block_size)
+    )
+
+
+def get_model_streamer(device: str, conditioning_path: str) -> Streamer:
+    # TODO: Make a good way to query an attack type. For now, I'm going to hard code this.
+    model_dir = os.path.join(MODELS_DIR, 'voicebox')
+    checkpoint_path = os.path.join(model_dir, 'voicebox_final.pt')
+    config_path = os.path.join(model_dir, 'voicebox_final.yaml')
+
+    with open(config_path) as f:
+        config = yaml.safe_load(f)
+
+    state_dict = torch.load(checkpoint_path, map_location=device)
+    condition_tensor = torch.load(conditioning_path, map_location=device)
+    model = VoiceBoxStreamer(
+        **config
+    )
+    model.load_state_dict(state_dict)
+    model.condition_vector = condition_tensor.reshape(1, 1, -1)
+
+    streamer = Streamer(
+        model=model,
+        device=device,
+        lookahead_frames=config['bottleneck_lookahead_frames'],
+        recurrent=True
+    )
+    return streamer
+
+
+def to_model(x: np.ndarray, device: str) -> torch.Tensor:
+    return torch.Tensor(x).view(1, 1, -1).to(device)
+
+
+def from_model(x: torch.Tensor) -> np.ndarray:
+    return x.detach().cpu().view(-1, 1).numpy()
+
+
+@argbind.bind(without_prefix=True)
+def main(
+    input: str = None,
+    output: str = '',
+    device: str = 'cpu',
+    num_frames: int = 4,
+    pass_through: bool = False,
+    conditioning_path: str = CONDITIONING_FILENAME
+):
+    f"""
+    Uses a streaming implementation of an attack to perturb incoming audio
+
+    :param input: Index or name of input audio interface. Defaults to current device
+    :type input: str, optional
+    :param output: Index of name output audio interface. Defaults to 0
+    :type output: str, optional
+    :param device: Device to processing attack. Should be either 'cpu' or 'cuda:X'
+        Defaults to 'cpu'.
+    :type device: str, optional
+    :param pass_through: If True, the voicebox perturbation is not applied and the input will be
+        identical to the output. This is for demo purposes. The input and output audio will
+        remain at 16 kHz.
+    :type pass_through: bool, optional
+    :type device: str, optional
+    :param num_frames: Number of overlapping model frames to process at one iteration.
+        Defaults to 1
+    :type num_frames: int
+    :param conditioning_path: Path to conditioning tensor. Default: {CONDITIONING_FILENAME}
+    :type conditioning_path: str
+    """
+    streamer = get_model_streamer(device, conditioning_path)
+    input_stream, output_stream = get_streams(input, output, streamer.hop_length)
+    if streamer.win_type in ['hann', 'triangular']:
+        input_samples = (num_frames - 1) * streamer.hop_length + streamer.window_length
+    else:
+        input_samples = streamer.hop_length
+    print("Ready to process audio")
+    input_stream.start()
+    output_stream.start()
+    try:
+        while True:
+            frames, overflow = input_stream.read(input_samples)
+            if pass_through:
+                output_stream.write(frames)
+                continue
+            out = streamer.feed(to_model(frames, device))
+            out = from_model(out)
+            underflow = output_stream.write(out)
+    except KeyboardInterrupt:
+        print("Stopping")
+        input_stream.stop()
+        output_stream.stop()
+
+
+if __name__ == "__main__":
+    args = argbind.parse_args()
+    with argbind.scope(args):
+        main()

voicebox/setup.py

@@ -0,0 +1,20 @@
+from setuptools import setup
+
+with open('README.md') as file:
+    long_description = file.read()
+
+setup(
+    name='src',
+    description='Code for VoiceBox',
+    version='0.0.1',
+    author='',
+    author_email='',
+    url='',
+    install_requires=[],
+    packages=['src'],
+    long_description=long_description,
+    long_description_content_type='text.markdown',
+    keywords=[],
+    classifiers=['License :: OSI Approved :: MIT License'],
+    license='MIT'
+)

voicebox/src.egg-info/PKG-INFO

@@ -0,0 +1,148 @@
+Metadata-Version: 2.1
+Name: src
+Version: 0.0.1
+Summary: Code for VoiceBox
+Home-page: 
+Author: 
+Author-email: 
+License: MIT
+Classifier: License :: OSI Approved :: MIT License
+Description-Content-Type: text.markdown
+License-File: LICENSE
+
+<h1 align="center">VoiceBlock</h1>
+<h4 align="center"> Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models</h4>
+<div align="center">
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/???/???.ipynb)
+[![Demo](https://img.shields.io/badge/Web-Demo-blue)](https://master.d3hvhbnf7qxjtf.amplifyapp.com/)
+[![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](/LICENSE)
+
+</div>
+<p align="center"><img src="./figures/vb_color_logo.png" width="200"/></p>
+
+
+## Contents
+
+* <a href="#install">Installation</a>
+* <a href="#reproduce">Reproducing Results</a>
+* <a href="#streamer">Streaming Implementation</a>
+* <a href="#citation">Citation</a>
+
+<h2 id="install">Installation</h2>
+
+1. Clone the repository:
+
+        git clone https://github.com/voiceboxneurips/voicebox.git
+
+2. We recommend working from a clean environment, e.g. using `conda`:
+
+        conda create --name voicebox python=3.9
+        source activate voicebox 
+
+3. Install dependencies:
+
+        cd voicebox
+        pip install -r requirements.txt
+        pip install -e .
+
+4. Grant permissions:
+
+        chmod -R u+x scripts/
+
+<h2 id="reproduce">Reproducing Results</h2>
+
+To reproduce our results, first download the corresponding data. Note that to download the [VoxCeleb1 dataset](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1.html), you must register and obtain a username and password.
+
+| Task | Dataset (Size) | Command |
+|---|---|---|
+| Objective evaluation | VoxCeleb1  (39G) | `python scripts/downloads/download_voxceleb.py --subset=1 --username=<VGG_USERNAME> --password=<VGG_PASSWORD>` |
+| WER / supplemental evaluations | LibriSpeech `train-clean-360` (23G) | `./scripts/downloads/download_librispeech_eval.sh` |
+| Train attacks | LibriSpeech `train-clean-100` (11G) | `./scripts/downloads/download_librispeech_train.sh` |
+
+
+We provide scripts to reproduce our experiments and save results, including generated audio, to named and time-stamped subdirectories within `runs/`. To reproduce our objective evaluation experiments using pre-trained attacks, run:
+
+```
+python scripts/experiments/evaluate.py
+```
+
+To reproduce our training, run:
+
+```
+python scripts/experiments/train.py
+```
+
+<h2 id="streamer">Streaming Implementation</h2>
+
+As a proof of concept, we provide a streaming implementation of VoiceBox capable of modifying user audio in real-time. Here, we provide installation instructions for MacOS and Ubuntu 20.04.
+
+<h3 id="streamer-mac">MacOS</h3>
+
+See video below:
+
+<a href="https://youtu.be/LcNjO5E7F3E">
+<p align="center"><img src="./figures/demo_thumbnail.png" width="500"/></p>
+</a>
+
+<h3 id="streamer-ubuntu">Ubuntu 20.04</h3>
+
+
+1. Open a terminal and follow the [installation instructions](#install) above. Change directory to the root of this repository.
+
+2. Run the following command:
+   
+        pacmd load-module module-null-sink sink_name=voicebox sink_properties=device.description=voicebox
+
+   If you are using PipeWire instead of PulseAudio:
+        
+        pactl load-module module-null-sink media.class=Audio/Sink sink_name=voicebox sink_properties=device.description=voicebox
+        
+   PulseAudio is the default on Ubuntu. If you haven't changed your system defaults, you are probably using PulseAudio. This will add "voicebox" as an output device. Select it as the input to your chosen audio software.
+
+3. Find which audio device to read and write from. In your conda environment, run:
+  
+        python -m sounddevice
+
+   You will get output similar to this:
+  
+          0 HDA Intel HDMI: 0 (hw:0,3), ALSA (0 in, 8 out)
+          1 HDA Intel HDMI: 1 (hw:0,7), ALSA (0 in, 8 out)
+          2 HDA Intel HDMI: 2 (hw:0,8), ALSA (0 in, 8 out)
+          3 HDA Intel HDMI: 3 (hw:0,9), ALSA (0 in, 8 out)
+          4 HDA Intel HDMI: 4 (hw:0,10), ALSA (0 in, 8 out)
+          5 hdmi, ALSA (0 in, 8 out)
+          6 jack, ALSA (2 in, 2 out)
+          7 pipewire, ALSA (64 in, 64 out)
+          8 pulse, ALSA (32 in, 32 out)
+        * 9 default, ALSA (32 in, 32 out)
+  
+   In this example, we are going to route the audio through PipeWire (channel 7). This will be our INPUT_NUM and OUTPUT_NUM
+
+4. First, we need to create a conditioning embedding. To do this, run the enrollment script and follow its on-screen instructions:
+  
+        python scripts/streamer/enroll.py --input INPUT_NUM
+
+5. We can now use the streamer. Run:
+  
+        python scripts/stream.py --input INPUT_NUM --output OUTPUT_NUM
+
+6. Once the streamer is running, open `pavucontrol`.
+
+   a. In `pavucontrol`, go to the "Playback" tab and find "ALSA pug-in [python3.9]: ALSA Playback on". Set the output to "voicebox".
+
+   b. Then, go to "Recording" and find "ALSA pug-in [python3.9]: ALSA Playback from", and set the input to your desired microphone device.
+
+<h2 id="citation">Citation</h2>
+
+If you use this your academic research, please cite the following:
+
+```
+@inproceedings{authors2022voicelock,
+title={VoiceBlock: Privacy through Real-Time Adversarial Attacks with Audio-to-Audio Models},
+author={Patrick O'Reilly, Andreas Bugler, Keshav Bhandari, Max Morrison, Bryan Pardo},
+booktitle={Neural Information Processing Systems},
+month={November},
+year={2022}
+}
+```

voicebox/src.egg-info/SOURCES.txt

@@ -0,0 +1,9 @@
+LICENSE
+README.md
+setup.py
+src/__init__.py
+src/constants.py
+src.egg-info/PKG-INFO
+src.egg-info/SOURCES.txt
+src.egg-info/dependency_links.txt
+src.egg-info/top_level.txt

voicebox/src.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

voicebox/src.egg-info/top_level.txt

@@ -0,0 +1 @@
+src