test: update python backend test coverage, remove some unused classes and functions (see README.md)

README.md

@@ -1,31 +1,56 @@
 
-# AI Pronunciation Trainer 
-This tool uses AI to evaluate your pronunciation so you can improve it and be understood more clearly. You can go straight test the tool at https://aipronunciationtr.com (please use the chrome browser for desktop and have some patience for it to "warm-up" :) ). 
+# AI Pronunciation Trainer
+
+This tool uses AI to evaluate your pronunciation so you can improve it and be understood more clearly. You can go straight test the tool at <https://aipronunciationtr.com> (please use the chrome browser for desktop and have some patience for it to "warm-up" :) ).
 
 ![](images/MainScreen.jpg)
 
-## Installation 
+## Installation
+
 To run the program locally, you need to install the requirements and run the main python file:
-```
+
+```bash
 pip install -r requirements.txt
 python webApp.py
 ```
-You'll also need ffmpeg, which you can download from here https://ffmpeg.org/download.html. On Windows, it may be needed to add the ffmpeg "bin" folder to your PATH environment variable. On Mac, you can also just run "brew install ffmpeg".
+
+You'll also need ffmpeg, which you can download from here <https://ffmpeg.org/download.html>. On Windows, it may be needed to add the ffmpeg "bin" folder to your PATH environment variable. On Mac, you can also just run "brew install ffmpeg".
 
 You should be able to run it locally without any major issues as long as you’re using a recent python 3.X version.  
 
+## Changes on [trincadev's](https://github.com/trincadev/) [repository](https://github.com/trincadev/ai-pronunciation-trainer)
+
+I upgraded the frontend (jquery@3.7.1, bootstrap@5.3.3) and backend (pytorch==1.13.1, numpy<2.0.0) libraries where possible: for example right now this project doesn't work with pytorch > 2.0.0, then we are locked with pytorch == 1.13.1.
+
+### Unused classes and functions now removed
+
+- `aip_trainer.lambdas.lambdaTTS.*`
+- `aip_trainer.models.models.getTTSModel()`
+- `aip_trainer.models.models.getTranslationModel()`
+- `aip_trainer.models.AllModels.NeuralTTS`
+- `aip_trainer.models.AllModels.NeuralTranslator`
+
+### TODO
+
+- add e2e tests with playwright
+- move from pytorch to onnxruntime
+- refactor frontend with something more modern (e.g. vuejs)
+- refactor css style with tailwindcss
+
 ## Online version
-For the people who don’t feel comfortable running code or just want to have a quick way to use the tool, I hosted an online version of it at https://aipronunciationtr.com. It should work well in desktop-chrome, any other browser is not officially supported, although most of the functionality should work fine. 
- 
-Please be aware that the usage is limited by day (I’m still not rich ;)). If, for some reason, you would like to avoid the daily usage limit, just enter in contact and we see what we can do. 
+
+For the people who don’t feel comfortable running code or just want to have a quick way to use the tool, I hosted an online version of it at <https://aipronunciationtr.com>. It should work well in desktop-chrome, any other browser is not officially supported, although most of the functionality should work fine.
+
+Please be aware that the usage is limited by day (I’m still not rich ;)). If, for some reason, you would like to avoid the daily usage limit, just enter in contact and we see what we can do.
 
 ## Motivation
 
-Often, when we want to improve our pronunciation, it is very difficult to self-assess how good we’re speaking. Asking a native, or language instructor, to constantly correct us is either impractical, due to monetary constrains, or annoying due to simply being too boring for this other person. Additionally, they may often say “it sounds good” after your 10th try to not discourage you, even though you may still have some mistakes in your pronunciation. 
+Often, when we want to improve our pronunciation, it is very difficult to self-assess how good we’re speaking. Asking a native, or language instructor, to constantly correct us is either impractical, due to monetary constrains, or annoying due to simply being too boring for this other person. Additionally, they may often say “it sounds good” after your 10th try to not discourage you, even though you may still have some mistakes in your pronunciation.
+
+The AI pronunciation trainer is a way to provide objective feedback on how well your pronunciation is in an automatic and scalable fashion, so the only limit to your improvement is your own dedication.
 
-The AI pronunciation trainer is a way to provide objective feedback on how well your pronunciation is in an automatic and scalable fashion, so the only limit to your improvement is your own dedication. 
+This project originated from a small program that I did to improve my own pronunciation.  When I finished it, I believed it could be a useful tool also for other people trying to be better understood, so I decided to make a simple, more user-friendly version of it.
 
-This project originated from a small program that I did to improve my own pronunciation.  When I finished it, I believed it could be a useful tool also for other people trying to be better understood, so I decided to make a simple, more user-friendly version of it. 
+## Disclaimer
 
-## Disclaimer 
 This is a simple project that I made in my free time with the goal to be useful to some people. It is not perfect, thus be aware that some small bugs may be present. In case you find something is not working, all feedback is welcome, and issues may be addressed depending on their severity.

aip_trainer/WordMetrics.py

@@ -3,35 +3,6 @@ import numpy as np
 from aip_trainer import app_logger
 
 
-# ref from https://gitlab.com/-/snippets/1948157
-# For some variants, look here https://en.wikibooks.org/wiki/Algorithm_Implementation/Strings/Levenshtein_distance#Python
-
-
-# Pure python
-def edit_distance_python2(a, b):
-    # This version is commutative, so as an optimization we force |a|>=|b|
-    if len(a) < len(b):
-        return edit_distance_python(b, a)
-    if len(b) == 0:  # Can deal with empty sequences faster
-        return len(a)
-    # Only two rows are really needed: the one currently filled in, and the previous
-    distances = [
-        [i for i in range(len(b) + 1)],
-        [0 for _ in range(len(b) + 1)]
-    ]
-    # We can prefill the first row:
-    costs = [0 for _ in range(3)]
-    for i, a_token in enumerate(a, start=1):
-        distances[1][0] += 1  # Deals with the first column.
-        for j, b_token in enumerate(b, start=1):
-            costs[0] = distances[1][j-1] + 1
-            costs[1] = distances[0][j] + 1
-            costs[2] = distances[0][j-1] + (0 if a_token == b_token else 1)
-            distances[1][j] = min(costs)
-        # Move to the next row:
-        distances[0][:] = distances[1][:]
-    return distances[1][len(b)]
-
 # https://stackabuse.com/levenshtein-distance-and-text-similarity-in-python/
 def edit_distance_python(seq1, seq2):
     size_x = len(seq1) + 1

aip_trainer/lambdas/lambdaGetSample.py

@@ -1,6 +1,5 @@
 import json
 import pickle
-import random
 from pathlib import Path
 
 import epitran
@@ -89,21 +88,20 @@ def getSentenceCategory(sentence) -> int:
             return category + 1
 
 
-if __name__ == "__main__":
-    import pandas as pd
-    with open(sample_folder / 'data_de_en_2.pickle', 'rb') as handle:
-        df = pickle.load(handle)
+def get_pickle2json_dataframe(
+        custom_pickle_filename_no_ext: Path | str = 'data_de_en_2',
+        custom_folder: Path = sample_folder
+    ):
+    custom_folder = Path(custom_folder)
+    with open(custom_folder / f'{custom_pickle_filename_no_ext}.pickle', 'rb') as handle:
+        df2 = pickle.load(handle)
         pass
-        df["de_category"] = df["de_sentence"].apply(getSentenceCategory)
+        df2["de_category"] = df2["de_sentence"].apply(getSentenceCategory)
         print("de_category added")
-        df["en_category"] = df["en_sentence"].apply(getSentenceCategory)
+        df2["en_category"] = df2["en_sentence"].apply(getSentenceCategory)
         print("en_category added")
-    df_json = df.to_json()
-    with open(sample_folder / 'data_de_en_with_categories.json', 'w') as dst:
+    df_json = df2.to_json()
+    with open(custom_folder / f'{custom_pickle_filename_no_ext}.json', 'w') as dst:
         dst.write(df_json)
         print("data_de_en_with_categories.json written")
-    with open(sample_folder / 'data_de_en_with_categories.json', 'r') as src:
-        jj = json.load(src)
-        print("jj:", jj)
-        df2 = pd.read_json(json.dumps(jj))
-        print(df2)
+

aip_trainer/lambdas/lambdaTTS.py

@@ -1,48 +0,0 @@
-
-import base64
-import json
-import tempfile
-
-import soundfile as sf
-
-from aip_trainer import app_logger
-from aip_trainer.models.models import getTTSModel
-from aip_trainer.models.AIModels import NeuralTTS
-
-
-sampling_rate = 16000
-model_de = getTTSModel('de')
-model_TTS_lambda = NeuralTTS(model_de, sampling_rate)
-
-
-def lambda_handler(event, context):
-
-    body = json.loads(event['body'])
-
-    text_string = body['value']
-
-    linear_factor = 0.2
-    audio = model_TTS_lambda.getAudioFromSentence(
-        text_string).detach().numpy()*linear_factor
-    with tempfile.TemporaryFile(prefix="temp_sound_", suffix=".wav") as f1:
-        app_logger.info(f"Saving temp audio to {f1.name}...")
-        # random_file_name = utilsFileIO.generateRandomString(20) + '.wav'
-        # sf.write('./'+random_file_name, audio, 16000)
-
-        sf.write(f1.name, audio, sampling_rate)
-        with open(f1.name, "rb") as f:
-            audio_byte_array = f.read()
-        # os.remove(random_file_name)
-        return {
-            'statusCode': 200,
-            'headers': {
-                'Access-Control-Allow-Headers': '*',
-                'Access-Control-Allow-Origin': 'http://127.0.0.1:3000/',
-                'Access-Control-Allow-Methods': 'OPTIONS,POST,GET'
-            },
-            'body': json.dumps(
-                {
-                    "wavBase64": str(base64.b64encode(audio_byte_array))[2:-1],
-                },
-            )
-        }

aip_trainer/models/AIModels.py

@@ -34,33 +34,3 @@ class NeuralASR(ModelInterfaces.IASRModel):
 
             self.audio_transcript, self.word_locations_in_samples = self.decoder(
                 nn_output[0, :, :].detach(), audio_length_in_samples, word_align=True)
-
-
-class NeuralTTS(ModelInterfaces.ITextToSpeechModel):
-    def __init__(self, model: torch.nn.Module, sampling_rate: int) -> None:
-        super().__init__()
-        self.model = model
-        self.sampling_rate = sampling_rate
-
-    def getAudioFromSentence(self, sentence: str) -> np.array:
-        with torch.inference_mode():
-            audio_transcript = self.model.apply_tts(texts=[sentence],
-                                                    sample_rate=self.sampling_rate)[0]
-
-        return audio_transcript
-
-
-class NeuralTranslator(ModelInterfaces.ITranslationModel):
-    def __init__(self, model: torch.nn.Module, tokenizer) -> None:
-        super().__init__()
-        self.model = model
-        self.tokenizer = tokenizer
-
-    def translateSentence(self, sentence: str) -> str:
-        """Get the transcripts of the process audio"""
-        tokenized_text = self.tokenizer(sentence, return_tensors='pt')
-        translation = self.model.generate(**tokenized_text)
-        translated_text = self.tokenizer.batch_decode(
-            translation, skip_special_tokens=True)[0]
-
-        return translated_text

aip_trainer/models/models.py

@@ -1,10 +1,12 @@
+from typing import Any
+
 import torch
 import torch.nn as nn
 
-import pickle
 
+# second returned type here is the custom class src.silero.utils.Decoder from snakers4/silero-models
+def getASRModel(language: str) -> tuple[nn.Module, Any]:
 
-def getASRModel(language: str) -> nn.Module:
 
     if language == 'de':
 
@@ -18,51 +20,7 @@ def getASRModel(language: str) -> nn.Module:
                                                model='silero_stt',
                                                language='en',
                                                device=torch.device('cpu'))
-    elif language == 'fr':
-        model, decoder, utils = torch.hub.load(repo_or_dir='snakers4/silero-models',
-                                               model='silero_stt',
-                                               language='fr',
-                                               device=torch.device('cpu'))
-
-    return (model, decoder)
-
-
-def getTTSModel(language: str) -> nn.Module:
-
-    if language == 'de':
-
-        speaker = 'thorsten_v2'  # 16 kHz
-        model, _ = torch.hub.load(repo_or_dir='snakers4/silero-models',
-                                  model='silero_tts',
-                                  language=language,
-                                  speaker=speaker)
-
-    elif language == 'en':
-        speaker = 'lj_16khz'  # 16 kHz
-        model = torch.hub.load(repo_or_dir='snakers4/silero-models',
-                               model='silero_tts',
-                               language=language,
-                               speaker=speaker)
-    else:
-        raise ValueError('Language not implemented')
-
-    return model
-
-
-def getTranslationModel(language: str) -> nn.Module:
-    from transformers import AutoTokenizer
-    from transformers import AutoModelForSeq2SeqLM
-    if language == 'de':
-        model = AutoModelForSeq2SeqLM.from_pretrained(
-            "Helsinki-NLP/opus-mt-de-en")
-        tokenizer = AutoTokenizer.from_pretrained(
-            "Helsinki-NLP/opus-mt-de-en")
-        # Cache models to avoid Hugging face processing
-        with open('translation_model_de.pickle', 'wb') as handle:
-            pickle.dump(model, handle)
-        with open('translation_tokenizer_de.pickle', 'wb') as handle:
-            pickle.dump(tokenizer, handle)
     else:
-        raise ValueError('Language not implemented')
+        raise NotImplementedError("currenty works only for 'de' and 'en' languages, not for '{}'.".format(language))
 
-    return model, tokenizer
+    return model, decoder

tests/test_dataset.py

@@ -2,7 +2,7 @@ import json
 import unittest
 
 from aip_trainer.lambdas import lambdaGetSample
-from tests import test_logger
+from tests import test_logger, TEST_ROOT_FOLDER
 
 
 def helper_category(category: int, threshold_min: int, threshold_max: int, n: int = 1000):
@@ -32,6 +32,18 @@ class TestDataset(unittest.TestCase):
     def test_hard_sentences(self):
         helper_category(3, 20, 10000)
 
+    def test_get_pickle2json_dataframe(self):
+        import os
+
+        custom_filename = 'test_data_de_en_2'
+        lambdaGetSample.get_pickle2json_dataframe(custom_filename, TEST_ROOT_FOLDER)
+        with open(TEST_ROOT_FOLDER / f'{custom_filename}.json', 'r') as src1:
+            with open(TEST_ROOT_FOLDER / f'{custom_filename}_expected.json', 'r') as src2:
+                json1 = json.load(src1)
+                json2 = json.load(src2)
+                assert json1 == json2
+        os.remove(TEST_ROOT_FOLDER / f'{custom_filename}.json')
+
 
 if __name__ == '__main__':
     unittest.main()

webApp.py

@@ -7,7 +7,6 @@ from flask_cors import CORS
 
 from aip_trainer.lambdas import lambdaGetSample
 from aip_trainer.lambdas import lambdaSpeechToScore
-from aip_trainer.lambdas import lambdaTTS
 
 
 app = Flask(__name__, template_folder="static")
@@ -22,40 +21,6 @@ def main():
     return render_template('main.html')
 
 
-@app.route(rootPath+'/getAllSamples')
-def getDataDeEnAll():
-    import pickle
-    from pathlib import Path
-    sample_folder = Path(PROJECT_ROOT_FOLDER / "aip_trainer" / "lambdas")
-    with open(sample_folder / 'data_de_en_2.pickle', 'rb') as handle:
-        df = pickle.load(handle)
-        j = df.to_json()
-        return Response(j, mimetype='application/json')
-
-
-@app.route(rootPath+'/getSampleSearch', methods=['POST'])
-def getDataDeEnSearch():
-    import pickle
-    from pathlib import Path
-    sample_folder = Path(PROJECT_ROOT_FOLDER / "aip_trainer" / "lambdas")
-    with open(sample_folder / 'data_de_en_2.pickle', 'rb') as handle:
-        event = request.get_json(force=True)
-        df = pickle.load(handle)
-        lang = event.get('language')
-        filter_key = event.get('search')
-        df_by_language = df[f"{lang}_sentence"]
-        filter_obj = df_by_language.str.contains(filter_key)
-        filtered = df_by_language[filter_obj]
-        j = filtered.to_json()
-        return Response(j, mimetype='application/json')
-
-
-@app.route(rootPath+'/getAudioFromText', methods=['POST'])
-def getAudioFromText():
-    event = {'body': json.dumps(request.get_json(force=True))}
-    return lambdaTTS.lambda_handler(event, [])
-
-
 @app.route(rootPath+'/getSample', methods=['POST'])
 def getNext():
     event = {'body':  json.dumps(request.get_json(force=True))}