Create README.md

README.md

@@ -0,0 +1,40 @@
+---
+datasets:
+- kresnik/zeroth_korean
+- mozilla-foundation/common_voice_17_0
+- PolyAI/minds14
+metrics:
+- bleu
+- cer
+base_model:
+- microsoft/Phi-4-multimodal-instruct
+language:
+- ko
+license: mit
+tags:
+- korean
+- stt
+- custom_code
+- phi
+- phi-4-multimodal
+---
+
+# Phi-4-multimodal-finetune-ko-speech
+
+This is a fine-tuned model for Korean speech-to-text translation, from [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct) on the following datasets:
+
+- kresnik/zeroth_korean
+- mozilla-foundation/common_voice_17_0
+- PolyAI/minds14
+- Custom dataset on my own (Recorded Korean speech sentences and transcribed using Azure Speech-to-text API). The speech was a mix of fast and slow speech, with some modulation using [audiomentations](https://github.com/iver56/audiomentations).
+
+Total 35K samples. Each sample is a pair of Korean speech and its transcription. Dataset was sampled 16kHz.
+
+The model was trained on a single A100 80GB GPU for 1 epoch with a batch size of 16 using the `sample_finetune_speech.py` script from [microsoft/Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct)
+
+Note that this model is just a PoC/experimental purpose, and not intended to be used in production.
+Phi-4-multimodal model is strong in multimodal tasks, especially in speech-to-text and high potential in Korean language tasks. Thus if you are interested in Korean speech-to-text task, this model can be a good starting point.
+
+## Evaluation
+
+ASR (Automatic Speech Recognition) on zeroth-test set and Speech translation on fleurs ko <-> en speech translation result. Script is retrieved from [here](https://gist.github.com/seastar105/d1d8983b27611370528e3b194dcc5577#file-evaluate-py).