End of training

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+predictions_common_voice_13_en_common_voice_13_en_test_wer33.93_hyp.trn.snt.utt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,120 @@
+---
+base_model: Lakoc/DeCRED_small_cv
+tags:
+- generated_from_trainer
+datasets:
+- common_voice_13_0
+metrics:
+- wer
+model-index:
+- name: DeCRED_small_cv_linear_mixing
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# DeCRED_small_cv_linear_mixing
+
+This model is a fine-tuned version of [Lakoc/DeCRED_small_cv](https://huggingface.co/Lakoc/DeCRED_small_cv) on the common_voice_13_0 dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.5358
+- Cer: 0.1334
+- Wer: 0.2616
+- Mer: 0.2546
+- Wil: 0.4037
+- Wip: 0.5963
+- Hits: 34174
+- Substitutions: 8492
+- Deletions: 1958
+- Insertions: 1223
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.0005
+- train_batch_size: 128
+- eval_batch_size: 128
+- seed: 42
+- distributed_type: multi-GPU
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 512
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 50.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Cer    | Wer    | Mer    | Wil    | Wip    | Hits  | Substitutions | Deletions | Insertions |
+|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:------:|:------:|:-----:|:-------------:|:---------:|:----------:|
+| 3.0556        | 0.98  | 22   | 3.0178          | 1.2906 | 1.4035 | 0.9218 | 0.9902 | 0.0098 | 5310  | 35886         | 3428      | 23314      |
+| 2.9712        | 2.0   | 45   | 2.9272          | 1.0482 | 1.2086 | 0.8876 | 0.9816 | 0.0184 | 6832  | 33809         | 3983      | 16140      |
+| 2.8778        | 2.98  | 67   | 2.8449          | 0.8931 | 1.0776 | 0.8519 | 0.9699 | 0.0301 | 8362  | 31916         | 4346      | 11823      |
+| 2.8329        | 4.0   | 90   | 2.7632          | 0.7772 | 0.9734 | 0.8097 | 0.9523 | 0.0477 | 10207 | 29712         | 4705      | 9021       |
+| 2.7126        | 4.98  | 112  | 2.6890          | 0.6915 | 0.9029 | 0.7722 | 0.9331 | 0.0669 | 11886 | 27857         | 4881      | 7553       |
+| 2.7523        | 6.0   | 135  | 2.6153          | 0.5987 | 0.8231 | 0.7289 | 0.9078 | 0.0922 | 13658 | 25936         | 5030      | 5764       |
+| 2.5892        | 6.98  | 157  | 2.5484          | 0.5370 | 0.7637 | 0.6887 | 0.8801 | 0.1199 | 15403 | 24075         | 5146      | 4857       |
+| 2.5555        | 8.0   | 180  | 2.4819          | 0.4797 | 0.7039 | 0.6449 | 0.8463 | 0.1537 | 17294 | 22237         | 5093      | 4083       |
+| 2.4556        | 8.98  | 202  | 2.4215          | 0.4348 | 0.6531 | 0.6033 | 0.8103 | 0.1897 | 19160 | 20536         | 4928      | 3678       |
+| 2.4722        | 10.0  | 225  | 2.3615          | 0.3908 | 0.5995 | 0.5587 | 0.7687 | 0.2313 | 21129 | 18865         | 4630      | 3255       |
+| 2.3578        | 10.98 | 247  | 2.3072          | 0.3534 | 0.5561 | 0.5215 | 0.7310 | 0.2690 | 22766 | 17449         | 4409      | 2958       |
+| 2.317         | 12.0  | 270  | 2.2533          | 0.3181 | 0.5166 | 0.4877 | 0.6952 | 0.3048 | 24219 | 16266         | 4139      | 2647       |
+| 2.2757        | 12.98 | 292  | 2.2044          | 0.2935 | 0.4833 | 0.4587 | 0.6633 | 0.3367 | 25452 | 15275         | 3897      | 2393       |
+| 2.2332        | 14.0  | 315  | 2.1560          | 0.2705 | 0.4522 | 0.4312 | 0.6319 | 0.3681 | 26614 | 14339         | 3671      | 2169       |
+| 2.1887        | 14.98 | 337  | 2.1122          | 0.2526 | 0.4289 | 0.4106 | 0.6072 | 0.3928 | 27479 | 13605         | 3540      | 1996       |
+| 2.1393        | 16.0  | 360  | 2.0689          | 0.2352 | 0.4049 | 0.3888 | 0.5811 | 0.4189 | 28408 | 12909         | 3307      | 1854       |
+| 2.0646        | 16.98 | 382  | 2.0297          | 0.2213 | 0.3855 | 0.3712 | 0.5598 | 0.4402 | 29141 | 12367         | 3116      | 1719       |
+| 2.0398        | 18.0  | 405  | 1.9911          | 0.2090 | 0.3687 | 0.3557 | 0.5407 | 0.4593 | 29799 | 11895         | 2930      | 1629       |
+| 2.0011        | 18.98 | 427  | 1.9562          | 0.1994 | 0.3552 | 0.3432 | 0.5243 | 0.4757 | 30335 | 11455         | 2834      | 1561       |
+| 2.0078        | 20.0  | 450  | 1.9219          | 0.1911 | 0.3435 | 0.3325 | 0.5108 | 0.4892 | 30776 | 11131         | 2717      | 1482       |
+| 1.9608        | 20.98 | 472  | 1.8910          | 0.1843 | 0.3340 | 0.3236 | 0.4992 | 0.5008 | 31153 | 10840         | 2631      | 1435       |
+| 1.9634        | 22.0  | 495  | 1.8605          | 0.1782 | 0.3258 | 0.3158 | 0.4888 | 0.5112 | 31498 | 10582         | 2544      | 1413       |
+| 1.8888        | 22.98 | 517  | 1.8332          | 0.1730 | 0.3181 | 0.3086 | 0.4790 | 0.5210 | 31806 | 10335         | 2483      | 1375       |
+| 1.9019        | 24.0  | 540  | 1.8064          | 0.1688 | 0.3118 | 0.3026 | 0.4711 | 0.5289 | 32063 | 10141         | 2420      | 1353       |
+| 1.8107        | 24.98 | 562  | 1.7823          | 0.1660 | 0.3077 | 0.2987 | 0.4658 | 0.5342 | 32237 | 10016         | 2371      | 1342       |
+| 1.8168        | 26.0  | 585  | 1.7588          | 0.1639 | 0.3029 | 0.2942 | 0.4598 | 0.5402 | 32427 | 9875          | 2322      | 1318       |
+| 1.7804        | 26.98 | 607  | 1.7377          | 0.1612 | 0.2988 | 0.2903 | 0.4543 | 0.5457 | 32593 | 9734          | 2297      | 1301       |
+| 1.8311        | 28.0  | 630  | 1.7171          | 0.1585 | 0.2954 | 0.2871 | 0.4499 | 0.5501 | 32726 | 9624          | 2274      | 1283       |
+| 1.7236        | 28.98 | 652  | 1.6988          | 0.1564 | 0.2928 | 0.2846 | 0.4466 | 0.5534 | 32837 | 9549          | 2238      | 1278       |
+| 1.741         | 30.0  | 675  | 1.6809          | 0.1548 | 0.2902 | 0.2822 | 0.4433 | 0.5567 | 32947 | 9474          | 2203      | 1274       |
+| 1.7035        | 30.98 | 697  | 1.6650          | 0.1523 | 0.2880 | 0.2801 | 0.4403 | 0.5597 | 33036 | 9398          | 2190      | 1263       |
+| 1.7518        | 32.0  | 720  | 1.6496          | 0.1504 | 0.2856 | 0.2778 | 0.4371 | 0.5629 | 33135 | 9320          | 2169      | 1255       |
+| 1.6684        | 32.98 | 742  | 1.6360          | 0.1491 | 0.2837 | 0.2760 | 0.4345 | 0.5655 | 33214 | 9255          | 2155      | 1250       |
+| 1.7251        | 34.0  | 765  | 1.6228          | 0.1474 | 0.2810 | 0.2734 | 0.4311 | 0.5689 | 33320 | 9175          | 2129      | 1236       |
+| 1.6414        | 34.98 | 787  | 1.6112          | 0.1457 | 0.2788 | 0.2713 | 0.4280 | 0.5720 | 33416 | 9099          | 2109      | 1232       |
+| 1.6707        | 36.0  | 810  | 1.6001          | 0.1444 | 0.2770 | 0.2696 | 0.4255 | 0.5745 | 33490 | 9034          | 2100      | 1229       |
+| 1.6509        | 36.98 | 832  | 1.5904          | 0.1428 | 0.2749 | 0.2675 | 0.4225 | 0.5775 | 33588 | 8959          | 2077      | 1233       |
+| 1.654         | 38.0  | 855  | 1.5812          | 0.1416 | 0.2732 | 0.2658 | 0.4199 | 0.5801 | 33670 | 8889          | 2065      | 1237       |
+| 1.6305        | 38.98 | 877  | 1.5732          | 0.1403 | 0.2718 | 0.2645 | 0.4180 | 0.5820 | 33730 | 8841          | 2053      | 1235       |
+| 1.6381        | 40.0  | 900  | 1.5658          | 0.1391 | 0.2703 | 0.2630 | 0.4158 | 0.5842 | 33811 | 8790          | 2023      | 1250       |
+| 1.6393        | 40.98 | 922  | 1.5595          | 0.1382 | 0.2689 | 0.2616 | 0.4138 | 0.5862 | 33870 | 8741          | 2013      | 1246       |
+| 1.6111        | 42.0  | 945  | 1.5537          | 0.1372 | 0.2675 | 0.2602 | 0.4117 | 0.5883 | 33937 | 8685          | 2002      | 1250       |
+| 1.6041        | 42.98 | 967  | 1.5490          | 0.1362 | 0.2659 | 0.2587 | 0.4096 | 0.5904 | 34000 | 8636          | 1988      | 1240       |
+| 1.6301        | 44.0  | 990  | 1.5448          | 0.1353 | 0.2647 | 0.2575 | 0.4079 | 0.5921 | 34052 | 8593          | 1979      | 1239       |
+| 1.5736        | 44.98 | 1012 | 1.5416          | 0.1347 | 0.2638 | 0.2567 | 0.4066 | 0.5934 | 34087 | 8561          | 1976      | 1235       |
+| 1.6209        | 46.0  | 1035 | 1.5389          | 0.1341 | 0.2626 | 0.2556 | 0.4050 | 0.5950 | 34135 | 8522          | 1967      | 1230       |
+| 1.5892        | 46.98 | 1057 | 1.5372          | 0.1337 | 0.2620 | 0.2550 | 0.4042 | 0.5958 | 34156 | 8502          | 1966      | 1225       |
+| 1.6066        | 48.0  | 1080 | 1.5361          | 0.1335 | 0.2617 | 0.2548 | 0.4039 | 0.5961 | 34167 | 8496          | 1961      | 1223       |
+| 1.6445        | 48.89 | 1100 | 1.5358          | 0.1334 | 0.2616 | 0.2546 | 0.4037 | 0.5963 | 34174 | 8492          | 1958      | 1223       |
+
+
+### Framework versions
+
+- Transformers 4.40.0.dev0
+- Pytorch 2.2.0+rocm5.6
+- Datasets 2.18.0
+- Tokenizers 0.15.2

config.json

@@ -0,0 +1,276 @@
+{
+  "_name_or_path": "Lakoc/DeCRED_small_cv",
+  "architectures": [
+    "JointCTCAttentionEncoderDecoder"
+  ],
+  "bos_token_id": 0,
+  "ctc_weight": 0.3,
+  "decoder": {
+    "_name_or_path": "Lakoc/gpt2_256h_6l_add_head3_04",
+    "activation_function": "gelu_new",
+    "add_cross_attention": true,
+    "architectures": null,
+    "attn_pdrop": 0.1,
+    "average_logits": false,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 0,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "early_stopping": false,
+    "embd_pdrop": 0.1,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 1,
+    "exponential_decay_length_penalty": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "head_locations": [
+      3
+    ],
+    "head_weights": [
+      0.6,
+      0.4
+    ],
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "is_decoder": true,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_epsilon": 1e-05,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "gpt2-multi-head",
+    "n_embd": 256,
+    "n_head": 4,
+    "n_inner": 2048,
+    "n_layer": 6,
+    "n_positions": 1024,
+    "no_repeat_ngram_size": 0,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "pos_emb_fixed": true,
+    "prefix": null,
+    "problem_type": null,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "reorder_and_upcast_attn": false,
+    "repetition_penalty": 1.0,
+    "resid_pdrop": 0.1,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "scale_attn_by_inverse_layer_idx": false,
+    "scale_attn_weights": true,
+    "sep_token_id": null,
+    "summary_activation": null,
+    "summary_first_dropout": 0.1,
+    "summary_proj_to_labels": true,
+    "summary_type": "cls_index",
+    "summary_use_proj": true,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_additional_weights": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": false,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_cache": true,
+    "vocab_size": 1000
+  },
+  "decoder_start_token_id": 0,
+  "encoder": {
+    "_name_or_path": "Lakoc/fisher_ebranchformer_enc_12_layers_fixed",
+    "activation_dropout": 0.1,
+    "adapter_attn_dim": null,
+    "adapter_kernel_size": 3,
+    "adapter_stride": 2,
+    "add_adapter": false,
+    "add_cross_attention": false,
+    "apply_spec_augment": false,
+    "architectures": null,
+    "attention_dropout": 0.1,
+    "bad_words_ids": null,
+    "begin_suppress_tokens": null,
+    "bos_token_id": 1,
+    "chunk_size_feed_forward": 0,
+    "classifier_proj_size": 256,
+    "codevector_dim": 256,
+    "conformer_conv_dropout": 0.1,
+    "contrastive_logits_temperature": 0.1,
+    "conv_bias": false,
+    "conv_depthwise_kernel_size": 31,
+    "conv_dim": [
+      256,
+      256
+    ],
+    "conv_kernel": [
+      3,
+      3
+    ],
+    "conv_padding": [
+      1,
+      1
+    ],
+    "conv_stride": [
+      2,
+      2
+    ],
+    "cross_attention_hidden_size": null,
+    "csgu_activation": "identity",
+    "csgu_conv_dropout": 0.1,
+    "csgu_kernel_size": 31,
+    "csgu_use_linear_after_conv": false,
+    "ctc_loss_reduction": "mean",
+    "ctc_zero_infinity": true,
+    "decoder_start_token_id": null,
+    "diversity_loss_weight": 0.1,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "do_stable_layer_norm": false,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 2,
+    "exponential_decay_length_penalty": null,
+    "fe_position_embeddings": false,
+    "feat_extract_activation": "gelu",
+    "feat_extract_norm": "group",
+    "feat_proj_dropout": 0.0,
+    "feat_quantizer_dropout": 0.0,
+    "final_dropout": 0.1,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "gelu",
+    "hidden_dropout": 0.1,
+    "hidden_size": 256,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 1024,
+    "is_causal": false,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-05,
+    "layerdrop": 0.0,
+    "length_penalty": 1.0,
+    "mask_feature_length": 10,
+    "mask_feature_min_masks": 0,
+    "mask_feature_prob": 0.0,
+    "mask_time_length": 10,
+    "mask_time_min_masks": 2,
+    "mask_time_prob": 0.05,
+    "max_length": 20,
+    "max_source_positions": 1024,
+    "merge_conv_kernel": 31,
+    "min_length": 0,
+    "model_type": "wav2vec2-ebranchformer",
+    "no_repeat_ngram_size": 0,
+    "num_adapter_layers": 3,
+    "num_attention_heads": 4,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_codevector_groups": 2,
+    "num_codevectors_per_group": 320,
+    "num_conv_pos_embedding_groups": 16,
+    "num_conv_pos_embeddings": 128,
+    "num_fbanks": 80,
+    "num_feat_extract_layers": 2,
+    "num_hidden_layers": 12,
+    "num_mel_bins": 80,
+    "num_negatives": 100,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_size": 256,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 3,
+    "position_embeddings_type": "relative",
+    "prefix": null,
+    "problem_type": null,
+    "proj_codevector_dim": 256,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "rotary_embedding_base": 10000,
+    "sep_token_id": null,
+    "suppress_tokens": null,
+    "task_specific_params": null,
+    "tdnn_dilation": [
+      1,
+      2,
+      3,
+      1,
+      1
+    ],
+    "tdnn_dim": [
+      512,
+      512,
+      512,
+      512,
+      1500
+    ],
+    "tdnn_kernel": [
+      5,
+      3,
+      3,
+      1,
+      1
+    ],
+    "temperature": 1.0,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "typical_p": 1.0,
+    "use_bfloat16": false,
+    "use_fbanks": true,
+    "use_macaron_ff": true,
+    "use_weighted_layer_sum": false,
+    "vocab_size": 1000,
+    "xvector_output_dim": 512
+  },
+  "eos_token_id": 1,
+  "is_encoder_decoder": true,
+  "lsm_factor": 0.1,
+  "mask_token_id": 4,
+  "model_type": "joint_aed_ctc_speech-encoder-decoder",
+  "pad_token_id": 3,
+  "shared_lm_head": false,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.40.0.dev0"
+}

generation_config.json

@@ -0,0 +1,15 @@
+{
+  "apply_eos_space_trick": false,
+  "bos_token_id": 0,
+  "ctc_margin": 0,
+  "ctc_weight": 0.0,
+  "decoder_start_token_id": 0,
+  "eos_space_trick_weight": 0.0,
+  "eos_token_id": 1,
+  "lm_model": null,
+  "lm_weight": 0.0,
+  "max_length": 512,
+  "pad_token_id": 3,
+  "space_token_id": -1,
+  "transformers_version": "4.40.0.dev0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59ceeff8442b84c6d63e13954aa0fa700b69f5104d0e434103083e623e55c5c0
+size 144251296

predictions_common_voice_13_en_common_voice_13_en_test_wer33.93_hyp.trn.snt.utt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94e2218a63c4e1f25573dab6a6d476bf41a24d255721bb6ec93b0cbccd06785a
+size 11523797

predictions_common_voice_13_en_common_voice_13_en_test_wer33.93_hyp.trn.sys

@@ -0,0 +1,18 @@
+
+
+
+                     SYSTEM SUMMARY PERCENTAGES by SPEAKER                      
+
+,---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------.
+|/scratch/project_465000836/ipoloka/huggingface_asr/experiments/decred/commonvoice/DeCRED_small_cv_linear_mixing/predictions_common_voice_13_en_common_voice_13_en_test_wer33.93_hyp.trn|
+|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+|       SPKR          |        # Snt                # Wrd         |       Corr                 Sub                Del                 Ins                 Err              S.Err        |
+|---------------------+-------------------------------------------+---------------------------------------------------------------------------------------------------------------------|
+|       utt           |       16365                144023         |       70.1                23.1                6.8                 4.0                33.9               84.7        |
+|=======================================================================================================================================================================================|
+|       Sum/Avg       |       16365                144023         |       70.1                23.1                6.8                 4.0                33.9               84.7        |
+|=======================================================================================================================================================================================|
+|        Mean         |      16365.0              144023.0        |       70.1                23.1                6.8                 4.0                33.9               84.7        |
+|        S.D.         |         0.0                  0.0          |        0.0                 0.0                0.0                 0.0                 0.0                0.0        |
+|       Median        |      16365.0              144023.0        |       70.1                23.1                6.8                 4.0                33.9               84.7        |
+`---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------'

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ffe3693f7af5b4ec582655ec02ad41dfe4f75810491dfd0ff82c2c8e47aadc63
+size 5688