flan-t5-large-v1

README.md

@@ -0,0 +1,75 @@
+---
+license: apache-2.0
+base_model: google/flan-t5-large
+tags:
+- generated_from_trainer
+metrics:
+- rouge
+model-index:
+- name: flan-t5-large-v1
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# flan-t5-large-v1
+
+This model is a fine-tuned version of [google/flan-t5-large](https://huggingface.co/google/flan-t5-large) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.1673
+- Rouge1: 74.1287
+- Rouge2: 66.4339
+- Rougel: 72.8596
+- Rougelsum: 73.8679
+- Gen Len: 16.3241
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 8
+- eval_batch_size: 16
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 200
+- num_epochs: 10
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
+|:-------------:|:-----:|:----:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
+| 12.4521       | 0.85  | 200  | 0.2565          | 72.2564 | 62.6872 | 70.741  | 72.0604   | 15.9467 |
+| 0.2538        | 1.7   | 400  | 0.1877          | 72.8835 | 64.0804 | 71.5277 | 72.642    | 16.3582 |
+| 0.1804        | 2.55  | 600  | 0.1715          | 73.307  | 64.5027 | 72.2345 | 73.098    | 16.1429 |
+| 0.1516        | 3.4   | 800  | 0.1675          | 73.9648 | 65.6244 | 72.8421 | 73.8516   | 16.2026 |
+| 0.1331        | 4.26  | 1000 | 0.1609          | 73.7382 | 65.6094 | 72.5124 | 73.5658   | 16.3198 |
+| 0.1205        | 5.11  | 1200 | 0.1656          | 74.2505 | 66.5083 | 73.1059 | 74.0956   | 16.3795 |
+| 0.1113        | 5.96  | 1400 | 0.1593          | 74.2997 | 66.2497 | 73.2158 | 74.1265   | 16.3326 |
+| 0.1031        | 6.81  | 1600 | 0.1643          | 74.2861 | 66.3972 | 73.1252 | 74.0796   | 16.2729 |
+| 0.0909        | 7.66  | 1800 | 0.1638          | 73.7071 | 65.61   | 72.4082 | 73.5071   | 16.3262 |
+| 0.0876        | 8.51  | 2000 | 0.1667          | 74.1477 | 66.0628 | 72.9115 | 73.9177   | 16.3198 |
+| 0.0911        | 9.36  | 2200 | 0.1673          | 74.1287 | 66.4339 | 72.8596 | 73.8679   | 16.3241 |
+
+
+### Framework versions
+
+- Transformers 4.32.1
+- Pytorch 2.0.1+cu117
+- Datasets 2.14.4
+- Tokenizers 0.13.3

config.json

@@ -0,0 +1,33 @@
+{
+  "_name_or_path": "google/flan-t5-large",
+  "architectures": [
+    "T5ForConditionalGeneration"
+  ],
+  "classifier_dropout": 0.0,
+  "d_ff": 2816,
+  "d_kv": 64,
+  "d_model": 1024,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "n_positions": 512,
+  "num_decoder_layers": 24,
+  "num_heads": 16,
+  "num_layers": 24,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.32.1",
+  "use_cache": true,
+  "vocab_size": 32128
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "decoder_start_token_id": 0,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.32.1"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae35c5d1d3179bef12461be552a484761e45c1bcc54c7f3e7afd4a4d53bbebb3
+size 3132793669

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb6eb0f00056553a850f2029f4baef5ad61c6b311398b6a11c5b628ef2a43d91
+size 4155