Update README.md

Added basic details on finetuning parameters, ROUGE1 score etc.

README.md

@@ -5,33 +5,78 @@ base_model: google/flan-t5-base
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+This is a flan-t5-base model finetuned using QLoRA (PEFT)
+on dialogSum dataset : https://huggingface.co/datasets/knkarthick/dialogsum
 
 
 
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
 
+## Model Details
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
+### Training Details:
+
+This is just a basic fine tuned model using below training args and params
+
+lora_config = LoraConfig(
+    r=16, 
+    lora_alpha=32, 
+    target_modules=['q','k','v','o'],
+    lora_dropout=.05,
+    bias='none',
+    task_type=TaskType.SEQ_2_SEQ_LM #flan-t5
+)
+
+
+output_dir = f'/kaggle/working/qlora-peft-flant5-base-dialogue-summary-training-{str(int(time.time()))}'
+
+peft_training_args_4bit = TrainingArguments(
+    output_dir=output_dir,
+    auto_find_batch_size=True,
+    learning_rate=1e-3, # Higher learning rate than full fine-tuning.
+    num_train_epochs=200,
+    logging_steps=10,
+    max_steps=200    
+)
+    
+peft_trainer_4bit = Trainer(
+    model=peft_model_4bit,
+    args=peft_training_args_4bit,
+    train_dataset=tokenized_dataset_cleaned["train"],
+    eval_dataset=tokenized_dataset_cleaned['validation']
+)
+
+
+Recorded training loss as below:
+
+Step	Training Loss
+10	29.131100
+20	4.856900
+30	3.241400
+40	1.346500
+50	0.560900
+60	0.344000
+70	0.258600
+80	0.201600
+90	0.202900
+100	0.198700
+110	0.185000
+120	0.177200
+130	0.161400
+140	0.164200
+150	0.164300
+160	0.165800
+170	0.168700
+180	0.155100
+190	0.161200
+200	0.170300
+
+
+Rouge1 score for 100 test dataset(out of 1500) is : 
+ORIGINAL MODEL:
+{'rouge1': 0.2232663790087573, 'rouge2': 0.06084131871447254, 'rougeL': 0.1936115999187245, 'rougeLsum': 0.19319411133637282}
+PEFT MODEL:
+{'rouge1': 0.34502805897556865, 'rouge2': 0.11517693222074701, 'rougeL': 0.2800665095598698, 'rougeLsum': 0.27941257109947587}
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
 
 ## Uses