Hakulani/t5-end2end-questions-generation

Model: Hakulani/t5-end2end-questions-generation Ruuning on RTX3090

Description: A T5-based model for end-to-end question generation, trained on a large corpus of text data.

Intended Use: This model is intended for generating questions based on a given input text. It can be used for various natural language processing tasks, such as text summarization, translation, and information retrieval.

Input: The model takes a text input in the form of a string.

Output: The model generates a list of questions related to the input text.

Training Data: The model was trained on a large corpus of text data from various sources, including news articles, web pages, and academic papers.

we're going to learn to fine-tune the 🤗 t5 model to generate questions without providing answers and use Weight and Biases for measurements and logs.

Dataset 🛢️ As dataset we use SQuAD v1.1: Stanford Question Answering Dataset is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.

Requirements This is not an introduction to Hugging Face Transformer library, it's a hands-on on how to fine tune t5 for this specific task. If you're not familiar with Hugging Face, you can watch the HF Course on Transformer models (it's free) here 🏗️ This notebook is a work in progress, some elements (check todo at the end) will change. Sources 📚 Transformer-based End-to-End Question Generation's Paper Patil Suraj's work on question generation

Parameters: _name_or_path: t5-base architectures: T5ForConditionalGeneration d_ff: 3072 d_kv: 64 d_model: 768 decoder_start_token_id: 0 dense_act_fn: relu dropout_rate: 0.1 eos_token_id: 1 feed_forward_proj: relu initializer_factor: 1.0 is_encoder_decoder: true is_gated_act: false layer_norm_epsilon: 1e-06 model_type: t5 n_positions: 512 num_decoder_layers: 12 num_heads: 12 num_layers: 12 output_past: true pad_token_id: 0 relative_attention_max_distance: 128 relative_attention_num_buckets: 32 task_specific_params: {"summarization": {"early_stopping": true, "length_penalty": 2.0, "max_length": 200, "min_length": 30, "no_repeat_ngram_size": 3, "num_beams": 4, "prefix": "summarize: "}, "translation_en_to_de": {"early_stopping": true, "max_length": 300, "num_beams": 4, "prefix": "translate English to German: "}, "translation_en_to_fr": {"early_stopping": true, "max_length": 300, "num_beams": 4, "prefix": "translate English to French: "}, "translation_en_to_ro": {"early_stopping": true, "max_length": 300, "num_beams": 4, "prefix": "translate English to Romanian: "}} torch_dtype: float32 transformers_version: 4.24.0 use_cache: true vocab_size: 32101

Hyperparameters:

early_stopping: true length_penalty: 2.0 max_length: 200 min_length: 30 no_repeat_ngram_size: 3 num_beams: 4 prefix: "summarize: " for summarization, "translate English to German: " for translation_en_to_de, "translate English to French: " for translation_en_to_fr, "translate English to Romanian: " for translation_en_to_ro

text = "Forrest Gump is a 1994 American comedy-drama film directed by Robert Zemeckis and written by Eric Roth.
It is based on the 1986 novel of the same name by Winston Groom and stars Tom Hanks, Robin Wright, Gary Sinise,
Mykelti Williamson and Sally Field. The story depicts several decades in the life of Forrest Gump (Hanks),
a slow-witted but kind-hearted man from Alabama who witnesses and unwittingly influences several defining
historical events in the 20th century United States. The film differs substantially from the novel."

checkpoint = "t5-base"
model = T5ForConditionalGeneration.from_pretrained(checkpoint)
tokenizer = T5TokenizerFast.from_pretrained(checkpoint)
tokenizer.sep_token = '<sep>'
tokenizer.add_tokens(['<sep>'])
model.resize_token_embeddings(len(tokenizer))
# Check the sep_token_id to verify that it was added to the tokenizer
tokenizer.sep_token_id
from transformers import T5ForConditionalGeneration, T5TokenizerFast

hfmodel = T5ForConditionalGeneration.from_pretrained("Hakulani/t5-end2end-questions-generation")
def hf_run_model(input_string, **generator_args):
  generator_args = {
  "max_length": 256,
  "num_beams": 4,
  "length_penalty": 1.5,
  "no_repeat_ngram_size": 3,
  "early_stopping": True,
  }
  input_string = "generate questions: " + input_string + " </s>"
  input_ids = tokenizer.encode(input_string, return_tensors="pt")
  res = hfmodel.generate(input_ids, **generator_args)
  output = tokenizer.batch_decode(res, skip_special_tokens=True)
  output = [item.split("<sep>") for item in output]
  return output


text = "Forrest Gump is a 1994 American comedy-drama film directed by Robert Zemeckis and written by Eric Roth. \
It is based on the 1986 novel of the same name by Winston Groom and stars Tom Hanks, Robin Wright, Gary Sinise, \
Mykelti Williamson and Sally Field. The story depicts several decades in the life of Forrest Gump (Hanks), \
a slow-witted but kind-hearted man from Alabama who witnesses and unwittingly influences several defining \
historical events in the 20th century United States. The film differs substantially from the novel."

hf_run_model(text)

[['Who directed the 1994 film Forrest Gump?',
  ' Who wrote the film for the film?',
  ' What is the film based on?',
  ' Which actor stars in the movie?',
  ' In what year was the novel of the same name written?',
  '']]