VictorSanh commited on
Commit
87af389
1 Parent(s): ad8b3e5

Update pronunciation + pre-filled examples

Browse files
Files changed (1) hide show
  1. README.md +50 -9
README.md CHANGED
@@ -6,21 +6,43 @@ license: apache-2.0
6
  widget:
7
  - text: "A is the son's of B's uncle. What is the family relationship between A and B?"
8
  - text: "Reorder the words in this sentence: justin and name bieber years is my am I 27 old."
9
- - text: "It's rainy today but it will stop in a few hours, when should I go for my run?"
10
- - text: "How many hydrogen atoms are in a water molecule?"
11
  - text: "Task: copy but say the opposite.\n
12
  PSG won its match against Barca."
13
  - text: "Is this review positive or negative? Review: Best cast iron skillet you will every buy."
14
- - text: "Question A:How is air traffic controlled?
15
  \nQuestion B: How do you become an air traffic controller?\nPick one: these questions are duplicates or not duplicates."
16
  - text: "Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He chose her because she had foreign affairs experience as a former First Lady.
17
  \nIn the previous sentence, decide who 'her' is referring to."
18
  - text: "Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.\n
19
  Select the category for the above sentence from: mobile, website, billing, account access."
20
- - text: "I don't have the proper tool to whisk a bowl of eggs.
21
- \nWhat should I use instead? Choose between a knife, a pen and a pair of chopsticks."
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
  ---
23
 
 
 
24
  # Model Description
25
 
26
  T0* is a series of encoder-decoder models trained on a large set of different tasks specified in natural language prompts. We convert numerous English supervised datasets into prompts, each with multiple templates using varying formulations. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. To obtain T0*, we fine-tune a pretrained language model on this multitask mixture covering many different NLP tasks.
@@ -29,9 +51,28 @@ T0* is a series of encoder-decoder models trained on a large set of different ta
29
 
30
  You can use the models to perform inference on tasks by specifying your query in natural language, and the models will generate a prediction. For instance, you can ask *"Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy"*, and the model will hopefully generate *"Positive"*.
31
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
  # How to use
33
 
34
- We make available the models presented in our [paper](https://arxiv.org/abs/2110.08207) along with the ablation models. We recommend using the [T0pp](https://huggingface.co/bigscience/T0pp) (pronounce "T zero plus plus") checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.
35
 
36
  |Model|Number of parameters|
37
  |-|-|
@@ -58,7 +99,7 @@ If you want to use another checkpoint, please replace the path in `AutoTokenizer
58
 
59
  # Training procedure
60
 
61
- T0* models are based on [T5](https://huggingface.co/google/t5-v1_1-large), a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on [C4](https://huggingface.co/datasets/c4). We use the publicly available [language model-adapated T5 checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) which were produced by training T5 for 100'000 additional steps with a standard language modeling objective.
62
 
63
  At a high level, the input text is fed to the encoder and the target text is produced by the decoder. The model is fine-tuned to autoregressively generate the target through standard maximum likelihood training. It is never trained to generate the input. We detail our training data in the next section.
64
 
@@ -119,7 +160,7 @@ We also evaluate T0, T0p and T0pp on the a subset of the [BIG-bench benchmark](h
119
 
120
  # Limitations
121
 
122
- - The models of the T0* series are quite large (3B or 11B parameters). Loading them and performing inference requires non-trivial computational ressources. When using multiple GPUs, it is possible to use [.parallelize()](https://huggingface.co/transformers/parallelism.html).
123
  - We have observed that different prompts can lead to varying performances. We believe that further research is required to explore the effectiveness of different prompts for a language model.
124
  - Due to design choices in the tokenization, the models are unable to perform inference for tasks involving code or non English text.
125
 
@@ -256,7 +297,7 @@ To measure the extent to which our model reproduces gender biases, we evaluate o
256
 
257
  ```bibtex
258
  @misc{sanh2021multitask,
259
- title={Multitask Prompted Training Enables Zero-Shot Task Generalization},
260
  author={Victor Sanh and Albert Webson and Colin Raffel and Stephen H. Bach and Lintang Sutawika and Zaid Alyafeai and Antoine Chaffin and Arnaud Stiegler and Teven Le Scao and Arun Raja and Manan Dey and M Saiful Bari and Canwen Xu and Urmish Thakker and Shanya Sharma Sharma and Eliza Szczechla and Taewoon Kim and Gunjan Chhablani and Nihal Nayak and Debajyoti Datta and Jonathan Chang and Mike Tian-Jian Jiang and Han Wang and Matteo Manica and Sheng Shen and Zheng Xin Yong and Harshit Pandey and Rachel Bawden and Thomas Wang and Trishala Neeraj and Jos Rozen and Abheesht Sharma and Andrea Santilli and Thibault Fevry and Jason Alan Fries and Ryan Teehan and Stella Biderman and Leo Gao and Tali Bers and Thomas Wolf and Alexander M. Rush},
261
  year={2021},
262
  eprint={2110.08207},
 
6
  widget:
7
  - text: "A is the son's of B's uncle. What is the family relationship between A and B?"
8
  - text: "Reorder the words in this sentence: justin and name bieber years is my am I 27 old."
 
 
9
  - text: "Task: copy but say the opposite.\n
10
  PSG won its match against Barca."
11
  - text: "Is this review positive or negative? Review: Best cast iron skillet you will every buy."
12
+ - text: "Question A: How is air traffic controlled?
13
  \nQuestion B: How do you become an air traffic controller?\nPick one: these questions are duplicates or not duplicates."
14
  - text: "Barack Obama nominated Hilary Clinton as his secretary of state on Monday. He chose her because she had foreign affairs experience as a former First Lady.
15
  \nIn the previous sentence, decide who 'her' is referring to."
16
  - text: "Last week I upgraded my iOS version and ever since then my phone has been overheating whenever I use your app.\n
17
  Select the category for the above sentence from: mobile, website, billing, account access."
18
+ - text: "Sentence 1: Gyorgy Heizler, head of the local disaster unit, said the coach was carrying 38 passengers.\n
19
+ Sentence 2: The head of the local disaster unit, Gyorgy Heizler, said the bus was full except for 38 empty seats.\n\n
20
+ Do sentences 1 and 2 have the same meaning?"
21
+ - text: "Here's the beginning of an article, choose a tag that best describes the topic of the article: business, cinema, politics, health, travel, sports.\n\n
22
+ The best and worst fo 007 as 'No time to die' marks Daniel Craig's exit.\n
23
+ (CNN) Some 007 math: 60 years, 25 movies (with a small asterisk) and six James Bonds. For a Cold War creation, Ian Fleming's suave spy has certainly gotten around, but despite different guises in the tuxedo and occasional scuba gear, when it comes to Bond ratings, there really shouldn't be much argument about who wore it best."
24
+ - text: "Max: Know any good websites to buy clothes from?\n
25
+ Payton: Sure :) LINK 1, LINK 2, LINK 3\n
26
+ Max: That's a lot of them!\n
27
+ Payton: Yeah, but they have different things so I usually buy things from 2 or 3 of them.\n
28
+ Max: I'll check them out. Thanks.\n\n
29
+ Who or what are Payton and Max referring to when they say 'them'?"
30
+ - text: "Is the word 'table' used in the same meaning in the two previous sentences?\n\n
31
+ Sentence A: you can leave the books on the table over there.\n
32
+ Sentence B: the tables in this book are very hard to read."
33
+ - text: "On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book.\n
34
+ The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.\n\n
35
+ Which book is the leftmost book?"
36
+ - text: "The word 'binne' means any animal that is furry and has four legs, and the word 'bam' means a simple sort of dwelling.\n\n
37
+ Which of the following best characterizes binne bams?\n
38
+ - Sentence 1: Binne bams are for pets.\n
39
+ - Sentence 2: Binne bams are typically furnished with sofas and televisions.\n
40
+ - Sentence 3: Binne bams are luxurious apartments.\n
41
+ - Sentence 4: Binne bams are places where people live."
42
  ---
43
 
44
+ **How do I pronounce the name of the model?** T0 should be pronounced "T Zero" and any "p" stands for "Plus", so "T0pp" should be pronounced "T Zero Plus Plus"!
45
+
46
  # Model Description
47
 
48
  T0* is a series of encoder-decoder models trained on a large set of different tasks specified in natural language prompts. We convert numerous English supervised datasets into prompts, each with multiple templates using varying formulations. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. To obtain T0*, we fine-tune a pretrained language model on this multitask mixture covering many different NLP tasks.
 
51
 
52
  You can use the models to perform inference on tasks by specifying your query in natural language, and the models will generate a prediction. For instance, you can ask *"Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy"*, and the model will hopefully generate *"Positive"*.
53
 
54
+ A few other examples that you can try:
55
+ - *A is the son's of B's uncle. What is the family relationship between A and B?*
56
+ - *Question A: How is air traffic controlled?<br>
57
+ Question B: How do you become an air traffic controller?<br>
58
+ Pick one: these questions are duplicates or not duplicates.*
59
+ - *Is the word 'table' used in the same meaning in the two previous sentences?<br><br>
60
+ Sentence A: you can leave the books on the table over there.<br>
61
+ Sentence B: the tables in this book are very hard to read.*
62
+ - *Max: Know any good websites to buy clothes from?<br>
63
+ Payton: Sure :) LINK 1, LINK 2, LINK 3<br>
64
+ Max: That's a lot of them!<br>
65
+ Payton: Yeah, but they have different things so I usually buy things from 2 or 3 of them.<br>
66
+ Max: I'll check them out. Thanks.<br><br>
67
+ Who or what are Payton and Max referring to when they say 'them'?*
68
+ - *On a shelf, there are five books: a gray book, a red book, a purple book, a blue book, and a black book.<br>
69
+ The red book is to the right of the gray book. The black book is to the left of the blue book. The blue book is to the left of the gray book. The purple book is the second from the right.<br><br>
70
+ Which book is the leftmost book?*
71
+ - *Reorder the words in this sentence: justin and name bieber years is my am I 27 old.*
72
+
73
  # How to use
74
 
75
+ We make available the models presented in our [paper](https://arxiv.org/abs/2110.08207) along with the ablation models. We recommend using the [T0pp](https://huggingface.co/bigscience/T0pp) (pronounce "T Zero Plus Plus") checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.
76
 
77
  |Model|Number of parameters|
78
  |-|-|
 
99
 
100
  # Training procedure
101
 
102
+ T0* models are based on [T5](https://huggingface.co/google/t5-v1_1-large), a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on [C4](https://huggingface.co/datasets/c4). We use the publicly available [language model-adapted T5 checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) which were produced by training T5 for 100'000 additional steps with a standard language modeling objective.
103
 
104
  At a high level, the input text is fed to the encoder and the target text is produced by the decoder. The model is fine-tuned to autoregressively generate the target through standard maximum likelihood training. It is never trained to generate the input. We detail our training data in the next section.
105
 
 
160
 
161
  # Limitations
162
 
163
+ - The models of the T0* series are quite large (3B or 11B parameters). Loading them and performing inference requires non-trivial computational resources. When using multiple GPUs, it is possible to use [.parallelize()](https://huggingface.co/transformers/parallelism.html).
164
  - We have observed that different prompts can lead to varying performances. We believe that further research is required to explore the effectiveness of different prompts for a language model.
165
  - Due to design choices in the tokenization, the models are unable to perform inference for tasks involving code or non English text.
166
 
 
297
 
298
  ```bibtex
299
  @misc{sanh2021multitask,
300
+ title={Multitask Prompted Training Enables Zero-Shot Task Generalization},
301
  author={Victor Sanh and Albert Webson and Colin Raffel and Stephen H. Bach and Lintang Sutawika and Zaid Alyafeai and Antoine Chaffin and Arnaud Stiegler and Teven Le Scao and Arun Raja and Manan Dey and M Saiful Bari and Canwen Xu and Urmish Thakker and Shanya Sharma Sharma and Eliza Szczechla and Taewoon Kim and Gunjan Chhablani and Nihal Nayak and Debajyoti Datta and Jonathan Chang and Mike Tian-Jian Jiang and Han Wang and Matteo Manica and Sheng Shen and Zheng Xin Yong and Harshit Pandey and Rachel Bawden and Thomas Wang and Trishala Neeraj and Jos Rozen and Abheesht Sharma and Andrea Santilli and Thibault Fevry and Jason Alan Fries and Ryan Teehan and Stella Biderman and Leo Gao and Tali Bers and Thomas Wolf and Alexander M. Rush},
302
  year={2021},
303
  eprint={2110.08207},