Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Model continually pre-trained on MS-MARCO (T5 span corruption task).
|
| 2 |
+
|
| 3 |
+
Starting model: google-t5/t5-base
|
| 4 |
+
|
| 5 |
+
Training script: follows nanoT5 using the following huggingface class:
|
| 6 |
+
T5ForConditionalGenerationRoPE: https://github.com/cxcscmu/LongEmbeddingAnalysis/blob/main/OpenMatch/src/openmatch/modeling/rope_t5.py
|