Model continually pre-trained on MS-MARCO (T5 span corruption task).
Starting model: google-t5/t5-base
Training script: follows nanoT5 using the following huggingface class:
T5ForConditionalGenerationRoPE: https://github.com/cxcscmu/LongEmbeddingAnalysis/blob/main/OpenMatch/src/openmatch/modeling/rope_t5.py