deliciouscat/deberta-v3-base-encoder-decoder-v0.3

Encoder-Decoder model with DeBERTa encoder

pre-trained models

• deliciouscat/deberta-v3-base-encoder-decoder-v0.2

-> 297511524(298M) params

Data used

• HuggingFaceFW/fineweb

• AiHub ko-en translation corpus (English part)

• Some papers that I kept

Training hparams

• optimizer: AdamW, lr=3e-5, betas=(0.875, 0.997)

• batch size: 12

-> training on denoising objective (BART), 29523 step

How to use

from transformers import AutoTokenizer, EncoderDecoderModel

model = EncoderDecoderModel.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.3")
tokenizer = AutoTokenizer.from_pretrained("deliciouscat/deberta-v3-base-encoder-decoder-v0.3")

Future work!

• train more scientific data

• fine-tune on keyword extraction task