This is an adapted tokenizer from GPT2 that can recognize tokens to do with Python coding. It is part of the huggingfaceNLP course exercise. It uses the method train_new_from_iterator()

Downloads last month: -; Downloads are not tracked for this model. How to track

Skier8402
/

code-search-net-tokenizer

Dataset used to train Skier8402/code-search-net-tokenizer