Update README.md

README.md

@@ -1,3 +1,34 @@
----
-license: mit
----
+---
+license: mit
+language:
+- tr
+library_name: transformers
+---
+
+# TS-Corpus WordPiece Tokenizer (256k, Cased)
+
+## Overview
+This repository contains a WordPiece tokenizer with a vocabulary size of 256,000, trained uncased on various datasets from the TS Corpus website. It is designed to handle Turkish text, leveraging rich and diverse sources to provide a robust tool for natural language processing tasks.
+
+## Dataset Sources
+The tokenizer was trained using multiple corpora from the TS Corpus, specifically:
+- [TS Corpus V2](https://tscorpus.com/corpora/ts-corpus-v2/)
+- [TS Wikipedia Corpus](https://tscorpus.com/corpora/ts-wikipedia-corpus/)
+- [TS Abstract Corpus](https://tscorpus.com/corpora/ts-abstract-corpus/)
+- [TS Idioms and Proverbs Corpus](https://tscorpus.com/corpora/ts-idioms-and-proverbs-corpus/)
+- [Syllable Corpus](https://tscorpus.com/corpora/syllable-corpus/)
+- [Turkish Constitution Corpus](https://tscorpus.com/corpora/turkish-constitution-corpus/)
+
+These diverse sources include a wide range of texts from encyclopedic articles to legal documents, providing a comprehensive linguistic foundation for the tokenizer.
+
+## Tokenizer Model
+The tokenizer uses the WordPiece model, which is widely utilized in many modern NLP systems. It is particularly effective in handling languages with rich morphology like Turkish due to its subword segmentation approach. This tokenizer differentiates between uppercase and lowercase letters.
+
+## Usage
+To use this tokenizer, you can load it via the Hugging Face `transformers` library as follows:
+
+```python
+from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("tahaenesaslanturk/ts-corpus-wordpiece-256k-cased")
+```
+