---
title: README
emoji: 📉
colorFrom: green
colorTo: yellow
sdk: static
pinned: false
---

# German Wikipedia LMs (GWLMs)

We present Language Models ([BERT](https://huggingface.co/gwlms/bert-base-dewiki-v1), [BERT with Token Dropping](https://huggingface.co/gwlms/bert-base-token-dropping-dewiki-v1), [TEAMS](https://huggingface.co/gwlms/teams-base-dewiki-v1-discriminator), [T5](https://huggingface.co/gwlms/t5-efficient-large-dewiki-v1)) pretrained on German Wikipedia.

This is an ongoing project!

# German Wikipedia Corpus

We use a recent Wikipedia Dump, that can can be accessed [here](https://huggingface.co/datasets/gwlms/dewiki-20230701). Additionally, a sentence-segmented (using NLTK) is available [here](https://huggingface.co/datasets/gwlms/dewiki-20230701-nltk-corpus).

# Fine-tuned Models

We fine-tuned NER models using [SpanMarker](https://github.com/tomaarsen/SpanMarkerNER) library on GermEval 2014 NER dataset and upload the best models:

* [GermEval 2014 NER Model with BERT as backbone LM](https://huggingface.co/gwlms/span-marker-bert-germeval14)
* [GermEval 2014 NER with BERT + Token Dropping as backbone LM](https://huggingface.co/gwlms/span-marker-token-dropping-bert-germeval14)
* [GermEval 2014 NER Model with TEAMS as backbone LM](https://huggingface.co/gwlms/span-marker-teams-germeval14)

# Acknowledgements

Research supported with Cloud TPUs from Google's [TPU Research Cloud](https://sites.research.google/trc/about/) (TRC).
Many Thanks for providing access to the TPUs ❤️