Multilingual LAMA
multilingual version of lama.  The underlying idea of LAMA is to query knowledge from pretrained LMs using templates without any finetuning
LMU Munich
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
lm probing
nan
Nora Kassner, Philipp Dufter, Hinrich Schutze
Recently, it has been found that monolingual English language models can be used as
knowledge bases. Instead of structural knowledge base queries, masked sentences such as
“Paris is the capital of [MASK]” are used as
probes. We translate the established benchmarks TREx and GoogleRE into 53 languages.
Working with mBERT, we investigate three
questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only
considers English. Extending research to multiple languages is important for diversity and
accessibility. (ii) Is mBERT’s performance
as knowledge base language-independent or
does it vary from language to language? (iii)
A multilingual model is trained on more text,
e.g., mBERT is trained on 104 Wikipedias.
Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across
languages and pooling predictions across languages improves performance. Conversely,
mBERT exhibits a language bias; e.g., when
queried in Italian, it tends to predict Italy as
the country of origin.
