Create README.md

README.md

@@ -0,0 +1,7 @@
+This model was pretrained on the bookcorpus dataset using knowledge distillation.
+
+The particularity of this model is that even though it shares the same architecture as BERT, it has a hidden size of 384 (half the hidden size of BERT) and 6 attention heads.
+The knowledge distillation was performed using multiple loss functions.
+The weights of the model were initialized from scratch.
+
+PS : the tokenizer is the same as the one of the model bert-base-uncased.