---
license: apache-2.0
---

This is the model for my Course Project at 3 Year of HSE FCS. Python notebooks for my
research can be found here: https://github.com/mastavtsev/PM_NLP/tree/main.

Model is specified for the task of unsupervised anomaly detection using SqueezeBERT architecture.
The model is trained using masked language modeling using traces of normal program execution. Thus, the model produces a representation
of trace tokens corresponding to normal program execution. 

Meta info:
- Tokenizer LOA 13 - 20000 dictionary size, 300 max. token length
- 512 context window size
- 2.5e-3 learning rate
- LAMB optimizer
- 300 epochs
- 43.6 million parameters
- 1.5 hours in Google Colab with GPU A100

Results of the model on the test data:
![image/png](https://cdn-uploads.huggingface.co/production/uploads/661915f3fdcb7df8a6e2fdaa/bsZ8dXWN6IoAH4zXfwzi2.png)