# Pico-OpenLAiNN-testing 🤗

Hey there fellow researchers, developers, and AI enthusiasts! Today I'm releasing a *smol* open LLM. This is mainly just a test and I plan to release actually usable models in the near future.
## Models Overview

- **Pico-OpenLAiNN-100M-SmallData**: The smallest of the bunch, this 100M parameter model is perfect for quick experiments and applications where computational resources are *extremely* limited.

## Pretraining Details

This specific version of Pico LAiNN was trained on just 8 billion tokens of the fineweb dataset.

## Other information:

- **Compatibility**: Built to be compatible with existing projects that use LLAMA 2's tokenizer and architecture.
- **Ease of Use**: No need to reinvent the wheel. These models are ready to be plugged into your applications.
- **Open Source**: Fully open source, so you can tweak, tune, and twist them to your heart's content.
## Getting Started

To start using these models, you can simply load them via the Hugging Face `transformers` library:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


MODEL_NAME = "UUFO-Aigis/Pico-OpenLAiNN-100M" #Replace 100M with 250M or 500M if you prefer those models.

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME)

def generate_text(prompt, model, tokenizer, max_length=512, temperature=1, top_k=50, top_p=0.95):
    inputs = tokenizer.encode(prompt, return_tensors="pt")

    outputs = model.generate(
        inputs,
        max_length=max_length,
        temperature=temperature,
        top_k=top_k,
        top_p=top_p,
        do_sample=True
    )


    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return generated_text

def main():
    # Define your prompt
    prompt = "According to all known laws of aviation, there is no way a bee should be able to fly."

    generated_text = generate_text(prompt, model, tokenizer)

    print(generated_text)

if __name__ == "__main__":
    main()
```
## Benchy :3 
|    Tasks     | Value |   |Stderr|
|--------------|------:|---|-----:|
|arc_challenge | 0.1826|±  |0.0113|
|arc_easy      | 0.3859|±  |0.0100|
|boolq         | 0.5804|±  |0.0086|
|hellaswag     | 0.2791|±  |0.0045|
|lambada_openai| 0.2437|±  |0.0060|
|piqa          | 0.6159|±  |0.0113|
|winogrande    | 0.5067|±  |0.0141|



## Future Plans

- **More Models**: I'm currenetly training the bigger siblings of this models, including a 1B parameter version and beyond. 2-4 Billion parameter versions are planned.
- **New architecture**: This is still up in the air and I'm still developing it, and will release if I deem it to be actually useful, so stay tuned!
- **Paper**: A detailed paper will be posted at some point. 

## Credit Where Credit's Due

If you find these models useful and decide to use these models, a link to this repository would be highly appreciated. I am a one man show running this. Thanks 🤗
## Contact
If you have questions, Please reach out to me at urlsys32dll@gmail.com

<p align="center">
  <img src="UUFO.png" alt="U.U.F.O Research Logo" width="250"/>
</p>