README.md · mllmTeam/PhoneLM-0.5B at main

metadata

license: apache-2.0
datasets:
  - mlfoundations/dclm-baseline-1.0-parquet
  - bigcode/starcoderdata
  - open-web-math/open-web-math
  - allenai/dolma
language:
  - en
library_name: transformers

PhoneLM-0.5B is a 0.5 billion parameter decoder-only language model pre-trained on 1.1 trillion tokens.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = 'mllmTeam/PhoneLM-0.5B'
model = AutoModelForCausalLM.from_pretrained(model_name, device_map='cuda', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)

inp = tokenizer("Machine Learning is ", return_tensors="pt")
inp = {k: v.to('cuda') for k, v in inp.items()}
out = model.generate(**inp, 
                     max_length=256,
                     do_sample=True,
                     temperature=0.7,
                     top_p=0.7
                     )
text = tokenizer.decode(out[0], skip_special_tokens=True)
print(text)

Model Details

Developed by: mllmTeam
Model type: PhoneLM 0.5B models are auto-regressive language models based on the transformer decoder architecture.
Language(s): English
Paper: PhoneLM Technical Report
Library: PhoneLM

Model Architecture

The model is a decoder-only transformer architecture with the following modifications:

Hidden Size	Layers	Heads	Sequence Length
1024	24	16	2048

Position Embeddings: Rotary Position Embeddings (Su et al., 2021) applied to the first 25% of head embedding dimensions for improved throughput following Black et al. (2022). PhoneLM quantized the sin and cos values in Rotary Position Embeddings to 8-bit integers.
Normalization: LayerNorm (Ba et al., 2016) with learned bias terms as opposed to RMSNorm (Zhang & Sennrich, 2019).
Biases: We remove all bias terms from the feed-forward networks and multi-head self-attention layers, except for the biases of the query, key, and value projections (Bai et al., 2023).
ReLU Activation Function: ReLU(Glorot et al., 2011) activation functions are adopted in feed-forward networks.
Tokenizer: We use the SmolLM(Allal et al., 2024)'s tokenizer with a vocabulary size of 49,152.

Training Dataset

The training dataset PhoneLM used is comprised of a filtered mixture of open-source large-scale datasets available on the HuggingFace Hub: DCLM-baseline(Li et al., 2024), StarCoder (Li et al., 2023), OpenWebMath (Paster et al., 2023) and Dolma (Soldaini et al., 2024).

Evaluation Results

Model	HellaSwag	WinoGrande	PIQA	SciQ	BoolQ	ARC Easy	ARC Challenge	Average
PhoneLM-0.5B	54.0	57.9	73.3	85.1	60.7	60.4	31.6	60.43
Pythia-410M	40.6	53.7	66.9	72.4	60.3	45.9	24.5	52.04
OPT-350M	36.8	52.3	64.3	68.5	57.6	40.1	23.7	49.04
BLOOM-560M	36.9	51.7	65.0	71.7	53.3	41.8	23.7	49.16
MobiLlama-500M	51.1	53.4	70.9	76.4	55.7	46.0	26.6	54.30
OpenELM-450M	54.0	58.0	72.3	79.4	55.8	48.1	27.6	56.46
SmolLM-360M	53.5	56.8	71.5	84.2	55.4	63.8	36.0	60.17
Qwen 1.5-500M	49.2	55.7	69.5	82.5	49.5	52.3	29.4	55.44
Cerebras-GPT-590M	32.3	49.8	62.8	68.2	59.2	41.2	23.5	48.14

License

This repository is released under the Apache-2.0 License.

Citation

@misc{yi2024phonelmanefficientcapablesmall,
      title={PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training}, 
      author={Rongjie Yi and Xiang Li and Weikai Xie and Zhenyan Lu and Chenghua Wang and Ao Zhou and Shangguang Wang and Xiwen Zhang and Mengwei Xu},
      year={2024},
      eprint={2411.05046},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.05046}, 
}