metadata
language:
- en
- fr
- es
- pt
tags:
- falcon3
license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
Falcon3-10B-Base
Falcon3 family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B.
This repository contains the Falcon3-10B-Base. It achieves state of art results (at release's time) on reasoning, language understanding, instruction following, code and mathematics tasks. Falcon3-10B-Base supports 4 languages (english, french, spanish, portuguese) and a context length up to 32K.
⚠️ This is a raw, pretrained model, which should be further finetuned using SFT, RLHF, continued pretraining, etc. for most usecases.
Model Details
- Architecture
- Transformer based causal decoder only architecture
- 40 decoder blocks
- Grouped query attention (GQA) for faster inference: 12 query heads and 4 key value heads
- Wider head dimension: 256
- High RoPE value to support long context understanding: 1000042
- Use SwiGLu and RMSNorm
- 32K context length
- 131K vocab size
- Depth-up-scaled from Falcon3-7B-Base with 2 Teratokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 2048 H100 GPU chips
- Supports EN, FR, ES, PT
- Developed by Technology Innovation Institute
- License: TII Falcon-LLM License 2.0
- Model Release Date: December 2024
Getting started
Click to expand
import torch
from transformers import pipeline
pipe = pipeline(
"text-generation",
model="tiiuae/Falcon3-10B-Base",
torch_dtype=torch.bfloat16,
device_map="auto"
)
response = pipe("Question: How many hours in one day? Answer: ")
print(response[0]['generated_text'])
Benchmarks
We report in the following table our internal pipeline benchmarks:
| Category | Benchmark | Gemma2-9B | Yi1.5-9B | Mistral-NeMo-12B | Falcon3-10B-Base |
|---|---|---|---|---|---|
| General | MMLU (5-shot) | 70.8 | 69.6 | 68.8 | 73.1 |
| MMLU-PRO (5-shot) | 41.4 | 39.3 | 34.7 | 42.5 | |
| IFEval | 21.2 | 29.1 | 16.1 | 36.4 | |
| Math | GSM8K (5-shot) | 69.1 | 63.8 | 55.3 | 81.4 |
| MATH(4-shot) | 10.5 | 9.2 | 4.9 | 22.9 | |
| Reasoning | Arc Challenge (25-shot) | 67.5 | 61.7 | 64.4 | 66.8 |
| GPQA (0-shot) | 33.4 | 36.6 | 28.8 | 34.1 | |
| MUSR (0-shot) | 45.2 | 43.3 | 39.2 | 44.2 | |
| BBH (3-shot) | 54.3 | 51.3 | 50.2 | 59.7 | |
| CommonSense Understanding | PIQA (0-shot) | 83.0 | 80.5 | 82.1 | 79.5 |
| SciQ (0-shot) | 97.1 | 95.2 | 95.2 | 93.5 | |
| Winogrande (0-shot) | 74.2 | 72.7 | 73.2 | 73.6 | |
| OpenbookQA (0-shot) | 47.2 | 45.2 | 47.2 | 45.0 |
Technical Report
Coming soon....
Citation
If Falcon3 family were helpful to your work, feel free to give us a cite.
@misc{Falcon3,
title = {The Falcon 3 family of Open Models},
author = {TII Team},
month = {December},
year = {2024}
}