---
language:
- en
- fr
- es
- pt
tags:
- falcon3
license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
---
# Falcon3-10B-Base
**Falcon3** family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B parameters.
This repository contains the **Falcon3-10B-Base**. It achieves state-of-the-art results (at release's time) on reasoning, language understanding, instruction following, code and mathematics tasks.
Falcon3-10B-Base supports 4 languages (English, French, Spanish, Portuguese) and a context length of up to 32K.
⚠️ **This is a raw, pretrained model, which should be further finetuned using SFT, RLHF, continued pretraining, etc. for most use cases.**
## Model Details
- Architecture
- Transformer-based causal decoder-only architecture
- 40 decoder blocks
- Grouped Query Attention (GQA) for faster inference: 12 query heads and 4 key-value heads
- Wider head dimension: 256
- High RoPE value to support long context understanding: 1000042
- Uses SwiGLu and RMSNorm
- 32K context length
- 131K vocab size
- Depth up-scaled from **Falcon3-7B-Base** with 2 Teratokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 2048 H100 GPU chips
- Supports EN, FR, ES, PT
- Developed by [Technology Innovation Institute](https://www.tii.ae)
- License: TII Falcon-LLM License 2.0
- Model Release Date: December 2024
## Getting started
Click to expand
```python
import torch
from transformers import pipeline
pipe = pipeline(
"text-generation",
model="tiiuae/Falcon3-10B-Base",
torch_dtype=torch.bfloat16,
device_map="auto"
)
response = pipe("Question: How many hours in one day? Answer: ")
print(response[0]['generated_text'])
```
## Benchmarks
We report in the following table our internal pipeline benchmarks:
Category | Benchmark | Gemma2-9B | Yi1.5-9B | Mistral-NeMo-12B | Falcon3-10B-Base |
---|---|---|---|---|---|
General | MMLU (5-shot) | 70.8 | 69.6 | 68.8 | 73.1 |
MMLU-PRO (5-shot) | 41.4 | 39.3 | 34.7 | 42.5 | |
IFEval | 21.2 | 29.1 | 16.1 | 36.4 | |
Math | GSM8K (5-shot) | 69.1 | 63.8 | 55.3 | 81.4 |
MATH(4-shot) | 10.5 | 9.2 | 4.9 | 22.9 | |
Reasoning | Arc Challenge (25-shot) | 67.5 | 61.7 | 64.4 | 66.8 |
GPQA (0-shot) | 33.4 | 36.6 | 28.8 | 34.1 | |
MUSR (0-shot) | 45.2 | 43.3 | 39.2 | 44.2 | |
BBH (3-shot) | 54.3 | 51.3 | 50.2 | 59.7 | |
CommonSense Understanding | PIQA (0-shot) | 83.0 | 80.5 | 82.1 | 79.5 |
SciQ (0-shot) | 97.1 | 95.2 | 95.2 | 93.5 | |
Winogrande (0-shot) | 74.2 | 72.7 | 73.2 | 73.6 | |
OpenbookQA (0-shot) | 47.2 | 45.2 | 47.2 | 45.0 |