metadata
language:
- en
- fr
- es
- pt
tags:
- falcon3
Falcon3-10B-Base
Falcon3 family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B.
This repository contains the Falcon3-10B-Base. It achieves state of art results (at release's time) on reasoning, language understanding, instruction following, code and mathematics tasks. Falcon3-10B-Base supports 4 languages (english, french, spanish, portuguese) and a context length up to 32K.
⚠️ This is a raw, pretrained model, which should be further finetuned for most usecases.
Model Details
- Architecture
- transformer based causal decoder only architecture
- 40 decoder blocks
- grouped query attention (GQA) for faster inference: 12 query heads and 4 KV heads
- wider head dimension: 256
- high RoPE value to support long context understanding: 1000042
- 32k context length
- 131k vocab size
- Depth-up-scaled from Falcon3-7B-Base with 2 Gigatokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 2048 H100 GPU chips
- Supports EN, FR, ES, PT
- Developed by Technology Innovation Institute
- License: TII Falcon-LLM License 2.0
- Model Release Date: December 2024
Getting started
Click to expand
import torch
from transformers import pipeline
pipe = pipeline(
"text-generation",
model="tiiuae/Falcon3-10B-Base",
torch_dtype=torch.bfloat16,
device_map="auto"
)
response = pipe("Question: How many hours in one day? Answer: ")
print(response[0]['generated_text'])
Benchmarks
We report in the following table our internal pipeline benchmarks:
| Category | Benchmark | Gemma2-9B | Yi1.5-9B | Mistral-NeMo-12B | Falcon3-10B-Base |
|---|---|---|---|---|---|
| General | MMLU (5-shot) | 0 | 69.6 | 68.8 | 73.1 |
| MMLU-PRO (5-shot) | 0 | 39.3 | 34.7 | 42.5 | |
| IFEval | 0 | 29.1 | 16.1 | 36.4 | |
| Math | GSM8K (5-shot) | 69.1 | 63.8 | 55.3 | 81.4 |
| MATH(4-shot) | 0 | 9.2 | 4.9 | 22.9 | |
| Reasoning | Arc Challenge (25-shot) | 63.7 | 58.2 | 60.6 | 62.6 |
| GPQA (0-shot) | 0 | 36.6 | 28.8 | 34.1 | |
| MUSR (0-shot) | 0 | 43.3 | 39.2 | 44.2 | |
| BBH (3-shot) | 0 | 51.3 | 50.2 | 59.7 | |
| CommonSense Understanding | PIQA (0-shot) | 81.4 | 79.8 | 81.4 | 79.1 |
| SciQ (0-shot) | 97.2 | 95.8 | 96.4 | 96.0 | |
| Winogrande (0-shot) | 74.2 | 72.7 | 73.2 | 73.6 | |
| OpenbookQA (0-shot) | 34.0 | 35.4 | 36.4 | 34.0 |
Citation
If Falcon3 family were helpful to your work, feel free to give us a cite.
@misc{Falcon3,
title = {The Falcon 3 family of Open Models},
author = {TII Team},
month = {December},
year = {2024}
}