--- language: - en - fr - es - pt tags: - falcon3 --- # Falcon3-10B-Base **Falcon3** family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B. This repository contains the **Falcon3-10B-Base**. It achieves state of art results (at release's time) on reasoning, language understanding, instruction following, code and mathematics tasks. Falcon3-10B-Base supports 4 languages (english, french, spanish, portuguese) and a context length up to 32K. ⚠️ **This is a raw, pretrained model, which should be further finetuned for most usecases.** ## Model Details - Architecture - transformer based causal decoder only architecture - 40 decoder blocks - grouped query attention (GQA) for faster inference: 12 query heads and 4 KV heads - wider head dimension: 256 - high RoPE value to support long context understanding: 1000042 - 32k context length - 131k vocab size - Depth-up-scaled from **Falcon3-7B-Base** with 2 Gigatokens of datasets comprising of web, code, STEM, high quality and mutlilingual data using 2048 H100 GPU chips - Supports EN, FR, ES, PT - Developed by [Technology Innovation Institute](https://www.tii.ae) - License: TII Falcon-LLM License 2.0 - Model Release Date: December 2024 ## Getting started
Click to expand ```python import torch from transformers import pipeline pipe = pipeline( "text-generation", model="tiiuae/Falcon3-10B-Base", torch_dtype=torch.bfloat16, device_map="auto" ) response = pipe("Question: How many hours in one day? Answer: ") print(response[0]['generated_text']) ```

# Benchmarks We report in the following table our internal pipeline benchmarks:
Category Benchmark Gemma2-9B Yi1.5-9B Mistral-NeMo-12B Falcon3-10B-Base
General MMLU (5-shot) 0 69.6 68.8 73.1
MMLU-PRO (5-shot) 0 39.3 34.7 42.5
IFEval 0 29.1 16.1 36.4
Math GSM8K (5-shot) 69.1 63.8 55.3 81.4
MATH(4-shot) 0 9.2 4.9 22.9
Reasoning Arc Challenge (25-shot) 63.7 58.2 60.6 62.6
GPQA (0-shot) 0 36.6 28.8 34.1
MUSR (0-shot) 0 43.3 39.2 44.2
BBH (3-shot) 0 51.3 50.2 59.7
CommonSense Understanding PIQA (0-shot) 81.4 79.8 81.4 79.1
SciQ (0-shot) 97.2 95.8 96.4 96.0
Winogrande (0-shot) 74.2 72.7 73.2 73.6
OpenbookQA (0-shot) 34.0 35.4 36.4 34.0
# Citation If Falcon3 family were helpful to your work, feel free to give us a cite. ``` @misc{Falcon3, title = {Falcon 3 family of Open Foundation Models}, author = {TII Team}, month = {December}, year = {2024} } ```