|
--- |
|
pipeline_tag: text-generation |
|
inference: false |
|
license: apache-2.0 |
|
model-index: |
|
- name: ibm/PowerMoE-3b |
|
results: |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: ARC |
|
metrics: |
|
- name: accuracy-norm |
|
type: accuracy-norm |
|
value: 58.1 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: BoolQ |
|
metrics: |
|
- name: accuracy |
|
type: accuracy |
|
value: 65 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: Hellaswag |
|
metrics: |
|
- name: accuracy-norm |
|
type: accuracy-norm |
|
value: 71.5 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: OpenBookQA |
|
metrics: |
|
- name: accuracy-norm |
|
type: accuracy-norm |
|
value: 41 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: PIQA |
|
metrics: |
|
- name: accuracy-norm |
|
type: accuracy-norm |
|
value: 79.1 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: Winogrande |
|
metrics: |
|
- name: accuracy-norm |
|
type: accuracy-norm |
|
value: 65 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: MMLU (5 shot) |
|
metrics: |
|
- name: accuracy |
|
type: accuracy |
|
value: 42.8 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: GSM8k (5 shot) |
|
metrics: |
|
- name: accuracy |
|
type: accuracy |
|
value: 25.9 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: lm-eval-harness |
|
name: math (4 shot) |
|
metrics: |
|
- name: accuracy |
|
type: accuracy |
|
value: 14.8 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: bigcode-eval |
|
name: humaneval |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 20.1 |
|
verified: false |
|
- task: |
|
type: text-generation |
|
dataset: |
|
type: bigcode-eval |
|
name: MBPP |
|
metrics: |
|
- name: pass@1 |
|
type: pass@1 |
|
value: 32.4 |
|
verified: false |
|
base_model: |
|
- ibm/PowerMoE-3b |
|
--- |
|
|
|
## Model Summary |
|
PowerMoE-3B is a 3B sparse Mixture-of-Experts (sMoE) language model trained with the Power learning rate scheduler. It sparsely activates 800M parameters for each token. It is trained on a mix of open-source and proprietary datasets. PowerMoE-3B has shown promising results compared to other dense models with 2x activate parameters across various benchmarks, including natural language multi-choices, code generation, and math reasoning. |
|
Paper: https://arxiv.org/abs/2408.13359 |
|
|
|
This is a GGUF quantized version. |
|
|
|
## Usage |
|
Requires latest llama.cpp to run. |
|
|
|
### Generation |
|
This is a simple example of how to use the PowerMoe GGUF: |
|
|
|
./llama-cli -m PowerMoE4x800M_q3km.gguf -p "How about a snack?" |