init: weights

README.md

@@ -1,3 +1,171 @@
 ---
+library_name: transformers
 license: apache-2.0
+tags:
+- jamba
+- mamba
+- moe
 ---
+
+# Jamba-v0.1-9B
+
+A dense version of [Jamba-v0.1](https://huggingface.co/ai21labs/Jamba-v0.1), which extracts the weights of the first expert.
+It no longer uses MoE. Please refer to [this script](https://github.com/TechxGenus/Jamba-utils/blob/main/dense_downcycling.py) for details.
+It can use single 3090/4090 for inference, and the usage method is exactly the same as Jamba-v0.1.
+
+---
+
+# Original Model Card for Jamba
+
+Jamba is a state-of-the-art, hybrid SSM-Transformer LLM. It delivers throughput gains over traditional Transformer-based models, while outperforming or matching the leading models of its size class on most common benchmarks.
+
+Jamba is the first production-scale Mamba implementation, which opens up interesting research and application opportunities. While this initial experimentation shows encouraging gains, we expect these to be further enhanced with future optimizations and explorations.
+
+This model card is for the base version of Jamba. It’s a pretrained, mixture-of-experts (MoE) generative text model, with 12B active parameters and a total of 52B parameters across all experts. It supports a 256K context length, and can fit up to 140K tokens on a single 80GB GPU.
+
+For full details of this model please read the [release blog post](https://www.ai21.com/blog/announcing-jamba).
+
+## Model Details
+
+- **Developed by:** [AI21](https://www.ai21.com)
+- **Model type:** Joint Attention and Mamba (Jamba)
+- **License:** Apache 2.0
+- **Context length:** 256K
+- **Knowledge cutoff date:** March 5, 2024
+
+## Usage
+### Presequities
+Jamba requires you use `transformers` version 4.39.0 or higher:
+```bash
+pip install transformers>=4.39.0
+```
+
+In order to run optimized Mamba implementations, you first need to install `mamba-ssm` and `causal-conv1d`:
+```bash
+pip install mamba-ssm causal-conv1d>=1.2.0
+```
+You also have to have the model on a CUDA device.
+
+You can run the model not using the optimized Mamba kernels, but it is **not** recommended as it will result in significantly lower latencies. In order to do that, you'll need to specify `use_mamba_kernels=False` when loading the model.
+
+### Run the model
+Please note that, at the moment, `trust_remote_code=True` is required for running the new Jamba architecture.
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1",
+                                             trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1")
+
+input_ids = tokenizer("In the recent Super Bowl LVIII,", return_tensors='pt').to(model.device)["input_ids"]
+
+outputs = model.generate(input_ids, max_new_tokens=216)
+
+print(tokenizer.batch_decode(outputs))
+# ["<|startoftext|>In the recent Super Bowl LVIII, the Kansas City Chiefs emerged victorious, defeating the San Francisco 49ers in a thrilling overtime showdown. The game was a nail-biter, with both teams showcasing their skills and determination.\n\nThe Chiefs, led by their star quarterback Patrick Mahomes, displayed their offensive prowess, while the 49ers, led by their strong defense, put up a tough fight. The game went into overtime, with the Chiefs ultimately securing the win with a touchdown.\n\nThe victory marked the Chiefs' second Super Bowl win in four years, solidifying their status as one of the top teams in the NFL. The game was a testament to the skill and talent of both teams, and a thrilling end to the NFL season.\n\nThe Super Bowl is not just about the game itself, but also about the halftime show and the commercials. This year's halftime show featured a star-studded lineup, including Usher, Alicia Keys, and Lil Jon. The show was a spectacle of music and dance, with the performers delivering an energetic and entertaining performance.\n"]
+```
+
+<details>
+<summary><strong>Loading the model in half precision</strong></summary>
+  
+  The published checkpoint is saved in BF16. In order to load it into RAM in BF16/FP16, you need to specify `torch_dtype`:
+  
+```python
+from transformers import AutoModelForCausalLM
+import torch
+model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1",
+                                             trust_remote_code=True,
+                                             torch_dtype=torch.bfloat16)    # you can also use torch_dtype=torch.float16
+```
+
+When using half precision, you can enable the [FlashAttention2](https://github.com/Dao-AILab/flash-attention) implementation of the Attention blocks. In order to use it, you also need the model on a CUDA device. Since in this precision the model is to big to fit on a single 80GB GPU, you'll also need to parallelize it using [accelerate](https://huggingface.co/docs/accelerate/index):
+```python
+from transformers import AutoModelForCausalLM
+import torch
+model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1",
+                                             trust_remote_code=True,
+                                             torch_dtype=torch.bfloat16,
+                                             attn_implementation="flash_attention_2",
+                                             device_map="auto")
+```
+
+</details>
+<details><summary><strong>Load the model in 8-bit</strong></summary>
+  
+  **Using 8-bit precision, it is possible to fit up to 140K sequence lengths on a single 80GB GPU.** You can easily quantize the model to 8-bit using [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index). In order to not degrade model quality, we recommend to exclude the Mamba blocks from the quantization:
+
+```python
+from transformers import AutoModelForCausalLM, BitsAndBytesConfig
+quantization_config = BitsAndBytesConfig(load_in_8bit=True,
+                                         llm_int8_skip_modules=["mamba"])
+model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1",
+                                             trust_remote_code=True,
+                                             torch_dtype=torch.bfloat16,
+                                             attn_implementation="flash_attention_2",
+                                             quantization_config=quantization_config)
+```
+</details>
+
+### Fine-tuning example
+Jamba is a base model that can be fine-tuned for custom solutions (including for chat/instruct versions). You can fine-tune it using any technique of your choice. Here is an example of fine-tuning with the [PEFT](https://huggingface.co/docs/peft/index) library:
+
+```python
+from datasets import load_dataset
+from trl import SFTTrainer
+from peft import LoraConfig
+from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
+
+tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1")
+model = AutoModelForCausalLM.from_pretrained("ai21labs/Jamba-v0.1", trust_remote_code=True, device_map='auto')
+
+dataset = load_dataset("Abirate/english_quotes", split="train")
+training_args = TrainingArguments(
+    output_dir="./results",
+    num_train_epochs=3,
+    per_device_train_batch_size=4,
+    logging_dir='./logs',
+    logging_steps=10,
+    learning_rate=2e-3
+)
+lora_config = LoraConfig(
+    r=8,
+    target_modules=["embed_tokens", "x_proj", "in_proj", "out_proj"],
+    task_type="CAUSAL_LM",
+    bias="none"
+)
+trainer = SFTTrainer(
+    model=model,
+    tokenizer=tokenizer,
+    args=training_args,
+    peft_config=lora_config,
+    train_dataset=dataset,
+    dataset_text_field="quote",
+)
+
+trainer.train()
+```
+
+## Results on common benchmarks
+| Benchmark    | Score |
+|--------------|:-----:|
+| HellaSwag    | 87.1% |
+| Arc Challenge | 64.4% |
+| WinoGrande   | 82.5% |
+| PIQA        | 83.2% |
+| MMLU       | 67.4% |
+| BBH            | 45.4% |
+| TruthfulQA          | 46.4% |
+| GSM8K (CoT)            | 59.9% |
+
+It's crucial that the 'BOS' token is added to all prompts, which might not be enabled by default in all eval frameworks.
+
+
+## Notice
+Jamba is a pretrained base model and did not undergo any alignment for instruct/chat interactions. 
+
+As a base model, Jamba is intended for use as a foundation layer for fine tuning, training, and developing custom solutions. Jamba does not have safety moderation mechanisms and guardrails should be added for responsible and safe use.
+
+## About AI21
+AI21 builds reliable, practical, and scalable AI solutions for the enterprise.
+
+Jamba is the first in AI21’s new family of models, and the Instruct version of Jamba is available in beta via the [AI21 platform](https://www.ai21.com/studio). 

config.json

@@ -0,0 +1,48 @@
+{
+  "architectures": [
+    "JambaForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "attn_layer_offset": 4,
+  "attn_layer_period": 8,
+  "auto_map": {
+    "AutoConfig": "configuration_jamba.JambaConfig",
+    "AutoModel": "modeling_jamba.JambaModel",
+    "AutoModelForCausalLM": "modeling_jamba.JambaForCausalLM",
+    "AutoModelForSequenceClassification": "model.JambaForSequenceClassification"
+  },
+  "bos_token_id": 1,
+  "calc_logits_for_entire_prompt": false,
+  "eos_token_id": 2,
+  "expert_layer_offset": 1,
+  "expert_layer_period": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "mamba_conv_bias": true,
+  "mamba_d_conv": 4,
+  "mamba_d_state": 16,
+  "mamba_dt_rank": 256,
+  "mamba_expand": 2,
+  "mamba_inner_layernorms": true,
+  "mamba_proj_bias": false,
+  "model_type": "jamba",
+  "n_ctx": 262144,
+  "num_attention_heads": 32,
+  "num_experts": 1,
+  "num_experts_per_tok": 1,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "output_router_logits": false,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "router_aux_loss_coef": 0.001,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.39.3",
+  "use_cache": true,
+  "use_mamba_kernels": true,
+  "vocab_size": 65536
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "transformers_version": "4.39.3"
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c60b5778b6856888ca24447b3880c39a777e523e5fb94dd18ed33d9532d01f0
+size 4916251136

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58aa641dcb763378138fd1b4151c0726c8e06969417f1ddaccbcb04a6de69c79
+size 4941976032

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45ab7424c95e53b2a769629468ecfb953cdc2f65790d5fa2acea4f27d567b538
+size 4941976080

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9698548b8ec18b2258676c33df63f0c96986aa9a5c0846decc31c58e9d8a1e5c
+size 3788575776

model.safetensors.index.json

@@ -0,0 +1,522 @@
+{
+  "metadata": {
+    "total_size": 18588720896
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "model.final_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.B_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.C_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.dt_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.dt_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.dt_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mamba.x_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.moe.experts.0.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.moe.experts.0.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.moe.experts.0.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.pre_moe_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.B_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.C_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.dt_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.dt_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.dt_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mamba.x_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.moe.experts.0.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.moe.experts.0.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.moe.experts.0.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.pre_moe_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.B_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.C_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.dt_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.dt_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.dt_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mamba.x_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.B_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.C_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.dt_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.dt_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.dt_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mamba.x_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.B_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.C_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.dt_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.dt_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.dt_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mamba.x_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.B_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.C_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.dt_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.dt_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.dt_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mamba.x_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.B_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.C_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.dt_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.dt_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.dt_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mamba.x_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.B_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.C_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.dt_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.dt_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.dt_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mamba.x_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.17.mamba.B_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mamba.C_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.17.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.17.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mamba.dt_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mamba.dt_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.17.mamba.dt_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mamba.x_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.B_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.C_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.dt_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.dt_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.dt_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mamba.x_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.B_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.C_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.dt_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.dt_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.dt_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mamba.x_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.B_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.C_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.dt_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.dt_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.dt_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mamba.x_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.moe.experts.0.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.moe.experts.0.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.moe.experts.0.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.pre_moe_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.B_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.C_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.dt_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.dt_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.dt_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mamba.x_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.B_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.C_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.dt_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.dt_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.dt_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mamba.x_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.B_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.C_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.dt_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.dt_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.dt_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mamba.x_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.B_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.C_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.dt_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.dt_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.dt_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mamba.x_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.B_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.C_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.dt_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.dt_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.dt_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.in_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.out_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mamba.x_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.moe.experts.0.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.moe.experts.0.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.moe.experts.0.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.pre_moe_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mamba.A_log": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.B_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mamba.C_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mamba.D": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.conv1d.bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.conv1d.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mamba.dt_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mamba.dt_proj.bias": "model-00004-of-00004.safetensors",
+    "model.layers.26.mamba.dt_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mamba.x_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.moe.experts.0.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.moe.experts.0.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.moe.experts.0.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.pre_moe_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.A_log": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.B_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.C_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.D": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.dt_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.dt_proj.bias": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.dt_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mamba.x_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.moe.experts.0.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.moe.experts.0.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.moe.experts.0.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.pre_moe_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.moe.experts.0.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.moe.experts.0.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.moe.experts.0.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.pre_moe_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.A_log": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.B_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.C_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.D": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.dt_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.dt_proj.bias": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.dt_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.mamba.x_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.moe.experts.0.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.moe.experts.0.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.moe.experts.0.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.29.pre_moe_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.B_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.C_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.dt_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.dt_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.dt_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mamba.x_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.moe.experts.0.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.moe.experts.0.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.moe.experts.0.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.pre_moe_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.A_log": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.B_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.C_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.D": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.dt_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.dt_proj.bias": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.dt_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.mamba.x_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.moe.experts.0.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.moe.experts.0.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.moe.experts.0.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.30.pre_moe_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.A_log": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.B_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.C_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.D": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.conv1d.bias": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.conv1d.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.dt_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.dt_proj.bias": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.dt_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.in_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.out_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.mamba.x_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.moe.experts.0.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.moe.experts.0.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.moe.experts.0.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.31.pre_moe_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.moe.experts.0.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.moe.experts.0.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.moe.experts.0.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.pre_moe_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.B_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.C_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.dt_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.dt_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.dt_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mamba.x_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.moe.experts.0.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.moe.experts.0.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.moe.experts.0.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.pre_moe_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.B_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.C_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.dt_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.dt_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.dt_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mamba.x_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.moe.experts.0.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.moe.experts.0.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.moe.experts.0.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.pre_moe_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.B_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.C_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.dt_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.dt_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.dt_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.in_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.out_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mamba.x_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.moe.experts.0.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.moe.experts.0.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.moe.experts.0.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.pre_moe_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mamba.A_log": "model-00001-of-00004.safetensors",
+    "model.layers.8.mamba.B_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mamba.C_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mamba.D": "model-00001-of-00004.safetensors",
+    "model.layers.8.mamba.conv1d.bias": "model-00001-of-00004.safetensors",
+    "model.layers.8.mamba.conv1d.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mamba.dt_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mamba.dt_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.8.mamba.dt_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mamba.x_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.A_log": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.B_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.C_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.D": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.conv1d.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.conv1d.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.dt_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.dt_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.dt_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.in_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.out_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mamba.x_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.moe.experts.0.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.moe.experts.0.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.moe.experts.0.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.pre_moe_layernorm.weight": "model-00002-of-00004.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|unk|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02fd6530b8ede0eedd8e509fcab32da7b1dd04c8119f8498c787100f13112713
+size 1124742

tokenizer_config.json

@@ -0,0 +1,47 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|startoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|unk|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<|unk|>",
+  "use_default_system_prompt": false
+}