Add files using upload-large-folder tool

README.md

@@ -0,0 +1,274 @@
+---
+license: other
+license_name: mrl
+license_link: https://mistral.ai/licenses/MRL-0.1.md
+language:
+  - en
+  - fr
+  - de
+  - es
+  - it
+  - pt
+  - zh
+  - ja
+  - ru 
+  - ko
+
+extra_gated_description: If you want to learn more about how we process your personal data, please read our <a href="https://mistral.ai/terms/">Privacy Policy</a>.
+---
+
+# Model Card for Mistral-Large-Instruct-2407
+
+Mistral-Large-Instruct-2407 is an advanced dense Large Language Model (LLM) of 123B parameters with state-of-the-art reasoning, knowledge and coding capabilities.
+
+For more details about this model please refer to our release [blog post](https://mistral.ai/news/mistral-large-2407/).
+
+## Key features
+- **Multi-lingual by design:** Dozens of languages supported, including English, French, German, Spanish, Italian, Chinese, Japanese, Korean, Portuguese, Dutch and Polish.
+- **Proficient in coding:** Trained on 80+ coding languages such as Python, Java, C, C++, Javacsript, and Bash. Also trained on more specific languages such as Swift and Fortran.
+- **Agentic-centric:** Best-in-class agentic capabilities with native function calling and JSON outputting.
+- **Advanced Reasoning:** State-of-the-art mathematical and reasoning capabilities.
+- **Mistral Research License:** Allows usage and modification for research and non-commercial usages.
+- **Large Context:** A large 128k context window.
+
+## Metrics
+
+### Base Pretrained Benchmarks
+
+| Benchmark | Score |
+| --- | --- |
+| MMLU | 84.0% |
+
+
+### Base Pretrained Multilingual Benchmarks (MMLU)
+| Benchmark | Score |
+| --- | --- |
+| French | 82.8% |
+| German | 81.6% |
+| Spanish | 82.7% |
+| Italian | 82.7% |
+| Dutch | 80.7% |
+| Portuguese | 81.6% |
+| Russian | 79.0% |
+| Korean | 60.1% |
+| Japanese | 78.8% |
+| Chinese | 74.8% |
+
+
+### Instruction Benchmarks
+
+| Benchmark | Score |
+| --- | --- |
+| MT Bench | 8.63 |
+| Wild Bench | 56.3 |
+| Arena Hard| 73.2 |
+
+### Code & Reasoning Benchmarks
+| Benchmark | Score |
+| --- | --- |
+| Human Eval | 92% |
+| Human Eval Plus| 87% |
+| MBPP Base| 80% |
+| MBPP Plus| 69% |
+
+### Math Benchmarks
+
+| Benchmark | Score |
+| --- | --- |
+| GSM8K | 93% |
+| Math Instruct (0-shot, no CoT) | 70% |
+| Math Instruct (0-shot, CoT)| 71.5% |
+
+## Usage
+
+The model can be used with two different frameworks
+
+- [`mistral_inference`](https://github.com/mistralai/mistral-inference): See [here](#mistral-inference)
+- [`transformers`](https://github.com/huggingface/transformers): See [here](#transformers)
+
+### Mistral Inference
+
+#### Install
+
+It is recommended to use `mistralai/Mistral-Large-Instruct-2407` with [mistral-inference](https://github.com/mistralai/mistral-inference). For HF transformers code snippets, please keep scrolling.
+
+```
+pip install mistral_inference
+```
+
+#### Download
+
+```py
+from huggingface_hub import snapshot_download
+from pathlib import Path
+
+mistral_models_path = Path.home().joinpath('mistral_models', 'Large')
+mistral_models_path.mkdir(parents=True, exist_ok=True)
+
+snapshot_download(repo_id="mistralai/Mistral-Large-Instruct-2407", allow_patterns=["params.json", "consolidated-*.safetensors", "tokenizer.model.v3"], local_dir=mistral_models_path)
+```
+
+#### Chat
+
+After installing `mistral_inference`, a `mistral-chat` CLI command should be available in your environment.
+Given the size of this model, you will need a node with several GPUs (more than 300GB cumulated vRAM).
+If you have 8 GPUs on your machine, you can chat with the model using
+
+```
+torchrun --nproc-per-node 8 --no-python mistral-chat $HOME/mistral_models/Large --instruct --max_tokens 256 --temperature 0.7
+```
+
+*E.g.* Try out something like:
+```
+How expensive would it be to ask a window cleaner to clean all windows in Paris. Make a reasonable guess in US Dollar.
+```
+
+#### Instruct following
+
+```py
+from mistral_inference.transformer import Transformer
+from mistral_inference.generate import generate
+
+from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
+from mistral_common.protocol.instruct.messages import UserMessage
+from mistral_common.protocol.instruct.request import ChatCompletionRequest
+
+tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tokenizer.model.v3")
+model = Transformer.from_folder(mistral_models_path)
+
+prompt = "How expensive would it be to ask a window cleaner to clean all windows in Paris. Make a reasonable guess in US Dollar."
+
+completion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)])
+
+tokens = tokenizer.encode_chat_completion(completion_request).tokens
+
+out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.7, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
+result = tokenizer.decode(out_tokens[0])
+
+print(result)
+```
+
+#### Function calling
+
+```py
+from mistral_common.protocol.instruct.tool_calls import Function, Tool
+from mistral_inference.transformer import Transformer
+from mistral_inference.generate import generate
+
+from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
+from mistral_common.protocol.instruct.messages import UserMessage
+from mistral_common.protocol.instruct.request import ChatCompletionRequest
+
+
+tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tokenizer.model.v3")
+model = Transformer.from_folder(mistral_models_path)
+
+completion_request = ChatCompletionRequest(
+    tools=[
+        Tool(
+            function=Function(
+                name="get_current_weather",
+                description="Get the current weather",
+                parameters={
+                    "type": "object",
+                    "properties": {
+                        "location": {
+                            "type": "string",
+                            "description": "The city and state, e.g. San Francisco, CA",
+                        },
+                        "format": {
+                            "type": "string",
+                            "enum": ["celsius", "fahrenheit"],
+                            "description": "The temperature unit to use. Infer this from the users location.",
+                        },
+                    },
+                    "required": ["location", "format"],
+                },
+            )
+        )
+    ],
+    messages=[
+        UserMessage(content="What's the weather like today in Paris?"),
+        ],
+)
+
+tokens = tokenizer.encode_chat_completion(completion_request).tokens
+
+out_tokens, _ = generate([tokens], model, max_tokens=256, temperature=0.7, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
+result = tokenizer.decode(out_tokens[0])
+
+print(result)
+```
+
+### Transformers
+
+If you want to use Hugging Face `transformers` to generate text, you can do something like this.
+
+```py
+from transformers import pipeline
+
+messages = [
+    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
+    {"role": "user", "content": "Who are you?"},
+]
+chatbot = pipeline("text-generation", model="mistralai/Mistral-Large-Instruct-2407")
+chatbot(messages)
+```
+
+## Function calling with `transformers`
+
+To use this example, you'll need `transformers` version 4.42.0 or higher. Please see the 
+[function calling guide](https://huggingface.co/docs/transformers/main/chat_templating#advanced-tool-use--function-calling)
+in the `transformers` docs for more information.
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+model_id = "mistralai/Mistral-Large-Instruct-2407"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+def get_current_weather(location: str, format: str):
+    """
+    Get the current weather
+
+    Args:
+        location: The city and state, e.g. San Francisco, CA
+        format: The temperature unit to use. Infer this from the users location. (choices: ["celsius", "fahrenheit"])
+    """
+    pass
+
+conversation = [{"role": "user", "content": "What's the weather like in Paris?"}]
+tools = [get_current_weather]
+
+# format and tokenize the tool use prompt 
+inputs = tokenizer.apply_chat_template(
+            conversation,
+            tools=tools,
+            add_generation_prompt=True,
+            return_dict=True,
+            return_tensors="pt",
+)
+
+model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
+
+inputs.to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=1000)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+Note that, for reasons of space, this example does not show a complete cycle of calling a tool and adding the tool call and tool
+results to the chat history so that the model can use them in its next generation. For a full tool calling example, please
+see the [function calling guide](https://huggingface.co/docs/transformers/main/chat_templating#advanced-tool-use--function-calling), 
+and note that Mistral **does** use tool call IDs, so these must be included in your tool calls and tool results. They should be
+exactly 9 alphanumeric characters.
+
+## Limitations
+
+The Mistral Large model is a quick demonstration that the base model can be easily fine-tuned to achieve compelling performance. 
+It does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to
+make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.
+
+## The Mistral AI Team
+
+Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Alok Kothari, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Augustin Garreau, Austin Birky, Bam4d, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Carole Rambaud, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Diogo Costa, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gaspard Blanchet, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Henri Roussez, Hichem Sattouf, Ian Mack, Jean-Malo Delignon, Jessica Chudnovsky, Justus Murke, Kartik Khandelwal, Lawrence Stewart, Louis Martin, Louis Ternon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Marjorie Janiewicz, Mickaël Seznec, Nicolas Schuhl, Niklas Muhs, Olivier de Garrigues, Patrick von Platen, Paul Jacob, Pauline Buche, Pavan Kumar Reddy, Perry Savas, Pierre Stock, Romain Sauvestre, Sagar Vaze, Sandeep Subramanian, Saurabh Garg, Sophia Yang, Szymon Antoniak, Teven Le Scao, Thibault Schueller, Thibaut Lavril, Thomas Wang, Théophile Gervet, Timothée Lacroix, Valera Nemychnikova, Wendy Shang, William El Sayed, William Marshall

config.json

@@ -0,0 +1,36 @@
+{
+    "architectures": [
+        "MistralForCausalLM"
+    ],
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "hidden_act": "silu",
+    "hidden_size": 12288,
+    "initializer_range": 0.02,
+    "intermediate_size": 28672,
+    "max_position_embeddings": 131072,
+    "model_type": "mistral",
+    "num_attention_heads": 96,
+    "num_hidden_layers": 88,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-05,
+    "rope_theta": 1000000.0,
+    "sliding_window": null,
+    "tie_word_embeddings": false,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.42.3",
+    "use_cache": true,
+    "vocab_size": 32768,
+    "quantization_config": {
+        "quant_method": "exl2",
+        "version": "0.2.1",
+        "bits": 2.85,
+        "head_bits": 6,
+        "calibration": {
+            "rows": 115,
+            "length": 2048,
+            "dataset": "(default)"
+        }
+    }
+}

config.yml

@@ -0,0 +1,228 @@
+# Sample YAML file for configuration.
+# Comment and uncomment values as needed. Every value has a default within the application.
+# This file serves to be a drop in for config.yml
+
+# Unless specified in the comments, DO NOT put these options in quotes!
+# You can use https://www.yamllint.com/ if you want to check your YAML formatting.
+
+# Options for networking
+network:
+  # The IP to host on (default: 127.0.0.1).
+  # Use 0.0.0.0 to expose on all network adapters
+  host: 0.0.0.0
+
+  # The port to host on (default: 5000)
+  port: 5000
+
+  # Disable HTTP token authenticaion with requests
+  # WARNING: This will make your instance vulnerable!
+  # Turn on this option if you are ONLY connecting from localhost
+  disable_auth: False
+
+  # Send tracebacks over the API to clients (default: False)
+  # NOTE: Only enable this for debug purposes
+  send_tracebacks: False
+
+  # Select API servers to enable (default: ["OAI"])
+  # Possible values: OAI
+  api_servers: ["OAI"]
+
+# Options for logging
+logging:
+  # Enable prompt logging (default: False)
+  prompt: False
+
+  # Enable generation parameter logging (default: False)
+  generation_params: False
+
+  # Enable request logging (default: False)
+  # NOTE: Only use this for debugging!
+  requests: False
+
+# Options for sampling
+sampling:
+  # Override preset name. Find this in the sampler-overrides folder (default: None)
+  # This overrides default fallbacks for sampler values that are passed to the API
+  # Server-side overrides are NOT needed by default
+  # WARNING: Using this can result in a generation speed penalty
+  #override_preset: 
+
+# Options for development and experimentation
+developer:
+  # Skips exllamav2 version check (default: False)
+  # It's highly recommended to update your dependencies rather than enabling this flag
+  # WARNING: Don't set this unless you know what you're doing!
+  #unsafe_launch: False
+
+  # Disable all request streaming (default: False)
+  # A kill switch for turning off SSE in the API server
+  #disable_request_streaming: False
+
+  # Enable the torch CUDA malloc backend (default: False)
+  # This can save a few MBs of VRAM, but has a risk of errors. Use at your own risk.
+  cuda_malloc_backend: True
+
+  # Enable Uvloop or Winloop (default: False)
+  # Make the program utilize a faster async event loop which can improve performance
+  # NOTE: It's recommended to enable this, but if something breaks, turn this off.
+  uvloop: True
+
+  # Set process to use a higher priority
+  # For realtime process priority, run as administrator or sudo
+  # Otherwise, the priority will be set to high
+  realtime_process_priority: True
+
+# Options for model overrides and loading
+# Please read the comments to understand how arguments are handled between initial and API loads
+model:
+  # Overrides the directory to look for models (default: models)
+  # Windows users, DO NOT put this path in quotes! This directory will be invalid otherwise.
+  model_dir: models
+
+  # Sends dummy model names when the models endpoint is queried
+  # Enable this if the program is looking for a specific OAI model
+  #use_dummy_models: False
+
+  # An initial model to load. Make sure the model is located in the model directory!
+  # A model can be loaded later via the API.
+  # REQUIRED: This must be filled out to load a model on startup!
+  model_name: Mistral-Large-Instruct-2407_exl2_2.85bpw
+
+  # The below parameters only apply for initial loads
+  # All API based loads do NOT inherit these settings unless specified in use_as_default
+
+  # Names of args to use as a default fallback for API load requests (default: [])
+  # For example, if you always want cache_mode to be Q4 instead of on the inital model load,
+  # Add "cache_mode" to this array
+  # Ex. ["max_seq_len", "cache_mode"]
+  #use_as_default: []
+
+  # The below parameters apply only if model_name is set
+
+  # Max sequence length (default: Empty)
+  # Fetched from the model's base sequence length in config.json by default
+  max_seq_len: 32768
+
+  # Overrides base model context length (default: Empty)
+  # WARNING: Don't set this unless you know what you're doing!
+  # Again, do NOT use this for configuring context length, use max_seq_len above ^
+  # Only use this if the model's base sequence length in config.json is incorrect (ex. Mistral 7B)
+  #override_base_seq_len:
+
+  # Load model with tensor parallelism
+  # If a GPU split isn't provided, the TP loader will fallback to autosplit
+  # Enabling ignores the gpu_split_auto and autosplit_reserve values
+  #tensor_parallel: True
+
+  # Automatically allocate resources to GPUs (default: True)
+  # NOTE: Not parsed for single GPU users
+  gpu_split_auto: True
+
+  # Reserve VRAM used for autosplit loading (default: 96 MB on GPU 0)
+  # This is represented as an array of MB per GPU used
+  autosplit_reserve: [0]
+
+  # An integer array of GBs of vram to split between GPUs (default: [])
+  # Used with tensor parallelism
+  # NOTE: Not parsed for single GPU users
+  #gpu_split: [20.6, 24]
+
+  # Rope scale (default: 1.0)
+  # Same thing as compress_pos_emb
+  # Only use if your model was trained on long context with rope (check config.json)
+  # Leave blank to pull the value from the model
+  #rope_scale: 1.0
+
+  # Rope alpha (default: 1.0)
+  # Same thing as alpha_value
+  # Leave blank to automatically calculate alpha
+  #rope_alpha: 1.0
+
+  # Enable different cache modes for VRAM savings (slight performance hit).
+  # Possible values FP16, Q8, Q6, Q4. (default: FP16)
+  cache_mode: Q4
+
+  # Size of the prompt cache to allocate (default: max_seq_len)
+  # This must be a multiple of 256. A larger cache uses more VRAM, but allows for more prompts to be processed at once.
+  # NOTE: Cache size should not be less than max_seq_len.
+  # For CFG, set this to 2 * max_seq_len to make room for both positive and negative prompts.
+  # cache_size:
+
+  # Chunk size for prompt ingestion. A lower value reduces VRAM usage at the cost of ingestion speed (default: 2048)
+  # NOTE: Effects vary depending on the model. An ideal value is between 512 and 4096
+  chunk_size: 1024
+
+  # Set the maximum amount of prompts to process at one time (default: None/Automatic)
+  # This will be automatically calculated if left blank.
+  # A max batch size of 1 processes prompts one at a time.
+  # NOTE: Only available for Nvidia ampere (30 series) and above GPUs
+  #max_batch_size:
+
+  # Set the prompt template for this model. If empty, attempts to look for the model's chat template. (default: None)
+  # If a model contains multiple templates in its tokenizer_config.json, set prompt_template to the name
+  # of the template you want to use.
+  # NOTE: Only works with chat completion message lists!
+  #prompt_template:
+
+  # Number of experts to use PER TOKEN. Fetched from the model's config.json if not specified (default: Empty)
+  # WARNING: Don't set this unless you know what you're doing!
+  # NOTE: For MoE models (ex. Mixtral) only!
+  #num_experts_per_token:
+
+  # Enables fasttensors to possibly increase model loading speeds (default: False)
+  fasttensors: true
+
+  # Options for draft models (speculative decoding). This will use more VRAM!
+  #draft:
+    # Overrides the directory to look for draft (default: models)
+    #draft_model_dir: models
+
+    # An initial draft model to load. Make sure this model is located in the model directory!
+    # A draft model can be loaded later via the API.
+    #draft_model_name: A model name
+  
+    # The below parameters only apply for initial loads
+    # All API based loads do NOT inherit these settings unless specified in use_as_default
+
+    # Rope scale for draft models (default: 1.0)
+    # Same thing as compress_pos_emb
+    # Only use if your draft model was trained on long context with rope (check config.json)
+    #draft_rope_scale: 1.0
+
+    # Rope alpha for draft model (default: 1.0)
+    # Same thing as alpha_value
+    # Leave blank to automatically calculate alpha value
+    #draft_rope_alpha: 1.0
+
+    # Enable different draft model cache modes for VRAM savings (slight performance hit).
+    # Possible values FP16, Q8, Q6, Q4. (default: FP16)
+    #draft_cache_mode: FP16
+  
+  # Options for loras
+  #lora:
+    # Overrides the directory to look for loras (default: loras)
+    #lora_dir: loras
+    
+    # List of loras to load and associated scaling factors (default: 1.0). Comment out unused entries or add more rows as needed.
+    #loras:
+    #- name: lora1
+    #  scaling: 1.0
+
+# Options for embedding models and loading.
+# NOTE: Embeddings requires the "extras" feature to be installed
+# Install it via "pip install .[extras]"
+embeddings:
+  # Overrides directory to look for embedding models (default: models)
+  embedding_model_dir: models
+
+  # Device to load embedding models on (default: cpu)
+  # Possible values: cpu, auto, cuda
+  # NOTE: It's recommended to load embedding models on the CPU.
+  # If you'd like to load on an AMD gpu, set this value to "cuda" as well.
+  embeddings_device: cpu
+
+  # The below parameters only apply for initial loads
+  # All API based loads do NOT inherit these settings unless specified in use_as_default
+
+  # An initial embedding model to load on the infinity backend (default: None)
+  embedding_model_name:

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "transformers_version": "4.42.3"
+}

model.safetensors.index.json

@@ -0,0 +1,802 @@
+{
+  "metadata": {
+    "total_size": 245220139008
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00051-of-00051.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00002-of-00051.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00002-of-00051.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00051.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00007-of-00051.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00007-of-00051.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00007-of-00051.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00007-of-00051.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00007-of-00051.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00008-of-00051.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00008-of-00051.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00009-of-00051.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00009-of-00051.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00008-of-00051.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00009-of-00051.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00009-of-00051.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00010-of-00051.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00010-of-00051.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00009-of-00051.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00010-of-00051.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00010-of-00051.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00011-of-00051.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00011-of-00051.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00010-of-00051.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00011-of-00051.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00011-of-00051.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00011-of-00051.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00012-of-00051.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00012-of-00051.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00002-of-00051.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00002-of-00051.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00013-of-00051.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00013-of-00051.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00012-of-00051.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00013-of-00051.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00013-of-00051.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00014-of-00051.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00014-of-00051.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00013-of-00051.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00014-of-00051.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00014-of-00051.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00015-of-00051.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00015-of-00051.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00014-of-00051.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00015-of-00051.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00015-of-00051.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00015-of-00051.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00016-of-00051.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00016-of-00051.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00017-of-00051.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00017-of-00051.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00016-of-00051.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00017-of-00051.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00017-of-00051.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00018-of-00051.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00018-of-00051.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00017-of-00051.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00003-of-00051.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00003-of-00051.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00002-of-00051.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00018-of-00051.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00018-of-00051.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00019-of-00051.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00019-of-00051.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00018-of-00051.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00019-of-00051.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00019-of-00051.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00019-of-00051.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00020-of-00051.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00020-of-00051.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00021-of-00051.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00021-of-00051.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00020-of-00051.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00021-of-00051.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00021-of-00051.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.36.input_layernorm.weight": "model-00022-of-00051.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00022-of-00051.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00021-of-00051.safetensors",
+    "model.layers.37.input_layernorm.weight": "model-00022-of-00051.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00022-of-00051.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.38.input_layernorm.weight": "model-00023-of-00051.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00023-of-00051.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00022-of-00051.safetensors",
+    "model.layers.39.input_layernorm.weight": "model-00023-of-00051.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00023-of-00051.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00023-of-00051.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00003-of-00051.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00003-of-00051.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00003-of-00051.safetensors",
+    "model.layers.40.input_layernorm.weight": "model-00024-of-00051.safetensors",
+    "model.layers.40.mlp.down_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.40.mlp.gate_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.40.mlp.up_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00024-of-00051.safetensors",
+    "model.layers.40.self_attn.k_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.40.self_attn.q_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.40.self_attn.v_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.41.input_layernorm.weight": "model-00025-of-00051.safetensors",
+    "model.layers.41.mlp.down_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.41.mlp.gate_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.41.mlp.up_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00025-of-00051.safetensors",
+    "model.layers.41.self_attn.k_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.41.self_attn.q_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.41.self_attn.v_proj.weight": "model-00024-of-00051.safetensors",
+    "model.layers.42.input_layernorm.weight": "model-00025-of-00051.safetensors",
+    "model.layers.42.mlp.down_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.42.mlp.gate_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.42.mlp.up_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00025-of-00051.safetensors",
+    "model.layers.42.self_attn.k_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.42.self_attn.q_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.42.self_attn.v_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.43.input_layernorm.weight": "model-00026-of-00051.safetensors",
+    "model.layers.43.mlp.down_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.43.mlp.gate_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.43.mlp.up_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00026-of-00051.safetensors",
+    "model.layers.43.self_attn.k_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.43.self_attn.q_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.43.self_attn.v_proj.weight": "model-00025-of-00051.safetensors",
+    "model.layers.44.input_layernorm.weight": "model-00026-of-00051.safetensors",
+    "model.layers.44.mlp.down_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.44.mlp.gate_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.44.mlp.up_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00026-of-00051.safetensors",
+    "model.layers.44.self_attn.k_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.44.self_attn.q_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.44.self_attn.v_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.45.input_layernorm.weight": "model-00027-of-00051.safetensors",
+    "model.layers.45.mlp.down_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.45.mlp.gate_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.45.mlp.up_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00027-of-00051.safetensors",
+    "model.layers.45.self_attn.k_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.45.self_attn.q_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.45.self_attn.v_proj.weight": "model-00026-of-00051.safetensors",
+    "model.layers.46.input_layernorm.weight": "model-00027-of-00051.safetensors",
+    "model.layers.46.mlp.down_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.46.mlp.gate_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.46.mlp.up_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00027-of-00051.safetensors",
+    "model.layers.46.self_attn.k_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.46.self_attn.q_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.46.self_attn.v_proj.weight": "model-00027-of-00051.safetensors",
+    "model.layers.47.input_layernorm.weight": "model-00028-of-00051.safetensors",
+    "model.layers.47.mlp.down_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.47.mlp.gate_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.47.mlp.up_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00028-of-00051.safetensors",
+    "model.layers.47.self_attn.k_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.47.self_attn.q_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.47.self_attn.v_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.48.input_layernorm.weight": "model-00029-of-00051.safetensors",
+    "model.layers.48.mlp.down_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.48.mlp.gate_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.48.mlp.up_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.48.post_attention_layernorm.weight": "model-00029-of-00051.safetensors",
+    "model.layers.48.self_attn.k_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.48.self_attn.o_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.48.self_attn.q_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.48.self_attn.v_proj.weight": "model-00028-of-00051.safetensors",
+    "model.layers.49.input_layernorm.weight": "model-00029-of-00051.safetensors",
+    "model.layers.49.mlp.down_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.49.mlp.gate_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.49.mlp.up_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.49.post_attention_layernorm.weight": "model-00029-of-00051.safetensors",
+    "model.layers.49.self_attn.k_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.49.self_attn.o_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.49.self_attn.q_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.49.self_attn.v_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00004-of-00051.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00004-of-00051.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.50.input_layernorm.weight": "model-00030-of-00051.safetensors",
+    "model.layers.50.mlp.down_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.50.mlp.gate_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.50.mlp.up_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.50.post_attention_layernorm.weight": "model-00030-of-00051.safetensors",
+    "model.layers.50.self_attn.k_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.50.self_attn.o_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.50.self_attn.q_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.50.self_attn.v_proj.weight": "model-00029-of-00051.safetensors",
+    "model.layers.51.input_layernorm.weight": "model-00030-of-00051.safetensors",
+    "model.layers.51.mlp.down_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.51.mlp.gate_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.51.mlp.up_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.51.post_attention_layernorm.weight": "model-00030-of-00051.safetensors",
+    "model.layers.51.self_attn.k_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.51.self_attn.o_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.51.self_attn.q_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.51.self_attn.v_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.52.input_layernorm.weight": "model-00031-of-00051.safetensors",
+    "model.layers.52.mlp.down_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.52.mlp.gate_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.52.mlp.up_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.52.post_attention_layernorm.weight": "model-00031-of-00051.safetensors",
+    "model.layers.52.self_attn.k_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.52.self_attn.o_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.52.self_attn.q_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.52.self_attn.v_proj.weight": "model-00030-of-00051.safetensors",
+    "model.layers.53.input_layernorm.weight": "model-00031-of-00051.safetensors",
+    "model.layers.53.mlp.down_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.53.mlp.gate_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.53.mlp.up_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.53.post_attention_layernorm.weight": "model-00031-of-00051.safetensors",
+    "model.layers.53.self_attn.k_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.53.self_attn.o_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.53.self_attn.q_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.53.self_attn.v_proj.weight": "model-00031-of-00051.safetensors",
+    "model.layers.54.input_layernorm.weight": "model-00032-of-00051.safetensors",
+    "model.layers.54.mlp.down_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.54.mlp.gate_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.54.mlp.up_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.54.post_attention_layernorm.weight": "model-00032-of-00051.safetensors",
+    "model.layers.54.self_attn.k_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.54.self_attn.o_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.54.self_attn.q_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.54.self_attn.v_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.55.input_layernorm.weight": "model-00033-of-00051.safetensors",
+    "model.layers.55.mlp.down_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.55.mlp.gate_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.55.mlp.up_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.55.post_attention_layernorm.weight": "model-00033-of-00051.safetensors",
+    "model.layers.55.self_attn.k_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.55.self_attn.o_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.55.self_attn.q_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.55.self_attn.v_proj.weight": "model-00032-of-00051.safetensors",
+    "model.layers.56.input_layernorm.weight": "model-00033-of-00051.safetensors",
+    "model.layers.56.mlp.down_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.56.mlp.gate_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.56.mlp.up_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.56.post_attention_layernorm.weight": "model-00033-of-00051.safetensors",
+    "model.layers.56.self_attn.k_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.56.self_attn.o_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.56.self_attn.q_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.56.self_attn.v_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.57.input_layernorm.weight": "model-00034-of-00051.safetensors",
+    "model.layers.57.mlp.down_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.57.mlp.gate_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.57.mlp.up_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.57.post_attention_layernorm.weight": "model-00034-of-00051.safetensors",
+    "model.layers.57.self_attn.k_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.57.self_attn.o_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.57.self_attn.q_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.57.self_attn.v_proj.weight": "model-00033-of-00051.safetensors",
+    "model.layers.58.input_layernorm.weight": "model-00034-of-00051.safetensors",
+    "model.layers.58.mlp.down_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.58.mlp.gate_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.58.mlp.up_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.58.post_attention_layernorm.weight": "model-00034-of-00051.safetensors",
+    "model.layers.58.self_attn.k_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.58.self_attn.o_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.58.self_attn.q_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.58.self_attn.v_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.59.input_layernorm.weight": "model-00035-of-00051.safetensors",
+    "model.layers.59.mlp.down_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.59.mlp.gate_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.59.mlp.up_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.59.post_attention_layernorm.weight": "model-00035-of-00051.safetensors",
+    "model.layers.59.self_attn.k_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.59.self_attn.o_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.59.self_attn.q_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.59.self_attn.v_proj.weight": "model-00034-of-00051.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00005-of-00051.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00005-of-00051.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00004-of-00051.safetensors",
+    "model.layers.60.input_layernorm.weight": "model-00035-of-00051.safetensors",
+    "model.layers.60.mlp.down_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.60.mlp.gate_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.60.mlp.up_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.60.post_attention_layernorm.weight": "model-00035-of-00051.safetensors",
+    "model.layers.60.self_attn.k_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.60.self_attn.o_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.60.self_attn.q_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.60.self_attn.v_proj.weight": "model-00035-of-00051.safetensors",
+    "model.layers.61.input_layernorm.weight": "model-00036-of-00051.safetensors",
+    "model.layers.61.mlp.down_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.61.mlp.gate_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.61.mlp.up_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.61.post_attention_layernorm.weight": "model-00036-of-00051.safetensors",
+    "model.layers.61.self_attn.k_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.61.self_attn.o_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.61.self_attn.q_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.61.self_attn.v_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.62.input_layernorm.weight": "model-00037-of-00051.safetensors",
+    "model.layers.62.mlp.down_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.62.mlp.gate_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.62.mlp.up_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.62.post_attention_layernorm.weight": "model-00037-of-00051.safetensors",
+    "model.layers.62.self_attn.k_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.62.self_attn.o_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.62.self_attn.q_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.62.self_attn.v_proj.weight": "model-00036-of-00051.safetensors",
+    "model.layers.63.input_layernorm.weight": "model-00037-of-00051.safetensors",
+    "model.layers.63.mlp.down_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.63.mlp.gate_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.63.mlp.up_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.63.post_attention_layernorm.weight": "model-00037-of-00051.safetensors",
+    "model.layers.63.self_attn.k_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.63.self_attn.o_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.63.self_attn.q_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.63.self_attn.v_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.64.input_layernorm.weight": "model-00038-of-00051.safetensors",
+    "model.layers.64.mlp.down_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.64.mlp.gate_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.64.mlp.up_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.64.post_attention_layernorm.weight": "model-00038-of-00051.safetensors",
+    "model.layers.64.self_attn.k_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.64.self_attn.o_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.64.self_attn.q_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.64.self_attn.v_proj.weight": "model-00037-of-00051.safetensors",
+    "model.layers.65.input_layernorm.weight": "model-00038-of-00051.safetensors",
+    "model.layers.65.mlp.down_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.65.mlp.gate_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.65.mlp.up_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.65.post_attention_layernorm.weight": "model-00038-of-00051.safetensors",
+    "model.layers.65.self_attn.k_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.65.self_attn.o_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.65.self_attn.q_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.65.self_attn.v_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.66.input_layernorm.weight": "model-00039-of-00051.safetensors",
+    "model.layers.66.mlp.down_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.66.mlp.gate_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.66.mlp.up_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.66.post_attention_layernorm.weight": "model-00039-of-00051.safetensors",
+    "model.layers.66.self_attn.k_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.66.self_attn.o_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.66.self_attn.q_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.66.self_attn.v_proj.weight": "model-00038-of-00051.safetensors",
+    "model.layers.67.input_layernorm.weight": "model-00039-of-00051.safetensors",
+    "model.layers.67.mlp.down_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.67.mlp.gate_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.67.mlp.up_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.67.post_attention_layernorm.weight": "model-00039-of-00051.safetensors",
+    "model.layers.67.self_attn.k_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.67.self_attn.o_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.67.self_attn.q_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.67.self_attn.v_proj.weight": "model-00039-of-00051.safetensors",
+    "model.layers.68.input_layernorm.weight": "model-00040-of-00051.safetensors",
+    "model.layers.68.mlp.down_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.68.mlp.gate_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.68.mlp.up_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.68.post_attention_layernorm.weight": "model-00040-of-00051.safetensors",
+    "model.layers.68.self_attn.k_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.68.self_attn.o_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.68.self_attn.q_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.68.self_attn.v_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.69.input_layernorm.weight": "model-00041-of-00051.safetensors",
+    "model.layers.69.mlp.down_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.69.mlp.gate_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.69.mlp.up_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.69.post_attention_layernorm.weight": "model-00041-of-00051.safetensors",
+    "model.layers.69.self_attn.k_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.69.self_attn.o_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.69.self_attn.q_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.69.self_attn.v_proj.weight": "model-00040-of-00051.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00005-of-00051.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00005-of-00051.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.70.input_layernorm.weight": "model-00041-of-00051.safetensors",
+    "model.layers.70.mlp.down_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.70.mlp.gate_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.70.mlp.up_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.70.post_attention_layernorm.weight": "model-00041-of-00051.safetensors",
+    "model.layers.70.self_attn.k_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.70.self_attn.o_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.70.self_attn.q_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.70.self_attn.v_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.71.input_layernorm.weight": "model-00042-of-00051.safetensors",
+    "model.layers.71.mlp.down_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.71.mlp.gate_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.71.mlp.up_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.71.post_attention_layernorm.weight": "model-00042-of-00051.safetensors",
+    "model.layers.71.self_attn.k_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.71.self_attn.o_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.71.self_attn.q_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.71.self_attn.v_proj.weight": "model-00041-of-00051.safetensors",
+    "model.layers.72.input_layernorm.weight": "model-00042-of-00051.safetensors",
+    "model.layers.72.mlp.down_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.72.mlp.gate_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.72.mlp.up_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.72.post_attention_layernorm.weight": "model-00042-of-00051.safetensors",
+    "model.layers.72.self_attn.k_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.72.self_attn.o_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.72.self_attn.q_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.72.self_attn.v_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.73.input_layernorm.weight": "model-00043-of-00051.safetensors",
+    "model.layers.73.mlp.down_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.73.mlp.gate_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.73.mlp.up_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.73.post_attention_layernorm.weight": "model-00043-of-00051.safetensors",
+    "model.layers.73.self_attn.k_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.73.self_attn.o_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.73.self_attn.q_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.73.self_attn.v_proj.weight": "model-00042-of-00051.safetensors",
+    "model.layers.74.input_layernorm.weight": "model-00043-of-00051.safetensors",
+    "model.layers.74.mlp.down_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.74.mlp.gate_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.74.mlp.up_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.74.post_attention_layernorm.weight": "model-00043-of-00051.safetensors",
+    "model.layers.74.self_attn.k_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.74.self_attn.o_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.74.self_attn.q_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.74.self_attn.v_proj.weight": "model-00043-of-00051.safetensors",
+    "model.layers.75.input_layernorm.weight": "model-00044-of-00051.safetensors",
+    "model.layers.75.mlp.down_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.75.mlp.gate_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.75.mlp.up_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.75.post_attention_layernorm.weight": "model-00044-of-00051.safetensors",
+    "model.layers.75.self_attn.k_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.75.self_attn.o_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.75.self_attn.q_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.75.self_attn.v_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.76.input_layernorm.weight": "model-00045-of-00051.safetensors",
+    "model.layers.76.mlp.down_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.76.mlp.gate_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.76.mlp.up_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.76.post_attention_layernorm.weight": "model-00045-of-00051.safetensors",
+    "model.layers.76.self_attn.k_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.76.self_attn.o_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.76.self_attn.q_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.76.self_attn.v_proj.weight": "model-00044-of-00051.safetensors",
+    "model.layers.77.input_layernorm.weight": "model-00045-of-00051.safetensors",
+    "model.layers.77.mlp.down_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.77.mlp.gate_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.77.mlp.up_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.77.post_attention_layernorm.weight": "model-00045-of-00051.safetensors",
+    "model.layers.77.self_attn.k_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.77.self_attn.o_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.77.self_attn.q_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.77.self_attn.v_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.78.input_layernorm.weight": "model-00046-of-00051.safetensors",
+    "model.layers.78.mlp.down_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.78.mlp.gate_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.78.mlp.up_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.78.post_attention_layernorm.weight": "model-00046-of-00051.safetensors",
+    "model.layers.78.self_attn.k_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.78.self_attn.o_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.78.self_attn.q_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.78.self_attn.v_proj.weight": "model-00045-of-00051.safetensors",
+    "model.layers.79.input_layernorm.weight": "model-00046-of-00051.safetensors",
+    "model.layers.79.mlp.down_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.79.mlp.gate_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.79.mlp.up_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.79.post_attention_layernorm.weight": "model-00046-of-00051.safetensors",
+    "model.layers.79.self_attn.k_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.79.self_attn.o_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.79.self_attn.q_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.79.self_attn.v_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00006-of-00051.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00006-of-00051.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00005-of-00051.safetensors",
+    "model.layers.80.input_layernorm.weight": "model-00047-of-00051.safetensors",
+    "model.layers.80.mlp.down_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.80.mlp.gate_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.80.mlp.up_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.80.post_attention_layernorm.weight": "model-00047-of-00051.safetensors",
+    "model.layers.80.self_attn.k_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.80.self_attn.o_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.80.self_attn.q_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.80.self_attn.v_proj.weight": "model-00046-of-00051.safetensors",
+    "model.layers.81.input_layernorm.weight": "model-00047-of-00051.safetensors",
+    "model.layers.81.mlp.down_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.81.mlp.gate_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.81.mlp.up_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.81.post_attention_layernorm.weight": "model-00047-of-00051.safetensors",
+    "model.layers.81.self_attn.k_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.81.self_attn.o_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.81.self_attn.q_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.81.self_attn.v_proj.weight": "model-00047-of-00051.safetensors",
+    "model.layers.82.input_layernorm.weight": "model-00048-of-00051.safetensors",
+    "model.layers.82.mlp.down_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.82.mlp.gate_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.82.mlp.up_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.82.post_attention_layernorm.weight": "model-00048-of-00051.safetensors",
+    "model.layers.82.self_attn.k_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.82.self_attn.o_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.82.self_attn.q_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.82.self_attn.v_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.83.input_layernorm.weight": "model-00049-of-00051.safetensors",
+    "model.layers.83.mlp.down_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.83.mlp.gate_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.83.mlp.up_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.83.post_attention_layernorm.weight": "model-00049-of-00051.safetensors",
+    "model.layers.83.self_attn.k_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.83.self_attn.o_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.83.self_attn.q_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.83.self_attn.v_proj.weight": "model-00048-of-00051.safetensors",
+    "model.layers.84.input_layernorm.weight": "model-00049-of-00051.safetensors",
+    "model.layers.84.mlp.down_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.84.mlp.gate_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.84.mlp.up_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.84.post_attention_layernorm.weight": "model-00049-of-00051.safetensors",
+    "model.layers.84.self_attn.k_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.84.self_attn.o_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.84.self_attn.q_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.84.self_attn.v_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.85.input_layernorm.weight": "model-00050-of-00051.safetensors",
+    "model.layers.85.mlp.down_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.85.mlp.gate_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.85.mlp.up_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.85.post_attention_layernorm.weight": "model-00050-of-00051.safetensors",
+    "model.layers.85.self_attn.k_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.85.self_attn.o_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.85.self_attn.q_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.85.self_attn.v_proj.weight": "model-00049-of-00051.safetensors",
+    "model.layers.86.input_layernorm.weight": "model-00050-of-00051.safetensors",
+    "model.layers.86.mlp.down_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.86.mlp.gate_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.86.mlp.up_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.86.post_attention_layernorm.weight": "model-00050-of-00051.safetensors",
+    "model.layers.86.self_attn.k_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.86.self_attn.o_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.86.self_attn.q_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.86.self_attn.v_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.87.input_layernorm.weight": "model-00051-of-00051.safetensors",
+    "model.layers.87.mlp.down_proj.weight": "model-00051-of-00051.safetensors",
+    "model.layers.87.mlp.gate_proj.weight": "model-00051-of-00051.safetensors",
+    "model.layers.87.mlp.up_proj.weight": "model-00051-of-00051.safetensors",
+    "model.layers.87.post_attention_layernorm.weight": "model-00051-of-00051.safetensors",
+    "model.layers.87.self_attn.k_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.87.self_attn.o_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.87.self_attn.q_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.87.self_attn.v_proj.weight": "model-00050-of-00051.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00006-of-00051.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00006-of-00051.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00006-of-00051.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00006-of-00051.safetensors",
+    "model.norm.weight": "model-00051-of-00051.safetensors"
+  }
+}

output-00001-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:904ebc593a37172a1859ba3ab4f1600f7cfd239059abf5340d42f8b5bb344e38
+size 8542303534

output-00002-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec9329d43017108b0f356fa95821ffb6905100bf1474c59601ccefa2d8efa7d9
+size 8444257302

output-00003-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8b6888f7e80666294ec77a48ddd01f5678fa8be0166a4e0e1ac2a6849cc0e47
+size 8477914022

output-00004-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93aace360a0720cbffcfe11818f5ed7ca566853b20e47b65091f73e384da5964
+size 8571883530

output-00005-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28912fa047378865f68eb97882411891bcbf8688a059713e54bf211a1cf354ea
+size 8506114580

output-00006-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81b778f7aa381cfc1344f832bf850cedfe24dd56dc8711842750eb11e75cf3a4
+size 1967327076

params.json

@@ -0,0 +1,11 @@
+{
+    "dim": 12288,
+    "n_layers": 88,
+    "head_dim": 128,
+    "hidden_dim": 28672,
+    "n_heads": 96,
+    "n_kv_heads": 8,
+    "norm_eps": 1e-05,
+    "vocab_size": 32768,
+    "rope_theta": 1000000.0
+}

test.py

@@ -0,0 +1,26 @@
+import json
+from typing import Dict
+
+from safetensors.torch import load_file, save_file
+from huggingface_hub import split_torch_state_dict_into_shards
+import torch
+import os
+
+def save_state_dict(state_dict: Dict[str, torch.Tensor], save_directory: str):
+    state_dict_split = split_torch_state_dict_into_shards(state_dict, filename_pattern='consolidated{suffix}.safetensors')
+    for filename, tensors in state_dict_split.filename_to_tensors.items():
+        shard = {tensor: state_dict[tensor] for tensor in tensors}
+        print("Saving", save_directory, filename)
+        save_file(shard, os.path.join(save_directory, filename))
+    if state_dict_split.is_sharded:
+        index = {
+            "metadata": state_dict_split.metadata,
+            "weight_map": state_dict_split.tensor_to_filename,
+        }
+        with open(os.path.join(save_directory, "consolidated.safetensors.index.json"), "w") as f:
+            f.write(json.dumps(index, indent=2))
+
+big_file = 'consolidated.safetensors'
+loaded = load_file(big_file)
+
+save_state_dict(loaded, save_directory=f'.')

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59f95e28944c062244741268596badc900df86c7f5ded05088d2da22a7379e06
+size 587583

upload.py

@@ -0,0 +1,45 @@
+from huggingface_hub import HfApi
+from pathlib import Path
+
+# Define the parameters for uploading
+repo_id = "DBMe/Mistral-Large-Instruct-2407-2.85bpw-h6-exl2"  # Replace with your actual repo ID
+folder_path = "/home/asusws-x570-ace/programs/tabbyAPI-new/models/Mistral-Large-Instruct-2407_exl2_2.85bpw/"  # Replace with your folder path
+repo_type = "model"  # Change to "model" or "space" if applicable
+revision = "main"  # Optional: specify the branch or use "main"
+private = False  # Set to True if the repository should be private
+allow_patterns = None  # Optional: specify patterns of files to include
+ignore_patterns = None  # Optional: specify patterns of files to exclude
+num_workers = 4  # Set based on your system; lower if your internet is unstable
+print_report = True  # Enable progress reporting
+print_report_every = 60  # Report frequency in seconds
+
+# Initialize the Hugging Face API client
+api = HfApi()
+
+# Function to upload the folder in a resumable manner
+def upload_resumable():
+    try:
+        print("Starting upload process...")
+
+        # Perform the upload with the provided parameters
+        api.upload_large_folder(
+            repo_id=repo_id,
+            folder_path=Path(folder_path),
+            repo_type=repo_type,
+            revision=revision,
+            private=private,
+            allow_patterns=allow_patterns,
+            ignore_patterns=ignore_patterns,
+            num_workers=num_workers,
+            print_report=print_report,
+            print_report_every=print_report_every,
+        )
+
+        print("Upload completed successfully!")
+
+    except Exception as e:
+        print(f"Upload interrupted due to error: {e}")
+        print("You can resume the upload by running the script again.")
+
+# Call the function to start the upload
+upload_resumable()