Upload folder using huggingface_hub

README.md

@@ -12,21 +12,13 @@ language:
 - 'no'
 - nn
 - is
-base_model: LumiOpen/Viking-7B
 ---
-NOTE THIS IS BASED ON THE 1000B VERSION! I will update this soon.
-
-This is an ExLlamaV2 quantized model in 4bpw of [LumiOpen/Viking-7B](https://huggingface.co/LumiOpen/Viking-7B) using the default calibration dataset.
-
-# Original Model card:
 
 # Viking 7B
 
-**NOTE: We are aware of an incompatibility with HF transformers that impacts finetuning and are working to correct it.**
-
-_**NOTE:** This is a **research checkpoint** of a model for which **training has not been completed.** It is being provided in its current state for research and testing purposes. **Care should be taken when using the outputs of the model.** Once pretraining has completed we intend to release additional instruction-tuned and chat-tuned varieties._
-
-Viking 7B is a 7B parameter decoder-only transformer pretrained on Finnish, English, Swedish, Danish, Norwegian, Icelandic and code.  It is being trained on 2 trillion tokens (1 trillion as of this release). Viking 7B is a fully open source model and is made available under the Apache 2.0 License.
+Viking 7B is a 7B parameter decoder-only transformer pretrained on Finnish,
+English, Swedish, Danish, Norwegian, Icelandic and code. It has been trained
+on 2 trillion tokens. Viking 7B is a fully open source model and is made available under the Apache 2.0 License.
 
 Viking was created in a collaboration between the [TurkuNLP group](https://turkunlp.org/) of the University of Turku, [SiloGen](https://www.silo.ai/silogen) from [Silo AI](https://www.silo.ai/),and [High Performance Language Technologies](https://hplt-project.org/) (HPLT). Training was conducted on the [LUMI supercomputer](https://www.lumi-supercomputer.eu/), using compute resources generously provided by [CSC](https://csc.fi/) - IT Center for Science, Finland.
 
@@ -45,7 +37,7 @@ Viking is the second set of models released by LumiOpen and is available at
 
 
 ## Model Overview
-_**NOTE:** In addition to being an early research release, Viking is a base model which needs further fine tuning for most use cases._
+_**NOTE:** Viking is a base model which needs further fine tuning for most use cases._
 
 Viking is a generative pretrained transformer using a LLaMA-like GPT architecture, and makes use of rotary positional embeddings and flash attention.
 
@@ -103,14 +95,24 @@ Training Checkpoints are available as branches in the repository.  Checkpoints w
 * [800B](https://huggingface.co/LumiOpen/Viking-7B/tree/800B)
 * [900B](https://huggingface.co/LumiOpen/Viking-7B/tree/900B)
 * [1000B](https://huggingface.co/LumiOpen/Viking-7B/tree/1000B)
+* [1100B](https://huggingface.co/LumiOpen/Viking-7B/tree/1100B)
+* [1200B](https://huggingface.co/LumiOpen/Viking-7B/tree/1200B)
+* [1300B](https://huggingface.co/LumiOpen/Viking-7B/tree/1300B)
+* [1400B](https://huggingface.co/LumiOpen/Viking-7B/tree/1400B)
+* [1500B](https://huggingface.co/LumiOpen/Viking-7B/tree/1500B)
+* [1600B](https://huggingface.co/LumiOpen/Viking-7B/tree/1600B)
+* [1700B](https://huggingface.co/LumiOpen/Viking-7B/tree/1700B)
+* [1800B](https://huggingface.co/LumiOpen/Viking-7B/tree/1800B)
+* [1900B](https://huggingface.co/LumiOpen/Viking-7B/tree/1900B)
+* [2000B](https://huggingface.co/LumiOpen/Viking-7B/tree/2000B)
 
 The transformers library allows you to load a checkpoint from a branch as follows:
 
 ```python
-branch = "200B"
+branch = "2000B"
 model = transformers.AutoModelForCausalLM.from_pretrained(
     "LumiOpen/Viking-7B",
-    torch_dtype=torch.bfloat16,
+    torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
     revision=branch,
 )
 ```

cal_data.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:040e73fd812b7abb24844b6a5bd261114780b01ea3c1cf908f009e3734e35c2c
+size 1884248

config.json

@@ -1,40 +1,29 @@
 {
-    "_name_or_path": "/scratch/project_462000319/general-tools/viking_v2_checkpoints/viking_v2_7B_iter_0239000_bfloat16",
-    "architectures": [
-        "LlamaForCausalLM"
-    ],
-    "attention_bias": false,
-    "attention_dropout": 0.0,
-    "bos_token_id": 1,
-    "eos_token_id": 2,
-    "hidden_act": "silu",
-    "hidden_size": 4096,
-    "initializer_range": 0.02,
-    "intermediate_size": 11008,
-    "max_position_embeddings": 4096,
-    "model_type": "llama",
-    "num_attention_heads": 32,
-    "num_hidden_layers": 32,
-    "num_key_value_heads": 32,
-    "pretraining_tp": 1,
-    "rms_norm_eps": 1e-05,
-    "rope_scaling": null,
-    "rope_theta": 10000.0,
-    "tie_word_embeddings": false,
-    "torch_dtype": "bfloat16",
-    "transformers_version": "4.37.2",
-    "untie_embeddings_and_output_weights": true,
-    "use_cache": true,
-    "vocab_size": 131072,
-    "quantization_config": {
-        "quant_method": "exl2",
-        "version": "0.0.17",
-        "bits": 4.0,
-        "head_bits": 6,
-        "calibration": {
-            "rows": 100,
-            "length": 2048,
-            "dataset": "(default)"
-        }
-    }
-}
+  "_name_or_path": "/scratch/project_462000319/general-tools/viking_v2_checkpoints/viking_v2_7B_iter_0476837_bfloat16",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 11008,
+  "max_position_embeddings": 4096,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 32,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.40.0",
+  "untie_embeddings_and_output_weights": true,
+  "use_cache": true,
+  "vocab_size": 131072
+}

generation_config.json

@@ -2,5 +2,5 @@
   "_from_model_config": true,
   "bos_token_id": 1,
   "eos_token_id": 2,
-  "transformers_version": "4.37.2"
+  "transformers_version": "4.40.0"
 }

hidden_states.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:401a35eac0cac36c6086067383005a154c330ed7ce6b4185d0a24be43056fe2f
+size 1929389952

out_tensor/lm_head.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0298b05a794f4d85a43f1e76a2170d4c0e15833c86d8c1df133db2a712b4471e
+size 425738872

out_tensor/model.layers.0.mlp.down_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90211f69276cd8e0a7fde44f40f425050db1ace7caf42c87e02f237aca409ffe
+size 24563824

out_tensor/model.layers.0.mlp.gate_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b78a448d824f34eb79f324507c4aa1134ef6e116d72ea37225c5affa50d6855c
+size 23442136

out_tensor/model.layers.0.mlp.up_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3b52d9a3b9a597101a46160f4d555b6671407d02e08fc3c3749e990bde3ed6e
+size 24146632

out_tensor/model.layers.0.self_attn.k_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45f16f4dfa35cb9969ed3476624d9a900d7263edaacc87ec0b585bc906023e8c
+size 8766368

out_tensor/model.layers.0.self_attn.o_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83500b7f6aeaef36abe4139bd6fbcdf1e3dc80ee2d9dc389b407288b4ff4cd1f
+size 8766368

out_tensor/model.layers.0.self_attn.q_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6aacfee4be69b35559fdee27d2563412433f4bd6af70ef9fca56a428a58dc8d
+size 8766368

out_tensor/model.layers.0.self_attn.v_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b48d2280c745ef92eaaf5640fad72c101613b88aa0effe0b67b239583b76923c
+size 8881440

out_tensor/model.layers.1.mlp.down_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c36a73be58b884d6b0d406b7aaa9061066a189e41522232cc70e0bb9669ab97
+size 25048616

out_tensor/model.layers.1.mlp.gate_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8e59b26e8bff000d92534b0168d779a45c953ebf740754c1e8c5c4f5a57f0e6
+size 23839000

out_tensor/model.layers.1.mlp.up_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86d591b8804a110148bea485d33a4f14fd00dd5616a064e9487f7fd3b6eda22b
+size 24675600

out_tensor/model.layers.1.self_attn.k_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:644f2496a3bbf754352920cf4063bd1622110e953455ab2b694492b476e61852
+size 10978600

out_tensor/model.layers.1.self_attn.o_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79001786ac13e199ffa842803e99318471d109b83f92534eb256405c93798120
+size 10978600

out_tensor/model.layers.1.self_attn.q_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b44ebdcf5968f4c8334fe04af1da3c89443d08fdd8ae4f3a02987b8a2be32fc5
+size 10978600

out_tensor/model.layers.1.self_attn.v_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b7c1da24826c5dc0ac466dd4cfcdcbb5e37d0348900068b13f17b16030bd337
+size 12862760

out_tensor/model.layers.10.mlp.down_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9708845897428da99c5fef71c7c6c26df2f45b18d3e5ebc675fbae7c05b2c389
+size 25048624

out_tensor/model.layers.10.mlp.gate_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4de879a37787ee49721c8d711acc333dc5af83ce05ce1845340384004f4b09d
+size 23839008

out_tensor/model.layers.10.mlp.up_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0598ac643c5984df4f5cd2012f6682b5e5a4266b40d7ba3b4e70b69b0338723
+size 24675600

out_tensor/model.layers.10.self_attn.k_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5306ec8dab319218fb9663b4dfc3d46cd232d256bdfc66f10a97adf490c19bc6
+size 4473760

out_tensor/model.layers.10.self_attn.o_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f235b7633698a0f273ffa6292d1a3dc3ecb952b51cc85435139aded45e4b6184
+size 4473760

out_tensor/model.layers.10.self_attn.q_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d52f703a09df4833c8ed0d46ee0b0734e5658acc6632854eecf03a55e30bc385
+size 4473760

out_tensor/model.layers.10.self_attn.v_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:875ee32a475329c7c8ee6b6f061e9b4aeb3fc7b8797b8f479e2691b77d0962c5
+size 4473760

out_tensor/model.layers.11.mlp.down_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:816fb8f05ebe65dc252c5db573746ed97560d7b27e4e1ed00b3478502e063900
+size 23974008

out_tensor/model.layers.11.mlp.gate_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ee36f3604157d6319cc2c1155d37197d4c0372933b1785c8b703a6d1b624499
+size 22737624

out_tensor/model.layers.11.mlp.up_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17f2d21eaebab90f7e87fa6854b813232c7aa650eb1a0ac70c8f3884966950e6
+size 23266576

out_tensor/model.layers.11.self_attn.k_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b86d4f247f57932c3ebfe7a6e5b7a15698c8fa07b2b0bc4f829f22b868352082
+size 4473760

out_tensor/model.layers.11.self_attn.o_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:560fa25340c3ace64f3253dfb9b37c3794e7cdc382e404501e0b2a309cf0be52
+size 4473760

out_tensor/model.layers.11.self_attn.q_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc589f35a5a0d4b27bec00d59411d083eb44d1ca29f6b63a4f718ad5e1115c91
+size 4473760

out_tensor/model.layers.11.self_attn.v_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a3c1faba688ddeb2cadad78527fe4a40622b644edaaffb5c0476994ecd2c733
+size 4473760

out_tensor/model.layers.12.mlp.down_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c418a01203a0de4484f6bbcd1f99a2bf70fee86f77d104d0fd32ced80065e95
+size 25048624

out_tensor/model.layers.12.mlp.gate_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6a3459d8f824d99d1778702bf3047cc8a575b3bab0654abf568c3fd05195f6f
+size 23839008

out_tensor/model.layers.12.mlp.up_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb113bce78227454218f9914588e80e2cbc1848be8a14324b9d0115208203277
+size 24675600

out_tensor/model.layers.12.self_attn.k_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d951d64cb90e44dee8dfb7f8d1c653a27dd7403201dac344517a674e9647981e
+size 4572064

out_tensor/model.layers.12.self_attn.o_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f62db4a948c5fd7760961fae13d383c08092168bc1d2f6cbf3732b5c66361b9e
+size 4572064

out_tensor/model.layers.12.self_attn.q_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc060607b1ce6f66bed8c62019675e2ffb2b2b3adc430f9d05fe4bcdd50fdc74
+size 4572064

out_tensor/model.layers.12.self_attn.v_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b6f93b0d13c2aced2c2d46913b55f1bf99e6aa81c4119fb64d0c4186900be36d
+size 4866976

out_tensor/model.layers.13.mlp.down_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:447d6fa976edbbca31377be95fa8a65079bfeaea11b1868e549777edacf20f0b
+size 24475184

out_tensor/model.layers.13.mlp.gate_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f6849a263686ea0234eb0f10b308ff5d1e55aa96f8b80469b91c23447333a20
+size 23266592

out_tensor/model.layers.13.mlp.up_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01e8c831384957f8b7e54dc8ed7f8a1432bbf2b5dd97a67a291f440615a5e4ca
+size 23266576

out_tensor/model.layers.13.self_attn.k_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:daea163599f614595640da330768bde7deaf2a45387367319490ff4fab5af2ba
+size 4572064

out_tensor/model.layers.13.self_attn.o_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f04238d500245e59e8b375c97fbef3cc7b52d9774c7d764d60b80627cb010362
+size 4572064

out_tensor/model.layers.13.self_attn.q_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf46fd0b1dbdc5d7fa1fcb89214475faf6ca1f543f821b58d8a7341d166b8a50
+size 4572064

out_tensor/model.layers.13.self_attn.v_proj.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bed739b8f640b7f51c31ae778f3229678e7a49b84bc14f4f2b5d041aed52015b
+size 6636256