jukofyork/miqu-control-vectors_ALPHA-0.1

NOTE: See creative-writing-control-vectors-v3.0 for the current main control-vector repo.

---

NOTE: This is a very early test so don't be disheartened if it doesn't work that well:

• This uses a very conservative threshold for deciding which directions to keep so as to lessen the chance of buggering up the output
• I've also not filtered the layer range and miqu-1-70b-sf__darkness_control_vector.gguf looks to be changing a lot of the final layers.

Overall I'm pretty confident this will eventually work well and soon will be able to add other "dimensions" akin to the D&D alignment system, etc.

---

Initializing GGUFWriter with path: 'miqu-1-70b-sf__positivity_control_vector.gguf' and architecture: 'controlvector'
gguf: This GGUF file is for Little Endian only
- Adding model hint: 'llama'
- Adding layer count: '5'
-- Processing layer: 20 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 30 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 31 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 55 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 66 with tensor of shape: torch.Size([1, 8192])
Export completed

Initializing GGUFWriter with path: 'miqu-1-70b-sf__darkness_control_vector.gguf ' and architecture: 'controlvector'
gguf: This GGUF file is for Little Endian only
- Adding model hint: 'llama'
- Adding layer count: '30'
-- Processing layer: 19 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 22 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 42 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 43 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 44 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 45 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 50 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 51 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 52 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 53 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 54 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 55 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 56 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 57 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 58 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 59 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 60 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 61 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 63 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 66 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 68 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 69 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 70 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 71 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 73 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 74 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 75 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 76 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 78 with tensor of shape: torch.Size([1, 8192])
-- Processing layer: 79 with tensor of shape: torch.Size([1, 8192])
Export completed

---

You need to run llama.cpp with these commands added to use them:

--control-vector-scaled miqu-1-70b-sf__positivity_control_vector.gguf -1.0 --control-vector-scaled miqu-1-70b-sf__darkness_control_vector.gguf 1.0

The -1.0 and 1.0 are reasonable defaults, but you might want to try larger or smaller values, and possibly not use one or other of the control-vectors - just have a play and see the effect.

Here is my full command for running this using the original leaked miqu-1:70b file:

./llama-server --n-gpu-layers 1000 --ctx_size 32764 --flash-attn --model miqu-1:70b-q5_K_M.gguf --chat-template llama2 --control-vector-scaled miqu-1-70b-sf__positivity_control_vector.gguf -1.0 --control-vector-scaled miqu-1-70b-sf__darkness_control_vector.gguf 1.0

You can try using '--chat-template vicuna' too.

---

It might also be worth testing the effect of reducing the layer range:

         --control-vector FNAME   add a control vector
         --control-vector-scaled FNAME SCALE
                                  add a control vector with user defined scaling SCALE
         --control-vector-layer-range START END
                                  layer range to apply the control vector(s) to, start and end inclusive

eg:

--control-vector-layer-range 0 70 --control-vector-scaled miqu-1-70b-sf__darkness_control_vector.gguf 1.0

Will ensure the last 10 layers don't get the "darkness" control vector applied (this will effect all control vectors though; AFAIK there is no way to set the range for individual control vectors?)