You should try Phi-3.
https://huggingface.co/microsoft/Phi-3-mini-4k-instruct

Un-censoring a model is the first pass at a broader goal of knowledge programming in these AI models; we can shortly provide simple inference level ways to model internal layers of a model into a "thought space" and be able to fine tune or adjust arbitrary information in and out. This has immense applications for industry with non-public data and limited training resources and this work is thoroughly saluted.

How much VRAM did it take to load then train on 32 examples for the 70b? I am willing to put a machine to work on 512 examples the original post had.

I had issues trying to reimplement the inference-time intervention in llama.cpp that were a lot like this. The intervention would prevent some refusals, but not others.

Here's the original code I was using, which was my attempt at a faithful translation: A = A - (A ⋅ R^) x R^

struct ggml_tensor * llm_build_ablation(
        struct ggml_context * ctx,
         struct ggml_tensor * cur,
         struct ggml_tensor * dir) {
    if (dir == nullptr) {
        return cur;
    }

    struct ggml_tensor * out;

    // [embd toks] x [embd dirs] -> [toks dirs]
    out = ggml_mul_mat(ctx, cur, dir);

    // [dirs toks] x [dirs embd] -> [toks embd]
    out = ggml_mul_mat(ctx, ggml_cont(ctx, ggml_transpose(ctx, out)),
                            ggml_cont(ctx, ggml_transpose(ctx, dir)));

    // [toks embd] -> [embd toks]
    out = ggml_scale(ctx, ggml_cont(ctx, ggml_transpose(ctx, out)), -1.0f);

    cur = ggml_add(ctx, cur, out);

    return cur;
}

I changed the code slightly to this: A = A + relu(-1.0 x (A ⋅ R^)) x R^, and the intervention works much better in all test cases. At 32 examples, it works in almost all cases. I haven't looked into how to translate it to the weight orthogonalization math.

struct ggml_tensor * llm_build_ablation(
        struct ggml_context * ctx,
         struct ggml_tensor * cur,
         struct ggml_tensor * dir) {
    if (dir == nullptr) {
        return cur;
    }

    struct ggml_tensor * out;

    // [embd toks] x [embd dirs] -> [toks dirs]
    out = ggml_mul_mat(ctx, cur, dir);

    // if scalar here is positive, becomes incoherent
    // if no relu, intervention does not prevent refusal
    // ???
    out = ggml_relu(ctx, ggml_scale(ctx, out, -1.0f));

    // [dirs toks] x [dirs embd] -> [toks embd]
    out = ggml_mul_mat(ctx, ggml_cont(ctx, ggml_transpose(ctx, out)),
                            ggml_cont(ctx, ggml_transpose(ctx, dir)));

    // [toks embd] -> [embd toks]
    out = ggml_cont(ctx, ggml_transpose(ctx, out));

    cur = ggml_add(ctx, cur, out);

    return cur;
}