Weight diff visualization
#4
by
yumemio
- opened
As discussed in this Reddit thread, the original model was apparently based on Llama 3 (not Llama 3.1).
Hugely appreciate it if anyone could run StableFluffy's weight diff plot script on this weight.
yumemio
changed discussion title from
Anyone willing to do the weight diff visualization?
to Weight diff visualization
I ran the diff on the first ten layers:
Llama-3
Llama-3.1
It seems like there's some tiny difference in the layernorms based off the color (which isn't shown due to the numerical precision of the graph), although I believe that's due to the fp16 -> fp32 and bf16 -> fp32 conversion I did. It does seem to indeed be a LoRA trained on Llama-3.1-70B-Instruct, as stated in the model card.
@leafspark You're awesome - much appreciated!