Training Code

by Gatozu35 - opened Mar 9

Discussion

Gatozu35

Mar 9

Hello, is there code available for the training of prosparse models?

Raincleared

SparseLLMs org Mar 10

The training of ProSparse is based on BMTrain and an unreleased version of CPM-Live. Actually, our training paradigm is similar to general pre-training of LLaMA, except the ReLU activation and the progressive $L_{1}$ regularization loss on the FFN intermediate outputs (i.e., the output x in this line).

Gatozu35

Mar 10

Thank you for your reply! From this, I assume you are not going to release the code anytime soon. I appreciate the guidance though!

Gatozu35 changed discussion status to closed Mar 10

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment