Youliang commited on
Commit
56164c2
1 Parent(s): 0c266cb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +22 -1
README.md CHANGED
@@ -17,7 +17,7 @@ This model is a fine-tuned version of [meta-llama/Meta-Llama-3-70B](https://hugg
17
 
18
  ## Model description
19
 
20
- More information needed
21
 
22
  ## Intended uses & limitations
23
 
@@ -45,6 +45,27 @@ The following hyperparameters were used during training:
45
  - lr_scheduler_type: cosine
46
  - num_epochs: 2.0
47
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
48
  ### Training results
49
 
50
 
 
17
 
18
  ## Model description
19
 
20
+ Please refer to the paper [Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training](https://arxiv.org/abs/2407.09121) and GitHub [DeRTa](https://github.com/RobustNLP/DeRTa).
21
 
22
  ## Intended uses & limitations
23
 
 
45
  - lr_scheduler_type: cosine
46
  - num_epochs: 2.0
47
 
48
+
49
+ The lora config is:
50
+ ```
51
+ {
52
+ "lora_r": 96,
53
+ "lora_alpha": 16,
54
+ "lora_dropout": 0.05,
55
+ "lora_target_modules": [
56
+ "q_proj",
57
+ "v_proj",
58
+ "k_proj",
59
+ "o_proj",
60
+ "gate_proj",
61
+ "down_proj",
62
+ "up_proj",
63
+ "w1",
64
+ "w2",
65
+ "w3"
66
+ ]
67
+ }
68
+ ```
69
  ### Training results
70
 
71