Update README.md
Browse files
README.md
CHANGED
|
@@ -46,6 +46,8 @@ The training was done for 2 epochs. We used 8x [AMD Instinct™ MI300X Accelerat
|
|
| 46 |
We also trained with a weight decay of 0.01 to help further stabilize the loss trajectory and mitigate catastrophic forgetting, and utilize a peak learning rate of 4e-6 to prevent the 2nd epoch loss from dropping too significantly (as it is a strong indicator of overfitting).
|
| 47 |
|
| 48 |
|
|
|
|
|
|
|
| 49 |
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
| 50 |
|
| 51 |
## Safety
|
|
|
|
| 46 |
We also trained with a weight decay of 0.01 to help further stabilize the loss trajectory and mitigate catastrophic forgetting, and utilize a peak learning rate of 4e-6 to prevent the 2nd epoch loss from dropping too significantly (as it is a strong indicator of overfitting).
|
| 47 |
|
| 48 |
|
| 49 |
+
Sample Packing was done for 16k tokens rather than the 8k tokens used in our previous runs.
|
| 50 |
+
|
| 51 |
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
|
| 52 |
|
| 53 |
## Safety
|