Spaces:
Running
Running
Commit
·
f647865
1
Parent(s):
943319e
fix a few typos in intro
Browse files- src/index.html +2 -2
src/index.html
CHANGED
@@ -84,14 +84,14 @@
|
|
84 |
|
85 |
<aside>If you have questions or remarks open a discussion on the <a href="https://huggingface.co/spaces/nanotron/ultrascale-playbook/discussions?status=open&type=discussion">Community tab</a>!</aside>
|
86 |
|
87 |
-
<p>We'll
|
88 |
|
89 |
<aside>We are extremely thankful to the whole <a href="https://distill.pub/">distill.pub</a> team for creating
|
90 |
the template on which we based this blog post.</aside>
|
91 |
|
92 |
<p>The book is built on the following <strong>three general foundations</strong>:</p>
|
93 |
|
94 |
-
<p><strong>Quick intros on theory and concepts:</strong> before diving into code and experiments, we want to understand how each method works at a high level and what
|
95 |
<aside>Note that we're still missing Pipeline Parallelism in this widget. To be added as an exercise for the reader.</aside>
|
96 |
|
97 |
<div class="large-image-background-transparent">
|
|
|
84 |
|
85 |
<aside>If you have questions or remarks open a discussion on the <a href="https://huggingface.co/spaces/nanotron/ultrascale-playbook/discussions?status=open&type=discussion">Community tab</a>!</aside>
|
86 |
|
87 |
+
<p>We'll assume you have some simple basic knowledge about current LLM architecture and are roughtly familiar with how deep learning model are trained, but you can be generally new to distributed training. If needed, the basics of model training can be found in great courses found at <a href="https://www.deeplearning.ai">DeepLearning.ai</a> or on the <a href="https://pytorch.org/tutorials/beginner/basics/intro.html">PyTorch tutorial sections</a>. This book can be seen as the second part of a trilogy following our first blog on processing data for pre-training, the so-called “<a href="https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1">FineWeb blog post</a>”. Having read both blog posts, you should have almost all the core knowledge needed to fully understand how how performing LLMs are being built nowadays, just missing some final spices regarding data mixing and architecture choices to complete the recipe (stay tuned for part three…).</p>
|
88 |
|
89 |
<aside>We are extremely thankful to the whole <a href="https://distill.pub/">distill.pub</a> team for creating
|
90 |
the template on which we based this blog post.</aside>
|
91 |
|
92 |
<p>The book is built on the following <strong>three general foundations</strong>:</p>
|
93 |
|
94 |
+
<p><strong>Quick intros on theory and concepts:</strong> before diving into code and experiments, we want to understand how each method works at a high level and what its advantages and limits are. You’ll learn about which parts of a language model eat away your memory and when during training it happens. You’ll learn how we can solve memory constraints by parallelizing the models and increase the throughput by scaling up GPUs. As a result you'll understand how the following widget to compute the memory breakdown of a transformer model works: </p>
|
95 |
<aside>Note that we're still missing Pipeline Parallelism in this widget. To be added as an exercise for the reader.</aside>
|
96 |
|
97 |
<div class="large-image-background-transparent">
|