Add details on building a pipeline
Browse files- description2.md +22 -0
description2.md
CHANGED
@@ -1,3 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
## Alternative schedules
|
2 |
|
3 |
By utilizing the building block, we can search for different types of schedules depending on the need. We illustrate few of them here below:
|
|
|
1 |
+
## Building a pipeline
|
2 |
+
|
3 |
+
### Building Block
|
4 |
+
It starts by laying out the passes for a single microbatch, which we call a building
|
5 |
+
block. For example, the building block of 1F1B schedule is made of a sequence of forward passes
|
6 |
+
followed by backward passes in the reverse order.
|
7 |
+
|
8 |
+
### Repeating
|
9 |
+
More microbatches are then introduced. The building blocks are repeated and woven
|
10 |
+
together to form a pipeline. Notably, legit building blocks are required to repeat without a collision, namely, the
|
11 |
+
passes from two building blocks should not overlap with each other.
|
12 |
+
|
13 |
+
### Squeezing
|
14 |
+
Depending on the building block, there may be redundant bubbles in the pipeline, which
|
15 |
+
can be simply removed by squeezing without changing the order of the passes.
|
16 |
+
|
17 |
+
### Reordering (optional)
|
18 |
+
We can reorder the passes in the warm-up and cool-down phase to further
|
19 |
+
improve the computation throughput. Intuitively, the peak of memory happens in the stable phase of
|
20 |
+
the pipeline, while in the warm-up and cool-down phases the RAM is under utilized, leaving some
|
21 |
+
space for improving the computation throughput without changing peak memory.
|
22 |
+
|
23 |
## Alternative schedules
|
24 |
|
25 |
By utilizing the building block, we can search for different types of schedules depending on the need. We illustrate few of them here below:
|