Building a pipeline
Building Block
It starts by laying out the passes for a single microbatch, which we call a building block. For example, the building block of 1F1B schedule is made of a sequence of forward passes followed by backward passes in the reverse order.
Repeating
More microbatches are then introduced. The building blocks are repeated and woven together to form a pipeline. Notably, legit building blocks are required to repeat without a collision, namely, the passes from two building blocks should not overlap with each other.
Squeezing
Depending on the building block, there may be redundant bubbles in the pipeline, which can be simply removed by squeezing without changing the order of the passes.
Reordering (optional)
We can reorder the passes in the warm-up and cool-down phase to further improve the computation throughput. Intuitively, the peak of memory happens in the stable phase of the pipeline, while in the warm-up and cool-down phases the RAM is under utilized, leaving some space for improving the computation throughput without changing peak memory.
Alternative schedules
By utilizing the building block, we can search for different types of schedules depending on the need. We illustrate few of them here below:
- 1F1B-V schedule without doing any B-W split.
- Schedule with 2/3rd 1F1B memory by utilising B-W split. Note that two microbatches are included in a single building block to avoid collision.
- Variation of interleaved 1F1B with lower memory