}} Books 21GB
- Acadamic Journals 101GB
## Pretrained models are data-hungry {auto-animate=true}
训练GPT-3使用了大约0.75TB的文本数据
这样的训练量在如今看来并不算多
```{=html}
{{< include components/gpt.qmd >}}
```
## Pretrained models are data-hungry {auto-animate=true}
训练GPT-3使用了大约0.75TB的文本数据
这样的训练量在如今看来并不算多
::: {style="text-align:center;"}
![](./figures/2024-Alan-D-Thompson-AI-Bubbles-Planets-Rev-6.png){width=80%}
:::
## {auto-animate=true background-video="./figures/tokyo-walk.mp4"}
## Dawning of the World Model Era {auto-animate=true .smaller background-video="./figures/tokyo-walk.mp4" background-opacity=0.25}
How many data SORA uses?
:::: columns
::: {.column width="50%"}
> We take inspiration from large language models which acquire generalist capabilities by training on **internet-scale** data [^1]
:::
::: {.column width="50%"}
> 一个可供对比的数据量是:每分钟上传至 YouTube 的视频是 500h 的量级。则近五年的 YouTube 上的视频数据总量为:13亿小时 = 788亿分钟 。由于Diffusion模型训练text to video 需要高质量的标注视频,因此我们可以估计Sora 训练的视频量级为1亿分钟左右。
>
> 目前有一个比较准确的估计, 一分钟视频约为 1M tokens 。[^2]
:::
::::
::: {.notes}
- 1.3 word ~= 1 token
- 参考 Diffusion Transformer, 256x256 的图片会被划分为 32x32 个 patch。 我们假设 1920x1080 分辨率的高清图像经过下采样得到 512x256 大小的图片,假设一个 patch 为 8x8 的像素块,则得到 64x32 大小的 patch 矩阵, 一张图片则约为: 64x32=2048 个 patch。 高清视频 1s 约为 30 帧以上,但实际训练和推理也会做压缩,我们估计压缩后 1s 约为 9 帧。 则一分钟共 540 帧。 一分钟的视频一共有:64x32x540=1.1M
昆仑万维
:::
## Dawning of the World Model Era {auto-animate=true background-video="./figures/tokyo-walk.mp4" background-opacity=0.25}
:::: columns
::: {.column width="50%"}
> 一个可供对比的数据量是:每分钟上传至 YouTube 的视频是 500h 的量级。则近五年的 YouTube 上的视频数据总量为:13亿小时 = 788亿分钟 。由于Diffusion模型训练text to video 需要高质量的标注视频,因此我们可以估计Sora 训练的视频量级为1亿分钟左右。
:::
::: {.column width="50%"}
::: {.r-fit-text}
~[500TB]{style="background-color:#e31c54; color:white;"} trained data
~[500PB]{style="background-color:#e31c54; color:white;"} raw data
:::
:::
::::
::: {.notes}
以一分钟高清视频大小5MB计算,1亿分钟的视频数据量为500TB,而筛选岀这高质量的1亿分钟可能需要500PB的原始数据。
:::
## Dawning of the World Model Era {auto-animate=true background-video="./figures/tokyo-walk.mp4" background-opacity=0.25}
```{=html}
{{< include components/token-bar.qmd >}}
```
## Challenge {auto-animate=true background-video="./figures/tokyo-walk.mp4" background-opacity=0.25}
:::: columns
::: {.column width="50%"}
::: {.r-fit-text}
Training on
[internet-]{.flow}
scale data
:::
:::
::: {.column width="45%"}
::: {.r-fit-text}
Modeling
[ultra-long]{.flow}
sequence
:::
:::
::::
[^1]: [Video generation models as world simulators(SORA tech report)](https://openai.com/research/video-generation-models-as-world-simulators)
[^2]: [浅谈 Sora 未来的千倍推理算力需求](https://zhuanlan.zhihu.com/p/683636677)
# {.theme-section}
::: {.title}
Streaming Data Flow
:::
## Legacy training paradigm {auto-animate=true .smaller}
传统的训练方式通常是一次性将数据下载到本地,然后进行处理。
```{.python code-line-numbers="5-8|11-14"}
{{< include ./scripts/hf.py >}}
```
1. 下载数据集
2. 将数据集处理为模型输入,并保存到本地
3. 准备训练
::: notes
- 将全部数据下载到共享存储空间
- 一次性将数据处理为模型接受的形式
:::
## Legacy training paradigm {auto-animate=true}
:::: columns
::: {.column width="40%"}
![](./figures/etl-explain-large2.webp){width=90%}
:::
::: {.column width="60%" .fragment}
这种范式下ETL与模型训练完全串行,是一种简单明了的方式。
```{=html}
{{< include ./components/profile-old.qmd >}}
```
:::
::::
## What's the Problem? {auto-animate=true}
:::: columns
::: {.column width="60%"}
![](./figures/etl-ai.jpg)
:::
::: {.column width="40%"}
多模态大模型的ETL流程正变得越来越复杂
- E: 数据模态多,来源复杂,拉取时间长
- T: 数据处理流程复杂
- L: 存储占用高
:::
::::
## What's the Problem? {auto-animate=true}
多模态数据由于版权和存储原因,大多以下载链接的形式分发,获取速率受到限制
```{=html}
{{< include components/webvid.qmd >}}
```
::: {style="text-align:center; font-size: 0.4em;"}
webvid以url形式提供, 共包括10730233条数据
:::
::: {.notes}
- 这意味着国内需要使用昂贵的国际带宽来获取数据,对于一个小型数据中心, 下载相当于Sora训练量的数据集可能需要花费数年的时间。
- 即便只下载webvid这样中等规模的数据,下载和处理的时间可能也是训练的瓶颈.
:::
## What's the Problem? {auto-animate=true}
处理流程复杂耗时,甚至超过训练开销
:::: columns
::: {.column width="60%"}
::: {style="margin-top: 50px;"}
![](./figures/caption.jpg)
:::
:::
::: {.column width="40%"}
GPT-4V (20s/it)
::: {style="font-size: 0.4em;"}
An aerial video sweeps over a majestic ocean cliff with striatedrock formations in rich hues of red, green, and orange.The sun'srays enhance the colorful palette of the landscape, while the sea'sazure watersgently crash aqainst the clif's base. Visible are thetextured detais ofthe clif face and the contrast of the areeralaae and seaqrasscoating parts of the rock.Seabirds can beseen flying close to the water around the rocky outcrop. Thevideo conveys a serene yet dynamic coastal scenery, highlightingthe natural beauty and geological diversity of a rugged coastline
:::
🌋 **LLaVA-1.6-Yi-34B (3s/it)**
::: {style="font-size: 0.4em;"}
A breathtakina aerial view of a rocky cliff iutting out into theocean.The cliff. adorned with vibrant qreen moss and patches ofyellow and red lichen, is bathed in the warm glow of the sun. Theocean, a deep blue, is dotted with small white waves that crashagainst the cliff's base. The perspective of the video is fromabove, providing a bird's eye view of the cliff and the surroundingocean.The sun's rays illuminate the scene, casting a beautifullight on the cliff and the ocean. The video is a stunningrepresentation of the raw beauty of nature.
:::
:::
::::
## What's the Problem? {auto-animate=true}
> [Storage]{.red} plays an important role in AI training, and yet is one of the least talked-about aspects. As the GenAI training jobs become more multimodal over time, consuming large amounts of [ image, video, and text data ]{.red}, the need for data storage grows rapidly. [^llama3]
- 要从原始数据中筛选出一亿分钟数据,可能意味着原始数据量高达数十PB以上
- 对于一般的小型数据中心,没有能力搭建适应视频预训练的存储设施。
[^llama3]: [Building Meta’s GenAI Infrastructure](https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/)
## What's the Problem? {auto-animate=true}
:::: columns
::: {.column width="50%"}
![](./figures/etl-problem.webp)
:::
::: {.column width="50%"}
::: {.incremental}
- 数据来源复杂
- 数据源不能立即被拉取
- 数据处理流程复杂
- 数据处理和模型训练耦合
- 数据量过大,无法一次性处理
- ...
:::
:::
::::
## What's the Problem? {auto-animate=true}
:::: columns
::: {.column width="50%"}
- 数据流离模型训练越来越远
- 仍然使用传统的方式处理数据,
数据流将成为阻塞训练的瓶颈。
::: {style="margin-left: 54px;"}
![](./figures/etl-explain-small.webp){width=80%}
:::
:::
::: {.column width="50%"}
```{=html}
{{< include ./components/profile-naive.qmd >}}
```
:::
::::
## {auto-animate=true}
::: {.r-fit-text}
How to train
on [internet-scale]{.flow}
data?
:::
## {auto-animate=true}
:::: columns
::: {.column width="50%"}
::: {.r-fit-text}
How to train
on [internet-scale]{.flow}
data?
:::
:::
::: {.column width="45%"}
::: {.r-fit-text .fragment}
Just training on
[the internet]{.flow}!
:::
:::
::::
## Streaming to the rescue {auto-animate=true}
:::: columns
::: {.column width="50%"}
![](./figures/streaming.gif)
::: {.incremental}
- 流式传输数据可以解决这些问题
- 但流式数据传输只是一个开始,我们需要构建完全基于流式数据的训练框架
:::
:::
::: {.column width="50%"}
::: {.fragment}
![](./figures/decoup-data.png)
:::
:::
::::
## Streaming to the rescue {auto-animate=true}
![](./figures/plat.png){width=80%}
## Streaming to the rescue {auto-animate=true .smaller}
:::: columns
::: {.column width="60%"}
![](./figures/plat.png){width=80%}
:::
::: {.column width="40%"}
::: {.incremental}
- [x] 零启动开销
- [x] 数据处理进程和模型训练进程完全分离
- [x] 节点内通过`SharedMemory`通信, 节点间通过内存数据库通信
- [x] 数据处理集群拓扑与GPU拓扑无关, 可以动态调整
- [x] 定时sink数据库,允许回溯数据流
- [x] 确定性的数据切分和洗牌算法,确保回溯的一致性
:::
:::
::::
## {auto-animate=true background="./figures/mosaicml-streaming-dataset-img-1.gif"}
::: {.notes}
每个云上shard内的样本具备确定性的切分和洗牌算法,确保回溯的一致性, 并与训练拓扑无关
:::
## Training on the internet {auto-animate=true .smaller background="./figures/mosaicml-streaming-dataset-img-1.gif" background-opacity=0.25}
使用S3作为数据和权重的存储后端, 无缝进行不同规模的云迁移
```{=html}
{{< include components/cloud-switch.qmd >}}
```
## Training on the internet {auto-animate=true .smaller}
引入DPU集群,允许将数据直接传输到GPU, 消除内存数据库的开销
![](./figures/dpu.png){width=100%}
Powered by
::: {.notes}
与中立云服务商UCloud合作
:::
## Training on the internet {auto-animate=true .smaller}
![](./figures/streaming-data.webp){width=80%}
## Training on the internet {auto-animate=true .smaller}
:::: columns
::: {.column width="50%"}
![](./figures/streaming-data.webp){width=60%}
:::
::: {.column width="50%"}
- 进一步分离了数据处理和模型训练
- 使ETL与模型训练完全并行
:::
::::
::: {.fragment}
```{=html}
{{< include components/profile-stream.qmd >}}
```
:::
# {.theme-section}
::: {.title}
Scaling Exact Attention
:::
## Efficient distributed training infra {auto-animate="true"}
| | Flash-Attn-2 | FP8 (H100) | 3D Parallel + Zero | Padding Free | Fused Kernel | Static Graph | TGS[^l] |
|------------:|:------------:|:----------:|:------------------:|:------------:|:------------:|:------------:|:---:|
| Platformers | ✔️ | ✔️ | ✔️ | ✔️ | [100%]{style="color:red;"} | ✔️ | [3743]{style="color:red;"} |
| Megatron-LM | ✖️ | ✔️ | ✔️ | ✖️ | 80% | ✖️ | 3581 |
| Deepspeed | ✔️ | ✖️ | ✔️ | ✖️ | 60% | ✖️ |✖️ |
| Colossal-ai | ✖️ | ✖️ | ✔️ | ✖️ | 40% | ✖️ | 2610 |
[^l]: Training LLaMA2 7b on DGX (8*A100 40GB) with 4096 sequence Length
## Scaling exact attention to ultra long sequence {auto-animate="true"}
![](./figures/context_parallel.svg)
## Scaling exact attention to ultra long sequence {auto-animate="true"}
![](./figures/computation_reduce.svg){width=80%}
## Scaling exact attention to ultra long sequence {auto-animate="true"}
:::: columns
::: {.column width="50%"}
```{=html}
{{< include ./components/seq-time.qmd >}}
```
:::
::: {.column width="50%"}
```{=html}
{{< include ./components/seq-tflops.qmd >}}
```
:::
::::
## Scaling exact attention to ultra long sequence {auto-animate="true"}
```{=html}
{{< include mocha.qmd >}}
```
# {.theme-end}
::: columns
::: {.column width="50%"}
::: {.r-fit-text}
Thanks
:::
:::
::: {.column width="25%"}
::: {style="text-align:center;"}
![wechat](./figures/qr/code.png)
:::
:::
::: {.column width="25%"}
::: {style="text-align:center;"}
![e-mail](./figures/qr/mail-data.png)
:::
:::
:::