Workinggs
Browse files- index.html +5 -5
- llm_conf.qmd +5 -5
- llm_conf_files/libs/revealjs/dist/theme/quarto.css +1 -1
- style.css +2 -1
index.html
CHANGED
|
@@ -426,7 +426,7 @@
|
|
| 426 |
<li>Backward ~= 2x the model size</li>
|
| 427 |
<li>The optimizer step ~= 4x the model size (1x model, 1x gradients, 2x optimizer):</li>
|
| 428 |
</ul>
|
| 429 |
-
<div style="font-size: 50%;
|
| 430 |
<table>
|
| 431 |
<thead>
|
| 432 |
<tr class="header">
|
|
@@ -465,7 +465,7 @@
|
|
| 465 |
<p>This works fine for small models, we have cards with anywhere from 12-24GB of GPU memory (on the GPU-poor side).</p>
|
| 466 |
<p>But what happens as we scale?</p>
|
| 467 |
<p>Here’s <code>llama-3-8B</code> (8.03B parameters)</p>
|
| 468 |
-
<div style="font-size: 50%;
|
| 469 |
<table>
|
| 470 |
<thead>
|
| 471 |
<tr class="header">
|
|
@@ -698,7 +698,7 @@
|
|
| 698 |
<li>Rely on <code>config.yaml</code> files</li>
|
| 699 |
<li>Choose to either running <code>accelerate config</code> or write your own:</li>
|
| 700 |
</ul>
|
| 701 |
-
<div class="columns" style="font-size: 50%;padding-left:10%;
|
| 702 |
<div class="column" style="width:40%;">
|
| 703 |
<div class="code-with-filename">
|
| 704 |
<div class="code-with-filename-file">
|
|
@@ -804,7 +804,7 @@
|
|
| 804 |
<ul>
|
| 805 |
<li>Let’s tie that back up to the model estimator with neat tools like NVIDIA’s TransformerEngine</li>
|
| 806 |
</ul>
|
| 807 |
-
<div style="font-size: 60%;
|
| 808 |
<table style="width:100%;">
|
| 809 |
<colgroup>
|
| 810 |
<col style="width: 14%">
|
|
@@ -894,7 +894,7 @@
|
|
| 894 |
<ul>
|
| 895 |
<li>Extremely similar, however mostly used different naming conventions for items and slight tweaks in the implementation</li>
|
| 896 |
</ul>
|
| 897 |
-
<div style="font-size: 50%;
|
| 898 |
<table style="width:100%;">
|
| 899 |
<colgroup>
|
| 900 |
<col style="width: 16%">
|
|
|
|
| 426 |
<li>Backward ~= 2x the model size</li>
|
| 427 |
<li>The optimizer step ~= 4x the model size (1x model, 1x gradients, 2x optimizer):</li>
|
| 428 |
</ul>
|
| 429 |
+
<div style="font-size: 50%;">
|
| 430 |
<table>
|
| 431 |
<thead>
|
| 432 |
<tr class="header">
|
|
|
|
| 465 |
<p>This works fine for small models, we have cards with anywhere from 12-24GB of GPU memory (on the GPU-poor side).</p>
|
| 466 |
<p>But what happens as we scale?</p>
|
| 467 |
<p>Here’s <code>llama-3-8B</code> (8.03B parameters)</p>
|
| 468 |
+
<div style="font-size: 50%;">
|
| 469 |
<table>
|
| 470 |
<thead>
|
| 471 |
<tr class="header">
|
|
|
|
| 698 |
<li>Rely on <code>config.yaml</code> files</li>
|
| 699 |
<li>Choose to either running <code>accelerate config</code> or write your own:</li>
|
| 700 |
</ul>
|
| 701 |
+
<div class="columns" style="font-size: 50%;padding-left:10%;">
|
| 702 |
<div class="column" style="width:40%;">
|
| 703 |
<div class="code-with-filename">
|
| 704 |
<div class="code-with-filename-file">
|
|
|
|
| 804 |
<ul>
|
| 805 |
<li>Let’s tie that back up to the model estimator with neat tools like NVIDIA’s TransformerEngine</li>
|
| 806 |
</ul>
|
| 807 |
+
<div style="font-size: 60%;">
|
| 808 |
<table style="width:100%;">
|
| 809 |
<colgroup>
|
| 810 |
<col style="width: 14%">
|
|
|
|
| 894 |
<ul>
|
| 895 |
<li>Extremely similar, however mostly used different naming conventions for items and slight tweaks in the implementation</li>
|
| 896 |
</ul>
|
| 897 |
+
<div style="font-size: 50%;">
|
| 898 |
<table style="width:100%;">
|
| 899 |
<colgroup>
|
| 900 |
<col style="width: 16%">
|
llm_conf.qmd
CHANGED
|
@@ -28,7 +28,7 @@ General estimate (`bert-base-cased`, 108M params):
|
|
| 28 |
- Backward ~= 2x the model size
|
| 29 |
- The optimizer step ~= 4x the model size (1x model, 1x gradients, 2x optimizer):
|
| 30 |
|
| 31 |
-
::: {style="font-size: 50%;
|
| 32 |
| dtype | Model | Gradients | Backward pass | Optimizer step | Highest |
|
| 33 |
|---------|:-----|:------:|:------:|:------:|:------:|
|
| 34 |
| float32 | 413.18 MB | 413.18 MB | 826.36 MB | 1.61 GB | 1.61 GB |
|
|
@@ -45,7 +45,7 @@ But what happens as we scale?
|
|
| 45 |
|
| 46 |
Here's `llama-3-8B` (8.03B parameters)
|
| 47 |
|
| 48 |
-
::: {style="font-size: 50%;
|
| 49 |
| dtype | Model | Gradients | Backward pass | Optimizer step | Highest |
|
| 50 |
|---------|:-----|:------:|:------:|:------:|:------:|
|
| 51 |
| float32 | 28.21 GB | 28.21 GB | 56.43 GB | 112.84 GB | 112.84 GB |
|
|
@@ -202,7 +202,7 @@ accelerate launch script.py
|
|
| 202 |
* Rely on `config.yaml` files
|
| 203 |
* Choose to either running `accelerate config` or write your own:
|
| 204 |
|
| 205 |
-
:::: {.columns style="font-size: 50%;padding-left:10%;
|
| 206 |
::: {.column width="40%"}
|
| 207 |
```{.yaml filename=ddp_config.yaml}
|
| 208 |
compute_environment: LOCAL_MACHINE
|
|
@@ -302,7 +302,7 @@ for batch in dataloader:
|
|
| 302 |
|
| 303 |
* Let's tie that back up to the model estimator with neat tools like NVIDIA's TransformerEngine
|
| 304 |
|
| 305 |
-
::: {style="font-size: 60%;
|
| 306 |
| Optimization Level | Computation (GEMM) | Comm | Weight | Master Weight | Weight Gradient | Optimizer States |
|
| 307 |
| -- | -- | -- | -- | -- | -- | -- |
|
| 308 |
| FP16 AMP | FP16 | FP32 | FP32 | N/A | FP32 | FP32+FP32 |
|
|
@@ -326,7 +326,7 @@ What is actually happening:
|
|
| 326 |
|
| 327 |
* Extremely similar, however mostly used different naming conventions for items and slight tweaks in the implementation
|
| 328 |
|
| 329 |
-
::: {style="font-size: 50%;
|
| 330 |
Framework | Model Loading (`torch_dtype`) | Mixed Precision | Preparation (Local) | Training | Optimizer (Local)
|
| 331 |
--|--|--|--|--|--
|
| 332 |
FSDP | bf16 | default (none) | bf16 | bf16 | bf16
|
|
|
|
| 28 |
- Backward ~= 2x the model size
|
| 29 |
- The optimizer step ~= 4x the model size (1x model, 1x gradients, 2x optimizer):
|
| 30 |
|
| 31 |
+
::: {style="font-size: 50%;"}
|
| 32 |
| dtype | Model | Gradients | Backward pass | Optimizer step | Highest |
|
| 33 |
|---------|:-----|:------:|:------:|:------:|:------:|
|
| 34 |
| float32 | 413.18 MB | 413.18 MB | 826.36 MB | 1.61 GB | 1.61 GB |
|
|
|
|
| 45 |
|
| 46 |
Here's `llama-3-8B` (8.03B parameters)
|
| 47 |
|
| 48 |
+
::: {style="font-size: 50%;"}
|
| 49 |
| dtype | Model | Gradients | Backward pass | Optimizer step | Highest |
|
| 50 |
|---------|:-----|:------:|:------:|:------:|:------:|
|
| 51 |
| float32 | 28.21 GB | 28.21 GB | 56.43 GB | 112.84 GB | 112.84 GB |
|
|
|
|
| 202 |
* Rely on `config.yaml` files
|
| 203 |
* Choose to either running `accelerate config` or write your own:
|
| 204 |
|
| 205 |
+
:::: {.columns style="font-size: 50%;padding-left:10%;"}
|
| 206 |
::: {.column width="40%"}
|
| 207 |
```{.yaml filename=ddp_config.yaml}
|
| 208 |
compute_environment: LOCAL_MACHINE
|
|
|
|
| 302 |
|
| 303 |
* Let's tie that back up to the model estimator with neat tools like NVIDIA's TransformerEngine
|
| 304 |
|
| 305 |
+
::: {style="font-size: 60%;"}
|
| 306 |
| Optimization Level | Computation (GEMM) | Comm | Weight | Master Weight | Weight Gradient | Optimizer States |
|
| 307 |
| -- | -- | -- | -- | -- | -- | -- |
|
| 308 |
| FP16 AMP | FP16 | FP32 | FP32 | N/A | FP32 | FP32+FP32 |
|
|
|
|
| 326 |
|
| 327 |
* Extremely similar, however mostly used different naming conventions for items and slight tweaks in the implementation
|
| 328 |
|
| 329 |
+
::: {style="font-size: 50%;"}
|
| 330 |
Framework | Model Loading (`torch_dtype`) | Mixed Precision | Preparation (Local) | Training | Optimizer (Local)
|
| 331 |
--|--|--|--|--|--
|
| 332 |
FSDP | bf16 | default (none) | bf16 | bf16 | bf16
|
llm_conf_files/libs/revealjs/dist/theme/quarto.css
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
@import"./fonts/league-gothic/league-gothic.css";@import"https://fonts.googleapis.com/css?family=Lato:400,700,400italic,700italic";@import"./fonts/source-sans-pro/source-sans-pro.css";table{color:#
|
| 2 |
*
|
| 3 |
* ansi colors from IPython notebook's
|
| 4 |
*
|
|
|
|
| 1 |
+
@import"./fonts/league-gothic/league-gothic.css";@import"https://fonts.googleapis.com/css?family=Lato:400,700,400italic,700italic";@import"./fonts/source-sans-pro/source-sans-pro.css";table{color:#93a1a1;background-color:rgba(0,0,0,.1)}.card{max-width:620px;margin:0 auto;padding:16px;border:1px solid #d3d3d3;border-radius:16px}.card p:last-child{margin-bottom:0}:root{--r-background-color: #002b36;--r-main-font: Lato, sans-serif;--r-main-font-size: 40px;--r-main-color: #93a1a1;--r-block-margin: 12px;--r-heading-margin: 0 0 12px 0;--r-heading-font: League Gothic, sans-serif;--r-heading-color: #eee8d5;--r-heading-line-height: 1.2;--r-heading-letter-spacing: normal;--r-heading-text-transform: uppercase;--r-heading-text-shadow: none;--r-heading-font-weight: 600;--r-heading1-text-shadow: none;--r-heading1-size: 2.5em;--r-heading2-size: 1.6em;--r-heading3-size: 1.3em;--r-heading4-size: 1em;--r-code-font: SFMono-Regular, Menlo, Monaco, Consolas, Liberation Mono, Courier New, monospace;--r-link-color: #268bd2;--r-link-color-dark: #1a6091;--r-link-color-hover: #4ca2df;--r-selection-background-color: #d33682;--r-selection-color: #002b36}.reveal-viewport{background:#002b36;background-color:var(--r-background-color)}.reveal{font-family:var(--r-main-font);font-size:var(--r-main-font-size);font-weight:normal;color:var(--r-main-color)}.reveal ::selection{color:var(--r-selection-color);background:var(--r-selection-background-color);text-shadow:none}.reveal ::-moz-selection{color:var(--r-selection-color);background:var(--r-selection-background-color);text-shadow:none}.reveal .slides section,.reveal .slides section>section{line-height:1.3;font-weight:inherit}.reveal h1,.reveal h2,.reveal h3,.reveal h4,.reveal h5,.reveal h6{margin:var(--r-heading-margin);color:var(--r-heading-color);font-family:var(--r-heading-font);font-weight:var(--r-heading-font-weight);line-height:var(--r-heading-line-height);letter-spacing:var(--r-heading-letter-spacing);text-transform:var(--r-heading-text-transform);text-shadow:var(--r-heading-text-shadow);word-wrap:break-word}.reveal h1{font-size:var(--r-heading1-size)}.reveal h2{font-size:var(--r-heading2-size)}.reveal h3{font-size:var(--r-heading3-size)}.reveal h4{font-size:var(--r-heading4-size)}.reveal h1{text-shadow:var(--r-heading1-text-shadow)}.reveal p{margin:var(--r-block-margin) 0;line-height:1.3}.reveal h1:last-child,.reveal h2:last-child,.reveal h3:last-child,.reveal h4:last-child,.reveal h5:last-child,.reveal h6:last-child{margin-bottom:0}.reveal img,.reveal video,.reveal iframe{max-width:95%;max-height:95%}.reveal strong,.reveal b{font-weight:bold}.reveal em{font-style:italic}.reveal ol,.reveal dl,.reveal ul{display:inline-block;text-align:left;margin:0 0 0 1em}.reveal ol{list-style-type:decimal}.reveal ul{list-style-type:disc}.reveal ul ul{list-style-type:square}.reveal ul ul ul{list-style-type:circle}.reveal ul ul,.reveal ul ol,.reveal ol ol,.reveal ol ul{display:block;margin-left:40px}.reveal dt{font-weight:bold}.reveal dd{margin-left:40px}.reveal blockquote{display:block;position:relative;width:70%;margin:var(--r-block-margin) auto;padding:5px;font-style:italic;background:rgba(255,255,255,.05);box-shadow:0px 0px 2px rgba(0,0,0,.2)}.reveal blockquote p:first-child,.reveal blockquote p:last-child{display:inline-block}.reveal q{font-style:italic}.reveal pre{display:block;position:relative;width:90%;margin:var(--r-block-margin) auto;text-align:left;font-size:.55em;font-family:var(--r-code-font);line-height:1.2em;word-wrap:break-word;box-shadow:0px 5px 15px rgba(0,0,0,.15)}.reveal code{font-family:var(--r-code-font);text-transform:none;tab-size:2}.reveal pre code{display:block;padding:5px;overflow:auto;max-height:400px;word-wrap:normal}.reveal .code-wrapper{white-space:normal}.reveal .code-wrapper code{white-space:pre}.reveal table{margin:auto;border-collapse:collapse;border-spacing:0}.reveal table th{font-weight:bold}.reveal table th,.reveal table td{text-align:left;padding:.2em .5em .2em .5em;border-bottom:1px solid}.reveal table th[align=center],.reveal table td[align=center]{text-align:center}.reveal table th[align=right],.reveal table td[align=right]{text-align:right}.reveal table tbody tr:last-child th,.reveal table tbody tr:last-child td{border-bottom:none}.reveal sup{vertical-align:super;font-size:smaller}.reveal sub{vertical-align:sub;font-size:smaller}.reveal small{display:inline-block;font-size:.6em;line-height:1.2em;vertical-align:top}.reveal small *{vertical-align:top}.reveal img{margin:var(--r-block-margin) 0}.reveal a{color:var(--r-link-color);text-decoration:none;transition:color .15s ease}.reveal a:hover{color:var(--r-link-color-hover);text-shadow:none;border:none}.reveal .roll span:after{color:#fff;background:var(--r-link-color-dark)}.reveal .r-frame{border:4px solid var(--r-main-color);box-shadow:0 0 10px rgba(0,0,0,.15)}.reveal a .r-frame{transition:all .15s linear}.reveal a:hover .r-frame{border-color:var(--r-link-color);box-shadow:0 0 20px rgba(0,0,0,.55)}.reveal .controls{color:var(--r-link-color)}.reveal .progress{background:rgba(0,0,0,.2);color:var(--r-link-color)}@media print{.backgrounds{background-color:var(--r-background-color)}}.top-right{position:absolute;top:1em;right:1em}.hidden{display:none !important}.zindex-bottom{z-index:-1 !important}figure.figure{display:block}.quarto-layout-panel{margin-bottom:1em}.quarto-layout-panel>figure{width:100%}.quarto-layout-panel>figure>figcaption,.quarto-layout-panel>.panel-caption{margin-top:10pt}.quarto-layout-panel>.table-caption{margin-top:0px}.table-caption p{margin-bottom:.5em}.quarto-layout-row{display:flex;flex-direction:row;align-items:flex-start}.quarto-layout-valign-top{align-items:flex-start}.quarto-layout-valign-bottom{align-items:flex-end}.quarto-layout-valign-center{align-items:center}.quarto-layout-cell{position:relative;margin-right:20px}.quarto-layout-cell:last-child{margin-right:0}.quarto-layout-cell figure,.quarto-layout-cell>p{margin:.2em}.quarto-layout-cell img{max-width:100%}.quarto-layout-cell .html-widget{width:100% !important}.quarto-layout-cell div figure p{margin:0}.quarto-layout-cell figure{display:block;margin-inline-start:0;margin-inline-end:0}.quarto-layout-cell table{display:inline-table}.quarto-layout-cell-subref figcaption,figure .quarto-layout-row figure figcaption{text-align:center;font-style:italic}.quarto-figure{position:relative;margin-bottom:1em}.quarto-figure>figure{width:100%;margin-bottom:0}.quarto-figure-left>figure>p,.quarto-figure-left>figure>div{text-align:left}.quarto-figure-center>figure>p,.quarto-figure-center>figure>div{text-align:center}.quarto-figure-right>figure>p,.quarto-figure-right>figure>div{text-align:right}.quarto-figure>figure>div.cell-annotation,.quarto-figure>figure>div code{text-align:left}figure>p:empty{display:none}figure>p:first-child{margin-top:0;margin-bottom:0}figure>figcaption{margin-top:.5em}figure.quarto-float-lst>figcaption{margin-bottom:.5em}div[id^=tbl-]{position:relative}.quarto-figure>.anchorjs-link{position:absolute;top:.6em;right:.5em}div[id^=tbl-]>.anchorjs-link{position:absolute;top:.7em;right:.3em}.quarto-figure:hover>.anchorjs-link,div[id^=tbl-]:hover>.anchorjs-link,h2:hover>.anchorjs-link,h3:hover>.anchorjs-link,h4:hover>.anchorjs-link,h5:hover>.anchorjs-link,h6:hover>.anchorjs-link,.reveal-anchorjs-link>.anchorjs-link{opacity:1}#title-block-header{margin-block-end:1rem;position:relative;margin-top:-1px}#title-block-header .abstract{margin-block-start:1rem}#title-block-header .abstract .abstract-title{font-weight:600}#title-block-header a{text-decoration:none}#title-block-header .author,#title-block-header .date,#title-block-header .doi{margin-block-end:.2rem}#title-block-header .quarto-title-block>div{display:flex}#title-block-header .quarto-title-block>div>h1{flex-grow:1}#title-block-header .quarto-title-block>div>button{flex-shrink:0;height:2.25rem;margin-top:0}tr.header>th>p:last-of-type{margin-bottom:0px}table,.table{caption-side:top;margin-bottom:1.5rem}figure.quarto-float-tbl figcaption,caption,.table-caption{padding-top:.5rem;padding-bottom:.5rem;text-align:center}.utterances{max-width:none;margin-left:-8px}iframe{margin-bottom:1em}details{margin-bottom:1em}details[show]{margin-bottom:0}details>summary{color:#93a1a1}details>summary>p:only-child{display:inline}pre.sourceCode,code.sourceCode{position:relative}p code:not(.sourceCode){white-space:pre-wrap}code{white-space:pre}@media print{code{white-space:pre-wrap}}pre>code{display:block}pre>code.sourceCode{white-space:pre}pre>code.sourceCode>span>a:first-child::before{text-decoration:none}pre.code-overflow-wrap>code.sourceCode{white-space:pre-wrap}pre.code-overflow-scroll>code.sourceCode{white-space:pre}code a:any-link{color:inherit;text-decoration:none}code a:hover{color:inherit;text-decoration:underline}ul.task-list{padding-left:1em}[data-tippy-root]{display:inline-block}.tippy-content .footnote-back{display:none}.tippy-content{overflow-x:scroll}.quarto-embedded-source-code{display:none}.quarto-unresolved-ref{font-weight:600}.quarto-cover-image{max-width:35%;float:right;margin-left:30px}.cell-output-display .widget-subarea{margin-bottom:1em}.cell-output-display:not(.no-overflow-x),.knitsql-table:not(.no-overflow-x){overflow-x:auto}.panel-input{margin-bottom:1em}.panel-input>div,.panel-input>div>div{display:inline-block;vertical-align:top;padding-right:12px}.panel-input>p:last-child{margin-bottom:0}.layout-sidebar{margin-bottom:1em}.layout-sidebar .tab-content{border:none}.tab-content>.page-columns.active{display:grid}div.sourceCode>iframe{width:100%;height:300px;margin-bottom:-0.5em}a{text-underline-offset:3px}div.ansi-escaped-output{font-family:monospace;display:block}/*!
|
| 2 |
*
|
| 3 |
* ansi colors from IPython notebook's
|
| 4 |
*
|
style.css
CHANGED
|
@@ -1,6 +1,7 @@
|
|
| 1 |
/*-- scss:defaults --*/
|
| 2 |
table {
|
| 3 |
-
color:
|
|
|
|
| 4 |
}
|
| 5 |
|
| 6 |
|
|
|
|
| 1 |
/*-- scss:defaults --*/
|
| 2 |
table {
|
| 3 |
+
color: #93a1a1;
|
| 4 |
+
background-color: rgba(0,0,0,.1);
|
| 5 |
}
|
| 6 |
|
| 7 |
|