| I tried making groupsize 16 but that did not end well so I went with 32g. FYI I can run this with full context on my A6000. | |
| ``` | |
| 65B (act-order true-sequential groupsize) | |
| wikitext2 3.5319948196411133 (stock 16bit) | |
| wikitext2 3.610668182373047 (32g) | |
| wikitext2 3.650667667388916 (16g) | |
| wikitext2 3.6660284996032715 (128) | |
| ptb-new 7.66942024230957 (stock 16bit) | |
| ptb-new 7.71506929397583 (32g) | |
| ptb-new 7.762592792510986 (128) | |
| ptb-new 7.829207897186279 (16g) | |
| c4-new 5.8114824295043945 (stock 16bit) | |
| c4-new 5.859227657318115 (32g) | |
| c4-new 5.893154144287109 (128) | |
| c4-new 5.929086208343506 (16g) | |
| ``` |