| ## Part 1 | |
| Normal model | |
| Memory usage of model alone = 510.342192 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 838.783488 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:25<00:00, 18.97it/s] | |
| Loss = 26.38488006591797 | |
| Time taken: 25.795103549957275 | |
| Full model quant | |
| Memory usage of model alone = 294.250369 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 1465.776128 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:21<00:00, 22.39it/s] | |
| Loss = 26.954803466796875 | |
| Time taken: 21.855380058288574 | |
| Full model without lm_head | |
| Memory usage of model alone = 255.602736 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 1269.30176 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:21<00:00, 22.68it/s] | |
| Loss = 26.41402816772461 | |
| Time taken: 21.578929662704468 | |
| Only LM head | |
| Memory usage of model alone = 548.989825 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 1036.319744 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:20<00:00, 23.39it/s] | |
| Loss = 26.924053192138672 | |
| Time taken: 20.919220209121704 | |
| Last 4 attention layers | |
| Memory usage of model alone = 425.42904 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 983.949824 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:20<00:00, 23.40it/s] | |
| Loss = 26.39584732055664 | |
| Time taken: 20.912957668304443 | |
| Only q,k,v | |
| Memory usage of model alone = 425.425968 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 989.827584 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:21<00:00, 23.11it/s] | |
| Loss = 26.396583557128906 | |
| Time taken: 21.17274236679077 | |
| ## Part 2: | |
| 4 bit model | |
| Memory usage of model alone = 134.060568 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 308.803072 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:16<00:00, 29.78it/s] | |
| Loss = 31.296875 | |
| Time taken: 16.42749333381653 | |
| `low_cpu_mem_usage` was None, now set to True since model is quantized. | |
| 8 bit model | |
| Memory usage of model alone = 176.527896 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 494.142976 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:29<00:00, 16.70it/s] | |
| Loss = 26.5625 | |
| Time taken: 29.27569341659546 | |
| `low_cpu_mem_usage` was None, now set to True since model is quantized. | |
| 4 bit nf4 model | |
| Memory usage of model alone = 134.060568 | |
| 0%| | 0/491 [00:00<?, ?it/s]Memory usage at forward pass = 494.85824 | |
| 100%|█████████████████████████████████████▊| 489/491 [00:15<00:00, 30.64it/s] | |
| Loss = 28.375 | |
| Time taken: 15.961309671401978 | |