Hugging Face's logo

eminorhan
/
gpt_saycam

Model card Files Community
eminorhan commited on
Commit
d028f33
·
1 Parent(s): 22b9cac

Upload 10 files

Browse files
Files changed (10) hide show
  1. logs/a_gimel_log_0.out +200 -0
  2. logs/a_gimel_log_1.out +188 -0
  3. logs/s_gimel_log_0.out +200 -0
  4. logs/s_gimel_log_1.out +168 -0
  5. logs/say_gimel_log_0.out +166 -0
  6. logs/say_gimel_log_1.out +164 -0
  7. logs/say_gimel_log_2.out +158 -0
  8. logs/say_gimel_log_3.out +168 -0
  9. logs/y_gimel_log_0.out +168 -0
  10. logs/y_gimel_log_1.out +188 -0
logs/a_gimel_log_0.out ADDED
@@ -0,0 +1,200 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
50
+ warnings.warn(
51
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
52
+ warnings.warn(
53
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
54
+ warnings.warn(msg)
55
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
56
+ warnings.warn(msg)
57
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
58
+ warnings.warn(
59
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
60
+ warnings.warn(
61
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
62
+ warnings.warn(msg)
63
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
64
+ warnings.warn(msg)
65
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
66
+ warnings.warn(
67
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
68
+ warnings.warn(msg)
69
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
70
+ warnings.warn(
71
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
72
+ warnings.warn(
73
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
74
+ warnings.warn(
75
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
76
+ warnings.warn(msg)
77
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
78
+ warnings.warn(
79
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
80
+ warnings.warn(msg)
81
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
82
+ warnings.warn(
83
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
84
+ warnings.warn(msg)
85
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
86
+ warnings.warn(msg)
87
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
88
+ warnings.warn(msg)
89
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
90
+ warnings.warn(
91
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
92
+ warnings.warn(msg)
93
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
94
+ warnings.warn(
95
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
96
+ warnings.warn(msg)
97
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
98
+ warnings.warn(
99
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
100
+ warnings.warn(msg)
101
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
102
+ warnings.warn(
103
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
104
+ warnings.warn(msg)
105
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
106
+ warnings.warn(
107
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
108
+ warnings.warn(msg)
109
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
110
+ warnings.warn(
111
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
112
+ warnings.warn(msg)
113
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
114
+ VQLPIPSWithDiscriminator running with hinge loss.
115
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
116
+ warnings.warn(_create_warning_msg(
117
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
118
+ warnings.warn(_create_warning_msg(
119
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
120
+ warnings.warn(_create_warning_msg(
121
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
122
+ warnings.warn(_create_warning_msg(
123
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
124
+ warnings.warn(_create_warning_msg(
125
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
126
+ warnings.warn(_create_warning_msg(
127
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
128
+ warnings.warn(_create_warning_msg(
129
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
130
+ warnings.warn(_create_warning_msg(
131
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
132
+ warnings.warn(_create_warning_msg(
133
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
134
+ warnings.warn(_create_warning_msg(
135
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
136
+ warnings.warn(_create_warning_msg(
137
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
138
+ warnings.warn(_create_warning_msg(
139
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
140
+ warnings.warn(_create_warning_msg(
141
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
142
+ warnings.warn(_create_warning_msg(
143
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
144
+ warnings.warn(_create_warning_msg(
145
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
146
+ warnings.warn(_create_warning_msg(
147
+ Number of parameters: 730671360
148
+ Running on 16 GPUs total
149
+ => no checkpoint loaded, will train from scratch
150
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
151
+ warnings.warn(warning.format(ret))
152
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
153
+ warnings.warn(warning.format(ret))
154
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
155
+ warnings.warn(warning.format(ret))
156
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
157
+ warnings.warn(warning.format(ret))
158
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
159
+ warnings.warn(warning.format(ret))
160
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
161
+ warnings.warn(warning.format(ret))
162
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
163
+ warnings.warn(warning.format(ret))
164
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
165
+ warnings.warn(warning.format(ret))
166
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
167
+ warnings.warn(warning.format(ret))
168
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
169
+ warnings.warn(warning.format(ret))
170
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
171
+ warnings.warn(warning.format(ret))
172
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
173
+ warnings.warn(warning.format(ret))
174
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
175
+ warnings.warn(warning.format(ret))
176
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
177
+ warnings.warn(warning.format(ret))
178
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
179
+ warnings.warn(warning.format(ret))
180
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
181
+ warnings.warn(warning.format(ret))
182
+ Iteration: 0 | Training loss: 9.283171653747559
183
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_0.pt
184
+ Iteration: 10000 | Training loss: 5.8971640702486035
185
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_10000.pt
186
+ Iteration: 20000 | Training loss: 5.4968702283144
187
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_20000.pt
188
+ Iteration: 30000 | Training loss: 5.361090888786316
189
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_30000.pt
190
+ Iteration: 40000 | Training loss: 5.2772094738245015
191
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_40000.pt
192
+ Iteration: 50000 | Training loss: 5.21737945227623
193
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_50000.pt
194
+ Iteration: 60000 | Training loss: 5.180605615353584
195
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_60000.pt
196
+ Iteration: 70000 | Training loss: 5.144675075793266
197
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_70000.pt
198
+ slurmstepd: error: *** STEP 29613492.0 ON ga013 CANCELLED AT 2023-01-31T16:58:41 ***
199
+ slurmstepd: error: *** JOB 29613492 ON ga013 CANCELLED AT 2023-01-31T16:58:41 ***
200
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
logs/a_gimel_log_1.out ADDED
@@ -0,0 +1,188 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/A_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/a_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='a_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
50
+ warnings.warn(
51
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
52
+ warnings.warn(
53
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
54
+ warnings.warn(
55
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
56
+ warnings.warn(
57
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
58
+ warnings.warn(msg)
59
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
60
+ warnings.warn(msg)
61
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
62
+ warnings.warn(
63
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
64
+ warnings.warn(
65
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
66
+ warnings.warn(msg)
67
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
68
+ warnings.warn(msg)
69
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
70
+ warnings.warn(
71
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
72
+ warnings.warn(
73
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
74
+ warnings.warn(
75
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
76
+ warnings.warn(
77
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
78
+ warnings.warn(msg)
79
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
80
+ warnings.warn(
81
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
82
+ warnings.warn(msg)
83
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
84
+ warnings.warn(
85
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
86
+ warnings.warn(msg)
87
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
88
+ warnings.warn(msg)
89
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
90
+ warnings.warn(
91
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
92
+ warnings.warn(
93
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
94
+ warnings.warn(msg)
95
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
96
+ warnings.warn(msg)
97
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
98
+ warnings.warn(msg)
99
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
100
+ warnings.warn(msg)
101
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
102
+ warnings.warn(
103
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
104
+ warnings.warn(msg)
105
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
106
+ warnings.warn(msg)
107
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
108
+ warnings.warn(msg)
109
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
110
+ warnings.warn(
111
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
112
+ warnings.warn(msg)
113
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
114
+ VQLPIPSWithDiscriminator running with hinge loss.
115
+ Number of parameters: 730671360
116
+ Running on 16 GPUs total
117
+ => loaded model weights and optimizer state at checkpoint '/scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel.pt'
118
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
119
+ warnings.warn(warning.format(ret))
120
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
121
+ warnings.warn(warning.format(ret))
122
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
123
+ warnings.warn(warning.format(ret))
124
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
125
+ warnings.warn(warning.format(ret))
126
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
127
+ warnings.warn(warning.format(ret))
128
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
129
+ warnings.warn(warning.format(ret))
130
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
131
+ warnings.warn(warning.format(ret))
132
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
133
+ warnings.warn(warning.format(ret))
134
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
135
+ warnings.warn(warning.format(ret))
136
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
137
+ warnings.warn(warning.format(ret))
138
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
139
+ warnings.warn(warning.format(ret))
140
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
141
+ warnings.warn(warning.format(ret))
142
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
143
+ warnings.warn(warning.format(ret))
144
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
145
+ warnings.warn(warning.format(ret))
146
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
147
+ warnings.warn(warning.format(ret))
148
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
149
+ warnings.warn(warning.format(ret))
150
+ Iteration: 0 | Training loss: 4.586821556091309
151
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_0.pt
152
+ Iteration: 4000 | Training loss: 5.206405858278274
153
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_4000.pt
154
+ Iteration: 8000 | Training loss: 5.007484036445618
155
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_8000.pt
156
+ Iteration: 12000 | Training loss: 4.913228427648544
157
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_12000.pt
158
+ Iteration: 16000 | Training loss: 4.878944065392018
159
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_16000.pt
160
+ Iteration: 20000 | Training loss: 4.853666521191597
161
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_20000.pt
162
+ Iteration: 24000 | Training loss: 4.85290568870306
163
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_24000.pt
164
+ Iteration: 28000 | Training loss: 4.853768558263779
165
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_28000.pt
166
+ Iteration: 32000 | Training loss: 4.846220250189305
167
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_32000.pt
168
+ Iteration: 36000 | Training loss: 4.852501285433769
169
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_36000.pt
170
+ Iteration: 40000 | Training loss: 4.8531693903803825
171
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_40000.pt
172
+ Iteration: 44000 | Training loss: 4.841027199268341
173
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_44000.pt
174
+ Iteration: 48000 | Training loss: 4.852847850620747
175
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_48000.pt
176
+ Iteration: 52000 | Training loss: 4.848223978936672
177
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_52000.pt
178
+ Iteration: 56000 | Training loss: 4.845628525495529
179
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_56000.pt
180
+ Iteration: 60000 | Training loss: 4.834981297135353
181
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_60000.pt
182
+ Iteration: 64000 | Training loss: 4.828589959859848
183
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_64000.pt
184
+ Iteration: 68000 | Training loss: 4.826472416520119
185
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/a_gimel_68000.pt
186
+ slurmstepd: error: *** JOB 29938000 ON ga001 CANCELLED AT 2023-02-09T22:22:39 ***
187
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
188
+ slurmstepd: error: *** STEP 29938000.0 ON ga001 CANCELLED AT 2023-02-09T22:22:39 ***
logs/s_gimel_log_0.out ADDED
@@ -0,0 +1,200 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
50
+ warnings.warn(
51
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
52
+ warnings.warn(msg)
53
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
54
+ warnings.warn(
55
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
56
+ warnings.warn(msg)
57
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
58
+ warnings.warn(
59
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
60
+ warnings.warn(msg)
61
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
62
+ warnings.warn(
63
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
64
+ warnings.warn(msg)
65
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
66
+ warnings.warn(
67
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
68
+ warnings.warn(msg)
69
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
70
+ warnings.warn(
71
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
72
+ warnings.warn(msg)
73
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
74
+ warnings.warn(
75
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
76
+ warnings.warn(
77
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
78
+ warnings.warn(msg)
79
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
80
+ warnings.warn(msg)
81
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
82
+ warnings.warn(
83
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
84
+ warnings.warn(msg)
85
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
86
+ warnings.warn(
87
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
88
+ warnings.warn(msg)
89
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
90
+ warnings.warn(
91
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
92
+ warnings.warn(msg)
93
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
94
+ warnings.warn(
95
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
96
+ warnings.warn(msg)
97
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
98
+ warnings.warn(
99
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
100
+ warnings.warn(msg)
101
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
102
+ warnings.warn(
103
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
104
+ warnings.warn(msg)
105
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
106
+ warnings.warn(
107
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
108
+ warnings.warn(msg)
109
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
110
+ warnings.warn(
111
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
112
+ warnings.warn(msg)
113
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
114
+ VQLPIPSWithDiscriminator running with hinge loss.
115
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
116
+ warnings.warn(_create_warning_msg(
117
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
118
+ warnings.warn(_create_warning_msg(
119
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
120
+ warnings.warn(_create_warning_msg(
121
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
122
+ warnings.warn(_create_warning_msg(
123
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
124
+ warnings.warn(_create_warning_msg(
125
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
126
+ warnings.warn(_create_warning_msg(
127
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
128
+ warnings.warn(_create_warning_msg(
129
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
130
+ warnings.warn(_create_warning_msg(
131
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
132
+ warnings.warn(_create_warning_msg(
133
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
134
+ warnings.warn(_create_warning_msg(
135
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
136
+ warnings.warn(_create_warning_msg(
137
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
138
+ warnings.warn(_create_warning_msg(
139
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
140
+ warnings.warn(_create_warning_msg(
141
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
142
+ warnings.warn(_create_warning_msg(
143
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
144
+ warnings.warn(_create_warning_msg(
145
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/utils/data/dataloader.py:563: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
146
+ warnings.warn(_create_warning_msg(
147
+ Number of parameters: 730671360
148
+ Running on 16 GPUs total
149
+ => no checkpoint loaded, will train from scratch
150
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
151
+ warnings.warn(warning.format(ret))
152
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
153
+ warnings.warn(warning.format(ret))
154
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
155
+ warnings.warn(warning.format(ret))
156
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
157
+ warnings.warn(warning.format(ret))
158
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
159
+ warnings.warn(warning.format(ret))
160
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
161
+ warnings.warn(warning.format(ret))
162
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
163
+ warnings.warn(warning.format(ret))
164
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
165
+ warnings.warn(warning.format(ret))
166
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
167
+ warnings.warn(warning.format(ret))
168
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
169
+ warnings.warn(warning.format(ret))
170
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
171
+ warnings.warn(warning.format(ret))
172
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
173
+ warnings.warn(warning.format(ret))
174
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
175
+ warnings.warn(warning.format(ret))
176
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
177
+ warnings.warn(warning.format(ret))
178
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
179
+ warnings.warn(warning.format(ret))
180
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
181
+ warnings.warn(warning.format(ret))
182
+ Iteration: 0 | Training loss: 9.250845909118652
183
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_0_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
184
+ Iteration: 10000 | Training loss: 5.809336846494674
185
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_10000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
186
+ Iteration: 20000 | Training loss: 5.381569646072387
187
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_20000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
188
+ Iteration: 30000 | Training loss: 5.244765582513809
189
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_30000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
190
+ Iteration: 40000 | Training loss: 5.168160017490387
191
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_40000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
192
+ Iteration: 50000 | Training loss: 5.114514644646644
193
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_50000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
194
+ Iteration: 60000 | Training loss: 5.067785410428047
195
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_60000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
196
+ Iteration: 70000 | Training loss: 5.025504623293877
197
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_70000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
198
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
199
+ slurmstepd: error: *** JOB 29613439 ON ga016 CANCELLED AT 2023-01-31T07:27:35 ***
200
+ slurmstepd: error: *** STEP 29613439.0 ON ga016 CANCELLED AT 2023-01-31T07:27:35 ***
logs/s_gimel_log_1.out ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/Sfp_5fps_300s_{000000..000003}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/s_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='s_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
50
+ warnings.warn(
51
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
52
+ warnings.warn(
53
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
54
+ warnings.warn(
55
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
56
+ warnings.warn(
57
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
58
+ warnings.warn(
59
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
60
+ warnings.warn(
61
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
62
+ warnings.warn(
63
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
64
+ warnings.warn(
65
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
66
+ warnings.warn(msg)
67
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
68
+ warnings.warn(
69
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
70
+ warnings.warn(msg)
71
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
72
+ warnings.warn(msg)
73
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
74
+ warnings.warn(msg)
75
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
76
+ warnings.warn(msg)
77
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
78
+ warnings.warn(msg)
79
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
80
+ warnings.warn(msg)
81
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
82
+ warnings.warn(msg)
83
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
84
+ warnings.warn(
85
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
86
+ warnings.warn(
87
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
88
+ warnings.warn(
89
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
90
+ warnings.warn(msg)
91
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
92
+ warnings.warn(msg)
93
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
94
+ warnings.warn(msg)
95
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
96
+ warnings.warn(
97
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
98
+ warnings.warn(
99
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
100
+ warnings.warn(
101
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
102
+ warnings.warn(msg)
103
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
104
+ warnings.warn(msg)
105
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
106
+ warnings.warn(msg)
107
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
108
+ warnings.warn(msg)
109
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
110
+ warnings.warn(
111
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
112
+ warnings.warn(msg)
113
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
114
+ VQLPIPSWithDiscriminator running with hinge loss.
115
+ Number of parameters: 730671360
116
+ Running on 16 GPUs total
117
+ => loaded model weights and optimizer state at checkpoint '/scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel.pt'
118
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
119
+ warnings.warn(warning.format(ret))
120
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
121
+ warnings.warn(warning.format(ret))
122
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
123
+ warnings.warn(warning.format(ret))
124
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
125
+ warnings.warn(warning.format(ret))
126
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
127
+ warnings.warn(warning.format(ret))
128
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
129
+ warnings.warn(warning.format(ret))
130
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
131
+ warnings.warn(warning.format(ret))
132
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
133
+ warnings.warn(warning.format(ret))
134
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
135
+ warnings.warn(warning.format(ret))
136
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
137
+ warnings.warn(warning.format(ret))
138
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
139
+ warnings.warn(warning.format(ret))
140
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
141
+ warnings.warn(warning.format(ret))
142
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
143
+ warnings.warn(warning.format(ret))
144
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
145
+ warnings.warn(warning.format(ret))
146
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
147
+ warnings.warn(warning.format(ret))
148
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
149
+ warnings.warn(warning.format(ret))
150
+ Iteration: 0 | Training loss: 4.485539436340332
151
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel_0.pt
152
+ Iteration: 10000 | Training loss: 4.957391840171814
153
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel_10000.pt
154
+ Iteration: 20000 | Training loss: 4.82037622089386
155
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel_20000.pt
156
+ Iteration: 30000 | Training loss: 4.788349422216416
157
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel_30000.pt
158
+ Iteration: 40000 | Training loss: 4.782987071561814
159
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel_40000.pt
160
+ Iteration: 50000 | Training loss: 4.775392537903786
161
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel_50000.pt
162
+ Iteration: 60000 | Training loss: 4.761250894474983
163
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel_60000.pt
164
+ Iteration: 70000 | Training loss: 4.750863534975052
165
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/s_gimel_70000.pt
166
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
167
+ slurmstepd: error: *** JOB 29841181 ON ga024 CANCELLED AT 2023-02-09T10:03:19 ***
168
+ slurmstepd: error: *** STEP 29841181.0 ON ga024 CANCELLED AT 2023-02-09T10:03:19 ***
logs/say_gimel_log_0.out ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
50
+ VQLPIPSWithDiscriminator running with hinge loss.
51
+ Number of parameters: 730671360
52
+ Running on 16 GPUs total
53
+ => no checkpoint loaded, will train from scratch
54
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
55
+ warnings.warn(warning.format(ret))
56
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
57
+ warnings.warn(warning.format(ret))
58
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
59
+ warnings.warn(warning.format(ret))
60
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
61
+ warnings.warn(warning.format(ret))
62
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
63
+ warnings.warn(warning.format(ret))
64
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
65
+ warnings.warn(warning.format(ret))
66
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
67
+ warnings.warn(warning.format(ret))
68
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
69
+ warnings.warn(warning.format(ret))
70
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
71
+ warnings.warn(warning.format(ret))
72
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
73
+ warnings.warn(warning.format(ret))
74
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
75
+ warnings.warn(warning.format(ret))
76
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
77
+ warnings.warn(warning.format(ret))
78
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
79
+ warnings.warn(warning.format(ret))
80
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
81
+ warnings.warn(warning.format(ret))
82
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
83
+ warnings.warn(warning.format(ret))
84
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
85
+ warnings.warn(warning.format(ret))
86
+ Iteration: 0 | Training loss: 9.269855499267578
87
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_0_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
88
+ Iteration: 5000 | Training loss: 6.228412897777558
89
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_5000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
90
+ Iteration: 10000 | Training loss: 5.677635769701004
91
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_10000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
92
+ Iteration: 15000 | Training loss: 5.588409051990509
93
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_15000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
94
+ Iteration: 20000 | Training loss: 5.531276674461365
95
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_20000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
96
+ Iteration: 25000 | Training loss: 5.4793645988941195
97
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_25000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
98
+ Iteration: 30000 | Training loss: 5.441831717920303
99
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_30000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
100
+ Iteration: 35000 | Training loss: 5.411573132991791
101
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_35000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
102
+ Iteration: 40000 | Training loss: 5.390492297029495
103
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_40000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
104
+ Iteration: 45000 | Training loss: 5.360090880012512
105
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_45000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
106
+ Iteration: 50000 | Training loss: 5.344044271087647
107
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_50000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
108
+ Iteration: 55000 | Training loss: 5.335392073106766
109
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_55000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
110
+ Iteration: 60000 | Training loss: 5.314779483127594
111
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_60000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
112
+ Iteration: 65000 | Training loss: 5.292163775300979
113
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_65000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
114
+ Iteration: 70000 | Training loss: 5.281603284406662
115
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_70000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
116
+ Iteration: 75000 | Training loss: 5.264535038089752
117
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_75000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
118
+ Iteration: 80000 | Training loss: 5.256706160640716
119
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_80000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
120
+ Iteration: 85000 | Training loss: 5.249376992893219
121
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_85000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
122
+ Iteration: 90000 | Training loss: 5.237781165266037
123
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_90000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
124
+ Iteration: 95000 | Training loss: 5.229439442396164
125
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_95000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
126
+ Iteration: 100000 | Training loss: 5.222921891212463
127
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_100000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
128
+ Iteration: 105000 | Training loss: 5.214226239681244
129
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_105000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
130
+ Iteration: 110000 | Training loss: 5.207693300676346
131
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_110000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
132
+ Iteration: 115000 | Training loss: 5.194360571146011
133
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_115000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
134
+ Iteration: 120000 | Training loss: 5.1919717218875885
135
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_120000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
136
+ Iteration: 125000 | Training loss: 5.191810512542725
137
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_125000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
138
+ Iteration: 130000 | Training loss: 5.170573520088196
139
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_130000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
140
+ Iteration: 135000 | Training loss: 5.571219595909119
141
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_135000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
142
+ Iteration: 140000 | Training loss: 5.75581296043396
143
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_140000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
144
+ Iteration: 145000 | Training loss: 5.1849900501728055
145
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_145000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
146
+ Iteration: 150000 | Training loss: 5.168959885025024
147
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_150000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
148
+ Iteration: 155000 | Training loss: 5.159844536018372
149
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_155000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
150
+ Iteration: 160000 | Training loss: 5.3883808923721315
151
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_160000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
152
+ Iteration: 165000 | Training loss: 5.556298746442795
153
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_165000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
154
+ Iteration: 170000 | Training loss: 5.1485559203624724
155
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_170000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
156
+ Iteration: 175000 | Training loss: 5.140151729488373
157
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_175000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
158
+ Iteration: 180000 | Training loss: 5.137233607387543
159
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_180000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
160
+ Iteration: 185000 | Training loss: 5.1244073034763336
161
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_185000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
162
+ Iteration: 190000 | Training loss: 5.122972550725937
163
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
164
+ slurmstepd: error: *** JOB 24732571 ON ga002 CANCELLED AT 2022-09-13T09:42:28 DUE TO TIME LIMIT ***
165
+ slurmstepd: error: *** STEP 24732571.0 ON ga002 CANCELLED AT 2022-09-13T09:42:28 DUE TO TIME LIMIT ***
166
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
logs/say_gimel_log_1.out ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=5000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
50
+ VQLPIPSWithDiscriminator running with hinge loss.
51
+ Number of parameters: 730671360
52
+ Running on 16 GPUs total
53
+ => loaded model weights and optimizer state at checkpoint '/scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_190000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt'
54
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
55
+ warnings.warn(warning.format(ret))
56
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
57
+ warnings.warn(warning.format(ret))
58
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
59
+ warnings.warn(warning.format(ret))
60
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
61
+ warnings.warn(warning.format(ret))
62
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
63
+ warnings.warn(warning.format(ret))
64
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
65
+ warnings.warn(warning.format(ret))
66
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
67
+ warnings.warn(warning.format(ret))
68
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
69
+ warnings.warn(warning.format(ret))
70
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
71
+ warnings.warn(warning.format(ret))
72
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
73
+ warnings.warn(warning.format(ret))
74
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
75
+ warnings.warn(warning.format(ret))
76
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
77
+ warnings.warn(warning.format(ret))
78
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
79
+ warnings.warn(warning.format(ret))
80
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
81
+ warnings.warn(warning.format(ret))
82
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
83
+ warnings.warn(warning.format(ret))
84
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
85
+ warnings.warn(warning.format(ret))
86
+ Iteration: 0 | Training loss: 5.30007791519165
87
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_0_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
88
+ Iteration: 5000 | Training loss: 5.138480274868011
89
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_5000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
90
+ Iteration: 10000 | Training loss: 5.107799020719528
91
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_10000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
92
+ Iteration: 15000 | Training loss: 5.088881051254273
93
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_15000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
94
+ Iteration: 20000 | Training loss: 5.085070754432678
95
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_20000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
96
+ Iteration: 25000 | Training loss: 5.080795408391952
97
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_25000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
98
+ Iteration: 30000 | Training loss: 5.380685155773163
99
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_30000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
100
+ Iteration: 35000 | Training loss: 5.718056494235992
101
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_35000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
102
+ Iteration: 40000 | Training loss: 5.23170142364502
103
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_40000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
104
+ Iteration: 45000 | Training loss: 5.106258099412918
105
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_45000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
106
+ Iteration: 50000 | Training loss: 5.080553993320465
107
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_50000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
108
+ Iteration: 55000 | Training loss: 5.211881767654419
109
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_55000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
110
+ Iteration: 60000 | Training loss: 5.073302331018448
111
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_60000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
112
+ Iteration: 65000 | Training loss: 5.081954577064514
113
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_65000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
114
+ Iteration: 70000 | Training loss: 5.065998318481445
115
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_70000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
116
+ Iteration: 75000 | Training loss: 5.060894852352142
117
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_75000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
118
+ Iteration: 80000 | Training loss: 5.057493374586105
119
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_80000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
120
+ Iteration: 85000 | Training loss: 5.0525652509212495
121
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_85000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
122
+ Iteration: 90000 | Training loss: 5.048048374509811
123
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_90000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
124
+ Iteration: 95000 | Training loss: 5.045162040233612
125
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_95000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
126
+ Iteration: 100000 | Training loss: 5.047997813224793
127
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_100000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
128
+ Iteration: 105000 | Training loss: 5.035569662857056
129
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_105000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
130
+ Iteration: 110000 | Training loss: 5.030170858097076
131
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_110000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
132
+ Iteration: 115000 | Training loss: 5.020523439121247
133
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_115000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
134
+ Iteration: 120000 | Training loss: 5.033615832281113
135
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_120000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
136
+ Iteration: 125000 | Training loss: 5.024354502677918
137
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_125000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
138
+ Iteration: 130000 | Training loss: 5.030594699859619
139
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_130000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
140
+ Iteration: 135000 | Training loss: 5.01051748251915
141
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_135000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
142
+ Iteration: 140000 | Training loss: 5.021781470680237
143
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_140000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
144
+ Iteration: 145000 | Training loss: 5.01388137922287
145
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_145000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
146
+ Iteration: 150000 | Training loss: 5.007130924606323
147
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_150000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
148
+ Iteration: 155000 | Training loss: 5.010347195625306
149
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_155000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
150
+ Iteration: 160000 | Training loss: 5.009956067085266
151
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_160000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
152
+ Iteration: 165000 | Training loss: 5.00600462846756
153
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_165000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
154
+ Iteration: 170000 | Training loss: 5.006761898326873
155
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_170000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
156
+ Iteration: 175000 | Training loss: 4.996211072683335
157
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_175000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
158
+ Iteration: 180000 | Training loss: 4.990817209720611
159
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_180000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
160
+ Iteration: 185000 | Training loss: 4.992834660196304
161
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_185000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
162
+ slurmstepd: error: *** JOB 24809016 ON ga002 CANCELLED AT 2022-09-15T08:32:40 ***
163
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
164
+ slurmstepd: error: *** STEP 24809016.0 ON ga002 CANCELLED AT 2022-09-15T08:32:40 ***
logs/say_gimel_log_2.out ADDED
@@ -0,0 +1,158 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
50
+ VQLPIPSWithDiscriminator running with hinge loss.
51
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
52
+ warnings.warn(_create_warning_msg(
53
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
54
+ warnings.warn(_create_warning_msg(
55
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
56
+ warnings.warn(_create_warning_msg(
57
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
58
+ warnings.warn(_create_warning_msg(
59
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
60
+ warnings.warn(_create_warning_msg(
61
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
62
+ warnings.warn(_create_warning_msg(
63
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
64
+ warnings.warn(_create_warning_msg(
65
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
66
+ warnings.warn(_create_warning_msg(
67
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
68
+ warnings.warn(_create_warning_msg(
69
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
70
+ warnings.warn(_create_warning_msg(
71
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
72
+ warnings.warn(_create_warning_msg(
73
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
74
+ warnings.warn(_create_warning_msg(
75
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
76
+ warnings.warn(_create_warning_msg(
77
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
78
+ warnings.warn(_create_warning_msg(
79
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
80
+ warnings.warn(_create_warning_msg(
81
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py:478: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 4, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
82
+ warnings.warn(_create_warning_msg(
83
+ Number of parameters: 730671360
84
+ Running on 16 GPUs total
85
+ => loaded model weights and optimizer state at checkpoint '/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt'
86
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
87
+ warnings.warn(warning.format(ret))
88
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
89
+ warnings.warn(warning.format(ret))
90
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
91
+ warnings.warn(warning.format(ret))
92
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
93
+ warnings.warn(warning.format(ret))
94
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
95
+ warnings.warn(warning.format(ret))
96
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
97
+ warnings.warn(warning.format(ret))
98
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
99
+ warnings.warn(warning.format(ret))
100
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
101
+ warnings.warn(warning.format(ret))
102
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
103
+ warnings.warn(warning.format(ret))
104
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
105
+ warnings.warn(warning.format(ret))
106
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
107
+ warnings.warn(warning.format(ret))
108
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
109
+ warnings.warn(warning.format(ret))
110
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
111
+ warnings.warn(warning.format(ret))
112
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
113
+ warnings.warn(warning.format(ret))
114
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
115
+ warnings.warn(warning.format(ret))
116
+ /scratch/eo41/miniconda3/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
117
+ warnings.warn(warning.format(ret))
118
+ Iteration: 0 | Training loss: 5.033977508544922
119
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_0_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
120
+ Iteration: 10000 | Training loss: 5.000367347049713
121
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_10000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
122
+ Iteration: 20000 | Training loss: 4.979147749662399
123
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_20000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
124
+ Iteration: 30000 | Training loss: 4.972516257357597
125
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_30000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
126
+ Iteration: 40000 | Training loss: 4.970100594377517
127
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_40000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
128
+ Iteration: 50000 | Training loss: 5.405501440525055
129
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_50000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
130
+ Iteration: 60000 | Training loss: 5.353503350329399
131
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_60000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
132
+ Iteration: 70000 | Training loss: 5.220138521456718
133
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_70000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
134
+ Iteration: 80000 | Training loss: 5.138026001763344
135
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_80000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
136
+ Iteration: 90000 | Training loss: 4.997180798411369
137
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_90000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
138
+ Iteration: 100000 | Training loss: 4.972350775599479
139
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_100000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
140
+ Iteration: 110000 | Training loss: 4.9657788955450055
141
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_110000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
142
+ Iteration: 120000 | Training loss: 4.953705482721329
143
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_120000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
144
+ Iteration: 130000 | Training loss: 4.9448775294542315
145
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_130000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
146
+ Iteration: 140000 | Training loss: 4.944383617019653
147
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_140000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
148
+ Iteration: 150000 | Training loss: 4.944210085701942
149
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_150000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
150
+ Iteration: 160000 | Training loss: 4.938878140282631
151
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_160000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
152
+ Iteration: 170000 | Training loss: 4.931895919656753
153
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_170000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
154
+ Iteration: 180000 | Training loss: 4.933802534270287
155
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/model_180000_36l_20h_1280e_96b_0.0003lr_Adamo_0s.pt
156
+ slurmstepd: error: *** JOB 27300133 ON ga001 CANCELLED AT 2022-11-26T10:02:03 DUE TO TIME LIMIT ***
157
+ slurmstepd: error: *** STEP 27300133.0 ON ga001 CANCELLED AT 2022-11-26T10:02:03 DUE TO TIME LIMIT ***
158
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
logs/say_gimel_log_3.out ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/SAY_5fps_300s_{000000..000009}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/say_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='say_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
50
+ warnings.warn(
51
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
52
+ warnings.warn(
53
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
54
+ warnings.warn(
55
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
56
+ warnings.warn(
57
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
58
+ warnings.warn(
59
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
60
+ warnings.warn(
61
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
62
+ warnings.warn(msg)
63
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
64
+ warnings.warn(
65
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
66
+ warnings.warn(
67
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
68
+ warnings.warn(msg)
69
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
70
+ warnings.warn(msg)
71
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
72
+ warnings.warn(msg)
73
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
74
+ warnings.warn(
75
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
76
+ warnings.warn(
77
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
78
+ warnings.warn(
79
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
80
+ warnings.warn(msg)
81
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
82
+ warnings.warn(
83
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
84
+ warnings.warn(
85
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
86
+ warnings.warn(msg)
87
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
88
+ warnings.warn(msg)
89
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
90
+ warnings.warn(msg)
91
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
92
+ warnings.warn(msg)
93
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
94
+ warnings.warn(msg)
95
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
96
+ warnings.warn(msg)
97
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
98
+ warnings.warn(msg)
99
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
100
+ warnings.warn(
101
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
102
+ warnings.warn(
103
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
104
+ warnings.warn(msg)
105
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
106
+ warnings.warn(
107
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
108
+ warnings.warn(msg)
109
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
110
+ warnings.warn(msg)
111
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
112
+ warnings.warn(msg)
113
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
114
+ VQLPIPSWithDiscriminator running with hinge loss.
115
+ Number of parameters: 730671360
116
+ Running on 16 GPUs total
117
+ => loaded model weights and optimizer state at checkpoint '/scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel.pt'
118
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
119
+ warnings.warn(warning.format(ret))
120
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
121
+ warnings.warn(warning.format(ret))
122
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
123
+ warnings.warn(warning.format(ret))
124
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
125
+ warnings.warn(warning.format(ret))
126
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
127
+ warnings.warn(warning.format(ret))
128
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
129
+ warnings.warn(warning.format(ret))
130
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
131
+ warnings.warn(warning.format(ret))
132
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
133
+ warnings.warn(warning.format(ret))
134
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
135
+ warnings.warn(warning.format(ret))
136
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
137
+ warnings.warn(warning.format(ret))
138
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
139
+ warnings.warn(warning.format(ret))
140
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
141
+ warnings.warn(warning.format(ret))
142
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
143
+ warnings.warn(warning.format(ret))
144
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
145
+ warnings.warn(warning.format(ret))
146
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
147
+ warnings.warn(warning.format(ret))
148
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
149
+ warnings.warn(warning.format(ret))
150
+ Iteration: 0 | Training loss: 5.078606605529785
151
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel_0.pt
152
+ Iteration: 10000 | Training loss: 4.9276696452379225
153
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel_10000.pt
154
+ Iteration: 20000 | Training loss: 5.072141832995415
155
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel_20000.pt
156
+ Iteration: 30000 | Training loss: 4.912290808415413
157
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel_30000.pt
158
+ Iteration: 40000 | Training loss: 4.900061968159676
159
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel_40000.pt
160
+ Iteration: 50000 | Training loss: 4.893786135959625
161
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel_50000.pt
162
+ Iteration: 60000 | Training loss: 4.895893426060677
163
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel_60000.pt
164
+ Iteration: 70000 | Training loss: 5.258830540013314
165
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/say_gimel_70000.pt
166
+ slurmstepd: error: *** STEP 29780055.0 ON ga002 CANCELLED AT 2023-02-06T10:44:58 ***
167
+ slurmstepd: error: *** JOB 29780055 ON ga002 CANCELLED AT 2023-02-06T10:44:58 ***
168
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
logs/y_gimel_log_0.out ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=8, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=10000, lr=0.0003, optimizer='Adam', resume='', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
50
+ warnings.warn(
51
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
52
+ warnings.warn(
53
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
54
+ warnings.warn(
55
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
56
+ warnings.warn(msg)
57
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
58
+ warnings.warn(msg)
59
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
60
+ warnings.warn(msg)
61
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
62
+ warnings.warn(
63
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
64
+ warnings.warn(
65
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
66
+ warnings.warn(
67
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
68
+ warnings.warn(msg)
69
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
70
+ warnings.warn(
71
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
72
+ warnings.warn(msg)
73
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
74
+ warnings.warn(msg)
75
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
76
+ warnings.warn(
77
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
78
+ warnings.warn(msg)
79
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
80
+ warnings.warn(
81
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
82
+ warnings.warn(
83
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
84
+ warnings.warn(msg)
85
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
86
+ warnings.warn(
87
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
88
+ warnings.warn(msg)
89
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
90
+ warnings.warn(msg)
91
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
92
+ warnings.warn(msg)
93
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
94
+ warnings.warn(
95
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
96
+ warnings.warn(
97
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
98
+ warnings.warn(msg)
99
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
100
+ warnings.warn(
101
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
102
+ warnings.warn(msg)
103
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
104
+ warnings.warn(
105
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
106
+ warnings.warn(msg)
107
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
108
+ warnings.warn(msg)
109
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
110
+ warnings.warn(
111
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
112
+ warnings.warn(msg)
113
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
114
+ VQLPIPSWithDiscriminator running with hinge loss.
115
+ Number of parameters: 730671360
116
+ Running on 16 GPUs total
117
+ => no checkpoint loaded, will train from scratch
118
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
119
+ warnings.warn(warning.format(ret))
120
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
121
+ warnings.warn(warning.format(ret))
122
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
123
+ warnings.warn(warning.format(ret))
124
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
125
+ warnings.warn(warning.format(ret))
126
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
127
+ warnings.warn(warning.format(ret))
128
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
129
+ warnings.warn(warning.format(ret))
130
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
131
+ warnings.warn(warning.format(ret))
132
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
133
+ warnings.warn(warning.format(ret))
134
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
135
+ warnings.warn(warning.format(ret))
136
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
137
+ warnings.warn(warning.format(ret))
138
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
139
+ warnings.warn(warning.format(ret))
140
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
141
+ warnings.warn(warning.format(ret))
142
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
143
+ warnings.warn(warning.format(ret))
144
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
145
+ warnings.warn(warning.format(ret))
146
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
147
+ warnings.warn(warning.format(ret))
148
+ /scratch/eo41/miniconda3/envs/llm/lib/python3.9/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
149
+ warnings.warn(warning.format(ret))
150
+ Iteration: 0 | Training loss: 9.316059112548828
151
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_0.pt
152
+ Iteration: 10000 | Training loss: 5.527423759746552
153
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_10000.pt
154
+ Iteration: 20000 | Training loss: 4.884146058940887
155
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_20000.pt
156
+ Iteration: 30000 | Training loss: 4.6967807245969775
157
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_30000.pt
158
+ Iteration: 40000 | Training loss: 4.584212138175964
159
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_40000.pt
160
+ Iteration: 50000 | Training loss: 4.498861210989952
161
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_50000.pt
162
+ Iteration: 60000 | Training loss: 4.439975436353683
163
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_60000.pt
164
+ Iteration: 70000 | Training loss: 4.385554709815979
165
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_70000.pt
166
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
167
+ slurmstepd: error: *** JOB 29645373 ON ga025 CANCELLED AT 2023-02-02T13:40:23 ***
168
+ slurmstepd: error: *** STEP 29645373.0 ON ga025 CANCELLED AT 2023-02-02T13:40:23 ***
logs/y_gimel_log_1.out ADDED
@@ -0,0 +1,188 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
2
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
3
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
4
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
5
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
6
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
7
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
8
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
9
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
10
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
11
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
12
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
13
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
14
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
15
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
16
+ Namespace(data_path='/scratch/eo41/data/saycam/Y_5fps_300s_{000000..000002}.tar', vqconfig_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.yaml', vqmodel_path='/scratch/eo41/vqgan-gpt/vqgan_pretrained_models/y_32x32_8192.ckpt', num_workers=16, seed=0, save_dir='/scratch/eo41/vqgan-gpt/gpt_pretrained_models', save_prefix='y_gimel', gpt_config='GPT_gimel', vocab_size=8192, block_size=1023, batch_size=6, print_freq=4000, lr=0.0003, optimizer='Adam', resume='/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt', gpu=None, world_size=-1, rank=-1, dist_url='env://', dist_backend='nccl', local_rank=-1)
17
+ model:
18
+ base_learning_rate: 1.0e-05
19
+ params:
20
+ ddconfig:
21
+ attn_resolutions:
22
+ - 32
23
+ ch: 128
24
+ ch_mult:
25
+ - 1
26
+ - 1
27
+ - 2
28
+ - 4
29
+ double_z: false
30
+ dropout: 0.0
31
+ in_channels: 3
32
+ num_res_blocks: 2
33
+ out_ch: 3
34
+ resolution: 256
35
+ z_channels: 256
36
+ embed_dim: 256
37
+ lossconfig:
38
+ params:
39
+ codebook_weight: 1.0
40
+ disc_conditional: false
41
+ disc_in_channels: 3
42
+ disc_start: 100001
43
+ disc_weight: 0.2
44
+ target: vqloss.VQLPIPSWithDiscriminator
45
+ n_embed: 8192
46
+ target: vqmodel.VQModel
47
+
48
+ Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
49
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
50
+ warnings.warn(
51
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
52
+ warnings.warn(msg)
53
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
54
+ warnings.warn(
55
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
56
+ warnings.warn(msg)
57
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
58
+ warnings.warn(
59
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
60
+ warnings.warn(
61
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
62
+ warnings.warn(msg)
63
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
64
+ warnings.warn(msg)
65
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
66
+ warnings.warn(
67
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
68
+ warnings.warn(
69
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
70
+ warnings.warn(msg)
71
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
72
+ warnings.warn(msg)
73
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
74
+ warnings.warn(
75
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
76
+ warnings.warn(msg)
77
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
78
+ warnings.warn(
79
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
80
+ warnings.warn(msg)
81
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
82
+ warnings.warn(
83
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
84
+ warnings.warn(msg)
85
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
86
+ warnings.warn(
87
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
88
+ warnings.warn(msg)
89
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
90
+ warnings.warn(
91
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
92
+ warnings.warn(msg)
93
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
94
+ warnings.warn(
95
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
96
+ warnings.warn(msg)
97
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
98
+ warnings.warn(
99
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
100
+ warnings.warn(msg)
101
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
102
+ warnings.warn(
103
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
104
+ warnings.warn(msg)
105
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
106
+ warnings.warn(
107
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
108
+ warnings.warn(msg)
109
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
110
+ warnings.warn(
111
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
112
+ warnings.warn(msg)
113
+ loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth
114
+ VQLPIPSWithDiscriminator running with hinge loss.
115
+ Number of parameters: 730671360
116
+ Running on 16 GPUs total
117
+ => loaded model weights and optimizer state at checkpoint '/scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel.pt'
118
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
119
+ warnings.warn(warning.format(ret))
120
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
121
+ warnings.warn(warning.format(ret))
122
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
123
+ warnings.warn(warning.format(ret))
124
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
125
+ warnings.warn(warning.format(ret))
126
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
127
+ warnings.warn(warning.format(ret))
128
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
129
+ warnings.warn(warning.format(ret))
130
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
131
+ warnings.warn(warning.format(ret))
132
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
133
+ warnings.warn(warning.format(ret))
134
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
135
+ warnings.warn(warning.format(ret))
136
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
137
+ warnings.warn(warning.format(ret))
138
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
139
+ warnings.warn(warning.format(ret))
140
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
141
+ warnings.warn(warning.format(ret))
142
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
143
+ warnings.warn(warning.format(ret))
144
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
145
+ warnings.warn(warning.format(ret))
146
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
147
+ warnings.warn(warning.format(ret))
148
+ /scratch/eo41/miniconda3/lib/python3.10/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
149
+ warnings.warn(warning.format(ret))
150
+ Iteration: 0 | Training loss: 4.905966758728027
151
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_0.pt
152
+ Iteration: 4000 | Training loss: 4.476815513432026
153
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_4000.pt
154
+ Iteration: 8000 | Training loss: 4.2836167613267895
155
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_8000.pt
156
+ Iteration: 12000 | Training loss: 4.162596913158894
157
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_12000.pt
158
+ Iteration: 16000 | Training loss: 4.122706566154957
159
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_16000.pt
160
+ Iteration: 20000 | Training loss: 4.092839049637318
161
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_20000.pt
162
+ Iteration: 24000 | Training loss: 4.0750753880739214
163
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_24000.pt
164
+ Iteration: 28000 | Training loss: 4.081402803361416
165
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_28000.pt
166
+ Iteration: 32000 | Training loss: 4.082153795778751
167
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_32000.pt
168
+ Iteration: 36000 | Training loss: 4.073826059103012
169
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_36000.pt
170
+ Iteration: 40000 | Training loss: 4.061313236832619
171
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_40000.pt
172
+ Iteration: 44000 | Training loss: 4.067675892353058
173
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_44000.pt
174
+ Iteration: 48000 | Training loss: 4.0533216901421545
175
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_48000.pt
176
+ Iteration: 52000 | Training loss: 4.051194758296013
177
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_52000.pt
178
+ Iteration: 56000 | Training loss: 4.056093183994293
179
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_56000.pt
180
+ Iteration: 60000 | Training loss: 4.040914334952832
181
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_60000.pt
182
+ Iteration: 64000 | Training loss: 4.0225038761496545
183
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_64000.pt
184
+ Iteration: 68000 | Training loss: 4.029447127580642
185
+ Saving model to: /scratch/eo41/vqgan-gpt/gpt_pretrained_models/y_gimel_68000.pt
186
+ slurmstepd: error: *** JOB 29968993 ON ga001 CANCELLED AT 2023-02-11T22:23:36 DUE TO TIME LIMIT ***
187
+ slurmstepd: error: *** STEP 29968993.0 ON ga001 CANCELLED AT 2023-02-11T22:23:36 DUE TO TIME LIMIT ***
188
+ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.