[2024-03-05 03:22:29 hydro_simmim_pretrain] (main_simmim_pt.py 229): INFO Full config saved to output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/config.json [2024-03-05 03:22:29 hydro_simmim_pretrain] (main_simmim_pt.py 232): INFO AMP_ENABLE: true AMP_OPT_LEVEL: '' AUG: AUTO_AUGMENT: rand-m9-mstd0.5-inc1 COLOR_JITTER: 0.4 CUTMIX: 1.0 CUTMIX_MINMAX: null MIXUP: 0.8 MIXUP_MODE: batch MIXUP_PROB: 1.0 MIXUP_SWITCH_PROB: 0.5 RECOUNT: 1 REMODE: pixel REPROB: 0.25 BASE: - '' DATA: BANDS: all BATCH_SIZE: 64 CACHE_MODE: part CHANNELS: 12 DATASET: imagenet DATA_PATH: /workspace/storage/data/hydro/images/ IMG_SIZE: 256 INTERPOLATION: bicubic MASK_PATCH_SIZE: 32 MASK_RATIO: 0.6 MEAN: - 340.76769064 - 429.9430203 - 614.21682446 - 590.23569706 - 950.68368468 - 1792.46290469 - 2075.46795189 - 2218.94553375 - 2266.46036911 - 2246.0605464 - 1594.42694882 - 1009.32729131 NUM_WORKERS: 8 PIN_MEMORY: true STD: - 554.81258967 - 572.41639287 - 582.87945694 - 675.88746967 - 729.89827633 - 1096.01480586 - 1273.45393088 - 1365.45589904 - 1356.13789355 - 1302.3292881 - 1079.19066363 - 818.86747235 ZIP_MODE: false ENABLE_AMP: true EVAL_MODE: false FUSED_LAYERNORM: false FUSED_WINDOW_PROCESS: false LOCAL_RANK: 0 MODEL: DROP_PATH_RATE: 0.1 DROP_RATE: 0.0 IN_CHANS: 3 LABEL_SMOOTHING: 0.1 NAME: hydro_simmim_pretrain NUM_CLASSES: 1000 PRETRAINED: Swin-Transformer/output/hydro_simmim_pretrain1/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_670.pth RESUME: '' SIMMIM: NORM_TARGET: ENABLE: true PATCH_SIZE: 47 SWIN: APE: false DEPTHS: - 2 - 2 - 6 - 2 EMBED_DIM: 96 IN_CHANS: 3 MLP_RATIO: 4.0 NUM_HEADS: - 3 - 6 - 12 - 24 PATCH_NORM: true PATCH_SIZE: 4 QKV_BIAS: true QK_SCALE: null WINDOW_SIZE: 7 SWINV2: APE: false DEPTHS: - 2 - 2 - 18 - 2 EMBED_DIM: 128 IN_CHANS: 12 MLP_RATIO: 4.0 NUM_HEADS: - 4 - 8 - 16 - 32 PATCH_NORM: true PATCH_SIZE: 4 PRETRAINED_WINDOW_SIZES: - 0 - 0 - 0 - 0 QKV_BIAS: true WINDOW_SIZE: 16 SWIN_MLP: APE: false DEPTHS: - 2 - 2 - 6 - 2 EMBED_DIM: 96 IN_CHANS: 3 MLP_RATIO: 4.0 NUM_HEADS: - 3 - 6 - 12 - 24 PATCH_NORM: true PATCH_SIZE: 4 WINDOW_SIZE: 7 SWIN_MOE: APE: false AUX_LOSS_WEIGHT: 0.01 CAPACITY_FACTOR: 1.25 COSINE_ROUTER: false COSINE_ROUTER_DIM: 256 COSINE_ROUTER_INIT_T: 0.5 DEPTHS: - 2 - 2 - 6 - 2 EMBED_DIM: 96 GATE_NOISE: 1.0 INIT_STD: 0.02 IN_CHANS: 3 IS_GSHARD_LOSS: false MLP_FC2_BIAS: true MLP_RATIO: 4.0 MOE_BLOCKS: - - -1 - - -1 - - -1 - - -1 MOE_DROP: 0.0 NORMALIZE_GATE: false NUM_HEADS: - 3 - 6 - 12 - 24 NUM_LOCAL_EXPERTS: 1 PATCH_NORM: true PATCH_SIZE: 4 PRETRAINED_WINDOW_SIZES: - 0 - 0 - 0 - 0 QKV_BIAS: true QK_SCALE: null TOP_VALUE: 1 USE_BPR: true WINDOW_SIZE: 7 TYPE: swinv2 OUTPUT: output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep PRINT_FREQ: 100 SAVE_FREQ: 5 SEED: 0 TAG: hydro_simmim_pretrain_swinv2_base_img256_window16_800ep TEST: CROP: true SEQUENTIAL: false SHUFFLE: false THROUGHPUT_MODE: false TRAIN: ACCUMULATION_STEPS: 1 AUTO_RESUME: true BASE_LR: 2.5e-05 CLIP_GRAD: 5.0 EPOCHS: 800 LAYER_DECAY: 1.0 LR_SCHEDULER: DECAY_EPOCHS: 30 DECAY_RATE: 0.1 GAMMA: 0.1 MULTISTEPS: - 700 NAME: multistep WARMUP_PREFIX: true MIN_LR: 1.25e-06 MOE: SAVE_MASTER: false OPTIMIZER: BETAS: - 0.9 - 0.999 EPS: 1.0e-08 MOMENTUM: 0.9 NAME: adamw START_EPOCH: 0 USE_CHECKPOINT: false WARMUP_EPOCHS: 10 WARMUP_LR: 1.25e-07 WEIGHT_DECAY: 0.05 [2024-03-05 03:22:30 hydro_simmim_pretrain] (main_simmim_pt.py 73): INFO Creating model:swinv2/hydro_simmim_pretrain [2024-03-05 03:22:31 hydro_simmim_pretrain] (main_simmim_pt.py 76): INFO SimMIM( (encoder): SwinTransformerV2ForSimMIM( (patch_embed): PatchEmbed( (proj): Conv2d(12, 128, kernel_size=(4, 4), stride=(4, 4)) (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True) ) (pos_drop): Dropout(p=0.0, inplace=False) (layers): ModuleList( (0): BasicLayer( dim=128, input_resolution=(64, 64), depth=2 (blocks): ModuleList( (0): SwinTransformerBlock( dim=128, input_resolution=(64, 64), num_heads=4, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((128,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=128, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=4 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=4, bias=False) ) (qkv): Linear(in_features=128, out_features=384, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=128, out_features=128, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): Identity() (norm2): LayerNorm((128,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=128, out_features=512, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=512, out_features=128, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (1): SwinTransformerBlock( dim=128, input_resolution=(64, 64), num_heads=4, window_size=16, shift_size=8, mlp_ratio=4.0 (norm1): LayerNorm((128,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=128, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=4 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=4, bias=False) ) (qkv): Linear(in_features=128, out_features=384, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=128, out_features=128, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.004) (norm2): LayerNorm((128,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=128, out_features=512, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=512, out_features=128, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) ) (downsample): PatchMerging( input_resolution=(64, 64), dim=128 (reduction): Linear(in_features=512, out_features=256, bias=False) (norm): LayerNorm((256,), eps=1e-05, elementwise_affine=True) ) ) (1): BasicLayer( dim=256, input_resolution=(32, 32), depth=2 (blocks): ModuleList( (0): SwinTransformerBlock( dim=256, input_resolution=(32, 32), num_heads=8, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((256,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=256, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=8 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=8, bias=False) ) (qkv): Linear(in_features=256, out_features=768, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=256, out_features=256, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.009) (norm2): LayerNorm((256,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=256, out_features=1024, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=1024, out_features=256, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (1): SwinTransformerBlock( dim=256, input_resolution=(32, 32), num_heads=8, window_size=16, shift_size=8, mlp_ratio=4.0 (norm1): LayerNorm((256,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=256, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=8 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=8, bias=False) ) (qkv): Linear(in_features=256, out_features=768, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=256, out_features=256, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.013) (norm2): LayerNorm((256,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=256, out_features=1024, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=1024, out_features=256, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) ) (downsample): PatchMerging( input_resolution=(32, 32), dim=256 (reduction): Linear(in_features=1024, out_features=512, bias=False) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) ) ) (2): BasicLayer( dim=512, input_resolution=(16, 16), depth=18 (blocks): ModuleList( (0): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.017) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (1): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.022) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (2): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.026) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (3): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.030) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (4): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.035) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (5): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.039) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (6): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.043) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (7): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.048) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (8): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.052) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (9): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.057) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (10): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.061) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (11): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.065) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (12): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.070) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (13): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.074) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (14): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.078) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (15): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.083) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (16): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.087) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (17): SwinTransformerBlock( dim=512, input_resolution=(16, 16), num_heads=16, window_size=16, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=512, window_size=(16, 16), pretrained_window_size=(0, 0), num_heads=16 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=16, bias=False) ) (qkv): Linear(in_features=512, out_features=1536, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=512, out_features=512, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.091) (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=512, out_features=2048, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=2048, out_features=512, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) ) (downsample): PatchMerging( input_resolution=(16, 16), dim=512 (reduction): Linear(in_features=2048, out_features=1024, bias=False) (norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) ) ) (3): BasicLayer( dim=1024, input_resolution=(8, 8), depth=2 (blocks): ModuleList( (0): SwinTransformerBlock( dim=1024, input_resolution=(8, 8), num_heads=32, window_size=8, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=1024, window_size=(8, 8), pretrained_window_size=(0, 0), num_heads=32 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=32, bias=False) ) (qkv): Linear(in_features=1024, out_features=3072, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=1024, out_features=1024, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.096) (norm2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=1024, out_features=4096, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=4096, out_features=1024, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) (1): SwinTransformerBlock( dim=1024, input_resolution=(8, 8), num_heads=32, window_size=8, shift_size=0, mlp_ratio=4.0 (norm1): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (attn): WindowAttention( dim=1024, window_size=(8, 8), pretrained_window_size=(0, 0), num_heads=32 (cpb_mlp): Sequential( (0): Linear(in_features=2, out_features=512, bias=True) (1): ReLU(inplace=True) (2): Linear(in_features=512, out_features=32, bias=False) ) (qkv): Linear(in_features=1024, out_features=3072, bias=False) (attn_drop): Dropout(p=0.0, inplace=False) (proj): Linear(in_features=1024, out_features=1024, bias=True) (proj_drop): Dropout(p=0.0, inplace=False) (softmax): Softmax(dim=-1) ) (drop_path): DropPath(drop_prob=0.100) (norm2): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (mlp): Mlp( (fc1): Linear(in_features=1024, out_features=4096, bias=True) (act): GELU(approximate='none') (fc2): Linear(in_features=4096, out_features=1024, bias=True) (drop): Dropout(p=0.0, inplace=False) ) ) ) ) ) (norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (avgpool): AdaptiveAvgPool1d(output_size=1) (head): Identity() ) (decoder): Sequential( (0): Conv2d(1024, 12288, kernel_size=(1, 1), stride=(1, 1)) (1): PixelShuffle(upscale_factor=32) ) ) [2024-03-05 03:22:31 hydro_simmim_pretrain] (main_simmim_pt.py 83): INFO number of params: 99507576 [2024-03-05 03:22:31 hydro_simmim_pretrain] (utils_simmim.py 84): INFO All checkpoints founded in output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep: [] [2024-03-05 03:22:31 hydro_simmim_pretrain] (main_simmim_pt.py 101): INFO no checkpoint found in output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep, ignoring auto resume [2024-03-05 03:22:31 hydro_simmim_pretrain] (main_simmim_pt.py 106): INFO Start training [2024-03-05 03:22:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [0/800][0/402] eta 0:53:26 lr 0.000000 time 7.9757 (7.9757) loss 0.8431 (0.8431) grad_norm 0.3650 (0.3650) loss_scale 65536.0000 (65536.0000) mem 29441MB [2024-03-05 03:24:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [0/800][100/402] eta 0:04:48 lr 0.000001 time 0.8761 (0.9564) loss 0.8075 (0.8295) grad_norm 0.3477 (0.3556) loss_scale 65536.0000 (65536.0000) mem 30604MB [2024-03-05 03:25:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [0/800][200/402] eta 0:03:05 lr 0.000001 time 0.8764 (0.9168) loss 0.7504 (0.8111) grad_norm 0.2313 (0.3295) loss_scale 65536.0000 (65536.0000) mem 30604MB [2024-03-05 03:27:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [0/800][300/402] eta 0:01:32 lr 0.000002 time 0.8763 (0.9035) loss 0.7551 (0.7895) grad_norm 0.0951 (0.2699) loss_scale 65536.0000 (65536.0000) mem 30604MB [2024-03-05 03:28:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [0/800][400/402] eta 0:00:01 lr 0.000003 time 0.8742 (0.8967) loss 0.6866 (0.7694) grad_norm 0.0585 (0.2214) loss_scale 65536.0000 (65536.0000) mem 30604MB [2024-03-05 03:28:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 0 training takes 0:06:00 [2024-03-05 03:28:32 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_0.pth saving...... [2024-03-05 03:28:34 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_0.pth saved !!! [2024-03-05 03:28:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [1/800][0/402] eta 0:30:17 lr 0.000003 time 4.5205 (4.5205) loss 0.7011 (0.7011) grad_norm 0.0568 (0.0568) loss_scale 65536.0000 (65536.0000) mem 30604MB [2024-03-05 03:30:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [1/800][100/402] eta 0:04:37 lr 0.000003 time 0.8770 (0.9199) loss 0.6928 (0.6886) grad_norm 0.0421 (0.0481) loss_scale 65536.0000 (65536.0000) mem 30606MB [2024-03-05 03:31:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [1/800][200/402] eta 0:03:01 lr 0.000004 time 0.8777 (0.8984) loss 0.6766 (0.6867) grad_norm 0.0319 (0.0428) loss_scale 65536.0000 (65536.0000) mem 30606MB [2024-03-05 03:33:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [1/800][300/402] eta 0:01:30 lr 0.000004 time 0.8790 (0.8911) loss 0.6875 (0.6848) grad_norm 0.0591 (0.0402) loss_scale 65536.0000 (65536.0000) mem 30606MB [2024-03-05 03:34:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [1/800][400/402] eta 0:00:01 lr 0.000005 time 0.8747 (0.8874) loss 0.6433 (0.6842) grad_norm 0.0314 (0.0394) loss_scale 65536.0000 (65536.0000) mem 30606MB [2024-03-05 03:34:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 1 training takes 0:05:56 [2024-03-05 03:34:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [2/800][0/402] eta 0:28:55 lr 0.000005 time 4.3182 (4.3182) loss 0.6686 (0.6686) grad_norm 0.0340 (0.0340) loss_scale 65536.0000 (65536.0000) mem 30606MB [2024-03-05 03:36:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [2/800][100/402] eta 0:04:34 lr 0.000006 time 0.8762 (0.9106) loss 0.7078 (0.6781) grad_norm 0.0364 (0.0364) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:37:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [2/800][200/402] eta 0:03:00 lr 0.000006 time 0.8771 (0.8936) loss 0.6653 (0.6781) grad_norm 0.0371 (0.0363) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:38:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [2/800][300/402] eta 0:01:30 lr 0.000007 time 0.8767 (0.8879) loss 0.6786 (0.6792) grad_norm 0.0391 (0.0384) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:40:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [2/800][400/402] eta 0:00:01 lr 0.000008 time 0.8749 (0.8850) loss 0.6854 (0.6791) grad_norm 0.0699 (0.0415) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:40:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 2 training takes 0:05:55 [2024-03-05 03:40:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [3/800][0/402] eta 0:28:59 lr 0.000008 time 4.3280 (4.3280) loss 0.6794 (0.6794) grad_norm 0.0367 (0.0367) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:41:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [3/800][100/402] eta 0:04:35 lr 0.000008 time 0.8761 (0.9107) loss 0.6633 (0.6758) grad_norm 0.1913 (0.0889) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:43:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [3/800][200/402] eta 0:03:00 lr 0.000009 time 0.8763 (0.8938) loss 0.6906 (0.6778) grad_norm 0.3996 (0.1124) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:44:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [3/800][300/402] eta 0:01:30 lr 0.000009 time 0.8768 (0.8882) loss 0.6884 (0.6782) grad_norm 0.2392 (0.1446) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:46:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [3/800][400/402] eta 0:00:01 lr 0.000010 time 0.8753 (0.8854) loss 0.6354 (0.6775) grad_norm 0.1608 (0.1835) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:46:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 3 training takes 0:05:56 [2024-03-05 03:46:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [4/800][0/402] eta 0:29:36 lr 0.000010 time 4.4201 (4.4201) loss 0.6744 (0.6744) grad_norm 0.4939 (0.4939) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:47:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [4/800][100/402] eta 0:04:35 lr 0.000011 time 0.8777 (0.9121) loss 0.6741 (0.6762) grad_norm 0.4582 (0.3627) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:49:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [4/800][200/402] eta 0:03:00 lr 0.000011 time 0.8771 (0.8948) loss 0.7133 (0.6746) grad_norm 0.4557 (0.4078) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:50:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [4/800][300/402] eta 0:01:30 lr 0.000012 time 0.8770 (0.8890) loss 0.6887 (0.6741) grad_norm 0.3521 (0.4563) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 03:52:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [4/800][400/402] eta 0:00:01 lr 0.000013 time 0.8760 (0.8860) loss 0.6823 (0.6736) grad_norm 0.4251 (0.4988) loss_scale 131072.0000 (67170.3142) mem 30609MB [2024-03-05 03:52:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 4 training takes 0:05:56 [2024-03-05 03:52:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [5/800][0/402] eta 0:29:03 lr 0.000013 time 4.3372 (4.3372) loss 0.6813 (0.6813) grad_norm 0.6084 (0.6084) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 03:53:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [5/800][100/402] eta 0:04:35 lr 0.000013 time 0.8778 (0.9115) loss 0.6680 (0.6752) grad_norm 1.2277 (0.5558) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 03:55:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [5/800][200/402] eta 0:03:00 lr 0.000014 time 0.8772 (0.8945) loss 0.6814 (0.6729) grad_norm 0.1999 (0.4976) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 03:56:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [5/800][300/402] eta 0:01:30 lr 0.000014 time 0.8770 (0.8888) loss 0.6698 (0.6735) grad_norm 0.7094 (0.5087) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 03:58:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [5/800][400/402] eta 0:00:01 lr 0.000015 time 0.8753 (0.8858) loss 0.6744 (0.6728) grad_norm 0.1591 (0.5065) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 03:58:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 5 training takes 0:05:56 [2024-03-05 03:58:15 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_5.pth saving...... [2024-03-05 03:58:17 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_5.pth saved !!! [2024-03-05 03:58:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [6/800][0/402] eta 0:29:52 lr 0.000015 time 4.4592 (4.4592) loss 0.6735 (0.6735) grad_norm 0.6347 (0.6347) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 03:59:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [6/800][100/402] eta 0:04:35 lr 0.000016 time 0.8768 (0.9132) loss 0.6604 (0.6729) grad_norm 0.4926 (0.4953) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:01:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [6/800][200/402] eta 0:03:00 lr 0.000016 time 0.8783 (0.8956) loss 0.6854 (0.6729) grad_norm 0.6780 (0.5193) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:02:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [6/800][300/402] eta 0:01:30 lr 0.000017 time 0.8776 (0.8897) loss 0.6605 (0.6734) grad_norm 0.1132 (0.5341) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:04:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [6/800][400/402] eta 0:00:01 lr 0.000018 time 0.8768 (0.8867) loss 0.6787 (0.6732) grad_norm 0.2066 (0.5333) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:04:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 6 training takes 0:05:56 [2024-03-05 04:04:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [7/800][0/402] eta 0:29:28 lr 0.000018 time 4.3994 (4.3994) loss 0.6558 (0.6558) grad_norm 0.4424 (0.4424) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:05:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [7/800][100/402] eta 0:04:35 lr 0.000018 time 0.8772 (0.9121) loss 0.6662 (0.6711) grad_norm 0.8556 (0.4564) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:07:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [7/800][200/402] eta 0:03:00 lr 0.000019 time 0.8772 (0.8949) loss 0.6801 (0.6709) grad_norm 0.2666 (0.4859) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:08:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [7/800][300/402] eta 0:01:30 lr 0.000019 time 0.8774 (0.8891) loss 0.6798 (0.6719) grad_norm 1.0602 (0.4566) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:10:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [7/800][400/402] eta 0:00:01 lr 0.000020 time 0.8761 (0.8861) loss 0.6607 (0.6716) grad_norm 1.5453 (0.4797) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:10:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 7 training takes 0:05:56 [2024-03-05 04:10:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [8/800][0/402] eta 0:29:29 lr 0.000020 time 4.4028 (4.4028) loss 0.6832 (0.6832) grad_norm 1.5431 (1.5431) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:11:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [8/800][100/402] eta 0:04:35 lr 0.000021 time 0.8770 (0.9122) loss 0.6553 (0.6715) grad_norm 0.8681 (0.4815) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:13:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [8/800][200/402] eta 0:03:00 lr 0.000021 time 0.8775 (0.8948) loss 0.6490 (0.6711) grad_norm 0.2154 (0.4904) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:14:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [8/800][300/402] eta 0:01:30 lr 0.000022 time 0.8772 (0.8890) loss 0.6644 (0.6720) grad_norm 0.4745 (0.4825) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:16:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [8/800][400/402] eta 0:00:01 lr 0.000023 time 0.8757 (0.8860) loss 0.7087 (0.6724) grad_norm 0.2050 (0.4884) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:16:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 8 training takes 0:05:56 [2024-03-05 04:16:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [9/800][0/402] eta 0:29:09 lr 0.000023 time 4.3517 (4.3517) loss 0.6627 (0.6627) grad_norm 0.5405 (0.5405) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:17:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [9/800][100/402] eta 0:04:35 lr 0.000023 time 0.8769 (0.9119) loss 0.6668 (0.6721) grad_norm 0.6903 (0.5485) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:19:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [9/800][200/402] eta 0:03:00 lr 0.000024 time 0.8773 (0.8947) loss 0.6610 (0.6712) grad_norm 0.8862 (0.4925) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:20:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [9/800][300/402] eta 0:01:30 lr 0.000024 time 0.8776 (0.8889) loss 0.6728 (0.6718) grad_norm 0.4626 (0.4813) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:22:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [9/800][400/402] eta 0:00:01 lr 0.000025 time 0.8755 (0.8860) loss 0.6953 (0.6712) grad_norm 0.6977 (0.4880) loss_scale 262144.0000 (137609.2569) mem 30609MB [2024-03-05 04:22:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 9 training takes 0:05:56 [2024-03-05 04:22:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [10/800][0/402] eta 0:29:26 lr 0.000025 time 4.3934 (4.3934) loss 0.6704 (0.6704) grad_norm 0.6351 (0.6351) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:23:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [10/800][100/402] eta 0:04:35 lr 0.000025 time 0.8770 (0.9122) loss 0.6530 (0.6706) grad_norm 0.1787 (0.4384) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:25:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [10/800][200/402] eta 0:03:00 lr 0.000025 time 0.8770 (0.8949) loss 0.6341 (0.6730) grad_norm 0.3482 (0.4074) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:26:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [10/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8890) loss 0.6822 (0.6730) grad_norm 0.1403 (0.4149) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:27:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [10/800][400/402] eta 0:00:01 lr 0.000025 time 0.8759 (0.8861) loss 0.6561 (0.6723) grad_norm 0.4335 (0.4111) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:27:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 10 training takes 0:05:56 [2024-03-05 04:27:59 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_10.pth saving...... [2024-03-05 04:28:00 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_10.pth saved !!! [2024-03-05 04:28:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [11/800][0/402] eta 0:29:51 lr 0.000025 time 4.4565 (4.4565) loss 0.6812 (0.6812) grad_norm 0.2388 (0.2388) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:29:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [11/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9131) loss 0.7310 (0.6736) grad_norm 0.4935 (0.3877) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:31:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [11/800][200/402] eta 0:03:00 lr 0.000025 time 0.8809 (0.8955) loss 0.6751 (0.6731) grad_norm 0.1544 (0.3918) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:32:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [11/800][300/402] eta 0:01:30 lr 0.000025 time 0.8773 (0.8897) loss 0.6344 (0.6715) grad_norm 0.1798 (0.3982) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:33:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [11/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8867) loss 0.6818 (0.6705) grad_norm 0.1896 (0.3883) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:33:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 11 training takes 0:05:56 [2024-03-05 04:34:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [12/800][0/402] eta 0:29:18 lr 0.000025 time 4.3738 (4.3738) loss 0.6685 (0.6685) grad_norm 0.7950 (0.7950) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 04:35:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [12/800][100/402] eta 0:04:35 lr 0.000025 time 0.8767 (0.9118) loss 0.6987 (0.6738) grad_norm 0.6265 (inf) loss_scale 131072.0000 (192065.9010) mem 30609MB [2024-03-05 04:36:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [12/800][200/402] eta 0:03:00 lr 0.000025 time 0.8772 (0.8946) loss 0.6639 (0.6711) grad_norm 0.1586 (inf) loss_scale 131072.0000 (161720.6766) mem 30609MB [2024-03-05 04:38:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [12/800][300/402] eta 0:01:30 lr 0.000025 time 0.8773 (0.8888) loss 0.6771 (0.6709) grad_norm 0.2257 (inf) loss_scale 131072.0000 (151538.3920) mem 30609MB [2024-03-05 04:39:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [12/800][400/402] eta 0:00:01 lr 0.000025 time 0.8753 (0.8859) loss 0.6689 (0.6711) grad_norm 0.2012 (inf) loss_scale 131072.0000 (146434.5536) mem 30609MB [2024-03-05 04:39:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 12 training takes 0:05:56 [2024-03-05 04:39:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [13/800][0/402] eta 0:29:07 lr 0.000025 time 4.3479 (4.3479) loss 0.6682 (0.6682) grad_norm 0.3268 (0.3268) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:41:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [13/800][100/402] eta 0:04:35 lr 0.000025 time 0.8774 (0.9116) loss 0.6848 (0.6710) grad_norm 0.4826 (0.3531) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:42:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [13/800][200/402] eta 0:03:00 lr 0.000025 time 0.8773 (0.8946) loss 0.6698 (0.6717) grad_norm 0.4706 (0.3922) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:44:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [13/800][300/402] eta 0:01:30 lr 0.000025 time 0.8770 (0.8889) loss 0.6883 (0.6719) grad_norm 1.0256 (0.3831) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:45:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [13/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8860) loss 0.6654 (0.6711) grad_norm 0.2795 (0.3733) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:45:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 13 training takes 0:05:56 [2024-03-05 04:45:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [14/800][0/402] eta 0:29:06 lr 0.000025 time 4.3453 (4.3453) loss 0.6740 (0.6740) grad_norm 0.2927 (0.2927) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:47:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [14/800][100/402] eta 0:04:35 lr 0.000025 time 0.8773 (0.9116) loss 0.6584 (0.6694) grad_norm 0.2846 (0.3993) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:48:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [14/800][200/402] eta 0:03:00 lr 0.000025 time 0.8771 (0.8945) loss 0.6834 (0.6714) grad_norm 0.1310 (0.4048) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:50:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [14/800][300/402] eta 0:01:30 lr 0.000025 time 0.8771 (0.8888) loss 0.6884 (0.6712) grad_norm 0.3865 (0.4084) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:51:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [14/800][400/402] eta 0:00:01 lr 0.000025 time 0.8755 (0.8859) loss 0.6634 (0.6712) grad_norm 0.6005 (0.4006) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:51:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 14 training takes 0:05:56 [2024-03-05 04:51:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [15/800][0/402] eta 0:29:08 lr 0.000025 time 4.3484 (4.3484) loss 0.6398 (0.6398) grad_norm 0.2397 (0.2397) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:53:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [15/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9127) loss 0.6535 (0.6688) grad_norm 0.1570 (0.4469) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:54:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [15/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.6580 (0.6686) grad_norm 0.5291 (0.4425) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:56:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [15/800][300/402] eta 0:01:30 lr 0.000025 time 0.8775 (0.8899) loss 0.6867 (0.6693) grad_norm 0.3214 (0.4348) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:57:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [15/800][400/402] eta 0:00:01 lr 0.000025 time 0.8755 (0.8868) loss 0.6592 (0.6698) grad_norm 0.1444 (0.4150) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:57:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 15 training takes 0:05:56 [2024-03-05 04:57:43 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_15.pth saving...... [2024-03-05 04:57:44 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_15.pth saved !!! [2024-03-05 04:57:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [16/800][0/402] eta 0:26:57 lr 0.000025 time 4.0228 (4.0228) loss 0.6970 (0.6970) grad_norm 0.2663 (0.2663) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 04:59:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [16/800][100/402] eta 0:04:34 lr 0.000025 time 0.8771 (0.9088) loss 0.6910 (0.6718) grad_norm 0.3110 (0.4412) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:00:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [16/800][200/402] eta 0:03:00 lr 0.000025 time 0.8772 (0.8931) loss 0.6432 (0.6718) grad_norm 0.4977 (0.4288) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:02:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [16/800][300/402] eta 0:01:30 lr 0.000025 time 0.8776 (0.8879) loss 0.6885 (0.6706) grad_norm 0.7050 (0.4117) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:03:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [16/800][400/402] eta 0:00:01 lr 0.000025 time 0.8759 (0.8852) loss 0.6490 (0.6710) grad_norm 0.2085 (0.3905) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:03:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 16 training takes 0:05:56 [2024-03-05 05:03:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [17/800][0/402] eta 0:29:01 lr 0.000025 time 4.3331 (4.3331) loss 0.6603 (0.6603) grad_norm 0.5176 (0.5176) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:05:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [17/800][100/402] eta 0:04:35 lr 0.000025 time 0.8772 (0.9116) loss 0.6613 (0.6704) grad_norm 1.0260 (0.3719) loss_scale 262144.0000 (214127.5248) mem 30609MB [2024-03-05 05:06:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [17/800][200/402] eta 0:03:00 lr 0.000025 time 0.8773 (0.8946) loss 0.6871 (0.6713) grad_norm 0.1209 (0.3680) loss_scale 262144.0000 (238016.3184) mem 30609MB [2024-03-05 05:08:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [17/800][300/402] eta 0:01:30 lr 0.000025 time 0.8774 (0.8889) loss 0.6625 (0.6707) grad_norm 0.2976 (inf) loss_scale 131072.0000 (206405.7409) mem 30609MB [2024-03-05 05:09:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [17/800][400/402] eta 0:00:01 lr 0.000025 time 0.8757 (0.8860) loss 0.6856 (0.6705) grad_norm 0.2883 (inf) loss_scale 131072.0000 (187619.2718) mem 30609MB [2024-03-05 05:09:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 17 training takes 0:05:56 [2024-03-05 05:09:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [18/800][0/402] eta 0:28:53 lr 0.000025 time 4.3124 (4.3124) loss 0.6686 (0.6686) grad_norm 0.4618 (0.4618) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:11:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [18/800][100/402] eta 0:04:35 lr 0.000025 time 0.8772 (0.9115) loss 0.6856 (0.6669) grad_norm 0.2043 (0.3948) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:12:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [18/800][200/402] eta 0:03:00 lr 0.000025 time 0.8772 (0.8946) loss 0.6495 (0.6679) grad_norm 1.0633 (0.4272) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:14:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [18/800][300/402] eta 0:01:30 lr 0.000025 time 0.8776 (0.8890) loss 0.6607 (0.6677) grad_norm 0.6282 (0.4271) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:15:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [18/800][400/402] eta 0:00:01 lr 0.000025 time 0.8757 (0.8861) loss 0.6917 (0.6683) grad_norm 0.3400 (0.4206) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:15:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 18 training takes 0:05:56 [2024-03-05 05:15:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [19/800][0/402] eta 0:28:56 lr 0.000025 time 4.3207 (4.3207) loss 0.6530 (0.6530) grad_norm 0.1940 (0.1940) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:17:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [19/800][100/402] eta 0:04:35 lr 0.000025 time 0.8773 (0.9117) loss 0.6325 (0.6666) grad_norm 0.5572 (0.4390) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:18:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [19/800][200/402] eta 0:03:00 lr 0.000025 time 0.8773 (0.8947) loss 0.6730 (0.6672) grad_norm 0.9689 (0.4280) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:20:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [19/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8890) loss 0.6732 (0.6672) grad_norm 0.6042 (0.4628) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:21:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [19/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8861) loss 0.6548 (0.6676) grad_norm 0.6626 (0.4518) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:21:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 19 training takes 0:05:56 [2024-03-05 05:21:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [20/800][0/402] eta 0:29:18 lr 0.000025 time 4.3747 (4.3747) loss 0.6171 (0.6171) grad_norm 1.1221 (1.1221) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:23:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [20/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9122) loss 0.6601 (0.6649) grad_norm 0.4849 (0.5740) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:24:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [20/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8950) loss 0.6626 (0.6657) grad_norm 0.3118 (0.5316) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:25:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [20/800][300/402] eta 0:01:30 lr 0.000025 time 0.8775 (0.8893) loss 0.6427 (0.6664) grad_norm 0.3324 (0.5397) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:27:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [20/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8864) loss 0.6474 (0.6661) grad_norm 0.4704 (0.5426) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:27:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 20 training takes 0:05:56 [2024-03-05 05:27:26 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_20.pth saving...... [2024-03-05 05:27:28 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_20.pth saved !!! [2024-03-05 05:27:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [21/800][0/402] eta 0:28:22 lr 0.000025 time 4.2361 (4.2361) loss 0.6858 (0.6858) grad_norm 0.3975 (0.3975) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:29:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [21/800][100/402] eta 0:04:35 lr 0.000025 time 0.8777 (0.9112) loss 0.6787 (0.6644) grad_norm 0.8069 (0.5557) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:30:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [21/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8946) loss 0.6976 (0.6650) grad_norm 0.5431 (0.6195) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:31:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [21/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8892) loss 0.6788 (0.6641) grad_norm 0.3199 (0.6612) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:33:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [21/800][400/402] eta 0:00:01 lr 0.000025 time 0.8757 (0.8865) loss 0.6656 (0.6633) grad_norm 0.5823 (0.6590) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:33:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 21 training takes 0:05:56 [2024-03-05 05:33:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [22/800][0/402] eta 0:29:05 lr 0.000025 time 4.3433 (4.3433) loss 0.6663 (0.6663) grad_norm 0.7686 (0.7686) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:34:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [22/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9121) loss 0.6680 (0.6624) grad_norm 0.5846 (0.6427) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:36:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [22/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8951) loss 0.6609 (0.6603) grad_norm 0.6969 (0.6611) loss_scale 262144.0000 (131724.0995) mem 30609MB [2024-03-05 05:37:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [22/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8893) loss 0.6400 (0.6604) grad_norm 0.5839 (inf) loss_scale 131072.0000 (144135.6545) mem 30609MB [2024-03-05 05:39:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [22/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8864) loss 0.6464 (0.6591) grad_norm 0.5241 (inf) loss_scale 131072.0000 (140877.8853) mem 30609MB [2024-03-05 05:39:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 22 training takes 0:05:56 [2024-03-05 05:39:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [23/800][0/402] eta 0:29:05 lr 0.000025 time 4.3424 (4.3424) loss 0.6614 (0.6614) grad_norm 0.5763 (0.5763) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:40:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [23/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9123) loss 0.6410 (0.6566) grad_norm 0.8547 (0.6642) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:42:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [23/800][200/402] eta 0:03:00 lr 0.000025 time 0.8776 (0.8952) loss 0.6461 (0.6571) grad_norm 0.5284 (0.6783) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:43:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [23/800][300/402] eta 0:01:30 lr 0.000025 time 0.8774 (0.8894) loss 0.6420 (0.6577) grad_norm 0.7672 (0.6810) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:45:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [23/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8865) loss 0.6504 (0.6576) grad_norm 0.8898 (0.6869) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:45:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 23 training takes 0:05:56 [2024-03-05 05:45:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [24/800][0/402] eta 0:28:48 lr 0.000025 time 4.3008 (4.3008) loss 0.6685 (0.6685) grad_norm 0.7931 (0.7931) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:46:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [24/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9119) loss 0.6616 (0.6610) grad_norm 0.7878 (0.7412) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:48:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [24/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8950) loss 0.6580 (0.6583) grad_norm 0.5632 (0.7270) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:49:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [24/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8893) loss 0.6465 (0.6575) grad_norm 1.1532 (0.7194) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:51:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [24/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8865) loss 0.6609 (0.6567) grad_norm 0.6621 (0.7310) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:51:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 24 training takes 0:05:56 [2024-03-05 05:51:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [25/800][0/402] eta 0:28:54 lr 0.000025 time 4.3141 (4.3141) loss 0.6539 (0.6539) grad_norm 0.5168 (0.5168) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:52:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [25/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9120) loss 0.6799 (0.6562) grad_norm 0.6825 (0.8046) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 05:54:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [25/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8952) loss 0.6442 (0.6541) grad_norm 0.8153 (inf) loss_scale 65536.0000 (114117.4129) mem 30609MB [2024-03-05 05:55:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [25/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8895) loss 0.6596 (0.6540) grad_norm 0.6801 (inf) loss_scale 65536.0000 (97977.4086) mem 30609MB [2024-03-05 05:57:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [25/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8866) loss 0.6685 (0.6532) grad_norm 0.5678 (inf) loss_scale 65536.0000 (89887.2818) mem 30609MB [2024-03-05 05:57:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 25 training takes 0:05:56 [2024-03-05 05:57:10 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_25.pth saving...... [2024-03-05 05:57:12 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_25.pth saved !!! [2024-03-05 05:57:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [26/800][0/402] eta 0:29:50 lr 0.000025 time 4.4528 (4.4528) loss 0.6604 (0.6604) grad_norm 0.7916 (0.7916) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 05:58:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [26/800][100/402] eta 0:04:35 lr 0.000025 time 0.8775 (0.9136) loss 0.6787 (0.6501) grad_norm 0.9750 (0.8038) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:00:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [26/800][200/402] eta 0:03:00 lr 0.000025 time 0.8776 (0.8959) loss 0.6944 (0.6505) grad_norm 0.7716 (0.7952) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:01:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [26/800][300/402] eta 0:01:30 lr 0.000025 time 0.8776 (0.8900) loss 0.6335 (0.6505) grad_norm 0.7331 (0.8061) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:03:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [26/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8869) loss 0.6511 (0.6500) grad_norm 0.7018 (0.8232) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:03:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 26 training takes 0:05:56 [2024-03-05 06:03:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [27/800][0/402] eta 0:29:14 lr 0.000025 time 4.3644 (4.3644) loss 0.6423 (0.6423) grad_norm 0.8307 (0.8307) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:04:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [27/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9125) loss 0.6565 (0.6457) grad_norm 1.0000 (0.7906) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:06:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [27/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8953) loss 0.6398 (0.6481) grad_norm 0.9628 (0.8161) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:07:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [27/800][300/402] eta 0:01:30 lr 0.000025 time 0.8771 (0.8896) loss 0.6673 (0.6492) grad_norm 0.8542 (0.8168) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:09:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [27/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8866) loss 0.6354 (0.6495) grad_norm 1.5882 (0.8175) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:09:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 27 training takes 0:05:56 [2024-03-05 06:09:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [28/800][0/402] eta 0:29:20 lr 0.000025 time 4.3787 (4.3787) loss 0.6149 (0.6149) grad_norm 1.0351 (1.0351) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:10:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [28/800][100/402] eta 0:04:35 lr 0.000025 time 0.8777 (0.9126) loss 0.6567 (0.6488) grad_norm 1.0358 (0.8117) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:12:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [28/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8953) loss 0.6686 (0.6503) grad_norm 0.6204 (0.8208) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:13:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [28/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8896) loss 0.6575 (0.6493) grad_norm 0.5315 (0.8168) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:15:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [28/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8866) loss 0.6194 (0.6482) grad_norm 1.5675 (0.8243) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:15:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 28 training takes 0:05:56 [2024-03-05 06:15:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [29/800][0/402] eta 0:29:08 lr 0.000025 time 4.3484 (4.3484) loss 0.6543 (0.6543) grad_norm 1.2180 (1.2180) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:16:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [29/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9123) loss 0.6589 (0.6491) grad_norm 0.6414 (0.8622) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:18:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [29/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8952) loss 0.6190 (0.6485) grad_norm 0.7739 (0.8439) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:19:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [29/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8895) loss 0.6549 (0.6487) grad_norm 0.9593 (0.8448) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:20:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [29/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8866) loss 0.6126 (0.6486) grad_norm 0.7555 (0.8357) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:20:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 29 training takes 0:05:56 [2024-03-05 06:21:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [30/800][0/402] eta 0:29:03 lr 0.000025 time 4.3367 (4.3367) loss 0.6005 (0.6005) grad_norm 0.8215 (0.8215) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:22:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [30/800][100/402] eta 0:04:35 lr 0.000025 time 0.8789 (0.9132) loss 0.6413 (0.6452) grad_norm 0.5641 (0.8374) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:23:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [30/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8962) loss 0.6164 (0.6461) grad_norm 0.6060 (0.8267) loss_scale 131072.0000 (85751.0846) mem 30609MB [2024-03-05 06:25:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [30/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8906) loss 0.6795 (0.6470) grad_norm 0.8386 (0.8324) loss_scale 131072.0000 (100807.8671) mem 30609MB [2024-03-05 06:26:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [30/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8877) loss 0.6276 (0.6466) grad_norm 0.6086 (0.8332) loss_scale 131072.0000 (108355.0324) mem 30609MB [2024-03-05 06:26:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 30 training takes 0:05:57 [2024-03-05 06:26:55 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_30.pth saving...... [2024-03-05 06:26:57 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_30.pth saved !!! [2024-03-05 06:27:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [31/800][0/402] eta 0:28:07 lr 0.000025 time 4.1990 (4.1990) loss 0.6451 (0.6451) grad_norm 0.9015 (0.9015) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:28:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [31/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9111) loss 0.6544 (0.6466) grad_norm 0.7731 (0.8049) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:29:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [31/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8950) loss 0.6133 (0.6459) grad_norm 0.9665 (0.8159) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:31:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [31/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8896) loss 0.6350 (0.6454) grad_norm 0.7254 (0.8143) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:32:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [31/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8867) loss 0.6671 (0.6461) grad_norm 0.8959 (0.8259) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:32:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 31 training takes 0:05:56 [2024-03-05 06:32:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [32/800][0/402] eta 0:29:13 lr 0.000025 time 4.3618 (4.3618) loss 0.6603 (0.6603) grad_norm 0.6205 (0.6205) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:34:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [32/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9127) loss 0.6500 (0.6437) grad_norm 0.8413 (0.8189) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:35:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [32/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8954) loss 0.6218 (0.6443) grad_norm 0.8139 (0.8263) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:37:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [32/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8897) loss 0.6098 (0.6441) grad_norm 0.4707 (0.8223) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:38:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [32/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8867) loss 0.6310 (0.6445) grad_norm 0.6432 (0.8088) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:38:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 32 training takes 0:05:56 [2024-03-05 06:38:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [33/800][0/402] eta 0:29:00 lr 0.000025 time 4.3307 (4.3307) loss 0.6438 (0.6438) grad_norm 0.9738 (0.9738) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:40:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [33/800][100/402] eta 0:04:35 lr 0.000025 time 0.8777 (0.9122) loss 0.6110 (0.6441) grad_norm 0.7062 (0.8595) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:41:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [33/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8952) loss 0.6415 (0.6452) grad_norm 0.6440 (0.8190) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:43:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [33/800][300/402] eta 0:01:30 lr 0.000025 time 0.8775 (0.8895) loss 0.6154 (0.6440) grad_norm 0.6182 (0.8260) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:44:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [33/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8866) loss 0.6387 (0.6450) grad_norm 0.8779 (0.8329) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:44:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 33 training takes 0:05:56 [2024-03-05 06:44:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [34/800][0/402] eta 0:29:09 lr 0.000025 time 4.3529 (4.3529) loss 0.6187 (0.6187) grad_norm 0.8940 (0.8940) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:46:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [34/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9125) loss 0.6284 (0.6433) grad_norm 0.9330 (0.8165) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 06:47:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [34/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8953) loss 0.6388 (0.6440) grad_norm 0.7617 (inf) loss_scale 65536.0000 (124877.0547) mem 30609MB [2024-03-05 06:49:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [34/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8896) loss 0.6503 (0.6446) grad_norm 0.6428 (inf) loss_scale 65536.0000 (105162.4186) mem 30609MB [2024-03-05 06:50:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [34/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8867) loss 0.6460 (0.6440) grad_norm 0.9771 (inf) loss_scale 65536.0000 (95280.5187) mem 30609MB [2024-03-05 06:50:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 34 training takes 0:05:56 [2024-03-05 06:50:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [35/800][0/402] eta 0:29:05 lr 0.000025 time 4.3414 (4.3414) loss 0.6354 (0.6354) grad_norm 0.8352 (0.8352) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:52:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [35/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9123) loss 0.6413 (0.6436) grad_norm 1.0490 (0.8038) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:53:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [35/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8953) loss 0.6432 (0.6418) grad_norm 0.6486 (0.8011) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:55:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [35/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8896) loss 0.6573 (0.6423) grad_norm 0.6169 (0.8086) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:56:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [35/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8866) loss 0.6762 (0.6430) grad_norm 0.6473 (0.8170) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:56:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 35 training takes 0:05:56 [2024-03-05 06:56:40 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_35.pth saving...... [2024-03-05 06:56:42 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_35.pth saved !!! [2024-03-05 06:56:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [36/800][0/402] eta 0:29:18 lr 0.000025 time 4.3747 (4.3747) loss 0.6423 (0.6423) grad_norm 0.7076 (0.7076) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:58:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [36/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9128) loss 0.6148 (0.6383) grad_norm 0.7647 (0.8124) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 06:59:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [36/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8955) loss 0.6484 (0.6396) grad_norm 0.8339 (0.8177) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:01:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [36/800][300/402] eta 0:01:30 lr 0.000025 time 0.8778 (0.8897) loss 0.6566 (0.6394) grad_norm 0.5498 (0.8179) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:02:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [36/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8868) loss 0.6114 (0.6396) grad_norm 1.2168 (0.8376) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:02:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 36 training takes 0:05:56 [2024-03-05 07:02:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [37/800][0/402] eta 0:29:13 lr 0.000025 time 4.3631 (4.3631) loss 0.6360 (0.6360) grad_norm 0.9668 (0.9668) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:04:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [37/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9126) loss 0.6865 (0.6389) grad_norm 0.9046 (0.8376) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:05:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [37/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8955) loss 0.6425 (0.6405) grad_norm 0.7441 (0.8055) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:07:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [37/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8897) loss 0.6376 (0.6400) grad_norm 0.9360 (0.7976) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:08:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [37/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8868) loss 0.6381 (0.6401) grad_norm 0.7321 (0.8111) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:08:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 37 training takes 0:05:56 [2024-03-05 07:08:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [38/800][0/402] eta 0:29:24 lr 0.000025 time 4.3900 (4.3900) loss 0.6071 (0.6071) grad_norm 0.9488 (0.9488) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:10:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [38/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9128) loss 0.6292 (0.6364) grad_norm 0.7643 (0.8273) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:11:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [38/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8956) loss 0.6509 (0.6386) grad_norm 0.7125 (0.8189) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:13:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [38/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.6333 (0.6381) grad_norm 0.8735 (0.8064) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:14:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [38/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8868) loss 0.6485 (0.6385) grad_norm 0.7962 (0.8098) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:14:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 38 training takes 0:05:56 [2024-03-05 07:14:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [39/800][0/402] eta 0:29:06 lr 0.000025 time 4.3452 (4.3452) loss 0.6252 (0.6252) grad_norm 0.8809 (0.8809) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:16:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [39/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9125) loss 0.6538 (0.6401) grad_norm 0.7796 (0.8253) loss_scale 65536.0000 (65536.0000) mem 30609MB [2024-03-05 07:17:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [39/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8954) loss 0.6045 (0.6399) grad_norm 0.6868 (0.8054) loss_scale 131072.0000 (74991.4428) mem 30609MB [2024-03-05 07:18:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [39/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8898) loss 0.6354 (0.6399) grad_norm 0.7569 (0.8032) loss_scale 131072.0000 (93622.8571) mem 30609MB [2024-03-05 07:20:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [39/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8868) loss 0.6362 (0.6391) grad_norm 0.9542 (0.8056) loss_scale 131072.0000 (102961.7955) mem 30609MB [2024-03-05 07:20:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 39 training takes 0:05:56 [2024-03-05 07:20:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [40/800][0/402] eta 0:29:15 lr 0.000025 time 4.3664 (4.3664) loss 0.6491 (0.6491) grad_norm 0.7887 (0.7887) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:22:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [40/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9128) loss 0.6275 (0.6367) grad_norm 0.8530 (0.8401) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:23:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [40/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8956) loss 0.6180 (0.6381) grad_norm 1.2451 (0.8274) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:24:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [40/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8898) loss 0.6215 (0.6384) grad_norm 0.7462 (0.8048) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:26:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [40/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8869) loss 0.6475 (0.6385) grad_norm 0.7979 (0.7941) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:26:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 40 training takes 0:05:56 [2024-03-05 07:26:25 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_40.pth saving...... [2024-03-05 07:26:27 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_40.pth saved !!! [2024-03-05 07:26:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [41/800][0/402] eta 0:29:16 lr 0.000025 time 4.3695 (4.3695) loss 0.6354 (0.6354) grad_norm 0.5917 (0.5917) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:27:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [41/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9128) loss 0.6626 (0.6364) grad_norm 0.5969 (0.7996) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:29:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [41/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8958) loss 0.6405 (0.6375) grad_norm 0.5831 (0.7907) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:30:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [41/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8901) loss 0.6622 (0.6368) grad_norm 0.7905 (0.7757) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:32:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [41/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8872) loss 0.6288 (0.6357) grad_norm 0.8036 (0.7808) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:32:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 41 training takes 0:05:56 [2024-03-05 07:32:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [42/800][0/402] eta 0:29:21 lr 0.000025 time 4.3820 (4.3820) loss 0.6529 (0.6529) grad_norm 0.6426 (0.6426) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:33:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [42/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9129) loss 0.6262 (0.6396) grad_norm 0.6001 (0.7567) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:35:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [42/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8957) loss 0.6628 (0.6395) grad_norm 0.9193 (0.7631) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:36:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [42/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8899) loss 0.6369 (0.6382) grad_norm 0.5632 (0.7635) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:38:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [42/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8869) loss 0.6102 (0.6374) grad_norm 1.1429 (0.7608) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:38:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 42 training takes 0:05:56 [2024-03-05 07:38:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [43/800][0/402] eta 0:29:06 lr 0.000025 time 4.3442 (4.3442) loss 0.6194 (0.6194) grad_norm 0.9580 (0.9580) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:39:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [43/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9125) loss 0.6381 (0.6355) grad_norm 0.7387 (0.7754) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:41:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [43/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8954) loss 0.6647 (0.6362) grad_norm 0.6433 (0.7669) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:42:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [43/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.5959 (0.6359) grad_norm 0.7508 (0.7715) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:44:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [43/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8868) loss 0.6346 (0.6357) grad_norm 0.7347 (0.7521) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:44:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 43 training takes 0:05:56 [2024-03-05 07:44:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [44/800][0/402] eta 0:29:15 lr 0.000025 time 4.3675 (4.3675) loss 0.6648 (0.6648) grad_norm 0.7337 (0.7337) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:45:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [44/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9127) loss 0.6620 (0.6381) grad_norm 0.7630 (0.7703) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:47:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [44/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8955) loss 0.6055 (0.6361) grad_norm 0.7316 (inf) loss_scale 131072.0000 (137592.9950) mem 30609MB [2024-03-05 07:48:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [44/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8898) loss 0.6470 (0.6359) grad_norm 0.6510 (inf) loss_scale 131072.0000 (135426.5515) mem 30609MB [2024-03-05 07:50:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [44/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8868) loss 0.6274 (0.6359) grad_norm 0.9127 (inf) loss_scale 131072.0000 (134340.6284) mem 30609MB [2024-03-05 07:50:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 44 training takes 0:05:56 [2024-03-05 07:50:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [45/800][0/402] eta 0:29:06 lr 0.000025 time 4.3456 (4.3456) loss 0.6320 (0.6320) grad_norm 0.8023 (0.8023) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:51:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [45/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9125) loss 0.6492 (0.6309) grad_norm 0.7453 (0.7105) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:53:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [45/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8954) loss 0.6668 (0.6329) grad_norm 0.8098 (0.7155) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:54:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [45/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8897) loss 0.6317 (0.6340) grad_norm 0.6669 (0.7124) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:56:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [45/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8868) loss 0.6202 (0.6341) grad_norm 0.6159 (0.7159) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:56:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 45 training takes 0:05:56 [2024-03-05 07:56:10 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_45.pth saving...... [2024-03-05 07:56:12 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_45.pth saved !!! [2024-03-05 07:56:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [46/800][0/402] eta 0:28:51 lr 0.000025 time 4.3079 (4.3079) loss 0.6697 (0.6697) grad_norm 0.7071 (0.7071) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:57:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [46/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9124) loss 0.6554 (0.6362) grad_norm 0.8591 (0.7626) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 07:59:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [46/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8954) loss 0.6120 (0.6353) grad_norm 0.5592 (0.7281) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:00:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [46/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8897) loss 0.6697 (0.6347) grad_norm 0.6558 (0.7259) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:02:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [46/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8868) loss 0.6134 (0.6340) grad_norm 0.6954 (0.7215) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:02:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 46 training takes 0:05:56 [2024-03-05 08:02:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [47/800][0/402] eta 0:26:35 lr 0.000025 time 3.9691 (3.9691) loss 0.5887 (0.5887) grad_norm 0.5384 (0.5384) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:03:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [47/800][100/402] eta 0:04:34 lr 0.000025 time 0.8781 (0.9088) loss 0.6803 (0.6354) grad_norm 0.6099 (0.7450) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:05:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [47/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8936) loss 0.5909 (0.6331) grad_norm 0.5351 (0.7172) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:06:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [47/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8884) loss 0.6069 (0.6329) grad_norm 0.8318 (0.7168) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:08:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [47/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8858) loss 0.6218 (0.6326) grad_norm 0.6779 (0.7144) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:08:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 47 training takes 0:05:56 [2024-03-05 08:08:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [48/800][0/402] eta 0:29:19 lr 0.000025 time 4.3777 (4.3777) loss 0.6411 (0.6411) grad_norm 0.5850 (0.5850) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:09:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [48/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9129) loss 0.6377 (0.6323) grad_norm 0.7850 (0.7186) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:11:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [48/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8958) loss 0.6052 (0.6324) grad_norm 0.8347 (0.7019) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:12:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [48/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.6388 (0.6326) grad_norm 0.5826 (0.7029) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:14:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [48/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.6277 (0.6325) grad_norm 1.0085 (0.7053) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:14:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 48 training takes 0:05:56 [2024-03-05 08:14:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [49/800][0/402] eta 0:29:17 lr 0.000025 time 4.3723 (4.3723) loss 0.6227 (0.6227) grad_norm 0.7912 (0.7912) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:15:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [49/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9128) loss 0.6133 (0.6307) grad_norm 0.8490 (0.6725) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:17:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [49/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8956) loss 0.6273 (0.6306) grad_norm 0.8897 (0.6879) loss_scale 262144.0000 (156503.8806) mem 30609MB [2024-03-05 08:18:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [49/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8899) loss 0.6769 (0.6311) grad_norm 0.7390 (0.6956) loss_scale 262144.0000 (191600.2658) mem 30609MB [2024-03-05 08:19:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [49/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.6268 (0.6315) grad_norm 0.7149 (0.6946) loss_scale 262144.0000 (209192.2195) mem 30609MB [2024-03-05 08:19:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 49 training takes 0:05:56 [2024-03-05 08:20:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [50/800][0/402] eta 0:29:02 lr 0.000025 time 4.3356 (4.3356) loss 0.6420 (0.6420) grad_norm 0.6844 (0.6844) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 08:21:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [50/800][100/402] eta 0:04:35 lr 0.000025 time 0.8790 (0.9123) loss 0.5957 (0.6331) grad_norm 0.7630 (inf) loss_scale 131072.0000 (144049.4257) mem 30609MB [2024-03-05 08:22:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [50/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8954) loss 0.6420 (0.6318) grad_norm 0.7854 (inf) loss_scale 131072.0000 (137592.9950) mem 30609MB [2024-03-05 08:24:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [50/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.6248 (0.6314) grad_norm 0.6592 (inf) loss_scale 131072.0000 (135426.5515) mem 30609MB [2024-03-05 08:25:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [50/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.6243 (0.6314) grad_norm 0.6017 (inf) loss_scale 131072.0000 (134340.6284) mem 30609MB [2024-03-05 08:25:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 50 training takes 0:05:56 [2024-03-05 08:25:55 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_50.pth saving...... [2024-03-05 08:25:56 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_50.pth saved !!! [2024-03-05 08:26:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [51/800][0/402] eta 0:29:39 lr 0.000025 time 4.4258 (4.4258) loss 0.6320 (0.6320) grad_norm 0.7584 (0.7584) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:27:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [51/800][100/402] eta 0:04:35 lr 0.000025 time 0.8794 (0.9134) loss 0.6124 (0.6313) grad_norm 0.6302 (0.6917) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:28:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [51/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8959) loss 0.6078 (0.6293) grad_norm 0.6777 (0.6698) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:30:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [51/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8901) loss 0.6295 (0.6300) grad_norm 0.8268 (0.6797) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:31:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [51/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8871) loss 0.6220 (0.6302) grad_norm 0.7155 (0.6781) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:31:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 51 training takes 0:05:56 [2024-03-05 08:31:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [52/800][0/402] eta 0:29:12 lr 0.000025 time 4.3597 (4.3597) loss 0.6409 (0.6409) grad_norm 0.7221 (0.7221) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:33:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [52/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9127) loss 0.6092 (0.6307) grad_norm 0.5618 (0.6983) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:34:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [52/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8956) loss 0.6409 (0.6305) grad_norm 0.6199 (0.6840) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:36:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [52/800][300/402] eta 0:01:30 lr 0.000025 time 0.8776 (0.8898) loss 0.6240 (0.6304) grad_norm 0.7213 (0.6762) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:37:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [52/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8869) loss 0.6295 (0.6303) grad_norm 0.5399 (0.6745) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:37:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 52 training takes 0:05:56 [2024-03-05 08:37:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [53/800][0/402] eta 0:29:13 lr 0.000025 time 4.3612 (4.3612) loss 0.5711 (0.5711) grad_norm 0.7317 (0.7317) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:39:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [53/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9127) loss 0.6439 (0.6276) grad_norm 0.5920 (0.6793) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:40:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [53/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8956) loss 0.6465 (0.6295) grad_norm 0.6496 (0.6735) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:42:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [53/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.6410 (0.6288) grad_norm 0.6472 (0.6781) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:43:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [53/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.5992 (0.6286) grad_norm 0.7089 (0.6716) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:43:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 53 training takes 0:05:56 [2024-03-05 08:43:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [54/800][0/402] eta 0:29:24 lr 0.000025 time 4.3891 (4.3891) loss 0.6224 (0.6224) grad_norm 0.6366 (0.6366) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:45:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [54/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9129) loss 0.6431 (0.6294) grad_norm 0.5374 (0.6454) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:46:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [54/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8956) loss 0.6311 (0.6287) grad_norm 0.6950 (0.6531) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:48:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [54/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8898) loss 0.6061 (0.6291) grad_norm 0.5256 (0.6492) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:49:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [54/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8869) loss 0.5921 (0.6295) grad_norm 0.7235 (0.6601) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:49:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 54 training takes 0:05:56 [2024-03-05 08:49:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [55/800][0/402] eta 0:29:18 lr 0.000025 time 4.3750 (4.3750) loss 0.6215 (0.6215) grad_norm 0.8397 (0.8397) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 08:51:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [55/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9130) loss 0.6316 (0.6264) grad_norm 0.6703 (inf) loss_scale 131072.0000 (212829.7822) mem 30609MB [2024-03-05 08:52:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [55/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8957) loss 0.6582 (0.6265) grad_norm 0.6251 (inf) loss_scale 131072.0000 (172154.2687) mem 30609MB [2024-03-05 08:54:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [55/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8899) loss 0.6182 (0.6272) grad_norm 0.8214 (inf) loss_scale 131072.0000 (158505.6744) mem 30609MB [2024-03-05 08:55:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [55/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8870) loss 0.6077 (0.6275) grad_norm 0.7398 (inf) loss_scale 131072.0000 (151664.3591) mem 30609MB [2024-03-05 08:55:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 55 training takes 0:05:56 [2024-03-05 08:55:40 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_55.pth saving...... [2024-03-05 08:55:42 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_55.pth saved !!! [2024-03-05 08:55:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [56/800][0/402] eta 0:29:35 lr 0.000025 time 4.4165 (4.4165) loss 0.6008 (0.6008) grad_norm 0.5773 (0.5773) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:57:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [56/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9132) loss 0.6181 (0.6243) grad_norm 0.7053 (0.6318) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 08:58:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [56/800][200/402] eta 0:03:00 lr 0.000025 time 0.8777 (0.8959) loss 0.6517 (0.6276) grad_norm 0.6583 (0.6532) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:00:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [56/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8901) loss 0.6512 (0.6265) grad_norm 0.5847 (0.6586) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:01:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [56/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8871) loss 0.6301 (0.6265) grad_norm 0.5848 (0.6572) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:01:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 56 training takes 0:05:56 [2024-03-05 09:01:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [57/800][0/402] eta 0:29:14 lr 0.000025 time 4.3651 (4.3651) loss 0.6269 (0.6269) grad_norm 0.6076 (0.6076) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:03:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [57/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9128) loss 0.6117 (0.6227) grad_norm 0.6321 (0.6323) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:04:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [57/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8957) loss 0.5942 (0.6247) grad_norm 0.6661 (0.6430) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:06:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [57/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.6348 (0.6260) grad_norm 0.5926 (0.6447) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:07:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [57/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8870) loss 0.6201 (0.6258) grad_norm 0.4450 (0.6435) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:07:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 57 training takes 0:05:56 [2024-03-05 09:07:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [58/800][0/402] eta 0:29:04 lr 0.000025 time 4.3383 (4.3383) loss 0.6522 (0.6522) grad_norm 0.7358 (0.7358) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:09:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [58/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9124) loss 0.6050 (0.6290) grad_norm 0.7027 (0.6334) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:10:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [58/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8954) loss 0.6094 (0.6279) grad_norm 0.6326 (0.6477) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:12:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [58/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8897) loss 0.6171 (0.6273) grad_norm 0.5795 (0.6442) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:13:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [58/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8868) loss 0.6099 (0.6267) grad_norm 0.6499 (0.6450) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:13:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 58 training takes 0:05:56 [2024-03-05 09:13:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [59/800][0/402] eta 0:28:47 lr 0.000025 time 4.2974 (4.2974) loss 0.6234 (0.6234) grad_norm 0.5817 (0.5817) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:15:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [59/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9121) loss 0.6190 (0.6285) grad_norm 0.5318 (0.6221) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:16:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [59/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8953) loss 0.5981 (0.6269) grad_norm 0.5150 (0.6309) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:18:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [59/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8897) loss 0.6454 (0.6266) grad_norm 0.5486 (0.6349) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:19:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [59/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8868) loss 0.6308 (0.6264) grad_norm 0.6268 (0.6386) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:19:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 59 training takes 0:05:56 [2024-03-05 09:19:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [60/800][0/402] eta 0:29:19 lr 0.000025 time 4.3767 (4.3767) loss 0.6162 (0.6162) grad_norm 0.6390 (0.6390) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 09:21:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [60/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9128) loss 0.6327 (0.6268) grad_norm 0.5879 (0.6268) loss_scale 262144.0000 (193363.6436) mem 30609MB [2024-03-05 09:22:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [60/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8956) loss 0.6079 (0.6253) grad_norm 0.5964 (0.6324) loss_scale 262144.0000 (227582.7264) mem 30609MB [2024-03-05 09:23:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [60/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.6387 (0.6244) grad_norm 0.6070 (0.6301) loss_scale 262144.0000 (239064.8771) mem 30609MB [2024-03-05 09:25:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [60/800][400/402] eta 0:00:01 lr 0.000025 time 0.8759 (0.8869) loss 0.6333 (0.6247) grad_norm 0.6792 (0.6274) loss_scale 262144.0000 (244820.2693) mem 30609MB [2024-03-05 09:25:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 60 training takes 0:05:56 [2024-03-05 09:25:25 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_60.pth saving...... [2024-03-05 09:25:27 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_60.pth saved !!! [2024-03-05 09:25:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [61/800][0/402] eta 0:29:03 lr 0.000025 time 4.3358 (4.3358) loss 0.5703 (0.5703) grad_norm 0.4778 (0.4778) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:26:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [61/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9126) loss 0.6286 (0.6265) grad_norm 0.6313 (0.6464) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:28:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [61/800][200/402] eta 0:03:00 lr 0.000025 time 0.8777 (0.8956) loss 0.6281 (0.6250) grad_norm 0.5689 (0.6244) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:29:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [61/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.6392 (0.6248) grad_norm 0.5773 (0.6186) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:31:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [61/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.6458 (0.6253) grad_norm 0.5508 (0.6110) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:31:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 61 training takes 0:05:56 [2024-03-05 09:31:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [62/800][0/402] eta 0:28:31 lr 0.000025 time 4.2583 (4.2583) loss 0.6188 (0.6188) grad_norm 0.5029 (0.5029) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:32:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [62/800][100/402] eta 0:04:35 lr 0.000025 time 0.8796 (0.9121) loss 0.6193 (0.6238) grad_norm 0.5866 (0.6109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:34:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [62/800][200/402] eta 0:03:00 lr 0.000025 time 0.8787 (0.8958) loss 0.6237 (0.6245) grad_norm 0.5706 (0.6190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:35:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [62/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8903) loss 0.6211 (0.6234) grad_norm 0.6738 (0.6158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:37:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [62/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8875) loss 0.5949 (0.6234) grad_norm 0.6066 (0.6070) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:37:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 62 training takes 0:05:56 [2024-03-05 09:37:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [63/800][0/402] eta 0:29:13 lr 0.000025 time 4.3609 (4.3609) loss 0.6286 (0.6286) grad_norm 0.5145 (0.5145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:38:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [63/800][100/402] eta 0:04:35 lr 0.000025 time 0.8798 (0.9138) loss 0.5968 (0.6231) grad_norm 0.5855 (0.6127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:40:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [63/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8967) loss 0.6408 (0.6236) grad_norm 0.5703 (0.6110) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:41:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [63/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8909) loss 0.6422 (0.6244) grad_norm 0.6909 (0.6041) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:43:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [63/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8880) loss 0.6109 (0.6235) grad_norm 0.5252 (0.5968) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:43:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 63 training takes 0:05:57 [2024-03-05 09:43:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [64/800][0/402] eta 0:29:10 lr 0.000025 time 4.3533 (4.3533) loss 0.6215 (0.6215) grad_norm 0.6200 (0.6200) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:44:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [64/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9127) loss 0.6085 (0.6262) grad_norm 0.6768 (0.6026) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:46:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [64/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.6229 (0.6264) grad_norm 0.5151 (0.5961) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:47:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [64/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.5702 (0.6246) grad_norm 0.5815 (0.5990) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:49:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [64/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8870) loss 0.5970 (0.6232) grad_norm 0.4764 (0.5955) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:49:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 64 training takes 0:05:56 [2024-03-05 09:49:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [65/800][0/402] eta 0:28:56 lr 0.000025 time 4.3189 (4.3189) loss 0.6183 (0.6183) grad_norm 0.5086 (0.5086) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:50:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [65/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9121) loss 0.6250 (0.6229) grad_norm 0.6464 (inf) loss_scale 262144.0000 (264739.4851) mem 30609MB [2024-03-05 09:52:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [65/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8952) loss 0.6086 (0.6233) grad_norm 0.5613 (inf) loss_scale 262144.0000 (263448.1990) mem 30609MB [2024-03-05 09:53:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [65/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8896) loss 0.6313 (0.6225) grad_norm 0.6013 (inf) loss_scale 262144.0000 (263014.9103) mem 30609MB [2024-03-05 09:55:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [65/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8867) loss 0.6111 (0.6224) grad_norm 0.5258 (inf) loss_scale 262144.0000 (262797.7257) mem 30609MB [2024-03-05 09:55:11 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 65 training takes 0:05:56 [2024-03-05 09:55:11 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_65.pth saving...... [2024-03-05 09:55:13 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_65.pth saved !!! [2024-03-05 09:55:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [66/800][0/402] eta 0:29:55 lr 0.000025 time 4.4652 (4.4652) loss 0.6355 (0.6355) grad_norm 0.6422 (0.6422) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:56:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [66/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9137) loss 0.5950 (0.6205) grad_norm 0.4965 (0.5620) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:58:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [66/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8961) loss 0.6140 (0.6219) grad_norm 0.5148 (0.5712) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 09:59:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [66/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8903) loss 0.5916 (0.6217) grad_norm 0.5808 (0.5743) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:01:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [66/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8873) loss 0.5878 (0.6217) grad_norm 0.5964 (0.5780) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:01:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 66 training takes 0:05:56 [2024-03-05 10:01:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [67/800][0/402] eta 0:28:17 lr 0.000025 time 4.2230 (4.2230) loss 0.6222 (0.6222) grad_norm 0.5316 (0.5316) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:02:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [67/800][100/402] eta 0:04:35 lr 0.000025 time 0.8777 (0.9114) loss 0.6153 (0.6185) grad_norm 0.5386 (0.5863) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:04:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [67/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8949) loss 0.6549 (0.6212) grad_norm 0.6813 (0.5820) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:05:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [67/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8894) loss 0.6038 (0.6205) grad_norm 0.5407 (0.5831) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:07:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [67/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8866) loss 0.6172 (0.6215) grad_norm 0.5398 (0.5817) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:07:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 67 training takes 0:05:56 [2024-03-05 10:07:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [68/800][0/402] eta 0:28:57 lr 0.000025 time 4.3216 (4.3216) loss 0.6353 (0.6353) grad_norm 0.5653 (0.5653) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:08:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [68/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9121) loss 0.6178 (0.6217) grad_norm 0.5973 (nan) loss_scale 131072.0000 (163515.5644) mem 30609MB [2024-03-05 10:10:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [68/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8953) loss 0.6374 (0.6205) grad_norm 0.5260 (nan) loss_scale 131072.0000 (147374.4876) mem 30609MB [2024-03-05 10:11:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [68/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8897) loss 0.6565 (0.6209) grad_norm 0.5106 (nan) loss_scale 131072.0000 (141958.3787) mem 30609MB [2024-03-05 10:13:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [68/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8868) loss 0.5997 (0.6215) grad_norm 0.5487 (nan) loss_scale 131072.0000 (139243.5711) mem 30609MB [2024-03-05 10:13:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 68 training takes 0:05:56 [2024-03-05 10:13:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [69/800][0/402] eta 0:29:10 lr 0.000025 time 4.3554 (4.3554) loss 0.6289 (0.6289) grad_norm 0.4832 (0.4832) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:14:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [69/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9126) loss 0.5922 (0.6222) grad_norm 0.4927 (0.5380) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:16:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [69/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8955) loss 0.6589 (0.6215) grad_norm 0.3914 (0.5588) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:17:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [69/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8898) loss 0.6146 (0.6214) grad_norm 0.6308 (0.5686) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:18:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [69/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.6420 (0.6207) grad_norm 0.5347 (0.5716) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:18:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 69 training takes 0:05:56 [2024-03-05 10:19:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [70/800][0/402] eta 0:29:09 lr 0.000025 time 4.3523 (4.3523) loss 0.6022 (0.6022) grad_norm 0.5170 (0.5170) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:20:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [70/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9126) loss 0.6043 (0.6217) grad_norm 0.5510 (0.5395) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:22:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [70/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8955) loss 0.6228 (0.6210) grad_norm 0.6113 (0.5450) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:23:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [70/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.6554 (0.6213) grad_norm 0.5269 (0.5525) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:24:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [70/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.6182 (0.6220) grad_norm 0.5051 (0.5505) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:24:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 70 training takes 0:05:56 [2024-03-05 10:24:56 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_70.pth saving...... [2024-03-05 10:24:58 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_70.pth saved !!! [2024-03-05 10:25:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [71/800][0/402] eta 0:29:23 lr 0.000025 time 4.3857 (4.3857) loss 0.6571 (0.6571) grad_norm 0.5172 (0.5172) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:26:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [71/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9131) loss 0.6407 (0.6209) grad_norm 0.5145 (0.5638) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:27:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [71/800][200/402] eta 0:03:00 lr 0.000025 time 0.8792 (0.8959) loss 0.6093 (0.6213) grad_norm 0.6150 (0.5512) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:29:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [71/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8901) loss 0.6192 (0.6206) grad_norm 0.5848 (0.5473) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:30:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [71/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8871) loss 0.6402 (0.6203) grad_norm 0.6877 (0.5475) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:30:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 71 training takes 0:05:56 [2024-03-05 10:30:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [72/800][0/402] eta 0:29:06 lr 0.000025 time 4.3456 (4.3456) loss 0.6237 (0.6237) grad_norm 0.6660 (0.6660) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:32:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [72/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9126) loss 0.6383 (0.6247) grad_norm 0.5724 (0.5618) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:33:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [72/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.6173 (0.6217) grad_norm 0.5590 (0.5557) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:35:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [72/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8898) loss 0.6091 (0.6215) grad_norm 0.4980 (0.5433) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:36:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [72/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.6041 (0.6202) grad_norm 0.4155 (0.5465) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:36:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 72 training takes 0:05:56 [2024-03-05 10:36:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [73/800][0/402] eta 0:29:04 lr 0.000025 time 4.3392 (4.3392) loss 0.6189 (0.6189) grad_norm 0.4796 (0.4796) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 10:38:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [73/800][100/402] eta 0:04:35 lr 0.000025 time 0.8794 (0.9128) loss 0.6113 (0.6198) grad_norm 0.5793 (0.5260) loss_scale 262144.0000 (242677.8614) mem 30609MB [2024-03-05 10:39:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [73/800][200/402] eta 0:03:01 lr 0.000025 time 0.8795 (0.8962) loss 0.6284 (0.6215) grad_norm 0.4722 (0.5304) loss_scale 262144.0000 (252362.5075) mem 30609MB [2024-03-05 10:41:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [73/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8907) loss 0.5800 (0.6201) grad_norm 0.5060 (0.5424) loss_scale 262144.0000 (255612.1728) mem 30609MB [2024-03-05 10:42:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [73/800][400/402] eta 0:00:01 lr 0.000025 time 0.8781 (0.8878) loss 0.6308 (0.6202) grad_norm 0.5133 (0.5401) loss_scale 262144.0000 (257241.0574) mem 30609MB [2024-03-05 10:42:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 73 training takes 0:05:57 [2024-03-05 10:42:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [74/800][0/402] eta 0:29:39 lr 0.000025 time 4.4259 (4.4259) loss 0.6214 (0.6214) grad_norm 0.4539 (0.4539) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:44:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [74/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9134) loss 0.6339 (0.6210) grad_norm 0.5137 (0.5370) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:45:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [74/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8960) loss 0.6417 (0.6196) grad_norm 0.5981 (0.5413) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:47:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [74/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8901) loss 0.6174 (0.6195) grad_norm 0.4739 (0.5408) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:48:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [74/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8871) loss 0.5957 (0.6181) grad_norm 0.4353 (0.5378) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:48:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 74 training takes 0:05:56 [2024-03-05 10:48:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [75/800][0/402] eta 0:29:21 lr 0.000025 time 4.3822 (4.3822) loss 0.6411 (0.6411) grad_norm 0.4971 (0.4971) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:50:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [75/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9131) loss 0.6491 (0.6202) grad_norm 0.5198 (0.5354) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:51:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [75/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8958) loss 0.6272 (0.6192) grad_norm 0.7097 (0.5374) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:53:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [75/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8900) loss 0.6188 (0.6198) grad_norm 0.5423 (0.5304) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:54:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [75/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8871) loss 0.6078 (0.6190) grad_norm 0.5323 (0.5275) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:54:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 75 training takes 0:05:56 [2024-03-05 10:54:42 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_75.pth saving...... [2024-03-05 10:54:44 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_75.pth saved !!! [2024-03-05 10:54:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [76/800][0/402] eta 0:29:38 lr 0.000025 time 4.4239 (4.4239) loss 0.6421 (0.6421) grad_norm 0.4906 (0.4906) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:56:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [76/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9134) loss 0.6206 (0.6183) grad_norm 0.7218 (0.5125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:57:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [76/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8959) loss 0.6289 (0.6191) grad_norm 0.5782 (0.5197) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 10:59:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [76/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8901) loss 0.6114 (0.6178) grad_norm 0.3981 (0.5196) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:00:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [76/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8871) loss 0.6522 (0.6179) grad_norm 0.4507 (0.5163) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:00:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 76 training takes 0:05:56 [2024-03-05 11:00:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [77/800][0/402] eta 0:28:48 lr 0.000025 time 4.2992 (4.2992) loss 0.6212 (0.6212) grad_norm 0.5471 (0.5471) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:02:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [77/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9123) loss 0.6187 (0.6207) grad_norm 0.5880 (0.5119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:03:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [77/800][200/402] eta 0:03:00 lr 0.000025 time 0.8776 (0.8954) loss 0.6097 (0.6186) grad_norm 0.4380 (0.5214) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:05:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [77/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8897) loss 0.6166 (0.6180) grad_norm 0.5092 (0.5192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:06:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [77/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8868) loss 0.6369 (0.6182) grad_norm 0.4337 (0.5175) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:06:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 77 training takes 0:05:56 [2024-03-05 11:06:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [78/800][0/402] eta 0:29:08 lr 0.000025 time 4.3497 (4.3497) loss 0.6166 (0.6166) grad_norm 0.4991 (0.4991) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:08:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [78/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9126) loss 0.6139 (0.6150) grad_norm 0.6109 (inf) loss_scale 262144.0000 (264739.4851) mem 30609MB [2024-03-05 11:09:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [78/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.6248 (0.6149) grad_norm 0.5312 (inf) loss_scale 262144.0000 (263448.1990) mem 30609MB [2024-03-05 11:11:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [78/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8898) loss 0.6032 (0.6149) grad_norm 0.4334 (inf) loss_scale 262144.0000 (263014.9103) mem 30609MB [2024-03-05 11:12:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [78/800][400/402] eta 0:00:01 lr 0.000025 time 0.8785 (0.8869) loss 0.6347 (0.6160) grad_norm 0.5135 (inf) loss_scale 262144.0000 (262797.7257) mem 30609MB [2024-03-05 11:12:34 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 78 training takes 0:05:56 [2024-03-05 11:12:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [79/800][0/402] eta 0:29:11 lr 0.000025 time 4.3576 (4.3576) loss 0.6360 (0.6360) grad_norm 0.3937 (0.3937) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:14:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [79/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9126) loss 0.6030 (0.6174) grad_norm 0.5541 (0.5179) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:15:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [79/800][200/402] eta 0:03:00 lr 0.000025 time 0.8789 (0.8956) loss 0.6058 (0.6190) grad_norm 0.4084 (0.5165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:17:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [79/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8899) loss 0.6269 (0.6180) grad_norm 0.4814 (0.5141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:18:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [79/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.5913 (0.6178) grad_norm 0.5036 (0.5104) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:18:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 79 training takes 0:05:56 [2024-03-05 11:18:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [80/800][0/402] eta 0:29:04 lr 0.000025 time 4.3386 (4.3386) loss 0.5882 (0.5882) grad_norm 0.5101 (0.5101) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:20:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [80/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9125) loss 0.6226 (0.6140) grad_norm 0.3552 (0.4992) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:21:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [80/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8955) loss 0.6102 (0.6148) grad_norm 0.4775 (0.4976) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:22:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [80/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8898) loss 0.6096 (0.6156) grad_norm 0.4905 (0.4947) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:24:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [80/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8869) loss 0.6160 (0.6167) grad_norm 0.5079 (0.4950) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:24:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 80 training takes 0:05:56 [2024-03-05 11:24:27 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_80.pth saving...... [2024-03-05 11:24:29 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_80.pth saved !!! [2024-03-05 11:24:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [81/800][0/402] eta 0:29:28 lr 0.000025 time 4.4002 (4.4002) loss 0.5997 (0.5997) grad_norm 0.5335 (0.5335) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:26:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [81/800][100/402] eta 0:04:36 lr 0.000025 time 0.8794 (0.9144) loss 0.6226 (0.6148) grad_norm 0.4860 (0.4827) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:27:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [81/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8966) loss 0.6055 (0.6156) grad_norm 0.5681 (0.4837) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:28:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [81/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8905) loss 0.6610 (0.6170) grad_norm 0.5551 (0.4829) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:30:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [81/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8874) loss 0.6090 (0.6160) grad_norm 0.4082 (0.4810) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:30:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 81 training takes 0:05:56 [2024-03-05 11:30:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [82/800][0/402] eta 0:28:47 lr 0.000025 time 4.2961 (4.2961) loss 0.5997 (0.5997) grad_norm 0.4211 (0.4211) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:31:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [82/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9121) loss 0.6237 (0.6154) grad_norm 0.5716 (0.4870) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:33:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [82/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8953) loss 0.6314 (0.6144) grad_norm 0.5772 (0.4863) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:34:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [82/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8897) loss 0.6205 (0.6151) grad_norm 0.5780 (0.4883) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:36:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [82/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8868) loss 0.6334 (0.6155) grad_norm 0.5982 (inf) loss_scale 262144.0000 (262797.7257) mem 30609MB [2024-03-05 11:36:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 82 training takes 0:05:56 [2024-03-05 11:36:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [83/800][0/402] eta 0:28:59 lr 0.000025 time 4.3280 (4.3280) loss 0.6015 (0.6015) grad_norm 0.4132 (0.4132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:37:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [83/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9124) loss 0.6281 (0.6160) grad_norm 0.4393 (0.4881) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:39:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [83/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.5817 (0.6166) grad_norm 0.4724 (0.4863) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:40:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [83/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.6223 (0.6167) grad_norm 0.4656 (0.4804) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:42:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [83/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.6405 (0.6161) grad_norm 0.4517 (0.4816) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:42:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 83 training takes 0:05:56 [2024-03-05 11:42:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [84/800][0/402] eta 0:29:26 lr 0.000025 time 4.3950 (4.3950) loss 0.6154 (0.6154) grad_norm 0.4930 (0.4930) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:43:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [84/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9130) loss 0.6208 (0.6158) grad_norm 0.4734 (0.5103) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:45:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [84/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8958) loss 0.6076 (0.6152) grad_norm 0.5731 (0.4883) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 11:46:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [84/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8899) loss 0.6316 (0.6150) grad_norm 0.4428 (inf) loss_scale 131072.0000 (254305.8073) mem 30609MB [2024-03-05 11:48:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [84/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8869) loss 0.6160 (0.6143) grad_norm 0.5546 (inf) loss_scale 131072.0000 (223574.1845) mem 30609MB [2024-03-05 11:48:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 84 training takes 0:05:56 [2024-03-05 11:48:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [85/800][0/402] eta 0:28:52 lr 0.000025 time 4.3102 (4.3102) loss 0.5976 (0.5976) grad_norm 0.4320 (0.4320) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 11:49:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [85/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9123) loss 0.5937 (0.6153) grad_norm 0.4065 (0.4677) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 11:51:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [85/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8954) loss 0.6066 (0.6155) grad_norm 0.3921 (0.4738) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 11:52:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [85/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8897) loss 0.6085 (0.6158) grad_norm 0.3894 (0.4721) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 11:54:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [85/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8868) loss 0.6103 (0.6156) grad_norm 0.4505 (0.4711) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 11:54:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 85 training takes 0:05:56 [2024-03-05 11:54:12 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_85.pth saving...... [2024-03-05 11:54:14 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_85.pth saved !!! [2024-03-05 11:54:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [86/800][0/402] eta 0:29:40 lr 0.000025 time 4.4286 (4.4286) loss 0.6036 (0.6036) grad_norm 0.5421 (0.5421) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 11:55:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [86/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9134) loss 0.6041 (0.6158) grad_norm 0.5160 (0.4828) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 11:57:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [86/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8960) loss 0.6181 (0.6172) grad_norm 0.4373 (0.4787) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 11:58:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [86/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8901) loss 0.5844 (0.6156) grad_norm 0.4927 (0.4775) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:00:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [86/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8871) loss 0.5793 (0.6159) grad_norm 0.4039 (0.4739) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:00:11 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 86 training takes 0:05:56 [2024-03-05 12:00:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [87/800][0/402] eta 0:29:09 lr 0.000025 time 4.3514 (4.3514) loss 0.6094 (0.6094) grad_norm 0.4337 (0.4337) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:01:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [87/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9126) loss 0.6050 (0.6133) grad_norm 0.3963 (0.4628) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:03:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [87/800][200/402] eta 0:03:00 lr 0.000025 time 0.8789 (0.8956) loss 0.6306 (0.6143) grad_norm 0.4642 (0.4608) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:04:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [87/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8899) loss 0.6229 (0.6133) grad_norm 0.3993 (0.4527) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:06:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [87/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.6331 (0.6147) grad_norm 0.4331 (0.4571) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:06:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 87 training takes 0:05:56 [2024-03-05 12:06:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [88/800][0/402] eta 0:29:25 lr 0.000025 time 4.3925 (4.3925) loss 0.6410 (0.6410) grad_norm 0.5805 (0.5805) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:07:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [88/800][100/402] eta 0:04:35 lr 0.000025 time 0.8789 (0.9131) loss 0.6228 (0.6155) grad_norm 0.4991 (0.4642) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:09:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [88/800][200/402] eta 0:03:00 lr 0.000025 time 0.8789 (0.8957) loss 0.6060 (0.6145) grad_norm 0.4738 (0.4656) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:10:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [88/800][300/402] eta 0:01:30 lr 0.000025 time 0.8776 (0.8900) loss 0.6235 (0.6141) grad_norm 0.3852 (0.4639) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:12:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [88/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8870) loss 0.6127 (0.6132) grad_norm 0.4458 (0.4630) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:12:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 88 training takes 0:05:56 [2024-03-05 12:12:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [89/800][0/402] eta 0:29:06 lr 0.000025 time 4.3454 (4.3454) loss 0.5994 (0.5994) grad_norm 0.5263 (0.5263) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:13:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [89/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9126) loss 0.6205 (0.6113) grad_norm 0.4862 (0.4565) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:15:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [89/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8955) loss 0.6231 (0.6119) grad_norm 0.4731 (0.4551) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 12:16:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [89/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.6026 (0.6123) grad_norm 0.4741 (0.4542) loss_scale 262144.0000 (143264.7442) mem 30609MB [2024-03-05 12:18:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [89/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8869) loss 0.6467 (0.6135) grad_norm 0.5309 (0.4482) loss_scale 262144.0000 (172910.4439) mem 30609MB [2024-03-05 12:18:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 89 training takes 0:05:56 [2024-03-05 12:18:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [90/800][0/402] eta 0:29:22 lr 0.000025 time 4.3836 (4.3836) loss 0.6134 (0.6134) grad_norm 0.4380 (0.4380) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:19:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [90/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9130) loss 0.6268 (0.6145) grad_norm 0.4712 (0.4641) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:21:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [90/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8958) loss 0.6042 (0.6141) grad_norm 0.5462 (0.4662) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:22:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [90/800][300/402] eta 0:01:30 lr 0.000025 time 0.8774 (0.8900) loss 0.6082 (0.6144) grad_norm 0.4356 (0.4584) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:23:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [90/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8870) loss 0.6388 (0.6139) grad_norm 0.4111 (0.4559) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:23:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 90 training takes 0:05:56 [2024-03-05 12:23:58 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_90.pth saving...... [2024-03-05 12:23:59 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_90.pth saved !!! [2024-03-05 12:24:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [91/800][0/402] eta 0:29:02 lr 0.000025 time 4.3345 (4.3345) loss 0.5821 (0.5821) grad_norm 0.5197 (0.5197) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:25:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [91/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9125) loss 0.6204 (0.6143) grad_norm 0.3806 (0.4577) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:26:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [91/800][200/402] eta 0:03:00 lr 0.000025 time 0.8787 (0.8955) loss 0.6146 (0.6138) grad_norm 0.4405 (0.4482) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:28:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [91/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.6246 (0.6135) grad_norm 0.4498 (0.4470) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:29:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [91/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.6000 (0.6129) grad_norm 0.4593 (0.4424) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:29:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 91 training takes 0:05:56 [2024-03-05 12:30:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [92/800][0/402] eta 0:28:52 lr 0.000025 time 4.3108 (4.3108) loss 0.6263 (0.6263) grad_norm 0.5199 (0.5199) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:31:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [92/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9122) loss 0.6100 (0.6129) grad_norm 0.4282 (0.4578) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:32:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [92/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8954) loss 0.6173 (0.6107) grad_norm 0.4590 (0.4493) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:34:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [92/800][300/402] eta 0:01:30 lr 0.000025 time 0.8778 (0.8898) loss 0.5932 (0.6114) grad_norm 0.5238 (0.4506) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:35:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [92/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8869) loss 0.6150 (0.6118) grad_norm 0.4542 (0.4480) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:35:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 92 training takes 0:05:56 [2024-03-05 12:35:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [93/800][0/402] eta 0:29:18 lr 0.000025 time 4.3731 (4.3731) loss 0.6119 (0.6119) grad_norm 0.4152 (0.4152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:37:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [93/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9128) loss 0.6115 (0.6150) grad_norm 0.4264 (0.4409) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:38:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [93/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8957) loss 0.6080 (0.6137) grad_norm 0.4231 (0.4392) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:40:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [93/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8899) loss 0.6008 (0.6124) grad_norm 0.3846 (0.4414) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:41:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [93/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8870) loss 0.6635 (0.6125) grad_norm 0.3776 (0.4373) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:41:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 93 training takes 0:05:56 [2024-03-05 12:41:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [94/800][0/402] eta 0:29:24 lr 0.000025 time 4.3885 (4.3885) loss 0.6005 (0.6005) grad_norm 0.4087 (0.4087) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:43:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [94/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9131) loss 0.6205 (0.6091) grad_norm 0.4819 (0.4296) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:44:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [94/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8958) loss 0.5959 (0.6097) grad_norm 0.4533 (0.4332) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:46:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [94/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8901) loss 0.5974 (0.6116) grad_norm 0.3721 (inf) loss_scale 262144.0000 (264756.7309) mem 30609MB [2024-03-05 12:47:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [94/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8874) loss 0.6512 (0.6116) grad_norm 0.4067 (inf) loss_scale 262144.0000 (264105.1771) mem 30609MB [2024-03-05 12:47:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 94 training takes 0:05:56 [2024-03-05 12:47:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [95/800][0/402] eta 0:29:16 lr 0.000025 time 4.3698 (4.3698) loss 0.5965 (0.5965) grad_norm 0.3599 (0.3599) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:49:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [95/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9128) loss 0.6336 (0.6147) grad_norm 0.4343 (0.4345) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:50:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [95/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8956) loss 0.6395 (0.6149) grad_norm 0.3971 (0.4343) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:52:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [95/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8898) loss 0.6408 (0.6135) grad_norm 0.4795 (0.4307) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:53:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [95/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.5863 (0.6131) grad_norm 0.5142 (0.4316) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:53:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 95 training takes 0:05:56 [2024-03-05 12:53:43 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_95.pth saving...... [2024-03-05 12:53:45 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_95.pth saved !!! [2024-03-05 12:53:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [96/800][0/402] eta 0:30:07 lr 0.000025 time 4.4973 (4.4973) loss 0.6460 (0.6460) grad_norm 0.5028 (0.5028) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:55:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [96/800][100/402] eta 0:04:36 lr 0.000025 time 0.8779 (0.9141) loss 0.6054 (0.6119) grad_norm 0.4154 (0.4254) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:56:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [96/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8963) loss 0.6030 (0.6111) grad_norm 0.4105 (0.4252) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:58:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [96/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8904) loss 0.5883 (0.6114) grad_norm 0.4752 (0.4284) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:59:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [96/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8874) loss 0.6024 (0.6122) grad_norm 0.4515 (0.4263) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 12:59:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 96 training takes 0:05:56 [2024-03-05 12:59:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [97/800][0/402] eta 0:29:13 lr 0.000025 time 4.3615 (4.3615) loss 0.6162 (0.6162) grad_norm 0.3771 (0.3771) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:01:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [97/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9127) loss 0.6079 (0.6123) grad_norm 0.4512 (0.4212) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:02:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [97/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8956) loss 0.6201 (0.6105) grad_norm 0.4244 (0.4320) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:04:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [97/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.6226 (0.6126) grad_norm 0.3494 (0.4293) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:05:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [97/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.5884 (0.6126) grad_norm 0.3835 (0.4267) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:05:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 97 training takes 0:05:56 [2024-03-05 13:05:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [98/800][0/402] eta 0:29:24 lr 0.000025 time 4.3888 (4.3888) loss 0.6244 (0.6244) grad_norm 0.4710 (0.4710) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:07:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [98/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9130) loss 0.5965 (0.6120) grad_norm 0.4485 (0.4228) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:08:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [98/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8957) loss 0.5819 (0.6116) grad_norm 0.4165 (0.4213) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:10:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [98/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8899) loss 0.5999 (0.6110) grad_norm 0.4159 (0.4205) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:11:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [98/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8870) loss 0.6252 (0.6110) grad_norm 0.4370 (0.4201) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:11:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 98 training takes 0:05:56 [2024-03-05 13:11:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [99/800][0/402] eta 0:29:07 lr 0.000025 time 4.3481 (4.3481) loss 0.6301 (0.6301) grad_norm 0.3356 (0.3356) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:13:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [99/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9128) loss 0.5991 (0.6102) grad_norm 0.3773 (0.4037) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:14:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [99/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8956) loss 0.6070 (0.6108) grad_norm 0.4790 (0.4052) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:16:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [99/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.6073 (0.6114) grad_norm 0.4400 (inf) loss_scale 262144.0000 (269111.2824) mem 30609MB [2024-03-05 13:17:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [99/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8869) loss 0.6010 (0.6114) grad_norm 0.4161 (inf) loss_scale 262144.0000 (267373.8055) mem 30609MB [2024-03-05 13:17:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 99 training takes 0:05:56 [2024-03-05 13:17:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [100/800][0/402] eta 0:29:23 lr 0.000025 time 4.3859 (4.3859) loss 0.6123 (0.6123) grad_norm 0.3597 (0.3597) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:19:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [100/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9130) loss 0.6028 (0.6076) grad_norm 0.3805 (0.4146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:20:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [100/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8957) loss 0.5899 (0.6093) grad_norm 0.4134 (0.4160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:22:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [100/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8899) loss 0.6088 (0.6107) grad_norm 0.4531 (0.4140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:23:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [100/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.5971 (0.6107) grad_norm 0.3354 (0.4086) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:23:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 100 training takes 0:05:56 [2024-03-05 13:23:28 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_100.pth saving...... [2024-03-05 13:23:30 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_100.pth saved !!! [2024-03-05 13:23:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [101/800][0/402] eta 0:29:26 lr 0.000025 time 4.3938 (4.3938) loss 0.5832 (0.5832) grad_norm 0.3104 (0.3104) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:25:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [101/800][100/402] eta 0:04:35 lr 0.000025 time 0.8775 (0.9138) loss 0.5949 (0.6095) grad_norm 0.4040 (0.4032) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:26:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [101/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8962) loss 0.6067 (0.6095) grad_norm 0.3871 (0.4071) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:27:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [101/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8903) loss 0.5612 (0.6103) grad_norm 0.3823 (0.4099) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:29:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [101/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8873) loss 0.6277 (0.6105) grad_norm 0.4035 (0.4081) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:29:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 101 training takes 0:05:56 [2024-03-05 13:29:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [102/800][0/402] eta 0:29:07 lr 0.000025 time 4.3480 (4.3480) loss 0.5747 (0.5747) grad_norm 0.3970 (0.3970) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:30:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [102/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9125) loss 0.6308 (0.6085) grad_norm 0.4022 (0.4161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:32:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [102/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.6132 (0.6105) grad_norm 0.4097 (0.4130) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:33:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [102/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8898) loss 0.5693 (0.6107) grad_norm 0.3692 (0.4103) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:35:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [102/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8868) loss 0.6071 (0.6107) grad_norm 0.3310 (0.4125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:35:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 102 training takes 0:05:56 [2024-03-05 13:35:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [103/800][0/402] eta 0:29:05 lr 0.000025 time 4.3429 (4.3429) loss 0.6145 (0.6145) grad_norm 0.3754 (0.3754) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:36:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [103/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9125) loss 0.5974 (0.6145) grad_norm 0.4684 (0.3964) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:38:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [103/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8956) loss 0.6146 (0.6120) grad_norm 0.4126 (0.3993) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:39:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [103/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.6383 (0.6114) grad_norm 0.3971 (0.3976) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:41:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [103/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8869) loss 0.6168 (0.6108) grad_norm 0.4616 (0.3998) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:41:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 103 training takes 0:05:56 [2024-03-05 13:41:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [104/800][0/402] eta 0:29:18 lr 0.000025 time 4.3743 (4.3743) loss 0.6122 (0.6122) grad_norm 0.3656 (0.3656) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:42:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [104/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9128) loss 0.5975 (0.6116) grad_norm 0.3774 (0.3989) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:44:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [104/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8957) loss 0.6121 (0.6103) grad_norm 0.4215 (0.3945) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:45:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [104/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8899) loss 0.5879 (0.6106) grad_norm 0.3656 (inf) loss_scale 262144.0000 (263885.8206) mem 30609MB [2024-03-05 13:47:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [104/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8869) loss 0.6007 (0.6114) grad_norm 0.3865 (inf) loss_scale 262144.0000 (263451.4514) mem 30609MB [2024-03-05 13:47:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 104 training takes 0:05:56 [2024-03-05 13:47:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [105/800][0/402] eta 0:29:01 lr 0.000025 time 4.3321 (4.3321) loss 0.5844 (0.5844) grad_norm 0.4335 (0.4335) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:48:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [105/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9124) loss 0.6090 (0.6085) grad_norm 0.4577 (0.3885) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:50:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [105/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8954) loss 0.6260 (0.6090) grad_norm 0.3636 (0.3929) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:51:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [105/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8897) loss 0.6052 (0.6096) grad_norm 0.3729 (0.3928) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:53:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [105/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.6349 (0.6095) grad_norm 0.3958 (0.3945) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:53:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 105 training takes 0:05:56 [2024-03-05 13:53:14 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_105.pth saving...... [2024-03-05 13:53:15 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_105.pth saved !!! [2024-03-05 13:53:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [106/800][0/402] eta 0:27:39 lr 0.000025 time 4.1279 (4.1279) loss 0.5787 (0.5787) grad_norm 0.3944 (0.3944) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:54:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [106/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9107) loss 0.6394 (0.6097) grad_norm 0.4327 (0.3864) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:56:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [106/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8947) loss 0.6001 (0.6098) grad_norm 0.4030 (0.3842) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:57:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [106/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8893) loss 0.5750 (0.6093) grad_norm 0.4664 (0.3890) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:59:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [106/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8865) loss 0.6459 (0.6087) grad_norm 0.3450 (0.3905) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 13:59:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 106 training takes 0:05:56 [2024-03-05 13:59:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [107/800][0/402] eta 0:29:21 lr 0.000025 time 4.3816 (4.3816) loss 0.5933 (0.5933) grad_norm 0.3956 (0.3956) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:00:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [107/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9130) loss 0.5875 (0.6046) grad_norm 0.3724 (0.3791) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:02:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [107/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8958) loss 0.6225 (0.6071) grad_norm 0.3298 (0.3861) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:03:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [107/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.6054 (0.6068) grad_norm 0.3040 (0.3887) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:05:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [107/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8871) loss 0.6333 (0.6080) grad_norm 0.4806 (0.3910) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:05:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 107 training takes 0:05:56 [2024-03-05 14:05:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [108/800][0/402] eta 0:29:06 lr 0.000025 time 4.3442 (4.3442) loss 0.6002 (0.6002) grad_norm 0.4010 (0.4010) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:06:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [108/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9128) loss 0.5982 (0.6071) grad_norm 0.3527 (0.3776) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:08:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [108/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.5959 (0.6077) grad_norm 0.4631 (0.3802) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:09:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [108/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8899) loss 0.5969 (0.6078) grad_norm 0.3456 (0.3793) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:11:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [108/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8870) loss 0.6250 (0.6066) grad_norm 0.4112 (0.3803) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:11:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 108 training takes 0:05:56 [2024-03-05 14:11:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [109/800][0/402] eta 0:29:05 lr 0.000025 time 4.3421 (4.3421) loss 0.5958 (0.5958) grad_norm 0.3545 (0.3545) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:12:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [109/800][100/402] eta 0:04:35 lr 0.000025 time 0.8797 (0.9136) loss 0.6099 (0.6071) grad_norm 0.3712 (0.3827) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:14:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [109/800][200/402] eta 0:03:01 lr 0.000025 time 0.8796 (0.8966) loss 0.6366 (0.6078) grad_norm 0.3766 (0.3870) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:15:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [109/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8909) loss 0.5943 (0.6077) grad_norm 0.3960 (0.3858) loss_scale 524288.0000 (310044.0664) mem 30609MB [2024-03-05 14:17:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [109/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8880) loss 0.6027 (0.6085) grad_norm 0.3788 (inf) loss_scale 262144.0000 (302021.2668) mem 30609MB [2024-03-05 14:17:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 109 training takes 0:05:57 [2024-03-05 14:17:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [110/800][0/402] eta 0:29:19 lr 0.000025 time 4.3766 (4.3766) loss 0.5980 (0.5980) grad_norm 0.5537 (0.5537) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:18:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [110/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9130) loss 0.5825 (0.6077) grad_norm 0.3509 (0.3745) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:20:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [110/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8958) loss 0.5884 (0.6075) grad_norm 0.3952 (0.3871) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:21:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [110/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8900) loss 0.6027 (0.6078) grad_norm 0.4003 (0.3841) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:22:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [110/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8871) loss 0.5878 (0.6080) grad_norm 0.3291 (0.3790) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:22:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 110 training takes 0:05:56 [2024-03-05 14:22:59 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_110.pth saving...... [2024-03-05 14:23:01 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_110.pth saved !!! [2024-03-05 14:23:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [111/800][0/402] eta 0:28:07 lr 0.000025 time 4.1979 (4.1979) loss 0.6106 (0.6106) grad_norm 0.3968 (0.3968) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:24:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [111/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9113) loss 0.6374 (0.6076) grad_norm 0.3711 (0.3803) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:26:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [111/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8950) loss 0.5994 (0.6082) grad_norm 0.3456 (0.3712) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:27:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [111/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8895) loss 0.6084 (0.6085) grad_norm 0.3776 (0.3737) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:28:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [111/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8867) loss 0.5795 (0.6086) grad_norm 0.4708 (0.3737) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:28:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 111 training takes 0:05:56 [2024-03-05 14:29:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [112/800][0/402] eta 0:29:30 lr 0.000025 time 4.4035 (4.4035) loss 0.5941 (0.5941) grad_norm 0.3785 (0.3785) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:30:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [112/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9132) loss 0.6315 (0.6080) grad_norm 0.4045 (0.3826) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:31:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [112/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8959) loss 0.5724 (0.6061) grad_norm 0.3370 (0.3722) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:33:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [112/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8901) loss 0.6385 (0.6078) grad_norm 0.4191 (0.3705) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:34:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [112/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8871) loss 0.6221 (0.6068) grad_norm 0.3978 (0.3693) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:34:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 112 training takes 0:05:56 [2024-03-05 14:34:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [113/800][0/402] eta 0:27:23 lr 0.000025 time 4.0885 (4.0885) loss 0.5607 (0.5607) grad_norm 0.3770 (0.3770) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:36:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [113/800][100/402] eta 0:04:34 lr 0.000025 time 0.8782 (0.9100) loss 0.5893 (0.6062) grad_norm 0.3293 (0.3798) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:37:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [113/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8943) loss 0.5906 (0.6060) grad_norm 0.3224 (0.3733) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:39:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [113/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8890) loss 0.6187 (0.6070) grad_norm 0.4327 (0.3741) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:40:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [113/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8862) loss 0.6177 (0.6067) grad_norm 0.3156 (0.3734) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:40:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 113 training takes 0:05:56 [2024-03-05 14:40:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [114/800][0/402] eta 0:29:02 lr 0.000025 time 4.3356 (4.3356) loss 0.6025 (0.6025) grad_norm 0.3657 (0.3657) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:42:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [114/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9125) loss 0.5946 (0.6086) grad_norm 0.3410 (0.3614) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 14:43:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [114/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8954) loss 0.6421 (0.6076) grad_norm 0.4267 (nan) loss_scale 131072.0000 (226930.6269) mem 30609MB [2024-03-05 14:45:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [114/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.6220 (0.6081) grad_norm 0.3853 (nan) loss_scale 131072.0000 (195083.9070) mem 30609MB [2024-03-05 14:46:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [114/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8869) loss 0.6085 (0.6080) grad_norm 0.3469 (nan) loss_scale 131072.0000 (179120.8379) mem 30609MB [2024-03-05 14:46:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 114 training takes 0:05:56 [2024-03-05 14:46:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [115/800][0/402] eta 0:29:00 lr 0.000025 time 4.3306 (4.3306) loss 0.6243 (0.6243) grad_norm 0.3420 (0.3420) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:48:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [115/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9124) loss 0.5832 (0.6075) grad_norm 0.3071 (0.3604) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:49:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [115/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8955) loss 0.5767 (0.6065) grad_norm 0.3318 (0.3662) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:51:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [115/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.5934 (0.6062) grad_norm 0.3593 (0.3656) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:52:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [115/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.6081 (0.6064) grad_norm 0.4796 (0.3661) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:52:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 115 training takes 0:05:56 [2024-03-05 14:52:44 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_115.pth saving...... [2024-03-05 14:52:46 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_115.pth saved !!! [2024-03-05 14:52:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [116/800][0/402] eta 0:26:04 lr 0.000025 time 3.8912 (3.8912) loss 0.6123 (0.6123) grad_norm 0.3606 (0.3606) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:54:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [116/800][100/402] eta 0:04:34 lr 0.000025 time 0.8780 (0.9081) loss 0.5736 (0.6036) grad_norm 0.3930 (0.3640) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:55:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [116/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8933) loss 0.5997 (0.6056) grad_norm 0.3632 (0.3602) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:57:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [116/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8883) loss 0.5519 (0.6056) grad_norm 0.3108 (0.3588) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:58:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [116/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8860) loss 0.6047 (0.6058) grad_norm 0.3804 (0.3586) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 14:58:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 116 training takes 0:05:56 [2024-03-05 14:58:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [117/800][0/402] eta 0:29:11 lr 0.000025 time 4.3578 (4.3578) loss 0.6139 (0.6139) grad_norm 0.3831 (0.3831) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:00:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [117/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9133) loss 0.5749 (0.6053) grad_norm 0.3577 (0.3682) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:01:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [117/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8959) loss 0.5769 (0.6066) grad_norm 0.4613 (0.3654) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:03:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [117/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8901) loss 0.6369 (0.6063) grad_norm 0.3278 (0.3625) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:04:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [117/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8871) loss 0.5835 (0.6059) grad_norm 0.3229 (0.3621) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:04:39 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 117 training takes 0:05:56 [2024-03-05 15:04:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [118/800][0/402] eta 0:29:19 lr 0.000025 time 4.3759 (4.3759) loss 0.6332 (0.6332) grad_norm 0.3625 (0.3625) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:06:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [118/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9129) loss 0.6303 (0.6064) grad_norm 0.3089 (0.3651) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:07:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [118/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8957) loss 0.6366 (0.6069) grad_norm 0.3078 (0.3632) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:09:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [118/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8899) loss 0.6203 (0.6063) grad_norm 0.3152 (0.3629) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:10:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [118/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8870) loss 0.5848 (0.6062) grad_norm 0.3740 (0.3583) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:10:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 118 training takes 0:05:56 [2024-03-05 15:10:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [119/800][0/402] eta 0:29:05 lr 0.000025 time 4.3423 (4.3423) loss 0.6757 (0.6757) grad_norm 0.3290 (0.3290) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:12:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [119/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9135) loss 0.6043 (0.6089) grad_norm 0.3966 (0.3587) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-05 15:13:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [119/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8960) loss 0.6128 (0.6070) grad_norm 0.3301 (0.3542) loss_scale 262144.0000 (172806.3682) mem 30609MB [2024-03-05 15:15:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [119/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8902) loss 0.5899 (0.6073) grad_norm 0.3533 (0.3553) loss_scale 262144.0000 (202486.6445) mem 30609MB [2024-03-05 15:16:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [119/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8872) loss 0.6033 (0.6063) grad_norm 0.3705 (0.3542) loss_scale 262144.0000 (217363.7905) mem 30609MB [2024-03-05 15:16:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 119 training takes 0:05:56 [2024-03-05 15:16:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [120/800][0/402] eta 0:29:29 lr 0.000025 time 4.4017 (4.4017) loss 0.6065 (0.6065) grad_norm 0.3807 (0.3807) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:18:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [120/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9132) loss 0.6114 (0.6045) grad_norm 0.3394 (0.3562) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:19:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [120/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8959) loss 0.5786 (0.6052) grad_norm 0.3561 (0.3510) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:21:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [120/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8900) loss 0.6238 (0.6038) grad_norm 0.3616 (0.3542) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:22:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [120/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8871) loss 0.5988 (0.6054) grad_norm 0.3853 (0.3528) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:22:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 120 training takes 0:05:56 [2024-03-05 15:22:29 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_120.pth saving...... [2024-03-05 15:22:31 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_120.pth saved !!! [2024-03-05 15:22:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [121/800][0/402] eta 0:29:33 lr 0.000025 time 4.4125 (4.4125) loss 0.6325 (0.6325) grad_norm 0.3079 (0.3079) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:24:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [121/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9133) loss 0.5876 (0.6061) grad_norm 0.3802 (0.3535) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:25:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [121/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8963) loss 0.6088 (0.6050) grad_norm 0.3252 (0.3551) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:26:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [121/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8903) loss 0.5775 (0.6052) grad_norm 0.3256 (0.3535) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:28:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [121/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8873) loss 0.6003 (0.6059) grad_norm 0.3325 (0.3504) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:28:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 121 training takes 0:05:56 [2024-03-05 15:28:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [122/800][0/402] eta 0:29:07 lr 0.000025 time 4.3468 (4.3468) loss 0.5995 (0.5995) grad_norm 0.3046 (0.3046) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:30:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [122/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9126) loss 0.5958 (0.6061) grad_norm 0.3207 (0.3465) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:31:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [122/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8956) loss 0.6303 (0.6045) grad_norm 0.2976 (0.3450) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:32:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [122/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8899) loss 0.5812 (0.6044) grad_norm 0.3527 (0.3425) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:34:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [122/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8870) loss 0.6185 (0.6043) grad_norm 0.3640 (0.3412) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:34:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 122 training takes 0:05:56 [2024-03-05 15:34:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [123/800][0/402] eta 0:29:07 lr 0.000025 time 4.3467 (4.3467) loss 0.6135 (0.6135) grad_norm 0.3231 (0.3231) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:35:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [123/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9126) loss 0.6165 (0.6032) grad_norm 0.3287 (0.3520) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:37:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [123/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8956) loss 0.6024 (0.6043) grad_norm 0.4344 (0.3509) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:38:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [123/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8902) loss 0.6088 (0.6057) grad_norm 0.3251 (0.3524) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:40:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [123/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8872) loss 0.5671 (0.6056) grad_norm 0.3917 (0.3526) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:40:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 123 training takes 0:05:56 [2024-03-05 15:40:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [124/800][0/402] eta 0:29:07 lr 0.000025 time 4.3473 (4.3473) loss 0.6650 (0.6650) grad_norm 0.3486 (0.3486) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:41:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [124/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9126) loss 0.6078 (0.6061) grad_norm 0.3374 (0.3444) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:43:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [124/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8956) loss 0.5644 (0.6060) grad_norm 0.3206 (0.3423) loss_scale 524288.0000 (358654.7264) mem 30609MB [2024-03-05 15:44:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [124/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8899) loss 0.6071 (0.6057) grad_norm 0.3373 (0.3415) loss_scale 524288.0000 (413682.3920) mem 30609MB [2024-03-05 15:46:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [124/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.6094 (0.6054) grad_norm 0.4048 (inf) loss_scale 262144.0000 (437996.2095) mem 30609MB [2024-03-05 15:46:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 124 training takes 0:05:56 [2024-03-05 15:46:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [125/800][0/402] eta 0:29:13 lr 0.000025 time 4.3625 (4.3625) loss 0.6003 (0.6003) grad_norm 0.3297 (0.3297) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:47:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [125/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9128) loss 0.6246 (0.6044) grad_norm 0.2619 (0.3455) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:49:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [125/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.6018 (0.6051) grad_norm 0.3351 (0.3361) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:50:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [125/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8899) loss 0.6136 (0.6054) grad_norm 0.3856 (0.3381) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:52:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [125/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8872) loss 0.6224 (0.6050) grad_norm 0.3472 (0.3396) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:52:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 125 training takes 0:05:56 [2024-03-05 15:52:15 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_125.pth saving...... [2024-03-05 15:52:17 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_125.pth saved !!! [2024-03-05 15:52:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [126/800][0/402] eta 0:29:23 lr 0.000025 time 4.3871 (4.3871) loss 0.6187 (0.6187) grad_norm 0.4167 (0.4167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:53:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [126/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9133) loss 0.5860 (0.6046) grad_norm 0.3582 (0.3412) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:55:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [126/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8960) loss 0.6226 (0.6044) grad_norm 0.3180 (0.3414) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:56:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [126/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8903) loss 0.6376 (0.6053) grad_norm 0.3580 (0.3404) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:58:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [126/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8873) loss 0.5755 (0.6047) grad_norm 0.3908 (0.3406) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:58:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 126 training takes 0:05:56 [2024-03-05 15:58:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [127/800][0/402] eta 0:29:11 lr 0.000025 time 4.3566 (4.3566) loss 0.5985 (0.5985) grad_norm 0.3483 (0.3483) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 15:59:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [127/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9128) loss 0.5767 (0.6039) grad_norm 0.3168 (0.3428) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:01:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [127/800][200/402] eta 0:03:00 lr 0.000025 time 0.8789 (0.8957) loss 0.5989 (0.6036) grad_norm 0.3267 (0.3418) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:02:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [127/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8899) loss 0.5647 (0.6034) grad_norm 0.3673 (0.3426) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:04:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [127/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8872) loss 0.5986 (0.6045) grad_norm 0.3504 (0.3408) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:04:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 127 training takes 0:05:56 [2024-03-05 16:04:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [128/800][0/402] eta 0:29:10 lr 0.000025 time 4.3546 (4.3546) loss 0.5731 (0.5731) grad_norm 0.3594 (0.3594) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:05:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [128/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9127) loss 0.5835 (0.6012) grad_norm 0.3957 (0.3351) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:07:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [128/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8957) loss 0.5949 (0.6030) grad_norm 0.2852 (0.3343) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:08:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [128/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8899) loss 0.6031 (0.6039) grad_norm 0.4298 (0.3392) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:10:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [128/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.6117 (0.6039) grad_norm 0.3318 (0.3396) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:10:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 128 training takes 0:05:56 [2024-03-05 16:10:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [129/800][0/402] eta 0:29:02 lr 0.000025 time 4.3351 (4.3351) loss 0.6303 (0.6303) grad_norm 0.3173 (0.3173) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:11:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [129/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9125) loss 0.5956 (0.6048) grad_norm 0.3583 (0.3412) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:13:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [129/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8955) loss 0.5745 (0.6043) grad_norm 0.3102 (0.3423) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:14:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [129/800][300/402] eta 0:01:30 lr 0.000025 time 0.8778 (0.8899) loss 0.5987 (0.6037) grad_norm 0.3036 (0.3385) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:16:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [129/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8869) loss 0.5826 (0.6040) grad_norm 0.3395 (0.3387) loss_scale 524288.0000 (271949.8853) mem 30609MB [2024-03-05 16:16:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 129 training takes 0:05:56 [2024-03-05 16:16:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [130/800][0/402] eta 0:28:51 lr 0.000025 time 4.3061 (4.3061) loss 0.6144 (0.6144) grad_norm 0.2975 (0.2975) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 16:17:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [130/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9131) loss 0.6019 (0.6032) grad_norm 0.3596 (0.3287) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 16:19:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [130/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8958) loss 0.6247 (0.6023) grad_norm 0.2927 (inf) loss_scale 262144.0000 (456469.6517) mem 30609MB [2024-03-05 16:20:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [130/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8900) loss 0.5995 (0.6028) grad_norm 0.3331 (inf) loss_scale 262144.0000 (391909.6346) mem 30609MB [2024-03-05 16:22:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [130/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8873) loss 0.6472 (0.6033) grad_norm 0.3402 (inf) loss_scale 262144.0000 (359549.1272) mem 30609MB [2024-03-05 16:22:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 130 training takes 0:05:56 [2024-03-05 16:22:01 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_130.pth saving...... [2024-03-05 16:22:03 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_130.pth saved !!! [2024-03-05 16:22:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [131/800][0/402] eta 0:27:37 lr 0.000025 time 4.1226 (4.1226) loss 0.6325 (0.6325) grad_norm 0.2769 (0.2769) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:23:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [131/800][100/402] eta 0:04:35 lr 0.000025 time 0.8797 (0.9113) loss 0.6226 (0.6018) grad_norm 0.3364 (0.3306) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:25:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [131/800][200/402] eta 0:03:00 lr 0.000025 time 0.8795 (0.8952) loss 0.5649 (0.6036) grad_norm 0.2995 (0.3328) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:26:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [131/800][300/402] eta 0:01:30 lr 0.000025 time 0.8793 (0.8898) loss 0.5996 (0.6040) grad_norm 0.2848 (0.3333) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:27:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [131/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8870) loss 0.5976 (0.6038) grad_norm 0.3277 (0.3306) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:27:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 131 training takes 0:05:56 [2024-03-05 16:28:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [132/800][0/402] eta 0:31:09 lr 0.000025 time 4.6513 (4.6513) loss 0.6160 (0.6160) grad_norm 0.3537 (0.3537) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:29:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [132/800][100/402] eta 0:04:36 lr 0.000025 time 0.8779 (0.9167) loss 0.6319 (0.6022) grad_norm 0.3151 (0.3327) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:31:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [132/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8978) loss 0.5966 (0.6038) grad_norm 0.3077 (0.3320) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:32:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [132/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8914) loss 0.6137 (0.6037) grad_norm 0.3356 (0.3302) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:33:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [132/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8882) loss 0.6106 (0.6037) grad_norm 0.3090 (0.3326) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:33:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 132 training takes 0:05:57 [2024-03-05 16:34:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [133/800][0/402] eta 0:29:41 lr 0.000025 time 4.4313 (4.4313) loss 0.6145 (0.6145) grad_norm 0.5028 (0.5028) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:35:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [133/800][100/402] eta 0:04:36 lr 0.000025 time 0.8797 (0.9141) loss 0.5861 (0.6032) grad_norm 0.2617 (0.3391) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:36:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [133/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8969) loss 0.6173 (0.6023) grad_norm 0.3245 (0.3353) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:38:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [133/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8909) loss 0.5907 (0.6037) grad_norm 0.4038 (0.3355) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:39:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [133/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8878) loss 0.6113 (0.6035) grad_norm 0.3496 (0.3348) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:39:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 133 training takes 0:05:57 [2024-03-05 16:39:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [134/800][0/402] eta 0:31:01 lr 0.000025 time 4.6294 (4.6294) loss 0.5752 (0.5752) grad_norm 0.3597 (0.3597) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:41:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [134/800][100/402] eta 0:04:37 lr 0.000025 time 0.9040 (0.9178) loss 0.6028 (0.6046) grad_norm 0.3306 (0.3290) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:42:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [134/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8987) loss 0.6059 (0.6020) grad_norm 0.3211 (0.3277) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:44:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [134/800][300/402] eta 0:01:31 lr 0.000025 time 0.8790 (0.8923) loss 0.6025 (0.6022) grad_norm 0.3353 (0.3277) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:45:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [134/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8888) loss 0.5880 (0.6029) grad_norm 0.3947 (0.3275) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:45:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 134 training takes 0:05:57 [2024-03-05 16:45:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [135/800][0/402] eta 0:30:49 lr 0.000025 time 4.6003 (4.6003) loss 0.6329 (0.6329) grad_norm 0.2864 (0.2864) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:47:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [135/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9153) loss 0.6168 (0.6046) grad_norm 0.3382 (0.3327) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:48:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [135/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8969) loss 0.6088 (0.6043) grad_norm 0.3514 (0.3306) loss_scale 524288.0000 (343004.3383) mem 30609MB [2024-03-05 16:50:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [135/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8917) loss 0.5664 (0.6031) grad_norm 0.3375 (0.3305) loss_scale 524288.0000 (403231.4684) mem 30609MB [2024-03-05 16:51:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [135/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8884) loss 0.5747 (0.6030) grad_norm 0.2665 (inf) loss_scale 262144.0000 (391581.6858) mem 30609MB [2024-03-05 16:51:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 135 training takes 0:05:57 [2024-03-05 16:51:49 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_135.pth saving...... [2024-03-05 16:51:50 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_135.pth saved !!! [2024-03-05 16:51:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [136/800][0/402] eta 0:30:23 lr 0.000025 time 4.5365 (4.5365) loss 0.5405 (0.5405) grad_norm 0.3007 (0.3007) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:53:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [136/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9150) loss 0.5588 (0.6011) grad_norm 0.2779 (0.3334) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:54:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [136/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8971) loss 0.5780 (0.6033) grad_norm 0.3324 (0.3285) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:56:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [136/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8913) loss 0.6334 (0.6033) grad_norm 0.3814 (0.3287) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:57:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [136/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8885) loss 0.5903 (0.6027) grad_norm 0.2850 (0.3264) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:57:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 136 training takes 0:05:57 [2024-03-05 16:57:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [137/800][0/402] eta 0:32:31 lr 0.000025 time 4.8534 (4.8534) loss 0.5936 (0.5936) grad_norm 0.3268 (0.3268) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 16:59:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [137/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9182) loss 0.5910 (0.6025) grad_norm 0.3739 (0.3308) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:00:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [137/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.9002) loss 0.5885 (0.6020) grad_norm 0.2640 (0.3308) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:02:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [137/800][300/402] eta 0:01:31 lr 0.000025 time 0.8782 (0.8930) loss 0.5833 (0.6024) grad_norm 0.3457 (0.3279) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:03:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [137/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8893) loss 0.5767 (0.6026) grad_norm 0.2837 (0.3263) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:03:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 137 training takes 0:05:57 [2024-03-05 17:03:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [138/800][0/402] eta 0:29:45 lr 0.000025 time 4.4415 (4.4415) loss 0.5750 (0.5750) grad_norm 0.3423 (0.3423) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:05:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [138/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9138) loss 0.6234 (0.6022) grad_norm 0.3171 (0.3258) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:06:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [138/800][200/402] eta 0:03:01 lr 0.000025 time 0.8802 (0.8962) loss 0.5801 (0.6015) grad_norm 0.3056 (0.3302) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:08:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [138/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8903) loss 0.5765 (0.6020) grad_norm 0.3164 (0.3255) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:09:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [138/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8875) loss 0.6214 (0.6031) grad_norm 0.3352 (0.3264) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:09:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 138 training takes 0:05:56 [2024-03-05 17:09:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [139/800][0/402] eta 0:29:30 lr 0.000025 time 4.4043 (4.4043) loss 0.6120 (0.6120) grad_norm 0.3297 (0.3297) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:11:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [139/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9131) loss 0.5728 (0.6007) grad_norm 0.2927 (0.3157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:12:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [139/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8958) loss 0.5903 (0.6014) grad_norm 0.2953 (0.3179) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:14:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [139/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8900) loss 0.5988 (0.6013) grad_norm 0.2808 (0.3203) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:15:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [139/800][400/402] eta 0:00:01 lr 0.000025 time 0.8776 (0.8871) loss 0.6313 (0.6018) grad_norm 0.3054 (0.3198) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:15:39 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 139 training takes 0:05:56 [2024-03-05 17:15:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [140/800][0/402] eta 0:29:43 lr 0.000025 time 4.4371 (4.4371) loss 0.6100 (0.6100) grad_norm 0.2667 (0.2667) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:17:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [140/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9136) loss 0.5991 (0.6038) grad_norm 0.2928 (0.3207) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:18:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [140/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8964) loss 0.6220 (0.6035) grad_norm 0.3641 (0.3174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:20:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [140/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8904) loss 0.6149 (0.6049) grad_norm 0.3348 (0.3227) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:21:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [140/800][400/402] eta 0:00:01 lr 0.000025 time 0.8780 (0.8887) loss 0.6066 (0.6035) grad_norm 0.3452 (0.3219) loss_scale 524288.0000 (310519.7007) mem 30609MB [2024-03-05 17:21:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 140 training takes 0:05:57 [2024-03-05 17:21:37 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_140.pth saving...... [2024-03-05 17:21:39 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_140.pth saved !!! [2024-03-05 17:21:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [141/800][0/402] eta 0:28:01 lr 0.000025 time 4.1834 (4.1834) loss 0.5472 (0.5472) grad_norm 0.3067 (0.3067) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 17:23:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [141/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9112) loss 0.5895 (0.5997) grad_norm 0.2918 (inf) loss_scale 262144.0000 (443827.9604) mem 30609MB [2024-03-05 17:24:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [141/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8963) loss 0.5708 (0.6012) grad_norm 0.2848 (inf) loss_scale 262144.0000 (353437.9303) mem 30609MB [2024-03-05 17:26:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [141/800][300/402] eta 0:01:31 lr 0.000025 time 0.8794 (0.8939) loss 0.6226 (0.6005) grad_norm 0.3514 (inf) loss_scale 262144.0000 (323107.7209) mem 30609MB [2024-03-05 17:27:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [141/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8902) loss 0.6084 (0.6010) grad_norm 0.2906 (inf) loss_scale 262144.0000 (307904.7980) mem 30609MB [2024-03-05 17:27:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 141 training takes 0:05:58 [2024-03-05 17:27:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [142/800][0/402] eta 0:29:52 lr 0.000025 time 4.4583 (4.4583) loss 0.5842 (0.5842) grad_norm 0.3232 (0.3232) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:29:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [142/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9137) loss 0.5836 (0.6013) grad_norm 0.3203 (0.3142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:30:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [142/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8961) loss 0.6224 (0.6016) grad_norm 0.3330 (0.3164) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:32:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [142/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8902) loss 0.5972 (0.6016) grad_norm 0.3162 (0.3132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:33:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [142/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8872) loss 0.5914 (0.6019) grad_norm 0.3536 (0.3158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:33:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 142 training takes 0:05:56 [2024-03-05 17:33:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [143/800][0/402] eta 0:29:34 lr 0.000025 time 4.4138 (4.4138) loss 0.5981 (0.5981) grad_norm 0.2899 (0.2899) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:35:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [143/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9132) loss 0.6057 (0.6003) grad_norm 0.3551 (0.3118) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:36:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [143/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8959) loss 0.6234 (0.6005) grad_norm 0.3190 (0.3157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:38:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [143/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8901) loss 0.6124 (0.6014) grad_norm 0.3005 (0.3160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:39:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [143/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8871) loss 0.6184 (0.6019) grad_norm 0.3100 (0.3139) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:39:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 143 training takes 0:05:56 [2024-03-05 17:39:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [144/800][0/402] eta 0:29:10 lr 0.000025 time 4.3535 (4.3535) loss 0.5936 (0.5936) grad_norm 0.2743 (0.2743) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:41:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [144/800][100/402] eta 0:04:35 lr 0.000025 time 0.8777 (0.9127) loss 0.6065 (0.5976) grad_norm 0.2707 (0.3144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:42:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [144/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8956) loss 0.5935 (0.5986) grad_norm 0.3168 (0.3150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:43:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [144/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8899) loss 0.6118 (0.5997) grad_norm 0.2844 (0.3145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:45:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [144/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.6035 (0.6008) grad_norm 0.3583 (0.3131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:45:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 144 training takes 0:05:56 [2024-03-05 17:45:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [145/800][0/402] eta 0:29:39 lr 0.000025 time 4.4276 (4.4276) loss 0.5926 (0.5926) grad_norm 0.2690 (0.2690) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:46:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [145/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9134) loss 0.6279 (0.6002) grad_norm 0.3053 (0.3156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:48:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [145/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8960) loss 0.6168 (0.6000) grad_norm 0.3625 (0.3140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:49:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [145/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8902) loss 0.5777 (0.6007) grad_norm 0.3155 (0.3137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:51:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [145/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8871) loss 0.5620 (0.6008) grad_norm 0.3211 (0.3114) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:51:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 145 training takes 0:05:56 [2024-03-05 17:51:24 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_145.pth saving...... [2024-03-05 17:51:25 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_145.pth saved !!! [2024-03-05 17:51:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [146/800][0/402] eta 0:26:23 lr 0.000025 time 3.9392 (3.9392) loss 0.6010 (0.6010) grad_norm 0.3241 (0.3241) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 17:52:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [146/800][100/402] eta 0:04:34 lr 0.000025 time 0.8782 (0.9086) loss 0.6156 (0.6026) grad_norm 0.2719 (0.3141) loss_scale 524288.0000 (368558.8911) mem 30609MB [2024-03-05 17:54:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [146/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8936) loss 0.6151 (0.6014) grad_norm 0.3544 (0.3128) loss_scale 524288.0000 (446036.0597) mem 30609MB [2024-03-05 17:55:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [146/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8886) loss 0.5762 (0.6019) grad_norm 0.3100 (0.3122) loss_scale 524288.0000 (472033.3821) mem 30609MB [2024-03-05 17:57:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [146/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8860) loss 0.5899 (0.6007) grad_norm 0.2705 (0.3110) loss_scale 524288.0000 (485064.4589) mem 30609MB [2024-03-05 17:57:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 146 training takes 0:05:56 [2024-03-05 17:57:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [147/800][0/402] eta 0:29:33 lr 0.000025 time 4.4114 (4.4114) loss 0.5834 (0.5834) grad_norm 0.2580 (0.2580) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 17:58:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [147/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9135) loss 0.5959 (0.6051) grad_norm 0.2921 (0.3157) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:00:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [147/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8960) loss 0.6080 (0.6034) grad_norm 0.2888 (0.3154) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:01:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [147/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8902) loss 0.5770 (0.6022) grad_norm 0.2979 (0.3114) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:03:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [147/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8872) loss 0.6020 (0.6023) grad_norm 0.2778 (0.3113) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:03:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 147 training takes 0:05:56 [2024-03-05 18:03:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [148/800][0/402] eta 0:29:45 lr 0.000025 time 4.4424 (4.4424) loss 0.5704 (0.5704) grad_norm 0.2838 (0.2838) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:04:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [148/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9135) loss 0.6236 (0.6028) grad_norm 0.3029 (0.3027) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:06:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [148/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8960) loss 0.5956 (0.6039) grad_norm 0.3164 (0.3104) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:07:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [148/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8902) loss 0.5625 (0.6021) grad_norm 0.3081 (0.3113) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:09:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [148/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8872) loss 0.5966 (0.6011) grad_norm 0.2932 (0.3086) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:09:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 148 training takes 0:05:56 [2024-03-05 18:09:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [149/800][0/402] eta 0:29:18 lr 0.000025 time 4.3755 (4.3755) loss 0.6097 (0.6097) grad_norm 0.3161 (0.3161) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:10:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [149/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9129) loss 0.5568 (0.5991) grad_norm 0.3830 (0.3149) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:12:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [149/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8958) loss 0.6163 (0.6000) grad_norm 0.2980 (0.3166) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:13:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [149/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8900) loss 0.6270 (0.6004) grad_norm 0.3007 (0.3163) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:15:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [149/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8871) loss 0.6172 (0.6003) grad_norm 0.2701 (0.3133) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:15:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 149 training takes 0:05:56 [2024-03-05 18:15:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [150/800][0/402] eta 0:29:32 lr 0.000025 time 4.4092 (4.4092) loss 0.5750 (0.5750) grad_norm 0.2881 (0.2881) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:16:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [150/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9135) loss 0.5683 (0.5976) grad_norm 0.3467 (0.3138) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:18:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [150/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8960) loss 0.6256 (0.5987) grad_norm 0.2832 (nan) loss_scale 262144.0000 (397780.6965) mem 30609MB [2024-03-05 18:19:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [150/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8901) loss 0.6013 (0.5984) grad_norm 0.3153 (nan) loss_scale 262144.0000 (352718.6711) mem 30609MB [2024-03-05 18:21:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [150/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8871) loss 0.6296 (0.5995) grad_norm 0.2912 (nan) loss_scale 262144.0000 (330131.4713) mem 30609MB [2024-03-05 18:21:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 150 training takes 0:05:56 [2024-03-05 18:21:09 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_150.pth saving...... [2024-03-05 18:21:11 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_150.pth saved !!! [2024-03-05 18:21:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [151/800][0/402] eta 0:29:58 lr 0.000025 time 4.4750 (4.4750) loss 0.6032 (0.6032) grad_norm 0.3087 (0.3087) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:22:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [151/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9141) loss 0.5663 (0.6004) grad_norm 0.2817 (0.3071) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:24:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [151/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8964) loss 0.6207 (0.6018) grad_norm 0.3125 (0.3073) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:25:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [151/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8904) loss 0.6176 (0.6010) grad_norm 0.3354 (0.3082) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:27:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [151/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8874) loss 0.5962 (0.6013) grad_norm 0.2784 (0.3079) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:27:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 151 training takes 0:05:56 [2024-03-05 18:27:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [152/800][0/402] eta 0:29:40 lr 0.000025 time 4.4287 (4.4287) loss 0.5975 (0.5975) grad_norm 0.2619 (0.2619) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:28:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [152/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9137) loss 0.6002 (0.6000) grad_norm 0.3534 (0.3104) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:30:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [152/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8961) loss 0.5818 (0.5998) grad_norm 0.2882 (0.3057) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:31:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [152/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8902) loss 0.6076 (0.5994) grad_norm 0.2975 (0.3064) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:33:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [152/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8873) loss 0.5990 (0.6001) grad_norm 0.2836 (0.3064) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:33:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 152 training takes 0:05:56 [2024-03-05 18:33:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [153/800][0/402] eta 0:29:35 lr 0.000025 time 4.4172 (4.4172) loss 0.5960 (0.5960) grad_norm 0.3232 (0.3232) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:34:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [153/800][100/402] eta 0:04:35 lr 0.000025 time 0.8789 (0.9134) loss 0.6018 (0.6006) grad_norm 0.3025 (0.3101) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:36:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [153/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8960) loss 0.6128 (0.5985) grad_norm 0.2559 (0.3091) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:37:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [153/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8902) loss 0.6169 (0.5992) grad_norm 0.3096 (0.3089) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:39:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [153/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8872) loss 0.5802 (0.5989) grad_norm 0.2983 (0.3070) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:39:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 153 training takes 0:05:56 [2024-03-05 18:39:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [154/800][0/402] eta 0:29:29 lr 0.000025 time 4.4007 (4.4007) loss 0.6162 (0.6162) grad_norm 0.3184 (0.3184) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:40:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [154/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9134) loss 0.5859 (0.5982) grad_norm 0.3276 (0.3001) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:42:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [154/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8961) loss 0.5984 (0.6010) grad_norm 0.3439 (0.3011) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:43:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [154/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8902) loss 0.6021 (0.6015) grad_norm 0.2907 (0.3038) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:44:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [154/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8872) loss 0.5791 (0.6013) grad_norm 0.2854 (0.3036) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:44:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 154 training takes 0:05:56 [2024-03-05 18:45:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [155/800][0/402] eta 0:29:41 lr 0.000025 time 4.4325 (4.4325) loss 0.5714 (0.5714) grad_norm 0.3497 (0.3497) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:46:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [155/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9135) loss 0.5872 (0.6025) grad_norm 0.2993 (0.3042) loss_scale 524288.0000 (280312.3960) mem 30609MB [2024-03-05 18:47:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [155/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8960) loss 0.6211 (0.6021) grad_norm 0.3152 (0.2977) loss_scale 524288.0000 (401693.2935) mem 30609MB [2024-03-05 18:49:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [155/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8901) loss 0.6319 (0.6012) grad_norm 0.2629 (0.3006) loss_scale 524288.0000 (442422.4319) mem 30609MB [2024-03-05 18:50:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [155/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8872) loss 0.6070 (0.6006) grad_norm 0.2881 (0.3026) loss_scale 524288.0000 (462837.7855) mem 30609MB [2024-03-05 18:50:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 155 training takes 0:05:56 [2024-03-05 18:50:55 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_155.pth saving...... [2024-03-05 18:50:56 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_155.pth saved !!! [2024-03-05 18:51:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [156/800][0/402] eta 0:29:37 lr 0.000025 time 4.4223 (4.4223) loss 0.6155 (0.6155) grad_norm 0.3095 (0.3095) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:52:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [156/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9139) loss 0.6128 (0.5992) grad_norm 0.3735 (0.3030) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 18:53:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [156/800][200/402] eta 0:03:01 lr 0.000025 time 0.8798 (0.8967) loss 0.5844 (0.5978) grad_norm 0.3237 (inf) loss_scale 262144.0000 (460382.2488) mem 30609MB [2024-03-05 18:55:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [156/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8910) loss 0.6016 (0.5983) grad_norm 0.3191 (inf) loss_scale 262144.0000 (394522.3654) mem 30609MB [2024-03-05 18:56:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [156/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8880) loss 0.6165 (0.5986) grad_norm 0.3111 (inf) loss_scale 262144.0000 (361510.3042) mem 30609MB [2024-03-05 18:56:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 156 training takes 0:05:57 [2024-03-05 18:56:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [157/800][0/402] eta 0:29:20 lr 0.000025 time 4.3789 (4.3789) loss 0.5894 (0.5894) grad_norm 0.2769 (0.2769) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:58:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [157/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9129) loss 0.5962 (0.5981) grad_norm 0.2685 (0.2930) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 18:59:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [157/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.6248 (0.5997) grad_norm 0.3023 (0.3017) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:01:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [157/800][300/402] eta 0:01:30 lr 0.000025 time 0.8940 (0.8915) loss 0.5648 (0.6003) grad_norm 0.3107 (0.3016) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:02:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [157/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8905) loss 0.5898 (0.5996) grad_norm 0.3459 (0.3015) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:02:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 157 training takes 0:05:58 [2024-03-05 19:02:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [158/800][0/402] eta 0:29:26 lr 0.000025 time 4.3939 (4.3939) loss 0.5921 (0.5921) grad_norm 0.3612 (0.3612) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:04:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [158/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9132) loss 0.5899 (0.5979) grad_norm 0.3323 (0.2975) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:05:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [158/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8959) loss 0.6405 (0.5984) grad_norm 0.3374 (0.2969) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:07:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [158/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8901) loss 0.6265 (0.5988) grad_norm 0.3280 (0.2963) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:08:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [158/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8872) loss 0.5935 (0.5991) grad_norm 0.2878 (0.2984) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:08:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 158 training takes 0:05:56 [2024-03-05 19:08:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [159/800][0/402] eta 0:29:19 lr 0.000025 time 4.3767 (4.3767) loss 0.6038 (0.6038) grad_norm 0.3088 (0.3088) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:10:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [159/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9129) loss 0.5827 (0.5982) grad_norm 0.3745 (0.3057) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:11:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [159/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8957) loss 0.5888 (0.5988) grad_norm 0.3122 (0.3072) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:13:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [159/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.6237 (0.5991) grad_norm 0.2783 (0.3048) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:14:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [159/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.5963 (0.5985) grad_norm 0.2761 (0.3015) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:14:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 159 training takes 0:05:56 [2024-03-05 19:14:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [160/800][0/402] eta 0:29:16 lr 0.000025 time 4.3685 (4.3685) loss 0.6131 (0.6131) grad_norm 0.3217 (0.3217) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:16:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [160/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9128) loss 0.6237 (0.5994) grad_norm 0.2638 (0.3044) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:17:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [160/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8957) loss 0.6139 (0.6001) grad_norm 0.2576 (0.2988) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:19:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [160/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8900) loss 0.5964 (0.5988) grad_norm 0.2922 (0.2981) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:20:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [160/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8870) loss 0.6228 (0.5993) grad_norm 0.2849 (0.2962) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:20:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 160 training takes 0:05:56 [2024-03-05 19:20:42 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_160.pth saving...... [2024-03-05 19:20:44 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_160.pth saved !!! [2024-03-05 19:20:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [161/800][0/402] eta 0:29:00 lr 0.000025 time 4.3305 (4.3305) loss 0.5524 (0.5524) grad_norm 0.3190 (0.3190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:22:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [161/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9126) loss 0.5872 (0.6012) grad_norm 0.3483 (0.2953) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:23:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [161/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8958) loss 0.6015 (0.5994) grad_norm 0.2535 (0.2909) loss_scale 524288.0000 (339091.7413) mem 30609MB [2024-03-05 19:25:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [161/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8901) loss 0.5999 (0.5995) grad_norm 0.3354 (0.2917) loss_scale 524288.0000 (400618.7375) mem 30609MB [2024-03-05 19:26:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [161/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8871) loss 0.6025 (0.5998) grad_norm 0.3735 (inf) loss_scale 262144.0000 (377853.4464) mem 30609MB [2024-03-05 19:26:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 161 training takes 0:05:56 [2024-03-05 19:26:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [162/800][0/402] eta 0:29:29 lr 0.000025 time 4.4018 (4.4018) loss 0.5996 (0.5996) grad_norm 0.2846 (0.2846) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:28:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [162/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9132) loss 0.6011 (0.5994) grad_norm 0.2803 (0.2937) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:29:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [162/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8959) loss 0.6079 (0.6000) grad_norm 0.2569 (0.2958) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:31:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [162/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8901) loss 0.6144 (0.6004) grad_norm 0.3327 (0.2949) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:32:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [162/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8871) loss 0.6048 (0.5995) grad_norm 0.3289 (0.2944) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:32:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 162 training takes 0:05:56 [2024-03-05 19:32:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [163/800][0/402] eta 0:29:15 lr 0.000025 time 4.3682 (4.3682) loss 0.5995 (0.5995) grad_norm 0.2540 (0.2540) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:34:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [163/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9128) loss 0.5867 (0.5977) grad_norm 0.2908 (0.3018) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:35:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [163/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8957) loss 0.6091 (0.5988) grad_norm 0.3530 (0.3015) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:37:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [163/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8899) loss 0.6008 (0.5985) grad_norm 0.2708 (0.2973) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:38:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [163/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8870) loss 0.6267 (0.5981) grad_norm 0.2891 (0.2968) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:38:34 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 163 training takes 0:05:56 [2024-03-05 19:38:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [164/800][0/402] eta 0:29:29 lr 0.000025 time 4.4005 (4.4005) loss 0.5813 (0.5813) grad_norm 0.3162 (0.3162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:40:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [164/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9132) loss 0.6248 (0.5991) grad_norm 0.3288 (0.2939) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:41:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [164/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8959) loss 0.6081 (0.5985) grad_norm 0.2928 (0.2920) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:43:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [164/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8900) loss 0.5921 (0.5988) grad_norm 0.3143 (0.2924) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:44:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [164/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8871) loss 0.5855 (0.5991) grad_norm 0.3017 (0.2945) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:44:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 164 training takes 0:05:56 [2024-03-05 19:44:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [165/800][0/402] eta 0:29:19 lr 0.000025 time 4.3761 (4.3761) loss 0.5907 (0.5907) grad_norm 0.2527 (0.2527) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:46:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [165/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9129) loss 0.5779 (0.5982) grad_norm 0.2625 (0.2893) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:47:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [165/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8957) loss 0.5943 (0.5978) grad_norm 0.3129 (0.2901) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:48:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [165/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8900) loss 0.6135 (0.5988) grad_norm 0.2347 (0.2874) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:50:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [165/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8870) loss 0.5830 (0.5992) grad_norm 0.2622 (0.2883) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:50:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 165 training takes 0:05:56 [2024-03-05 19:50:28 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_165.pth saving...... [2024-03-05 19:50:29 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_165.pth saved !!! [2024-03-05 19:50:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [166/800][0/402] eta 0:29:45 lr 0.000025 time 4.4423 (4.4423) loss 0.5916 (0.5916) grad_norm 0.2948 (0.2948) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:52:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [166/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9135) loss 0.6108 (0.5985) grad_norm 0.3365 (0.2994) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:53:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [166/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8961) loss 0.5916 (0.5993) grad_norm 0.3181 (0.2948) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:54:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [166/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8902) loss 0.6276 (0.5982) grad_norm 0.2845 (0.2901) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 19:56:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [166/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8872) loss 0.5369 (0.5974) grad_norm 0.2835 (0.2912) loss_scale 524288.0000 (322286.7631) mem 30609MB [2024-03-05 19:56:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 166 training takes 0:05:56 [2024-03-05 19:56:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [167/800][0/402] eta 0:29:21 lr 0.000025 time 4.3821 (4.3821) loss 0.6359 (0.6359) grad_norm 0.2618 (0.2618) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 19:57:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [167/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9129) loss 0.5812 (0.5989) grad_norm 0.2964 (inf) loss_scale 262144.0000 (340008.5545) mem 30609MB [2024-03-05 19:59:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [167/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.5835 (0.5984) grad_norm 0.3481 (inf) loss_scale 262144.0000 (301269.9701) mem 30609MB [2024-03-05 20:00:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [167/800][300/402] eta 0:01:30 lr 0.000025 time 0.8776 (0.8900) loss 0.5961 (0.5982) grad_norm 0.2662 (inf) loss_scale 262144.0000 (288271.3090) mem 30609MB [2024-03-05 20:02:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [167/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8871) loss 0.6165 (0.5991) grad_norm 0.2747 (inf) loss_scale 262144.0000 (281755.7706) mem 30609MB [2024-03-05 20:02:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 167 training takes 0:05:56 [2024-03-05 20:02:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [168/800][0/402] eta 0:29:19 lr 0.000025 time 4.3778 (4.3778) loss 0.6202 (0.6202) grad_norm 0.2974 (0.2974) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:03:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [168/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9129) loss 0.5994 (0.5972) grad_norm 0.2654 (0.2971) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:05:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [168/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8957) loss 0.6112 (0.5987) grad_norm 0.2825 (0.2908) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:06:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [168/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.5811 (0.5986) grad_norm 0.2912 (0.2899) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:08:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [168/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8870) loss 0.5974 (0.5987) grad_norm 0.2908 (0.2884) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:08:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 168 training takes 0:05:56 [2024-03-05 20:08:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [169/800][0/402] eta 0:29:17 lr 0.000025 time 4.3719 (4.3719) loss 0.5657 (0.5657) grad_norm 0.2891 (0.2891) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:09:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [169/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9129) loss 0.6100 (0.5990) grad_norm 0.2680 (0.2815) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:11:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [169/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8957) loss 0.5923 (0.5995) grad_norm 0.3265 (0.2813) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:12:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [169/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.6088 (0.5989) grad_norm 0.2723 (0.2849) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:14:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [169/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8870) loss 0.6008 (0.5989) grad_norm 0.2584 (0.2845) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:14:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 169 training takes 0:05:56 [2024-03-05 20:14:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [170/800][0/402] eta 0:29:03 lr 0.000025 time 4.3374 (4.3374) loss 0.6295 (0.6295) grad_norm 0.3103 (0.3103) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:15:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [170/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9127) loss 0.5835 (0.5980) grad_norm 0.4088 (0.2910) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:17:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [170/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.6249 (0.5984) grad_norm 0.2505 (0.2925) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:18:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [170/800][300/402] eta 0:01:30 lr 0.000025 time 0.8791 (0.8899) loss 0.5965 (0.5974) grad_norm 0.2632 (0.2911) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:20:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [170/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.6115 (0.5973) grad_norm 0.3072 (0.2902) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:20:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 170 training takes 0:05:56 [2024-03-05 20:20:13 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_170.pth saving...... [2024-03-05 20:20:15 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_170.pth saved !!! [2024-03-05 20:20:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [171/800][0/402] eta 0:29:42 lr 0.000025 time 4.4330 (4.4330) loss 0.5729 (0.5729) grad_norm 0.2496 (0.2496) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:21:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [171/800][100/402] eta 0:04:35 lr 0.000025 time 0.8798 (0.9136) loss 0.5787 (0.5975) grad_norm 0.2365 (0.2776) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:23:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [171/800][200/402] eta 0:03:01 lr 0.000025 time 0.8807 (0.8963) loss 0.6030 (0.5975) grad_norm 0.2630 (0.2787) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:24:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [171/800][300/402] eta 0:01:30 lr 0.000025 time 0.8807 (0.8905) loss 0.6268 (0.5983) grad_norm 0.3004 (0.2827) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:26:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [171/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8875) loss 0.5898 (0.5978) grad_norm 0.2704 (0.2822) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:26:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 171 training takes 0:05:56 [2024-03-05 20:26:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [172/800][0/402] eta 0:29:35 lr 0.000025 time 4.4159 (4.4159) loss 0.5975 (0.5975) grad_norm 0.2936 (0.2936) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:27:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [172/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9135) loss 0.5930 (0.5984) grad_norm 0.2994 (0.2743) loss_scale 524288.0000 (472378.2970) mem 30609MB [2024-03-05 20:29:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [172/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8961) loss 0.6104 (0.5979) grad_norm 0.2994 (0.2816) loss_scale 524288.0000 (498204.0199) mem 30609MB [2024-03-05 20:30:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [172/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8903) loss 0.6158 (0.5975) grad_norm 0.2950 (0.2845) loss_scale 524288.0000 (506869.7940) mem 30609MB [2024-03-05 20:32:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [172/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8873) loss 0.5658 (0.5977) grad_norm 0.2392 (0.2835) loss_scale 524288.0000 (511213.4863) mem 30609MB [2024-03-05 20:32:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 172 training takes 0:05:56 [2024-03-05 20:32:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [173/800][0/402] eta 0:29:19 lr 0.000025 time 4.3766 (4.3766) loss 0.5757 (0.5757) grad_norm 0.2919 (0.2919) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:33:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [173/800][100/402] eta 0:04:36 lr 0.000025 time 0.8798 (0.9141) loss 0.5910 (0.5955) grad_norm 0.2909 (0.2869) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:35:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [173/800][200/402] eta 0:03:01 lr 0.000025 time 0.8796 (0.8969) loss 0.6137 (0.5967) grad_norm 0.2735 (0.2864) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:36:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [173/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8911) loss 0.5633 (0.5960) grad_norm 0.4125 (0.2852) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:38:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [173/800][400/402] eta 0:00:01 lr 0.000025 time 0.8779 (0.8882) loss 0.6132 (0.5966) grad_norm 0.2469 (0.2864) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:38:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 173 training takes 0:05:57 [2024-03-05 20:38:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [174/800][0/402] eta 0:29:18 lr 0.000025 time 4.3741 (4.3741) loss 0.6291 (0.6291) grad_norm 0.2727 (0.2727) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:39:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [174/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9129) loss 0.5713 (0.5987) grad_norm 0.2557 (0.2853) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:41:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [174/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8958) loss 0.6011 (0.5991) grad_norm 0.3575 (0.2834) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:42:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [174/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8900) loss 0.5824 (0.5983) grad_norm 0.3177 (0.2827) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:44:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [174/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8871) loss 0.5945 (0.5986) grad_norm 0.2621 (0.2823) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:44:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 174 training takes 0:05:56 [2024-03-05 20:44:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [175/800][0/402] eta 0:29:30 lr 0.000025 time 4.4049 (4.4049) loss 0.6106 (0.6106) grad_norm 0.2545 (0.2545) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:45:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [175/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9131) loss 0.6144 (0.5986) grad_norm 0.2756 (0.2775) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:47:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [175/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8958) loss 0.6069 (0.5978) grad_norm 0.2852 (0.2831) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:48:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [175/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8900) loss 0.5721 (0.5989) grad_norm 0.2565 (0.2824) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:49:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [175/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8871) loss 0.6183 (0.5977) grad_norm 0.3260 (0.2805) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:49:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 175 training takes 0:05:56 [2024-03-05 20:49:59 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_175.pth saving...... [2024-03-05 20:50:01 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_175.pth saved !!! [2024-03-05 20:50:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [176/800][0/402] eta 0:27:45 lr 0.000025 time 4.1443 (4.1443) loss 0.5781 (0.5781) grad_norm 0.3091 (0.3091) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:51:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [176/800][100/402] eta 0:04:35 lr 0.000025 time 0.8793 (0.9110) loss 0.6088 (0.5986) grad_norm 0.2944 (0.2828) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 20:53:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [176/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8948) loss 0.5928 (0.5979) grad_norm 0.2746 (nan) loss_scale 262144.0000 (455165.4527) mem 30609MB [2024-03-05 20:54:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [176/800][300/402] eta 0:01:30 lr 0.000025 time 0.8792 (0.8894) loss 0.5826 (0.5973) grad_norm 0.2527 (nan) loss_scale 262144.0000 (391038.7243) mem 30609MB [2024-03-05 20:55:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [176/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8867) loss 0.5610 (0.5969) grad_norm 0.2937 (nan) loss_scale 262144.0000 (358895.4015) mem 30609MB [2024-03-05 20:55:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 176 training takes 0:05:56 [2024-03-05 20:56:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [177/800][0/402] eta 0:29:26 lr 0.000025 time 4.3944 (4.3944) loss 0.6280 (0.6280) grad_norm 0.2618 (0.2618) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:57:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [177/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9133) loss 0.5960 (0.5954) grad_norm 0.2712 (0.2879) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 20:58:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [177/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8959) loss 0.6140 (0.5957) grad_norm 0.2419 (0.2840) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:00:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [177/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.5709 (0.5961) grad_norm 0.2965 (0.2812) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:01:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [177/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8871) loss 0.6133 (0.5957) grad_norm 0.2326 (0.2817) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:01:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 177 training takes 0:05:56 [2024-03-05 21:01:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [178/800][0/402] eta 0:29:19 lr 0.000025 time 4.3769 (4.3769) loss 0.6019 (0.6019) grad_norm 0.2654 (0.2654) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:03:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [178/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9129) loss 0.6079 (0.5960) grad_norm 0.2310 (0.2742) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:04:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [178/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8957) loss 0.5808 (0.5948) grad_norm 0.2871 (0.2760) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:06:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [178/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.6540 (0.5962) grad_norm 0.2241 (0.2803) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:07:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [178/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8870) loss 0.5751 (0.5955) grad_norm 0.2500 (0.2776) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:07:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 178 training takes 0:05:56 [2024-03-05 21:07:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [179/800][0/402] eta 0:29:32 lr 0.000025 time 4.4104 (4.4104) loss 0.6054 (0.6054) grad_norm 0.2375 (0.2375) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:09:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [179/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9132) loss 0.6173 (0.5961) grad_norm 0.2949 (0.2765) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:10:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [179/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8959) loss 0.5836 (0.5973) grad_norm 0.2867 (0.2779) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:12:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [179/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8901) loss 0.5536 (0.5973) grad_norm 0.2863 (0.2778) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:13:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [179/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8871) loss 0.5989 (0.5971) grad_norm 0.2704 (0.2756) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:13:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 179 training takes 0:05:56 [2024-03-05 21:13:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [180/800][0/402] eta 0:29:41 lr 0.000025 time 4.4304 (4.4304) loss 0.5885 (0.5885) grad_norm 0.2523 (0.2523) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:15:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [180/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9134) loss 0.5509 (0.5955) grad_norm 0.3353 (0.2844) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:16:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [180/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8959) loss 0.6016 (0.5958) grad_norm 0.2850 (0.2811) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:18:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [180/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8901) loss 0.5758 (0.5958) grad_norm 0.3056 (0.2792) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:19:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [180/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8871) loss 0.6010 (0.5962) grad_norm 0.2348 (0.2790) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:19:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 180 training takes 0:05:56 [2024-03-05 21:19:45 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_180.pth saving...... [2024-03-05 21:19:46 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_180.pth saved !!! [2024-03-05 21:19:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [181/800][0/402] eta 0:28:54 lr 0.000025 time 4.3139 (4.3139) loss 0.6230 (0.6230) grad_norm 0.2597 (0.2597) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:21:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [181/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9126) loss 0.5818 (0.5971) grad_norm 0.2891 (0.2811) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:22:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [181/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8956) loss 0.6117 (0.5971) grad_norm 0.2703 (0.2818) loss_scale 524288.0000 (344308.5373) mem 30609MB [2024-03-05 21:24:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [181/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8901) loss 0.6247 (0.5974) grad_norm 0.2630 (0.2749) loss_scale 524288.0000 (404102.3787) mem 30609MB [2024-03-05 21:25:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [181/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8872) loss 0.6106 (0.5968) grad_norm 0.2368 (0.2751) loss_scale 524288.0000 (434073.8554) mem 30609MB [2024-03-05 21:25:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 181 training takes 0:05:56 [2024-03-05 21:25:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [182/800][0/402] eta 0:29:39 lr 0.000025 time 4.4255 (4.4255) loss 0.5947 (0.5947) grad_norm 0.2964 (0.2964) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:27:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [182/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9133) loss 0.5940 (0.5970) grad_norm 0.2783 (0.2739) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:28:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [182/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8959) loss 0.6177 (0.5988) grad_norm 0.2391 (0.2768) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:30:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [182/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8900) loss 0.6255 (0.5983) grad_norm 0.3090 (0.2742) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:31:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [182/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.6149 (0.5972) grad_norm 0.2674 (0.2747) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:31:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 182 training takes 0:05:56 [2024-03-05 21:31:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [183/800][0/402] eta 0:29:33 lr 0.000025 time 4.4113 (4.4113) loss 0.5759 (0.5759) grad_norm 0.2841 (0.2841) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:33:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [183/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9132) loss 0.5946 (0.5960) grad_norm 0.2763 (0.2669) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:34:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [183/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8959) loss 0.6201 (0.5959) grad_norm 0.3069 (0.2664) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:36:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [183/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8900) loss 0.5971 (0.5954) grad_norm 0.2825 (0.2695) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:37:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [183/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8871) loss 0.5893 (0.5960) grad_norm 0.2378 (0.2709) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:37:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 183 training takes 0:05:56 [2024-03-05 21:37:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [184/800][0/402] eta 0:29:22 lr 0.000025 time 4.3838 (4.3838) loss 0.5901 (0.5901) grad_norm 0.2587 (0.2587) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:39:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [184/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9131) loss 0.5872 (0.5947) grad_norm 0.2653 (0.2747) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:40:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [184/800][200/402] eta 0:03:00 lr 0.000025 time 0.8793 (0.8958) loss 0.5972 (0.5946) grad_norm 0.2414 (0.2751) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:42:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [184/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8900) loss 0.6096 (0.5950) grad_norm 0.2764 (0.2767) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:43:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [184/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8870) loss 0.6094 (0.5947) grad_norm 0.2659 (0.2758) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:43:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 184 training takes 0:05:56 [2024-03-05 21:43:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [185/800][0/402] eta 0:29:28 lr 0.000025 time 4.3999 (4.3999) loss 0.5672 (0.5672) grad_norm 0.2792 (0.2792) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:45:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [185/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9131) loss 0.6140 (0.5987) grad_norm 0.2362 (0.2729) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:46:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [185/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8958) loss 0.5979 (0.5989) grad_norm 0.2521 (0.2685) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:48:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [185/800][300/402] eta 0:01:30 lr 0.000025 time 0.8773 (0.8900) loss 0.5795 (0.5985) grad_norm 0.2774 (0.2692) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 21:49:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [185/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8870) loss 0.5901 (0.5975) grad_norm 0.2994 (nan) loss_scale 262144.0000 (506637.4065) mem 30609MB [2024-03-05 21:49:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 185 training takes 0:05:56 [2024-03-05 21:49:30 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_185.pth saving...... [2024-03-05 21:49:32 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_185.pth saved !!! [2024-03-05 21:49:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [186/800][0/402] eta 0:29:47 lr 0.000025 time 4.4453 (4.4453) loss 0.6115 (0.6115) grad_norm 0.2290 (0.2290) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:51:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [186/800][100/402] eta 0:04:35 lr 0.000025 time 0.8790 (0.9138) loss 0.5751 (0.5965) grad_norm 0.3107 (0.2683) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:52:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [186/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8964) loss 0.6091 (0.5969) grad_norm 0.3005 (0.2673) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:54:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [186/800][300/402] eta 0:01:30 lr 0.000025 time 0.8795 (0.8906) loss 0.5992 (0.5969) grad_norm 0.2728 (0.2690) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:55:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [186/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8876) loss 0.5926 (0.5967) grad_norm 0.2949 (0.2705) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:55:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 186 training takes 0:05:56 [2024-03-05 21:55:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [187/800][0/402] eta 0:29:20 lr 0.000025 time 4.3796 (4.3796) loss 0.6071 (0.6071) grad_norm 0.2684 (0.2684) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:57:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [187/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9129) loss 0.6084 (0.5957) grad_norm 0.2435 (0.2717) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:58:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [187/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8957) loss 0.5699 (0.5946) grad_norm 0.2483 (0.2715) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 21:59:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [187/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8900) loss 0.5707 (0.5952) grad_norm 0.2568 (0.2729) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:01:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [187/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8870) loss 0.5772 (0.5948) grad_norm 0.3038 (0.2736) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:01:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 187 training takes 0:05:56 [2024-03-05 22:01:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [188/800][0/402] eta 0:29:12 lr 0.000025 time 4.3590 (4.3590) loss 0.5665 (0.5665) grad_norm 0.2786 (0.2786) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:02:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [188/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9126) loss 0.6017 (0.5953) grad_norm 0.2926 (0.2708) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:04:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [188/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8955) loss 0.6013 (0.5941) grad_norm 0.2786 (0.2705) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:05:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [188/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.6085 (0.5934) grad_norm 0.2395 (0.2702) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:07:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [188/800][400/402] eta 0:00:01 lr 0.000025 time 0.8780 (0.8870) loss 0.5800 (0.5946) grad_norm 0.3114 (0.2726) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:07:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 188 training takes 0:05:56 [2024-03-05 22:07:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [189/800][0/402] eta 0:29:09 lr 0.000025 time 4.3521 (4.3521) loss 0.6131 (0.6131) grad_norm 0.2609 (0.2609) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:08:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [189/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9126) loss 0.5909 (0.5947) grad_norm 0.2607 (0.2651) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:10:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [189/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8955) loss 0.6133 (0.5941) grad_norm 0.2398 (0.2657) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:11:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [189/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.6079 (0.5960) grad_norm 0.2379 (0.2656) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:13:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [189/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.5964 (0.5963) grad_norm 0.2948 (0.2661) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:13:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 189 training takes 0:05:56 [2024-03-05 22:13:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [190/800][0/402] eta 0:29:10 lr 0.000025 time 4.3539 (4.3539) loss 0.5927 (0.5927) grad_norm 0.2603 (0.2603) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:14:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [190/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9127) loss 0.5753 (0.5926) grad_norm 0.2287 (0.2710) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:16:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [190/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8956) loss 0.5514 (0.5942) grad_norm 0.3120 (0.2706) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:17:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [190/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8899) loss 0.5897 (0.5946) grad_norm 0.2619 (0.2691) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 22:19:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [190/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.5859 (0.5948) grad_norm 0.2382 (0.2669) loss_scale 524288.0000 (286331.8504) mem 30609MB [2024-03-05 22:19:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 190 training takes 0:05:56 [2024-03-05 22:19:16 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_190.pth saving...... [2024-03-05 22:19:17 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_190.pth saved !!! [2024-03-05 22:19:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [191/800][0/402] eta 0:29:25 lr 0.000025 time 4.3930 (4.3930) loss 0.5804 (0.5804) grad_norm 0.3181 (0.3181) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:20:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [191/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9131) loss 0.5896 (0.5946) grad_norm 0.2528 (0.2670) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:22:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [191/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8958) loss 0.6110 (0.5951) grad_norm 0.2425 (0.2646) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:23:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [191/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8900) loss 0.5542 (0.5950) grad_norm 0.2377 (0.2639) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:25:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [191/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8871) loss 0.6417 (0.5949) grad_norm 0.2511 (0.2632) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:25:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 191 training takes 0:05:56 [2024-03-05 22:25:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [192/800][0/402] eta 0:29:05 lr 0.000025 time 4.3416 (4.3416) loss 0.5957 (0.5957) grad_norm 0.2887 (0.2887) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:26:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [192/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9127) loss 0.6099 (0.5930) grad_norm 0.2600 (0.2658) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:28:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [192/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8956) loss 0.5936 (0.5939) grad_norm 0.2421 (0.2683) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:29:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [192/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8899) loss 0.5851 (0.5950) grad_norm 0.2464 (0.2683) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:31:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [192/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8870) loss 0.5910 (0.5949) grad_norm 0.2355 (0.2662) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:31:11 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 192 training takes 0:05:56 [2024-03-05 22:31:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [193/800][0/402] eta 0:29:12 lr 0.000025 time 4.3595 (4.3595) loss 0.5984 (0.5984) grad_norm 0.3044 (0.3044) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:32:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [193/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9129) loss 0.5987 (0.5937) grad_norm 0.2648 (0.2604) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:34:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [193/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8957) loss 0.5931 (0.5938) grad_norm 0.2750 (0.2594) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:35:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [193/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8900) loss 0.6209 (0.5944) grad_norm 0.2293 (0.2622) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:37:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [193/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8870) loss 0.5648 (0.5952) grad_norm 0.2422 (0.2646) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:37:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 193 training takes 0:05:56 [2024-03-05 22:37:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [194/800][0/402] eta 0:29:03 lr 0.000025 time 4.3381 (4.3381) loss 0.5973 (0.5973) grad_norm 0.2374 (0.2374) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:38:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [194/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9125) loss 0.5789 (0.5933) grad_norm 0.2440 (0.2602) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:40:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [194/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8956) loss 0.6205 (0.5953) grad_norm 0.2379 (0.2616) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:41:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [194/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.5952 (0.5966) grad_norm 0.3033 (0.2633) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:43:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [194/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8869) loss 0.5776 (0.5951) grad_norm 0.2540 (0.2644) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:43:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 194 training takes 0:05:56 [2024-03-05 22:43:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [195/800][0/402] eta 0:29:10 lr 0.000025 time 4.3554 (4.3554) loss 0.5704 (0.5704) grad_norm 0.2489 (0.2489) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:44:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [195/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9127) loss 0.5890 (0.5954) grad_norm 0.2715 (0.2614) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:46:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [195/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8956) loss 0.5952 (0.5943) grad_norm 0.2255 (0.2616) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:47:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [195/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.6061 (0.5950) grad_norm 0.2606 (0.2614) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:49:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [195/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.5937 (0.5956) grad_norm 0.2534 (inf) loss_scale 524288.0000 (533440.1596) mem 30609MB [2024-03-05 22:49:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 195 training takes 0:05:56 [2024-03-05 22:49:01 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_195.pth saving...... [2024-03-05 22:49:03 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_195.pth saved !!! [2024-03-05 22:49:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [196/800][0/402] eta 0:28:48 lr 0.000025 time 4.3002 (4.3002) loss 0.5732 (0.5732) grad_norm 0.2813 (0.2813) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:50:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [196/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9122) loss 0.5853 (0.5927) grad_norm 0.3286 (0.2626) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:52:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [196/800][200/402] eta 0:03:00 lr 0.000025 time 0.8788 (0.8957) loss 0.5838 (0.5951) grad_norm 0.2618 (0.2605) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:53:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [196/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8899) loss 0.6111 (0.5942) grad_norm 0.2420 (0.2636) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:54:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [196/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8870) loss 0.5777 (0.5944) grad_norm 0.2393 (0.2643) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:54:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 196 training takes 0:05:56 [2024-03-05 22:55:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [197/800][0/402] eta 0:28:58 lr 0.000025 time 4.3234 (4.3234) loss 0.6365 (0.6365) grad_norm 0.2670 (0.2670) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:56:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [197/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9123) loss 0.5470 (0.5958) grad_norm 0.2621 (0.2613) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:57:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [197/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8954) loss 0.5854 (0.5953) grad_norm 0.2381 (0.2642) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 22:59:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [197/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8897) loss 0.5795 (0.5942) grad_norm 0.2531 (0.2646) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:00:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [197/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8868) loss 0.5931 (0.5944) grad_norm 0.2451 (0.2641) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:00:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 197 training takes 0:05:56 [2024-03-05 23:01:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [198/800][0/402] eta 0:29:29 lr 0.000025 time 4.4008 (4.4008) loss 0.5853 (0.5853) grad_norm 0.2901 (0.2901) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:02:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [198/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9132) loss 0.6175 (0.5951) grad_norm 0.2205 (0.2589) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:03:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [198/800][200/402] eta 0:03:00 lr 0.000025 time 0.8790 (0.8959) loss 0.6154 (0.5952) grad_norm 0.2513 (0.2636) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:05:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [198/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8901) loss 0.5842 (0.5947) grad_norm 0.2728 (0.2654) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:06:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [198/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8871) loss 0.6112 (0.5956) grad_norm 0.2641 (0.2646) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:06:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 198 training takes 0:05:56 [2024-03-05 23:06:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [199/800][0/402] eta 0:28:37 lr 0.000025 time 4.2733 (4.2733) loss 0.5872 (0.5872) grad_norm 0.2472 (0.2472) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:08:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [199/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9118) loss 0.5849 (0.5940) grad_norm 0.2345 (0.2634) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:09:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [199/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8951) loss 0.5901 (0.5940) grad_norm 0.2889 (0.2610) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:11:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [199/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8895) loss 0.5514 (0.5941) grad_norm 0.3099 (nan) loss_scale 262144.0000 (472904.2924) mem 30609MB [2024-03-05 23:12:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [199/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8866) loss 0.5888 (0.5946) grad_norm 0.2559 (nan) loss_scale 262144.0000 (420345.6160) mem 30609MB [2024-03-05 23:12:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 199 training takes 0:05:56 [2024-03-05 23:12:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [200/800][0/402] eta 0:29:15 lr 0.000025 time 4.3681 (4.3681) loss 0.5945 (0.5945) grad_norm 0.2314 (0.2314) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:14:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [200/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9129) loss 0.5823 (0.5933) grad_norm 0.2414 (0.2591) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:15:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [200/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.6278 (0.5950) grad_norm 0.2628 (0.2591) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:17:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [200/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8899) loss 0.6172 (0.5949) grad_norm 0.2470 (0.2594) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:18:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [200/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8870) loss 0.5533 (0.5949) grad_norm 0.2305 (0.2577) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:18:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 200 training takes 0:05:56 [2024-03-05 23:18:46 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_200.pth saving...... [2024-03-05 23:18:48 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_200.pth saved !!! [2024-03-05 23:18:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [201/800][0/402] eta 0:29:09 lr 0.000025 time 4.3513 (4.3513) loss 0.5544 (0.5544) grad_norm 0.2313 (0.2313) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:20:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [201/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9127) loss 0.5661 (0.5933) grad_norm 0.2521 (0.2620) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:21:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [201/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8956) loss 0.6225 (0.5935) grad_norm 0.2535 (0.2627) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:23:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [201/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8898) loss 0.6557 (0.5939) grad_norm 0.2258 (0.2608) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:24:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [201/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.5964 (0.5943) grad_norm 0.2937 (0.2601) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:24:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 201 training takes 0:05:56 [2024-03-05 23:24:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [202/800][0/402] eta 0:29:33 lr 0.000025 time 4.4107 (4.4107) loss 0.5827 (0.5827) grad_norm 0.2426 (0.2426) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:26:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [202/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9132) loss 0.5872 (0.5948) grad_norm 0.2798 (0.2681) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:27:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [202/800][200/402] eta 0:03:00 lr 0.000025 time 0.8787 (0.8959) loss 0.5950 (0.5941) grad_norm 0.2536 (0.2613) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:29:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [202/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8901) loss 0.6252 (0.5939) grad_norm 0.2684 (0.2606) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:30:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [202/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8871) loss 0.5987 (0.5947) grad_norm 0.2560 (0.2598) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:30:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 202 training takes 0:05:56 [2024-03-05 23:30:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [203/800][0/402] eta 0:29:23 lr 0.000025 time 4.3872 (4.3872) loss 0.6143 (0.6143) grad_norm 0.2967 (0.2967) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:32:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [203/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9130) loss 0.6325 (0.5960) grad_norm 0.2426 (0.2711) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:33:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [203/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8957) loss 0.6128 (0.5954) grad_norm 0.2377 (0.2637) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:35:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [203/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.5491 (0.5944) grad_norm 0.2691 (0.2629) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:36:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [203/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.6031 (0.5945) grad_norm 0.2186 (0.2605) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:36:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 203 training takes 0:05:56 [2024-03-05 23:36:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [204/800][0/402] eta 0:29:14 lr 0.000025 time 4.3648 (4.3648) loss 0.5813 (0.5813) grad_norm 0.2348 (0.2348) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:38:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [204/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9127) loss 0.5763 (0.5909) grad_norm 0.2483 (0.2561) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:39:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [204/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8956) loss 0.6034 (0.5937) grad_norm 0.2572 (0.2614) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-05 23:41:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [204/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8899) loss 0.5870 (0.5942) grad_norm 0.2674 (0.2604) loss_scale 524288.0000 (322236.8106) mem 30609MB [2024-03-05 23:42:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [204/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8870) loss 0.6041 (0.5937) grad_norm 0.2953 (0.2599) loss_scale 524288.0000 (372623.6409) mem 30609MB [2024-03-05 23:42:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 204 training takes 0:05:56 [2024-03-05 23:42:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [205/800][0/402] eta 0:28:47 lr 0.000025 time 4.2978 (4.2978) loss 0.6344 (0.6344) grad_norm 0.2626 (0.2626) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:44:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [205/800][100/402] eta 0:04:35 lr 0.000025 time 0.8789 (0.9121) loss 0.5901 (0.5942) grad_norm 0.2937 (0.2676) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:45:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [205/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8953) loss 0.5816 (0.5946) grad_norm 0.2430 (0.2608) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:47:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [205/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8897) loss 0.5901 (0.5942) grad_norm 0.2745 (0.2565) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:48:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [205/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8868) loss 0.6067 (0.5946) grad_norm 0.2443 (0.2564) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:48:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 205 training takes 0:05:56 [2024-03-05 23:48:32 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_205.pth saving...... [2024-03-05 23:48:33 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_205.pth saved !!! [2024-03-05 23:48:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [206/800][0/402] eta 0:30:01 lr 0.000025 time 4.4824 (4.4824) loss 0.5880 (0.5880) grad_norm 0.2744 (0.2744) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:50:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [206/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9145) loss 0.5962 (0.5936) grad_norm 0.2256 (0.2640) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:51:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [206/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8967) loss 0.5897 (0.5920) grad_norm 0.2408 (0.2597) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:53:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [206/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8907) loss 0.5859 (0.5924) grad_norm 0.3224 (0.2609) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:54:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [206/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8876) loss 0.6039 (0.5931) grad_norm 0.2465 (0.2586) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:54:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 206 training takes 0:05:56 [2024-03-05 23:54:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [207/800][0/402] eta 0:29:19 lr 0.000025 time 4.3778 (4.3778) loss 0.5790 (0.5790) grad_norm 0.2387 (0.2387) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:56:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [207/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9133) loss 0.6256 (0.5939) grad_norm 0.2286 (0.2500) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:57:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [207/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8960) loss 0.5929 (0.5937) grad_norm 0.2343 (0.2546) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-05 23:58:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [207/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8901) loss 0.6036 (0.5942) grad_norm 0.2458 (0.2530) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:00:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [207/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8871) loss 0.5799 (0.5934) grad_norm 0.2833 (0.2564) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:00:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 207 training takes 0:05:56 [2024-03-06 00:00:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [208/800][0/402] eta 0:29:10 lr 0.000025 time 4.3540 (4.3540) loss 0.6125 (0.6125) grad_norm 0.2303 (0.2303) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:01:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [208/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9127) loss 0.5968 (0.5967) grad_norm 0.2186 (0.2507) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:03:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [208/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8956) loss 0.5665 (0.5947) grad_norm 0.2356 (0.2503) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:04:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [208/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8898) loss 0.6101 (0.5948) grad_norm 0.2572 (0.2542) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:06:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [208/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8869) loss 0.5994 (0.5952) grad_norm 0.2846 (0.2561) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:06:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 208 training takes 0:05:56 [2024-03-06 00:06:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [209/800][0/402] eta 0:29:05 lr 0.000025 time 4.3409 (4.3409) loss 0.5762 (0.5762) grad_norm 0.2758 (0.2758) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:07:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [209/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9125) loss 0.5844 (0.5913) grad_norm 0.2503 (0.2517) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:09:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [209/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8956) loss 0.5998 (0.5918) grad_norm 0.2431 (0.2545) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:10:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [209/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.5885 (0.5930) grad_norm 0.2301 (inf) loss_scale 524288.0000 (534738.9236) mem 30609MB [2024-03-06 00:12:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [209/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8869) loss 0.5980 (0.5929) grad_norm 0.2428 (inf) loss_scale 524288.0000 (532132.7082) mem 30609MB [2024-03-06 00:12:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 209 training takes 0:05:56 [2024-03-06 00:12:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [210/800][0/402] eta 0:29:09 lr 0.000025 time 4.3532 (4.3532) loss 0.5705 (0.5705) grad_norm 0.2628 (0.2628) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:13:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [210/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9127) loss 0.5994 (0.5940) grad_norm 0.2167 (0.2497) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:15:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [210/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8956) loss 0.5541 (0.5945) grad_norm 0.2812 (0.2531) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:16:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [210/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8899) loss 0.6195 (0.5946) grad_norm 0.2394 (0.2530) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:18:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [210/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8869) loss 0.5918 (0.5940) grad_norm 0.2475 (0.2538) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:18:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 210 training takes 0:05:56 [2024-03-06 00:18:17 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_210.pth saving...... [2024-03-06 00:18:19 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_210.pth saved !!! [2024-03-06 00:18:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [211/800][0/402] eta 0:29:35 lr 0.000025 time 4.4172 (4.4172) loss 0.5955 (0.5955) grad_norm 0.2520 (0.2520) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:19:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [211/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9133) loss 0.5884 (0.5967) grad_norm 0.2678 (0.2569) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:21:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [211/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8959) loss 0.5540 (0.5950) grad_norm 0.2583 (0.2568) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:22:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [211/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8902) loss 0.6042 (0.5949) grad_norm 0.2543 (0.2561) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:24:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [211/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8872) loss 0.5831 (0.5940) grad_norm 0.2522 (0.2554) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:24:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 211 training takes 0:05:56 [2024-03-06 00:24:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [212/800][0/402] eta 0:29:14 lr 0.000025 time 4.3643 (4.3643) loss 0.6076 (0.6076) grad_norm 0.2214 (0.2214) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:25:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [212/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9128) loss 0.5921 (0.5943) grad_norm 0.2400 (0.2503) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:27:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [212/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8957) loss 0.5712 (0.5937) grad_norm 0.2473 (0.2526) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 00:28:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [212/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8899) loss 0.6199 (0.5939) grad_norm 0.2437 (inf) loss_scale 262144.0000 (509482.5249) mem 30609MB [2024-03-06 00:30:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [212/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8869) loss 0.5965 (0.5937) grad_norm 0.2143 (inf) loss_scale 262144.0000 (447802.0948) mem 30609MB [2024-03-06 00:30:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 212 training takes 0:05:56 [2024-03-06 00:30:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [213/800][0/402] eta 0:29:10 lr 0.000025 time 4.3533 (4.3533) loss 0.5861 (0.5861) grad_norm 0.2802 (0.2802) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:31:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [213/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9126) loss 0.5913 (0.5919) grad_norm 0.2521 (0.2589) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:33:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [213/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8956) loss 0.6044 (0.5930) grad_norm 0.2365 (0.2572) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:34:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [213/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.5783 (0.5932) grad_norm 0.2503 (0.2553) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:36:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [213/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8870) loss 0.5844 (0.5932) grad_norm 0.2362 (0.2526) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:36:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 213 training takes 0:05:56 [2024-03-06 00:36:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [214/800][0/402] eta 0:29:21 lr 0.000025 time 4.3816 (4.3816) loss 0.6000 (0.6000) grad_norm 0.2419 (0.2419) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:37:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [214/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9131) loss 0.5996 (0.5922) grad_norm 0.2385 (0.2459) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:39:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [214/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8958) loss 0.5834 (0.5917) grad_norm 0.2545 (0.2522) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:40:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [214/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8900) loss 0.5793 (0.5921) grad_norm 0.2310 (0.2533) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:42:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [214/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.5930 (0.5926) grad_norm 0.2744 (0.2533) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:42:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 214 training takes 0:05:56 [2024-03-06 00:42:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [215/800][0/402] eta 0:28:59 lr 0.000025 time 4.3268 (4.3268) loss 0.5594 (0.5594) grad_norm 0.2414 (0.2414) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:43:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [215/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9124) loss 0.5950 (0.5934) grad_norm 0.2705 (0.2529) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:45:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [215/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.5990 (0.5932) grad_norm 0.2247 (0.2511) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:46:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [215/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8898) loss 0.5939 (0.5934) grad_norm 0.2343 (0.2526) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:48:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [215/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.5989 (0.5932) grad_norm 0.2905 (0.2529) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:48:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 215 training takes 0:05:56 [2024-03-06 00:48:02 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_215.pth saving...... [2024-03-06 00:48:04 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_215.pth saved !!! [2024-03-06 00:48:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [216/800][0/402] eta 0:28:28 lr 0.000025 time 4.2498 (4.2498) loss 0.6077 (0.6077) grad_norm 0.2542 (0.2542) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:49:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [216/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9120) loss 0.5520 (0.5954) grad_norm 0.2470 (0.2572) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:51:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [216/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8953) loss 0.6225 (0.5940) grad_norm 0.2553 (0.2553) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:52:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [216/800][300/402] eta 0:01:30 lr 0.000025 time 0.8775 (0.8897) loss 0.5916 (0.5935) grad_norm 0.2724 (0.2570) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:54:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [216/800][400/402] eta 0:00:01 lr 0.000025 time 0.8783 (0.8869) loss 0.5988 (0.5931) grad_norm 0.2518 (0.2548) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:54:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 216 training takes 0:05:56 [2024-03-06 00:54:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [217/800][0/402] eta 0:29:38 lr 0.000025 time 4.4233 (4.4233) loss 0.5876 (0.5876) grad_norm 0.2565 (0.2565) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:55:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [217/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9134) loss 0.5976 (0.5971) grad_norm 0.2760 (0.2508) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:57:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [217/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8960) loss 0.5781 (0.5950) grad_norm 0.2489 (0.2528) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 00:58:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [217/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8901) loss 0.6234 (0.5937) grad_norm 0.2445 (0.2514) loss_scale 524288.0000 (285658.5781) mem 30609MB [2024-03-06 00:59:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [217/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8871) loss 0.6089 (0.5929) grad_norm 0.2390 (0.2504) loss_scale 524288.0000 (345167.1621) mem 30609MB [2024-03-06 00:59:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 217 training takes 0:05:56 [2024-03-06 01:00:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [218/800][0/402] eta 0:29:07 lr 0.000025 time 4.3469 (4.3469) loss 0.6091 (0.6091) grad_norm 0.2465 (0.2465) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:01:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [218/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9127) loss 0.6064 (0.5945) grad_norm 0.2854 (0.2606) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:02:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [218/800][200/402] eta 0:03:00 lr 0.000025 time 0.8791 (0.8957) loss 0.6019 (0.5946) grad_norm 0.2603 (0.2568) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:04:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [218/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.5588 (0.5945) grad_norm 0.2503 (0.2542) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:05:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [218/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8870) loss 0.5738 (0.5941) grad_norm 0.2429 (0.2532) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:05:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 218 training takes 0:05:56 [2024-03-06 01:05:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [219/800][0/402] eta 0:28:57 lr 0.000025 time 4.3225 (4.3225) loss 0.5844 (0.5844) grad_norm 0.2004 (0.2004) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:07:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [219/800][100/402] eta 0:04:35 lr 0.000025 time 0.8775 (0.9124) loss 0.5955 (0.5913) grad_norm 0.2327 (0.2560) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:08:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [219/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8955) loss 0.5759 (0.5935) grad_norm 0.2505 (0.2523) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:10:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [219/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8898) loss 0.5713 (0.5934) grad_norm 0.2673 (0.2513) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:11:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [219/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8869) loss 0.6001 (0.5931) grad_norm 0.2632 (0.2519) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:11:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 219 training takes 0:05:56 [2024-03-06 01:11:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [220/800][0/402] eta 0:29:05 lr 0.000025 time 4.3423 (4.3423) loss 0.5762 (0.5762) grad_norm 0.2280 (0.2280) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:13:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [220/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9126) loss 0.5922 (0.5929) grad_norm 0.2423 (0.2561) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:14:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [220/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8955) loss 0.6185 (0.5956) grad_norm 0.2219 (0.2503) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:16:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [220/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.6108 (0.5947) grad_norm 0.2560 (0.2518) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:17:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [220/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8870) loss 0.6101 (0.5940) grad_norm 0.2475 (0.2533) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:17:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 220 training takes 0:05:56 [2024-03-06 01:17:48 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_220.pth saving...... [2024-03-06 01:17:49 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_220.pth saved !!! [2024-03-06 01:17:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [221/800][0/402] eta 0:29:59 lr 0.000025 time 4.4767 (4.4767) loss 0.6017 (0.6017) grad_norm 0.2247 (0.2247) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:19:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [221/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9141) loss 0.6348 (0.5918) grad_norm 0.2285 (0.2433) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:20:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [221/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8964) loss 0.6161 (0.5936) grad_norm 0.2375 (0.2430) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:22:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [221/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8904) loss 0.6003 (0.5933) grad_norm 0.2416 (0.2429) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:23:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [221/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8874) loss 0.5568 (0.5924) grad_norm 0.2793 (0.2445) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:23:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 221 training takes 0:05:56 [2024-03-06 01:23:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [222/800][0/402] eta 0:29:24 lr 0.000025 time 4.3890 (4.3890) loss 0.6005 (0.6005) grad_norm 0.2378 (0.2378) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:25:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [222/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9131) loss 0.5924 (0.5922) grad_norm 0.2398 (0.2494) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:26:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [222/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8958) loss 0.6105 (0.5932) grad_norm 0.2159 (0.2493) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:28:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [222/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8900) loss 0.5931 (0.5925) grad_norm 0.2119 (inf) loss_scale 524288.0000 (559124.4120) mem 30609MB [2024-03-06 01:29:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [222/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8870) loss 0.5875 (0.5922) grad_norm 0.2621 (inf) loss_scale 524288.0000 (550437.0274) mem 30609MB [2024-03-06 01:29:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 222 training takes 0:05:56 [2024-03-06 01:29:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [223/800][0/402] eta 0:29:14 lr 0.000025 time 4.3638 (4.3638) loss 0.5677 (0.5677) grad_norm 0.2371 (0.2371) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:31:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [223/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9130) loss 0.6136 (0.5893) grad_norm 0.2723 (0.2481) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:32:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [223/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8957) loss 0.5978 (0.5908) grad_norm 0.2206 (0.2508) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:34:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [223/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8900) loss 0.5998 (0.5912) grad_norm 0.2274 (0.2510) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:35:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [223/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.6010 (0.5919) grad_norm 0.2362 (0.2510) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:35:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 223 training takes 0:05:56 [2024-03-06 01:35:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [224/800][0/402] eta 0:28:56 lr 0.000025 time 4.3205 (4.3205) loss 0.6074 (0.6074) grad_norm 0.2640 (0.2640) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:37:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [224/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9124) loss 0.5629 (0.5928) grad_norm 0.2560 (0.2527) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:38:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [224/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8955) loss 0.6066 (0.5911) grad_norm 0.2198 (0.2486) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:40:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [224/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.5622 (0.5906) grad_norm 0.2487 (0.2457) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:41:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [224/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.6152 (0.5910) grad_norm 0.2308 (0.2473) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:41:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 224 training takes 0:05:56 [2024-03-06 01:41:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [225/800][0/402] eta 0:29:19 lr 0.000025 time 4.3764 (4.3764) loss 0.5809 (0.5809) grad_norm 0.2530 (0.2530) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:43:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [225/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9131) loss 0.5664 (0.5925) grad_norm 0.2476 (0.2520) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:44:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [225/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8959) loss 0.6340 (0.5952) grad_norm 0.2188 (0.2488) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 01:46:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [225/800][300/402] eta 0:01:30 lr 0.000025 time 0.8654 (0.8900) loss 0.5831 (0.5939) grad_norm nan (nan) loss_scale 262144.0000 (523417.0897) mem 30609MB [2024-03-06 01:47:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [225/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8871) loss 0.5925 (0.5934) grad_norm 0.2711 (nan) loss_scale 262144.0000 (458261.7057) mem 30609MB [2024-03-06 01:47:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 225 training takes 0:05:56 [2024-03-06 01:47:33 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_225.pth saving...... [2024-03-06 01:47:35 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_225.pth saved !!! [2024-03-06 01:47:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [226/800][0/402] eta 0:29:38 lr 0.000025 time 4.4244 (4.4244) loss 0.6113 (0.6113) grad_norm 0.2244 (0.2244) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:49:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [226/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9135) loss 0.5940 (0.5943) grad_norm 0.2589 (0.2426) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:50:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [226/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8960) loss 0.5822 (0.5935) grad_norm 0.2249 (0.2436) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:52:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [226/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8901) loss 0.5939 (0.5934) grad_norm 0.2409 (0.2460) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:53:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [226/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8871) loss 0.5496 (0.5926) grad_norm 0.3086 (0.2449) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:53:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 226 training takes 0:05:56 [2024-03-06 01:53:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [227/800][0/402] eta 0:29:16 lr 0.000025 time 4.3693 (4.3693) loss 0.6274 (0.6274) grad_norm 0.1990 (0.1990) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:55:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [227/800][100/402] eta 0:04:35 lr 0.000025 time 0.8773 (0.9128) loss 0.5897 (0.5922) grad_norm 0.3070 (0.2571) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:56:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [227/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8956) loss 0.5875 (0.5936) grad_norm 0.3261 (0.2516) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:57:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [227/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8899) loss 0.5696 (0.5924) grad_norm 0.2143 (0.2506) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:59:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [227/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.5972 (0.5930) grad_norm 0.2431 (0.2483) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 01:59:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 227 training takes 0:05:56 [2024-03-06 01:59:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [228/800][0/402] eta 0:28:57 lr 0.000025 time 4.3217 (4.3217) loss 0.5980 (0.5980) grad_norm 0.2536 (0.2536) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:01:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [228/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9124) loss 0.5907 (0.5914) grad_norm 0.1919 (0.2425) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:02:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [228/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8954) loss 0.5924 (0.5916) grad_norm 0.2716 (0.2458) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:03:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [228/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8897) loss 0.6078 (0.5914) grad_norm 0.2472 (0.2467) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:05:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [228/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8868) loss 0.6032 (0.5912) grad_norm 0.2645 (0.2480) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:05:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 228 training takes 0:05:56 [2024-03-06 02:05:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [229/800][0/402] eta 0:29:09 lr 0.000025 time 4.3518 (4.3518) loss 0.6226 (0.6226) grad_norm 0.2575 (0.2575) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:06:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [229/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9126) loss 0.6386 (0.5925) grad_norm 0.2709 (0.2451) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:08:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [229/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8955) loss 0.5934 (0.5911) grad_norm 0.2139 (0.2432) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:09:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [229/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8898) loss 0.6107 (0.5916) grad_norm 0.3222 (0.2466) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:11:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [229/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8869) loss 0.5990 (0.5917) grad_norm 0.2559 (0.2475) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:11:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 229 training takes 0:05:56 [2024-03-06 02:11:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [230/800][0/402] eta 0:29:01 lr 0.000025 time 4.3330 (4.3330) loss 0.6350 (0.6350) grad_norm 0.2748 (0.2748) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:12:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [230/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9126) loss 0.5982 (0.5896) grad_norm 0.2292 (0.2483) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:14:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [230/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.6120 (0.5920) grad_norm 0.2752 (0.2464) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:15:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [230/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8898) loss 0.5621 (0.5910) grad_norm 0.2493 (0.2472) loss_scale 524288.0000 (271724.0133) mem 30609MB [2024-03-06 02:17:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [230/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.5854 (0.5914) grad_norm 0.2539 (0.2464) loss_scale 524288.0000 (334707.5511) mem 30609MB [2024-03-06 02:17:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 230 training takes 0:05:56 [2024-03-06 02:17:18 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_230.pth saving...... [2024-03-06 02:17:20 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_230.pth saved !!! [2024-03-06 02:17:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [231/800][0/402] eta 0:29:31 lr 0.000025 time 4.4064 (4.4064) loss 0.6047 (0.6047) grad_norm 0.2181 (0.2181) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:18:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [231/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9135) loss 0.5549 (0.5900) grad_norm 0.2340 (0.2494) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:20:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [231/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8961) loss 0.5683 (0.5916) grad_norm 0.2878 (0.2491) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:21:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [231/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8902) loss 0.6105 (0.5930) grad_norm 0.2413 (0.2484) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:23:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [231/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8872) loss 0.5648 (0.5919) grad_norm 0.2367 (0.2489) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:23:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 231 training takes 0:05:56 [2024-03-06 02:23:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [232/800][0/402] eta 0:29:09 lr 0.000025 time 4.3511 (4.3511) loss 0.6084 (0.6084) grad_norm 0.2560 (0.2560) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:24:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [232/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9127) loss 0.5928 (0.5912) grad_norm 0.2417 (0.2452) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:26:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [232/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8957) loss 0.6106 (0.5925) grad_norm 0.2573 (0.2458) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:27:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [232/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8899) loss 0.5280 (0.5920) grad_norm 0.2405 (0.2431) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:29:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [232/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8870) loss 0.5978 (0.5920) grad_norm 0.2514 (0.2427) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:29:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 232 training takes 0:05:56 [2024-03-06 02:29:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [233/800][0/402] eta 0:29:02 lr 0.000025 time 4.3352 (4.3352) loss 0.6024 (0.6024) grad_norm 0.2374 (0.2374) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:30:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [233/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9124) loss 0.5794 (0.5892) grad_norm 0.2470 (0.2415) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:32:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [233/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8955) loss 0.5874 (0.5924) grad_norm 0.2846 (0.2412) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:33:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [233/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8898) loss 0.5781 (0.5913) grad_norm 0.2447 (0.2436) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:35:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [233/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8869) loss 0.6069 (0.5918) grad_norm 0.2318 (0.2452) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:35:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 233 training takes 0:05:56 [2024-03-06 02:35:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [234/800][0/402] eta 0:29:17 lr 0.000025 time 4.3723 (4.3723) loss 0.5882 (0.5882) grad_norm 0.2449 (0.2449) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:36:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [234/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9130) loss 0.5842 (0.5897) grad_norm 0.2162 (0.2480) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:38:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [234/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8957) loss 0.5713 (0.5904) grad_norm 0.2248 (0.2443) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 02:39:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [234/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8899) loss 0.6058 (0.5913) grad_norm 0.2348 (nan) loss_scale 262144.0000 (459840.6379) mem 30609MB [2024-03-06 02:41:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [234/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8870) loss 0.5834 (0.5914) grad_norm 0.2878 (nan) loss_scale 262144.0000 (410539.7307) mem 30609MB [2024-03-06 02:41:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 234 training takes 0:05:56 [2024-03-06 02:41:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [235/800][0/402] eta 0:29:08 lr 0.000025 time 4.3483 (4.3483) loss 0.5958 (0.5958) grad_norm 0.2963 (0.2963) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:42:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [235/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9125) loss 0.5968 (0.5924) grad_norm 0.2538 (0.2448) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:44:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [235/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.5917 (0.5911) grad_norm 0.2370 (0.2471) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:45:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [235/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.5958 (0.5910) grad_norm 0.2412 (0.2467) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:47:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [235/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8869) loss 0.6229 (0.5913) grad_norm 0.2324 (0.2468) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:47:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 235 training takes 0:05:56 [2024-03-06 02:47:04 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_235.pth saving...... [2024-03-06 02:47:05 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_235.pth saved !!! [2024-03-06 02:47:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [236/800][0/402] eta 0:30:05 lr 0.000025 time 4.4916 (4.4916) loss 0.5612 (0.5612) grad_norm 0.2299 (0.2299) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:48:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [236/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9142) loss 0.5706 (0.5915) grad_norm 0.2425 (0.2413) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:50:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [236/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8965) loss 0.6033 (0.5912) grad_norm 0.2805 (0.2414) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:51:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [236/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8905) loss 0.5815 (0.5907) grad_norm 0.2527 (0.2436) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:53:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [236/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8875) loss 0.5989 (0.5920) grad_norm 0.2452 (0.2428) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:53:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 236 training takes 0:05:56 [2024-03-06 02:53:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [237/800][0/402] eta 0:29:21 lr 0.000025 time 4.3810 (4.3810) loss 0.5662 (0.5662) grad_norm 0.2204 (0.2204) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:54:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [237/800][100/402] eta 0:04:36 lr 0.000025 time 0.8794 (0.9140) loss 0.5689 (0.5891) grad_norm 0.2111 (0.2406) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:56:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [237/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8968) loss 0.5966 (0.5908) grad_norm 0.2152 (0.2415) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:57:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [237/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8909) loss 0.6053 (0.5918) grad_norm 0.2541 (0.2422) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:58:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [237/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.5816 (0.5923) grad_norm 0.2191 (0.2423) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 02:58:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 237 training takes 0:05:57 [2024-03-06 02:59:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [238/800][0/402] eta 0:29:22 lr 0.000025 time 4.3846 (4.3846) loss 0.5748 (0.5748) grad_norm 0.2370 (0.2370) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:00:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [238/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9129) loss 0.5833 (0.5929) grad_norm 0.2610 (0.2388) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:01:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [238/800][200/402] eta 0:03:00 lr 0.000025 time 0.8779 (0.8957) loss 0.5796 (0.5908) grad_norm 0.2165 (0.2409) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:03:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [238/800][300/402] eta 0:01:30 lr 0.000025 time 0.8778 (0.8899) loss 0.5911 (0.5915) grad_norm 0.2274 (0.2406) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:04:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [238/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8870) loss 0.5921 (0.5917) grad_norm 0.2260 (0.2408) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:04:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 238 training takes 0:05:56 [2024-03-06 03:05:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [239/800][0/402] eta 0:29:00 lr 0.000025 time 4.3284 (4.3284) loss 0.5890 (0.5890) grad_norm 0.2351 (0.2351) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:06:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [239/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9124) loss 0.6072 (0.5910) grad_norm 0.2360 (0.2442) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:07:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [239/800][200/402] eta 0:03:00 lr 0.000025 time 0.8798 (0.8955) loss 0.6192 (0.5904) grad_norm 0.2711 (0.2437) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:09:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [239/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8898) loss 0.6078 (0.5921) grad_norm 0.2295 (0.2432) loss_scale 524288.0000 (335300.4651) mem 30609MB [2024-03-06 03:10:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [239/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8869) loss 0.6111 (0.5918) grad_norm 0.2160 (0.2433) loss_scale 524288.0000 (382429.5262) mem 30609MB [2024-03-06 03:10:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 239 training takes 0:05:56 [2024-03-06 03:10:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [240/800][0/402] eta 0:29:17 lr 0.000025 time 4.3718 (4.3718) loss 0.5589 (0.5589) grad_norm 0.2863 (0.2863) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:12:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [240/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9128) loss 0.5973 (0.5917) grad_norm 0.2275 (0.2540) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:13:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [240/800][200/402] eta 0:03:00 lr 0.000025 time 0.8802 (0.8957) loss 0.6109 (0.5925) grad_norm 0.2587 (0.2473) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:15:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [240/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8899) loss 0.5887 (0.5930) grad_norm 0.2342 (0.2459) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:16:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [240/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8870) loss 0.5576 (0.5928) grad_norm 0.2123 (0.2446) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:16:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 240 training takes 0:05:56 [2024-03-06 03:16:50 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_240.pth saving...... [2024-03-06 03:16:51 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_240.pth saved !!! [2024-03-06 03:16:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [241/800][0/402] eta 0:30:47 lr 0.000025 time 4.5952 (4.5952) loss 0.5816 (0.5816) grad_norm 0.2030 (0.2030) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:18:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [241/800][100/402] eta 0:04:36 lr 0.000025 time 0.8789 (0.9152) loss 0.6021 (0.5921) grad_norm 0.2739 (nan) loss_scale 262144.0000 (391918.2574) mem 30609MB [2024-03-06 03:19:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [241/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8970) loss 0.5937 (0.5919) grad_norm 0.2603 (nan) loss_scale 262144.0000 (327353.9502) mem 30609MB [2024-03-06 03:21:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [241/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8910) loss 0.6036 (0.5927) grad_norm 0.2228 (nan) loss_scale 262144.0000 (305689.5150) mem 30609MB [2024-03-06 03:22:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [241/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8878) loss 0.5835 (0.5918) grad_norm 0.2535 (nan) loss_scale 262144.0000 (294830.2843) mem 30609MB [2024-03-06 03:22:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 241 training takes 0:05:57 [2024-03-06 03:22:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [242/800][0/402] eta 0:29:17 lr 0.000025 time 4.3711 (4.3711) loss 0.6057 (0.6057) grad_norm 0.2271 (0.2271) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:24:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [242/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9129) loss 0.5955 (0.5932) grad_norm 0.2185 (0.2366) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:25:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [242/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8957) loss 0.6034 (0.5911) grad_norm 0.2259 (0.2344) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:27:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [242/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8899) loss 0.6009 (0.5907) grad_norm 0.2073 (0.2376) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:28:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [242/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.5804 (0.5912) grad_norm 0.2382 (0.2390) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:28:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 242 training takes 0:05:56 [2024-03-06 03:28:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [243/800][0/402] eta 0:29:25 lr 0.000025 time 4.3917 (4.3917) loss 0.5891 (0.5891) grad_norm 0.2300 (0.2300) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:30:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [243/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9130) loss 0.5983 (0.5919) grad_norm 0.2153 (0.2423) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:31:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [243/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8958) loss 0.5919 (0.5940) grad_norm 0.2217 (0.2437) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:33:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [243/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.5907 (0.5927) grad_norm 0.2326 (0.2431) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:34:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [243/800][400/402] eta 0:00:01 lr 0.000025 time 0.8784 (0.8870) loss 0.6385 (0.5924) grad_norm 0.3151 (0.2421) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:34:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 243 training takes 0:05:56 [2024-03-06 03:34:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [244/800][0/402] eta 0:29:10 lr 0.000025 time 4.3543 (4.3543) loss 0.6112 (0.6112) grad_norm 0.2355 (0.2355) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:36:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [244/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9126) loss 0.5828 (0.5943) grad_norm 0.2255 (0.2430) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:37:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [244/800][200/402] eta 0:03:00 lr 0.000025 time 0.8787 (0.8956) loss 0.5387 (0.5920) grad_norm 0.2165 (0.2403) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:39:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [244/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.5858 (0.5925) grad_norm 0.2670 (0.2392) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:40:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [244/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8869) loss 0.5680 (0.5928) grad_norm 0.2888 (0.2396) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:40:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 244 training takes 0:05:56 [2024-03-06 03:40:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [245/800][0/402] eta 0:29:29 lr 0.000025 time 4.4013 (4.4013) loss 0.6161 (0.6161) grad_norm 0.2451 (0.2451) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:42:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [245/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9132) loss 0.5825 (0.5929) grad_norm 0.2471 (0.2379) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:43:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [245/800][200/402] eta 0:03:00 lr 0.000025 time 0.8778 (0.8959) loss 0.5709 (0.5903) grad_norm 0.2250 (0.2385) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:45:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [245/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8900) loss 0.5819 (0.5904) grad_norm 0.3248 (0.2395) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:46:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [245/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8870) loss 0.6114 (0.5907) grad_norm 0.2404 (0.2420) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:46:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 245 training takes 0:05:56 [2024-03-06 03:46:35 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_245.pth saving...... [2024-03-06 03:46:37 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_245.pth saved !!! [2024-03-06 03:46:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [246/800][0/402] eta 0:28:40 lr 0.000025 time 4.2797 (4.2797) loss 0.5857 (0.5857) grad_norm 0.2408 (0.2408) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 03:48:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [246/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9120) loss 0.5305 (0.5920) grad_norm 0.2696 (0.2387) loss_scale 524288.0000 (420468.5941) mem 30609MB [2024-03-06 03:49:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [246/800][200/402] eta 0:03:00 lr 0.000025 time 0.8803 (0.8953) loss 0.5747 (0.5915) grad_norm 0.3002 (0.2409) loss_scale 524288.0000 (472120.0398) mem 30609MB [2024-03-06 03:51:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [246/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8896) loss 0.5855 (0.5907) grad_norm 0.2564 (0.2404) loss_scale 524288.0000 (489451.5880) mem 30609MB [2024-03-06 03:52:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [246/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8868) loss 0.5915 (0.5913) grad_norm 0.1992 (0.2394) loss_scale 524288.0000 (498138.9726) mem 30609MB [2024-03-06 03:52:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 246 training takes 0:05:56 [2024-03-06 03:52:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [247/800][0/402] eta 0:29:16 lr 0.000025 time 4.3689 (4.3689) loss 0.5443 (0.5443) grad_norm 0.2691 (0.2691) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:54:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [247/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9129) loss 0.5885 (0.5925) grad_norm 0.2468 (0.2385) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:55:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [247/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8957) loss 0.5774 (0.5909) grad_norm 0.2828 (0.2416) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:57:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [247/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8899) loss 0.5923 (0.5909) grad_norm 0.1963 (0.2399) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:58:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [247/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8870) loss 0.6280 (0.5908) grad_norm 0.2338 (0.2393) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 03:58:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 247 training takes 0:05:56 [2024-03-06 03:58:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [248/800][0/402] eta 0:29:05 lr 0.000025 time 4.3409 (4.3409) loss 0.5934 (0.5934) grad_norm 0.2134 (0.2134) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:00:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [248/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9125) loss 0.5871 (0.5897) grad_norm 0.2544 (0.2393) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:01:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [248/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.5853 (0.5899) grad_norm 0.2884 (0.2395) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:02:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [248/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8898) loss 0.6185 (0.5897) grad_norm 0.2324 (0.2405) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:04:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [248/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.5826 (0.5898) grad_norm 0.2997 (0.2397) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:04:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 248 training takes 0:05:56 [2024-03-06 04:04:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [249/800][0/402] eta 0:29:13 lr 0.000025 time 4.3618 (4.3618) loss 0.5867 (0.5867) grad_norm 0.2570 (0.2570) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:05:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [249/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9127) loss 0.5850 (0.5927) grad_norm 0.2578 (0.2358) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:07:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [249/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8957) loss 0.5943 (0.5924) grad_norm 0.2582 (0.2369) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:08:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [249/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8900) loss 0.5312 (0.5925) grad_norm 0.2666 (0.2384) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:10:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [249/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8870) loss 0.5896 (0.5917) grad_norm 0.2329 (0.2368) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:10:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 249 training takes 0:05:56 [2024-03-06 04:10:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [250/800][0/402] eta 0:29:07 lr 0.000025 time 4.3482 (4.3482) loss 0.5755 (0.5755) grad_norm 0.2321 (0.2321) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:11:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [250/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9126) loss 0.5900 (0.5934) grad_norm 0.2332 (0.2367) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:13:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [250/800][200/402] eta 0:03:00 lr 0.000025 time 0.8787 (0.8956) loss 0.5682 (0.5916) grad_norm 0.2180 (0.2353) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:14:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [250/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8899) loss 0.5715 (0.5917) grad_norm 0.2054 (0.2366) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:16:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [250/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8869) loss 0.5515 (0.5912) grad_norm 0.2899 (0.2368) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:16:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 250 training takes 0:05:56 [2024-03-06 04:16:20 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_250.pth saving...... [2024-03-06 04:16:22 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_250.pth saved !!! [2024-03-06 04:16:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [251/800][0/402] eta 0:28:24 lr 0.000025 time 4.2410 (4.2410) loss 0.5670 (0.5670) grad_norm 0.2261 (0.2261) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:17:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [251/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9115) loss 0.6212 (0.5867) grad_norm 0.2425 (inf) loss_scale 524288.0000 (705971.9604) mem 30609MB [2024-03-06 04:19:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [251/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8951) loss 0.6071 (0.5898) grad_norm 0.2332 (inf) loss_scale 524288.0000 (615581.9303) mem 30609MB [2024-03-06 04:20:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [251/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8895) loss 0.5815 (0.5896) grad_norm 0.2669 (inf) loss_scale 524288.0000 (585251.7209) mem 30609MB [2024-03-06 04:22:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [251/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8867) loss 0.5803 (0.5901) grad_norm 0.2586 (inf) loss_scale 524288.0000 (570048.7980) mem 30609MB [2024-03-06 04:22:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 251 training takes 0:05:56 [2024-03-06 04:22:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [252/800][0/402] eta 0:28:43 lr 0.000025 time 4.2868 (4.2868) loss 0.6165 (0.6165) grad_norm 0.2307 (0.2307) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:23:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [252/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9119) loss 0.5884 (0.5908) grad_norm 0.2015 (0.2342) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:25:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [252/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8952) loss 0.6138 (0.5896) grad_norm 0.2406 (0.2362) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:26:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [252/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8896) loss 0.6013 (0.5897) grad_norm 0.2372 (0.2352) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:28:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [252/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8868) loss 0.5819 (0.5902) grad_norm 0.2541 (0.2362) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:28:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 252 training takes 0:05:56 [2024-03-06 04:28:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [253/800][0/402] eta 0:29:13 lr 0.000025 time 4.3628 (4.3628) loss 0.5824 (0.5824) grad_norm 0.2395 (0.2395) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:29:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [253/800][100/402] eta 0:04:35 lr 0.000025 time 0.8790 (0.9130) loss 0.5995 (0.5917) grad_norm 0.2232 (0.2354) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:31:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [253/800][200/402] eta 0:03:00 lr 0.000025 time 0.8788 (0.8958) loss 0.5907 (0.5930) grad_norm 0.2217 (0.2374) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:32:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [253/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8900) loss 0.5691 (0.5930) grad_norm 0.2739 (0.2369) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:34:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [253/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8871) loss 0.5474 (0.5914) grad_norm 0.2258 (0.2384) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:34:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 253 training takes 0:05:56 [2024-03-06 04:34:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [254/800][0/402] eta 0:29:18 lr 0.000025 time 4.3747 (4.3747) loss 0.5685 (0.5685) grad_norm 0.2277 (0.2277) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:35:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [254/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9130) loss 0.5557 (0.5900) grad_norm 0.2437 (0.2431) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 04:37:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [254/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8956) loss 0.6047 (0.5896) grad_norm 0.2508 (nan) loss_scale 262144.0000 (395172.2985) mem 30609MB [2024-03-06 04:38:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [254/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8899) loss 0.6003 (0.5905) grad_norm 0.2001 (nan) loss_scale 262144.0000 (350976.8505) mem 30609MB [2024-03-06 04:40:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [254/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8869) loss 0.6230 (0.5909) grad_norm 0.2138 (nan) loss_scale 262144.0000 (328824.0200) mem 30609MB [2024-03-06 04:40:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 254 training takes 0:05:56 [2024-03-06 04:40:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [255/800][0/402] eta 0:29:26 lr 0.000025 time 4.3936 (4.3936) loss 0.5702 (0.5702) grad_norm 0.2431 (0.2431) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:41:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [255/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9130) loss 0.5621 (0.5879) grad_norm 0.2359 (0.2354) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:43:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [255/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8958) loss 0.5936 (0.5901) grad_norm 0.2264 (0.2351) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:44:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [255/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.5690 (0.5895) grad_norm 0.2260 (0.2357) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:46:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [255/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.6068 (0.5903) grad_norm 0.2768 (0.2372) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:46:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 255 training takes 0:05:56 [2024-03-06 04:46:05 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_255.pth saving...... [2024-03-06 04:46:07 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_255.pth saved !!! [2024-03-06 04:46:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [256/800][0/402] eta 0:29:18 lr 0.000025 time 4.3754 (4.3754) loss 0.5703 (0.5703) grad_norm 0.2290 (0.2290) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:47:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [256/800][100/402] eta 0:04:36 lr 0.000025 time 0.8794 (0.9140) loss 0.5739 (0.5920) grad_norm 0.2402 (0.2407) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:49:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [256/800][200/402] eta 0:03:01 lr 0.000025 time 0.8796 (0.8968) loss 0.5859 (0.5911) grad_norm 0.2009 (0.2396) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:50:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [256/800][300/402] eta 0:01:30 lr 0.000025 time 0.8793 (0.8910) loss 0.5685 (0.5907) grad_norm 0.2214 (0.2417) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:52:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [256/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8881) loss 0.5948 (0.5915) grad_norm 0.2649 (0.2402) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:52:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 256 training takes 0:05:57 [2024-03-06 04:52:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [257/800][0/402] eta 0:29:13 lr 0.000025 time 4.3612 (4.3612) loss 0.5913 (0.5913) grad_norm 0.2242 (0.2242) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:53:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [257/800][100/402] eta 0:04:35 lr 0.000025 time 0.8792 (0.9138) loss 0.6144 (0.5907) grad_norm 0.2435 (0.2353) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:55:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [257/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8967) loss 0.6037 (0.5902) grad_norm 0.2507 (0.2385) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:56:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [257/800][300/402] eta 0:01:30 lr 0.000025 time 0.8796 (0.8910) loss 0.5910 (0.5896) grad_norm 0.2303 (0.2372) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:58:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [257/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8879) loss 0.5608 (0.5897) grad_norm 0.2030 (0.2364) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:58:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 257 training takes 0:05:57 [2024-03-06 04:58:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [258/800][0/402] eta 0:29:01 lr 0.000025 time 4.3327 (4.3327) loss 0.5604 (0.5604) grad_norm 0.2832 (0.2832) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 04:59:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [258/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9124) loss 0.6270 (0.5909) grad_norm 0.2080 (0.2354) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:01:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [258/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8955) loss 0.5759 (0.5900) grad_norm 0.2272 (0.2362) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:02:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [258/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8898) loss 0.6418 (0.5905) grad_norm 0.2640 (0.2355) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:03:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [258/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8869) loss 0.5728 (0.5902) grad_norm 0.2515 (0.2360) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:03:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 258 training takes 0:05:56 [2024-03-06 05:04:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [259/800][0/402] eta 0:29:06 lr 0.000025 time 4.3453 (4.3453) loss 0.5479 (0.5479) grad_norm 0.2764 (0.2764) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:05:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [259/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9126) loss 0.6017 (0.5911) grad_norm 0.1980 (0.2378) loss_scale 524288.0000 (285503.3663) mem 30609MB [2024-03-06 05:06:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [259/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8955) loss 0.5836 (0.5905) grad_norm 0.2398 (0.2378) loss_scale 524288.0000 (404301.6915) mem 30609MB [2024-03-06 05:08:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [259/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.5971 (0.5901) grad_norm 0.2214 (0.2374) loss_scale 524288.0000 (444164.2525) mem 30609MB [2024-03-06 05:09:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [259/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8869) loss 0.6156 (0.5904) grad_norm 0.2283 (0.2378) loss_scale 524288.0000 (464145.2369) mem 30609MB [2024-03-06 05:09:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 259 training takes 0:05:56 [2024-03-06 05:09:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [260/800][0/402] eta 0:29:07 lr 0.000025 time 4.3476 (4.3476) loss 0.6130 (0.6130) grad_norm 0.2216 (0.2216) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:11:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [260/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9127) loss 0.6114 (0.5916) grad_norm 0.2052 (0.2393) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:12:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [260/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8956) loss 0.5937 (0.5900) grad_norm 0.2580 (0.2382) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:14:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [260/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.5757 (0.5904) grad_norm 0.2343 (0.2392) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:15:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [260/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8870) loss 0.6168 (0.5905) grad_norm 0.2607 (0.2385) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:15:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 260 training takes 0:05:56 [2024-03-06 05:15:52 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_260.pth saving...... [2024-03-06 05:15:53 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_260.pth saved !!! [2024-03-06 05:15:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [261/800][0/402] eta 0:29:45 lr 0.000025 time 4.4411 (4.4411) loss 0.5717 (0.5717) grad_norm 0.2633 (0.2633) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:17:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [261/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9137) loss 0.6040 (0.5885) grad_norm 0.2430 (0.2381) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:18:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [261/800][200/402] eta 0:03:01 lr 0.000025 time 0.8800 (0.8966) loss 0.5924 (0.5898) grad_norm 0.1999 (0.2364) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:20:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [261/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8906) loss 0.5942 (0.5900) grad_norm 0.2129 (0.2359) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:21:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [261/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8875) loss 0.6065 (0.5901) grad_norm 0.1937 (0.2354) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:21:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 261 training takes 0:05:56 [2024-03-06 05:21:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [262/800][0/402] eta 0:29:11 lr 0.000025 time 4.3560 (4.3560) loss 0.5928 (0.5928) grad_norm 0.2050 (0.2050) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:23:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [262/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9127) loss 0.5550 (0.5903) grad_norm 0.2355 (0.2314) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:24:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [262/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8956) loss 0.5650 (0.5908) grad_norm 0.2154 (0.2376) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:26:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [262/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8899) loss 0.5953 (0.5901) grad_norm 0.1880 (0.2371) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:27:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [262/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.5794 (0.5902) grad_norm 0.2034 (0.2371) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:27:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 262 training takes 0:05:56 [2024-03-06 05:27:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [263/800][0/402] eta 0:28:48 lr 0.000025 time 4.2995 (4.2995) loss 0.5566 (0.5566) grad_norm 0.2109 (0.2109) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:29:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [263/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9121) loss 0.5829 (0.5932) grad_norm 0.2420 (0.2398) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:30:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [263/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8953) loss 0.5726 (0.5932) grad_norm 0.2304 (0.2366) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:32:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [263/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8897) loss 0.5888 (0.5913) grad_norm 0.2136 (0.2357) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:33:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [263/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8868) loss 0.6072 (0.5914) grad_norm 0.2888 (0.2369) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:33:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 263 training takes 0:05:56 [2024-03-06 05:33:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [264/800][0/402] eta 0:28:59 lr 0.000025 time 4.3263 (4.3263) loss 0.5867 (0.5867) grad_norm 0.2235 (0.2235) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:35:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [264/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9123) loss 0.5900 (0.5917) grad_norm 0.2467 (0.2270) loss_scale 1048576.0000 (622916.4356) mem 30609MB [2024-03-06 05:36:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [264/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8953) loss 0.5891 (0.5914) grad_norm 0.2369 (inf) loss_scale 524288.0000 (618190.3284) mem 30609MB [2024-03-06 05:38:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [264/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8897) loss 0.5958 (0.5920) grad_norm 0.2434 (inf) loss_scale 524288.0000 (586993.5415) mem 30609MB [2024-03-06 05:39:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [264/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8868) loss 0.5750 (0.5915) grad_norm 0.2007 (inf) loss_scale 524288.0000 (571356.2494) mem 30609MB [2024-03-06 05:39:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 264 training takes 0:05:56 [2024-03-06 05:39:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [265/800][0/402] eta 0:29:10 lr 0.000025 time 4.3557 (4.3557) loss 0.6079 (0.6079) grad_norm 0.2199 (0.2199) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:41:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [265/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9128) loss 0.5927 (0.5891) grad_norm 0.2296 (0.2345) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:42:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [265/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8956) loss 0.5977 (0.5897) grad_norm 0.2317 (0.2304) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:44:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [265/800][300/402] eta 0:01:30 lr 0.000025 time 0.8778 (0.8899) loss 0.5726 (0.5900) grad_norm 0.2041 (0.2310) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 05:45:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [265/800][400/402] eta 0:00:01 lr 0.000025 time 0.8756 (0.8869) loss 0.5637 (0.5904) grad_norm 0.2189 (nan) loss_scale 262144.0000 (510559.7606) mem 30609MB [2024-03-06 05:45:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 265 training takes 0:05:56 [2024-03-06 05:45:37 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_265.pth saving...... [2024-03-06 05:45:38 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_265.pth saved !!! [2024-03-06 05:45:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [266/800][0/402] eta 0:29:44 lr 0.000025 time 4.4402 (4.4402) loss 0.5994 (0.5994) grad_norm 0.2409 (0.2409) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:47:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [266/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9137) loss 0.6001 (0.5905) grad_norm 0.2179 (0.2273) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:48:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [266/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8961) loss 0.5944 (0.5919) grad_norm 0.2284 (0.2342) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:50:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [266/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8902) loss 0.6036 (0.5905) grad_norm 0.2283 (0.2372) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:51:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [266/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8873) loss 0.5745 (0.5902) grad_norm 0.2464 (0.2357) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:51:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 266 training takes 0:05:56 [2024-03-06 05:51:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [267/800][0/402] eta 0:29:10 lr 0.000025 time 4.3552 (4.3552) loss 0.5710 (0.5710) grad_norm 0.2350 (0.2350) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:53:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [267/800][100/402] eta 0:04:36 lr 0.000025 time 0.8791 (0.9139) loss 0.6070 (0.5904) grad_norm 0.2112 (0.2262) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:54:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [267/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8968) loss 0.5725 (0.5903) grad_norm 0.2225 (0.2278) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:56:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [267/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8910) loss 0.6153 (0.5898) grad_norm 0.2016 (0.2321) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:57:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [267/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8880) loss 0.6080 (0.5899) grad_norm 0.2268 (0.2331) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:57:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 267 training takes 0:05:57 [2024-03-06 05:57:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [268/800][0/402] eta 0:29:13 lr 0.000025 time 4.3614 (4.3614) loss 0.5595 (0.5595) grad_norm 0.1974 (0.1974) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 05:59:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [268/800][100/402] eta 0:04:35 lr 0.000025 time 0.8792 (0.9138) loss 0.5940 (0.5898) grad_norm 0.2063 (0.2313) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:00:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [268/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8967) loss 0.5629 (0.5904) grad_norm 0.2368 (0.2290) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:02:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [268/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8909) loss 0.5854 (0.5896) grad_norm 0.2400 (0.2317) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:03:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [268/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8880) loss 0.5895 (0.5893) grad_norm 0.2165 (0.2300) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:03:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 268 training takes 0:05:57 [2024-03-06 06:03:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [269/800][0/402] eta 0:29:03 lr 0.000025 time 4.3382 (4.3382) loss 0.5974 (0.5974) grad_norm 0.2489 (0.2489) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:05:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [269/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9136) loss 0.5996 (0.5903) grad_norm 0.3166 (0.2357) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:06:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [269/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8961) loss 0.5736 (0.5880) grad_norm 0.2063 (0.2380) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:07:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [269/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8902) loss 0.5410 (0.5888) grad_norm 0.2513 (0.2364) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:09:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [269/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8872) loss 0.5881 (0.5892) grad_norm 0.2326 (0.2367) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:09:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 269 training takes 0:05:56 [2024-03-06 06:09:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [270/800][0/402] eta 0:28:52 lr 0.000025 time 4.3109 (4.3109) loss 0.5705 (0.5705) grad_norm 0.2106 (0.2106) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:10:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [270/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9123) loss 0.6056 (0.5900) grad_norm 0.2126 (0.2244) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:12:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [270/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8954) loss 0.5691 (0.5889) grad_norm 0.2226 (0.2292) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:13:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [270/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8897) loss 0.6144 (0.5897) grad_norm 0.2393 (0.2322) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:15:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [270/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8868) loss 0.5855 (0.5891) grad_norm 0.2230 (0.2334) loss_scale 524288.0000 (282409.4963) mem 30609MB [2024-03-06 06:15:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 270 training takes 0:05:56 [2024-03-06 06:15:23 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_270.pth saving...... [2024-03-06 06:15:25 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_270.pth saved !!! [2024-03-06 06:15:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [271/800][0/402] eta 0:28:47 lr 0.000025 time 4.2977 (4.2977) loss 0.5990 (0.5990) grad_norm 0.2066 (0.2066) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:16:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [271/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9122) loss 0.5619 (0.5864) grad_norm 0.2145 (0.2396) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:18:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [271/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8955) loss 0.5842 (0.5875) grad_norm 0.2131 (0.2379) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:19:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [271/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.6381 (0.5896) grad_norm 0.2345 (0.2353) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:21:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [271/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8869) loss 0.6223 (0.5898) grad_norm 0.2223 (0.2357) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:21:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 271 training takes 0:05:56 [2024-03-06 06:21:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [272/800][0/402] eta 0:29:00 lr 0.000025 time 4.3295 (4.3295) loss 0.5751 (0.5751) grad_norm 0.2287 (0.2287) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:22:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [272/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9124) loss 0.5885 (0.5904) grad_norm 0.2336 (0.2301) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:24:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [272/800][200/402] eta 0:03:00 lr 0.000025 time 0.8787 (0.8955) loss 0.5678 (0.5909) grad_norm 0.2410 (0.2324) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:25:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [272/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8898) loss 0.5871 (0.5899) grad_norm 0.3367 (0.2347) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:27:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [272/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8869) loss 0.5949 (0.5894) grad_norm 0.1967 (0.2329) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:27:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 272 training takes 0:05:56 [2024-03-06 06:27:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [273/800][0/402] eta 0:29:13 lr 0.000025 time 4.3619 (4.3619) loss 0.6062 (0.6062) grad_norm 0.2305 (0.2305) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:28:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [273/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9128) loss 0.6013 (0.5932) grad_norm 0.2186 (0.2340) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:30:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [273/800][200/402] eta 0:03:00 lr 0.000025 time 0.8781 (0.8956) loss 0.5938 (0.5906) grad_norm 0.2515 (0.2353) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:31:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [273/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8900) loss 0.6081 (0.5892) grad_norm 0.1906 (0.2364) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:33:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [273/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8870) loss 0.6027 (0.5893) grad_norm 0.2266 (0.2352) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:33:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 273 training takes 0:05:56 [2024-03-06 06:33:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [274/800][0/402] eta 0:29:05 lr 0.000025 time 4.3422 (4.3422) loss 0.5565 (0.5565) grad_norm 0.2270 (0.2270) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:34:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [274/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9126) loss 0.6102 (0.5906) grad_norm 0.2098 (0.2315) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:36:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [274/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8956) loss 0.5960 (0.5901) grad_norm 0.1944 (0.2341) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 06:37:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [274/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8898) loss 0.5793 (0.5900) grad_norm 0.2334 (nan) loss_scale 262144.0000 (520804.3588) mem 30609MB [2024-03-06 06:39:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [274/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8869) loss 0.5876 (0.5895) grad_norm 0.2095 (nan) loss_scale 262144.0000 (456300.5287) mem 30609MB [2024-03-06 06:39:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 274 training takes 0:05:56 [2024-03-06 06:39:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [275/800][0/402] eta 0:29:03 lr 0.000025 time 4.3367 (4.3367) loss 0.5740 (0.5740) grad_norm 0.2707 (0.2707) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:40:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [275/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9125) loss 0.5884 (0.5893) grad_norm 0.2750 (0.2468) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:42:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [275/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8955) loss 0.5473 (0.5892) grad_norm 0.2436 (0.2399) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:43:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [275/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8898) loss 0.5762 (0.5884) grad_norm 0.2472 (0.2361) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:45:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [275/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8868) loss 0.6207 (0.5894) grad_norm 0.2466 (0.2346) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:45:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 275 training takes 0:05:56 [2024-03-06 06:45:08 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_275.pth saving...... [2024-03-06 06:45:10 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_275.pth saved !!! [2024-03-06 06:45:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [276/800][0/402] eta 0:28:58 lr 0.000025 time 4.3240 (4.3240) loss 0.6133 (0.6133) grad_norm 0.2276 (0.2276) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:46:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [276/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9125) loss 0.5928 (0.5905) grad_norm 0.2163 (0.2334) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:48:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [276/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8955) loss 0.6027 (0.5908) grad_norm 0.1796 (0.2349) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:49:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [276/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.6042 (0.5901) grad_norm 0.2442 (0.2350) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:51:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [276/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8868) loss 0.5785 (0.5903) grad_norm 0.2079 (0.2349) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:51:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 276 training takes 0:05:56 [2024-03-06 06:51:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [277/800][0/402] eta 0:29:17 lr 0.000025 time 4.3708 (4.3708) loss 0.5956 (0.5956) grad_norm 0.2211 (0.2211) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:52:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [277/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9128) loss 0.5574 (0.5865) grad_norm 0.2055 (0.2314) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:54:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [277/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8956) loss 0.5967 (0.5889) grad_norm 0.2644 (0.2291) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:55:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [277/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8899) loss 0.5965 (0.5882) grad_norm 0.2636 (0.2348) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:57:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [277/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8869) loss 0.6038 (0.5880) grad_norm 0.2225 (0.2345) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:57:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 277 training takes 0:05:56 [2024-03-06 06:57:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [278/800][0/402] eta 0:29:20 lr 0.000025 time 4.3787 (4.3787) loss 0.6162 (0.6162) grad_norm 0.2724 (0.2724) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 06:58:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [278/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9128) loss 0.5775 (0.5917) grad_norm 0.2907 (0.2340) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:00:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [278/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8957) loss 0.6160 (0.5890) grad_norm 0.2162 (0.2313) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:01:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [278/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8899) loss 0.5393 (0.5886) grad_norm 0.2466 (0.2306) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:02:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [278/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.5744 (0.5888) grad_norm 0.2249 (0.2295) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:03:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 278 training takes 0:05:56 [2024-03-06 07:03:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [279/800][0/402] eta 0:29:02 lr 0.000025 time 4.3340 (4.3340) loss 0.5844 (0.5844) grad_norm 0.2364 (0.2364) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:04:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [279/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9124) loss 0.5770 (0.5895) grad_norm 0.2227 (0.2362) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:06:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [279/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8954) loss 0.6065 (0.5881) grad_norm 0.1934 (0.2297) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:07:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [279/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8897) loss 0.5846 (0.5882) grad_norm 0.2227 (0.2303) loss_scale 524288.0000 (274336.7442) mem 30609MB [2024-03-06 07:08:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [279/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8868) loss 0.6382 (0.5878) grad_norm 0.2164 (0.2313) loss_scale 524288.0000 (336668.7282) mem 30609MB [2024-03-06 07:08:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 279 training takes 0:05:56 [2024-03-06 07:09:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [280/800][0/402] eta 0:29:02 lr 0.000025 time 4.3352 (4.3352) loss 0.5950 (0.5950) grad_norm 0.2406 (0.2406) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:10:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [280/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9125) loss 0.5895 (0.5897) grad_norm 0.2358 (0.2324) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:11:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [280/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8955) loss 0.6007 (0.5882) grad_norm 0.1844 (0.2315) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:13:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [280/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8898) loss 0.5733 (0.5891) grad_norm 0.2782 (0.2331) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:14:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [280/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8869) loss 0.5797 (0.5896) grad_norm 0.2567 (0.2316) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:14:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 280 training takes 0:05:56 [2024-03-06 07:14:53 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_280.pth saving...... [2024-03-06 07:14:55 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_280.pth saved !!! [2024-03-06 07:14:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [281/800][0/402] eta 0:29:09 lr 0.000025 time 4.3528 (4.3528) loss 0.5922 (0.5922) grad_norm 0.2079 (0.2079) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:16:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [281/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9126) loss 0.6028 (0.5869) grad_norm 0.1940 (0.2370) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:17:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [281/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8955) loss 0.6006 (0.5893) grad_norm 0.2098 (0.2386) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:19:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [281/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.6021 (0.5881) grad_norm 0.2326 (0.2358) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:20:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [281/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8869) loss 0.6040 (0.5888) grad_norm 0.2511 (0.2364) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:20:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 281 training takes 0:05:56 [2024-03-06 07:20:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [282/800][0/402] eta 0:28:59 lr 0.000025 time 4.3278 (4.3278) loss 0.5962 (0.5962) grad_norm 0.1986 (0.1986) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:22:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [282/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9124) loss 0.5506 (0.5915) grad_norm 0.2909 (0.2326) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:23:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [282/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8955) loss 0.5775 (0.5899) grad_norm 0.2778 (0.2301) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:25:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [282/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8898) loss 0.6090 (0.5894) grad_norm 0.2181 (0.2286) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:26:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [282/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8870) loss 0.6174 (0.5888) grad_norm 0.2053 (0.2280) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:26:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 282 training takes 0:05:56 [2024-03-06 07:26:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [283/800][0/402] eta 0:28:43 lr 0.000025 time 4.2885 (4.2885) loss 0.6103 (0.6103) grad_norm 0.2376 (0.2376) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:28:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [283/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9120) loss 0.5911 (0.5880) grad_norm 0.2082 (0.2283) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:29:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [283/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8953) loss 0.5929 (0.5890) grad_norm 0.2156 (0.2315) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:31:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [283/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8896) loss 0.6121 (0.5899) grad_norm 0.2295 (0.2333) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:32:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [283/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8867) loss 0.6038 (0.5892) grad_norm 0.3112 (0.2343) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:32:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 283 training takes 0:05:56 [2024-03-06 07:32:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [284/800][0/402] eta 0:28:58 lr 0.000025 time 4.3239 (4.3239) loss 0.5518 (0.5518) grad_norm 0.2364 (0.2364) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:34:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [284/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9124) loss 0.5655 (0.5896) grad_norm 0.2468 (0.2387) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:35:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [284/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8954) loss 0.6103 (0.5890) grad_norm 0.2155 (0.2383) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 07:37:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [284/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8897) loss 0.6183 (0.5885) grad_norm 0.2052 (0.2361) loss_scale 1048576.0000 (566091.6944) mem 30609MB [2024-03-06 07:38:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [284/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8868) loss 0.5704 (0.5887) grad_norm 0.2249 (nan) loss_scale 262144.0000 (565472.7182) mem 30609MB [2024-03-06 07:38:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 284 training takes 0:05:56 [2024-03-06 07:38:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [285/800][0/402] eta 0:29:07 lr 0.000025 time 4.3470 (4.3470) loss 0.5417 (0.5417) grad_norm 0.2474 (0.2474) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:40:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [285/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9127) loss 0.6031 (0.5891) grad_norm 0.2152 (0.2310) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:41:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [285/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8956) loss 0.6101 (0.5895) grad_norm 0.2658 (0.2321) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:43:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [285/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8898) loss 0.5978 (0.5895) grad_norm 0.1982 (0.2337) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:44:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [285/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.5708 (0.5892) grad_norm 0.2064 (0.2311) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:44:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 285 training takes 0:05:56 [2024-03-06 07:44:38 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_285.pth saving...... [2024-03-06 07:44:40 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_285.pth saved !!! [2024-03-06 07:44:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [286/800][0/402] eta 0:29:09 lr 0.000025 time 4.3525 (4.3525) loss 0.6037 (0.6037) grad_norm 0.2356 (0.2356) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:46:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [286/800][100/402] eta 0:04:35 lr 0.000025 time 0.8793 (0.9133) loss 0.5839 (0.5898) grad_norm 0.2229 (0.2260) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:47:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [286/800][200/402] eta 0:03:01 lr 0.000025 time 0.8798 (0.8963) loss 0.5472 (0.5877) grad_norm 0.2447 (0.2301) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:49:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [286/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8905) loss 0.5805 (0.5892) grad_norm 0.2526 (0.2295) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:50:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [286/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8874) loss 0.6529 (0.5883) grad_norm 0.2421 (0.2337) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:50:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 286 training takes 0:05:56 [2024-03-06 07:50:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [287/800][0/402] eta 0:28:55 lr 0.000025 time 4.3175 (4.3175) loss 0.5742 (0.5742) grad_norm 0.2717 (0.2717) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:52:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [287/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9125) loss 0.6238 (0.5861) grad_norm 0.2426 (0.2271) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:53:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [287/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8955) loss 0.5828 (0.5871) grad_norm 0.1990 (0.2320) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:55:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [287/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8898) loss 0.5719 (0.5881) grad_norm 0.2342 (0.2312) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:56:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [287/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8869) loss 0.6033 (0.5880) grad_norm 0.2954 (0.2306) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:56:34 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 287 training takes 0:05:56 [2024-03-06 07:56:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [288/800][0/402] eta 0:28:59 lr 0.000025 time 4.3276 (4.3276) loss 0.5965 (0.5965) grad_norm 0.2377 (0.2377) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:58:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [288/800][100/402] eta 0:04:35 lr 0.000025 time 0.8793 (0.9134) loss 0.6248 (0.5891) grad_norm 0.1963 (0.2303) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 07:59:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [288/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8965) loss 0.6080 (0.5882) grad_norm 0.2137 (0.2316) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:01:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [288/800][300/402] eta 0:01:30 lr 0.000025 time 0.8795 (0.8908) loss 0.5917 (0.5881) grad_norm 0.2380 (0.2334) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:02:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [288/800][400/402] eta 0:00:01 lr 0.000025 time 0.8781 (0.8879) loss 0.5935 (0.5877) grad_norm 0.2181 (0.2317) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:02:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 288 training takes 0:05:57 [2024-03-06 08:02:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [289/800][0/402] eta 0:29:12 lr 0.000025 time 4.3592 (4.3592) loss 0.6107 (0.6107) grad_norm 0.2328 (0.2328) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:04:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [289/800][100/402] eta 0:04:35 lr 0.000025 time 0.8794 (0.9138) loss 0.5690 (0.5886) grad_norm 0.2130 (0.2273) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:05:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [289/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8967) loss 0.6320 (0.5889) grad_norm 0.2715 (0.2272) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:06:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [289/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8908) loss 0.5828 (0.5892) grad_norm 0.3025 (0.2282) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:08:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [289/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8876) loss 0.5747 (0.5887) grad_norm 0.2635 (0.2297) loss_scale 524288.0000 (275872.2394) mem 30609MB [2024-03-06 08:08:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 289 training takes 0:05:56 [2024-03-06 08:08:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [290/800][0/402] eta 0:29:10 lr 0.000025 time 4.3538 (4.3538) loss 0.5697 (0.5697) grad_norm 0.2215 (0.2215) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:10:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [290/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9127) loss 0.5623 (0.5846) grad_norm 0.2431 (0.2299) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:11:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [290/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8956) loss 0.5984 (0.5869) grad_norm 0.2113 (0.2348) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:12:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [290/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.5967 (0.5872) grad_norm 0.2863 (0.2335) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:14:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [290/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8869) loss 0.5788 (0.5868) grad_norm 0.1965 (0.2343) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:14:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 290 training takes 0:05:56 [2024-03-06 08:14:24 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_290.pth saving...... [2024-03-06 08:14:26 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_290.pth saved !!! [2024-03-06 08:14:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [291/800][0/402] eta 0:29:17 lr 0.000025 time 4.3717 (4.3717) loss 0.5816 (0.5816) grad_norm 0.2275 (0.2275) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:15:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [291/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9128) loss 0.6018 (0.5876) grad_norm 0.2870 (0.2296) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:17:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [291/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8957) loss 0.6005 (0.5879) grad_norm 0.2448 (0.2283) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:18:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [291/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8900) loss 0.6117 (0.5885) grad_norm 0.2258 (0.2271) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:20:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [291/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8870) loss 0.5543 (0.5884) grad_norm 0.2643 (inf) loss_scale 262144.0000 (511213.4863) mem 30609MB [2024-03-06 08:20:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 291 training takes 0:05:56 [2024-03-06 08:20:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [292/800][0/402] eta 0:29:12 lr 0.000025 time 4.3601 (4.3601) loss 0.5951 (0.5951) grad_norm 0.2000 (0.2000) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:21:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [292/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9128) loss 0.5881 (0.5891) grad_norm 0.2720 (0.2342) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:23:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [292/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8956) loss 0.5709 (0.5888) grad_norm 0.2438 (0.2304) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:24:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [292/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8899) loss 0.6233 (0.5884) grad_norm 0.2365 (0.2310) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:26:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [292/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8869) loss 0.5935 (0.5889) grad_norm 0.2098 (0.2299) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:26:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 292 training takes 0:05:56 [2024-03-06 08:26:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [293/800][0/402] eta 0:28:56 lr 0.000025 time 4.3206 (4.3206) loss 0.5922 (0.5922) grad_norm 0.2058 (0.2058) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:27:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [293/800][100/402] eta 0:04:35 lr 0.000025 time 0.8777 (0.9122) loss 0.5995 (0.5876) grad_norm 0.2009 (0.2302) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:29:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [293/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8953) loss 0.5599 (0.5871) grad_norm 0.1967 (0.2268) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:30:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [293/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8897) loss 0.5640 (0.5882) grad_norm 0.2314 (0.2265) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:32:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [293/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8868) loss 0.5939 (0.5884) grad_norm 0.2011 (0.2274) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:32:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 293 training takes 0:05:56 [2024-03-06 08:32:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [294/800][0/402] eta 0:29:09 lr 0.000025 time 4.3530 (4.3530) loss 0.5814 (0.5814) grad_norm 0.2473 (0.2473) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:33:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [294/800][100/402] eta 0:04:35 lr 0.000025 time 0.8779 (0.9126) loss 0.6172 (0.5906) grad_norm 0.2208 (0.2325) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:35:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [294/800][200/402] eta 0:03:00 lr 0.000025 time 0.8780 (0.8955) loss 0.5917 (0.5883) grad_norm 0.2192 (0.2388) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:36:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [294/800][300/402] eta 0:01:30 lr 0.000025 time 0.8774 (0.8898) loss 0.6233 (0.5885) grad_norm 0.2305 (0.2356) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:38:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [294/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.5695 (0.5885) grad_norm 0.2098 (0.2340) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:38:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 294 training takes 0:05:56 [2024-03-06 08:38:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [295/800][0/402] eta 0:29:25 lr 0.000025 time 4.3929 (4.3929) loss 0.5842 (0.5842) grad_norm 0.2839 (0.2839) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:39:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [295/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9130) loss 0.6167 (0.5908) grad_norm 0.2258 (0.2322) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:41:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [295/800][200/402] eta 0:03:00 lr 0.000025 time 0.8794 (0.8957) loss 0.5901 (0.5899) grad_norm 0.2233 (0.2299) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:42:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [295/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8899) loss 0.6147 (0.5883) grad_norm 0.2108 (0.2298) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:44:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [295/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8869) loss 0.5593 (0.5883) grad_norm 0.2163 (0.2285) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:44:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 295 training takes 0:05:56 [2024-03-06 08:44:10 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_295.pth saving...... [2024-03-06 08:44:11 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_295.pth saved !!! [2024-03-06 08:44:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [296/800][0/402] eta 0:28:16 lr 0.000025 time 4.2205 (4.2205) loss 0.5816 (0.5816) grad_norm 0.2350 (0.2350) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:45:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [296/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9113) loss 0.6016 (0.5885) grad_norm 0.2494 (0.2405) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:47:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [296/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8949) loss 0.5890 (0.5894) grad_norm 0.2722 (0.2319) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:48:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [296/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8894) loss 0.5958 (0.5885) grad_norm 0.2155 (0.2291) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 08:50:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [296/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8866) loss 0.5773 (0.5884) grad_norm 0.2247 (0.2294) loss_scale 524288.0000 (281755.7706) mem 30609MB [2024-03-06 08:50:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 296 training takes 0:05:56 [2024-03-06 08:50:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [297/800][0/402] eta 0:29:20 lr 0.000025 time 4.3793 (4.3793) loss 0.6127 (0.6127) grad_norm 0.2288 (0.2288) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:51:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [297/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9129) loss 0.5795 (0.5898) grad_norm 0.3128 (0.2357) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:53:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [297/800][200/402] eta 0:03:00 lr 0.000025 time 0.8777 (0.8957) loss 0.5988 (0.5876) grad_norm 0.2399 (0.2325) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:54:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [297/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8899) loss 0.6028 (0.5875) grad_norm 0.2353 (0.2307) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:56:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [297/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.5743 (0.5875) grad_norm 0.2201 (0.2296) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:56:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 297 training takes 0:05:56 [2024-03-06 08:56:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [298/800][0/402] eta 0:29:34 lr 0.000025 time 4.4143 (4.4143) loss 0.6049 (0.6049) grad_norm 0.2389 (0.2389) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:57:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [298/800][100/402] eta 0:04:35 lr 0.000025 time 0.8782 (0.9132) loss 0.6014 (0.5879) grad_norm 0.2049 (0.2327) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 08:59:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [298/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8962) loss 0.5625 (0.5882) grad_norm 0.2155 (0.2290) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:00:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [298/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8902) loss 0.5873 (0.5870) grad_norm 0.2345 (nan) loss_scale 262144.0000 (506869.7940) mem 30609MB [2024-03-06 09:02:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [298/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8872) loss 0.5847 (0.5877) grad_norm 0.2404 (nan) loss_scale 262144.0000 (445840.9177) mem 30609MB [2024-03-06 09:02:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 298 training takes 0:05:56 [2024-03-06 09:02:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [299/800][0/402] eta 0:29:15 lr 0.000025 time 4.3670 (4.3670) loss 0.5849 (0.5849) grad_norm 0.2112 (0.2112) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:03:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [299/800][100/402] eta 0:04:35 lr 0.000025 time 0.8780 (0.9129) loss 0.6097 (0.5843) grad_norm 0.2000 (0.2244) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:05:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [299/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8957) loss 0.5800 (0.5857) grad_norm 0.2149 (0.2257) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:06:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [299/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8899) loss 0.5739 (0.5869) grad_norm 0.2618 (0.2273) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:07:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [299/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8870) loss 0.5901 (0.5867) grad_norm 0.2136 (0.2275) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:07:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 299 training takes 0:05:56 [2024-03-06 09:08:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [300/800][0/402] eta 0:29:24 lr 0.000025 time 4.3898 (4.3898) loss 0.5592 (0.5592) grad_norm 0.2116 (0.2116) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:09:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [300/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9130) loss 0.6001 (0.5854) grad_norm 0.2143 (0.2325) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:10:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [300/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8957) loss 0.6034 (0.5864) grad_norm 0.2089 (0.2279) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:12:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [300/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8903) loss 0.5927 (0.5867) grad_norm 0.2025 (0.2276) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:13:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [300/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8873) loss 0.6038 (0.5873) grad_norm 0.2542 (0.2271) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:13:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 300 training takes 0:05:56 [2024-03-06 09:13:55 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_300.pth saving...... [2024-03-06 09:13:57 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_300.pth saved !!! [2024-03-06 09:14:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [301/800][0/402] eta 0:29:42 lr 0.000025 time 4.4345 (4.4345) loss 0.5541 (0.5541) grad_norm 0.2282 (0.2282) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:15:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [301/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9139) loss 0.5676 (0.5862) grad_norm 0.2162 (0.2203) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:16:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [301/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8964) loss 0.5553 (0.5874) grad_norm 0.2158 (0.2231) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:18:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [301/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8905) loss 0.5752 (0.5865) grad_norm 0.1802 (0.2256) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:19:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [301/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8875) loss 0.5835 (0.5871) grad_norm 0.2398 (0.2256) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:19:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 301 training takes 0:05:56 [2024-03-06 09:19:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [302/800][0/402] eta 0:29:21 lr 0.000025 time 4.3814 (4.3814) loss 0.6143 (0.6143) grad_norm 0.2206 (0.2206) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:21:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [302/800][100/402] eta 0:04:35 lr 0.000025 time 0.8790 (0.9128) loss 0.5659 (0.5861) grad_norm 0.2753 (0.2306) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:22:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [302/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8957) loss 0.5808 (0.5887) grad_norm 0.2132 (0.2317) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:24:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [302/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8902) loss 0.5643 (0.5883) grad_norm 0.2297 (0.2321) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:25:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [302/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8872) loss 0.5602 (0.5881) grad_norm 0.2158 (0.2318) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:25:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 302 training takes 0:05:56 [2024-03-06 09:25:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [303/800][0/402] eta 0:29:00 lr 0.000025 time 4.3290 (4.3290) loss 0.5929 (0.5929) grad_norm 0.2146 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:27:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [303/800][100/402] eta 0:04:35 lr 0.000025 time 0.8789 (0.9124) loss 0.6039 (0.5858) grad_norm 0.2452 (0.2251) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:28:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [303/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8955) loss 0.5619 (0.5862) grad_norm 0.2199 (0.2251) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:30:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [303/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8899) loss 0.5580 (0.5876) grad_norm 0.2134 (0.2267) loss_scale 524288.0000 (288271.3090) mem 30609MB [2024-03-06 09:31:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [303/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8869) loss 0.5842 (0.5873) grad_norm 0.1987 (0.2293) loss_scale 524288.0000 (347128.3392) mem 30609MB [2024-03-06 09:31:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 303 training takes 0:05:56 [2024-03-06 09:31:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [304/800][0/402] eta 0:29:13 lr 0.000025 time 4.3625 (4.3625) loss 0.5961 (0.5961) grad_norm 0.2324 (0.2324) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:33:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [304/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9128) loss 0.5638 (0.5881) grad_norm 0.2509 (0.2348) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:34:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [304/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8957) loss 0.6300 (0.5888) grad_norm 0.2667 (0.2313) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:36:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [304/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8899) loss 0.6105 (0.5891) grad_norm 0.2227 (0.2291) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:37:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [304/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8870) loss 0.5837 (0.5890) grad_norm 0.2055 (0.2304) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:37:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 304 training takes 0:05:56 [2024-03-06 09:37:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [305/800][0/402] eta 0:29:15 lr 0.000025 time 4.3670 (4.3670) loss 0.6281 (0.6281) grad_norm 0.2363 (0.2363) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:39:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [305/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9139) loss 0.5772 (0.5858) grad_norm 0.2269 (0.2307) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:40:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [305/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8962) loss 0.5994 (0.5871) grad_norm 0.2266 (0.2271) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:42:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [305/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8903) loss 0.6016 (0.5876) grad_norm 0.2448 (0.2292) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:43:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [305/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8873) loss 0.5568 (0.5883) grad_norm 0.2281 (0.2286) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:43:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 305 training takes 0:05:56 [2024-03-06 09:43:41 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_305.pth saving...... [2024-03-06 09:43:42 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_305.pth saved !!! [2024-03-06 09:43:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [306/800][0/402] eta 0:30:52 lr 0.000025 time 4.6089 (4.6089) loss 0.5727 (0.5727) grad_norm 0.2049 (0.2049) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:45:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [306/800][100/402] eta 0:04:36 lr 0.000025 time 0.8798 (0.9159) loss 0.5582 (0.5888) grad_norm 0.2070 (0.2273) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:46:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [306/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8976) loss 0.5444 (0.5878) grad_norm 0.2669 (0.2281) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:48:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [306/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8913) loss 0.6036 (0.5881) grad_norm 0.2268 (0.2276) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:49:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [306/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8880) loss 0.5758 (0.5882) grad_norm 0.2275 (0.2281) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:49:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 306 training takes 0:05:57 [2024-03-06 09:49:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [307/800][0/402] eta 0:29:36 lr 0.000025 time 4.4187 (4.4187) loss 0.6091 (0.6091) grad_norm 0.2422 (0.2422) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 09:51:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [307/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9138) loss 0.5865 (0.5902) grad_norm 0.2128 (inf) loss_scale 262144.0000 (420468.5941) mem 30609MB [2024-03-06 09:52:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [307/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8961) loss 0.5732 (0.5882) grad_norm 0.2243 (inf) loss_scale 262144.0000 (341700.1393) mem 30609MB [2024-03-06 09:54:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [307/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8902) loss 0.5884 (0.5878) grad_norm 0.2270 (inf) loss_scale 262144.0000 (315269.5282) mem 30609MB [2024-03-06 09:55:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [307/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8872) loss 0.6111 (0.5874) grad_norm 0.2518 (inf) loss_scale 262144.0000 (302021.2668) mem 30609MB [2024-03-06 09:55:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 307 training takes 0:05:56 [2024-03-06 09:55:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [308/800][0/402] eta 0:29:30 lr 0.000025 time 4.4050 (4.4050) loss 0.5361 (0.5361) grad_norm 0.3005 (0.3005) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:57:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [308/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9132) loss 0.5971 (0.5846) grad_norm 0.2053 (0.2303) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 09:58:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [308/800][200/402] eta 0:03:00 lr 0.000025 time 0.8777 (0.8959) loss 0.5743 (0.5863) grad_norm 0.2611 (0.2311) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:00:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [308/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8901) loss 0.5878 (0.5870) grad_norm 0.2335 (0.2293) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:01:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [308/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8871) loss 0.5780 (0.5864) grad_norm 0.2283 (0.2288) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:01:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 308 training takes 0:05:56 [2024-03-06 10:01:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [309/800][0/402] eta 0:29:36 lr 0.000025 time 4.4188 (4.4188) loss 0.5513 (0.5513) grad_norm 0.2232 (0.2232) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:03:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [309/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9133) loss 0.5885 (0.5882) grad_norm 0.2368 (0.2194) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:04:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [309/800][200/402] eta 0:03:01 lr 0.000025 time 0.8775 (0.8963) loss 0.5917 (0.5883) grad_norm 0.2261 (0.2233) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:06:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [309/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8904) loss 0.6033 (0.5880) grad_norm 0.2296 (0.2255) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:07:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [309/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8873) loss 0.5732 (0.5878) grad_norm 0.2139 (0.2251) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:07:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 309 training takes 0:05:56 [2024-03-06 10:07:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [310/800][0/402] eta 0:29:14 lr 0.000025 time 4.3640 (4.3640) loss 0.5955 (0.5955) grad_norm 0.2174 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:09:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [310/800][100/402] eta 0:04:35 lr 0.000025 time 0.8778 (0.9127) loss 0.5775 (0.5867) grad_norm 0.2103 (0.2257) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:10:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [310/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8956) loss 0.5238 (0.5858) grad_norm 0.2899 (0.2280) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:11:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [310/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8899) loss 0.5930 (0.5861) grad_norm 0.2241 (0.2277) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:13:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [310/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.5717 (0.5864) grad_norm 0.2332 (0.2273) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:13:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 310 training takes 0:05:56 [2024-03-06 10:13:27 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_310.pth saving...... [2024-03-06 10:13:29 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_310.pth saved !!! [2024-03-06 10:13:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [311/800][0/402] eta 0:28:33 lr 0.000025 time 4.2621 (4.2621) loss 0.5248 (0.5248) grad_norm 0.2405 (0.2405) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:15:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [311/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9122) loss 0.5644 (0.5826) grad_norm 0.2182 (0.2296) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:16:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [311/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8958) loss 0.5882 (0.5860) grad_norm 0.2139 (0.2290) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:17:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [311/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8902) loss 0.6408 (0.5868) grad_norm 0.2293 (0.2284) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:19:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [311/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8873) loss 0.5689 (0.5867) grad_norm 0.2340 (0.2276) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:19:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 311 training takes 0:05:56 [2024-03-06 10:19:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [312/800][0/402] eta 0:29:52 lr 0.000025 time 4.4586 (4.4586) loss 0.6036 (0.6036) grad_norm 0.2282 (0.2282) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:20:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [312/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9138) loss 0.5722 (0.5868) grad_norm 0.2110 (0.2288) loss_scale 524288.0000 (391918.2574) mem 30609MB [2024-03-06 10:22:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [312/800][200/402] eta 0:03:01 lr 0.000025 time 0.8797 (0.8965) loss 0.6004 (0.5866) grad_norm 0.2067 (0.2270) loss_scale 524288.0000 (457773.8507) mem 30609MB [2024-03-06 10:23:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [312/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8905) loss 0.6078 (0.5866) grad_norm 0.2177 (0.2257) loss_scale 524288.0000 (479871.5748) mem 30609MB [2024-03-06 10:25:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [312/800][400/402] eta 0:00:01 lr 0.000025 time 0.8779 (0.8875) loss 0.5721 (0.5868) grad_norm 0.2282 (0.2252) loss_scale 524288.0000 (490947.9900) mem 30609MB [2024-03-06 10:25:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 312 training takes 0:05:57 [2024-03-06 10:25:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [313/800][0/402] eta 0:30:49 lr 0.000025 time 4.5996 (4.5996) loss 0.5751 (0.5751) grad_norm 0.2000 (0.2000) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 10:26:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [313/800][100/402] eta 0:04:36 lr 0.000025 time 0.8801 (0.9152) loss 0.5921 (0.5878) grad_norm 0.2813 (0.2354) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 10:28:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [313/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8969) loss 0.5740 (0.5883) grad_norm 0.2659 (0.2316) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 10:29:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [313/800][300/402] eta 0:01:30 lr 0.000025 time 0.8817 (0.8910) loss 0.5455 (0.5886) grad_norm 0.2020 (0.2278) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 10:31:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [313/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8879) loss 0.5808 (0.5883) grad_norm 0.2187 (0.2287) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 10:31:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 313 training takes 0:05:57 [2024-03-06 10:31:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [314/800][0/402] eta 0:30:52 lr 0.000025 time 4.6075 (4.6075) loss 0.6122 (0.6122) grad_norm 0.2400 (0.2400) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 10:32:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [314/800][100/402] eta 0:04:36 lr 0.000025 time 0.8789 (0.9155) loss 0.5807 (0.5873) grad_norm 0.2112 (0.2298) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 10:34:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [314/800][200/402] eta 0:03:01 lr 0.000025 time 0.8794 (0.8970) loss 0.5640 (0.5863) grad_norm 0.2358 (nan) loss_scale 262144.0000 (470815.8408) mem 30609MB [2024-03-06 10:35:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [314/800][300/402] eta 0:01:30 lr 0.000025 time 0.8799 (0.8910) loss 0.5735 (0.5862) grad_norm 0.1892 (nan) loss_scale 262144.0000 (401489.6478) mem 30609MB [2024-03-06 10:37:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [314/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8879) loss 0.5893 (0.5870) grad_norm 0.2469 (nan) loss_scale 262144.0000 (366740.1097) mem 30609MB [2024-03-06 10:37:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 314 training takes 0:05:57 [2024-03-06 10:37:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [315/800][0/402] eta 0:29:24 lr 0.000025 time 4.3888 (4.3888) loss 0.5994 (0.5994) grad_norm 0.2796 (0.2796) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:38:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [315/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9133) loss 0.5994 (0.5862) grad_norm 0.2257 (0.2252) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:40:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [315/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8961) loss 0.5331 (0.5846) grad_norm 0.2273 (0.2260) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:41:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [315/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8906) loss 0.5732 (0.5859) grad_norm 0.2428 (0.2249) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:43:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [315/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8877) loss 0.6009 (0.5865) grad_norm 0.2422 (0.2246) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:43:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 315 training takes 0:05:57 [2024-03-06 10:43:14 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_315.pth saving...... [2024-03-06 10:43:16 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_315.pth saved !!! [2024-03-06 10:43:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [316/800][0/402] eta 0:28:39 lr 0.000025 time 4.2770 (4.2770) loss 0.5835 (0.5835) grad_norm 0.2049 (0.2049) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:44:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [316/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9124) loss 0.6027 (0.5844) grad_norm 0.2404 (0.2265) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:46:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [316/800][200/402] eta 0:03:00 lr 0.000025 time 0.8791 (0.8957) loss 0.6161 (0.5870) grad_norm 0.2315 (0.2298) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:47:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [316/800][300/402] eta 0:01:30 lr 0.000025 time 0.8792 (0.8902) loss 0.5830 (0.5866) grad_norm 0.2160 (0.2286) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:49:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [316/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8874) loss 0.5956 (0.5871) grad_norm 0.2148 (0.2287) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:49:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 316 training takes 0:05:56 [2024-03-06 10:49:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [317/800][0/402] eta 0:32:42 lr 0.000025 time 4.8818 (4.8818) loss 0.5715 (0.5715) grad_norm 0.2132 (0.2132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:50:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [317/800][100/402] eta 0:04:37 lr 0.000025 time 0.8784 (0.9182) loss 0.5622 (0.5865) grad_norm 0.1772 (0.2267) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:52:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [317/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8985) loss 0.6027 (0.5866) grad_norm 0.2006 (0.2216) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:53:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [317/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8919) loss 0.5636 (0.5866) grad_norm 0.2342 (0.2237) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:55:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [317/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8886) loss 0.5584 (0.5872) grad_norm 0.2221 (0.2231) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:55:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 317 training takes 0:05:57 [2024-03-06 10:55:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [318/800][0/402] eta 0:32:18 lr 0.000025 time 4.8225 (4.8225) loss 0.6036 (0.6036) grad_norm 0.2378 (0.2378) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:56:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [318/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9181) loss 0.5508 (0.5839) grad_norm 0.2630 (0.2246) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:58:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [318/800][200/402] eta 0:03:01 lr 0.000025 time 0.8805 (0.8986) loss 0.6083 (0.5850) grad_norm 0.2418 (0.2256) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 10:59:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [318/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8920) loss 0.5845 (0.5856) grad_norm 0.2100 (0.2258) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:01:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [318/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8886) loss 0.5557 (0.5864) grad_norm 0.2434 (0.2262) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:01:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 318 training takes 0:05:57 [2024-03-06 11:01:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [319/800][0/402] eta 0:32:30 lr 0.000025 time 4.8521 (4.8521) loss 0.6114 (0.6114) grad_norm 0.1997 (0.1997) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:02:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [319/800][100/402] eta 0:04:37 lr 0.000025 time 0.8790 (0.9182) loss 0.5818 (0.5875) grad_norm 0.2067 (0.2381) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:04:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [319/800][200/402] eta 0:03:01 lr 0.000025 time 0.8772 (0.8987) loss 0.5797 (0.5863) grad_norm 0.2155 (0.2298) loss_scale 524288.0000 (328658.1493) mem 30609MB [2024-03-06 11:05:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [319/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8920) loss 0.5802 (0.5857) grad_norm 0.2022 (0.2292) loss_scale 524288.0000 (393651.4551) mem 30609MB [2024-03-06 11:07:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [319/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8887) loss 0.5734 (0.5865) grad_norm 0.2179 (0.2280) loss_scale 524288.0000 (426229.1471) mem 30609MB [2024-03-06 11:07:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 319 training takes 0:05:57 [2024-03-06 11:07:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [320/800][0/402] eta 0:32:44 lr 0.000025 time 4.8863 (4.8863) loss 0.5753 (0.5753) grad_norm 0.2000 (0.2000) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:08:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [320/800][100/402] eta 0:04:37 lr 0.000025 time 0.8795 (0.9184) loss 0.5872 (0.5855) grad_norm 0.1951 (0.2203) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:10:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [320/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8989) loss 0.5998 (0.5854) grad_norm 0.2046 (0.2245) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:11:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [320/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8922) loss 0.5926 (0.5859) grad_norm 0.2369 (0.2274) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:13:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [320/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8888) loss 0.6097 (0.5868) grad_norm 0.2349 (nan) loss_scale 262144.0000 (478527.2020) mem 30609MB [2024-03-06 11:13:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 320 training takes 0:05:57 [2024-03-06 11:13:03 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_320.pth saving...... [2024-03-06 11:13:04 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_320.pth saved !!! [2024-03-06 11:13:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [321/800][0/402] eta 0:31:32 lr 0.000025 time 4.7075 (4.7075) loss 0.5968 (0.5968) grad_norm 0.1913 (0.1913) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:14:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [321/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9165) loss 0.6195 (0.5855) grad_norm 0.2090 (0.2227) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:16:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [321/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8978) loss 0.5746 (0.5860) grad_norm 0.2002 (0.2261) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:17:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [321/800][300/402] eta 0:01:30 lr 0.000025 time 0.8776 (0.8914) loss 0.6181 (0.5859) grad_norm 0.2094 (0.2234) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:19:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [321/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8881) loss 0.5978 (0.5868) grad_norm 0.2516 (0.2231) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:19:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 321 training takes 0:05:57 [2024-03-06 11:19:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [322/800][0/402] eta 0:35:23 lr 0.000025 time 5.2822 (5.2822) loss 0.5780 (0.5780) grad_norm 0.2195 (0.2195) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:20:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [322/800][100/402] eta 0:04:38 lr 0.000025 time 0.8784 (0.9229) loss 0.5949 (0.5833) grad_norm 0.2490 (0.2313) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:22:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [322/800][200/402] eta 0:03:02 lr 0.000025 time 0.8780 (0.9010) loss 0.5931 (0.5845) grad_norm 0.2002 (0.2288) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:23:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [322/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8934) loss 0.5825 (0.5855) grad_norm 0.2055 (0.2306) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:24:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [322/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8896) loss 0.5807 (0.5866) grad_norm 0.2219 (0.2277) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:25:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 322 training takes 0:05:57 [2024-03-06 11:25:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [323/800][0/402] eta 0:32:17 lr 0.000025 time 4.8184 (4.8184) loss 0.5601 (0.5601) grad_norm 0.2183 (0.2183) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:26:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [323/800][100/402] eta 0:04:37 lr 0.000025 time 0.8786 (0.9175) loss 0.6073 (0.5899) grad_norm 0.2220 (0.2238) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:28:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [323/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8980) loss 0.6177 (0.5879) grad_norm 0.2043 (0.2225) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:29:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [323/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8915) loss 0.5744 (0.5879) grad_norm 0.2212 (0.2234) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:30:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [323/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8881) loss 0.5990 (0.5870) grad_norm 0.1949 (0.2242) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:30:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 323 training takes 0:05:57 [2024-03-06 11:31:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [324/800][0/402] eta 0:34:29 lr 0.000025 time 5.1474 (5.1474) loss 0.5759 (0.5759) grad_norm 0.1886 (0.1886) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:32:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [324/800][100/402] eta 0:04:38 lr 0.000025 time 0.8784 (0.9207) loss 0.5883 (0.5854) grad_norm 0.2156 (0.2233) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:33:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [324/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8996) loss 0.5885 (0.5867) grad_norm 0.2194 (0.2258) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:35:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [324/800][300/402] eta 0:01:31 lr 0.000025 time 0.8782 (0.8932) loss 0.6023 (0.5861) grad_norm 0.1788 (0.2223) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:36:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [324/800][400/402] eta 0:00:01 lr 0.000025 time 0.8805 (0.8894) loss 0.5941 (0.5868) grad_norm 0.2732 (0.2241) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:36:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 324 training takes 0:05:57 [2024-03-06 11:36:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [325/800][0/402] eta 0:31:21 lr 0.000025 time 4.6800 (4.6800) loss 0.5793 (0.5793) grad_norm 0.2953 (0.2953) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:38:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [325/800][100/402] eta 0:04:36 lr 0.000025 time 0.8807 (0.9165) loss 0.5922 (0.5845) grad_norm 0.1982 (0.2232) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:39:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [325/800][200/402] eta 0:03:01 lr 0.000025 time 0.8775 (0.8977) loss 0.5743 (0.5850) grad_norm 0.1804 (0.2247) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:41:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [325/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8912) loss 0.5832 (0.5852) grad_norm 0.2053 (0.2246) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 11:42:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [325/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8880) loss 0.6056 (0.5862) grad_norm 0.4919 (0.2252) loss_scale 524288.0000 (314442.0549) mem 30609MB [2024-03-06 11:42:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 325 training takes 0:05:57 [2024-03-06 11:42:52 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_325.pth saving...... [2024-03-06 11:42:54 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_325.pth saved !!! [2024-03-06 11:42:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [326/800][0/402] eta 0:33:45 lr 0.000025 time 5.0392 (5.0392) loss 0.6090 (0.6090) grad_norm 0.2565 (0.2565) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:44:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [326/800][100/402] eta 0:04:37 lr 0.000025 time 0.8789 (0.9195) loss 0.5820 (0.5862) grad_norm 0.2158 (0.2265) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:45:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [326/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8991) loss 0.5710 (0.5878) grad_norm 0.2385 (0.2267) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:47:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [326/800][300/402] eta 0:01:31 lr 0.000025 time 0.8783 (0.8924) loss 0.5704 (0.5875) grad_norm 0.2006 (0.2272) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:48:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [326/800][400/402] eta 0:00:01 lr 0.000025 time 0.8794 (0.8891) loss 0.5854 (0.5869) grad_norm 0.2397 (0.2271) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:48:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 326 training takes 0:05:57 [2024-03-06 11:48:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [327/800][0/402] eta 0:32:15 lr 0.000025 time 4.8139 (4.8139) loss 0.6028 (0.6028) grad_norm 0.2705 (0.2705) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:50:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [327/800][100/402] eta 0:04:37 lr 0.000025 time 0.8781 (0.9174) loss 0.6173 (0.5901) grad_norm 0.2364 (0.2223) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:51:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [327/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8979) loss 0.5963 (0.5882) grad_norm 0.2032 (0.2255) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:53:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [327/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8913) loss 0.5983 (0.5870) grad_norm 0.2274 (0.2257) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:54:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [327/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8880) loss 0.6102 (0.5871) grad_norm 0.2309 (0.2255) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:54:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 327 training takes 0:05:57 [2024-03-06 11:54:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [328/800][0/402] eta 0:33:03 lr 0.000025 time 4.9347 (4.9347) loss 0.5886 (0.5886) grad_norm 0.2293 (0.2293) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:56:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [328/800][100/402] eta 0:04:37 lr 0.000025 time 0.8791 (0.9189) loss 0.5884 (0.5847) grad_norm 0.2186 (0.2255) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 11:57:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [328/800][200/402] eta 0:03:01 lr 0.000025 time 0.8792 (0.8988) loss 0.6128 (0.5863) grad_norm 0.2084 (nan) loss_scale 262144.0000 (473424.2388) mem 30609MB [2024-03-06 11:59:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [328/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8920) loss 0.5766 (0.5861) grad_norm 0.2216 (nan) loss_scale 262144.0000 (403231.4684) mem 30609MB [2024-03-06 12:00:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [328/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8887) loss 0.5691 (0.5863) grad_norm 0.2432 (nan) loss_scale 262144.0000 (368047.5611) mem 30609MB [2024-03-06 12:00:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 328 training takes 0:05:57 [2024-03-06 12:00:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [329/800][0/402] eta 0:35:31 lr 0.000025 time 5.3019 (5.3019) loss 0.5753 (0.5753) grad_norm 0.2148 (0.2148) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:02:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [329/800][100/402] eta 0:04:39 lr 0.000025 time 0.8785 (0.9240) loss 0.6079 (0.5882) grad_norm 0.2144 (0.2169) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:03:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [329/800][200/402] eta 0:03:02 lr 0.000025 time 0.8786 (0.9016) loss 0.5772 (0.5859) grad_norm 0.2046 (0.2220) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:05:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [329/800][300/402] eta 0:01:31 lr 0.000025 time 0.8785 (0.8940) loss 0.6261 (0.5869) grad_norm 0.2263 (0.2237) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:06:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [329/800][400/402] eta 0:00:01 lr 0.000025 time 0.8776 (0.8902) loss 0.5937 (0.5869) grad_norm 0.2432 (0.2242) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:06:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 329 training takes 0:05:58 [2024-03-06 12:06:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [330/800][0/402] eta 0:34:58 lr 0.000025 time 5.2197 (5.2197) loss 0.6071 (0.6071) grad_norm 0.2076 (0.2076) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:08:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [330/800][100/402] eta 0:04:38 lr 0.000025 time 0.8782 (0.9216) loss 0.5730 (0.5860) grad_norm 0.2403 (0.2296) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:09:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [330/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8999) loss 0.5674 (0.5863) grad_norm 0.2050 (0.2292) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:11:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [330/800][300/402] eta 0:01:31 lr 0.000025 time 0.8778 (0.8927) loss 0.5710 (0.5856) grad_norm 0.2070 (0.2260) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:12:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [330/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8891) loss 0.5649 (0.5862) grad_norm 0.1894 (0.2264) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:12:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 330 training takes 0:05:57 [2024-03-06 12:12:42 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_330.pth saving...... [2024-03-06 12:12:44 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_330.pth saved !!! [2024-03-06 12:12:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [331/800][0/402] eta 0:35:15 lr 0.000025 time 5.2622 (5.2622) loss 0.5332 (0.5332) grad_norm 0.1970 (0.1970) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:14:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [331/800][100/402] eta 0:04:38 lr 0.000025 time 0.8830 (0.9229) loss 0.5867 (0.5848) grad_norm 0.1956 (0.2318) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:15:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [331/800][200/402] eta 0:03:02 lr 0.000025 time 0.8783 (0.9011) loss 0.6038 (0.5850) grad_norm 0.2407 (0.2295) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:17:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [331/800][300/402] eta 0:01:31 lr 0.000025 time 0.8826 (0.8937) loss 0.6025 (0.5856) grad_norm 0.2792 (0.2266) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:18:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [331/800][400/402] eta 0:00:01 lr 0.000025 time 0.8798 (0.8899) loss 0.5732 (0.5853) grad_norm 0.2175 (0.2262) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:18:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 331 training takes 0:05:58 [2024-03-06 12:18:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [332/800][0/402] eta 0:33:32 lr 0.000025 time 5.0074 (5.0074) loss 0.5764 (0.5764) grad_norm 0.2138 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:20:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [332/800][100/402] eta 0:04:37 lr 0.000025 time 0.8802 (0.9202) loss 0.5781 (0.5848) grad_norm 0.2192 (0.2203) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:21:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [332/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8996) loss 0.5925 (0.5853) grad_norm 0.2170 (0.2212) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:23:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [332/800][300/402] eta 0:01:31 lr 0.000025 time 0.8788 (0.8927) loss 0.6077 (0.5858) grad_norm 0.2653 (0.2234) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:24:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [332/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8892) loss 0.5693 (0.5858) grad_norm 0.2626 (0.2230) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:24:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 332 training takes 0:05:57 [2024-03-06 12:24:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [333/800][0/402] eta 0:34:26 lr 0.000025 time 5.1407 (5.1407) loss 0.5834 (0.5834) grad_norm 0.2147 (0.2147) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:26:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [333/800][100/402] eta 0:04:38 lr 0.000025 time 0.8803 (0.9214) loss 0.5933 (0.5863) grad_norm 0.2290 (0.2252) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:27:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [333/800][200/402] eta 0:03:01 lr 0.000025 time 0.8810 (0.9007) loss 0.5584 (0.5850) grad_norm 0.2338 (0.2267) loss_scale 524288.0000 (326049.7512) mem 30609MB [2024-03-06 12:29:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [333/800][300/402] eta 0:01:31 lr 0.000025 time 0.8783 (0.8936) loss 0.6047 (0.5853) grad_norm 0.1925 (0.2249) loss_scale 524288.0000 (391909.6346) mem 30609MB [2024-03-06 12:30:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [333/800][400/402] eta 0:00:01 lr 0.000025 time 0.8774 (0.8898) loss 0.5800 (0.5864) grad_norm 0.2172 (0.2249) loss_scale 524288.0000 (424921.6958) mem 30609MB [2024-03-06 12:30:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 333 training takes 0:05:57 [2024-03-06 12:30:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [334/800][0/402] eta 0:31:49 lr 0.000025 time 4.7490 (4.7490) loss 0.5881 (0.5881) grad_norm 0.2175 (0.2175) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 12:32:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [334/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9172) loss 0.5836 (0.5835) grad_norm 0.2162 (0.2325) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 12:33:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [334/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8981) loss 0.6033 (0.5849) grad_norm 0.1913 (0.2271) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 12:35:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [334/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8917) loss 0.5734 (0.5858) grad_norm 0.2286 (0.2283) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 12:36:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [334/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8884) loss 0.5891 (0.5860) grad_norm 0.2446 (0.2267) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 12:36:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 334 training takes 0:05:57 [2024-03-06 12:36:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [335/800][0/402] eta 0:31:32 lr 0.000025 time 4.7072 (4.7072) loss 0.5896 (0.5896) grad_norm 0.1819 (0.1819) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 12:38:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [335/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9168) loss 0.5947 (0.5879) grad_norm 0.2006 (0.2259) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 12:39:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [335/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8977) loss 0.5761 (0.5875) grad_norm 0.2071 (0.2213) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 12:41:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [335/800][300/402] eta 0:01:30 lr 0.000025 time 0.8776 (0.8914) loss 0.5899 (0.5869) grad_norm 0.2003 (nan) loss_scale 262144.0000 (458098.8173) mem 30609MB [2024-03-06 12:42:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [335/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8881) loss 0.5934 (0.5863) grad_norm 0.2410 (nan) loss_scale 262144.0000 (409232.2793) mem 30609MB [2024-03-06 12:42:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 335 training takes 0:05:57 [2024-03-06 12:42:33 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_335.pth saving...... [2024-03-06 12:42:35 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_335.pth saved !!! [2024-03-06 12:42:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [336/800][0/402] eta 0:34:59 lr 0.000025 time 5.2231 (5.2231) loss 0.6144 (0.6144) grad_norm 0.1939 (0.1939) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:44:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [336/800][100/402] eta 0:04:38 lr 0.000025 time 0.8787 (0.9220) loss 0.5816 (0.5831) grad_norm 0.2117 (0.2280) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:45:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [336/800][200/402] eta 0:03:01 lr 0.000025 time 0.8799 (0.9006) loss 0.5711 (0.5847) grad_norm 0.2130 (0.2260) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:47:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [336/800][300/402] eta 0:01:31 lr 0.000025 time 0.8785 (0.8934) loss 0.5730 (0.5862) grad_norm 0.1998 (0.2256) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:48:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [336/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8897) loss 0.6010 (0.5859) grad_norm 0.2639 (0.2256) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:48:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 336 training takes 0:05:57 [2024-03-06 12:48:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [337/800][0/402] eta 0:35:02 lr 0.000025 time 5.2289 (5.2289) loss 0.5842 (0.5842) grad_norm 0.2088 (0.2088) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:50:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [337/800][100/402] eta 0:04:38 lr 0.000025 time 0.8807 (0.9221) loss 0.5980 (0.5873) grad_norm 0.2177 (0.2228) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:51:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [337/800][200/402] eta 0:03:01 lr 0.000025 time 0.8811 (0.9006) loss 0.5603 (0.5852) grad_norm 0.2219 (0.2216) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:53:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [337/800][300/402] eta 0:01:31 lr 0.000025 time 0.8797 (0.8933) loss 0.6122 (0.5858) grad_norm 0.2136 (0.2220) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:54:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [337/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8897) loss 0.6200 (0.5866) grad_norm 0.2247 (0.2213) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:54:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 337 training takes 0:05:57 [2024-03-06 12:54:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [338/800][0/402] eta 0:31:58 lr 0.000025 time 4.7733 (4.7733) loss 0.5737 (0.5737) grad_norm 0.2077 (0.2077) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:56:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [338/800][100/402] eta 0:04:36 lr 0.000025 time 0.8778 (0.9170) loss 0.5928 (0.5836) grad_norm 0.2214 (0.2239) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:57:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [338/800][200/402] eta 0:03:01 lr 0.000025 time 0.8769 (0.8980) loss 0.5756 (0.5853) grad_norm 0.2240 (0.2297) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 12:58:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [338/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8914) loss 0.5801 (0.5862) grad_norm 0.2112 (0.2276) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:00:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [338/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8881) loss 0.6124 (0.5854) grad_norm 0.2106 (0.2264) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:00:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 338 training takes 0:05:57 [2024-03-06 13:00:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [339/800][0/402] eta 0:32:26 lr 0.000025 time 4.8410 (4.8410) loss 0.5817 (0.5817) grad_norm 0.2209 (0.2209) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:02:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [339/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9179) loss 0.5879 (0.5823) grad_norm 0.2277 (0.2206) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:03:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [339/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8983) loss 0.5991 (0.5844) grad_norm 0.2059 (0.2226) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:04:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [339/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8917) loss 0.5747 (0.5854) grad_norm 0.2311 (0.2250) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:06:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [339/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8883) loss 0.6041 (0.5864) grad_norm 0.1998 (0.2233) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:06:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 339 training takes 0:05:57 [2024-03-06 13:06:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [340/800][0/402] eta 0:35:58 lr 0.000025 time 5.3682 (5.3682) loss 0.5639 (0.5639) grad_norm 0.2052 (0.2052) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:07:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [340/800][100/402] eta 0:04:38 lr 0.000025 time 0.8784 (0.9234) loss 0.5920 (0.5870) grad_norm 0.2141 (inf) loss_scale 131072.0000 (240082.3762) mem 30609MB [2024-03-06 13:09:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [340/800][200/402] eta 0:03:02 lr 0.000025 time 0.8793 (0.9013) loss 0.5629 (0.5878) grad_norm 0.2056 (inf) loss_scale 131072.0000 (185848.3582) mem 30609MB [2024-03-06 13:10:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [340/800][300/402] eta 0:01:31 lr 0.000025 time 0.8796 (0.8940) loss 0.6002 (0.5874) grad_norm 0.2267 (inf) loss_scale 131072.0000 (167650.2326) mem 30609MB [2024-03-06 13:12:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [340/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8902) loss 0.5922 (0.5871) grad_norm 0.2111 (inf) loss_scale 131072.0000 (158528.4788) mem 30609MB [2024-03-06 13:12:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 340 training takes 0:05:58 [2024-03-06 13:12:23 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_340.pth saving...... [2024-03-06 13:12:25 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_340.pth saved !!! [2024-03-06 13:12:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [341/800][0/402] eta 0:31:53 lr 0.000025 time 4.7593 (4.7593) loss 0.5819 (0.5819) grad_norm 0.2136 (0.2136) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:13:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [341/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9172) loss 0.5838 (0.5855) grad_norm 0.2566 (0.2198) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:15:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [341/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8980) loss 0.5642 (0.5855) grad_norm 0.2333 (0.2186) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:16:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [341/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8916) loss 0.6000 (0.5854) grad_norm 0.2048 (0.2202) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:18:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [341/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8883) loss 0.6183 (0.5859) grad_norm 0.2319 (0.2214) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:18:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 341 training takes 0:05:57 [2024-03-06 13:18:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [342/800][0/402] eta 0:33:52 lr 0.000025 time 5.0549 (5.0549) loss 0.5719 (0.5719) grad_norm 0.2626 (0.2626) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:19:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [342/800][100/402] eta 0:04:38 lr 0.000025 time 0.8785 (0.9210) loss 0.6102 (0.5841) grad_norm 0.2169 (0.2180) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:21:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [342/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.9001) loss 0.6064 (0.5863) grad_norm 0.1988 (0.2244) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:22:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [342/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8930) loss 0.5694 (0.5856) grad_norm 0.2078 (0.2222) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:24:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [342/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8895) loss 0.5781 (0.5854) grad_norm 0.2342 (0.2227) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:24:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 342 training takes 0:05:57 [2024-03-06 13:24:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [343/800][0/402] eta 0:36:05 lr 0.000025 time 5.3863 (5.3863) loss 0.5763 (0.5763) grad_norm 0.2604 (0.2604) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:25:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [343/800][100/402] eta 0:04:38 lr 0.000025 time 0.8781 (0.9234) loss 0.6062 (0.5854) grad_norm 0.1988 (0.2230) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:27:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [343/800][200/402] eta 0:03:02 lr 0.000025 time 0.8786 (0.9012) loss 0.5875 (0.5876) grad_norm 0.2557 (0.2215) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:28:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [343/800][300/402] eta 0:01:31 lr 0.000025 time 0.8781 (0.8937) loss 0.5765 (0.5854) grad_norm 0.2212 (0.2225) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:30:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [343/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8899) loss 0.5621 (0.5859) grad_norm 0.2001 (0.2220) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:30:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 343 training takes 0:05:57 [2024-03-06 13:30:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [344/800][0/402] eta 0:35:35 lr 0.000025 time 5.3125 (5.3125) loss 0.6193 (0.6193) grad_norm 0.1880 (0.1880) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:31:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [344/800][100/402] eta 0:04:38 lr 0.000025 time 0.8786 (0.9229) loss 0.5716 (0.5859) grad_norm 0.2047 (0.2230) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:33:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [344/800][200/402] eta 0:03:02 lr 0.000025 time 0.8787 (0.9010) loss 0.5543 (0.5849) grad_norm 0.2019 (0.2215) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:34:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [344/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8939) loss 0.5754 (0.5853) grad_norm 0.1919 (0.2221) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:36:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [344/800][400/402] eta 0:00:01 lr 0.000025 time 0.8799 (0.8901) loss 0.6078 (0.5862) grad_norm 0.2102 (0.2225) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:36:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 344 training takes 0:05:58 [2024-03-06 13:36:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [345/800][0/402] eta 0:33:42 lr 0.000025 time 5.0300 (5.0300) loss 0.5783 (0.5783) grad_norm 0.1868 (0.1868) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 13:37:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [345/800][100/402] eta 0:04:38 lr 0.000025 time 0.8783 (0.9209) loss 0.5988 (0.5869) grad_norm 0.2062 (0.2253) loss_scale 262144.0000 (166111.0495) mem 30609MB [2024-03-06 13:39:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [345/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8998) loss 0.5761 (0.5861) grad_norm 0.2057 (0.2226) loss_scale 262144.0000 (213888.6368) mem 30609MB [2024-03-06 13:40:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [345/800][300/402] eta 0:01:31 lr 0.000025 time 0.8779 (0.8927) loss 0.5779 (0.5866) grad_norm 0.2090 (0.2237) loss_scale 262144.0000 (229920.3189) mem 30609MB [2024-03-06 13:42:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [345/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8891) loss 0.6090 (0.5866) grad_norm 0.2469 (0.2245) loss_scale 262144.0000 (237956.1496) mem 30609MB [2024-03-06 13:42:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 345 training takes 0:05:57 [2024-03-06 13:42:14 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_345.pth saving...... [2024-03-06 13:42:16 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_345.pth saved !!! [2024-03-06 13:42:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [346/800][0/402] eta 0:31:35 lr 0.000025 time 4.7146 (4.7146) loss 0.5795 (0.5795) grad_norm 0.2318 (0.2318) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:43:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [346/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9166) loss 0.5608 (0.5832) grad_norm 0.2110 (0.2213) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:45:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [346/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8978) loss 0.6406 (0.5842) grad_norm 0.2217 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:46:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [346/800][300/402] eta 0:01:30 lr 0.000025 time 0.8792 (0.8916) loss 0.5865 (0.5847) grad_norm 0.2105 (0.2216) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:48:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [346/800][400/402] eta 0:00:01 lr 0.000025 time 0.8776 (0.8883) loss 0.6062 (0.5848) grad_norm 0.2031 (0.2212) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:48:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 346 training takes 0:05:57 [2024-03-06 13:48:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [347/800][0/402] eta 0:34:21 lr 0.000025 time 5.1291 (5.1291) loss 0.5695 (0.5695) grad_norm 0.2386 (0.2386) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:49:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [347/800][100/402] eta 0:04:38 lr 0.000025 time 0.8782 (0.9206) loss 0.5750 (0.5860) grad_norm 0.2179 (0.2257) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:51:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [347/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8999) loss 0.6178 (0.5850) grad_norm 0.1808 (0.2238) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:52:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [347/800][300/402] eta 0:01:31 lr 0.000025 time 0.8783 (0.8928) loss 0.5682 (0.5853) grad_norm 0.2152 (0.2227) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:54:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [347/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8892) loss 0.5595 (0.5851) grad_norm 0.2142 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:54:11 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 347 training takes 0:05:57 [2024-03-06 13:54:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [348/800][0/402] eta 0:31:12 lr 0.000025 time 4.6580 (4.6580) loss 0.5944 (0.5944) grad_norm 0.2108 (0.2108) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:55:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [348/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9160) loss 0.6047 (0.5842) grad_norm 0.2420 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:57:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [348/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8974) loss 0.6210 (0.5845) grad_norm 0.1936 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 13:58:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [348/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8912) loss 0.5902 (0.5849) grad_norm 0.2608 (0.2192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:00:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [348/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8880) loss 0.5761 (0.5849) grad_norm 0.2266 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:00:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 348 training takes 0:05:57 [2024-03-06 14:00:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [349/800][0/402] eta 0:31:26 lr 0.000025 time 4.6927 (4.6927) loss 0.5765 (0.5765) grad_norm 0.2289 (0.2289) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:01:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [349/800][100/402] eta 0:04:37 lr 0.000025 time 0.8780 (0.9177) loss 0.5875 (0.5848) grad_norm 0.2147 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:03:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [349/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8986) loss 0.5571 (0.5849) grad_norm 0.2499 (0.2221) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:04:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [349/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8920) loss 0.5928 (0.5852) grad_norm 0.2218 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:06:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [349/800][400/402] eta 0:00:01 lr 0.000025 time 0.8774 (0.8889) loss 0.5684 (0.5853) grad_norm 0.2387 (0.2221) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:06:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 349 training takes 0:05:57 [2024-03-06 14:06:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [350/800][0/402] eta 0:34:38 lr 0.000025 time 5.1704 (5.1704) loss 0.6061 (0.6061) grad_norm 0.2129 (0.2129) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:07:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [350/800][100/402] eta 0:04:38 lr 0.000025 time 0.8783 (0.9213) loss 0.6012 (0.5867) grad_norm 0.2233 (0.2163) loss_scale 524288.0000 (358176.9505) mem 30609MB [2024-03-06 14:09:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [350/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.9003) loss 0.5575 (0.5870) grad_norm 0.1889 (0.2195) loss_scale 524288.0000 (440819.2637) mem 30609MB [2024-03-06 14:10:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [350/800][300/402] eta 0:01:31 lr 0.000025 time 0.8783 (0.8931) loss 0.5528 (0.5866) grad_norm 0.2080 (0.2211) loss_scale 524288.0000 (468549.7409) mem 30609MB [2024-03-06 14:12:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [350/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8895) loss 0.6001 (0.5862) grad_norm 0.2198 (0.2222) loss_scale 524288.0000 (482449.5561) mem 30609MB [2024-03-06 14:12:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 350 training takes 0:05:57 [2024-03-06 14:12:04 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_350.pth saving...... [2024-03-06 14:12:06 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_350.pth saved !!! [2024-03-06 14:12:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [351/800][0/402] eta 0:32:26 lr 0.000025 time 4.8433 (4.8433) loss 0.6068 (0.6068) grad_norm 0.2544 (0.2544) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 14:13:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [351/800][100/402] eta 0:04:37 lr 0.000025 time 0.8784 (0.9184) loss 0.5579 (0.5851) grad_norm 0.2011 (inf) loss_scale 262144.0000 (290694.3366) mem 30609MB [2024-03-06 14:15:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [351/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8986) loss 0.5602 (0.5842) grad_norm 0.1909 (inf) loss_scale 262144.0000 (276490.1891) mem 30609MB [2024-03-06 14:16:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [351/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8919) loss 0.6314 (0.5856) grad_norm 0.2025 (inf) loss_scale 262144.0000 (271724.0133) mem 30609MB [2024-03-06 14:18:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [351/800][400/402] eta 0:00:01 lr 0.000025 time 0.8779 (0.8885) loss 0.5887 (0.5853) grad_norm 0.2226 (inf) loss_scale 262144.0000 (269334.9825) mem 30609MB [2024-03-06 14:18:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 351 training takes 0:05:57 [2024-03-06 14:18:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [352/800][0/402] eta 0:34:19 lr 0.000025 time 5.1230 (5.1230) loss 0.5830 (0.5830) grad_norm 0.2274 (0.2274) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:19:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [352/800][100/402] eta 0:04:38 lr 0.000025 time 0.8780 (0.9211) loss 0.6097 (0.5838) grad_norm 0.1790 (0.2192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:21:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [352/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8997) loss 0.5906 (0.5851) grad_norm 0.2145 (0.2194) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:22:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [352/800][300/402] eta 0:01:31 lr 0.000025 time 0.8782 (0.8925) loss 0.5861 (0.5857) grad_norm 0.2453 (0.2212) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:24:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [352/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8888) loss 0.5975 (0.5852) grad_norm 0.1996 (0.2212) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:24:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 352 training takes 0:05:57 [2024-03-06 14:24:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [353/800][0/402] eta 0:32:30 lr 0.000025 time 4.8511 (4.8511) loss 0.5656 (0.5656) grad_norm 0.2549 (0.2549) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:25:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [353/800][100/402] eta 0:04:37 lr 0.000025 time 0.8780 (0.9181) loss 0.6146 (0.5845) grad_norm 0.2362 (0.2223) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:27:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [353/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8985) loss 0.6193 (0.5849) grad_norm 0.2177 (0.2209) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:28:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [353/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8921) loss 0.5828 (0.5858) grad_norm 0.1852 (0.2185) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:29:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [353/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8887) loss 0.5479 (0.5855) grad_norm 0.2028 (0.2226) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:29:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 353 training takes 0:05:57 [2024-03-06 14:30:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [354/800][0/402] eta 0:36:02 lr 0.000025 time 5.3796 (5.3796) loss 0.5761 (0.5761) grad_norm 0.2348 (0.2348) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:31:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [354/800][100/402] eta 0:04:38 lr 0.000025 time 0.8774 (0.9232) loss 0.5671 (0.5878) grad_norm 0.2378 (0.2204) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:32:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [354/800][200/402] eta 0:03:02 lr 0.000025 time 0.8784 (0.9013) loss 0.5383 (0.5865) grad_norm 0.2232 (0.2237) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:34:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [354/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8937) loss 0.5816 (0.5866) grad_norm 0.2420 (0.2224) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:35:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [354/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8898) loss 0.6276 (0.5866) grad_norm 0.2289 (0.2204) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:35:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 354 training takes 0:05:57 [2024-03-06 14:36:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [355/800][0/402] eta 0:35:17 lr 0.000025 time 5.2677 (5.2677) loss 0.5876 (0.5876) grad_norm 0.2394 (0.2394) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:37:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [355/800][100/402] eta 0:04:38 lr 0.000025 time 0.8774 (0.9222) loss 0.6167 (0.5875) grad_norm 0.2423 (0.2245) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:38:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [355/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.9006) loss 0.5869 (0.5849) grad_norm 0.2344 (0.2227) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:40:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [355/800][300/402] eta 0:01:31 lr 0.000025 time 0.8780 (0.8933) loss 0.5735 (0.5849) grad_norm 0.2033 (0.2214) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:41:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [355/800][400/402] eta 0:00:01 lr 0.000025 time 0.8788 (0.8898) loss 0.5934 (0.5849) grad_norm 0.2198 (0.2215) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:41:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 355 training takes 0:05:57 [2024-03-06 14:41:54 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_355.pth saving...... [2024-03-06 14:41:56 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_355.pth saved !!! [2024-03-06 14:42:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [356/800][0/402] eta 0:35:10 lr 0.000025 time 5.2501 (5.2501) loss 0.5864 (0.5864) grad_norm 0.1951 (0.1951) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:43:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [356/800][100/402] eta 0:04:38 lr 0.000025 time 0.8796 (0.9221) loss 0.5837 (0.5874) grad_norm 0.2376 (0.2211) loss_scale 524288.0000 (521692.5149) mem 30609MB [2024-03-06 14:44:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [356/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.9004) loss 0.5729 (0.5869) grad_norm 0.2133 (inf) loss_scale 262144.0000 (499508.2189) mem 30609MB [2024-03-06 14:46:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [356/800][300/402] eta 0:01:31 lr 0.000025 time 0.8786 (0.8934) loss 0.5899 (0.5854) grad_norm 0.2254 (inf) loss_scale 262144.0000 (420649.6744) mem 30609MB [2024-03-06 14:47:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [356/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8897) loss 0.5502 (0.5853) grad_norm 0.2039 (inf) loss_scale 262144.0000 (381122.0748) mem 30609MB [2024-03-06 14:47:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 356 training takes 0:05:57 [2024-03-06 14:47:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [357/800][0/402] eta 0:34:01 lr 0.000025 time 5.0780 (5.0780) loss 0.5817 (0.5817) grad_norm 0.2168 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:49:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [357/800][100/402] eta 0:04:37 lr 0.000025 time 0.8777 (0.9204) loss 0.5834 (0.5848) grad_norm 0.2065 (0.2205) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:50:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [357/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8995) loss 0.5892 (0.5859) grad_norm 0.2361 (0.2212) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:52:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [357/800][300/402] eta 0:01:31 lr 0.000025 time 0.8788 (0.8926) loss 0.5594 (0.5857) grad_norm 0.2250 (0.2201) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:53:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [357/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8891) loss 0.6188 (0.5852) grad_norm 0.1667 (0.2218) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:53:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 357 training takes 0:05:57 [2024-03-06 14:53:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [358/800][0/402] eta 0:34:06 lr 0.000025 time 5.0914 (5.0914) loss 0.6023 (0.6023) grad_norm 0.2490 (0.2490) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:55:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [358/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9204) loss 0.6110 (0.5860) grad_norm 0.1921 (0.2210) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:56:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [358/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8997) loss 0.5729 (0.5857) grad_norm 0.2217 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:58:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [358/800][300/402] eta 0:01:31 lr 0.000025 time 0.8792 (0.8927) loss 0.5949 (0.5859) grad_norm 0.2079 (0.2190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:59:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [358/800][400/402] eta 0:00:01 lr 0.000025 time 0.8778 (0.8894) loss 0.5562 (0.5862) grad_norm 0.2122 (0.2197) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 14:59:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 358 training takes 0:05:57 [2024-03-06 14:59:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [359/800][0/402] eta 0:35:25 lr 0.000025 time 5.2863 (5.2863) loss 0.5746 (0.5746) grad_norm 0.2375 (0.2375) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:01:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [359/800][100/402] eta 0:04:38 lr 0.000025 time 0.8806 (0.9227) loss 0.5601 (0.5831) grad_norm 0.1920 (0.2209) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:02:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [359/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.9009) loss 0.5817 (0.5847) grad_norm 0.2208 (0.2213) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:04:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [359/800][300/402] eta 0:01:31 lr 0.000025 time 0.8807 (0.8935) loss 0.5871 (0.5851) grad_norm 0.2329 (0.2215) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:05:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [359/800][400/402] eta 0:00:01 lr 0.000025 time 0.8788 (0.8898) loss 0.5661 (0.5856) grad_norm 0.2183 (0.2202) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:05:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 359 training takes 0:05:57 [2024-03-06 15:05:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [360/800][0/402] eta 0:34:51 lr 0.000025 time 5.2033 (5.2033) loss 0.5880 (0.5880) grad_norm 0.2128 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:07:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [360/800][100/402] eta 0:04:38 lr 0.000025 time 0.8777 (0.9226) loss 0.6019 (0.5827) grad_norm 0.2070 (0.2158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:08:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [360/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.9009) loss 0.5500 (0.5845) grad_norm 0.2429 (0.2184) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:10:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [360/800][300/402] eta 0:01:31 lr 0.000025 time 0.8788 (0.8936) loss 0.5746 (0.5853) grad_norm 0.2421 (0.2194) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:11:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [360/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8898) loss 0.5779 (0.5856) grad_norm 0.2180 (0.2210) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:11:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 360 training takes 0:05:57 [2024-03-06 15:11:46 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_360.pth saving...... [2024-03-06 15:11:48 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_360.pth saved !!! [2024-03-06 15:11:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [361/800][0/402] eta 0:34:32 lr 0.000025 time 5.1544 (5.1544) loss 0.6031 (0.6031) grad_norm 0.2075 (0.2075) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:13:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [361/800][100/402] eta 0:04:38 lr 0.000025 time 0.8807 (0.9217) loss 0.6001 (0.5844) grad_norm 0.2045 (0.2178) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:14:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [361/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.9004) loss 0.6011 (0.5857) grad_norm 0.2266 (0.2185) loss_scale 524288.0000 (298661.5721) mem 30609MB [2024-03-06 15:16:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [361/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8932) loss 0.5969 (0.5849) grad_norm 0.2262 (0.2187) loss_scale 524288.0000 (373620.5183) mem 30609MB [2024-03-06 15:17:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [361/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8895) loss 0.5816 (0.5848) grad_norm 0.2226 (nan) loss_scale 262144.0000 (363471.4813) mem 30609MB [2024-03-06 15:17:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 361 training takes 0:05:57 [2024-03-06 15:17:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [362/800][0/402] eta 0:32:01 lr 0.000025 time 4.7790 (4.7790) loss 0.5561 (0.5561) grad_norm 0.2028 (0.2028) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:19:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [362/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9180) loss 0.5894 (0.5870) grad_norm 0.2162 (0.2200) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:20:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [362/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8982) loss 0.5686 (0.5852) grad_norm 0.2343 (0.2193) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:22:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [362/800][300/402] eta 0:01:30 lr 0.000025 time 0.8770 (0.8915) loss 0.5932 (0.5857) grad_norm 0.2235 (0.2195) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:23:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [362/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8882) loss 0.6050 (0.5850) grad_norm 0.2048 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:23:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 362 training takes 0:05:57 [2024-03-06 15:23:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [363/800][0/402] eta 0:32:13 lr 0.000025 time 4.8091 (4.8091) loss 0.5380 (0.5380) grad_norm 0.2415 (0.2415) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:25:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [363/800][100/402] eta 0:04:37 lr 0.000025 time 0.8807 (0.9177) loss 0.5592 (0.5829) grad_norm 0.2052 (0.2191) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:26:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [363/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8983) loss 0.5908 (0.5843) grad_norm 0.2719 (0.2302) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:28:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [363/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8919) loss 0.6066 (0.5842) grad_norm 0.2154 (0.2257) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:29:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [363/800][400/402] eta 0:00:01 lr 0.000025 time 0.8776 (0.8885) loss 0.6026 (0.5844) grad_norm 0.2095 (0.2234) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:29:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 363 training takes 0:05:57 [2024-03-06 15:29:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [364/800][0/402] eta 0:29:43 lr 0.000025 time 4.4369 (4.4369) loss 0.5837 (0.5837) grad_norm 0.2364 (0.2364) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:31:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [364/800][100/402] eta 0:04:36 lr 0.000025 time 0.8776 (0.9143) loss 0.5897 (0.5842) grad_norm 0.2292 (0.2392) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:32:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [364/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8968) loss 0.5760 (0.5854) grad_norm 0.2186 (0.2271) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:34:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [364/800][300/402] eta 0:01:30 lr 0.000025 time 0.8799 (0.8908) loss 0.5796 (0.5846) grad_norm 0.2433 (0.2251) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:35:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [364/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8878) loss 0.5972 (0.5839) grad_norm 0.2022 (0.2255) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:35:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 364 training takes 0:05:57 [2024-03-06 15:35:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [365/800][0/402] eta 0:36:51 lr 0.000025 time 5.5018 (5.5018) loss 0.5899 (0.5899) grad_norm 0.2138 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:37:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [365/800][100/402] eta 0:04:39 lr 0.000025 time 0.8779 (0.9245) loss 0.5936 (0.5851) grad_norm 0.2231 (0.2169) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:38:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [365/800][200/402] eta 0:03:02 lr 0.000025 time 0.8786 (0.9017) loss 0.5941 (0.5853) grad_norm 0.2003 (0.2178) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:40:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [365/800][300/402] eta 0:01:31 lr 0.000025 time 0.8805 (0.8941) loss 0.5868 (0.5853) grad_norm 0.1959 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:41:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [365/800][400/402] eta 0:00:01 lr 0.000025 time 0.8779 (0.8904) loss 0.5772 (0.5844) grad_norm 0.2305 (0.2184) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:41:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 365 training takes 0:05:58 [2024-03-06 15:41:36 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_365.pth saving...... [2024-03-06 15:41:38 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_365.pth saved !!! [2024-03-06 15:41:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [366/800][0/402] eta 0:33:37 lr 0.000025 time 5.0198 (5.0198) loss 0.5682 (0.5682) grad_norm 0.2298 (0.2298) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:43:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [366/800][100/402] eta 0:04:37 lr 0.000025 time 0.8784 (0.9198) loss 0.5659 (0.5845) grad_norm 0.2294 (0.2139) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:44:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [366/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8998) loss 0.5716 (0.5851) grad_norm 0.2369 (0.2201) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:46:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [366/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8928) loss 0.6009 (0.5867) grad_norm 0.2402 (0.2191) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 15:47:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [366/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8893) loss 0.5741 (0.5863) grad_norm 0.1817 (0.2194) loss_scale 524288.0000 (316403.2319) mem 30609MB [2024-03-06 15:47:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 366 training takes 0:05:57 [2024-03-06 15:47:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [367/800][0/402] eta 0:31:53 lr 0.000025 time 4.7595 (4.7595) loss 0.6004 (0.6004) grad_norm 0.2333 (0.2333) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:49:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [367/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9172) loss 0.5809 (0.5839) grad_norm 0.2111 (0.2189) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:50:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [367/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8982) loss 0.5956 (0.5844) grad_norm 0.1958 (0.2265) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:52:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [367/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8918) loss 0.5959 (0.5839) grad_norm 0.2445 (0.2244) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:53:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [367/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8885) loss 0.5897 (0.5846) grad_norm 0.2039 (0.2223) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:53:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 367 training takes 0:05:57 [2024-03-06 15:53:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [368/800][0/402] eta 0:32:19 lr 0.000025 time 4.8252 (4.8252) loss 0.5946 (0.5946) grad_norm 0.2258 (0.2258) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:55:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [368/800][100/402] eta 0:04:37 lr 0.000025 time 0.8804 (0.9185) loss 0.5917 (0.5846) grad_norm 0.2180 (0.2157) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:56:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [368/800][200/402] eta 0:03:01 lr 0.000025 time 0.8800 (0.8989) loss 0.5888 (0.5853) grad_norm 0.1973 (0.2196) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:58:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [368/800][300/402] eta 0:01:31 lr 0.000025 time 0.8781 (0.8927) loss 0.5523 (0.5845) grad_norm 0.2845 (0.2222) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 15:59:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [368/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8892) loss 0.5757 (0.5843) grad_norm 0.2190 (nan) loss_scale 262144.0000 (502715.0524) mem 30609MB [2024-03-06 15:59:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 368 training takes 0:05:57 [2024-03-06 15:59:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [369/800][0/402] eta 0:32:35 lr 0.000025 time 4.8651 (4.8651) loss 0.6165 (0.6165) grad_norm 0.1806 (0.1806) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:01:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [369/800][100/402] eta 0:04:37 lr 0.000025 time 0.8807 (0.9183) loss 0.6145 (0.5884) grad_norm 0.2132 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:02:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [369/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8986) loss 0.5769 (0.5853) grad_norm 0.1929 (0.2158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:03:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [369/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8920) loss 0.5861 (0.5840) grad_norm 0.2063 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:05:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [369/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8887) loss 0.5916 (0.5840) grad_norm 0.2091 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:05:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 369 training takes 0:05:57 [2024-03-06 16:05:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [370/800][0/402] eta 0:30:56 lr 0.000025 time 4.6176 (4.6176) loss 0.5902 (0.5902) grad_norm 0.2165 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:07:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [370/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9159) loss 0.6018 (0.5813) grad_norm 0.2219 (0.2233) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:08:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [370/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8978) loss 0.5426 (0.5820) grad_norm 0.2095 (inf) loss_scale 131072.0000 (231495.3234) mem 30609MB [2024-03-06 16:09:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [370/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8916) loss 0.6108 (0.5833) grad_norm 0.2039 (inf) loss_scale 131072.0000 (198132.0930) mem 30609MB [2024-03-06 16:11:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [370/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8884) loss 0.6065 (0.5844) grad_norm 0.1714 (inf) loss_scale 131072.0000 (181408.8778) mem 30609MB [2024-03-06 16:11:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 370 training takes 0:05:57 [2024-03-06 16:11:26 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_370.pth saving...... [2024-03-06 16:11:28 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_370.pth saved !!! [2024-03-06 16:11:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [371/800][0/402] eta 0:35:03 lr 0.000025 time 5.2325 (5.2325) loss 0.5922 (0.5922) grad_norm 0.2161 (0.2161) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:13:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [371/800][100/402] eta 0:04:38 lr 0.000025 time 0.8787 (0.9220) loss 0.6072 (0.5861) grad_norm 0.2072 (0.2158) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:14:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [371/800][200/402] eta 0:03:01 lr 0.000025 time 0.8776 (0.9007) loss 0.5969 (0.5849) grad_norm 0.2330 (0.2165) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:15:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [371/800][300/402] eta 0:01:31 lr 0.000025 time 0.8789 (0.8934) loss 0.6095 (0.5847) grad_norm 0.1766 (0.2169) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:17:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [371/800][400/402] eta 0:00:01 lr 0.000025 time 0.8778 (0.8897) loss 0.5634 (0.5844) grad_norm 0.2663 (0.2174) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:17:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 371 training takes 0:05:57 [2024-03-06 16:17:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [372/800][0/402] eta 0:32:02 lr 0.000025 time 4.7829 (4.7829) loss 0.5770 (0.5770) grad_norm 0.1789 (0.1789) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:18:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [372/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9176) loss 0.5683 (0.5849) grad_norm 0.2417 (0.2207) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:20:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [372/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8984) loss 0.5970 (0.5843) grad_norm 0.2196 (0.2182) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:21:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [372/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8920) loss 0.5663 (0.5848) grad_norm 0.2652 (0.2212) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:23:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [372/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8886) loss 0.5909 (0.5853) grad_norm 0.2167 (0.2202) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:23:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 372 training takes 0:05:57 [2024-03-06 16:23:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [373/800][0/402] eta 0:31:41 lr 0.000025 time 4.7292 (4.7292) loss 0.5574 (0.5574) grad_norm 0.2206 (0.2206) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:24:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [373/800][100/402] eta 0:04:37 lr 0.000025 time 0.8788 (0.9175) loss 0.5874 (0.5855) grad_norm 0.2406 (0.2174) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:26:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [373/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8981) loss 0.5643 (0.5849) grad_norm 0.2050 (0.2161) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:27:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [373/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8916) loss 0.5555 (0.5847) grad_norm 0.2001 (0.2178) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:29:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [373/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8884) loss 0.5825 (0.5844) grad_norm 0.2647 (0.2161) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:29:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 373 training takes 0:05:57 [2024-03-06 16:29:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [374/800][0/402] eta 0:32:00 lr 0.000025 time 4.7781 (4.7781) loss 0.5978 (0.5978) grad_norm 0.2384 (0.2384) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:30:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [374/800][100/402] eta 0:04:37 lr 0.000025 time 0.8783 (0.9175) loss 0.6087 (0.5858) grad_norm 0.1840 (0.2302) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:32:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [374/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8984) loss 0.5910 (0.5845) grad_norm 0.2168 (0.2229) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:33:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [374/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8918) loss 0.5700 (0.5835) grad_norm 0.2134 (0.2219) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:35:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [374/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8885) loss 0.5487 (0.5839) grad_norm 0.2377 (0.2208) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:35:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 374 training takes 0:05:57 [2024-03-06 16:35:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [375/800][0/402] eta 0:31:32 lr 0.000025 time 4.7083 (4.7083) loss 0.5899 (0.5899) grad_norm 0.1899 (0.1899) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:36:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [375/800][100/402] eta 0:04:36 lr 0.000025 time 0.8797 (0.9167) loss 0.5805 (0.5842) grad_norm 0.2706 (0.2184) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-06 16:38:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [375/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8985) loss 0.5568 (0.5836) grad_norm 0.2025 (0.2261) loss_scale 262144.0000 (168241.6716) mem 30609MB [2024-03-06 16:39:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [375/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8918) loss 0.6038 (0.5842) grad_norm 0.2067 (0.2230) loss_scale 262144.0000 (199438.4585) mem 30609MB [2024-03-06 16:41:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [375/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8885) loss 0.5830 (0.5847) grad_norm 0.2404 (0.2219) loss_scale 262144.0000 (215075.7506) mem 30609MB [2024-03-06 16:41:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 375 training takes 0:05:57 [2024-03-06 16:41:15 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_375.pth saving...... [2024-03-06 16:41:17 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_375.pth saved !!! [2024-03-06 16:41:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [376/800][0/402] eta 0:32:16 lr 0.000025 time 4.8167 (4.8167) loss 0.6173 (0.6173) grad_norm 0.2081 (0.2081) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:42:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [376/800][100/402] eta 0:04:37 lr 0.000025 time 0.8788 (0.9176) loss 0.5728 (0.5828) grad_norm 0.2095 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:44:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [376/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8983) loss 0.5427 (0.5843) grad_norm 0.2526 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:45:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [376/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8917) loss 0.5632 (0.5844) grad_norm 0.2089 (0.2136) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:47:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [376/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8883) loss 0.5933 (0.5840) grad_norm 0.2303 (0.2178) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:47:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 376 training takes 0:05:57 [2024-03-06 16:47:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [377/800][0/402] eta 0:32:15 lr 0.000025 time 4.8137 (4.8137) loss 0.5820 (0.5820) grad_norm 0.2556 (0.2556) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:48:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [377/800][100/402] eta 0:04:37 lr 0.000025 time 0.8776 (0.9173) loss 0.5967 (0.5826) grad_norm 0.2238 (0.2216) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:50:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [377/800][200/402] eta 0:03:01 lr 0.000025 time 0.8794 (0.8987) loss 0.5948 (0.5842) grad_norm 0.2113 (0.2210) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:51:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [377/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8920) loss 0.5574 (0.5846) grad_norm 0.1898 (0.2193) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:53:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [377/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8885) loss 0.5830 (0.5854) grad_norm 0.2187 (0.2184) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:53:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 377 training takes 0:05:57 [2024-03-06 16:53:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [378/800][0/402] eta 0:31:29 lr 0.000025 time 4.7011 (4.7011) loss 0.5678 (0.5678) grad_norm 0.2174 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:54:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [378/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9164) loss 0.5874 (0.5848) grad_norm 0.2118 (0.2166) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:56:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [378/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8975) loss 0.5936 (0.5852) grad_norm 0.2307 (0.2173) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:57:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [378/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8912) loss 0.5952 (0.5846) grad_norm 0.2084 (0.2183) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:59:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [378/800][400/402] eta 0:00:01 lr 0.000025 time 0.8782 (0.8880) loss 0.6176 (0.5837) grad_norm 0.2168 (0.2198) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 16:59:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 378 training takes 0:05:57 [2024-03-06 16:59:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [379/800][0/402] eta 0:32:28 lr 0.000025 time 4.8466 (4.8466) loss 0.5763 (0.5763) grad_norm 0.2433 (0.2433) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:00:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [379/800][100/402] eta 0:04:37 lr 0.000025 time 0.8779 (0.9180) loss 0.5813 (0.5846) grad_norm 0.2013 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:02:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [379/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8984) loss 0.5978 (0.5845) grad_norm 0.1885 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:03:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [379/800][300/402] eta 0:01:31 lr 0.000025 time 0.8793 (0.8923) loss 0.5987 (0.5847) grad_norm 0.1943 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:05:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [379/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8888) loss 0.6141 (0.5843) grad_norm 0.1985 (0.2153) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:05:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 379 training takes 0:05:57 [2024-03-06 17:05:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [380/800][0/402] eta 0:32:23 lr 0.000025 time 4.8351 (4.8351) loss 0.5791 (0.5791) grad_norm 0.2241 (0.2241) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:06:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [380/800][100/402] eta 0:04:37 lr 0.000025 time 0.8795 (0.9178) loss 0.5792 (0.5862) grad_norm 0.1907 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:08:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [380/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8984) loss 0.5687 (0.5857) grad_norm 0.2182 (0.2175) loss_scale 524288.0000 (349525.3333) mem 30609MB [2024-03-06 17:09:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [380/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8918) loss 0.5993 (0.5844) grad_norm 0.2029 (inf) loss_scale 262144.0000 (403231.4684) mem 30609MB [2024-03-06 17:11:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [380/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8885) loss 0.6120 (0.5855) grad_norm 0.1945 (inf) loss_scale 262144.0000 (368047.5611) mem 30609MB [2024-03-06 17:11:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 380 training takes 0:05:57 [2024-03-06 17:11:04 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_380.pth saving...... [2024-03-06 17:11:06 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_380.pth saved !!! [2024-03-06 17:11:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [381/800][0/402] eta 0:30:36 lr 0.000025 time 4.5691 (4.5691) loss 0.5912 (0.5912) grad_norm 0.2071 (0.2071) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:12:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [381/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9152) loss 0.5864 (0.5828) grad_norm 0.2432 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:14:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [381/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8970) loss 0.6503 (0.5838) grad_norm 0.2254 (0.2193) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:15:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [381/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8909) loss 0.5794 (0.5852) grad_norm 0.2150 (0.2203) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:17:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [381/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8881) loss 0.5680 (0.5846) grad_norm 0.2303 (0.2218) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:17:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 381 training takes 0:05:57 [2024-03-06 17:17:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [382/800][0/402] eta 0:29:30 lr 0.000025 time 4.4052 (4.4052) loss 0.5636 (0.5636) grad_norm 0.2190 (0.2190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:18:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [382/800][100/402] eta 0:04:35 lr 0.000025 time 0.8791 (0.9137) loss 0.5995 (0.5872) grad_norm 0.2132 (0.2180) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:20:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [382/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8963) loss 0.6227 (0.5841) grad_norm 0.2487 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:21:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [382/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8905) loss 0.5840 (0.5836) grad_norm 0.1863 (0.2164) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:22:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [382/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8876) loss 0.5762 (0.5836) grad_norm 0.2116 (0.2158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:23:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 382 training takes 0:05:57 [2024-03-06 17:23:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [383/800][0/402] eta 0:29:59 lr 0.000025 time 4.4773 (4.4773) loss 0.6119 (0.6119) grad_norm 0.2333 (0.2333) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:24:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [383/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9144) loss 0.5714 (0.5835) grad_norm 0.2276 (0.2200) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:26:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [383/800][200/402] eta 0:03:01 lr 0.000025 time 0.8806 (0.8966) loss 0.5655 (0.5844) grad_norm 0.2361 (0.2204) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:27:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [383/800][300/402] eta 0:01:30 lr 0.000025 time 0.8803 (0.8908) loss 0.5849 (0.5846) grad_norm 0.2028 (0.2204) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:28:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [383/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8877) loss 0.5961 (0.5838) grad_norm 0.1895 (0.2202) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:28:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 383 training takes 0:05:57 [2024-03-06 17:29:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [384/800][0/402] eta 0:30:28 lr 0.000025 time 4.5479 (4.5479) loss 0.5875 (0.5875) grad_norm 0.2229 (0.2229) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:30:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [384/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9163) loss 0.5635 (0.5875) grad_norm 0.2399 (0.2234) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:31:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [384/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8974) loss 0.5748 (0.5863) grad_norm 0.2104 (0.2216) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:33:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [384/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8912) loss 0.5924 (0.5862) grad_norm 0.2079 (0.2197) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:34:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [384/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8879) loss 0.5707 (0.5853) grad_norm 0.2164 (0.2192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:34:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 384 training takes 0:05:57 [2024-03-06 17:34:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [385/800][0/402] eta 0:30:31 lr 0.000025 time 4.5551 (4.5551) loss 0.5765 (0.5765) grad_norm 0.2174 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:36:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [385/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9149) loss 0.5821 (0.5830) grad_norm 0.2141 (0.2109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:37:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [385/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.6083 (0.5835) grad_norm 0.2011 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:39:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [385/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8907) loss 0.5680 (0.5839) grad_norm 0.1995 (0.2170) loss_scale 524288.0000 (275207.6545) mem 30609MB [2024-03-06 17:40:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [385/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8876) loss 0.5862 (0.5844) grad_norm 0.2091 (0.2180) loss_scale 524288.0000 (337322.4539) mem 30609MB [2024-03-06 17:40:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 385 training takes 0:05:56 [2024-03-06 17:40:51 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_385.pth saving...... [2024-03-06 17:40:53 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_385.pth saved !!! [2024-03-06 17:40:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [386/800][0/402] eta 0:30:13 lr 0.000025 time 4.5108 (4.5108) loss 0.6183 (0.6183) grad_norm 0.2212 (0.2212) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 17:42:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [386/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9146) loss 0.5688 (0.5855) grad_norm 0.1838 (0.2209) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 17:43:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [386/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8967) loss 0.5568 (0.5847) grad_norm 0.2129 (0.2186) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 17:45:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [386/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8909) loss 0.5645 (0.5842) grad_norm 0.2210 (0.2171) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 17:46:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [386/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8877) loss 0.5698 (0.5844) grad_norm 0.2564 (0.2184) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 17:46:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 386 training takes 0:05:57 [2024-03-06 17:46:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [387/800][0/402] eta 0:30:54 lr 0.000025 time 4.6136 (4.6136) loss 0.5772 (0.5772) grad_norm 0.2028 (0.2028) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 17:48:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [387/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9153) loss 0.6081 (0.5847) grad_norm 0.2281 (nan) loss_scale 262144.0000 (342604.0396) mem 30609MB [2024-03-06 17:49:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [387/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8970) loss 0.5916 (0.5838) grad_norm 0.1852 (nan) loss_scale 262144.0000 (302574.1692) mem 30609MB [2024-03-06 17:51:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [387/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8909) loss 0.5590 (0.5842) grad_norm 0.2126 (nan) loss_scale 262144.0000 (289142.2193) mem 30609MB [2024-03-06 17:52:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [387/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5942 (0.5841) grad_norm 0.2173 (nan) loss_scale 262144.0000 (282409.4963) mem 30609MB [2024-03-06 17:52:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 387 training takes 0:05:57 [2024-03-06 17:52:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [388/800][0/402] eta 0:30:37 lr 0.000025 time 4.5703 (4.5703) loss 0.5491 (0.5491) grad_norm 0.2435 (0.2435) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:54:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [388/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9150) loss 0.5862 (0.5876) grad_norm 0.2285 (0.2207) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:55:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [388/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5987 (0.5852) grad_norm 0.2269 (0.2201) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:57:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [388/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8908) loss 0.5688 (0.5858) grad_norm 0.2192 (0.2178) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:58:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [388/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8878) loss 0.5827 (0.5849) grad_norm 0.2152 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 17:58:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 388 training takes 0:05:57 [2024-03-06 17:58:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [389/800][0/402] eta 0:30:22 lr 0.000025 time 4.5336 (4.5336) loss 0.5777 (0.5777) grad_norm 0.2277 (0.2277) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:00:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [389/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9146) loss 0.5961 (0.5870) grad_norm 0.2232 (0.2259) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:01:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [389/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8967) loss 0.6026 (0.5864) grad_norm 0.1980 (0.2218) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:03:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [389/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8906) loss 0.6060 (0.5862) grad_norm 0.1975 (0.2249) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:04:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [389/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8875) loss 0.5768 (0.5857) grad_norm 0.1980 (0.2233) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:04:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 389 training takes 0:05:56 [2024-03-06 18:04:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [390/800][0/402] eta 0:30:25 lr 0.000025 time 4.5404 (4.5404) loss 0.5625 (0.5625) grad_norm 0.1821 (0.1821) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:06:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [390/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9147) loss 0.6033 (0.5853) grad_norm 0.1989 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:07:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [390/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8967) loss 0.6357 (0.5858) grad_norm 0.1935 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:09:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [390/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8907) loss 0.5708 (0.5854) grad_norm 0.2148 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:10:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [390/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8876) loss 0.5645 (0.5844) grad_norm 0.1788 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:10:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 390 training takes 0:05:56 [2024-03-06 18:10:38 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_390.pth saving...... [2024-03-06 18:10:40 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_390.pth saved !!! [2024-03-06 18:10:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [391/800][0/402] eta 0:29:42 lr 0.000025 time 4.4333 (4.4333) loss 0.5621 (0.5621) grad_norm 0.1991 (0.1991) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:12:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [391/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9140) loss 0.5837 (0.5852) grad_norm 0.2164 (0.2194) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:13:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [391/800][200/402] eta 0:03:01 lr 0.000025 time 0.8796 (0.8964) loss 0.5594 (0.5830) grad_norm 0.2276 (0.2197) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:15:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [391/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8905) loss 0.5958 (0.5840) grad_norm 0.2051 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:16:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [391/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8875) loss 0.5769 (0.5846) grad_norm 0.2508 (0.2184) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:16:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 391 training takes 0:05:56 [2024-03-06 18:16:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [392/800][0/402] eta 0:30:47 lr 0.000025 time 4.5953 (4.5953) loss 0.5943 (0.5943) grad_norm 0.2131 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:18:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [392/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9153) loss 0.5844 (0.5818) grad_norm 0.2207 (0.2159) loss_scale 524288.0000 (469782.8119) mem 30609MB [2024-03-06 18:19:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [392/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8970) loss 0.5942 (0.5822) grad_norm 0.2628 (0.2183) loss_scale 524288.0000 (496899.8209) mem 30609MB [2024-03-06 18:21:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [392/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8908) loss 0.5670 (0.5824) grad_norm 0.1878 (nan) loss_scale 262144.0000 (493806.1395) mem 30609MB [2024-03-06 18:22:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [392/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8877) loss 0.5759 (0.5823) grad_norm 0.2102 (nan) loss_scale 262144.0000 (436035.0324) mem 30609MB [2024-03-06 18:22:34 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 392 training takes 0:05:57 [2024-03-06 18:22:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [393/800][0/402] eta 0:30:40 lr 0.000025 time 4.5775 (4.5775) loss 0.5730 (0.5730) grad_norm 0.2273 (0.2273) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:24:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [393/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9150) loss 0.5937 (0.5852) grad_norm 0.2545 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:25:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [393/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8969) loss 0.5435 (0.5844) grad_norm 0.1981 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:27:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [393/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8908) loss 0.5683 (0.5846) grad_norm 0.2159 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:28:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [393/800][400/402] eta 0:00:01 lr 0.000025 time 0.8759 (0.8877) loss 0.5955 (0.5844) grad_norm 0.2066 (0.2166) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:28:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 393 training takes 0:05:56 [2024-03-06 18:28:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [394/800][0/402] eta 0:30:20 lr 0.000025 time 4.5277 (4.5277) loss 0.5941 (0.5941) grad_norm 0.1965 (0.1965) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:30:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [394/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9146) loss 0.6223 (0.5834) grad_norm 0.2186 (0.2180) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:31:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [394/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8966) loss 0.5898 (0.5832) grad_norm 0.2020 (0.2208) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:32:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [394/800][300/402] eta 0:01:30 lr 0.000025 time 0.8778 (0.8906) loss 0.5840 (0.5833) grad_norm 0.2086 (0.2190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:34:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [394/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8875) loss 0.5883 (0.5830) grad_norm 0.1957 (0.2198) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:34:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 394 training takes 0:05:56 [2024-03-06 18:34:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [395/800][0/402] eta 0:30:30 lr 0.000025 time 4.5540 (4.5540) loss 0.6094 (0.6094) grad_norm 0.1920 (0.1920) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:36:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [395/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9149) loss 0.5740 (0.5810) grad_norm 0.2104 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:37:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [395/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8968) loss 0.5692 (0.5813) grad_norm 0.2054 (0.2193) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:38:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [395/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8908) loss 0.6046 (0.5821) grad_norm 0.2395 (0.2209) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:40:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [395/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8876) loss 0.5805 (0.5822) grad_norm 0.2279 (0.2215) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:40:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 395 training takes 0:05:56 [2024-03-06 18:40:25 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_395.pth saving...... [2024-03-06 18:40:26 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_395.pth saved !!! [2024-03-06 18:40:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [396/800][0/402] eta 0:30:15 lr 0.000025 time 4.5164 (4.5164) loss 0.5972 (0.5972) grad_norm 0.2181 (0.2181) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:41:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [396/800][100/402] eta 0:04:36 lr 0.000025 time 0.8801 (0.9147) loss 0.5714 (0.5836) grad_norm 0.2473 (0.2163) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:43:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [396/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8968) loss 0.6100 (0.5836) grad_norm 0.2153 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:44:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [396/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8908) loss 0.5869 (0.5845) grad_norm 0.1828 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:46:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [396/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8877) loss 0.6003 (0.5840) grad_norm 0.2065 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:46:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 396 training takes 0:05:57 [2024-03-06 18:46:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [397/800][0/402] eta 0:30:25 lr 0.000025 time 4.5400 (4.5400) loss 0.5789 (0.5789) grad_norm 0.2132 (0.2132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:47:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [397/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9147) loss 0.5655 (0.5823) grad_norm 0.1962 (0.2139) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:49:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [397/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8968) loss 0.5706 (0.5834) grad_norm 0.2090 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 18:50:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [397/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8907) loss 0.5556 (0.5830) grad_norm 0.1975 (0.2160) loss_scale 524288.0000 (283045.8472) mem 30609MB [2024-03-06 18:52:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [397/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8875) loss 0.5637 (0.5829) grad_norm 0.2341 (0.2164) loss_scale 524288.0000 (343205.9850) mem 30609MB [2024-03-06 18:52:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 397 training takes 0:05:56 [2024-03-06 18:52:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [398/800][0/402] eta 0:30:28 lr 0.000025 time 4.5481 (4.5481) loss 0.5878 (0.5878) grad_norm 0.2213 (0.2213) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 18:53:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [398/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9148) loss 0.6057 (0.5809) grad_norm 0.2111 (0.2116) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 18:55:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [398/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8968) loss 0.5711 (0.5846) grad_norm 0.1999 (0.2166) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 18:56:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [398/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8907) loss 0.5883 (0.5851) grad_norm 0.2161 (0.2149) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 18:58:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [398/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8876) loss 0.5743 (0.5847) grad_norm 0.2362 (0.2173) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 18:58:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 398 training takes 0:05:56 [2024-03-06 18:58:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [399/800][0/402] eta 0:30:10 lr 0.000025 time 4.5036 (4.5036) loss 0.5603 (0.5603) grad_norm 0.2200 (0.2200) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 18:59:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [399/800][100/402] eta 0:04:36 lr 0.000025 time 0.8778 (0.9143) loss 0.5778 (0.5827) grad_norm 0.2638 (0.2158) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:01:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [399/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8965) loss 0.5933 (0.5848) grad_norm 0.2280 (0.2196) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:02:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [399/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8906) loss 0.5660 (0.5856) grad_norm 0.1988 (0.2172) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:04:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [399/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8875) loss 0.5805 (0.5846) grad_norm 0.1994 (0.2178) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:04:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 399 training takes 0:05:56 [2024-03-06 19:04:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [400/800][0/402] eta 0:30:24 lr 0.000025 time 4.5389 (4.5389) loss 0.6096 (0.6096) grad_norm 0.2251 (0.2251) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:05:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [400/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9148) loss 0.5711 (0.5857) grad_norm 0.2172 (0.2250) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:07:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [400/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8967) loss 0.5750 (0.5860) grad_norm 0.2036 (inf) loss_scale 262144.0000 (473424.2388) mem 30609MB [2024-03-06 19:08:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [400/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8906) loss 0.5846 (0.5853) grad_norm 0.2111 (inf) loss_scale 262144.0000 (403231.4684) mem 30609MB [2024-03-06 19:10:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [400/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8875) loss 0.6175 (0.5845) grad_norm 0.2437 (inf) loss_scale 262144.0000 (368047.5611) mem 30609MB [2024-03-06 19:10:11 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 400 training takes 0:05:56 [2024-03-06 19:10:11 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_400.pth saving...... [2024-03-06 19:10:13 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_400.pth saved !!! [2024-03-06 19:10:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [401/800][0/402] eta 0:29:25 lr 0.000025 time 4.3927 (4.3927) loss 0.6079 (0.6079) grad_norm 0.1807 (0.1807) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:11:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [401/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9133) loss 0.5927 (0.5818) grad_norm 0.1947 (0.2122) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:13:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [401/800][200/402] eta 0:03:00 lr 0.000025 time 0.8785 (0.8960) loss 0.5664 (0.5830) grad_norm 0.2157 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:14:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [401/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8902) loss 0.5884 (0.5834) grad_norm 0.2099 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:16:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [401/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8873) loss 0.5774 (0.5831) grad_norm 0.1981 (0.2153) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:16:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 401 training takes 0:05:56 [2024-03-06 19:16:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [402/800][0/402] eta 0:30:52 lr 0.000025 time 4.6070 (4.6070) loss 0.5780 (0.5780) grad_norm 0.2326 (0.2326) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:17:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [402/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9154) loss 0.6005 (0.5830) grad_norm 0.2309 (0.2176) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:19:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [402/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8970) loss 0.5628 (0.5829) grad_norm 0.2239 (0.2164) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:20:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [402/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8909) loss 0.5595 (0.5838) grad_norm 0.2216 (0.2178) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:22:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [402/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8878) loss 0.5966 (0.5833) grad_norm 0.2102 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:22:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 402 training takes 0:05:57 [2024-03-06 19:22:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [403/800][0/402] eta 0:30:32 lr 0.000025 time 4.5589 (4.5589) loss 0.5718 (0.5718) grad_norm 0.2089 (0.2089) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:23:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [403/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9149) loss 0.5750 (0.5828) grad_norm 0.2121 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:25:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [403/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8968) loss 0.5899 (0.5819) grad_norm 0.1904 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:26:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [403/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8907) loss 0.6183 (0.5841) grad_norm 0.2073 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:28:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [403/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8876) loss 0.6064 (0.5841) grad_norm 0.1944 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:28:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 403 training takes 0:05:56 [2024-03-06 19:28:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [404/800][0/402] eta 0:30:06 lr 0.000025 time 4.4945 (4.4945) loss 0.6033 (0.6033) grad_norm 0.1912 (0.1912) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:29:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [404/800][100/402] eta 0:04:36 lr 0.000025 time 0.8808 (0.9147) loss 0.5883 (0.5810) grad_norm 0.2061 (0.2112) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:31:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [404/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8967) loss 0.5796 (0.5834) grad_norm 0.2362 (0.2164) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:32:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [404/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8907) loss 0.6027 (0.5835) grad_norm 0.2134 (0.2180) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:34:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [404/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8876) loss 0.5903 (0.5840) grad_norm 0.2623 (0.2178) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:34:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 404 training takes 0:05:56 [2024-03-06 19:34:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [405/800][0/402] eta 0:30:31 lr 0.000025 time 4.5566 (4.5566) loss 0.5749 (0.5749) grad_norm 0.2343 (0.2343) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:35:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [405/800][100/402] eta 0:04:36 lr 0.000025 time 0.8778 (0.9148) loss 0.5821 (0.5864) grad_norm 0.2055 (0.2195) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:37:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [405/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8968) loss 0.5667 (0.5861) grad_norm 0.2000 (0.2209) loss_scale 524288.0000 (326049.7512) mem 30609MB [2024-03-06 19:38:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [405/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8907) loss 0.5850 (0.5854) grad_norm 0.1894 (0.2185) loss_scale 524288.0000 (391909.6346) mem 30609MB [2024-03-06 19:39:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [405/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8876) loss 0.5739 (0.5845) grad_norm 0.2062 (0.2170) loss_scale 524288.0000 (424921.6958) mem 30609MB [2024-03-06 19:39:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 405 training takes 0:05:56 [2024-03-06 19:39:58 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_405.pth saving...... [2024-03-06 19:39:59 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_405.pth saved !!! [2024-03-06 19:40:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [406/800][0/402] eta 0:30:17 lr 0.000025 time 4.5206 (4.5206) loss 0.6021 (0.6021) grad_norm 0.2580 (0.2580) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:41:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [406/800][100/402] eta 0:04:36 lr 0.000025 time 0.8790 (0.9148) loss 0.5877 (0.5838) grad_norm 0.2251 (0.2240) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:43:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [406/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8970) loss 0.5678 (0.5829) grad_norm 0.2454 (0.2249) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:44:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [406/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8910) loss 0.6186 (0.5834) grad_norm 0.2044 (0.2230) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:45:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [406/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8879) loss 0.5790 (0.5824) grad_norm 0.2575 (0.2211) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:45:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 406 training takes 0:05:57 [2024-03-06 19:46:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [407/800][0/402] eta 0:30:57 lr 0.000025 time 4.6196 (4.6196) loss 0.5748 (0.5748) grad_norm 0.2309 (0.2309) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:47:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [407/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9158) loss 0.6059 (0.5827) grad_norm 0.1813 (0.2195) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:48:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [407/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8972) loss 0.5938 (0.5834) grad_norm 0.1903 (0.2149) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:50:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [407/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8910) loss 0.5970 (0.5835) grad_norm 0.2110 (0.2157) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:51:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [407/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8878) loss 0.5714 (0.5827) grad_norm 0.1986 (0.2162) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:51:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 407 training takes 0:05:57 [2024-03-06 19:51:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [408/800][0/402] eta 0:30:54 lr 0.000025 time 4.6142 (4.6142) loss 0.6108 (0.6108) grad_norm 0.2005 (0.2005) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:53:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [408/800][100/402] eta 0:04:36 lr 0.000025 time 0.8801 (0.9154) loss 0.6043 (0.5817) grad_norm 0.2018 (0.2156) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:54:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [408/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8970) loss 0.5469 (0.5826) grad_norm 0.2318 (0.2171) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 19:56:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [408/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8909) loss 0.5934 (0.5831) grad_norm 0.2302 (nan) loss_scale 262144.0000 (519062.5382) mem 30609MB [2024-03-06 19:57:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [408/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8878) loss 0.5986 (0.5825) grad_norm 0.1894 (nan) loss_scale 262144.0000 (454993.0773) mem 30609MB [2024-03-06 19:57:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 408 training takes 0:05:57 [2024-03-06 19:57:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [409/800][0/402] eta 0:30:31 lr 0.000025 time 4.5553 (4.5553) loss 0.5912 (0.5912) grad_norm 0.3373 (0.3373) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 19:59:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [409/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9149) loss 0.5893 (0.5838) grad_norm 0.1767 (0.2200) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:00:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [409/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8969) loss 0.5949 (0.5838) grad_norm 0.2086 (0.2199) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:02:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [409/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8908) loss 0.5665 (0.5832) grad_norm 0.2050 (0.2178) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:03:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [409/800][400/402] eta 0:00:01 lr 0.000025 time 0.8778 (0.8877) loss 0.5946 (0.5832) grad_norm 0.2261 (0.2173) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:03:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 409 training takes 0:05:57 [2024-03-06 20:03:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [410/800][0/402] eta 0:30:25 lr 0.000025 time 4.5423 (4.5423) loss 0.5827 (0.5827) grad_norm 0.2063 (0.2063) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:05:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [410/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9147) loss 0.5854 (0.5797) grad_norm 0.2062 (0.2236) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:06:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [410/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8967) loss 0.5670 (0.5815) grad_norm 0.1744 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:08:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [410/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8906) loss 0.5633 (0.5823) grad_norm 0.2092 (0.2179) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:09:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [410/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8876) loss 0.5961 (0.5836) grad_norm 0.2146 (0.2163) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:09:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 410 training takes 0:05:56 [2024-03-06 20:09:45 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_410.pth saving...... [2024-03-06 20:09:46 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_410.pth saved !!! [2024-03-06 20:09:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [411/800][0/402] eta 0:29:41 lr 0.000025 time 4.4311 (4.4311) loss 0.6049 (0.6049) grad_norm 0.2234 (0.2234) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:11:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [411/800][100/402] eta 0:04:35 lr 0.000025 time 0.8786 (0.9135) loss 0.5788 (0.5843) grad_norm 0.2149 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:12:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [411/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8961) loss 0.5887 (0.5835) grad_norm 0.2469 (0.2177) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:14:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [411/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8902) loss 0.5929 (0.5826) grad_norm 0.2386 (0.2186) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:15:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [411/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8873) loss 0.5746 (0.5838) grad_norm 0.2488 (0.2189) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:15:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 411 training takes 0:05:56 [2024-03-06 20:15:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [412/800][0/402] eta 0:30:36 lr 0.000025 time 4.5690 (4.5690) loss 0.5934 (0.5934) grad_norm 0.2129 (0.2129) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:17:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [412/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9150) loss 0.6052 (0.5846) grad_norm 0.2091 (0.2177) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:18:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [412/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8968) loss 0.6134 (0.5859) grad_norm 0.1854 (0.2197) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:20:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [412/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8907) loss 0.6071 (0.5836) grad_norm 0.1929 (0.2183) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:21:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [412/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8876) loss 0.5532 (0.5829) grad_norm 0.2054 (0.2164) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:21:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 412 training takes 0:05:56 [2024-03-06 20:21:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [413/800][0/402] eta 0:30:21 lr 0.000025 time 4.5304 (4.5304) loss 0.5301 (0.5301) grad_norm 0.2360 (0.2360) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:23:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [413/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9146) loss 0.5827 (0.5847) grad_norm 0.2428 (0.2100) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:24:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [413/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8966) loss 0.5617 (0.5852) grad_norm 0.2099 (0.2121) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:26:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [413/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8906) loss 0.5723 (0.5844) grad_norm 0.2417 (0.2157) loss_scale 524288.0000 (276078.5648) mem 30609MB [2024-03-06 20:27:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [413/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8875) loss 0.5849 (0.5841) grad_norm 0.2082 (0.2150) loss_scale 524288.0000 (337976.1796) mem 30609MB [2024-03-06 20:27:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 413 training takes 0:05:56 [2024-03-06 20:27:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [414/800][0/402] eta 0:30:34 lr 0.000025 time 4.5646 (4.5646) loss 0.6039 (0.6039) grad_norm 0.2126 (0.2126) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 20:29:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [414/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9153) loss 0.5656 (0.5843) grad_norm 0.1984 (0.2286) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 20:30:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [414/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8970) loss 0.5538 (0.5822) grad_norm 0.2063 (0.2224) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 20:32:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [414/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8909) loss 0.6044 (0.5819) grad_norm 0.1911 (0.2192) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 20:33:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [414/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.6382 (0.5833) grad_norm 0.2016 (nan) loss_scale 262144.0000 (496831.5212) mem 30609MB [2024-03-06 20:33:34 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 414 training takes 0:05:57 [2024-03-06 20:33:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [415/800][0/402] eta 0:30:23 lr 0.000025 time 4.5356 (4.5356) loss 0.5635 (0.5635) grad_norm 0.2630 (0.2630) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:35:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [415/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9147) loss 0.5909 (0.5869) grad_norm 0.2096 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:36:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [415/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8967) loss 0.5952 (0.5852) grad_norm 0.2086 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:38:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [415/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8907) loss 0.5639 (0.5838) grad_norm 0.2021 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:39:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [415/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8876) loss 0.5840 (0.5835) grad_norm 0.2289 (0.2158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:39:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 415 training takes 0:05:56 [2024-03-06 20:39:31 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_415.pth saving...... [2024-03-06 20:39:33 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_415.pth saved !!! [2024-03-06 20:39:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [416/800][0/402] eta 0:28:20 lr 0.000025 time 4.2311 (4.2311) loss 0.5782 (0.5782) grad_norm 0.1895 (0.1895) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:41:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [416/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9119) loss 0.5820 (0.5811) grad_norm 0.2209 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:42:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [416/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8956) loss 0.5879 (0.5838) grad_norm 0.2194 (0.2141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:44:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [416/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8900) loss 0.5832 (0.5847) grad_norm 0.2151 (0.2143) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:45:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [416/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8871) loss 0.5771 (0.5841) grad_norm 0.2564 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:45:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 416 training takes 0:05:56 [2024-03-06 20:45:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [417/800][0/402] eta 0:30:30 lr 0.000025 time 4.5524 (4.5524) loss 0.5863 (0.5863) grad_norm 0.1782 (0.1782) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:47:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [417/800][100/402] eta 0:04:36 lr 0.000025 time 0.8779 (0.9151) loss 0.5738 (0.5833) grad_norm 0.2457 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:48:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [417/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8968) loss 0.5883 (0.5833) grad_norm 0.1862 (0.2226) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:49:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [417/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8907) loss 0.6135 (0.5830) grad_norm 0.2039 (0.2199) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:51:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [417/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8876) loss 0.5337 (0.5833) grad_norm 0.2362 (0.2192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:51:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 417 training takes 0:05:56 [2024-03-06 20:51:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [418/800][0/402] eta 0:30:16 lr 0.000025 time 4.5182 (4.5182) loss 0.6166 (0.6166) grad_norm 0.2234 (0.2234) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:52:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [418/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9144) loss 0.5814 (0.5852) grad_norm 0.1755 (0.2298) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:54:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [418/800][200/402] eta 0:03:01 lr 0.000025 time 0.8776 (0.8965) loss 0.5759 (0.5841) grad_norm 0.2061 (0.2237) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:55:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [418/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8906) loss 0.5687 (0.5827) grad_norm 0.2076 (0.2217) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:57:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [418/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8875) loss 0.5970 (0.5830) grad_norm 0.1961 (0.2224) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:57:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 418 training takes 0:05:56 [2024-03-06 20:57:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [419/800][0/402] eta 0:30:16 lr 0.000025 time 4.5182 (4.5182) loss 0.5689 (0.5689) grad_norm 0.1841 (0.1841) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 20:58:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [419/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9145) loss 0.5832 (0.5876) grad_norm 0.1878 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:00:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [419/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8966) loss 0.6030 (0.5855) grad_norm 0.2569 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:01:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [419/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8907) loss 0.5394 (0.5851) grad_norm 0.2203 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:03:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [419/800][400/402] eta 0:00:01 lr 0.000025 time 0.8774 (0.8876) loss 0.6182 (0.5851) grad_norm 0.2261 (0.2144) loss_scale 524288.0000 (296137.7357) mem 30609MB [2024-03-06 21:03:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 419 training takes 0:05:56 [2024-03-06 21:03:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [420/800][0/402] eta 0:29:53 lr 0.000025 time 4.4609 (4.4609) loss 0.5833 (0.5833) grad_norm 0.2166 (0.2166) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:04:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [420/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9139) loss 0.5471 (0.5843) grad_norm 0.2383 (0.2172) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:06:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [420/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8963) loss 0.5722 (0.5834) grad_norm 0.2096 (0.2133) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:07:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [420/800][300/402] eta 0:01:30 lr 0.000025 time 0.8800 (0.8904) loss 0.6129 (0.5821) grad_norm 0.2231 (0.2158) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:09:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [420/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8874) loss 0.6015 (0.5825) grad_norm 0.2191 (0.2169) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:09:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 420 training takes 0:05:56 [2024-03-06 21:09:18 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_420.pth saving...... [2024-03-06 21:09:19 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_420.pth saved !!! [2024-03-06 21:09:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [421/800][0/402] eta 0:29:34 lr 0.000025 time 4.4134 (4.4134) loss 0.5653 (0.5653) grad_norm 0.1898 (0.1898) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:10:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [421/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9136) loss 0.5813 (0.5803) grad_norm 0.1862 (0.2156) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:12:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [421/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8962) loss 0.5736 (0.5809) grad_norm 0.2161 (0.2210) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:13:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [421/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8904) loss 0.5597 (0.5820) grad_norm 0.2107 (0.2188) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:15:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [421/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8874) loss 0.5678 (0.5825) grad_norm 0.2011 (0.2171) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:15:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 421 training takes 0:05:56 [2024-03-06 21:15:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [422/800][0/402] eta 0:30:20 lr 0.000025 time 4.5276 (4.5276) loss 0.5728 (0.5728) grad_norm 0.2197 (0.2197) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:16:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [422/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9147) loss 0.5889 (0.5827) grad_norm 0.2121 (0.2112) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:18:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [422/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8967) loss 0.6176 (0.5829) grad_norm 0.2187 (0.2129) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:19:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [422/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8907) loss 0.6030 (0.5828) grad_norm 0.2130 (0.2140) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:21:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [422/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8876) loss 0.5786 (0.5830) grad_norm 0.2607 (0.2136) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:21:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 422 training takes 0:05:56 [2024-03-06 21:21:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [423/800][0/402] eta 0:30:45 lr 0.000025 time 4.5918 (4.5918) loss 0.5646 (0.5646) grad_norm 0.2152 (0.2152) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:22:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [423/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9151) loss 0.5489 (0.5837) grad_norm 0.2645 (nan) loss_scale 262144.0000 (516501.5446) mem 30609MB [2024-03-06 21:24:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [423/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8971) loss 0.5909 (0.5828) grad_norm 0.2177 (nan) loss_scale 262144.0000 (389955.5025) mem 30609MB [2024-03-06 21:25:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [423/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8909) loss 0.5791 (0.5836) grad_norm 0.2373 (nan) loss_scale 262144.0000 (347493.2093) mem 30609MB [2024-03-06 21:27:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [423/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8878) loss 0.5951 (0.5842) grad_norm 0.2330 (nan) loss_scale 262144.0000 (326209.1172) mem 30609MB [2024-03-06 21:27:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 423 training takes 0:05:57 [2024-03-06 21:27:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [424/800][0/402] eta 0:30:33 lr 0.000025 time 4.5604 (4.5604) loss 0.5795 (0.5795) grad_norm 0.2197 (0.2197) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:28:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [424/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9150) loss 0.5705 (0.5831) grad_norm 0.2238 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:30:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [424/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5679 (0.5820) grad_norm 0.1932 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:31:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [424/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8908) loss 0.5959 (0.5826) grad_norm 0.2131 (0.2190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:33:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [424/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8877) loss 0.5937 (0.5827) grad_norm 0.2260 (0.2194) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:33:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 424 training takes 0:05:56 [2024-03-06 21:33:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [425/800][0/402] eta 0:30:30 lr 0.000025 time 4.5547 (4.5547) loss 0.5991 (0.5991) grad_norm 0.2064 (0.2064) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:34:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [425/800][100/402] eta 0:04:36 lr 0.000025 time 0.8817 (0.9149) loss 0.5898 (0.5848) grad_norm 0.1999 (0.2102) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:36:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [425/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5767 (0.5853) grad_norm 0.2170 (0.2135) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:37:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [425/800][300/402] eta 0:01:30 lr 0.000025 time 0.8802 (0.8908) loss 0.6075 (0.5853) grad_norm 0.2412 (0.2148) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:39:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [425/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8877) loss 0.5590 (0.5840) grad_norm 0.2027 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:39:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 425 training takes 0:05:57 [2024-03-06 21:39:04 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_425.pth saving...... [2024-03-06 21:39:06 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_425.pth saved !!! [2024-03-06 21:39:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [426/800][0/402] eta 0:31:06 lr 0.000025 time 4.6429 (4.6429) loss 0.5665 (0.5665) grad_norm 0.1997 (0.1997) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:40:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [426/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9158) loss 0.5838 (0.5800) grad_norm 0.2032 (0.2180) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:42:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [426/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8972) loss 0.5725 (0.5803) grad_norm 0.2053 (0.2190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:43:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [426/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8910) loss 0.5939 (0.5811) grad_norm 0.2022 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:45:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [426/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8878) loss 0.6160 (0.5814) grad_norm 0.2010 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:45:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 426 training takes 0:05:57 [2024-03-06 21:45:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [427/800][0/402] eta 0:30:31 lr 0.000025 time 4.5561 (4.5561) loss 0.5541 (0.5541) grad_norm 0.2279 (0.2279) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:46:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [427/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9148) loss 0.5901 (0.5803) grad_norm 0.1941 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:48:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [427/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5925 (0.5824) grad_norm 0.2101 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:49:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [427/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8907) loss 0.5923 (0.5821) grad_norm 0.2569 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:50:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [427/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8876) loss 0.5607 (0.5821) grad_norm 0.2335 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:51:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 427 training takes 0:05:56 [2024-03-06 21:51:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [428/800][0/402] eta 0:30:49 lr 0.000025 time 4.6006 (4.6006) loss 0.5929 (0.5929) grad_norm 0.2189 (0.2189) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 21:52:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [428/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9153) loss 0.5544 (0.5849) grad_norm 0.1942 (0.2151) loss_scale 524288.0000 (295885.3069) mem 30609MB [2024-03-06 21:54:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [428/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8970) loss 0.5451 (0.5838) grad_norm 0.2426 (0.2173) loss_scale 524288.0000 (409518.4876) mem 30609MB [2024-03-06 21:55:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [428/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8908) loss 0.6254 (0.5848) grad_norm 0.2053 (0.2178) loss_scale 524288.0000 (447647.8937) mem 30609MB [2024-03-06 21:56:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [428/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5554 (0.5842) grad_norm 0.2865 (0.2184) loss_scale 524288.0000 (466760.1397) mem 30609MB [2024-03-06 21:56:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 428 training takes 0:05:57 [2024-03-06 21:57:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [429/800][0/402] eta 0:30:45 lr 0.000025 time 4.5896 (4.5896) loss 0.5864 (0.5864) grad_norm 0.2092 (0.2092) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:58:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [429/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9151) loss 0.5963 (0.5826) grad_norm 0.2066 (0.2115) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 21:59:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [429/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8969) loss 0.5943 (0.5807) grad_norm 0.2378 (0.2144) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:01:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [429/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8908) loss 0.5922 (0.5816) grad_norm 0.1977 (0.2137) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:02:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [429/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8877) loss 0.6213 (0.5825) grad_norm 0.2344 (nan) loss_scale 262144.0000 (495524.0698) mem 30609MB [2024-03-06 22:02:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 429 training takes 0:05:56 [2024-03-06 22:02:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [430/800][0/402] eta 0:30:35 lr 0.000025 time 4.5653 (4.5653) loss 0.5718 (0.5718) grad_norm 0.2412 (0.2412) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:04:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [430/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9155) loss 0.5837 (0.5837) grad_norm 0.2367 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:05:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [430/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8971) loss 0.5432 (0.5839) grad_norm 0.1907 (0.2101) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:07:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [430/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8910) loss 0.5748 (0.5836) grad_norm 0.2201 (0.2111) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:08:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [430/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8878) loss 0.5876 (0.5836) grad_norm 0.2475 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:08:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 430 training takes 0:05:57 [2024-03-06 22:08:51 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_430.pth saving...... [2024-03-06 22:08:53 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_430.pth saved !!! [2024-03-06 22:08:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [431/800][0/402] eta 0:29:33 lr 0.000025 time 4.4112 (4.4112) loss 0.5621 (0.5621) grad_norm 0.2319 (0.2319) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:10:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [431/800][100/402] eta 0:04:35 lr 0.000025 time 0.8788 (0.9136) loss 0.5845 (0.5787) grad_norm 0.2233 (0.2181) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:11:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [431/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8961) loss 0.5517 (0.5805) grad_norm 0.2167 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:13:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [431/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8903) loss 0.5949 (0.5812) grad_norm 0.2228 (0.2147) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:14:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [431/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8873) loss 0.6037 (0.5824) grad_norm 0.2312 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:14:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 431 training takes 0:05:56 [2024-03-06 22:14:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [432/800][0/402] eta 0:30:21 lr 0.000025 time 4.5305 (4.5305) loss 0.5717 (0.5717) grad_norm 0.2304 (0.2304) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:16:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [432/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9146) loss 0.5813 (0.5829) grad_norm 0.2280 (0.2136) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:17:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [432/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8967) loss 0.6177 (0.5835) grad_norm 0.1872 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:19:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [432/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8908) loss 0.5950 (0.5835) grad_norm 0.2121 (0.2149) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:20:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [432/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8876) loss 0.6178 (0.5831) grad_norm 0.1992 (0.2143) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:20:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 432 training takes 0:05:56 [2024-03-06 22:20:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [433/800][0/402] eta 0:30:15 lr 0.000025 time 4.5164 (4.5164) loss 0.5545 (0.5545) grad_norm 0.2063 (0.2063) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:22:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [433/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9145) loss 0.5779 (0.5850) grad_norm 0.2067 (0.2090) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:23:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [433/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8966) loss 0.5761 (0.5852) grad_norm 0.1978 (0.2105) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:25:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [433/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8906) loss 0.5710 (0.5840) grad_norm 0.2150 (0.2115) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:26:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [433/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8875) loss 0.5823 (0.5837) grad_norm 0.2540 (0.2135) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:26:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 433 training takes 0:05:56 [2024-03-06 22:26:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [434/800][0/402] eta 0:30:24 lr 0.000025 time 4.5397 (4.5397) loss 0.5841 (0.5841) grad_norm 0.2146 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:28:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [434/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9147) loss 0.5518 (0.5804) grad_norm 0.2198 (0.2231) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:29:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [434/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5829 (0.5807) grad_norm 0.2427 (0.2183) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:31:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [434/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8907) loss 0.5701 (0.5815) grad_norm 0.1991 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:32:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [434/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8877) loss 0.6056 (0.5814) grad_norm 0.2143 (0.2162) loss_scale 524288.0000 (297445.1870) mem 30609MB [2024-03-06 22:32:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 434 training takes 0:05:56 [2024-03-06 22:32:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [435/800][0/402] eta 0:30:27 lr 0.000025 time 4.5454 (4.5454) loss 0.5837 (0.5837) grad_norm 0.1932 (0.1932) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:34:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [435/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9151) loss 0.5901 (0.5823) grad_norm 0.1885 (0.2135) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:35:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [435/800][200/402] eta 0:03:01 lr 0.000025 time 0.8777 (0.8969) loss 0.5904 (0.5822) grad_norm 0.1798 (0.2159) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:37:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [435/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8909) loss 0.6147 (0.5825) grad_norm 0.1909 (0.2171) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:38:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [435/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5926 (0.5822) grad_norm 0.2050 (0.2160) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:38:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 435 training takes 0:05:57 [2024-03-06 22:38:38 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_435.pth saving...... [2024-03-06 22:38:39 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_435.pth saved !!! [2024-03-06 22:38:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [436/800][0/402] eta 0:31:04 lr 0.000025 time 4.6390 (4.6390) loss 0.5572 (0.5572) grad_norm 0.1872 (0.1872) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:40:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [436/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9159) loss 0.5681 (0.5817) grad_norm 0.1992 (0.2156) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:41:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [436/800][200/402] eta 0:03:01 lr 0.000025 time 0.8798 (0.8975) loss 0.5718 (0.5810) grad_norm 0.2589 (0.2148) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:43:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [436/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8913) loss 0.5979 (0.5817) grad_norm 0.2208 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 22:44:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [436/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8881) loss 0.5627 (0.5824) grad_norm 0.2245 (inf) loss_scale 262144.0000 (512520.9377) mem 30609MB [2024-03-06 22:44:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 436 training takes 0:05:57 [2024-03-06 22:44:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [437/800][0/402] eta 0:30:34 lr 0.000025 time 4.5644 (4.5644) loss 0.5906 (0.5906) grad_norm 0.2067 (0.2067) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:46:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [437/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9157) loss 0.6056 (0.5824) grad_norm 0.2287 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:47:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [437/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8972) loss 0.5692 (0.5820) grad_norm 0.1939 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:49:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [437/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8910) loss 0.5972 (0.5821) grad_norm 0.2172 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:50:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [437/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8878) loss 0.5819 (0.5818) grad_norm 0.2435 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:50:34 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 437 training takes 0:05:57 [2024-03-06 22:50:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [438/800][0/402] eta 0:30:45 lr 0.000025 time 4.5912 (4.5912) loss 0.6027 (0.6027) grad_norm 0.2225 (0.2225) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:52:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [438/800][100/402] eta 0:04:36 lr 0.000025 time 0.8798 (0.9152) loss 0.5759 (0.5830) grad_norm 0.2396 (0.2125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:53:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [438/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8970) loss 0.5685 (0.5833) grad_norm 0.2359 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:55:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [438/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8909) loss 0.5995 (0.5836) grad_norm 0.1933 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:56:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [438/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5702 (0.5832) grad_norm 0.2139 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:56:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 438 training takes 0:05:57 [2024-03-06 22:56:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [439/800][0/402] eta 0:30:15 lr 0.000025 time 4.5159 (4.5159) loss 0.5803 (0.5803) grad_norm 0.2511 (0.2511) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:58:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [439/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9145) loss 0.5990 (0.5808) grad_norm 0.1847 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 22:59:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [439/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8969) loss 0.5849 (0.5812) grad_norm 0.2221 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:00:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [439/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8908) loss 0.6430 (0.5820) grad_norm 0.2009 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:02:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [439/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5806 (0.5828) grad_norm 0.2041 (0.2141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:02:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 439 training takes 0:05:56 [2024-03-06 23:02:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [440/800][0/402] eta 0:29:29 lr 0.000025 time 4.4027 (4.4027) loss 0.5758 (0.5758) grad_norm 0.2177 (0.2177) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:04:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [440/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9134) loss 0.6010 (0.5867) grad_norm 0.1885 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:05:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [440/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8961) loss 0.6059 (0.5853) grad_norm 0.2094 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:06:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [440/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8903) loss 0.5738 (0.5829) grad_norm 0.1882 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:08:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [440/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8873) loss 0.5894 (0.5829) grad_norm 0.2154 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:08:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 440 training takes 0:05:56 [2024-03-06 23:08:25 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_440.pth saving...... [2024-03-06 23:08:26 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_440.pth saved !!! [2024-03-06 23:08:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [441/800][0/402] eta 0:29:40 lr 0.000025 time 4.4292 (4.4292) loss 0.5840 (0.5840) grad_norm 0.2095 (0.2095) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:09:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [441/800][100/402] eta 0:04:35 lr 0.000025 time 0.8801 (0.9138) loss 0.5908 (0.5840) grad_norm 0.2060 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:11:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [441/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8964) loss 0.5446 (0.5825) grad_norm 0.2315 (0.2181) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:12:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [441/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8905) loss 0.5694 (0.5822) grad_norm 0.2103 (0.2153) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:14:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [441/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8875) loss 0.5887 (0.5815) grad_norm 0.2313 (0.2174) loss_scale 524288.0000 (280448.3192) mem 30609MB [2024-03-06 23:14:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 441 training takes 0:05:56 [2024-03-06 23:14:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [442/800][0/402] eta 0:30:43 lr 0.000025 time 4.5850 (4.5850) loss 0.6184 (0.6184) grad_norm 0.1921 (0.1921) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:15:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [442/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9152) loss 0.5896 (0.5787) grad_norm 0.2066 (0.2182) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:17:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [442/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8969) loss 0.5498 (0.5825) grad_norm 0.2032 (0.2147) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:18:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [442/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8908) loss 0.5871 (0.5819) grad_norm 0.2185 (0.2153) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:20:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [442/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8877) loss 0.6081 (0.5822) grad_norm 0.1926 (inf) loss_scale 262144.0000 (477219.7506) mem 30609MB [2024-03-06 23:20:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 442 training takes 0:05:57 [2024-03-06 23:20:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [443/800][0/402] eta 0:30:55 lr 0.000025 time 4.6148 (4.6148) loss 0.5898 (0.5898) grad_norm 0.2034 (0.2034) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:21:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [443/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9154) loss 0.5870 (0.5840) grad_norm 0.2002 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:23:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [443/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8971) loss 0.6148 (0.5844) grad_norm 0.1956 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:24:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [443/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8909) loss 0.5643 (0.5829) grad_norm 0.1931 (0.2148) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:26:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [443/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8879) loss 0.6042 (0.5823) grad_norm 0.1986 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:26:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 443 training takes 0:05:57 [2024-03-06 23:26:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [444/800][0/402] eta 0:30:25 lr 0.000025 time 4.5419 (4.5419) loss 0.5571 (0.5571) grad_norm 0.2995 (0.2995) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:27:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [444/800][100/402] eta 0:04:36 lr 0.000025 time 0.8789 (0.9147) loss 0.6048 (0.5832) grad_norm 0.2275 (0.2134) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:29:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [444/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8967) loss 0.5600 (0.5829) grad_norm 0.2144 (0.2179) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:30:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [444/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8907) loss 0.6063 (0.5819) grad_norm 0.1836 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:32:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [444/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8876) loss 0.5724 (0.5818) grad_norm 0.2154 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:32:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 444 training takes 0:05:56 [2024-03-06 23:32:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [445/800][0/402] eta 0:30:23 lr 0.000025 time 4.5358 (4.5358) loss 0.5759 (0.5759) grad_norm 0.2787 (0.2787) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:33:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [445/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9146) loss 0.5663 (0.5818) grad_norm 0.1911 (0.2109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:35:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [445/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8966) loss 0.6013 (0.5820) grad_norm 0.1906 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:36:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [445/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8906) loss 0.5801 (0.5821) grad_norm 0.2375 (0.2181) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:38:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [445/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8875) loss 0.5943 (0.5822) grad_norm 0.2064 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:38:11 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 445 training takes 0:05:56 [2024-03-06 23:38:11 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_445.pth saving...... [2024-03-06 23:38:13 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_445.pth saved !!! [2024-03-06 23:38:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [446/800][0/402] eta 0:30:42 lr 0.000025 time 4.5834 (4.5834) loss 0.6147 (0.6147) grad_norm 0.2109 (0.2109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:39:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [446/800][100/402] eta 0:04:36 lr 0.000025 time 0.8791 (0.9156) loss 0.5483 (0.5824) grad_norm 0.2100 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:41:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [446/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8972) loss 0.5551 (0.5828) grad_norm 0.1940 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:42:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [446/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8910) loss 0.5732 (0.5818) grad_norm 0.2238 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:44:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [446/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8878) loss 0.5541 (0.5818) grad_norm 0.2274 (0.2147) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:44:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 446 training takes 0:05:57 [2024-03-06 23:44:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [447/800][0/402] eta 0:30:25 lr 0.000025 time 4.5420 (4.5420) loss 0.5782 (0.5782) grad_norm 0.2511 (0.2511) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:45:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [447/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9147) loss 0.5837 (0.5790) grad_norm 0.2418 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:47:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [447/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8967) loss 0.6022 (0.5812) grad_norm 0.2366 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:48:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [447/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8907) loss 0.5890 (0.5811) grad_norm 0.1921 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-06 23:50:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [447/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8876) loss 0.5935 (0.5815) grad_norm 0.2386 (0.2160) loss_scale 524288.0000 (315749.5062) mem 30609MB [2024-03-06 23:50:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 447 training takes 0:05:56 [2024-03-06 23:50:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [448/800][0/402] eta 0:30:22 lr 0.000025 time 4.5333 (4.5333) loss 0.5709 (0.5709) grad_norm 0.2090 (0.2090) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:51:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [448/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9147) loss 0.5829 (0.5793) grad_norm 0.2427 (0.2197) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:53:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [448/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8967) loss 0.5554 (0.5800) grad_norm 0.2540 (0.2207) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:54:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [448/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8908) loss 0.5776 (0.5815) grad_norm 0.2207 (0.2169) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:56:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [448/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.5999 (0.5817) grad_norm 0.2042 (0.2169) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:56:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 448 training takes 0:05:57 [2024-03-06 23:56:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [449/800][0/402] eta 0:30:11 lr 0.000025 time 4.5057 (4.5057) loss 0.5753 (0.5753) grad_norm 0.2062 (0.2062) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:57:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [449/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9144) loss 0.5872 (0.5822) grad_norm 0.2545 (0.2149) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-06 23:59:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [449/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8966) loss 0.5779 (0.5819) grad_norm 0.1966 (0.2147) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:00:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [449/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8906) loss 0.5824 (0.5821) grad_norm 0.1942 (0.2134) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:02:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [449/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8875) loss 0.5761 (0.5827) grad_norm 0.2052 (0.2127) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:02:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 449 training takes 0:05:56 [2024-03-07 00:02:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [450/800][0/402] eta 0:30:30 lr 0.000025 time 4.5542 (4.5542) loss 0.5424 (0.5424) grad_norm 0.2228 (0.2228) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:03:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [450/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9148) loss 0.5442 (0.5838) grad_norm 0.2126 (0.2128) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:05:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [450/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5989 (0.5822) grad_norm 0.2002 (0.2154) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:06:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [450/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8908) loss 0.5729 (0.5807) grad_norm 0.1895 (0.2133) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:07:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [450/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8878) loss 0.5908 (0.5818) grad_norm 0.1717 (0.2141) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:07:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 450 training takes 0:05:57 [2024-03-07 00:07:58 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_450.pth saving...... [2024-03-07 00:08:00 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_450.pth saved !!! [2024-03-07 00:08:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [451/800][0/402] eta 0:30:49 lr 0.000025 time 4.6004 (4.6004) loss 0.5834 (0.5834) grad_norm 0.1896 (0.1896) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:09:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [451/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9154) loss 0.5950 (0.5808) grad_norm 0.1938 (0.2149) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:11:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [451/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8973) loss 0.6009 (0.5828) grad_norm 0.1974 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:12:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [451/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8911) loss 0.5968 (0.5826) grad_norm 0.1923 (0.2141) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:13:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [451/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8880) loss 0.5893 (0.5826) grad_norm 0.2023 (0.2142) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:13:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 451 training takes 0:05:57 [2024-03-07 00:14:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [452/800][0/402] eta 0:30:28 lr 0.000025 time 4.5474 (4.5474) loss 0.5959 (0.5959) grad_norm 0.2001 (0.2001) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:15:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [452/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9148) loss 0.5698 (0.5830) grad_norm 0.2964 (0.2114) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:16:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [452/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8968) loss 0.6077 (0.5836) grad_norm 0.2117 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:18:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [452/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8908) loss 0.5920 (0.5828) grad_norm 0.2158 (0.2156) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:19:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [452/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8877) loss 0.5450 (0.5824) grad_norm 0.1979 (0.2169) loss_scale 1048576.0000 (644573.5262) mem 30609MB [2024-03-07 00:19:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 452 training takes 0:05:56 [2024-03-07 00:19:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [453/800][0/402] eta 0:30:28 lr 0.000025 time 4.5480 (4.5480) loss 0.5835 (0.5835) grad_norm 0.1928 (0.1928) loss_scale 1048576.0000 (1048576.0000) mem 30609MB [2024-03-07 00:21:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [453/800][100/402] eta 0:04:36 lr 0.000025 time 0.8779 (0.9151) loss 0.6006 (0.5818) grad_norm 0.2121 (0.2124) loss_scale 1048576.0000 (1048576.0000) mem 30609MB [2024-03-07 00:22:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [453/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8970) loss 0.5807 (0.5820) grad_norm 0.2261 (0.2116) loss_scale 1048576.0000 (1048576.0000) mem 30609MB [2024-03-07 00:24:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [453/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8908) loss 0.5857 (0.5816) grad_norm 0.2136 (nan) loss_scale 524288.0000 (984128.6379) mem 30609MB [2024-03-07 00:25:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [453/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5676 (0.5820) grad_norm 0.1954 (nan) loss_scale 524288.0000 (869455.1621) mem 30609MB [2024-03-07 00:25:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 453 training takes 0:05:57 [2024-03-07 00:25:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [454/800][0/402] eta 0:30:23 lr 0.000025 time 4.5368 (4.5368) loss 0.5997 (0.5997) grad_norm 0.2128 (0.2128) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:27:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [454/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9147) loss 0.5862 (0.5834) grad_norm 0.2359 (0.2166) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:28:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [454/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8967) loss 0.5844 (0.5833) grad_norm 0.2048 (0.2150) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:30:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [454/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8907) loss 0.5781 (0.5835) grad_norm 0.2131 (0.2165) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:31:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [454/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8876) loss 0.5947 (0.5821) grad_norm 0.2019 (0.2165) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:31:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 454 training takes 0:05:56 [2024-03-07 00:31:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [455/800][0/402] eta 0:30:29 lr 0.000025 time 4.5511 (4.5511) loss 0.5569 (0.5569) grad_norm 0.2314 (0.2314) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:33:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [455/800][100/402] eta 0:04:36 lr 0.000025 time 0.8789 (0.9148) loss 0.5833 (0.5811) grad_norm 0.2376 (0.2142) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:34:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [455/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8969) loss 0.5517 (0.5814) grad_norm 0.2356 (0.2146) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:36:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [455/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8908) loss 0.5655 (0.5814) grad_norm 0.1757 (0.2138) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 00:37:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [455/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8877) loss 0.5767 (0.5810) grad_norm 0.2027 (nan) loss_scale 262144.0000 (472643.6708) mem 30609MB [2024-03-07 00:37:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 455 training takes 0:05:57 [2024-03-07 00:37:45 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_455.pth saving...... [2024-03-07 00:37:47 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_455.pth saved !!! [2024-03-07 00:37:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [456/800][0/402] eta 0:29:37 lr 0.000025 time 4.4208 (4.4208) loss 0.5804 (0.5804) grad_norm 0.2206 (0.2206) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:39:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [456/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9136) loss 0.5685 (0.5814) grad_norm 0.2703 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:40:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [456/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8962) loss 0.5598 (0.5820) grad_norm 0.1921 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:42:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [456/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8904) loss 0.5589 (0.5814) grad_norm 0.2480 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:43:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [456/800][400/402] eta 0:00:01 lr 0.000025 time 0.8774 (0.8874) loss 0.6056 (0.5816) grad_norm 0.2058 (0.2169) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:43:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 456 training takes 0:05:56 [2024-03-07 00:43:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [457/800][0/402] eta 0:29:49 lr 0.000025 time 4.4516 (4.4516) loss 0.5817 (0.5817) grad_norm 0.2261 (0.2261) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:45:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [457/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9143) loss 0.5836 (0.5798) grad_norm 0.1718 (0.2057) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:46:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [457/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8965) loss 0.6088 (0.5807) grad_norm 0.2348 (0.2071) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:48:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [457/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8905) loss 0.5864 (0.5810) grad_norm 0.1932 (0.2106) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:49:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [457/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8875) loss 0.5520 (0.5813) grad_norm 0.2281 (0.2110) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:49:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 457 training takes 0:05:56 [2024-03-07 00:49:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [458/800][0/402] eta 0:30:21 lr 0.000025 time 4.5319 (4.5319) loss 0.6151 (0.6151) grad_norm 0.2027 (0.2027) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:51:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [458/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9147) loss 0.5881 (0.5841) grad_norm 0.2027 (0.2102) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:52:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [458/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8967) loss 0.5875 (0.5817) grad_norm 0.2097 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:54:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [458/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8907) loss 0.5860 (0.5816) grad_norm 0.2171 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:55:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [458/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8876) loss 0.5643 (0.5817) grad_norm 0.1934 (0.2142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:55:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 458 training takes 0:05:56 [2024-03-07 00:55:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [459/800][0/402] eta 0:29:59 lr 0.000025 time 4.4753 (4.4753) loss 0.5799 (0.5799) grad_norm 0.1973 (0.1973) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:57:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [459/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9141) loss 0.5617 (0.5831) grad_norm 0.2281 (0.2110) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 00:58:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [459/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8968) loss 0.6012 (0.5829) grad_norm 0.1897 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:00:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [459/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8907) loss 0.5606 (0.5829) grad_norm 0.2191 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:01:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [459/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8876) loss 0.5848 (0.5829) grad_norm 0.2125 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:01:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 459 training takes 0:05:56 [2024-03-07 01:01:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [460/800][0/402] eta 0:29:22 lr 0.000025 time 4.3846 (4.3846) loss 0.5784 (0.5784) grad_norm 0.2224 (0.2224) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:03:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [460/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9136) loss 0.5441 (0.5834) grad_norm 0.1972 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:04:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [460/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8962) loss 0.5922 (0.5831) grad_norm 0.2147 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:06:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [460/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8903) loss 0.5859 (0.5833) grad_norm 0.2677 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:07:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [460/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8873) loss 0.5784 (0.5831) grad_norm 0.1960 (0.2167) loss_scale 524288.0000 (320325.5860) mem 30609MB [2024-03-07 01:07:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 460 training takes 0:05:56 [2024-03-07 01:07:31 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_460.pth saving...... [2024-03-07 01:07:33 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_460.pth saved !!! [2024-03-07 01:07:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [461/800][0/402] eta 0:29:43 lr 0.000025 time 4.4366 (4.4366) loss 0.5513 (0.5513) grad_norm 0.2235 (0.2235) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:09:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [461/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9140) loss 0.5983 (0.5842) grad_norm 0.2293 (0.2115) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:10:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [461/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8965) loss 0.6088 (0.5832) grad_norm 0.2767 (0.2117) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:12:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [461/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8907) loss 0.6080 (0.5837) grad_norm 0.2226 (0.2150) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:13:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [461/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.6200 (0.5830) grad_norm 0.1913 (0.2139) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:13:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 461 training takes 0:05:57 [2024-03-07 01:13:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [462/800][0/402] eta 0:30:20 lr 0.000025 time 4.5276 (4.5276) loss 0.5566 (0.5566) grad_norm 0.2272 (0.2272) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:15:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [462/800][100/402] eta 0:04:36 lr 0.000025 time 0.8794 (0.9145) loss 0.6102 (0.5806) grad_norm 0.2160 (nan) loss_scale 262144.0000 (275121.4257) mem 30609MB [2024-03-07 01:16:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [462/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8968) loss 0.5962 (0.5800) grad_norm 0.1870 (nan) loss_scale 262144.0000 (268664.9950) mem 30609MB [2024-03-07 01:17:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [462/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8907) loss 0.5824 (0.5800) grad_norm 0.2253 (nan) loss_scale 262144.0000 (266498.5515) mem 30609MB [2024-03-07 01:19:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [462/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8876) loss 0.5725 (0.5810) grad_norm 0.1923 (nan) loss_scale 262144.0000 (265412.6284) mem 30609MB [2024-03-07 01:19:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 462 training takes 0:05:57 [2024-03-07 01:19:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [463/800][0/402] eta 0:30:25 lr 0.000025 time 4.5422 (4.5422) loss 0.6158 (0.6158) grad_norm 0.2081 (0.2081) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:21:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [463/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9147) loss 0.5612 (0.5810) grad_norm 0.2246 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:22:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [463/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8967) loss 0.5399 (0.5821) grad_norm 0.2434 (0.2099) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:23:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [463/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8907) loss 0.6104 (0.5817) grad_norm 0.2102 (0.2099) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:25:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [463/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8878) loss 0.6347 (0.5815) grad_norm 0.1769 (0.2101) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:25:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 463 training takes 0:05:57 [2024-03-07 01:25:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [464/800][0/402] eta 0:30:03 lr 0.000025 time 4.4852 (4.4852) loss 0.5624 (0.5624) grad_norm 0.2573 (0.2573) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:26:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [464/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9142) loss 0.5581 (0.5824) grad_norm 0.1978 (0.2222) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:28:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [464/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8964) loss 0.5864 (0.5818) grad_norm 0.2067 (0.2193) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:29:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [464/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8906) loss 0.5573 (0.5823) grad_norm 0.2037 (0.2180) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:31:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [464/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8875) loss 0.6018 (0.5822) grad_norm 0.2118 (0.2169) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:31:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 464 training takes 0:05:56 [2024-03-07 01:31:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [465/800][0/402] eta 0:30:34 lr 0.000025 time 4.5634 (4.5634) loss 0.5179 (0.5179) grad_norm 0.2156 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:32:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [465/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9150) loss 0.6173 (0.5804) grad_norm 0.1921 (0.2149) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:34:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [465/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8969) loss 0.5859 (0.5783) grad_norm 0.2398 (0.2136) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:35:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [465/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8908) loss 0.6003 (0.5787) grad_norm 0.1986 (0.2130) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:37:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [465/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8877) loss 0.5528 (0.5791) grad_norm 0.1770 (0.2124) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:37:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 465 training takes 0:05:57 [2024-03-07 01:37:18 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_465.pth saving...... [2024-03-07 01:37:20 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_465.pth saved !!! [2024-03-07 01:37:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [466/800][0/402] eta 0:31:32 lr 0.000025 time 4.7083 (4.7083) loss 0.6183 (0.6183) grad_norm 0.2118 (0.2118) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:38:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [466/800][100/402] eta 0:04:37 lr 0.000025 time 0.8784 (0.9175) loss 0.5871 (0.5847) grad_norm 0.1988 (0.2116) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:40:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [466/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8984) loss 0.5637 (0.5835) grad_norm 0.2405 (0.2118) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:41:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [466/800][300/402] eta 0:01:30 lr 0.000025 time 0.8799 (0.8920) loss 0.6161 (0.5818) grad_norm 0.1839 (0.2140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 01:43:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [466/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8886) loss 0.5573 (0.5818) grad_norm 0.1982 (0.2143) loss_scale 524288.0000 (264758.9027) mem 30609MB [2024-03-07 01:43:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 466 training takes 0:05:57 [2024-03-07 01:43:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [467/800][0/402] eta 0:30:34 lr 0.000025 time 4.5645 (4.5645) loss 0.5963 (0.5963) grad_norm 0.2287 (0.2287) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:44:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [467/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9155) loss 0.6204 (0.5825) grad_norm 0.1796 (0.2095) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:46:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [467/800][200/402] eta 0:03:01 lr 0.000025 time 0.8814 (0.8973) loss 0.5384 (0.5819) grad_norm 0.1940 (0.2113) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:47:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [467/800][300/402] eta 0:01:30 lr 0.000025 time 0.8802 (0.8913) loss 0.5937 (0.5811) grad_norm 0.2071 (0.2121) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:49:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [467/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8882) loss 0.6377 (0.5817) grad_norm 0.2359 (0.2124) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:49:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 467 training takes 0:05:57 [2024-03-07 01:49:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [468/800][0/402] eta 0:30:50 lr 0.000025 time 4.6042 (4.6042) loss 0.6106 (0.6106) grad_norm 0.2063 (0.2063) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:50:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [468/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9154) loss 0.5687 (0.5859) grad_norm 0.2593 (0.2203) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:52:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [468/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8976) loss 0.5831 (0.5839) grad_norm 0.1860 (0.2205) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:53:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [468/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8913) loss 0.5508 (0.5837) grad_norm 0.2448 (0.2164) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:55:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [468/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8881) loss 0.6006 (0.5826) grad_norm 0.1890 (0.2138) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:55:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 468 training takes 0:05:57 [2024-03-07 01:55:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [469/800][0/402] eta 0:30:56 lr 0.000025 time 4.6190 (4.6190) loss 0.5817 (0.5817) grad_norm 0.2612 (0.2612) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:56:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [469/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9155) loss 0.5730 (0.5788) grad_norm 0.2214 (0.2122) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:58:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [469/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8973) loss 0.5843 (0.5813) grad_norm 0.2305 (0.2153) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 01:59:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [469/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8911) loss 0.5737 (0.5825) grad_norm 0.2072 (0.2138) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:01:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [469/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8879) loss 0.5860 (0.5821) grad_norm 0.2254 (0.2144) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:01:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 469 training takes 0:05:57 [2024-03-07 02:01:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [470/800][0/402] eta 0:30:49 lr 0.000025 time 4.6003 (4.6003) loss 0.5814 (0.5814) grad_norm 0.1951 (0.1951) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:02:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [470/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9153) loss 0.5774 (0.5814) grad_norm 0.2001 (0.2266) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:04:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [470/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8971) loss 0.5809 (0.5825) grad_norm 0.1973 (0.2187) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:05:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [470/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8913) loss 0.5996 (0.5815) grad_norm 0.1823 (0.2192) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:07:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [470/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8881) loss 0.6017 (0.5819) grad_norm 0.2006 (0.2163) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:07:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 470 training takes 0:05:57 [2024-03-07 02:07:06 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_470.pth saving...... [2024-03-07 02:07:08 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_470.pth saved !!! [2024-03-07 02:07:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [471/800][0/402] eta 0:29:35 lr 0.000025 time 4.4174 (4.4174) loss 0.5614 (0.5614) grad_norm 0.2076 (0.2076) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:08:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [471/800][100/402] eta 0:04:35 lr 0.000025 time 0.8785 (0.9137) loss 0.5468 (0.5809) grad_norm 0.2306 (inf) loss_scale 262144.0000 (454209.9010) mem 30609MB [2024-03-07 02:10:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [471/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8965) loss 0.5615 (0.5824) grad_norm 0.2182 (inf) loss_scale 262144.0000 (358654.7264) mem 30609MB [2024-03-07 02:11:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [471/800][300/402] eta 0:01:30 lr 0.000025 time 0.8835 (0.8909) loss 0.6251 (0.5811) grad_norm 0.2432 (inf) loss_scale 262144.0000 (326591.3621) mem 30609MB [2024-03-07 02:13:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [471/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8880) loss 0.5927 (0.5813) grad_norm 0.2252 (inf) loss_scale 262144.0000 (310519.7007) mem 30609MB [2024-03-07 02:13:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 471 training takes 0:05:57 [2024-03-07 02:13:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [472/800][0/402] eta 0:32:42 lr 0.000025 time 4.8821 (4.8821) loss 0.6131 (0.6131) grad_norm 0.2093 (0.2093) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:14:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [472/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9184) loss 0.5991 (0.5796) grad_norm 0.1869 (0.2163) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:16:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [472/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8987) loss 0.5768 (0.5820) grad_norm 0.2066 (0.2147) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:17:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [472/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8924) loss 0.5833 (0.5807) grad_norm 0.2123 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:19:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [472/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8889) loss 0.5669 (0.5803) grad_norm 0.2363 (0.2140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:19:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 472 training takes 0:05:57 [2024-03-07 02:19:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [473/800][0/402] eta 0:32:15 lr 0.000025 time 4.8149 (4.8149) loss 0.5753 (0.5753) grad_norm 0.2157 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:20:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [473/800][100/402] eta 0:04:37 lr 0.000025 time 0.8780 (0.9177) loss 0.5487 (0.5798) grad_norm 0.2016 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:22:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [473/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8984) loss 0.5557 (0.5803) grad_norm 0.2648 (0.2166) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:23:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [473/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8920) loss 0.5900 (0.5801) grad_norm 0.2405 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:24:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [473/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8888) loss 0.5846 (0.5812) grad_norm 0.2096 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:25:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 473 training takes 0:05:57 [2024-03-07 02:25:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [474/800][0/402] eta 0:32:23 lr 0.000025 time 4.8345 (4.8345) loss 0.5714 (0.5714) grad_norm 0.2595 (0.2595) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:26:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [474/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9179) loss 0.5693 (0.5808) grad_norm 0.2093 (0.2124) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:28:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [474/800][200/402] eta 0:03:01 lr 0.000025 time 0.8797 (0.8984) loss 0.5633 (0.5808) grad_norm 0.2366 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:29:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [474/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8919) loss 0.5985 (0.5809) grad_norm 0.2046 (0.2173) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:30:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [474/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8886) loss 0.5959 (0.5810) grad_norm 0.2127 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:30:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 474 training takes 0:05:57 [2024-03-07 02:31:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [475/800][0/402] eta 0:32:11 lr 0.000025 time 4.8043 (4.8043) loss 0.6126 (0.6126) grad_norm 0.2115 (0.2115) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:32:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [475/800][100/402] eta 0:04:37 lr 0.000025 time 0.8784 (0.9186) loss 0.5804 (0.5801) grad_norm 0.1989 (0.2244) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:33:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [475/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8989) loss 0.6090 (0.5827) grad_norm 0.2247 (0.2310) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:35:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [475/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8924) loss 0.6049 (0.5820) grad_norm 0.1957 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:36:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [475/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8889) loss 0.5774 (0.5813) grad_norm 0.1795 (0.2198) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:36:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 475 training takes 0:05:57 [2024-03-07 02:36:56 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_475.pth saving...... [2024-03-07 02:36:57 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_475.pth saved !!! [2024-03-07 02:37:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [476/800][0/402] eta 0:34:49 lr 0.000025 time 5.1984 (5.1984) loss 0.5675 (0.5675) grad_norm 0.1993 (0.1993) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 02:38:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [476/800][100/402] eta 0:04:38 lr 0.000025 time 0.8779 (0.9218) loss 0.6093 (0.5793) grad_norm 0.2248 (0.2173) loss_scale 524288.0000 (358176.9505) mem 30609MB [2024-03-07 02:39:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [476/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.9007) loss 0.6144 (0.5803) grad_norm 0.1691 (0.2247) loss_scale 524288.0000 (440819.2637) mem 30609MB [2024-03-07 02:41:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [476/800][300/402] eta 0:01:31 lr 0.000025 time 0.8785 (0.8936) loss 0.5713 (0.5807) grad_norm 0.1686 (0.2187) loss_scale 524288.0000 (468549.7409) mem 30609MB [2024-03-07 02:42:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [476/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8898) loss 0.5584 (0.5813) grad_norm 0.2014 (0.2170) loss_scale 524288.0000 (482449.5561) mem 30609MB [2024-03-07 02:42:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 476 training takes 0:05:57 [2024-03-07 02:43:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [477/800][0/402] eta 0:31:20 lr 0.000025 time 4.6777 (4.6777) loss 0.5667 (0.5667) grad_norm 0.2123 (0.2123) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:44:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [477/800][100/402] eta 0:04:37 lr 0.000025 time 0.8801 (0.9175) loss 0.6111 (0.5828) grad_norm 0.2035 (0.2133) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:45:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [477/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8983) loss 0.5450 (0.5813) grad_norm 0.2302 (0.2121) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:47:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [477/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8920) loss 0.5980 (0.5807) grad_norm 0.1856 (0.2139) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:48:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [477/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8886) loss 0.5831 (0.5821) grad_norm 0.1887 (0.2142) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:48:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 477 training takes 0:05:57 [2024-03-07 02:48:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [478/800][0/402] eta 0:33:07 lr 0.000025 time 4.9436 (4.9436) loss 0.5620 (0.5620) grad_norm 0.2143 (0.2143) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:50:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [478/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9191) loss 0.5903 (0.5828) grad_norm 0.1939 (0.2120) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:51:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [478/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8991) loss 0.5760 (0.5818) grad_norm 0.2271 (0.2132) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:53:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [478/800][300/402] eta 0:01:31 lr 0.000025 time 0.8782 (0.8926) loss 0.5728 (0.5807) grad_norm 0.2101 (0.2135) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:54:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [478/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8891) loss 0.5664 (0.5809) grad_norm 0.1975 (0.2123) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:54:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 478 training takes 0:05:57 [2024-03-07 02:54:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [479/800][0/402] eta 0:33:41 lr 0.000025 time 5.0276 (5.0276) loss 0.5872 (0.5872) grad_norm 0.1965 (0.1965) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:56:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [479/800][100/402] eta 0:04:37 lr 0.000025 time 0.8783 (0.9199) loss 0.5912 (0.5817) grad_norm 0.2096 (0.2128) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:57:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [479/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8999) loss 0.6008 (0.5812) grad_norm 0.1967 (0.2136) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 02:59:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [479/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8929) loss 0.5589 (0.5815) grad_norm 0.1819 (0.2124) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:00:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [479/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8894) loss 0.5764 (0.5811) grad_norm 0.2461 (0.2122) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:00:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 479 training takes 0:05:57 [2024-03-07 03:00:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [480/800][0/402] eta 0:31:30 lr 0.000025 time 4.7034 (4.7034) loss 0.5826 (0.5826) grad_norm 0.2173 (0.2173) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:02:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [480/800][100/402] eta 0:04:36 lr 0.000025 time 0.8790 (0.9169) loss 0.5725 (0.5813) grad_norm 0.2587 (0.2214) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:03:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [480/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8979) loss 0.5633 (0.5819) grad_norm 0.1925 (0.2173) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:05:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [480/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8916) loss 0.5844 (0.5817) grad_norm 0.2242 (0.2152) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:06:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [480/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8885) loss 0.5642 (0.5815) grad_norm 0.2199 (0.2158) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:06:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 480 training takes 0:05:57 [2024-03-07 03:06:46 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_480.pth saving...... [2024-03-07 03:06:48 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_480.pth saved !!! [2024-03-07 03:06:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [481/800][0/402] eta 0:35:35 lr 0.000025 time 5.3117 (5.3117) loss 0.5668 (0.5668) grad_norm 0.2625 (0.2625) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:08:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [481/800][100/402] eta 0:04:38 lr 0.000025 time 0.8807 (0.9228) loss 0.6081 (0.5791) grad_norm 0.2027 (0.2134) loss_scale 1048576.0000 (768263.6040) mem 30609MB [2024-03-07 03:09:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [481/800][200/402] eta 0:03:02 lr 0.000025 time 0.8812 (0.9012) loss 0.6140 (0.5795) grad_norm 0.2116 (inf) loss_scale 524288.0000 (795561.3930) mem 30609MB [2024-03-07 03:11:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [481/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8939) loss 0.5704 (0.5802) grad_norm 0.2003 (inf) loss_scale 524288.0000 (705437.3422) mem 30609MB [2024-03-07 03:12:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [481/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8900) loss 0.5606 (0.5807) grad_norm 0.1835 (inf) loss_scale 524288.0000 (660262.9426) mem 30609MB [2024-03-07 03:12:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 481 training takes 0:05:58 [2024-03-07 03:12:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [482/800][0/402] eta 0:36:25 lr 0.000025 time 5.4376 (5.4376) loss 0.5574 (0.5574) grad_norm 0.1980 (0.1980) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:14:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [482/800][100/402] eta 0:04:39 lr 0.000025 time 0.8786 (0.9241) loss 0.5867 (0.5784) grad_norm 0.1916 (0.2105) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 03:15:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [482/800][200/402] eta 0:03:02 lr 0.000025 time 0.8787 (0.9015) loss 0.5562 (0.5796) grad_norm 0.2193 (inf) loss_scale 262144.0000 (427777.2736) mem 30609MB [2024-03-07 03:17:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [482/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8939) loss 0.5532 (0.5801) grad_norm 0.1986 (inf) loss_scale 262144.0000 (372749.6080) mem 30609MB [2024-03-07 03:18:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [482/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8900) loss 0.5969 (0.5809) grad_norm 0.1962 (inf) loss_scale 262144.0000 (345167.1621) mem 30609MB [2024-03-07 03:18:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 482 training takes 0:05:58 [2024-03-07 03:18:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [483/800][0/402] eta 0:33:53 lr 0.000025 time 5.0578 (5.0578) loss 0.6044 (0.6044) grad_norm 0.2221 (0.2221) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:20:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [483/800][100/402] eta 0:04:38 lr 0.000025 time 0.8794 (0.9210) loss 0.6016 (0.5793) grad_norm 0.2015 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:21:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [483/800][200/402] eta 0:03:01 lr 0.000025 time 0.8796 (0.9002) loss 0.5796 (0.5811) grad_norm 0.2288 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:23:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [483/800][300/402] eta 0:01:31 lr 0.000025 time 0.8790 (0.8933) loss 0.5824 (0.5803) grad_norm 0.2252 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:24:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [483/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8897) loss 0.6034 (0.5808) grad_norm 0.2012 (0.2143) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:24:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 483 training takes 0:05:57 [2024-03-07 03:24:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [484/800][0/402] eta 0:35:15 lr 0.000025 time 5.2615 (5.2615) loss 0.6041 (0.6041) grad_norm 0.1930 (0.1930) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:26:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [484/800][100/402] eta 0:04:38 lr 0.000025 time 0.8790 (0.9223) loss 0.5788 (0.5808) grad_norm 0.2305 (0.2176) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:27:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [484/800][200/402] eta 0:03:01 lr 0.000025 time 0.8812 (0.9007) loss 0.5533 (0.5811) grad_norm 0.2060 (0.2126) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:29:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [484/800][300/402] eta 0:01:31 lr 0.000025 time 0.8795 (0.8935) loss 0.5857 (0.5822) grad_norm 0.2118 (0.2117) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:30:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [484/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8898) loss 0.5386 (0.5820) grad_norm 0.1991 (0.2108) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:30:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 484 training takes 0:05:57 [2024-03-07 03:30:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [485/800][0/402] eta 0:36:39 lr 0.000025 time 5.4720 (5.4720) loss 0.5478 (0.5478) grad_norm 0.1875 (0.1875) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:32:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [485/800][100/402] eta 0:04:39 lr 0.000025 time 0.8789 (0.9244) loss 0.5793 (0.5848) grad_norm 0.2280 (0.2088) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:33:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [485/800][200/402] eta 0:03:02 lr 0.000025 time 0.8788 (0.9020) loss 0.5439 (0.5818) grad_norm 0.2065 (0.2136) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:35:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [485/800][300/402] eta 0:01:31 lr 0.000025 time 0.8791 (0.8943) loss 0.6233 (0.5817) grad_norm 0.2072 (0.2126) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:36:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [485/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8906) loss 0.5627 (0.5816) grad_norm 0.2198 (0.2136) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:36:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 485 training takes 0:05:58 [2024-03-07 03:36:38 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_485.pth saving...... [2024-03-07 03:36:40 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_485.pth saved !!! [2024-03-07 03:36:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [486/800][0/402] eta 0:36:11 lr 0.000025 time 5.4027 (5.4027) loss 0.5727 (0.5727) grad_norm 0.1853 (0.1853) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:38:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [486/800][100/402] eta 0:04:38 lr 0.000025 time 0.8787 (0.9234) loss 0.5894 (0.5806) grad_norm 0.2100 (0.2109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:39:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [486/800][200/402] eta 0:03:02 lr 0.000025 time 0.8777 (0.9012) loss 0.5430 (0.5818) grad_norm 0.2383 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:41:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [486/800][300/402] eta 0:01:31 lr 0.000025 time 0.8794 (0.8937) loss 0.5796 (0.5806) grad_norm 0.2319 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:42:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [486/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8899) loss 0.5991 (0.5808) grad_norm 0.2065 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:42:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 486 training takes 0:05:57 [2024-03-07 03:42:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [487/800][0/402] eta 0:31:32 lr 0.000025 time 4.7083 (4.7083) loss 0.5668 (0.5668) grad_norm 0.1895 (0.1895) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:44:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [487/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9168) loss 0.6005 (0.5808) grad_norm 0.2024 (0.2118) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:45:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [487/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8980) loss 0.5717 (0.5799) grad_norm 0.2094 (0.2123) loss_scale 524288.0000 (371696.7164) mem 30609MB [2024-03-07 03:47:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [487/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8916) loss 0.5781 (0.5802) grad_norm 0.2322 (nan) loss_scale 262144.0000 (349235.0299) mem 30609MB [2024-03-07 03:48:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [487/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8885) loss 0.5611 (0.5797) grad_norm 0.2154 (nan) loss_scale 262144.0000 (327516.5686) mem 30609MB [2024-03-07 03:48:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 487 training takes 0:05:57 [2024-03-07 03:48:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [488/800][0/402] eta 0:33:22 lr 0.000025 time 4.9815 (4.9815) loss 0.6115 (0.6115) grad_norm 0.2405 (0.2405) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:50:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [488/800][100/402] eta 0:04:37 lr 0.000025 time 0.8783 (0.9196) loss 0.5483 (0.5820) grad_norm 0.1885 (0.2070) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:51:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [488/800][200/402] eta 0:03:01 lr 0.000025 time 0.8776 (0.8993) loss 0.5847 (0.5808) grad_norm 0.1821 (0.2099) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:53:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [488/800][300/402] eta 0:01:31 lr 0.000025 time 0.8788 (0.8925) loss 0.6018 (0.5812) grad_norm 0.1878 (0.2096) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:54:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [488/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8890) loss 0.6080 (0.5805) grad_norm 0.1955 (0.2117) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:54:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 488 training takes 0:05:57 [2024-03-07 03:54:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [489/800][0/402] eta 0:31:41 lr 0.000025 time 4.7302 (4.7302) loss 0.6045 (0.6045) grad_norm 0.2035 (0.2035) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:56:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [489/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9169) loss 0.5804 (0.5800) grad_norm 0.1962 (0.2103) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:57:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [489/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8979) loss 0.5705 (0.5805) grad_norm 0.2233 (0.2123) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 03:59:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [489/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8916) loss 0.5600 (0.5805) grad_norm 0.2282 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:00:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [489/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8883) loss 0.6072 (0.5818) grad_norm 0.1840 (0.2121) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:00:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 489 training takes 0:05:57 [2024-03-07 04:00:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [490/800][0/402] eta 0:33:27 lr 0.000025 time 4.9941 (4.9941) loss 0.5326 (0.5326) grad_norm 0.2354 (0.2354) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:02:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [490/800][100/402] eta 0:04:37 lr 0.000025 time 0.8796 (0.9200) loss 0.6170 (0.5808) grad_norm 0.2166 (0.2102) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:03:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [490/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8998) loss 0.6049 (0.5810) grad_norm 0.2460 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:04:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [490/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8928) loss 0.5830 (0.5804) grad_norm 0.2001 (0.2134) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:06:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [490/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8893) loss 0.5682 (0.5805) grad_norm 0.2336 (0.2141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:06:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 490 training takes 0:05:57 [2024-03-07 04:06:28 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_490.pth saving...... [2024-03-07 04:06:30 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_490.pth saved !!! [2024-03-07 04:06:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [491/800][0/402] eta 0:35:49 lr 0.000025 time 5.3476 (5.3476) loss 0.5868 (0.5868) grad_norm 0.1988 (0.1988) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:08:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [491/800][100/402] eta 0:04:38 lr 0.000025 time 0.8809 (0.9230) loss 0.6190 (0.5806) grad_norm 0.1999 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:09:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [491/800][200/402] eta 0:03:02 lr 0.000025 time 0.8787 (0.9011) loss 0.5910 (0.5817) grad_norm 0.2430 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:10:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [491/800][300/402] eta 0:01:31 lr 0.000025 time 0.8785 (0.8936) loss 0.5632 (0.5819) grad_norm 0.2064 (0.2125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:12:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [491/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8898) loss 0.5851 (0.5815) grad_norm 0.2011 (0.2112) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:12:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 491 training takes 0:05:57 [2024-03-07 04:12:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [492/800][0/402] eta 0:34:35 lr 0.000025 time 5.1626 (5.1626) loss 0.5537 (0.5537) grad_norm 0.2390 (0.2390) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:14:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [492/800][100/402] eta 0:04:38 lr 0.000025 time 0.8791 (0.9212) loss 0.5717 (0.5807) grad_norm 0.2614 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:15:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [492/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.9008) loss 0.5751 (0.5799) grad_norm 0.2042 (0.2176) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:16:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [492/800][300/402] eta 0:01:31 lr 0.000025 time 0.8792 (0.8935) loss 0.6299 (0.5811) grad_norm 0.1978 (0.2156) loss_scale 524288.0000 (344009.5681) mem 30609MB [2024-03-07 04:18:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [492/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8897) loss 0.5998 (0.5803) grad_norm 0.2163 (0.2156) loss_scale 524288.0000 (388966.7830) mem 30609MB [2024-03-07 04:18:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 492 training takes 0:05:57 [2024-03-07 04:18:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [493/800][0/402] eta 0:35:09 lr 0.000025 time 5.2468 (5.2468) loss 0.6058 (0.6058) grad_norm 0.1952 (0.1952) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 04:19:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [493/800][100/402] eta 0:04:38 lr 0.000025 time 0.8794 (0.9219) loss 0.5667 (0.5815) grad_norm 0.2031 (0.2079) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 04:21:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [493/800][200/402] eta 0:03:01 lr 0.000025 time 0.8798 (0.9004) loss 0.5958 (0.5812) grad_norm 0.2141 (0.2125) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 04:22:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [493/800][300/402] eta 0:01:31 lr 0.000025 time 0.8798 (0.8933) loss 0.5793 (0.5815) grad_norm 0.2207 (nan) loss_scale 262144.0000 (482484.3056) mem 30609MB [2024-03-07 04:24:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [493/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8896) loss 0.5702 (0.5809) grad_norm 0.2021 (nan) loss_scale 262144.0000 (427536.5985) mem 30609MB [2024-03-07 04:24:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 493 training takes 0:05:57 [2024-03-07 04:24:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [494/800][0/402] eta 0:35:07 lr 0.000025 time 5.2434 (5.2434) loss 0.5943 (0.5943) grad_norm 0.2274 (0.2274) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:25:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [494/800][100/402] eta 0:04:38 lr 0.000025 time 0.8790 (0.9221) loss 0.6113 (0.5807) grad_norm 0.1753 (0.2175) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:27:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [494/800][200/402] eta 0:03:01 lr 0.000025 time 0.8792 (0.9006) loss 0.5831 (0.5803) grad_norm 0.2591 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:28:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [494/800][300/402] eta 0:01:31 lr 0.000025 time 0.8782 (0.8939) loss 0.5830 (0.5801) grad_norm 0.1874 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:30:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [494/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8901) loss 0.5899 (0.5802) grad_norm 0.2049 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:30:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 494 training takes 0:05:58 [2024-03-07 04:30:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [495/800][0/402] eta 0:32:33 lr 0.000025 time 4.8602 (4.8602) loss 0.5985 (0.5985) grad_norm 0.2451 (0.2451) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:31:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [495/800][100/402] eta 0:04:37 lr 0.000025 time 0.8833 (0.9185) loss 0.5924 (0.5790) grad_norm 0.1822 (0.2123) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:33:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [495/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8988) loss 0.5706 (0.5803) grad_norm 0.2252 (0.2106) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:34:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [495/800][300/402] eta 0:01:30 lr 0.000025 time 0.8796 (0.8922) loss 0.5660 (0.5817) grad_norm 0.1889 (0.2096) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:36:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [495/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8888) loss 0.6055 (0.5814) grad_norm 0.1864 (0.2101) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:36:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 495 training takes 0:05:57 [2024-03-07 04:36:20 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_495.pth saving...... [2024-03-07 04:36:22 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_495.pth saved !!! [2024-03-07 04:36:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [496/800][0/402] eta 0:35:44 lr 0.000025 time 5.3340 (5.3340) loss 0.6046 (0.6046) grad_norm 0.2443 (0.2443) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:37:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [496/800][100/402] eta 0:04:38 lr 0.000025 time 0.8790 (0.9231) loss 0.5553 (0.5809) grad_norm 0.2137 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:39:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [496/800][200/402] eta 0:03:02 lr 0.000025 time 0.8787 (0.9010) loss 0.5873 (0.5812) grad_norm 0.2005 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:40:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [496/800][300/402] eta 0:01:31 lr 0.000025 time 0.8789 (0.8939) loss 0.5764 (0.5808) grad_norm 0.2219 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:42:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [496/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8901) loss 0.5964 (0.5813) grad_norm 0.2125 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:42:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 496 training takes 0:05:58 [2024-03-07 04:42:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [497/800][0/402] eta 0:36:52 lr 0.000025 time 5.5025 (5.5025) loss 0.5885 (0.5885) grad_norm 0.1960 (0.1960) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:43:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [497/800][100/402] eta 0:04:39 lr 0.000025 time 0.8791 (0.9247) loss 0.5881 (0.5814) grad_norm 0.2333 (0.2132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:45:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [497/800][200/402] eta 0:03:02 lr 0.000025 time 0.8785 (0.9019) loss 0.5920 (0.5818) grad_norm 0.1805 (0.2117) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:46:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [497/800][300/402] eta 0:01:31 lr 0.000025 time 0.8785 (0.8942) loss 0.5644 (0.5806) grad_norm 0.1989 (0.2125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:48:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [497/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8904) loss 0.5550 (0.5808) grad_norm 0.1981 (0.2142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:48:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 497 training takes 0:05:58 [2024-03-07 04:48:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [498/800][0/402] eta 0:33:00 lr 0.000025 time 4.9272 (4.9272) loss 0.5681 (0.5681) grad_norm 0.1976 (0.1976) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:49:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [498/800][100/402] eta 0:04:37 lr 0.000025 time 0.8783 (0.9189) loss 0.5807 (0.5761) grad_norm 0.1904 (0.2143) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:51:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [498/800][200/402] eta 0:03:01 lr 0.000025 time 0.8816 (0.8989) loss 0.5843 (0.5796) grad_norm 0.2231 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 04:52:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [498/800][300/402] eta 0:01:31 lr 0.000025 time 0.8782 (0.8923) loss 0.5791 (0.5797) grad_norm 0.2425 (0.2124) loss_scale 524288.0000 (312656.7973) mem 30609MB [2024-03-07 04:54:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [498/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8889) loss 0.5600 (0.5807) grad_norm 0.2110 (0.2139) loss_scale 524288.0000 (365432.6584) mem 30609MB [2024-03-07 04:54:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 498 training takes 0:05:57 [2024-03-07 04:54:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [499/800][0/402] eta 0:36:44 lr 0.000025 time 5.4842 (5.4842) loss 0.6070 (0.6070) grad_norm 0.2284 (0.2284) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 04:55:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [499/800][100/402] eta 0:04:39 lr 0.000025 time 0.8792 (0.9252) loss 0.5952 (0.5857) grad_norm 0.2125 (0.2165) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 04:57:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [499/800][200/402] eta 0:03:02 lr 0.000025 time 0.8783 (0.9021) loss 0.6022 (0.5833) grad_norm 0.1939 (0.2193) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 04:58:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [499/800][300/402] eta 0:01:31 lr 0.000025 time 0.8792 (0.8944) loss 0.5864 (0.5826) grad_norm 0.2170 (0.2176) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:00:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [499/800][400/402] eta 0:00:01 lr 0.000025 time 0.8783 (0.8905) loss 0.5294 (0.5819) grad_norm 0.2166 (0.2140) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:00:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 499 training takes 0:05:58 [2024-03-07 05:00:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [500/800][0/402] eta 0:34:44 lr 0.000025 time 5.1846 (5.1846) loss 0.5796 (0.5796) grad_norm 0.2026 (0.2026) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:01:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [500/800][100/402] eta 0:04:38 lr 0.000025 time 0.8794 (0.9214) loss 0.5870 (0.5778) grad_norm 0.2070 (0.2143) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:03:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [500/800][200/402] eta 0:03:01 lr 0.000025 time 0.8777 (0.9003) loss 0.5550 (0.5802) grad_norm 0.2294 (0.2159) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:04:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [500/800][300/402] eta 0:01:31 lr 0.000025 time 0.8776 (0.8931) loss 0.5715 (0.5806) grad_norm 0.2429 (0.2153) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:06:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [500/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8894) loss 0.5900 (0.5809) grad_norm 0.1983 (0.2148) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:06:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 500 training takes 0:05:57 [2024-03-07 05:06:12 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_500.pth saving...... [2024-03-07 05:06:14 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_500.pth saved !!! [2024-03-07 05:06:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [501/800][0/402] eta 0:35:25 lr 0.000025 time 5.2875 (5.2875) loss 0.5728 (0.5728) grad_norm 0.2119 (0.2119) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:07:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [501/800][100/402] eta 0:04:38 lr 0.000025 time 0.8785 (0.9227) loss 0.6112 (0.5790) grad_norm 0.1993 (0.2116) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:09:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [501/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.9010) loss 0.6321 (0.5803) grad_norm 0.1965 (0.2133) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:10:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [501/800][300/402] eta 0:01:31 lr 0.000025 time 0.8786 (0.8937) loss 0.5730 (0.5808) grad_norm 0.2533 (inf) loss_scale 262144.0000 (503386.1528) mem 30609MB [2024-03-07 05:12:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [501/800][400/402] eta 0:00:01 lr 0.000025 time 0.8774 (0.8899) loss 0.5920 (0.5811) grad_norm 0.1939 (inf) loss_scale 262144.0000 (443226.0150) mem 30609MB [2024-03-07 05:12:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 501 training takes 0:05:57 [2024-03-07 05:12:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [502/800][0/402] eta 0:32:49 lr 0.000025 time 4.9004 (4.9004) loss 0.5678 (0.5678) grad_norm 0.1901 (0.1901) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:13:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [502/800][100/402] eta 0:04:37 lr 0.000025 time 0.8794 (0.9188) loss 0.5700 (0.5808) grad_norm 0.1946 (0.2106) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:15:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [502/800][200/402] eta 0:03:01 lr 0.000025 time 0.8795 (0.8989) loss 0.5933 (0.5807) grad_norm 0.1926 (0.2106) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:16:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [502/800][300/402] eta 0:01:31 lr 0.000025 time 0.8791 (0.8923) loss 0.5803 (0.5805) grad_norm 0.1770 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:18:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [502/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8888) loss 0.6071 (0.5817) grad_norm 0.2214 (0.2136) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:18:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 502 training takes 0:05:57 [2024-03-07 05:18:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [503/800][0/402] eta 0:33:11 lr 0.000025 time 4.9529 (4.9529) loss 0.5870 (0.5870) grad_norm 0.1956 (0.1956) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:19:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [503/800][100/402] eta 0:04:37 lr 0.000025 time 0.8792 (0.9192) loss 0.5793 (0.5811) grad_norm 0.2274 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:21:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [503/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8993) loss 0.6178 (0.5813) grad_norm 0.2329 (0.2130) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:22:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [503/800][300/402] eta 0:01:31 lr 0.000025 time 0.8790 (0.8925) loss 0.5790 (0.5808) grad_norm 0.2400 (0.2110) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:24:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [503/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8891) loss 0.5889 (0.5810) grad_norm 0.2102 (0.2121) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:24:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 503 training takes 0:05:57 [2024-03-07 05:24:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [504/800][0/402] eta 0:31:45 lr 0.000025 time 4.7412 (4.7412) loss 0.5793 (0.5793) grad_norm 0.2157 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:25:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [504/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9172) loss 0.6040 (0.5790) grad_norm 0.1908 (0.2091) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:27:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [504/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8982) loss 0.5730 (0.5818) grad_norm 0.2105 (0.2149) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:28:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [504/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8917) loss 0.5908 (0.5812) grad_norm 0.2056 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:30:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [504/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8883) loss 0.5694 (0.5808) grad_norm 0.2075 (0.2114) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:30:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 504 training takes 0:05:57 [2024-03-07 05:30:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [505/800][0/402] eta 0:34:59 lr 0.000025 time 5.2232 (5.2232) loss 0.5615 (0.5615) grad_norm 0.2003 (0.2003) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:31:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [505/800][100/402] eta 0:04:38 lr 0.000025 time 0.8793 (0.9219) loss 0.5671 (0.5816) grad_norm 0.1928 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:33:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [505/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.9004) loss 0.6030 (0.5819) grad_norm 0.1987 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:34:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [505/800][300/402] eta 0:01:31 lr 0.000025 time 0.8783 (0.8934) loss 0.5702 (0.5809) grad_norm 0.2039 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:36:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [505/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8896) loss 0.5828 (0.5800) grad_norm 0.2177 (0.2216) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:36:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 505 training takes 0:05:57 [2024-03-07 05:36:03 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_505.pth saving...... [2024-03-07 05:36:04 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_505.pth saved !!! [2024-03-07 05:36:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [506/800][0/402] eta 0:34:18 lr 0.000025 time 5.1199 (5.1199) loss 0.5675 (0.5675) grad_norm 0.2058 (0.2058) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:37:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [506/800][100/402] eta 0:04:38 lr 0.000025 time 0.8794 (0.9216) loss 0.5646 (0.5793) grad_norm 0.2138 (0.2125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:39:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [506/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.9003) loss 0.5513 (0.5783) grad_norm 0.1762 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 05:40:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [506/800][300/402] eta 0:01:31 lr 0.000025 time 0.8793 (0.8931) loss 0.5893 (0.5791) grad_norm 0.2379 (0.2133) loss_scale 524288.0000 (291754.9502) mem 30609MB [2024-03-07 05:42:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [506/800][400/402] eta 0:00:01 lr 0.000025 time 0.8780 (0.8895) loss 0.5541 (0.5797) grad_norm 0.1978 (0.2134) loss_scale 524288.0000 (349743.2419) mem 30609MB [2024-03-07 05:42:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 506 training takes 0:05:57 [2024-03-07 05:42:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [507/800][0/402] eta 0:32:00 lr 0.000025 time 4.7765 (4.7765) loss 0.5839 (0.5839) grad_norm 0.1984 (0.1984) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:43:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [507/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9175) loss 0.5992 (0.5795) grad_norm 0.2905 (0.2134) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:45:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [507/800][200/402] eta 0:03:01 lr 0.000025 time 0.8792 (0.8983) loss 0.5705 (0.5810) grad_norm 0.2862 (0.2179) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:46:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [507/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8917) loss 0.6138 (0.5810) grad_norm 0.1948 (0.2159) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:47:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [507/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8886) loss 0.5731 (0.5811) grad_norm 0.1868 (0.2137) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:48:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 507 training takes 0:05:57 [2024-03-07 05:48:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [508/800][0/402] eta 0:31:29 lr 0.000025 time 4.6998 (4.6998) loss 0.5550 (0.5550) grad_norm 0.2052 (0.2052) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:49:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [508/800][100/402] eta 0:04:36 lr 0.000025 time 0.8794 (0.9167) loss 0.5747 (0.5785) grad_norm 0.2199 (0.2129) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:51:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [508/800][200/402] eta 0:03:01 lr 0.000025 time 0.8793 (0.8981) loss 0.5728 (0.5782) grad_norm 0.2207 (0.2165) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:52:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [508/800][300/402] eta 0:01:30 lr 0.000025 time 0.8792 (0.8918) loss 0.5862 (0.5793) grad_norm 0.2218 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:53:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [508/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8885) loss 0.5601 (0.5800) grad_norm 0.1915 (0.2150) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:53:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 508 training takes 0:05:57 [2024-03-07 05:54:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [509/800][0/402] eta 0:33:08 lr 0.000025 time 4.9475 (4.9475) loss 0.6259 (0.6259) grad_norm 0.2109 (0.2109) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:55:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [509/800][100/402] eta 0:04:37 lr 0.000025 time 0.8778 (0.9191) loss 0.5816 (0.5803) grad_norm 0.2082 (0.2121) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:56:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [509/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8990) loss 0.5830 (0.5788) grad_norm 0.2171 (0.2097) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 05:58:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [509/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8923) loss 0.6004 (0.5809) grad_norm 0.1978 (inf) loss_scale 262144.0000 (513837.0764) mem 30609MB [2024-03-07 05:59:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [509/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8888) loss 0.5848 (0.5816) grad_norm 0.2171 (inf) loss_scale 262144.0000 (451070.7232) mem 30609MB [2024-03-07 05:59:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 509 training takes 0:05:57 [2024-03-07 06:00:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [510/800][0/402] eta 0:34:40 lr 0.000025 time 5.1761 (5.1761) loss 0.5599 (0.5599) grad_norm 0.2389 (0.2389) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:01:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [510/800][100/402] eta 0:04:38 lr 0.000025 time 0.8791 (0.9217) loss 0.5812 (0.5798) grad_norm 0.1992 (0.2064) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:02:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [510/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.9004) loss 0.5733 (0.5789) grad_norm 0.2362 (0.2105) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:04:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [510/800][300/402] eta 0:01:31 lr 0.000025 time 0.9106 (0.8934) loss 0.5676 (0.5796) grad_norm 0.2010 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:05:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [510/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8897) loss 0.5499 (0.5797) grad_norm 0.1906 (0.2122) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:05:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 510 training takes 0:05:57 [2024-03-07 06:05:53 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_510.pth saving...... [2024-03-07 06:05:55 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_510.pth saved !!! [2024-03-07 06:06:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [511/800][0/402] eta 0:34:21 lr 0.000025 time 5.1269 (5.1269) loss 0.5832 (0.5832) grad_norm 0.2243 (0.2243) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:07:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [511/800][100/402] eta 0:04:38 lr 0.000025 time 0.8784 (0.9207) loss 0.6024 (0.5777) grad_norm 0.2093 (0.2098) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:08:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [511/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8999) loss 0.5780 (0.5793) grad_norm 0.2140 (0.2116) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:10:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [511/800][300/402] eta 0:01:31 lr 0.000025 time 0.8780 (0.8928) loss 0.5832 (0.5795) grad_norm 0.2010 (0.2108) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:11:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [511/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8892) loss 0.5853 (0.5799) grad_norm 0.2434 (0.2140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:11:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 511 training takes 0:05:57 [2024-03-07 06:11:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [512/800][0/402] eta 0:32:04 lr 0.000025 time 4.7871 (4.7871) loss 0.5884 (0.5884) grad_norm 0.2020 (0.2020) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:13:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [512/800][100/402] eta 0:04:37 lr 0.000025 time 0.8782 (0.9174) loss 0.5809 (0.5814) grad_norm 0.2083 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:14:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [512/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8981) loss 0.5918 (0.5795) grad_norm 0.2041 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:16:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [512/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8916) loss 0.5847 (0.5799) grad_norm 0.1977 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:17:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [512/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8882) loss 0.5985 (0.5797) grad_norm 0.1892 (0.2129) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:17:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 512 training takes 0:05:57 [2024-03-07 06:17:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [513/800][0/402] eta 0:31:51 lr 0.000025 time 4.7554 (4.7554) loss 0.5683 (0.5683) grad_norm 0.2291 (0.2291) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:19:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [513/800][100/402] eta 0:04:37 lr 0.000025 time 0.8781 (0.9173) loss 0.5915 (0.5795) grad_norm 0.1917 (0.2196) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:20:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [513/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8980) loss 0.5773 (0.5796) grad_norm 0.2178 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:22:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [513/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8915) loss 0.5837 (0.5799) grad_norm 0.2083 (0.2158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:23:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [513/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8882) loss 0.5809 (0.5792) grad_norm 0.2509 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:23:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 513 training takes 0:05:57 [2024-03-07 06:23:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [514/800][0/402] eta 0:31:41 lr 0.000025 time 4.7292 (4.7292) loss 0.5400 (0.5400) grad_norm 0.2227 (0.2227) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:25:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [514/800][100/402] eta 0:04:36 lr 0.000025 time 0.8790 (0.9167) loss 0.6127 (0.5830) grad_norm 0.2089 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:26:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [514/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8977) loss 0.5721 (0.5815) grad_norm 0.1884 (0.2142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:28:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [514/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8914) loss 0.5707 (0.5806) grad_norm 0.2756 (0.2155) loss_scale 524288.0000 (281304.0266) mem 30609MB [2024-03-07 06:29:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [514/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8881) loss 0.6179 (0.5814) grad_norm 0.1936 (nan) loss_scale 262144.0000 (322286.7631) mem 30609MB [2024-03-07 06:29:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 514 training takes 0:05:57 [2024-03-07 06:29:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [515/800][0/402] eta 0:32:25 lr 0.000025 time 4.8384 (4.8384) loss 0.5990 (0.5990) grad_norm 0.2124 (0.2124) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:31:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [515/800][100/402] eta 0:04:37 lr 0.000025 time 0.8781 (0.9176) loss 0.5986 (0.5813) grad_norm 0.2034 (0.2105) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:32:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [515/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8984) loss 0.5809 (0.5800) grad_norm 0.1744 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:34:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [515/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8918) loss 0.5856 (0.5815) grad_norm 0.1964 (0.2126) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:35:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [515/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8884) loss 0.5812 (0.5809) grad_norm 0.1904 (0.2124) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:35:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 515 training takes 0:05:57 [2024-03-07 06:35:42 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_515.pth saving...... [2024-03-07 06:35:43 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_515.pth saved !!! [2024-03-07 06:35:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [516/800][0/402] eta 0:31:31 lr 0.000025 time 4.7047 (4.7047) loss 0.5820 (0.5820) grad_norm 0.2276 (0.2276) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:37:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [516/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9163) loss 0.5542 (0.5784) grad_norm 0.1784 (0.2132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:38:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [516/800][200/402] eta 0:03:01 lr 0.000025 time 0.8803 (0.8974) loss 0.5652 (0.5777) grad_norm 0.2168 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:40:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [516/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8914) loss 0.5965 (0.5791) grad_norm 0.1930 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:41:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [516/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8881) loss 0.5915 (0.5796) grad_norm 0.1944 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:41:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 516 training takes 0:05:57 [2024-03-07 06:41:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [517/800][0/402] eta 0:32:01 lr 0.000025 time 4.7811 (4.7811) loss 0.5492 (0.5492) grad_norm 0.2480 (0.2480) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:43:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [517/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9169) loss 0.5549 (0.5813) grad_norm 0.2349 (0.2135) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:44:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [517/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8978) loss 0.6095 (0.5830) grad_norm 0.2096 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:46:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [517/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8917) loss 0.5861 (0.5813) grad_norm 0.1895 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:47:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [517/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8883) loss 0.6354 (0.5806) grad_norm 0.2540 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:47:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 517 training takes 0:05:57 [2024-03-07 06:47:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [518/800][0/402] eta 0:31:13 lr 0.000025 time 4.6614 (4.6614) loss 0.5584 (0.5584) grad_norm 0.1954 (0.1954) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:49:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [518/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9158) loss 0.6069 (0.5805) grad_norm 0.2321 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:50:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [518/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8972) loss 0.5583 (0.5811) grad_norm 0.1883 (0.2134) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:52:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [518/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8909) loss 0.5863 (0.5809) grad_norm 0.2055 (0.2126) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:53:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [518/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8878) loss 0.5910 (0.5798) grad_norm 0.2129 (0.2139) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:53:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 518 training takes 0:05:57 [2024-03-07 06:53:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [519/800][0/402] eta 0:32:02 lr 0.000025 time 4.7823 (4.7823) loss 0.6057 (0.6057) grad_norm 0.1907 (0.1907) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:55:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [519/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9172) loss 0.5832 (0.5820) grad_norm 0.2098 (0.2116) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:56:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [519/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8979) loss 0.5825 (0.5821) grad_norm 0.1926 (0.2112) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:58:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [519/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8914) loss 0.5831 (0.5814) grad_norm 0.2086 (0.2132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 06:59:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [519/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8882) loss 0.5709 (0.5808) grad_norm 0.2368 (0.2140) loss_scale 524288.0000 (288293.0274) mem 30609MB [2024-03-07 06:59:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 519 training takes 0:05:57 [2024-03-07 06:59:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [520/800][0/402] eta 0:30:56 lr 0.000025 time 4.6173 (4.6173) loss 0.5961 (0.5961) grad_norm 0.2172 (0.2172) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:01:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [520/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9155) loss 0.6161 (0.5803) grad_norm 0.2308 (0.2123) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:02:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [520/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8970) loss 0.5544 (0.5803) grad_norm 0.2501 (0.2170) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:04:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [520/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8908) loss 0.5784 (0.5797) grad_norm 0.1957 (0.2154) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:05:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [520/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8877) loss 0.5891 (0.5799) grad_norm 0.2398 (0.2153) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:05:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 520 training takes 0:05:57 [2024-03-07 07:05:30 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_520.pth saving...... [2024-03-07 07:05:31 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_520.pth saved !!! [2024-03-07 07:05:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [521/800][0/402] eta 0:33:49 lr 0.000025 time 5.0477 (5.0477) loss 0.5719 (0.5719) grad_norm 0.1987 (0.1987) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:07:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [521/800][100/402] eta 0:04:37 lr 0.000025 time 0.8782 (0.9197) loss 0.5381 (0.5827) grad_norm 0.1937 (0.2098) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:08:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [521/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8992) loss 0.6214 (0.5798) grad_norm 0.1777 (0.2097) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:10:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [521/800][300/402] eta 0:01:31 lr 0.000025 time 0.8785 (0.8923) loss 0.5883 (0.5802) grad_norm 0.2091 (0.2103) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:11:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [521/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8888) loss 0.6204 (0.5799) grad_norm 0.1800 (0.2114) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:11:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 521 training takes 0:05:57 [2024-03-07 07:11:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [522/800][0/402] eta 0:31:30 lr 0.000025 time 4.7019 (4.7019) loss 0.5914 (0.5914) grad_norm 0.2106 (0.2106) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:13:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [522/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9163) loss 0.5980 (0.5795) grad_norm 0.1830 (0.2117) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:14:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [522/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8976) loss 0.6129 (0.5816) grad_norm 0.2210 (0.2136) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:15:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [522/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8913) loss 0.5779 (0.5805) grad_norm 0.2127 (0.2134) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:17:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [522/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8880) loss 0.5639 (0.5801) grad_norm 0.2148 (nan) loss_scale 262144.0000 (483757.0075) mem 30609MB [2024-03-07 07:17:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 522 training takes 0:05:57 [2024-03-07 07:17:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [523/800][0/402] eta 0:32:13 lr 0.000025 time 4.8109 (4.8109) loss 0.5649 (0.5649) grad_norm 0.2006 (0.2006) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:18:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [523/800][100/402] eta 0:04:37 lr 0.000025 time 0.8783 (0.9173) loss 0.5859 (0.5801) grad_norm 0.2084 (0.2222) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:20:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [523/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8979) loss 0.6153 (0.5800) grad_norm 0.1885 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:21:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [523/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8915) loss 0.5780 (0.5799) grad_norm 0.2315 (0.2141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:23:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [523/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8882) loss 0.5818 (0.5803) grad_norm 0.2925 (0.2186) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:23:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 523 training takes 0:05:57 [2024-03-07 07:23:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [524/800][0/402] eta 0:32:28 lr 0.000025 time 4.8469 (4.8469) loss 0.5786 (0.5786) grad_norm 0.1801 (0.1801) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:24:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [524/800][100/402] eta 0:04:37 lr 0.000025 time 0.8776 (0.9178) loss 0.5908 (0.5814) grad_norm 0.2580 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:26:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [524/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8982) loss 0.6018 (0.5808) grad_norm 0.1919 (0.2114) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:27:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [524/800][300/402] eta 0:01:30 lr 0.000025 time 0.8810 (0.8920) loss 0.6118 (0.5805) grad_norm 0.2176 (0.2118) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:29:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [524/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8886) loss 0.5677 (0.5801) grad_norm 0.1928 (0.2109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:29:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 524 training takes 0:05:57 [2024-03-07 07:29:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [525/800][0/402] eta 0:31:50 lr 0.000025 time 4.7532 (4.7532) loss 0.5671 (0.5671) grad_norm 0.2095 (0.2095) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:30:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [525/800][100/402] eta 0:04:36 lr 0.000025 time 0.8807 (0.9168) loss 0.5818 (0.5797) grad_norm 0.2346 (0.2106) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:32:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [525/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8978) loss 0.6102 (0.5799) grad_norm 0.2617 (0.2153) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:33:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [525/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8913) loss 0.5533 (0.5799) grad_norm 0.2524 (0.2135) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:35:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [525/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8881) loss 0.6011 (0.5798) grad_norm 0.1932 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:35:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 525 training takes 0:05:57 [2024-03-07 07:35:18 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_525.pth saving...... [2024-03-07 07:35:20 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_525.pth saved !!! [2024-03-07 07:35:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [526/800][0/402] eta 0:32:31 lr 0.000025 time 4.8557 (4.8557) loss 0.5666 (0.5666) grad_norm 0.2167 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:36:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [526/800][100/402] eta 0:04:37 lr 0.000025 time 0.8784 (0.9176) loss 0.5937 (0.5816) grad_norm 0.1895 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:38:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [526/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8981) loss 0.5528 (0.5807) grad_norm 0.2203 (0.2108) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:39:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [526/800][300/402] eta 0:01:30 lr 0.000025 time 0.8778 (0.8916) loss 0.6027 (0.5805) grad_norm 0.1965 (0.2122) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:41:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [526/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8885) loss 0.5578 (0.5807) grad_norm 0.2230 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:41:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 526 training takes 0:05:57 [2024-03-07 07:41:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [527/800][0/402] eta 0:32:14 lr 0.000025 time 4.8115 (4.8115) loss 0.5748 (0.5748) grad_norm 0.2167 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:42:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [527/800][100/402] eta 0:04:37 lr 0.000025 time 0.8783 (0.9173) loss 0.5973 (0.5795) grad_norm 0.2197 (0.2103) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:44:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [527/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8981) loss 0.5996 (0.5809) grad_norm 0.2345 (0.2142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:45:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [527/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8915) loss 0.6008 (0.5797) grad_norm 0.2044 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 07:47:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [527/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8883) loss 0.5859 (0.5794) grad_norm 0.2142 (0.2158) loss_scale 524288.0000 (309212.2494) mem 30609MB [2024-03-07 07:47:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 527 training takes 0:05:57 [2024-03-07 07:47:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [528/800][0/402] eta 0:31:27 lr 0.000025 time 4.6945 (4.6945) loss 0.6056 (0.6056) grad_norm 0.1899 (0.1899) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:48:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [528/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9162) loss 0.5956 (0.5833) grad_norm 0.2077 (0.2130) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:50:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [528/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8974) loss 0.6112 (0.5807) grad_norm 0.2114 (0.2112) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:51:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [528/800][300/402] eta 0:01:30 lr 0.000025 time 0.8793 (0.8912) loss 0.5525 (0.5805) grad_norm 0.2204 (0.2137) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:53:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [528/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8880) loss 0.5703 (0.5801) grad_norm 0.1870 (0.2128) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:53:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 528 training takes 0:05:57 [2024-03-07 07:53:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [529/800][0/402] eta 0:31:41 lr 0.000025 time 4.7292 (4.7292) loss 0.5554 (0.5554) grad_norm 0.2130 (0.2130) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:54:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [529/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9170) loss 0.5735 (0.5773) grad_norm 0.2197 (0.2139) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:56:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [529/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8979) loss 0.5771 (0.5786) grad_norm 0.1819 (0.2125) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 07:57:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [529/800][300/402] eta 0:01:30 lr 0.000025 time 0.8775 (0.8914) loss 0.5518 (0.5794) grad_norm 0.2242 (nan) loss_scale 262144.0000 (474646.1130) mem 30609MB [2024-03-07 07:59:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [529/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8882) loss 0.5684 (0.5794) grad_norm 0.2198 (nan) loss_scale 262144.0000 (421653.0673) mem 30609MB [2024-03-07 07:59:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 529 training takes 0:05:57 [2024-03-07 07:59:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [530/800][0/402] eta 0:31:59 lr 0.000025 time 4.7739 (4.7739) loss 0.5747 (0.5747) grad_norm 0.2071 (0.2071) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:00:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [530/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9174) loss 0.5795 (0.5820) grad_norm 0.1916 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:02:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [530/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8981) loss 0.5545 (0.5795) grad_norm 0.1883 (0.2140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:03:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [530/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8916) loss 0.5815 (0.5802) grad_norm 0.2089 (0.2125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:05:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [530/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8883) loss 0.5702 (0.5803) grad_norm 0.1686 (0.2116) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:05:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 530 training takes 0:05:57 [2024-03-07 08:05:07 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_530.pth saving...... [2024-03-07 08:05:08 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_530.pth saved !!! [2024-03-07 08:05:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [531/800][0/402] eta 0:31:27 lr 0.000025 time 4.6956 (4.6956) loss 0.5847 (0.5847) grad_norm 0.2166 (0.2166) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:06:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [531/800][100/402] eta 0:04:36 lr 0.000025 time 0.8794 (0.9164) loss 0.6226 (0.5813) grad_norm 0.1897 (0.2094) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:08:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [531/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8979) loss 0.5883 (0.5797) grad_norm 0.2068 (0.2093) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:09:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [531/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8915) loss 0.5628 (0.5808) grad_norm 0.2398 (0.2100) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:11:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [531/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8882) loss 0.5791 (0.5797) grad_norm 0.2234 (0.2121) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:11:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 531 training takes 0:05:57 [2024-03-07 08:11:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [532/800][0/402] eta 0:32:35 lr 0.000025 time 4.8637 (4.8637) loss 0.5825 (0.5825) grad_norm 0.1980 (0.1980) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:12:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [532/800][100/402] eta 0:04:37 lr 0.000025 time 0.8779 (0.9183) loss 0.5976 (0.5808) grad_norm 0.1881 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:14:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [532/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8985) loss 0.5789 (0.5802) grad_norm 0.1957 (0.2134) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:15:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [532/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8918) loss 0.5682 (0.5801) grad_norm 0.2007 (0.2114) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:17:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [532/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8884) loss 0.5970 (0.5803) grad_norm 0.2002 (0.2122) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:17:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 532 training takes 0:05:57 [2024-03-07 08:17:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [533/800][0/402] eta 0:31:21 lr 0.000025 time 4.6801 (4.6801) loss 0.5909 (0.5909) grad_norm 0.2042 (0.2042) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:18:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [533/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9159) loss 0.5576 (0.5839) grad_norm 0.1965 (0.2130) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:20:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [533/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8973) loss 0.5852 (0.5825) grad_norm 0.2459 (0.2164) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:21:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [533/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8913) loss 0.5783 (0.5822) grad_norm 0.2121 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:22:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [533/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8881) loss 0.5752 (0.5817) grad_norm 0.1925 (0.2149) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:23:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 533 training takes 0:05:57 [2024-03-07 08:23:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [534/800][0/402] eta 0:31:50 lr 0.000025 time 4.7533 (4.7533) loss 0.5716 (0.5716) grad_norm 0.1822 (0.1822) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:24:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [534/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9167) loss 0.5627 (0.5808) grad_norm 0.2592 (0.2147) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:26:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [534/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8978) loss 0.5621 (0.5814) grad_norm 0.1778 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:27:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [534/800][300/402] eta 0:01:30 lr 0.000025 time 0.8798 (0.8914) loss 0.5458 (0.5801) grad_norm 0.1968 (0.2150) loss_scale 524288.0000 (320494.9900) mem 30609MB [2024-03-07 08:28:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [534/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8881) loss 0.5898 (0.5804) grad_norm 0.2068 (0.2141) loss_scale 524288.0000 (371316.1895) mem 30609MB [2024-03-07 08:28:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 534 training takes 0:05:57 [2024-03-07 08:29:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [535/800][0/402] eta 0:31:16 lr 0.000025 time 4.6668 (4.6668) loss 0.5570 (0.5570) grad_norm 0.2053 (0.2053) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:30:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [535/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9160) loss 0.5969 (0.5790) grad_norm 0.2080 (0.2131) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:31:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [535/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8974) loss 0.5982 (0.5785) grad_norm 0.1960 (0.2131) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:33:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [535/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8912) loss 0.5911 (0.5799) grad_norm 0.1847 (0.2116) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:34:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [535/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8880) loss 0.5975 (0.5806) grad_norm 0.2036 (0.2120) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:34:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 535 training takes 0:05:57 [2024-03-07 08:34:55 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_535.pth saving...... [2024-03-07 08:34:56 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_535.pth saved !!! [2024-03-07 08:35:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [536/800][0/402] eta 0:27:19 lr 0.000025 time 4.0795 (4.0795) loss 0.5487 (0.5487) grad_norm 0.2472 (0.2472) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:36:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [536/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9107) loss 0.5962 (0.5802) grad_norm 0.1889 (0.2159) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:37:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [536/800][200/402] eta 0:03:00 lr 0.000025 time 0.8788 (0.8948) loss 0.5859 (0.5797) grad_norm 0.2082 (0.2165) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:39:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [536/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8895) loss 0.6147 (0.5799) grad_norm 0.3189 (0.2172) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:40:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [536/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8867) loss 0.5917 (0.5797) grad_norm 0.2574 (0.2174) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:40:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 536 training takes 0:05:56 [2024-03-07 08:40:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [537/800][0/402] eta 0:32:03 lr 0.000025 time 4.7845 (4.7845) loss 0.5946 (0.5946) grad_norm 0.2154 (0.2154) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:42:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [537/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9171) loss 0.5635 (0.5781) grad_norm 0.2319 (0.2183) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:43:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [537/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8979) loss 0.5783 (0.5800) grad_norm 0.2277 (0.2142) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:45:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [537/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8914) loss 0.5578 (0.5809) grad_norm 0.2688 (0.2109) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 08:46:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [537/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8881) loss 0.5705 (0.5809) grad_norm 0.2233 (inf) loss_scale 262144.0000 (477873.4763) mem 30609MB [2024-03-07 08:46:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 537 training takes 0:05:57 [2024-03-07 08:46:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [538/800][0/402] eta 0:30:57 lr 0.000025 time 4.6205 (4.6205) loss 0.5895 (0.5895) grad_norm 0.2127 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:48:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [538/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9156) loss 0.5704 (0.5797) grad_norm 0.2195 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:49:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [538/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8976) loss 0.5927 (0.5805) grad_norm 0.2086 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:51:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [538/800][300/402] eta 0:01:30 lr 0.000025 time 0.8777 (0.8914) loss 0.5726 (0.5792) grad_norm 0.1898 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:52:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [538/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8881) loss 0.5849 (0.5791) grad_norm 0.1865 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:52:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 538 training takes 0:05:57 [2024-03-07 08:52:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [539/800][0/402] eta 0:32:18 lr 0.000025 time 4.8222 (4.8222) loss 0.5444 (0.5444) grad_norm 0.1892 (0.1892) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:54:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [539/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9175) loss 0.5685 (0.5803) grad_norm 0.2152 (0.2221) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:55:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [539/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8982) loss 0.5914 (0.5801) grad_norm 0.2036 (0.2147) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:57:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [539/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8917) loss 0.5883 (0.5795) grad_norm 0.2139 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:58:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [539/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8884) loss 0.5979 (0.5797) grad_norm 0.2235 (0.2147) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 08:58:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 539 training takes 0:05:57 [2024-03-07 08:58:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [540/800][0/402] eta 0:32:02 lr 0.000025 time 4.7820 (4.7820) loss 0.5572 (0.5572) grad_norm 0.1865 (0.1865) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:00:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [540/800][100/402] eta 0:04:36 lr 0.000025 time 0.8779 (0.9170) loss 0.5834 (0.5821) grad_norm 0.2219 (0.2215) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:01:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [540/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8977) loss 0.5680 (0.5819) grad_norm 0.1943 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:03:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [540/800][300/402] eta 0:01:30 lr 0.000025 time 0.8797 (0.8915) loss 0.6029 (0.5807) grad_norm 0.2333 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:04:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [540/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8882) loss 0.5861 (0.5804) grad_norm 0.1913 (0.2135) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:04:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 540 training takes 0:05:57 [2024-03-07 09:04:42 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_540.pth saving...... [2024-03-07 09:04:44 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_540.pth saved !!! [2024-03-07 09:04:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [541/800][0/402] eta 0:31:36 lr 0.000025 time 4.7168 (4.7168) loss 0.5960 (0.5960) grad_norm 0.1976 (0.1976) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:06:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [541/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9164) loss 0.5624 (0.5815) grad_norm 0.2097 (0.2104) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:07:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [541/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8976) loss 0.5915 (0.5804) grad_norm 0.2020 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:09:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [541/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8912) loss 0.5945 (0.5813) grad_norm 0.2297 (0.2139) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:10:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [541/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8880) loss 0.5751 (0.5805) grad_norm 0.1904 (0.2121) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:10:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 541 training takes 0:05:57 [2024-03-07 09:10:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [542/800][0/402] eta 0:30:37 lr 0.000025 time 4.5697 (4.5697) loss 0.5921 (0.5921) grad_norm 0.1997 (0.1997) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:12:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [542/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9149) loss 0.5456 (0.5793) grad_norm 0.2283 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:13:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [542/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8968) loss 0.5902 (0.5795) grad_norm 0.2105 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:15:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [542/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8908) loss 0.5621 (0.5793) grad_norm 0.2012 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:16:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [542/800][400/402] eta 0:00:01 lr 0.000025 time 0.8802 (0.8879) loss 0.5893 (0.5790) grad_norm 0.2155 (0.2140) loss_scale 524288.0000 (315095.7805) mem 30609MB [2024-03-07 09:16:39 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 542 training takes 0:05:57 [2024-03-07 09:16:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [543/800][0/402] eta 0:31:48 lr 0.000025 time 4.7470 (4.7470) loss 0.5622 (0.5622) grad_norm 0.2866 (0.2866) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:18:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [543/800][100/402] eta 0:04:37 lr 0.000025 time 0.8782 (0.9173) loss 0.5725 (0.5835) grad_norm 0.2148 (0.2102) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:19:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [543/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8984) loss 0.5767 (0.5810) grad_norm 0.2274 (0.2139) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:21:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [543/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8918) loss 0.5577 (0.5805) grad_norm 0.2147 (0.2129) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:22:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [543/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8884) loss 0.5988 (0.5804) grad_norm 0.2038 (0.2137) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:22:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 543 training takes 0:05:57 [2024-03-07 09:22:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [544/800][0/402] eta 0:32:10 lr 0.000025 time 4.8019 (4.8019) loss 0.5715 (0.5715) grad_norm 0.2215 (0.2215) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:24:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [544/800][100/402] eta 0:04:37 lr 0.000025 time 0.8782 (0.9173) loss 0.5782 (0.5805) grad_norm 0.1960 (0.2161) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:25:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [544/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8981) loss 0.5591 (0.5800) grad_norm 0.2242 (0.2147) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:27:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [544/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8917) loss 0.6010 (0.5801) grad_norm 0.1833 (0.2129) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:28:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [544/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8883) loss 0.5397 (0.5806) grad_norm 0.2396 (nan) loss_scale 262144.0000 (481142.1047) mem 30609MB [2024-03-07 09:28:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 544 training takes 0:05:57 [2024-03-07 09:28:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [545/800][0/402] eta 0:31:23 lr 0.000025 time 4.6852 (4.6852) loss 0.5662 (0.5662) grad_norm 0.2525 (0.2525) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:30:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [545/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9169) loss 0.5781 (0.5778) grad_norm 0.2164 (0.2098) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:31:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [545/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8979) loss 0.6056 (0.5794) grad_norm 0.1949 (0.2130) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:33:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [545/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8916) loss 0.6110 (0.5800) grad_norm 0.2176 (0.2126) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:34:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [545/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8883) loss 0.5525 (0.5799) grad_norm 0.2012 (0.2132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:34:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 545 training takes 0:05:57 [2024-03-07 09:34:31 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_545.pth saving...... [2024-03-07 09:34:32 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_545.pth saved !!! [2024-03-07 09:34:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [546/800][0/402] eta 0:31:40 lr 0.000025 time 4.7285 (4.7285) loss 0.5919 (0.5919) grad_norm 0.2109 (0.2109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:36:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [546/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9164) loss 0.5692 (0.5788) grad_norm 0.2031 (0.2196) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:37:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [546/800][200/402] eta 0:03:01 lr 0.000025 time 0.8794 (0.8977) loss 0.6016 (0.5804) grad_norm 0.2380 (0.2175) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:39:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [546/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8913) loss 0.6263 (0.5796) grad_norm 0.2102 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:40:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [546/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8880) loss 0.5370 (0.5792) grad_norm 0.2202 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:40:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 546 training takes 0:05:57 [2024-03-07 09:40:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [547/800][0/402] eta 0:31:59 lr 0.000025 time 4.7743 (4.7743) loss 0.5793 (0.5793) grad_norm 0.2159 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:42:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [547/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9170) loss 0.5787 (0.5789) grad_norm 0.2062 (0.2101) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:43:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [547/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8978) loss 0.5983 (0.5784) grad_norm 0.1872 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:44:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [547/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8918) loss 0.5871 (0.5789) grad_norm 0.2219 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:46:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [547/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8884) loss 0.5997 (0.5790) grad_norm 0.2202 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:46:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 547 training takes 0:05:57 [2024-03-07 09:46:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [548/800][0/402] eta 0:30:53 lr 0.000025 time 4.6102 (4.6102) loss 0.5668 (0.5668) grad_norm 0.2131 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:47:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [548/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9155) loss 0.5394 (0.5814) grad_norm 0.2195 (0.2196) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:49:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [548/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8971) loss 0.5554 (0.5809) grad_norm 0.2178 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:50:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [548/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8910) loss 0.6003 (0.5807) grad_norm 0.1900 (0.2142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:52:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [548/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8879) loss 0.5460 (0.5794) grad_norm 0.2248 (0.2135) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:52:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 548 training takes 0:05:57 [2024-03-07 09:52:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [549/800][0/402] eta 0:30:58 lr 0.000025 time 4.6227 (4.6227) loss 0.5954 (0.5954) grad_norm 0.1950 (0.1950) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:53:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [549/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9155) loss 0.5666 (0.5801) grad_norm 0.2025 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:55:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [549/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8972) loss 0.5441 (0.5807) grad_norm 0.2190 (0.2124) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:56:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [549/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8910) loss 0.5551 (0.5809) grad_norm 0.2835 (0.2128) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 09:58:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [549/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8880) loss 0.5979 (0.5810) grad_norm 0.1977 (0.2130) loss_scale 524288.0000 (311827.1521) mem 30609MB [2024-03-07 09:58:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 549 training takes 0:05:57 [2024-03-07 09:58:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [550/800][0/402] eta 0:32:04 lr 0.000025 time 4.7881 (4.7881) loss 0.5771 (0.5771) grad_norm 0.2192 (0.2192) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 09:59:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [550/800][100/402] eta 0:04:37 lr 0.000025 time 0.8773 (0.9174) loss 0.5884 (0.5821) grad_norm 0.2356 (0.2105) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:01:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [550/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8980) loss 0.5889 (0.5820) grad_norm 0.2347 (0.2134) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:02:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [550/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8915) loss 0.5867 (0.5815) grad_norm 0.2143 (0.2159) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:04:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [550/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8883) loss 0.5559 (0.5810) grad_norm 0.2469 (0.2158) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:04:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 550 training takes 0:05:57 [2024-03-07 10:04:19 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_550.pth saving...... [2024-03-07 10:04:20 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_550.pth saved !!! [2024-03-07 10:04:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [551/800][0/402] eta 0:30:46 lr 0.000025 time 4.5921 (4.5921) loss 0.5715 (0.5715) grad_norm 0.2084 (0.2084) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:05:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [551/800][100/402] eta 0:04:36 lr 0.000025 time 0.8790 (0.9154) loss 0.5472 (0.5798) grad_norm 0.2207 (0.2087) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:07:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [551/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8971) loss 0.5699 (0.5811) grad_norm 0.2850 (0.2133) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:08:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [551/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8909) loss 0.5694 (0.5800) grad_norm 0.2042 (0.2136) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:10:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [551/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8878) loss 0.6061 (0.5800) grad_norm 0.1787 (0.2135) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:10:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 551 training takes 0:05:57 [2024-03-07 10:10:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [552/800][0/402] eta 0:32:06 lr 0.000025 time 4.7922 (4.7922) loss 0.5937 (0.5937) grad_norm 0.2084 (0.2084) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:11:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [552/800][100/402] eta 0:04:37 lr 0.000025 time 0.8789 (0.9181) loss 0.5569 (0.5787) grad_norm 0.2333 (0.2141) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:13:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [552/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8984) loss 0.5749 (0.5793) grad_norm 0.2246 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:14:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [552/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8918) loss 0.5708 (0.5786) grad_norm 0.2005 (0.2137) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:16:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [552/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8884) loss 0.5558 (0.5785) grad_norm 0.1941 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:16:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 552 training takes 0:05:57 [2024-03-07 10:16:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [553/800][0/402] eta 0:30:07 lr 0.000025 time 4.4961 (4.4961) loss 0.5494 (0.5494) grad_norm 0.2166 (0.2166) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:17:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [553/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9141) loss 0.5809 (0.5753) grad_norm 0.2040 (nan) loss_scale 262144.0000 (340008.5545) mem 30609MB [2024-03-07 10:19:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [553/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8964) loss 0.5913 (0.5785) grad_norm 0.1877 (nan) loss_scale 262144.0000 (301269.9701) mem 30609MB [2024-03-07 10:20:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [553/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8905) loss 0.5615 (0.5793) grad_norm 0.2560 (nan) loss_scale 262144.0000 (288271.3090) mem 30609MB [2024-03-07 10:22:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [553/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8874) loss 0.5883 (0.5801) grad_norm 0.2344 (nan) loss_scale 262144.0000 (281755.7706) mem 30609MB [2024-03-07 10:22:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 553 training takes 0:05:56 [2024-03-07 10:22:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [554/800][0/402] eta 0:32:10 lr 0.000025 time 4.8011 (4.8011) loss 0.5770 (0.5770) grad_norm 0.2381 (0.2381) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:23:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [554/800][100/402] eta 0:04:37 lr 0.000025 time 0.8780 (0.9173) loss 0.5392 (0.5779) grad_norm 0.2137 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:25:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [554/800][200/402] eta 0:03:01 lr 0.000025 time 0.8776 (0.8988) loss 0.5624 (0.5788) grad_norm 0.2142 (0.2201) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:26:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [554/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8921) loss 0.5921 (0.5777) grad_norm 0.2240 (0.2194) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:28:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [554/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8888) loss 0.5982 (0.5788) grad_norm 0.2404 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:28:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 554 training takes 0:05:57 [2024-03-07 10:28:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [555/800][0/402] eta 0:31:38 lr 0.000025 time 4.7227 (4.7227) loss 0.5798 (0.5798) grad_norm 0.2331 (0.2331) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:29:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [555/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9170) loss 0.5729 (0.5779) grad_norm 0.1803 (0.2069) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:31:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [555/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8980) loss 0.5504 (0.5790) grad_norm 0.1787 (0.2114) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:32:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [555/800][300/402] eta 0:01:30 lr 0.000025 time 0.8791 (0.8916) loss 0.5673 (0.5792) grad_norm 0.1914 (0.2100) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:34:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [555/800][400/402] eta 0:00:01 lr 0.000025 time 0.8778 (0.8884) loss 0.6002 (0.5796) grad_norm 0.2357 (0.2112) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:34:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 555 training takes 0:05:57 [2024-03-07 10:34:07 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_555.pth saving...... [2024-03-07 10:34:08 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_555.pth saved !!! [2024-03-07 10:34:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [556/800][0/402] eta 0:30:21 lr 0.000025 time 4.5314 (4.5314) loss 0.5651 (0.5651) grad_norm 0.2068 (0.2068) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:35:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [556/800][100/402] eta 0:04:36 lr 0.000025 time 0.8779 (0.9147) loss 0.5539 (0.5773) grad_norm 0.2338 (0.2121) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:37:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [556/800][200/402] eta 0:03:01 lr 0.000025 time 0.8792 (0.8970) loss 0.5669 (0.5788) grad_norm 0.2263 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:38:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [556/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8911) loss 0.5712 (0.5788) grad_norm 0.2113 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:40:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [556/800][400/402] eta 0:00:01 lr 0.000025 time 0.8780 (0.8880) loss 0.5310 (0.5790) grad_norm 0.2480 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:40:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 556 training takes 0:05:57 [2024-03-07 10:40:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [557/800][0/402] eta 0:31:15 lr 0.000025 time 4.6651 (4.6651) loss 0.5647 (0.5647) grad_norm 0.2189 (0.2189) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:41:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [557/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9164) loss 0.5616 (0.5778) grad_norm 0.2362 (0.2135) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:43:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [557/800][200/402] eta 0:03:01 lr 0.000025 time 0.8801 (0.8976) loss 0.5996 (0.5792) grad_norm 0.2248 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:44:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [557/800][300/402] eta 0:01:30 lr 0.000025 time 0.8798 (0.8914) loss 0.6035 (0.5787) grad_norm 0.2038 (0.2148) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:46:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [557/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8881) loss 0.5881 (0.5789) grad_norm 0.2332 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:46:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 557 training takes 0:05:57 [2024-03-07 10:46:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [558/800][0/402] eta 0:30:47 lr 0.000025 time 4.5957 (4.5957) loss 0.5666 (0.5666) grad_norm 0.2305 (0.2305) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 10:47:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [558/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9152) loss 0.5738 (0.5792) grad_norm 0.2039 (0.2124) loss_scale 524288.0000 (472378.2970) mem 30609MB [2024-03-07 10:49:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [558/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8971) loss 0.5802 (0.5796) grad_norm 0.2224 (0.2099) loss_scale 524288.0000 (498204.0199) mem 30609MB [2024-03-07 10:50:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [558/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8909) loss 0.5957 (0.5790) grad_norm 0.2260 (0.2135) loss_scale 524288.0000 (506869.7940) mem 30609MB [2024-03-07 10:51:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [558/800][400/402] eta 0:00:01 lr 0.000025 time 0.8802 (0.8879) loss 0.5738 (0.5788) grad_norm 0.1912 (0.2167) loss_scale 524288.0000 (511213.4863) mem 30609MB [2024-03-07 10:52:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 558 training takes 0:05:57 [2024-03-07 10:52:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [559/800][0/402] eta 0:30:11 lr 0.000025 time 4.5053 (4.5053) loss 0.5654 (0.5654) grad_norm 0.2111 (0.2111) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:53:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [559/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9160) loss 0.6119 (0.5785) grad_norm 0.2088 (0.2089) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:55:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [559/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8975) loss 0.5932 (0.5787) grad_norm 0.2098 (0.2106) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:56:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [559/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8914) loss 0.5665 (0.5779) grad_norm 0.2236 (0.2143) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:57:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [559/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8882) loss 0.5614 (0.5793) grad_norm 0.2162 (0.2139) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:57:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 559 training takes 0:05:57 [2024-03-07 10:58:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [560/800][0/402] eta 0:31:52 lr 0.000025 time 4.7570 (4.7570) loss 0.5958 (0.5958) grad_norm 0.1887 (0.1887) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 10:59:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [560/800][100/402] eta 0:04:36 lr 0.000025 time 0.8812 (0.9172) loss 0.5713 (0.5797) grad_norm 0.2394 (0.2150) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:00:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [560/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8981) loss 0.5970 (0.5797) grad_norm 0.2436 (0.2126) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:02:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [560/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8917) loss 0.5865 (0.5805) grad_norm 0.2099 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:03:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [560/800][400/402] eta 0:00:01 lr 0.000025 time 0.8777 (0.8885) loss 0.5889 (0.5795) grad_norm 0.2246 (0.2144) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:03:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 560 training takes 0:05:57 [2024-03-07 11:03:55 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_560.pth saving...... [2024-03-07 11:03:57 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_560.pth saved !!! [2024-03-07 11:04:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [561/800][0/402] eta 0:31:03 lr 0.000025 time 4.6365 (4.6365) loss 0.5585 (0.5585) grad_norm 0.2225 (0.2225) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:05:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [561/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9157) loss 0.5442 (0.5784) grad_norm 0.1975 (0.2151) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:06:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [561/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8976) loss 0.5932 (0.5774) grad_norm 0.2021 (0.2135) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:08:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [561/800][300/402] eta 0:01:30 lr 0.000025 time 0.8779 (0.8913) loss 0.5952 (0.5781) grad_norm 0.2149 (0.2139) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:09:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [561/800][400/402] eta 0:00:01 lr 0.000025 time 0.8776 (0.8881) loss 0.6066 (0.5788) grad_norm 0.1909 (0.2133) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:09:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 561 training takes 0:05:57 [2024-03-07 11:09:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [562/800][0/402] eta 0:32:18 lr 0.000025 time 4.8226 (4.8226) loss 0.5914 (0.5914) grad_norm 0.1792 (0.1792) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:11:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [562/800][100/402] eta 0:04:37 lr 0.000025 time 0.8792 (0.9178) loss 0.5494 (0.5783) grad_norm 0.2537 (0.2126) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:12:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [562/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8984) loss 0.5574 (0.5786) grad_norm 0.2257 (0.2122) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:14:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [562/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8919) loss 0.5367 (0.5795) grad_norm 0.2360 (0.2105) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:15:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [562/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8886) loss 0.5726 (0.5793) grad_norm 0.2083 (0.2115) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:15:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 562 training takes 0:05:57 [2024-03-07 11:15:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [563/800][0/402] eta 0:31:17 lr 0.000025 time 4.6714 (4.6714) loss 0.5702 (0.5702) grad_norm 0.2056 (0.2056) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:17:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [563/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9160) loss 0.5432 (0.5768) grad_norm 0.2682 (0.2180) loss_scale 1048576.0000 (996666.2970) mem 30609MB [2024-03-07 11:18:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [563/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8974) loss 0.5746 (0.5784) grad_norm 0.2151 (inf) loss_scale 524288.0000 (949456.8756) mem 30609MB [2024-03-07 11:20:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [563/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8914) loss 0.5670 (0.5795) grad_norm 0.2144 (inf) loss_scale 524288.0000 (808204.7575) mem 30609MB [2024-03-07 11:21:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [563/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8881) loss 0.5622 (0.5790) grad_norm 0.2452 (inf) loss_scale 524288.0000 (737402.5736) mem 30609MB [2024-03-07 11:21:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 563 training takes 0:05:57 [2024-03-07 11:21:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [564/800][0/402] eta 0:32:19 lr 0.000025 time 4.8252 (4.8252) loss 0.5629 (0.5629) grad_norm 0.2116 (0.2116) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:23:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [564/800][100/402] eta 0:04:37 lr 0.000025 time 0.8782 (0.9177) loss 0.5545 (0.5790) grad_norm 0.2339 (0.2187) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:24:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [564/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8982) loss 0.6038 (0.5785) grad_norm 0.1984 (nan) loss_scale 262144.0000 (474728.4378) mem 30609MB [2024-03-07 11:26:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [564/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8917) loss 0.5671 (0.5785) grad_norm 0.1808 (nan) loss_scale 262144.0000 (404102.3787) mem 30609MB [2024-03-07 11:27:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [564/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8884) loss 0.5586 (0.5785) grad_norm 0.2239 (nan) loss_scale 262144.0000 (368701.2868) mem 30609MB [2024-03-07 11:27:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 564 training takes 0:05:57 [2024-03-07 11:27:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [565/800][0/402] eta 0:31:46 lr 0.000025 time 4.7414 (4.7414) loss 0.5542 (0.5542) grad_norm 0.2360 (0.2360) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:29:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [565/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9167) loss 0.5679 (0.5802) grad_norm 0.2137 (0.2158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:30:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [565/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8977) loss 0.5819 (0.5805) grad_norm 0.1941 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:32:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [565/800][300/402] eta 0:01:30 lr 0.000025 time 0.8796 (0.8915) loss 0.5684 (0.5795) grad_norm 0.2187 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:33:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [565/800][400/402] eta 0:00:01 lr 0.000025 time 0.8778 (0.8883) loss 0.5655 (0.5801) grad_norm 0.2706 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:33:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 565 training takes 0:05:57 [2024-03-07 11:33:44 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_565.pth saving...... [2024-03-07 11:33:45 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_565.pth saved !!! [2024-03-07 11:33:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [566/800][0/402] eta 0:34:51 lr 0.000025 time 5.2016 (5.2016) loss 0.5427 (0.5427) grad_norm 0.2045 (0.2045) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:35:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [566/800][100/402] eta 0:04:38 lr 0.000025 time 0.8799 (0.9213) loss 0.6074 (0.5778) grad_norm 0.2149 (0.2195) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:36:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [566/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.9001) loss 0.5588 (0.5784) grad_norm 0.2041 (0.2180) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:38:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [566/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8930) loss 0.5669 (0.5792) grad_norm 0.2214 (0.2185) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:39:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [566/800][400/402] eta 0:00:01 lr 0.000025 time 0.8779 (0.8894) loss 0.5805 (0.5796) grad_norm 0.1960 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:39:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 566 training takes 0:05:57 [2024-03-07 11:39:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [567/800][0/402] eta 0:33:44 lr 0.000025 time 5.0371 (5.0371) loss 0.5967 (0.5967) grad_norm 0.2318 (0.2318) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:41:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [567/800][100/402] eta 0:04:37 lr 0.000025 time 0.8789 (0.9199) loss 0.5711 (0.5781) grad_norm 0.2423 (0.2105) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:42:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [567/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8992) loss 0.5753 (0.5792) grad_norm 0.1935 (0.2113) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:44:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [567/800][300/402] eta 0:01:31 lr 0.000025 time 0.8783 (0.8927) loss 0.5735 (0.5784) grad_norm 0.1974 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:45:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [567/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8891) loss 0.6058 (0.5789) grad_norm 0.2414 (0.2142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:45:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 567 training takes 0:05:57 [2024-03-07 11:45:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [568/800][0/402] eta 0:32:49 lr 0.000025 time 4.8981 (4.8981) loss 0.5605 (0.5605) grad_norm 0.2630 (0.2630) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:47:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [568/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9195) loss 0.6092 (0.5791) grad_norm 0.2115 (0.2124) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:48:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [568/800][200/402] eta 0:03:01 lr 0.000025 time 0.8794 (0.8993) loss 0.5804 (0.5790) grad_norm 0.2235 (0.2114) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:50:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [568/800][300/402] eta 0:01:31 lr 0.000025 time 0.8781 (0.8925) loss 0.5807 (0.5802) grad_norm 0.1822 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:51:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [568/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8891) loss 0.5944 (0.5807) grad_norm 0.2042 (0.2143) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:51:39 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 568 training takes 0:05:57 [2024-03-07 11:51:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [569/800][0/402] eta 0:34:26 lr 0.000025 time 5.1400 (5.1400) loss 0.5683 (0.5683) grad_norm 0.2050 (0.2050) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:53:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [569/800][100/402] eta 0:04:38 lr 0.000025 time 0.8788 (0.9211) loss 0.5651 (0.5774) grad_norm 0.2267 (0.2179) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 11:54:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [569/800][200/402] eta 0:03:01 lr 0.000025 time 0.8798 (0.9001) loss 0.5922 (0.5788) grad_norm 0.2316 (0.2194) loss_scale 524288.0000 (324745.5522) mem 30609MB [2024-03-07 11:56:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [569/800][300/402] eta 0:01:31 lr 0.000025 time 0.8788 (0.8931) loss 0.6052 (0.5777) grad_norm 0.1859 (0.2168) loss_scale 524288.0000 (391038.7243) mem 30609MB [2024-03-07 11:57:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [569/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8895) loss 0.5872 (0.5779) grad_norm 0.2064 (0.2154) loss_scale 524288.0000 (424267.9701) mem 30609MB [2024-03-07 11:57:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 569 training takes 0:05:57 [2024-03-07 11:57:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [570/800][0/402] eta 0:33:42 lr 0.000025 time 5.0313 (5.0313) loss 0.6070 (0.6070) grad_norm 0.1716 (0.1716) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 11:59:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [570/800][100/402] eta 0:04:37 lr 0.000025 time 0.8788 (0.9199) loss 0.5366 (0.5793) grad_norm 0.2131 (0.2126) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:00:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [570/800][200/402] eta 0:03:01 lr 0.000025 time 0.8795 (0.8999) loss 0.5754 (0.5797) grad_norm 0.1966 (0.2130) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:02:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [570/800][300/402] eta 0:01:31 lr 0.000025 time 0.8788 (0.8929) loss 0.5563 (0.5802) grad_norm 0.1827 (0.2115) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:03:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [570/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8894) loss 0.5640 (0.5797) grad_norm 0.1904 (0.2119) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:03:34 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 570 training takes 0:05:57 [2024-03-07 12:03:34 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_570.pth saving...... [2024-03-07 12:03:36 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_570.pth saved !!! [2024-03-07 12:03:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [571/800][0/402] eta 0:35:01 lr 0.000025 time 5.2274 (5.2274) loss 0.5739 (0.5739) grad_norm 0.2605 (0.2605) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:05:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [571/800][100/402] eta 0:04:38 lr 0.000025 time 0.8793 (0.9219) loss 0.5333 (0.5765) grad_norm 0.1808 (0.2128) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:06:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [571/800][200/402] eta 0:03:01 lr 0.000025 time 0.8807 (0.9005) loss 0.5703 (0.5791) grad_norm 0.2064 (0.2135) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:08:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [571/800][300/402] eta 0:01:31 lr 0.000025 time 0.8793 (0.8933) loss 0.5659 (0.5783) grad_norm 0.1906 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:09:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [571/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8896) loss 0.5603 (0.5788) grad_norm 0.2247 (inf) loss_scale 262144.0000 (514482.1147) mem 30609MB [2024-03-07 12:09:34 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 571 training takes 0:05:57 [2024-03-07 12:09:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [572/800][0/402] eta 0:33:19 lr 0.000025 time 4.9751 (4.9751) loss 0.5708 (0.5708) grad_norm 0.2166 (0.2166) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:11:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [572/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9204) loss 0.5671 (0.5774) grad_norm 0.2240 (0.2117) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:12:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [572/800][200/402] eta 0:03:01 lr 0.000025 time 0.8797 (0.8997) loss 0.6109 (0.5807) grad_norm 0.1868 (0.2092) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:14:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [572/800][300/402] eta 0:01:31 lr 0.000025 time 0.8782 (0.8928) loss 0.5795 (0.5799) grad_norm 0.2184 (0.2100) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:15:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [572/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8894) loss 0.5802 (0.5796) grad_norm 0.2490 (0.2112) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:15:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 572 training takes 0:05:57 [2024-03-07 12:15:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [573/800][0/402] eta 0:31:48 lr 0.000025 time 4.7469 (4.7469) loss 0.5847 (0.5847) grad_norm 0.2334 (0.2334) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:17:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [573/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9174) loss 0.5797 (0.5786) grad_norm 0.1869 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:18:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [573/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8981) loss 0.5951 (0.5790) grad_norm 0.2134 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:20:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [573/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8917) loss 0.5781 (0.5789) grad_norm 0.2005 (0.2175) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:21:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [573/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8884) loss 0.5597 (0.5785) grad_norm 0.2163 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:21:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 573 training takes 0:05:57 [2024-03-07 12:21:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [574/800][0/402] eta 0:32:42 lr 0.000025 time 4.8827 (4.8827) loss 0.5608 (0.5608) grad_norm 0.2119 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:23:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [574/800][100/402] eta 0:04:37 lr 0.000025 time 0.8790 (0.9185) loss 0.5717 (0.5791) grad_norm 0.2037 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:24:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [574/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8991) loss 0.5850 (0.5786) grad_norm 0.1774 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:25:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [574/800][300/402] eta 0:01:31 lr 0.000025 time 0.8786 (0.8924) loss 0.5700 (0.5782) grad_norm 0.2853 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:27:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [574/800][400/402] eta 0:00:01 lr 0.000025 time 0.8774 (0.8889) loss 0.6221 (0.5785) grad_norm 0.2178 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:27:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 574 training takes 0:05:57 [2024-03-07 12:27:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [575/800][0/402] eta 0:33:56 lr 0.000025 time 5.0666 (5.0666) loss 0.5680 (0.5680) grad_norm 0.2151 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:29:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [575/800][100/402] eta 0:04:38 lr 0.000025 time 0.8787 (0.9209) loss 0.5705 (0.5765) grad_norm 0.1901 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:30:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [575/800][200/402] eta 0:03:01 lr 0.000025 time 0.8776 (0.9000) loss 0.5478 (0.5787) grad_norm 0.2118 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:31:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [575/800][300/402] eta 0:01:31 lr 0.000025 time 0.8792 (0.8930) loss 0.5865 (0.5787) grad_norm 0.1915 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:33:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [575/800][400/402] eta 0:00:01 lr 0.000025 time 0.8778 (0.8894) loss 0.5755 (0.5782) grad_norm 0.1714 (0.2130) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:33:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 575 training takes 0:05:57 [2024-03-07 12:33:25 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_575.pth saving...... [2024-03-07 12:33:27 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_575.pth saved !!! [2024-03-07 12:33:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [576/800][0/402] eta 0:28:04 lr 0.000025 time 4.1914 (4.1914) loss 0.5756 (0.5756) grad_norm 0.2045 (0.2045) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:34:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [576/800][100/402] eta 0:04:35 lr 0.000025 time 0.8804 (0.9114) loss 0.5600 (0.5794) grad_norm 0.2024 (0.2129) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:36:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [576/800][200/402] eta 0:03:00 lr 0.000025 time 0.8800 (0.8952) loss 0.5454 (0.5803) grad_norm 0.2307 (0.2273) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:37:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [576/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8899) loss 0.6019 (0.5796) grad_norm 0.2531 (0.2232) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:39:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [576/800][400/402] eta 0:00:01 lr 0.000025 time 0.8782 (0.8871) loss 0.5617 (0.5789) grad_norm 0.1830 (0.2211) loss_scale 524288.0000 (278487.1421) mem 30609MB [2024-03-07 12:39:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 576 training takes 0:05:56 [2024-03-07 12:39:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [577/800][0/402] eta 0:32:58 lr 0.000025 time 4.9205 (4.9205) loss 0.5585 (0.5585) grad_norm 0.2142 (0.2142) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:40:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [577/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9187) loss 0.5641 (0.5797) grad_norm 0.2144 (0.2110) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:42:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [577/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8992) loss 0.6029 (0.5800) grad_norm 0.2083 (0.2110) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:43:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [577/800][300/402] eta 0:01:31 lr 0.000025 time 0.8780 (0.8923) loss 0.5607 (0.5795) grad_norm 0.2236 (0.2118) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 12:45:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [577/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8888) loss 0.5749 (0.5795) grad_norm 0.2111 (nan) loss_scale 262144.0000 (478527.2020) mem 30609MB [2024-03-07 12:45:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 577 training takes 0:05:57 [2024-03-07 12:45:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [578/800][0/402] eta 0:35:59 lr 0.000025 time 5.3711 (5.3711) loss 0.5978 (0.5978) grad_norm 0.2029 (0.2029) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:46:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [578/800][100/402] eta 0:04:38 lr 0.000025 time 0.8792 (0.9235) loss 0.5705 (0.5787) grad_norm 0.2515 (0.2111) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:48:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [578/800][200/402] eta 0:03:02 lr 0.000025 time 0.8787 (0.9014) loss 0.5835 (0.5775) grad_norm 0.2239 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:49:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [578/800][300/402] eta 0:01:31 lr 0.000025 time 0.8789 (0.8939) loss 0.5659 (0.5768) grad_norm 0.2183 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:51:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [578/800][400/402] eta 0:00:01 lr 0.000025 time 0.8775 (0.8902) loss 0.5664 (0.5777) grad_norm 0.2230 (0.2135) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:51:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 578 training takes 0:05:58 [2024-03-07 12:51:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [579/800][0/402] eta 0:32:02 lr 0.000025 time 4.7833 (4.7833) loss 0.5791 (0.5791) grad_norm 0.2071 (0.2071) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:52:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [579/800][100/402] eta 0:04:37 lr 0.000025 time 0.8779 (0.9177) loss 0.5664 (0.5772) grad_norm 0.2013 (0.2110) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:54:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [579/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8982) loss 0.5378 (0.5765) grad_norm 0.2267 (0.2109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:55:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [579/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8919) loss 0.6015 (0.5779) grad_norm 0.2107 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:57:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [579/800][400/402] eta 0:00:01 lr 0.000025 time 0.8774 (0.8886) loss 0.5732 (0.5789) grad_norm 0.2098 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:57:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 579 training takes 0:05:57 [2024-03-07 12:57:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [580/800][0/402] eta 0:33:52 lr 0.000025 time 5.0566 (5.0566) loss 0.5810 (0.5810) grad_norm 0.1914 (0.1914) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 12:58:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [580/800][100/402] eta 0:04:37 lr 0.000025 time 0.8789 (0.9201) loss 0.5745 (0.5773) grad_norm 0.2065 (0.2139) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:00:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [580/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8996) loss 0.5577 (0.5774) grad_norm 0.2068 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:01:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [580/800][300/402] eta 0:01:31 lr 0.000025 time 0.8789 (0.8927) loss 0.5758 (0.5782) grad_norm 0.1911 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:03:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [580/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8892) loss 0.5877 (0.5784) grad_norm 0.2203 (0.2153) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:03:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 580 training takes 0:05:57 [2024-03-07 13:03:14 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_580.pth saving...... [2024-03-07 13:03:16 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_580.pth saved !!! [2024-03-07 13:03:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [581/800][0/402] eta 0:33:53 lr 0.000025 time 5.0596 (5.0596) loss 0.5536 (0.5536) grad_norm 0.1833 (0.1833) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:04:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [581/800][100/402] eta 0:04:38 lr 0.000025 time 0.8781 (0.9206) loss 0.5763 (0.5788) grad_norm 0.2156 (0.2132) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:06:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [581/800][200/402] eta 0:03:01 lr 0.000025 time 0.8794 (0.8998) loss 0.5923 (0.5777) grad_norm 0.1988 (0.2140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:07:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [581/800][300/402] eta 0:01:31 lr 0.000025 time 0.8778 (0.8929) loss 0.5584 (0.5769) grad_norm 0.2126 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:09:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [581/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8893) loss 0.5968 (0.5783) grad_norm 0.2118 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:09:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 581 training takes 0:05:57 [2024-03-07 13:09:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [582/800][0/402] eta 0:33:58 lr 0.000025 time 5.0716 (5.0716) loss 0.6141 (0.6141) grad_norm 0.2023 (0.2023) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:10:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [582/800][100/402] eta 0:04:37 lr 0.000025 time 0.8784 (0.9203) loss 0.5655 (0.5831) grad_norm 0.2318 (0.2186) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:12:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [582/800][200/402] eta 0:03:01 lr 0.000025 time 0.8795 (0.8998) loss 0.5882 (0.5820) grad_norm 0.1987 (0.2202) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:13:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [582/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8928) loss 0.5763 (0.5816) grad_norm 0.2268 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:15:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [582/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8892) loss 0.6145 (0.5807) grad_norm 0.1921 (0.2162) loss_scale 524288.0000 (314442.0549) mem 30609MB [2024-03-07 13:15:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 582 training takes 0:05:57 [2024-03-07 13:15:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [583/800][0/402] eta 0:34:02 lr 0.000025 time 5.0811 (5.0811) loss 0.6288 (0.6288) grad_norm 0.2418 (0.2418) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:16:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [583/800][100/402] eta 0:04:37 lr 0.000025 time 0.8785 (0.9204) loss 0.5342 (0.5785) grad_norm 0.2170 (0.2167) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:18:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [583/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.9000) loss 0.5939 (0.5795) grad_norm 0.2068 (0.2144) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:19:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [583/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8928) loss 0.5511 (0.5786) grad_norm 0.2114 (0.2157) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:21:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [583/800][400/402] eta 0:00:01 lr 0.000025 time 0.8776 (0.8892) loss 0.5545 (0.5793) grad_norm 0.2009 (0.2161) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:21:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 583 training takes 0:05:57 [2024-03-07 13:21:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [584/800][0/402] eta 0:33:06 lr 0.000025 time 4.9425 (4.9425) loss 0.5712 (0.5712) grad_norm 0.2209 (0.2209) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:22:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [584/800][100/402] eta 0:04:37 lr 0.000025 time 0.8781 (0.9189) loss 0.6486 (0.5798) grad_norm 0.1890 (0.2143) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:24:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [584/800][200/402] eta 0:03:01 lr 0.000025 time 0.8810 (0.8993) loss 0.5663 (0.5779) grad_norm 0.2194 (0.2134) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:25:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [584/800][300/402] eta 0:01:31 lr 0.000025 time 0.8786 (0.8925) loss 0.5695 (0.5787) grad_norm 0.2192 (0.2149) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:27:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [584/800][400/402] eta 0:00:01 lr 0.000025 time 0.8782 (0.8890) loss 0.5800 (0.5787) grad_norm 0.1934 (0.2138) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:27:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 584 training takes 0:05:57 [2024-03-07 13:27:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [585/800][0/402] eta 0:35:05 lr 0.000025 time 5.2364 (5.2364) loss 0.5504 (0.5504) grad_norm 0.1985 (0.1985) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:28:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [585/800][100/402] eta 0:04:38 lr 0.000025 time 0.8784 (0.9220) loss 0.5838 (0.5784) grad_norm 0.1984 (0.2110) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:30:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [585/800][200/402] eta 0:03:01 lr 0.000025 time 0.8792 (0.9005) loss 0.5994 (0.5781) grad_norm 0.1979 (0.2136) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:31:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [585/800][300/402] eta 0:01:31 lr 0.000025 time 0.8786 (0.8934) loss 0.5586 (0.5776) grad_norm 0.2092 (0.2153) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:33:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [585/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8897) loss 0.6111 (0.5785) grad_norm 0.2163 (0.2157) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:33:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 585 training takes 0:05:57 [2024-03-07 13:33:05 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_585.pth saving...... [2024-03-07 13:33:07 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_585.pth saved !!! [2024-03-07 13:33:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [586/800][0/402] eta 0:34:22 lr 0.000025 time 5.1309 (5.1309) loss 0.5841 (0.5841) grad_norm 0.2395 (0.2395) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:34:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [586/800][100/402] eta 0:04:38 lr 0.000025 time 0.8786 (0.9209) loss 0.5818 (0.5809) grad_norm 0.2144 (0.2199) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 13:36:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [586/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8999) loss 0.5993 (0.5802) grad_norm 0.1738 (nan) loss_scale 262144.0000 (494291.4229) mem 30609MB [2024-03-07 13:37:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [586/800][300/402] eta 0:01:31 lr 0.000025 time 0.8784 (0.8931) loss 0.5736 (0.5790) grad_norm 0.2166 (nan) loss_scale 262144.0000 (417166.0332) mem 30609MB [2024-03-07 13:39:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [586/800][400/402] eta 0:00:01 lr 0.000025 time 0.8773 (0.8895) loss 0.5907 (0.5784) grad_norm 0.1872 (nan) loss_scale 262144.0000 (378507.1721) mem 30609MB [2024-03-07 13:39:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 586 training takes 0:05:57 [2024-03-07 13:39:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [587/800][0/402] eta 0:34:53 lr 0.000025 time 5.2069 (5.2069) loss 0.6047 (0.6047) grad_norm 0.1728 (0.1728) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:40:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [587/800][100/402] eta 0:04:38 lr 0.000025 time 0.8793 (0.9215) loss 0.5752 (0.5833) grad_norm 0.2034 (0.2083) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:42:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [587/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.9002) loss 0.5767 (0.5803) grad_norm 0.2037 (0.2088) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:43:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [587/800][300/402] eta 0:01:31 lr 0.000025 time 0.8800 (0.8934) loss 0.5995 (0.5797) grad_norm 0.2345 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:45:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [587/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8897) loss 0.5583 (0.5793) grad_norm 0.2350 (0.2143) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:45:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 587 training takes 0:05:57 [2024-03-07 13:45:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [588/800][0/402] eta 0:31:59 lr 0.000025 time 4.7761 (4.7761) loss 0.5677 (0.5677) grad_norm 0.2119 (0.2119) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:46:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [588/800][100/402] eta 0:04:37 lr 0.000025 time 0.8795 (0.9175) loss 0.5787 (0.5786) grad_norm 0.2325 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:48:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [588/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8983) loss 0.5483 (0.5786) grad_norm 0.2605 (0.2125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:49:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [588/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8918) loss 0.5897 (0.5786) grad_norm 0.2758 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:50:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [588/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8886) loss 0.5779 (0.5785) grad_norm 0.1936 (0.2140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:51:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 588 training takes 0:05:57 [2024-03-07 13:51:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [589/800][0/402] eta 0:31:55 lr 0.000025 time 4.7638 (4.7638) loss 0.5445 (0.5445) grad_norm 0.1940 (0.1940) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:52:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [589/800][100/402] eta 0:04:37 lr 0.000025 time 0.8783 (0.9173) loss 0.5136 (0.5791) grad_norm 0.2622 (0.2107) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:54:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [589/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8981) loss 0.5698 (0.5796) grad_norm 0.2093 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:55:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [589/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8918) loss 0.5811 (0.5788) grad_norm 0.1928 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:56:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [589/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8885) loss 0.5815 (0.5786) grad_norm 0.1925 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:56:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 589 training takes 0:05:57 [2024-03-07 13:57:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [590/800][0/402] eta 0:34:24 lr 0.000025 time 5.1359 (5.1359) loss 0.5768 (0.5768) grad_norm 0.1994 (0.1994) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:58:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [590/800][100/402] eta 0:04:38 lr 0.000025 time 0.8789 (0.9218) loss 0.5493 (0.5792) grad_norm 0.2202 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 13:59:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [590/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.9005) loss 0.6022 (0.5788) grad_norm 0.1859 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:01:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [590/800][300/402] eta 0:01:31 lr 0.000025 time 0.8782 (0.8933) loss 0.5855 (0.5790) grad_norm 0.2065 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:02:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [590/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8896) loss 0.5975 (0.5791) grad_norm 0.2272 (0.2175) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:02:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 590 training takes 0:05:57 [2024-03-07 14:02:56 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_590.pth saving...... [2024-03-07 14:02:57 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_590.pth saved !!! [2024-03-07 14:03:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [591/800][0/402] eta 0:28:46 lr 0.000025 time 4.2953 (4.2953) loss 0.5726 (0.5726) grad_norm 0.1696 (0.1696) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:04:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [591/800][100/402] eta 0:04:35 lr 0.000025 time 0.8792 (0.9130) loss 0.5630 (0.5806) grad_norm 0.1904 (0.2070) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:05:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [591/800][200/402] eta 0:03:00 lr 0.000025 time 0.8798 (0.8959) loss 0.5738 (0.5798) grad_norm 0.2624 (0.2116) loss_scale 524288.0000 (305182.5672) mem 30609MB [2024-03-07 14:07:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [591/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8902) loss 0.5730 (0.5790) grad_norm 0.2437 (0.2141) loss_scale 524288.0000 (377975.0698) mem 30609MB [2024-03-07 14:08:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [591/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8873) loss 0.5825 (0.5792) grad_norm 0.1952 (0.2150) loss_scale 524288.0000 (414462.0848) mem 30609MB [2024-03-07 14:08:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 591 training takes 0:05:56 [2024-03-07 14:08:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [592/800][0/402] eta 0:30:43 lr 0.000025 time 4.5868 (4.5868) loss 0.5887 (0.5887) grad_norm 0.1942 (0.1942) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:10:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [592/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9157) loss 0.5570 (0.5813) grad_norm 0.2208 (0.2121) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:11:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [592/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8972) loss 0.5830 (0.5791) grad_norm 0.2177 (0.2115) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:13:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [592/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8910) loss 0.5499 (0.5791) grad_norm 0.2057 (0.2126) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:14:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [592/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8878) loss 0.5713 (0.5784) grad_norm 0.2357 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:14:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 592 training takes 0:05:57 [2024-03-07 14:14:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [593/800][0/402] eta 0:30:17 lr 0.000025 time 4.5214 (4.5214) loss 0.5926 (0.5926) grad_norm 0.1993 (0.1993) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:16:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [593/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9146) loss 0.5949 (0.5810) grad_norm 0.2386 (0.2134) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:17:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [593/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5919 (0.5805) grad_norm 0.2137 (0.2115) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:19:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [593/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8907) loss 0.5535 (0.5793) grad_norm 0.2295 (0.2129) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:20:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [593/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8876) loss 0.5835 (0.5791) grad_norm 0.1950 (0.2128) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:20:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 593 training takes 0:05:56 [2024-03-07 14:20:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [594/800][0/402] eta 0:30:17 lr 0.000025 time 4.5199 (4.5199) loss 0.5980 (0.5980) grad_norm 0.2126 (0.2126) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:22:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [594/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9146) loss 0.5738 (0.5783) grad_norm 0.2384 (0.2139) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:23:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [594/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8967) loss 0.5927 (0.5785) grad_norm 0.2150 (0.2128) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:25:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [594/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8907) loss 0.5706 (0.5779) grad_norm 0.1980 (0.2121) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:26:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [594/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8876) loss 0.5626 (0.5784) grad_norm 0.2297 (0.2123) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:26:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 594 training takes 0:05:56 [2024-03-07 14:26:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [595/800][0/402] eta 0:30:53 lr 0.000025 time 4.6112 (4.6112) loss 0.5705 (0.5705) grad_norm 0.2099 (0.2099) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 14:28:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [595/800][100/402] eta 0:04:36 lr 0.000025 time 0.8805 (0.9153) loss 0.5267 (0.5805) grad_norm 0.2015 (nan) loss_scale 262144.0000 (314053.7030) mem 30609MB [2024-03-07 14:29:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [595/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8970) loss 0.5708 (0.5795) grad_norm 0.1910 (nan) loss_scale 262144.0000 (288227.9801) mem 30609MB [2024-03-07 14:31:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [595/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8909) loss 0.5863 (0.5791) grad_norm 0.1943 (nan) loss_scale 262144.0000 (279562.2060) mem 30609MB [2024-03-07 14:32:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [595/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8878) loss 0.6004 (0.5784) grad_norm 0.2171 (nan) loss_scale 262144.0000 (275218.5137) mem 30609MB [2024-03-07 14:32:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 595 training takes 0:05:57 [2024-03-07 14:32:42 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_595.pth saving...... [2024-03-07 14:32:44 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_595.pth saved !!! [2024-03-07 14:32:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [596/800][0/402] eta 0:30:47 lr 0.000025 time 4.5957 (4.5957) loss 0.5603 (0.5603) grad_norm 0.2294 (0.2294) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:34:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [596/800][100/402] eta 0:04:36 lr 0.000025 time 0.8775 (0.9155) loss 0.5988 (0.5757) grad_norm 0.1915 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:35:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [596/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8974) loss 0.5693 (0.5771) grad_norm 0.2513 (0.2148) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:37:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [596/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8911) loss 0.5967 (0.5773) grad_norm 0.1902 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:38:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [596/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8879) loss 0.5926 (0.5782) grad_norm 0.2181 (0.2154) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:38:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 596 training takes 0:05:57 [2024-03-07 14:38:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [597/800][0/402] eta 0:30:45 lr 0.000025 time 4.5911 (4.5911) loss 0.5757 (0.5757) grad_norm 0.2213 (0.2213) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:40:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [597/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9152) loss 0.5700 (0.5795) grad_norm 0.2272 (0.2092) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:41:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [597/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8969) loss 0.6019 (0.5802) grad_norm 0.2242 (0.2121) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:43:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [597/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8908) loss 0.5752 (0.5798) grad_norm 0.2219 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:44:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [597/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.5687 (0.5794) grad_norm 0.2139 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:44:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 597 training takes 0:05:56 [2024-03-07 14:44:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [598/800][0/402] eta 0:30:12 lr 0.000025 time 4.5089 (4.5089) loss 0.5605 (0.5605) grad_norm 0.2617 (0.2617) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:46:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [598/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9148) loss 0.5890 (0.5792) grad_norm 0.2534 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:47:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [598/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8968) loss 0.5817 (0.5789) grad_norm 0.2276 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:49:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [598/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8908) loss 0.5551 (0.5786) grad_norm 0.2064 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:50:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [598/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8876) loss 0.5338 (0.5797) grad_norm 0.2020 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:50:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 598 training takes 0:05:56 [2024-03-07 14:50:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [599/800][0/402] eta 0:30:40 lr 0.000025 time 4.5790 (4.5790) loss 0.5620 (0.5620) grad_norm 0.2495 (0.2495) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:52:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [599/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9151) loss 0.5931 (0.5836) grad_norm 0.2380 (0.2183) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:53:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [599/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8969) loss 0.6187 (0.5806) grad_norm 0.2181 (0.2208) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:55:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [599/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8908) loss 0.5660 (0.5799) grad_norm 0.2078 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:56:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [599/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8876) loss 0.5621 (0.5788) grad_norm 0.2374 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:56:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 599 training takes 0:05:56 [2024-03-07 14:56:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [600/800][0/402] eta 0:30:28 lr 0.000025 time 4.5491 (4.5491) loss 0.6042 (0.6042) grad_norm 0.2034 (0.2034) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 14:58:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [600/800][100/402] eta 0:04:36 lr 0.000025 time 0.8801 (0.9148) loss 0.5913 (0.5762) grad_norm 0.2090 (0.2127) loss_scale 524288.0000 (498333.1485) mem 30609MB [2024-03-07 14:59:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [600/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8969) loss 0.5938 (0.5772) grad_norm 0.2044 (0.2119) loss_scale 524288.0000 (511246.0100) mem 30609MB [2024-03-07 15:01:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [600/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8908) loss 0.5674 (0.5767) grad_norm 0.1958 (0.2127) loss_scale 524288.0000 (515578.8970) mem 30609MB [2024-03-07 15:02:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [600/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5563 (0.5775) grad_norm 0.1949 (0.2130) loss_scale 524288.0000 (517750.7431) mem 30609MB [2024-03-07 15:02:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 600 training takes 0:05:57 [2024-03-07 15:02:29 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_600.pth saving...... [2024-03-07 15:02:31 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_600.pth saved !!! [2024-03-07 15:02:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [601/800][0/402] eta 0:30:20 lr 0.000025 time 4.5290 (4.5290) loss 0.5943 (0.5943) grad_norm 0.2091 (0.2091) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 15:04:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [601/800][100/402] eta 0:04:36 lr 0.000025 time 0.8806 (0.9149) loss 0.6093 (0.5749) grad_norm 0.2007 (nan) loss_scale 262144.0000 (319244.6733) mem 30609MB [2024-03-07 15:05:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [601/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8969) loss 0.5633 (0.5770) grad_norm 0.2334 (nan) loss_scale 262144.0000 (290836.3781) mem 30609MB [2024-03-07 15:06:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [601/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8910) loss 0.5485 (0.5774) grad_norm 0.2236 (nan) loss_scale 262144.0000 (281304.0266) mem 30609MB [2024-03-07 15:08:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [601/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8879) loss 0.6145 (0.5773) grad_norm 0.1978 (nan) loss_scale 262144.0000 (276525.9651) mem 30609MB [2024-03-07 15:08:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 601 training takes 0:05:57 [2024-03-07 15:08:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [602/800][0/402] eta 0:30:18 lr 0.000025 time 4.5229 (4.5229) loss 0.5767 (0.5767) grad_norm 0.2186 (0.2186) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:10:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [602/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9145) loss 0.5783 (0.5751) grad_norm 0.1812 (0.2099) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:11:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [602/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8967) loss 0.5725 (0.5765) grad_norm 0.2275 (0.2118) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:12:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [602/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8907) loss 0.5969 (0.5783) grad_norm 0.2358 (0.2127) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:14:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [602/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8876) loss 0.5347 (0.5788) grad_norm 0.2267 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:14:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 602 training takes 0:05:56 [2024-03-07 15:14:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [603/800][0/402] eta 0:30:10 lr 0.000025 time 4.5036 (4.5036) loss 0.5845 (0.5845) grad_norm 0.2724 (0.2724) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:15:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [603/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9144) loss 0.5846 (0.5753) grad_norm 0.1842 (0.2242) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:17:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [603/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8965) loss 0.5725 (0.5772) grad_norm 0.2324 (0.2177) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:18:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [603/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8905) loss 0.5528 (0.5785) grad_norm 0.1927 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:20:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [603/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8874) loss 0.5885 (0.5786) grad_norm 0.2181 (0.2164) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:20:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 603 training takes 0:05:56 [2024-03-07 15:20:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [604/800][0/402] eta 0:30:35 lr 0.000025 time 4.5650 (4.5650) loss 0.5937 (0.5937) grad_norm 0.2034 (0.2034) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:21:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [604/800][100/402] eta 0:04:36 lr 0.000025 time 0.8791 (0.9149) loss 0.5797 (0.5810) grad_norm 0.2072 (0.2143) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:23:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [604/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8968) loss 0.5880 (0.5789) grad_norm 0.1787 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:24:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [604/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8907) loss 0.5840 (0.5784) grad_norm 0.2184 (0.2169) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:26:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [604/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8876) loss 0.5999 (0.5783) grad_norm 0.2075 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:26:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 604 training takes 0:05:56 [2024-03-07 15:26:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [605/800][0/402] eta 0:29:53 lr 0.000025 time 4.4617 (4.4617) loss 0.5872 (0.5872) grad_norm 0.1977 (0.1977) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:27:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [605/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9139) loss 0.5964 (0.5807) grad_norm 0.2584 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:29:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [605/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8964) loss 0.5848 (0.5802) grad_norm 0.2089 (0.2141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:30:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [605/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8904) loss 0.5809 (0.5798) grad_norm 0.1979 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:32:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [605/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8874) loss 0.5543 (0.5798) grad_norm 0.2028 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:32:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 605 training takes 0:05:56 [2024-03-07 15:32:16 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_605.pth saving...... [2024-03-07 15:32:18 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_605.pth saved !!! [2024-03-07 15:32:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [606/800][0/402] eta 0:30:50 lr 0.000025 time 4.6025 (4.6025) loss 0.5901 (0.5901) grad_norm 0.2208 (0.2208) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:33:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [606/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9156) loss 0.5761 (0.5780) grad_norm 0.2732 (0.2101) loss_scale 524288.0000 (493142.1782) mem 30609MB [2024-03-07 15:35:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [606/800][200/402] eta 0:03:01 lr 0.000025 time 0.8801 (0.8971) loss 0.5745 (0.5787) grad_norm 0.2434 (inf) loss_scale 262144.0000 (383434.5075) mem 30609MB [2024-03-07 15:36:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [606/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8910) loss 0.5698 (0.5791) grad_norm 0.2056 (inf) loss_scale 262144.0000 (343138.6578) mem 30609MB [2024-03-07 15:38:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [606/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8878) loss 0.5913 (0.5788) grad_norm 0.2228 (inf) loss_scale 262144.0000 (322940.4888) mem 30609MB [2024-03-07 15:38:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 606 training takes 0:05:57 [2024-03-07 15:38:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [607/800][0/402] eta 0:30:30 lr 0.000025 time 4.5525 (4.5525) loss 0.5724 (0.5724) grad_norm 0.1961 (0.1961) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:39:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [607/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9148) loss 0.5797 (0.5749) grad_norm 0.1973 (0.2117) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:41:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [607/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8969) loss 0.5982 (0.5756) grad_norm 0.2323 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:42:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [607/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8908) loss 0.5797 (0.5771) grad_norm 0.1988 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:44:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [607/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8876) loss 0.5735 (0.5777) grad_norm 0.2521 (0.2144) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:44:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 607 training takes 0:05:56 [2024-03-07 15:44:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [608/800][0/402] eta 0:30:38 lr 0.000025 time 4.5743 (4.5743) loss 0.5861 (0.5861) grad_norm 0.2277 (0.2277) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:45:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [608/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9150) loss 0.5939 (0.5804) grad_norm 0.2036 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:47:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [608/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8968) loss 0.5358 (0.5781) grad_norm 0.1801 (0.2137) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:48:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [608/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8907) loss 0.5727 (0.5772) grad_norm 0.1994 (0.2122) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:50:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [608/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8876) loss 0.5730 (0.5771) grad_norm 0.2096 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:50:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 608 training takes 0:05:56 [2024-03-07 15:50:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [609/800][0/402] eta 0:30:35 lr 0.000025 time 4.5650 (4.5650) loss 0.5779 (0.5779) grad_norm 0.1927 (0.1927) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:51:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [609/800][100/402] eta 0:04:36 lr 0.000025 time 0.8790 (0.9149) loss 0.5594 (0.5814) grad_norm 0.1910 (0.2133) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:53:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [609/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8968) loss 0.5692 (0.5777) grad_norm 0.1974 (0.2138) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:54:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [609/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8909) loss 0.5736 (0.5774) grad_norm 0.2962 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:56:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [609/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5921 (0.5783) grad_norm 0.4505 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:56:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 609 training takes 0:05:57 [2024-03-07 15:56:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [610/800][0/402] eta 0:30:50 lr 0.000025 time 4.6036 (4.6036) loss 0.5952 (0.5952) grad_norm 0.2182 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:57:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [610/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9153) loss 0.5643 (0.5799) grad_norm 0.2205 (0.2190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 15:59:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [610/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8970) loss 0.5959 (0.5786) grad_norm 0.2093 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:00:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [610/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8908) loss 0.5772 (0.5784) grad_norm 0.1842 (0.2153) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:02:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [610/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5725 (0.5777) grad_norm 0.2283 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:02:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 610 training takes 0:05:57 [2024-03-07 16:02:03 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_610.pth saving...... [2024-03-07 16:02:04 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_610.pth saved !!! [2024-03-07 16:02:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [611/800][0/402] eta 0:30:18 lr 0.000025 time 4.5244 (4.5244) loss 0.5530 (0.5530) grad_norm 0.2188 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:03:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [611/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9145) loss 0.5761 (0.5758) grad_norm 0.1923 (0.2115) loss_scale 524288.0000 (277716.9109) mem 30609MB [2024-03-07 16:05:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [611/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8966) loss 0.5866 (0.5771) grad_norm 0.2100 (0.2157) loss_scale 524288.0000 (400389.0945) mem 30609MB [2024-03-07 16:06:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [611/800][300/402] eta 0:01:30 lr 0.000025 time 0.8792 (0.8906) loss 0.6029 (0.5787) grad_norm 0.1994 (0.2165) loss_scale 524288.0000 (441551.5216) mem 30609MB [2024-03-07 16:08:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [611/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8876) loss 0.5745 (0.5776) grad_norm 0.1976 (0.2169) loss_scale 524288.0000 (462184.0599) mem 30609MB [2024-03-07 16:08:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 611 training takes 0:05:56 [2024-03-07 16:08:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [612/800][0/402] eta 0:30:12 lr 0.000025 time 4.5087 (4.5087) loss 0.5921 (0.5921) grad_norm 0.2094 (0.2094) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:09:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [612/800][100/402] eta 0:04:36 lr 0.000025 time 0.8791 (0.9143) loss 0.5854 (0.5793) grad_norm 0.2221 (0.2115) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:11:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [612/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8965) loss 0.5571 (0.5766) grad_norm 0.2114 (0.2114) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:12:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [612/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8905) loss 0.5774 (0.5764) grad_norm 0.2166 (0.2123) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:13:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [612/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8874) loss 0.5666 (0.5774) grad_norm 0.2153 (0.2115) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:13:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 612 training takes 0:05:56 [2024-03-07 16:14:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [613/800][0/402] eta 0:30:35 lr 0.000025 time 4.5665 (4.5665) loss 0.5559 (0.5559) grad_norm 0.2457 (0.2457) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:15:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [613/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9149) loss 0.5608 (0.5765) grad_norm 0.2203 (0.2216) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:16:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [613/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8968) loss 0.5879 (0.5762) grad_norm 0.2023 (0.2203) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:18:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [613/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8907) loss 0.5533 (0.5769) grad_norm 0.2180 (0.2193) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:19:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [613/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8876) loss 0.5898 (0.5774) grad_norm 0.1990 (0.2183) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:19:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 613 training takes 0:05:56 [2024-03-07 16:20:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [614/800][0/402] eta 0:30:20 lr 0.000025 time 4.5275 (4.5275) loss 0.5958 (0.5958) grad_norm 0.2043 (0.2043) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:21:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [614/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9150) loss 0.6425 (0.5785) grad_norm 0.1835 (0.2129) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:22:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [614/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8968) loss 0.5677 (0.5783) grad_norm 0.2019 (0.2146) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:24:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [614/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8907) loss 0.5565 (0.5776) grad_norm 0.2402 (0.2146) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:25:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [614/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8876) loss 0.5665 (0.5784) grad_norm 0.2378 (0.2149) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:25:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 614 training takes 0:05:56 [2024-03-07 16:25:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [615/800][0/402] eta 0:30:43 lr 0.000025 time 4.5863 (4.5863) loss 0.5817 (0.5817) grad_norm 0.2565 (0.2565) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:27:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [615/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9152) loss 0.5562 (0.5820) grad_norm 0.2157 (0.2290) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:28:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [615/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8969) loss 0.5850 (0.5794) grad_norm 0.2067 (0.2207) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:30:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [615/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8908) loss 0.6205 (0.5798) grad_norm 0.2737 (0.2197) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:31:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [615/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8877) loss 0.5659 (0.5790) grad_norm 0.2213 (0.2192) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:31:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 615 training takes 0:05:57 [2024-03-07 16:31:49 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_615.pth saving...... [2024-03-07 16:31:51 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_615.pth saved !!! [2024-03-07 16:31:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [616/800][0/402] eta 0:28:43 lr 0.000025 time 4.2868 (4.2868) loss 0.5526 (0.5526) grad_norm 0.1716 (0.1716) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:33:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [616/800][100/402] eta 0:04:35 lr 0.000025 time 0.8793 (0.9127) loss 0.5741 (0.5765) grad_norm 0.2359 (nan) loss_scale 524288.0000 (539860.9109) mem 30609MB [2024-03-07 16:34:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [616/800][200/402] eta 0:03:01 lr 0.000025 time 0.8820 (0.8960) loss 0.5694 (0.5763) grad_norm 0.2390 (nan) loss_scale 524288.0000 (532113.1940) mem 30609MB [2024-03-07 16:36:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [616/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8903) loss 0.5965 (0.5770) grad_norm 0.2052 (nan) loss_scale 524288.0000 (529513.4618) mem 30609MB [2024-03-07 16:37:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [616/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8874) loss 0.6020 (0.5778) grad_norm 0.1968 (nan) loss_scale 524288.0000 (528210.3541) mem 30609MB [2024-03-07 16:37:48 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 616 training takes 0:05:56 [2024-03-07 16:37:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [617/800][0/402] eta 0:30:25 lr 0.000025 time 4.5404 (4.5404) loss 0.5436 (0.5436) grad_norm 0.2248 (0.2248) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:39:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [617/800][100/402] eta 0:04:36 lr 0.000025 time 0.8790 (0.9152) loss 0.6263 (0.5785) grad_norm 0.1868 (0.2147) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:40:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [617/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8972) loss 0.5882 (0.5801) grad_norm 0.2174 (0.2158) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:42:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [617/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8912) loss 0.5824 (0.5792) grad_norm 0.2048 (0.2177) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:43:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [617/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8881) loss 0.5648 (0.5782) grad_norm 0.2422 (0.2171) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:43:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 617 training takes 0:05:57 [2024-03-07 16:43:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [618/800][0/402] eta 0:30:54 lr 0.000025 time 4.6120 (4.6120) loss 0.5843 (0.5843) grad_norm 0.1897 (0.1897) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:45:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [618/800][100/402] eta 0:04:36 lr 0.000025 time 0.8789 (0.9159) loss 0.5634 (0.5782) grad_norm 0.1881 (0.2160) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:46:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [618/800][200/402] eta 0:03:01 lr 0.000025 time 0.8792 (0.8977) loss 0.5650 (0.5773) grad_norm 0.2265 (0.2167) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:48:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [618/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8915) loss 0.5883 (0.5762) grad_norm 0.2122 (0.2173) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 16:49:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [618/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8883) loss 0.5630 (0.5772) grad_norm 0.2215 (nan) loss_scale 262144.0000 (502061.3267) mem 30609MB [2024-03-07 16:49:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 618 training takes 0:05:57 [2024-03-07 16:49:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [619/800][0/402] eta 0:30:42 lr 0.000025 time 4.5838 (4.5838) loss 0.5236 (0.5236) grad_norm 0.1992 (0.1992) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:51:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [619/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9152) loss 0.6078 (0.5759) grad_norm 0.2114 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:52:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [619/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8970) loss 0.5428 (0.5750) grad_norm 0.2007 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:54:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [619/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8909) loss 0.5895 (0.5756) grad_norm 0.2140 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:55:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [619/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.6093 (0.5766) grad_norm 0.1759 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:55:39 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 619 training takes 0:05:57 [2024-03-07 16:55:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [620/800][0/402] eta 0:30:43 lr 0.000025 time 4.5864 (4.5864) loss 0.5732 (0.5732) grad_norm 0.2096 (0.2096) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:57:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [620/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9151) loss 0.5746 (0.5768) grad_norm 0.2691 (0.2142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 16:58:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [620/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8969) loss 0.5818 (0.5782) grad_norm 0.1894 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:00:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [620/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8909) loss 0.5776 (0.5784) grad_norm 0.2324 (0.2186) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:01:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [620/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8877) loss 0.6073 (0.5777) grad_norm 0.2463 (0.2192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:01:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 620 training takes 0:05:57 [2024-03-07 17:01:36 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_620.pth saving...... [2024-03-07 17:01:38 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_620.pth saved !!! [2024-03-07 17:01:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [621/800][0/402] eta 0:29:10 lr 0.000025 time 4.3540 (4.3540) loss 0.5857 (0.5857) grad_norm 0.2345 (0.2345) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:03:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [621/800][100/402] eta 0:04:35 lr 0.000025 time 0.8781 (0.9135) loss 0.5350 (0.5785) grad_norm 0.2116 (0.2126) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:04:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [621/800][200/402] eta 0:03:01 lr 0.000025 time 0.8780 (0.8961) loss 0.5910 (0.5786) grad_norm 0.2126 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:06:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [621/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8903) loss 0.6117 (0.5778) grad_norm 0.2327 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:07:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [621/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8874) loss 0.5711 (0.5771) grad_norm 0.2190 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:07:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 621 training takes 0:05:56 [2024-03-07 17:07:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [622/800][0/402] eta 0:30:46 lr 0.000025 time 4.5922 (4.5922) loss 0.5865 (0.5865) grad_norm 0.1796 (0.1796) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:09:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [622/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9152) loss 0.5758 (0.5783) grad_norm 0.2196 (0.2117) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:10:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [622/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8969) loss 0.5620 (0.5778) grad_norm 0.2103 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:12:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [622/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8908) loss 0.5670 (0.5786) grad_norm 0.1994 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:13:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [622/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.5879 (0.5784) grad_norm 0.1899 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:13:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 622 training takes 0:05:57 [2024-03-07 17:13:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [623/800][0/402] eta 0:30:57 lr 0.000025 time 4.6212 (4.6212) loss 0.5608 (0.5608) grad_norm 0.2172 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:15:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [623/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9154) loss 0.5866 (0.5751) grad_norm 0.2236 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:16:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [623/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8972) loss 0.6090 (0.5767) grad_norm 0.2437 (0.2174) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:18:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [623/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8910) loss 0.5661 (0.5773) grad_norm 0.2218 (0.2176) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 17:19:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [623/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8878) loss 0.5921 (0.5778) grad_norm 0.2157 (0.2174) loss_scale 524288.0000 (290907.9302) mem 30609MB [2024-03-07 17:19:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 623 training takes 0:05:57 [2024-03-07 17:19:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [624/800][0/402] eta 0:30:37 lr 0.000025 time 4.5705 (4.5705) loss 0.5419 (0.5419) grad_norm 0.2210 (0.2210) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:21:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [624/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9150) loss 0.5873 (0.5784) grad_norm 0.2160 (0.2144) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:22:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [624/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8968) loss 0.5377 (0.5771) grad_norm 0.2126 (0.2161) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:23:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [624/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8908) loss 0.5547 (0.5763) grad_norm 0.2293 (0.2179) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:25:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [624/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8876) loss 0.6163 (0.5766) grad_norm 0.1841 (0.2180) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:25:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 624 training takes 0:05:56 [2024-03-07 17:25:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [625/800][0/402] eta 0:29:54 lr 0.000025 time 4.4646 (4.4646) loss 0.5761 (0.5761) grad_norm 0.1958 (0.1958) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:26:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [625/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9143) loss 0.6141 (0.5766) grad_norm 0.2217 (0.2218) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:28:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [625/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8965) loss 0.5782 (0.5769) grad_norm 0.1984 (0.2192) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:29:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [625/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8906) loss 0.5708 (0.5781) grad_norm 0.1935 (0.2201) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:31:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [625/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8876) loss 0.5664 (0.5785) grad_norm 0.1931 (0.2185) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:31:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 625 training takes 0:05:56 [2024-03-07 17:31:23 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_625.pth saving...... [2024-03-07 17:31:25 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_625.pth saved !!! [2024-03-07 17:31:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [626/800][0/402] eta 0:28:44 lr 0.000025 time 4.2902 (4.2902) loss 0.6082 (0.6082) grad_norm 0.1917 (0.1917) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:32:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [626/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9123) loss 0.5757 (0.5775) grad_norm 0.1823 (0.2117) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:34:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [626/800][200/402] eta 0:03:00 lr 0.000025 time 0.8783 (0.8955) loss 0.6031 (0.5773) grad_norm 0.2571 (0.2168) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:35:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [626/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8899) loss 0.5789 (0.5787) grad_norm 0.2362 (0.2155) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:37:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [626/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8870) loss 0.5711 (0.5784) grad_norm 0.2691 (0.2161) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:37:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 626 training takes 0:05:56 [2024-03-07 17:37:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [627/800][0/402] eta 0:30:41 lr 0.000025 time 4.5801 (4.5801) loss 0.5804 (0.5804) grad_norm 0.2105 (0.2105) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:38:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [627/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9151) loss 0.5877 (0.5780) grad_norm 0.2195 (0.2147) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:40:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [627/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8970) loss 0.5427 (0.5779) grad_norm 0.2223 (0.2148) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:41:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [627/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8908) loss 0.5791 (0.5782) grad_norm 0.2047 (0.2171) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:43:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [627/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8877) loss 0.5575 (0.5777) grad_norm 0.1874 (0.2172) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:43:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 627 training takes 0:05:57 [2024-03-07 17:43:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [628/800][0/402] eta 0:30:27 lr 0.000025 time 4.5453 (4.5453) loss 0.5751 (0.5751) grad_norm 0.2440 (0.2440) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:44:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [628/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9151) loss 0.5829 (0.5749) grad_norm 0.2109 (0.2178) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:46:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [628/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8969) loss 0.6157 (0.5770) grad_norm 0.2498 (0.2157) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:47:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [628/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8908) loss 0.5350 (0.5761) grad_norm 0.2541 (0.2184) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:49:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [628/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8877) loss 0.5989 (0.5768) grad_norm 0.1886 (nan) loss_scale 524288.0000 (525595.4514) mem 30609MB [2024-03-07 17:49:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 628 training takes 0:05:56 [2024-03-07 17:49:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [629/800][0/402] eta 0:29:31 lr 0.000025 time 4.4078 (4.4078) loss 0.6193 (0.6193) grad_norm 0.2022 (0.2022) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:50:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [629/800][100/402] eta 0:04:35 lr 0.000025 time 0.8790 (0.9133) loss 0.5882 (0.5732) grad_norm 0.2194 (0.2220) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:52:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [629/800][200/402] eta 0:03:00 lr 0.000025 time 0.8782 (0.8960) loss 0.5955 (0.5762) grad_norm 0.1829 (0.2179) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:53:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [629/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8903) loss 0.5864 (0.5772) grad_norm 0.1620 (0.2187) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:55:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [629/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8873) loss 0.6177 (0.5779) grad_norm 0.2227 (0.2195) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:55:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 629 training takes 0:05:56 [2024-03-07 17:55:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [630/800][0/402] eta 0:30:35 lr 0.000025 time 4.5651 (4.5651) loss 0.6290 (0.6290) grad_norm 0.2235 (0.2235) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:56:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [630/800][100/402] eta 0:04:36 lr 0.000025 time 0.8790 (0.9149) loss 0.6085 (0.5755) grad_norm 0.2687 (0.2117) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:58:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [630/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8970) loss 0.5268 (0.5766) grad_norm 0.2561 (0.2117) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 17:59:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [630/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8908) loss 0.5819 (0.5774) grad_norm 0.2205 (0.2118) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:01:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [630/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8877) loss 0.5905 (0.5777) grad_norm 0.2133 (nan) loss_scale 262144.0000 (497485.2469) mem 30609MB [2024-03-07 18:01:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 630 training takes 0:05:56 [2024-03-07 18:01:09 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_630.pth saving...... [2024-03-07 18:01:11 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_630.pth saved !!! [2024-03-07 18:01:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [631/800][0/402] eta 0:30:09 lr 0.000025 time 4.5001 (4.5001) loss 0.6153 (0.6153) grad_norm 0.1823 (0.1823) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:02:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [631/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9148) loss 0.6123 (0.5815) grad_norm 0.1892 (0.2178) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:04:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [631/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5749 (0.5799) grad_norm 0.2106 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:05:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [631/800][300/402] eta 0:01:30 lr 0.000025 time 0.8780 (0.8908) loss 0.5850 (0.5804) grad_norm 0.1879 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:07:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [631/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8878) loss 0.5665 (0.5794) grad_norm 0.2103 (0.2184) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:07:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 631 training takes 0:05:57 [2024-03-07 18:07:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [632/800][0/402] eta 0:30:31 lr 0.000025 time 4.5568 (4.5568) loss 0.6115 (0.6115) grad_norm 0.2627 (0.2627) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:08:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [632/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9149) loss 0.5643 (0.5773) grad_norm 0.2059 (0.2140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:10:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [632/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8968) loss 0.5773 (0.5777) grad_norm 0.2122 (0.2142) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:11:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [632/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8908) loss 0.5671 (0.5778) grad_norm 0.2018 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:13:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [632/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8877) loss 0.5802 (0.5775) grad_norm 0.2184 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:13:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 632 training takes 0:05:56 [2024-03-07 18:13:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [633/800][0/402] eta 0:30:25 lr 0.000025 time 4.5413 (4.5413) loss 0.5757 (0.5757) grad_norm 0.1860 (0.1860) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:14:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [633/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9147) loss 0.5725 (0.5777) grad_norm 0.2184 (0.2120) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:16:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [633/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8967) loss 0.5871 (0.5770) grad_norm 0.2041 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:17:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [633/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8907) loss 0.5971 (0.5782) grad_norm 0.3904 (0.2247) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:19:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [633/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8876) loss 0.5919 (0.5781) grad_norm 0.2105 (0.2206) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:19:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 633 training takes 0:05:56 [2024-03-07 18:19:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [634/800][0/402] eta 0:30:55 lr 0.000025 time 4.6146 (4.6146) loss 0.5806 (0.5806) grad_norm 0.1955 (0.1955) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:20:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [634/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9159) loss 0.5647 (0.5761) grad_norm 0.2522 (0.2461) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:22:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [634/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8973) loss 0.5613 (0.5776) grad_norm 0.2424 (0.2301) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:23:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [634/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8911) loss 0.6197 (0.5779) grad_norm 0.2308 (0.2252) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:24:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [634/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8879) loss 0.6035 (0.5779) grad_norm 0.1970 (0.2223) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:24:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 634 training takes 0:05:57 [2024-03-07 18:25:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [635/800][0/402] eta 0:30:13 lr 0.000025 time 4.5118 (4.5118) loss 0.5654 (0.5654) grad_norm 0.2041 (0.2041) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:26:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [635/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9145) loss 0.5787 (0.5791) grad_norm 0.2013 (0.2205) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:27:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [635/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8966) loss 0.5760 (0.5764) grad_norm 0.2081 (0.2201) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:29:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [635/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8906) loss 0.5714 (0.5773) grad_norm 0.2067 (0.2186) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 18:30:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [635/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8875) loss 0.5884 (0.5778) grad_norm 0.2193 (0.2170) loss_scale 524288.0000 (295484.0100) mem 30609MB [2024-03-07 18:30:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 635 training takes 0:05:56 [2024-03-07 18:30:56 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_635.pth saving...... [2024-03-07 18:30:58 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_635.pth saved !!! [2024-03-07 18:31:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [636/800][0/402] eta 0:31:03 lr 0.000025 time 4.6365 (4.6365) loss 0.5635 (0.5635) grad_norm 0.2474 (0.2474) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:32:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [636/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9157) loss 0.5826 (0.5754) grad_norm 0.2122 (0.2156) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:33:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [636/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8973) loss 0.5853 (0.5760) grad_norm 0.1794 (0.2191) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:35:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [636/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8911) loss 0.5418 (0.5765) grad_norm 0.2286 (0.2184) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:36:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [636/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8879) loss 0.5567 (0.5768) grad_norm 0.2476 (0.2192) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:36:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 636 training takes 0:05:57 [2024-03-07 18:36:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [637/800][0/402] eta 0:30:36 lr 0.000025 time 4.5693 (4.5693) loss 0.6147 (0.6147) grad_norm 0.2248 (0.2248) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:38:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [637/800][100/402] eta 0:04:36 lr 0.000025 time 0.8794 (0.9153) loss 0.6062 (0.5766) grad_norm 0.2330 (0.2143) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:39:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [637/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8971) loss 0.5591 (0.5775) grad_norm 0.2047 (0.2201) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:41:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [637/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8909) loss 0.5913 (0.5782) grad_norm 0.2134 (0.2196) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:42:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [637/800][400/402] eta 0:00:01 lr 0.000025 time 0.8774 (0.8878) loss 0.5861 (0.5776) grad_norm 0.1948 (0.2177) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:42:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 637 training takes 0:05:57 [2024-03-07 18:42:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [638/800][0/402] eta 0:30:52 lr 0.000025 time 4.6080 (4.6080) loss 0.5851 (0.5851) grad_norm 0.2143 (0.2143) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:44:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [638/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9155) loss 0.5676 (0.5785) grad_norm 0.2256 (0.2176) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:45:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [638/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8971) loss 0.6046 (0.5788) grad_norm 0.1898 (0.2174) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:47:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [638/800][300/402] eta 0:01:30 lr 0.000025 time 0.8791 (0.8910) loss 0.5777 (0.5795) grad_norm 0.2292 (0.2173) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:48:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [638/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8879) loss 0.5825 (0.5789) grad_norm 0.2026 (0.2170) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:48:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 638 training takes 0:05:57 [2024-03-07 18:48:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [639/800][0/402] eta 0:30:33 lr 0.000025 time 4.5611 (4.5611) loss 0.5635 (0.5635) grad_norm 0.2090 (0.2090) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:50:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [639/800][100/402] eta 0:04:36 lr 0.000025 time 0.8789 (0.9149) loss 0.6113 (0.5770) grad_norm 0.2178 (0.2187) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:51:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [639/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8970) loss 0.5978 (0.5766) grad_norm 0.2191 (0.2156) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:53:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [639/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8909) loss 0.5689 (0.5770) grad_norm 0.2377 (0.2158) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:54:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [639/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8877) loss 0.5556 (0.5768) grad_norm 0.2413 (0.2150) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:54:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 639 training takes 0:05:57 [2024-03-07 18:54:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [640/800][0/402] eta 0:30:30 lr 0.000025 time 4.5540 (4.5540) loss 0.5428 (0.5428) grad_norm 0.1906 (0.1906) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:56:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [640/800][100/402] eta 0:04:36 lr 0.000025 time 0.8778 (0.9150) loss 0.5882 (0.5805) grad_norm 0.2108 (0.2160) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:57:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [640/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8969) loss 0.5254 (0.5792) grad_norm 0.2251 (0.2179) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 18:59:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [640/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8908) loss 0.5585 (0.5782) grad_norm 0.2523 (0.2170) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:00:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [640/800][400/402] eta 0:00:01 lr 0.000025 time 0.8759 (0.8877) loss 0.5847 (0.5772) grad_norm 0.1831 (nan) loss_scale 524288.0000 (529517.8055) mem 30609MB [2024-03-07 19:00:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 640 training takes 0:05:57 [2024-03-07 19:00:43 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_640.pth saving...... [2024-03-07 19:00:45 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_640.pth saved !!! [2024-03-07 19:00:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [641/800][0/402] eta 0:29:43 lr 0.000025 time 4.4363 (4.4363) loss 0.5934 (0.5934) grad_norm 0.2769 (0.2769) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:02:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [641/800][100/402] eta 0:04:35 lr 0.000025 time 0.8787 (0.9138) loss 0.5432 (0.5752) grad_norm 0.2174 (0.2130) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:03:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [641/800][200/402] eta 0:03:01 lr 0.000025 time 0.8795 (0.8963) loss 0.5709 (0.5769) grad_norm 0.2333 (0.2154) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:05:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [641/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8904) loss 0.5902 (0.5768) grad_norm 0.2021 (0.2148) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:06:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [641/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8875) loss 0.5680 (0.5771) grad_norm 0.2163 (0.2144) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:06:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 641 training takes 0:05:56 [2024-03-07 19:06:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [642/800][0/402] eta 0:30:32 lr 0.000025 time 4.5576 (4.5576) loss 0.5557 (0.5557) grad_norm 0.1747 (0.1747) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:08:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [642/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9149) loss 0.5742 (0.5788) grad_norm 0.1988 (0.2211) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:09:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [642/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8968) loss 0.5839 (0.5786) grad_norm 0.2334 (0.2179) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:11:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [642/800][300/402] eta 0:01:30 lr 0.000025 time 0.8778 (0.8908) loss 0.5887 (0.5775) grad_norm 0.2889 (0.2194) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:12:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [642/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8877) loss 0.5724 (0.5779) grad_norm 0.2395 (0.2197) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:12:39 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 642 training takes 0:05:57 [2024-03-07 19:12:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [643/800][0/402] eta 0:29:59 lr 0.000025 time 4.4762 (4.4762) loss 0.5962 (0.5962) grad_norm 0.2285 (0.2285) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:14:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [643/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9140) loss 0.5749 (0.5751) grad_norm 0.1961 (nan) loss_scale 262144.0000 (267334.9703) mem 30609MB [2024-03-07 19:15:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [643/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8964) loss 0.5641 (0.5766) grad_norm 0.2142 (nan) loss_scale 262144.0000 (264752.3980) mem 30609MB [2024-03-07 19:17:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [643/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8904) loss 0.5866 (0.5768) grad_norm 0.2156 (nan) loss_scale 262144.0000 (263885.8206) mem 30609MB [2024-03-07 19:18:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [643/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8874) loss 0.5579 (0.5770) grad_norm 0.2151 (nan) loss_scale 262144.0000 (263451.4514) mem 30609MB [2024-03-07 19:18:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 643 training takes 0:05:56 [2024-03-07 19:18:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [644/800][0/402] eta 0:30:21 lr 0.000025 time 4.5305 (4.5305) loss 0.5631 (0.5631) grad_norm 0.2122 (0.2122) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:20:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [644/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9151) loss 0.6050 (0.5800) grad_norm 0.2079 (0.2192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:21:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [644/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8969) loss 0.5679 (0.5780) grad_norm 0.2101 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:23:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [644/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8908) loss 0.5994 (0.5766) grad_norm 0.2076 (0.2176) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:24:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [644/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.5523 (0.5765) grad_norm 0.1963 (0.2162) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:24:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 644 training takes 0:05:56 [2024-03-07 19:24:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [645/800][0/402] eta 0:30:20 lr 0.000025 time 4.5297 (4.5297) loss 0.5637 (0.5637) grad_norm 0.1956 (0.1956) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:26:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [645/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9148) loss 0.5564 (0.5770) grad_norm 0.2476 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:27:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [645/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8968) loss 0.5645 (0.5777) grad_norm 0.2231 (0.2169) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:29:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [645/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8907) loss 0.6271 (0.5780) grad_norm 0.2179 (0.2166) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:30:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [645/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8876) loss 0.5719 (0.5774) grad_norm 0.2095 (0.2147) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:30:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 645 training takes 0:05:56 [2024-03-07 19:30:30 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_645.pth saving...... [2024-03-07 19:30:31 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_645.pth saved !!! [2024-03-07 19:30:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [646/800][0/402] eta 0:30:51 lr 0.000025 time 4.6048 (4.6048) loss 0.5605 (0.5605) grad_norm 0.2084 (0.2084) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:32:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [646/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9156) loss 0.5994 (0.5793) grad_norm 0.2173 (0.2130) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:33:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [646/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8973) loss 0.5743 (0.5784) grad_norm 0.2196 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:35:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [646/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8911) loss 0.5706 (0.5777) grad_norm 0.2396 (0.2195) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:36:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [646/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8879) loss 0.5984 (0.5779) grad_norm 0.2376 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:36:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 646 training takes 0:05:57 [2024-03-07 19:36:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [647/800][0/402] eta 0:30:43 lr 0.000025 time 4.5866 (4.5866) loss 0.5898 (0.5898) grad_norm 0.2219 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:38:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [647/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9155) loss 0.5686 (0.5765) grad_norm 0.2804 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:39:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [647/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8971) loss 0.5833 (0.5765) grad_norm 0.2085 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:40:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [647/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8910) loss 0.5435 (0.5762) grad_norm 0.2103 (0.2176) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 19:42:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [647/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8878) loss 0.5870 (0.5768) grad_norm 0.2249 (0.2157) loss_scale 524288.0000 (266720.0798) mem 30609MB [2024-03-07 19:42:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 647 training takes 0:05:57 [2024-03-07 19:42:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [648/800][0/402] eta 0:30:30 lr 0.000025 time 4.5528 (4.5528) loss 0.5780 (0.5780) grad_norm 0.1778 (0.1778) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:43:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [648/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9149) loss 0.5841 (0.5786) grad_norm 0.3124 (0.2236) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:45:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [648/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8968) loss 0.5848 (0.5783) grad_norm 0.1902 (0.2215) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:46:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [648/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8909) loss 0.5719 (0.5783) grad_norm 0.2185 (0.2219) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:48:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [648/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.5005 (0.5780) grad_norm 0.2548 (0.2207) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:48:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 648 training takes 0:05:57 [2024-03-07 19:48:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [649/800][0/402] eta 0:30:31 lr 0.000025 time 4.5552 (4.5552) loss 0.5838 (0.5838) grad_norm 0.2216 (0.2216) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:49:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [649/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9150) loss 0.5634 (0.5738) grad_norm 0.2127 (0.2177) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:51:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [649/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8970) loss 0.5849 (0.5757) grad_norm 0.2106 (0.2199) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:52:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [649/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8908) loss 0.6020 (0.5762) grad_norm 0.1997 (0.2195) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:54:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [649/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8877) loss 0.5629 (0.5773) grad_norm 0.2360 (0.2179) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:54:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 649 training takes 0:05:57 [2024-03-07 19:54:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [650/800][0/402] eta 0:30:28 lr 0.000025 time 4.5495 (4.5495) loss 0.5671 (0.5671) grad_norm 0.2274 (0.2274) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:55:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [650/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9148) loss 0.5521 (0.5770) grad_norm 0.2112 (0.2173) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:57:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [650/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.6185 (0.5782) grad_norm 0.2230 (0.2134) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 19:58:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [650/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8907) loss 0.5317 (0.5768) grad_norm 0.2184 (nan) loss_scale 262144.0000 (472033.3821) mem 30609MB [2024-03-07 20:00:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [650/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8876) loss 0.5474 (0.5768) grad_norm 0.2646 (nan) loss_scale 262144.0000 (419691.8903) mem 30609MB [2024-03-07 20:00:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 650 training takes 0:05:56 [2024-03-07 20:00:16 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_650.pth saving...... [2024-03-07 20:00:18 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_650.pth saved !!! [2024-03-07 20:00:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [651/800][0/402] eta 0:29:53 lr 0.000025 time 4.4618 (4.4618) loss 0.5753 (0.5753) grad_norm 0.1913 (0.1913) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:01:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [651/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9144) loss 0.5737 (0.5748) grad_norm 0.1924 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:03:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [651/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8967) loss 0.5556 (0.5770) grad_norm 0.2257 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:04:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [651/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8907) loss 0.5869 (0.5768) grad_norm 0.2152 (0.2177) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:06:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [651/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8877) loss 0.6002 (0.5772) grad_norm 0.2382 (0.2189) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:06:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 651 training takes 0:05:56 [2024-03-07 20:06:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [652/800][0/402] eta 0:30:55 lr 0.000025 time 4.6158 (4.6158) loss 0.5831 (0.5831) grad_norm 0.2541 (0.2541) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:07:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [652/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9155) loss 0.6090 (0.5749) grad_norm 0.1958 (0.2191) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:09:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [652/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8971) loss 0.6017 (0.5763) grad_norm 0.2069 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:10:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [652/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8909) loss 0.6066 (0.5771) grad_norm 0.2426 (0.2196) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:12:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [652/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8878) loss 0.6160 (0.5775) grad_norm 0.2215 (0.2185) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:12:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 652 training takes 0:05:57 [2024-03-07 20:12:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [653/800][0/402] eta 0:30:37 lr 0.000025 time 4.5699 (4.5699) loss 0.5511 (0.5511) grad_norm 0.2958 (0.2958) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:13:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [653/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9151) loss 0.5567 (0.5773) grad_norm 0.2829 (0.2266) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:15:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [653/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8971) loss 0.5643 (0.5771) grad_norm 0.2210 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:16:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [653/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8910) loss 0.6069 (0.5768) grad_norm 0.1933 (0.2197) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:18:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [653/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8878) loss 0.5742 (0.5773) grad_norm 0.1711 (0.2176) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:18:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 653 training takes 0:05:57 [2024-03-07 20:18:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [654/800][0/402] eta 0:30:35 lr 0.000025 time 4.5649 (4.5649) loss 0.5748 (0.5748) grad_norm 0.1978 (0.1978) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:19:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [654/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9150) loss 0.5814 (0.5770) grad_norm 0.1896 (0.2214) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:21:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [654/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8969) loss 0.5999 (0.5767) grad_norm 0.2039 (0.2196) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:22:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [654/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8908) loss 0.5572 (0.5760) grad_norm 0.2316 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:24:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [654/800][400/402] eta 0:00:01 lr 0.000025 time 0.8770 (0.8877) loss 0.5989 (0.5765) grad_norm 0.1919 (0.2158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:24:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 654 training takes 0:05:57 [2024-03-07 20:24:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [655/800][0/402] eta 0:30:31 lr 0.000025 time 4.5552 (4.5552) loss 0.5454 (0.5454) grad_norm 0.2307 (0.2307) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:25:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [655/800][100/402] eta 0:04:36 lr 0.000025 time 0.8789 (0.9149) loss 0.5856 (0.5787) grad_norm 0.2401 (0.2196) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:27:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [655/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8968) loss 0.5717 (0.5791) grad_norm 0.1976 (0.2184) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:28:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [655/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8909) loss 0.5918 (0.5792) grad_norm 0.1953 (0.2176) loss_scale 524288.0000 (323107.7209) mem 30609MB [2024-03-07 20:30:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [655/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8877) loss 0.5733 (0.5788) grad_norm 0.2162 (0.2179) loss_scale 524288.0000 (373277.3666) mem 30609MB [2024-03-07 20:30:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 655 training takes 0:05:57 [2024-03-07 20:30:03 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_655.pth saving...... [2024-03-07 20:30:05 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_655.pth saved !!! [2024-03-07 20:30:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [656/800][0/402] eta 0:30:09 lr 0.000025 time 4.5007 (4.5007) loss 0.6002 (0.6002) grad_norm 0.2078 (0.2078) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 20:31:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [656/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9145) loss 0.5686 (0.5795) grad_norm 0.2007 (0.2261) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 20:33:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [656/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8966) loss 0.5613 (0.5776) grad_norm 0.1946 (nan) loss_scale 262144.0000 (520375.4030) mem 30609MB [2024-03-07 20:34:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [656/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8906) loss 0.5884 (0.5758) grad_norm 0.1978 (nan) loss_scale 262144.0000 (434584.2392) mem 30609MB [2024-03-07 20:36:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [656/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8875) loss 0.5924 (0.5762) grad_norm 0.2192 (nan) loss_scale 262144.0000 (391581.6858) mem 30609MB [2024-03-07 20:36:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 656 training takes 0:05:56 [2024-03-07 20:36:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [657/800][0/402] eta 0:30:13 lr 0.000025 time 4.5121 (4.5121) loss 0.5651 (0.5651) grad_norm 0.2516 (0.2516) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:37:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [657/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9144) loss 0.5763 (0.5783) grad_norm 0.2031 (0.2226) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:39:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [657/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8965) loss 0.5411 (0.5769) grad_norm 0.2102 (0.2232) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:40:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [657/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8905) loss 0.5775 (0.5763) grad_norm 0.2273 (0.2209) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:41:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [657/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8875) loss 0.5708 (0.5769) grad_norm 0.1936 (0.2202) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:41:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 657 training takes 0:05:56 [2024-03-07 20:42:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [658/800][0/402] eta 0:30:45 lr 0.000025 time 4.5896 (4.5896) loss 0.5715 (0.5715) grad_norm 0.2454 (0.2454) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:43:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [658/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9152) loss 0.5606 (0.5785) grad_norm 0.2353 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:44:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [658/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8969) loss 0.6084 (0.5785) grad_norm 0.2168 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:46:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [658/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8908) loss 0.5769 (0.5776) grad_norm 0.2190 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:47:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [658/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.5565 (0.5773) grad_norm 0.2068 (0.2185) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:47:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 658 training takes 0:05:56 [2024-03-07 20:48:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [659/800][0/402] eta 0:30:38 lr 0.000025 time 4.5735 (4.5735) loss 0.5746 (0.5746) grad_norm 0.2175 (0.2175) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:49:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [659/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9150) loss 0.5591 (0.5750) grad_norm 0.2229 (0.2158) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:50:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [659/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8969) loss 0.5529 (0.5768) grad_norm 0.2285 (0.2238) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:52:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [659/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8907) loss 0.5855 (0.5781) grad_norm 0.2269 (0.2213) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:53:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [659/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8877) loss 0.5970 (0.5781) grad_norm 0.1899 (0.2206) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:53:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 659 training takes 0:05:56 [2024-03-07 20:53:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [660/800][0/402] eta 0:30:46 lr 0.000025 time 4.5942 (4.5942) loss 0.5594 (0.5594) grad_norm 0.1776 (0.1776) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:55:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [660/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9155) loss 0.6015 (0.5770) grad_norm 0.2095 (0.2167) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:56:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [660/800][200/402] eta 0:03:01 lr 0.000025 time 0.8778 (0.8971) loss 0.5714 (0.5770) grad_norm 0.2252 (0.2184) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:58:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [660/800][300/402] eta 0:01:30 lr 0.000025 time 0.8787 (0.8910) loss 0.5925 (0.5767) grad_norm 0.2097 (0.2210) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:59:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [660/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8878) loss 0.6114 (0.5777) grad_norm 0.2259 (0.2194) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 20:59:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 660 training takes 0:05:57 [2024-03-07 20:59:50 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_660.pth saving...... [2024-03-07 20:59:52 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_660.pth saved !!! [2024-03-07 20:59:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [661/800][0/402] eta 0:30:26 lr 0.000025 time 4.5427 (4.5427) loss 0.5494 (0.5494) grad_norm 0.2300 (0.2300) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:01:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [661/800][100/402] eta 0:04:36 lr 0.000025 time 0.8784 (0.9148) loss 0.5700 (0.5763) grad_norm 0.2248 (0.2183) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:02:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [661/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8968) loss 0.5374 (0.5769) grad_norm 0.2269 (0.2181) loss_scale 524288.0000 (279098.5871) mem 30609MB [2024-03-07 21:04:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [661/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8908) loss 0.5250 (0.5778) grad_norm 0.2418 (0.2189) loss_scale 524288.0000 (360556.8638) mem 30609MB [2024-03-07 21:05:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [661/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8876) loss 0.5654 (0.5779) grad_norm 0.2173 (0.2186) loss_scale 524288.0000 (401387.5711) mem 30609MB [2024-03-07 21:05:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 661 training takes 0:05:56 [2024-03-07 21:05:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [662/800][0/402] eta 0:30:37 lr 0.000025 time 4.5698 (4.5698) loss 0.5726 (0.5726) grad_norm 0.2054 (0.2054) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:07:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [662/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9151) loss 0.5843 (0.5778) grad_norm 0.2281 (0.2194) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:08:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [662/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8969) loss 0.5838 (0.5773) grad_norm 0.2519 (0.2277) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:10:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [662/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8910) loss 0.5779 (0.5764) grad_norm 0.2271 (0.2251) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:11:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [662/800][400/402] eta 0:00:01 lr 0.000025 time 0.8760 (0.8878) loss 0.5613 (0.5764) grad_norm 0.1857 (0.2216) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:11:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 662 training takes 0:05:57 [2024-03-07 21:11:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [663/800][0/402] eta 0:31:04 lr 0.000025 time 4.6374 (4.6374) loss 0.5612 (0.5612) grad_norm 0.2372 (0.2372) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:13:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [663/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9158) loss 0.5395 (0.5755) grad_norm 0.2223 (0.2180) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:14:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [663/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8973) loss 0.5876 (0.5745) grad_norm 0.2053 (0.2153) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:16:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [663/800][300/402] eta 0:01:30 lr 0.000025 time 0.8782 (0.8910) loss 0.5995 (0.5762) grad_norm 0.2218 (nan) loss_scale 262144.0000 (463324.2791) mem 30609MB [2024-03-07 21:17:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [663/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8878) loss 0.6077 (0.5761) grad_norm 0.2134 (nan) loss_scale 262144.0000 (413154.6334) mem 30609MB [2024-03-07 21:17:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 663 training takes 0:05:57 [2024-03-07 21:17:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [664/800][0/402] eta 0:30:43 lr 0.000025 time 4.5853 (4.5853) loss 0.5700 (0.5700) grad_norm 0.2239 (0.2239) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:19:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [664/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9152) loss 0.6077 (0.5770) grad_norm 0.1802 (0.2192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:20:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [664/800][200/402] eta 0:03:01 lr 0.000025 time 0.8781 (0.8969) loss 0.5299 (0.5758) grad_norm 0.2611 (0.2187) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:22:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [664/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8908) loss 0.5678 (0.5762) grad_norm 0.2448 (0.2192) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:23:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [664/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8878) loss 0.5646 (0.5767) grad_norm 0.1958 (0.2185) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:23:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 664 training takes 0:05:57 [2024-03-07 21:23:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [665/800][0/402] eta 0:30:31 lr 0.000025 time 4.5569 (4.5569) loss 0.5567 (0.5567) grad_norm 0.2273 (0.2273) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:25:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [665/800][100/402] eta 0:04:36 lr 0.000025 time 0.8791 (0.9151) loss 0.6057 (0.5783) grad_norm 0.2112 (0.2140) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:26:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [665/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8969) loss 0.5713 (0.5772) grad_norm 0.2076 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:28:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [665/800][300/402] eta 0:01:30 lr 0.000025 time 0.8790 (0.8908) loss 0.5682 (0.5778) grad_norm 0.2103 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:29:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [665/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8877) loss 0.6066 (0.5781) grad_norm 0.2013 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:29:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 665 training takes 0:05:57 [2024-03-07 21:29:37 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_665.pth saving...... [2024-03-07 21:29:39 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_665.pth saved !!! [2024-03-07 21:29:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [666/800][0/402] eta 0:30:23 lr 0.000025 time 4.5356 (4.5356) loss 0.5714 (0.5714) grad_norm 0.2191 (0.2191) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:31:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [666/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9147) loss 0.5366 (0.5779) grad_norm 0.1998 (0.2141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:32:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [666/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.8968) loss 0.5704 (0.5781) grad_norm 0.2009 (0.2141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:34:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [666/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8907) loss 0.5616 (0.5767) grad_norm 0.2155 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:35:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [666/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8876) loss 0.5896 (0.5771) grad_norm 0.2564 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:35:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 666 training takes 0:05:56 [2024-03-07 21:35:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [667/800][0/402] eta 0:30:43 lr 0.000025 time 4.5849 (4.5849) loss 0.5857 (0.5857) grad_norm 0.1818 (0.1818) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:37:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [667/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9155) loss 0.5973 (0.5773) grad_norm 0.2005 (0.2266) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:38:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [667/800][200/402] eta 0:03:01 lr 0.000025 time 0.8802 (0.8971) loss 0.5728 (0.5770) grad_norm 0.2018 (0.2239) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:40:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [667/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8910) loss 0.5632 (0.5781) grad_norm 0.2321 (0.2208) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:41:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [667/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8878) loss 0.5758 (0.5780) grad_norm 0.2195 (0.2234) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:41:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 667 training takes 0:05:57 [2024-03-07 21:41:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [668/800][0/402] eta 0:30:48 lr 0.000025 time 4.5980 (4.5980) loss 0.6010 (0.6010) grad_norm 0.1910 (0.1910) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:43:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [668/800][100/402] eta 0:04:36 lr 0.000025 time 0.8780 (0.9153) loss 0.5508 (0.5757) grad_norm 0.2471 (0.2240) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:44:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [668/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8971) loss 0.5802 (0.5760) grad_norm 0.2751 (0.2224) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:46:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [668/800][300/402] eta 0:01:30 lr 0.000025 time 0.8781 (0.8909) loss 0.6010 (0.5764) grad_norm 0.2141 (0.2213) loss_scale 524288.0000 (331816.8239) mem 30609MB [2024-03-07 21:47:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [668/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8878) loss 0.5888 (0.5757) grad_norm 0.2544 (0.2201) loss_scale 524288.0000 (379814.6234) mem 30609MB [2024-03-07 21:47:30 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 668 training takes 0:05:57 [2024-03-07 21:47:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [669/800][0/402] eta 0:30:32 lr 0.000025 time 4.5591 (4.5591) loss 0.5945 (0.5945) grad_norm 0.2370 (0.2370) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:49:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [669/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9150) loss 0.5703 (0.5776) grad_norm 0.2330 (0.2160) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 21:50:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [669/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8970) loss 0.5516 (0.5753) grad_norm 0.2320 (nan) loss_scale 262144.0000 (519071.2040) mem 30609MB [2024-03-07 21:51:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [669/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8909) loss 0.5738 (0.5760) grad_norm 0.2243 (nan) loss_scale 262144.0000 (433713.3289) mem 30609MB [2024-03-07 21:53:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [669/800][400/402] eta 0:00:01 lr 0.000025 time 0.8766 (0.8878) loss 0.5863 (0.5760) grad_norm 0.1847 (nan) loss_scale 262144.0000 (390927.9601) mem 30609MB [2024-03-07 21:53:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 669 training takes 0:05:57 [2024-03-07 21:53:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [670/800][0/402] eta 0:29:34 lr 0.000025 time 4.4144 (4.4144) loss 0.5929 (0.5929) grad_norm 0.2386 (0.2386) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:54:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [670/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9135) loss 0.5538 (0.5790) grad_norm 0.2014 (0.2125) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:56:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [670/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8961) loss 0.5772 (0.5788) grad_norm 0.2537 (0.2176) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:57:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [670/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8903) loss 0.6046 (0.5778) grad_norm 0.1867 (0.2195) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:59:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [670/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8873) loss 0.6028 (0.5774) grad_norm 0.2060 (0.2189) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 21:59:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 670 training takes 0:05:56 [2024-03-07 21:59:24 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_670.pth saving...... [2024-03-07 21:59:25 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_670.pth saved !!! [2024-03-07 21:59:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [671/800][0/402] eta 0:28:31 lr 0.000025 time 4.2583 (4.2583) loss 0.5947 (0.5947) grad_norm 0.2110 (0.2110) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:00:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [671/800][100/402] eta 0:04:35 lr 0.000025 time 0.8784 (0.9119) loss 0.5510 (0.5736) grad_norm 0.2242 (0.2238) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:02:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [671/800][200/402] eta 0:03:00 lr 0.000025 time 0.8786 (0.8954) loss 0.5684 (0.5758) grad_norm 0.2039 (0.2189) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:03:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [671/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8898) loss 0.5456 (0.5769) grad_norm 0.2138 (0.2195) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:05:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [671/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8870) loss 0.5941 (0.5765) grad_norm 0.2024 (0.2188) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:05:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 671 training takes 0:05:56 [2024-03-07 22:05:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [672/800][0/402] eta 0:30:33 lr 0.000025 time 4.5618 (4.5618) loss 0.5694 (0.5694) grad_norm 0.1947 (0.1947) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:06:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [672/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9150) loss 0.5908 (0.5762) grad_norm 0.2404 (0.2156) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:08:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [672/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8968) loss 0.5826 (0.5754) grad_norm 0.2214 (0.2175) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:09:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [672/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8907) loss 0.5759 (0.5761) grad_norm 0.2562 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:11:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [672/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8876) loss 0.5840 (0.5765) grad_norm 0.2147 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:11:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 672 training takes 0:05:56 [2024-03-07 22:11:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [673/800][0/402] eta 0:30:43 lr 0.000025 time 4.5857 (4.5857) loss 0.5639 (0.5639) grad_norm 0.2420 (0.2420) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:12:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [673/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9152) loss 0.5239 (0.5760) grad_norm 0.1971 (0.2145) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:14:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [673/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8970) loss 0.5614 (0.5772) grad_norm 0.2055 (0.2146) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:15:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [673/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8908) loss 0.5586 (0.5773) grad_norm 0.2176 (0.2148) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:17:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [673/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8878) loss 0.5585 (0.5774) grad_norm 0.2086 (0.2155) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:17:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 673 training takes 0:05:57 [2024-03-07 22:17:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [674/800][0/402] eta 0:30:34 lr 0.000025 time 4.5635 (4.5635) loss 0.5746 (0.5746) grad_norm 0.2061 (0.2061) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:18:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [674/800][100/402] eta 0:04:36 lr 0.000025 time 0.8790 (0.9154) loss 0.5606 (0.5768) grad_norm 0.2178 (0.2150) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:20:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [674/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8971) loss 0.5734 (0.5763) grad_norm 0.2033 (0.2186) loss_scale 524288.0000 (280402.7861) mem 30609MB [2024-03-07 22:21:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [674/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8909) loss 0.5982 (0.5770) grad_norm 0.2140 (nan) loss_scale 262144.0000 (296109.5017) mem 30609MB [2024-03-07 22:23:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [674/800][400/402] eta 0:00:01 lr 0.000025 time 0.8771 (0.8877) loss 0.5654 (0.5766) grad_norm 0.1827 (nan) loss_scale 262144.0000 (287639.3017) mem 30609MB [2024-03-07 22:23:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 674 training takes 0:05:57 [2024-03-07 22:23:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [675/800][0/402] eta 0:30:56 lr 0.000025 time 4.6186 (4.6186) loss 0.5530 (0.5530) grad_norm 0.2274 (0.2274) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:24:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [675/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9156) loss 0.6123 (0.5734) grad_norm 0.2039 (0.2177) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:26:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [675/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8972) loss 0.5736 (0.5745) grad_norm 0.1988 (0.2211) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:27:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [675/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8910) loss 0.5680 (0.5763) grad_norm 0.2262 (0.2195) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:29:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [675/800][400/402] eta 0:00:01 lr 0.000025 time 0.8762 (0.8879) loss 0.5861 (0.5761) grad_norm 0.2201 (0.2189) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:29:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 675 training takes 0:05:57 [2024-03-07 22:29:10 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_675.pth saving...... [2024-03-07 22:29:12 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_675.pth saved !!! [2024-03-07 22:29:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [676/800][0/402] eta 0:30:13 lr 0.000025 time 4.5104 (4.5104) loss 0.6094 (0.6094) grad_norm 0.1908 (0.1908) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:30:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [676/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9146) loss 0.5976 (0.5771) grad_norm 0.1938 (0.2253) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:32:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [676/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8969) loss 0.5964 (0.5758) grad_norm 0.1882 (0.2212) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:33:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [676/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8908) loss 0.5390 (0.5766) grad_norm 0.2655 (0.2214) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:35:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [676/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8877) loss 0.5967 (0.5769) grad_norm 0.2029 (0.2198) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:35:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 676 training takes 0:05:57 [2024-03-07 22:35:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [677/800][0/402] eta 0:30:46 lr 0.000025 time 4.5941 (4.5941) loss 0.5547 (0.5547) grad_norm 0.2292 (0.2292) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:36:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [677/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9153) loss 0.6022 (0.5755) grad_norm 0.2097 (0.2200) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:38:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [677/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8970) loss 0.5482 (0.5758) grad_norm 0.2027 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:39:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [677/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8909) loss 0.5697 (0.5760) grad_norm 0.3086 (0.2185) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:41:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [677/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8878) loss 0.5804 (0.5770) grad_norm 0.2175 (0.2208) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:41:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 677 training takes 0:05:57 [2024-03-07 22:41:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [678/800][0/402] eta 0:30:51 lr 0.000025 time 4.6045 (4.6045) loss 0.5853 (0.5853) grad_norm 0.2560 (0.2560) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:42:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [678/800][100/402] eta 0:04:36 lr 0.000025 time 0.8794 (0.9154) loss 0.5997 (0.5741) grad_norm 0.1977 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:44:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [678/800][200/402] eta 0:03:01 lr 0.000025 time 0.8787 (0.8974) loss 0.5856 (0.5764) grad_norm 0.2236 (0.2161) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:45:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [678/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8912) loss 0.5669 (0.5771) grad_norm 0.2088 (0.2152) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:47:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [678/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8880) loss 0.5874 (0.5767) grad_norm 0.2077 (0.2160) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:47:03 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 678 training takes 0:05:57 [2024-03-07 22:47:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [679/800][0/402] eta 0:30:32 lr 0.000025 time 4.5594 (4.5594) loss 0.5765 (0.5765) grad_norm 0.2289 (0.2289) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:48:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [679/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9150) loss 0.5482 (0.5801) grad_norm 0.1857 (0.2115) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:50:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [679/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8969) loss 0.5696 (0.5787) grad_norm 0.1961 (0.2122) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-07 22:51:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [679/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8908) loss 0.5605 (0.5787) grad_norm 0.1925 (0.2127) loss_scale 524288.0000 (336171.3754) mem 30609MB [2024-03-07 22:52:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [679/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8877) loss 0.5820 (0.5776) grad_norm 0.2002 (0.2139) loss_scale 524288.0000 (383083.2519) mem 30609MB [2024-03-07 22:53:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 679 training takes 0:05:57 [2024-03-07 22:53:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [680/800][0/402] eta 0:30:40 lr 0.000025 time 4.5790 (4.5790) loss 0.5968 (0.5968) grad_norm 0.2215 (0.2215) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 22:54:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [680/800][100/402] eta 0:04:36 lr 0.000025 time 0.8782 (0.9152) loss 0.6019 (0.5736) grad_norm 0.2100 (0.2211) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 22:56:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [680/800][200/402] eta 0:03:01 lr 0.000025 time 0.8779 (0.8970) loss 0.6024 (0.5767) grad_norm 0.2166 (0.2203) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 22:57:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [680/800][300/402] eta 0:01:30 lr 0.000025 time 0.8793 (0.8911) loss 0.5791 (0.5755) grad_norm 0.1863 (0.2195) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 22:58:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [680/800][400/402] eta 0:00:01 lr 0.000025 time 0.8761 (0.8880) loss 0.5801 (0.5765) grad_norm 0.1886 (0.2191) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 22:58:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 680 training takes 0:05:57 [2024-03-07 22:58:58 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_680.pth saving...... [2024-03-07 22:58:59 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_680.pth saved !!! [2024-03-07 22:59:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [681/800][0/402] eta 0:29:58 lr 0.000025 time 4.4734 (4.4734) loss 0.5668 (0.5668) grad_norm 0.2298 (0.2298) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:00:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [681/800][100/402] eta 0:04:36 lr 0.000025 time 0.8786 (0.9141) loss 0.5946 (0.5766) grad_norm 0.2208 (0.2168) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:01:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [681/800][200/402] eta 0:03:01 lr 0.000025 time 0.8807 (0.8964) loss 0.5686 (0.5751) grad_norm 0.1920 (0.2191) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:03:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [681/800][300/402] eta 0:01:30 lr 0.000025 time 0.8796 (0.8906) loss 0.5469 (0.5768) grad_norm 0.2082 (0.2183) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:04:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [681/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8876) loss 0.6026 (0.5769) grad_norm 0.2086 (0.2173) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:04:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 681 training takes 0:05:57 [2024-03-07 23:05:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [682/800][0/402] eta 0:30:44 lr 0.000025 time 4.5885 (4.5885) loss 0.5998 (0.5998) grad_norm 0.1993 (0.1993) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:06:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [682/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9152) loss 0.5798 (0.5795) grad_norm 0.2300 (0.2199) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:07:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [682/800][200/402] eta 0:03:01 lr 0.000025 time 0.8783 (0.8969) loss 0.5950 (0.5787) grad_norm 0.2521 (0.2237) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:09:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [682/800][300/402] eta 0:01:30 lr 0.000025 time 0.8784 (0.8911) loss 0.5923 (0.5778) grad_norm 0.2075 (0.2198) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:10:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [682/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8879) loss 0.5699 (0.5778) grad_norm 0.2378 (0.2192) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:10:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 682 training takes 0:05:57 [2024-03-07 23:10:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [683/800][0/402] eta 0:30:33 lr 0.000025 time 4.5605 (4.5605) loss 0.6000 (0.6000) grad_norm 0.1918 (0.1918) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:12:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [683/800][100/402] eta 0:04:36 lr 0.000025 time 0.8987 (0.9151) loss 0.5883 (0.5746) grad_norm 0.2041 (0.2153) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:13:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [683/800][200/402] eta 0:03:01 lr 0.000025 time 0.8788 (0.8969) loss 0.5764 (0.5746) grad_norm 0.2069 (0.2156) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:15:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [683/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8908) loss 0.5803 (0.5759) grad_norm 0.2360 (0.2153) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:16:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [683/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8877) loss 0.5557 (0.5766) grad_norm 0.2442 (0.2159) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:16:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 683 training takes 0:05:57 [2024-03-07 23:16:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [684/800][0/402] eta 0:30:27 lr 0.000025 time 4.5455 (4.5455) loss 0.5363 (0.5363) grad_norm 0.2652 (0.2652) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:18:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [684/800][100/402] eta 0:04:36 lr 0.000025 time 0.8783 (0.9148) loss 0.5778 (0.5768) grad_norm 0.2209 (0.2220) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:19:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [684/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.8967) loss 0.5447 (0.5765) grad_norm 0.1934 (0.2203) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:21:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [684/800][300/402] eta 0:01:30 lr 0.000025 time 0.8788 (0.8906) loss 0.5783 (0.5756) grad_norm 0.2061 (nan) loss_scale 524288.0000 (606153.5681) mem 30609MB [2024-03-07 23:22:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [684/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8876) loss 0.5363 (0.5758) grad_norm 0.2100 (nan) loss_scale 524288.0000 (585738.2145) mem 30609MB [2024-03-07 23:22:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 684 training takes 0:05:56 [2024-03-07 23:22:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [685/800][0/402] eta 0:30:20 lr 0.000025 time 4.5289 (4.5289) loss 0.5900 (0.5900) grad_norm 0.2078 (0.2078) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:24:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [685/800][100/402] eta 0:04:36 lr 0.000025 time 0.8785 (0.9152) loss 0.5953 (0.5781) grad_norm 0.2355 (0.2354) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:25:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [685/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8973) loss 0.6003 (0.5769) grad_norm 0.2220 (0.2243) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:27:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [685/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8913) loss 0.5587 (0.5763) grad_norm 0.2207 (0.2228) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:28:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [685/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8882) loss 0.5472 (0.5768) grad_norm 0.2517 (0.2237) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:28:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 685 training takes 0:05:57 [2024-03-07 23:28:45 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_685.pth saving...... [2024-03-07 23:28:46 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_685.pth saved !!! [2024-03-07 23:28:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [686/800][0/402] eta 0:27:23 lr 0.000025 time 4.0885 (4.0885) loss 0.5117 (0.5117) grad_norm 0.2818 (0.2818) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:30:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [686/800][100/402] eta 0:04:35 lr 0.000025 time 0.8783 (0.9106) loss 0.6047 (0.5781) grad_norm 0.2368 (0.2222) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:31:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [686/800][200/402] eta 0:03:00 lr 0.000025 time 0.8784 (0.8948) loss 0.6034 (0.5759) grad_norm 0.2013 (0.2212) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:33:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [686/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8895) loss 0.5875 (0.5760) grad_norm 0.2036 (0.2198) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:34:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [686/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8868) loss 0.5644 (0.5764) grad_norm 0.2078 (0.2204) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:34:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 686 training takes 0:05:56 [2024-03-07 23:34:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [687/800][0/402] eta 0:30:45 lr 0.000025 time 4.5904 (4.5904) loss 0.5813 (0.5813) grad_norm 0.1913 (0.1913) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:36:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [687/800][100/402] eta 0:04:36 lr 0.000025 time 0.8787 (0.9159) loss 0.6042 (0.5775) grad_norm 0.2065 (0.2181) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:37:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [687/800][200/402] eta 0:03:01 lr 0.000025 time 0.8794 (0.8975) loss 0.5512 (0.5775) grad_norm 0.1887 (0.2180) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:39:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [687/800][300/402] eta 0:01:30 lr 0.000025 time 0.8800 (0.8912) loss 0.5434 (0.5774) grad_norm 0.2095 (0.2188) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:40:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [687/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8881) loss 0.6087 (0.5774) grad_norm 0.2145 (0.2187) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:40:40 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 687 training takes 0:05:57 [2024-03-07 23:40:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [688/800][0/402] eta 0:31:09 lr 0.000025 time 4.6502 (4.6502) loss 0.6141 (0.6141) grad_norm 0.1957 (0.1957) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:42:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [688/800][100/402] eta 0:04:36 lr 0.000025 time 0.8791 (0.9159) loss 0.5677 (0.5775) grad_norm 0.2317 (0.2247) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:43:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [688/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8974) loss 0.5916 (0.5750) grad_norm 0.2458 (0.2236) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:45:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [688/800][300/402] eta 0:01:30 lr 0.000025 time 0.8785 (0.8912) loss 0.6332 (0.5757) grad_norm 0.2240 (0.2227) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:46:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [688/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8880) loss 0.5913 (0.5768) grad_norm 0.1936 (0.2206) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:46:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 688 training takes 0:05:57 [2024-03-07 23:46:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [689/800][0/402] eta 0:31:21 lr 0.000025 time 4.6814 (4.6814) loss 0.5660 (0.5660) grad_norm 0.2045 (0.2045) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:48:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [689/800][100/402] eta 0:04:36 lr 0.000025 time 0.8781 (0.9162) loss 0.5621 (0.5769) grad_norm 0.2603 (0.2163) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:49:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [689/800][200/402] eta 0:03:01 lr 0.000025 time 0.8784 (0.8978) loss 0.6009 (0.5766) grad_norm 0.1925 (0.2144) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:51:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [689/800][300/402] eta 0:01:30 lr 0.000025 time 0.8786 (0.8915) loss 0.5910 (0.5758) grad_norm 0.1934 (0.2175) loss_scale 1048576.0000 (625313.5947) mem 30609MB [2024-03-07 23:52:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [689/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8882) loss 0.5412 (0.5758) grad_norm 0.2025 (nan) loss_scale 524288.0000 (647188.4289) mem 30609MB [2024-03-07 23:52:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 689 training takes 0:05:57 [2024-03-07 23:52:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [690/800][0/402] eta 0:31:12 lr 0.000025 time 4.6583 (4.6583) loss 0.6107 (0.6107) grad_norm 0.1973 (0.1973) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:54:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [690/800][100/402] eta 0:04:36 lr 0.000025 time 0.8788 (0.9162) loss 0.5942 (0.5812) grad_norm 0.2103 (0.2173) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:55:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [690/800][200/402] eta 0:03:01 lr 0.000025 time 0.8791 (0.8976) loss 0.5572 (0.5774) grad_norm 0.2001 (0.2196) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:57:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [690/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8913) loss 0.5875 (0.5763) grad_norm 0.2242 (0.2232) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:58:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [690/800][400/402] eta 0:00:01 lr 0.000025 time 0.8767 (0.8881) loss 0.5907 (0.5772) grad_norm 0.1935 (0.2221) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-07 23:58:32 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 690 training takes 0:05:57 [2024-03-07 23:58:32 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_690.pth saving...... [2024-03-07 23:58:34 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_690.pth saved !!! [2024-03-07 23:58:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [691/800][0/402] eta 0:32:53 lr 0.000025 time 4.9087 (4.9087) loss 0.5820 (0.5820) grad_norm 0.2413 (0.2413) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:00:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [691/800][100/402] eta 0:04:37 lr 0.000025 time 0.8787 (0.9188) loss 0.5583 (0.5767) grad_norm 0.2395 (0.2182) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:01:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [691/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.8990) loss 0.5833 (0.5765) grad_norm 0.1746 (0.2180) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:03:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [691/800][300/402] eta 0:01:31 lr 0.000025 time 0.8798 (0.8925) loss 0.5588 (0.5762) grad_norm 0.1949 (0.2195) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:04:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [691/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8891) loss 0.5805 (0.5774) grad_norm 0.2204 (0.2179) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:04:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 691 training takes 0:05:57 [2024-03-08 00:04:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [692/800][0/402] eta 0:32:52 lr 0.000025 time 4.9055 (4.9055) loss 0.5772 (0.5772) grad_norm 0.2257 (0.2257) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:06:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [692/800][100/402] eta 0:04:37 lr 0.000025 time 0.8791 (0.9189) loss 0.5605 (0.5731) grad_norm 0.2009 (0.2165) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:07:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [692/800][200/402] eta 0:03:01 lr 0.000025 time 0.8803 (0.8990) loss 0.5695 (0.5741) grad_norm 0.1848 (nan) loss_scale 262144.0000 (444731.8607) mem 30609MB [2024-03-08 00:09:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [692/800][300/402] eta 0:01:31 lr 0.000025 time 0.8786 (0.8925) loss 0.5073 (0.5749) grad_norm 0.2244 (nan) loss_scale 262144.0000 (384071.4419) mem 30609MB [2024-03-08 00:10:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [692/800][400/402] eta 0:00:01 lr 0.000025 time 0.8772 (0.8890) loss 0.5888 (0.5752) grad_norm 0.2169 (nan) loss_scale 262144.0000 (353665.5960) mem 30609MB [2024-03-08 00:10:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 692 training takes 0:05:57 [2024-03-08 00:10:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [693/800][0/402] eta 0:32:18 lr 0.000025 time 4.8224 (4.8224) loss 0.5909 (0.5909) grad_norm 0.2257 (0.2257) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:12:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [693/800][100/402] eta 0:04:37 lr 0.000025 time 0.8786 (0.9179) loss 0.5768 (0.5761) grad_norm 0.2378 (0.2191) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:13:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [693/800][200/402] eta 0:03:01 lr 0.000025 time 0.8782 (0.8985) loss 0.5641 (0.5765) grad_norm 0.2479 (0.2207) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:14:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [693/800][300/402] eta 0:01:30 lr 0.000025 time 0.8783 (0.8920) loss 0.5377 (0.5760) grad_norm 0.2057 (0.2223) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:16:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [693/800][400/402] eta 0:00:01 lr 0.000025 time 0.8769 (0.8888) loss 0.5895 (0.5758) grad_norm 0.1966 (0.2221) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:16:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 693 training takes 0:05:57 [2024-03-08 00:16:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [694/800][0/402] eta 0:32:18 lr 0.000025 time 4.8228 (4.8228) loss 0.6186 (0.6186) grad_norm 0.1777 (0.1777) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:17:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [694/800][100/402] eta 0:04:37 lr 0.000025 time 0.8788 (0.9181) loss 0.5960 (0.5782) grad_norm 0.2132 (0.2220) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:19:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [694/800][200/402] eta 0:03:01 lr 0.000025 time 0.8796 (0.8984) loss 0.5902 (0.5771) grad_norm 0.2189 (0.2207) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:20:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [694/800][300/402] eta 0:01:30 lr 0.000025 time 0.8789 (0.8919) loss 0.5771 (0.5773) grad_norm 0.2589 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:22:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [694/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8889) loss 0.5698 (0.5770) grad_norm 0.2096 (0.2212) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:22:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 694 training takes 0:05:57 [2024-03-08 00:22:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [695/800][0/402] eta 0:31:03 lr 0.000025 time 4.6355 (4.6355) loss 0.5662 (0.5662) grad_norm 0.2050 (0.2050) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:23:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [695/800][100/402] eta 0:04:36 lr 0.000025 time 0.8793 (0.9159) loss 0.5775 (0.5761) grad_norm 0.2057 (0.2189) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:25:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [695/800][200/402] eta 0:03:01 lr 0.000025 time 0.8785 (0.8976) loss 0.5726 (0.5776) grad_norm 0.2371 (0.2206) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:26:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [695/800][300/402] eta 0:01:30 lr 0.000025 time 0.8794 (0.8914) loss 0.5819 (0.5780) grad_norm 0.2031 (0.2213) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:28:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [695/800][400/402] eta 0:00:01 lr 0.000025 time 0.8768 (0.8883) loss 0.5635 (0.5776) grad_norm 0.2229 (0.2208) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:28:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 695 training takes 0:05:57 [2024-03-08 00:28:22 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_695.pth saving...... [2024-03-08 00:28:23 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_695.pth saved !!! [2024-03-08 00:28:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [696/800][0/402] eta 0:35:05 lr 0.000025 time 5.2387 (5.2387) loss 0.6055 (0.6055) grad_norm 0.2097 (0.2097) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:29:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [696/800][100/402] eta 0:04:38 lr 0.000025 time 0.8788 (0.9227) loss 0.5751 (0.5757) grad_norm 0.2075 (0.2218) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:31:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [696/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.9009) loss 0.5941 (0.5761) grad_norm 0.2169 (0.2227) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:32:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [696/800][300/402] eta 0:01:31 lr 0.000025 time 0.8787 (0.8936) loss 0.5672 (0.5768) grad_norm 0.1766 (0.2200) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:34:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [696/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8898) loss 0.5789 (0.5761) grad_norm 0.2111 (0.2186) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:34:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 696 training takes 0:05:57 [2024-03-08 00:34:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [697/800][0/402] eta 0:34:44 lr 0.000025 time 5.1862 (5.1862) loss 0.6040 (0.6040) grad_norm 0.2295 (0.2295) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:35:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [697/800][100/402] eta 0:04:38 lr 0.000025 time 0.8781 (0.9224) loss 0.5962 (0.5735) grad_norm 0.2026 (0.2201) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:37:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [697/800][200/402] eta 0:03:01 lr 0.000025 time 0.8790 (0.9008) loss 0.5755 (0.5745) grad_norm 0.2367 (0.2197) loss_scale 524288.0000 (354742.1294) mem 30609MB [2024-03-08 00:38:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [697/800][300/402] eta 0:01:31 lr 0.000025 time 0.8789 (0.8935) loss 0.5902 (0.5755) grad_norm 0.2137 (0.2189) loss_scale 524288.0000 (411069.6611) mem 30609MB [2024-03-08 00:40:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [697/800][400/402] eta 0:00:01 lr 0.000025 time 0.8764 (0.8898) loss 0.5815 (0.5758) grad_norm 0.2295 (0.2195) loss_scale 524288.0000 (439303.6608) mem 30609MB [2024-03-08 00:40:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 697 training takes 0:05:57 [2024-03-08 00:40:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [698/800][0/402] eta 0:34:04 lr 0.000025 time 5.0855 (5.0855) loss 0.5988 (0.5988) grad_norm 0.1921 (0.1921) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:41:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [698/800][100/402] eta 0:04:38 lr 0.000025 time 0.8791 (0.9206) loss 0.5824 (0.5778) grad_norm 0.2114 (0.2309) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:43:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [698/800][200/402] eta 0:03:01 lr 0.000025 time 0.8789 (0.9003) loss 0.5834 (0.5771) grad_norm 0.2163 (0.2252) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:44:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [698/800][300/402] eta 0:01:31 lr 0.000025 time 0.8786 (0.8933) loss 0.5747 (0.5771) grad_norm 0.1908 (0.2231) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:46:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [698/800][400/402] eta 0:00:01 lr 0.000025 time 0.8763 (0.8896) loss 0.5476 (0.5760) grad_norm 0.2431 (0.2218) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:46:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 698 training takes 0:05:57 [2024-03-08 00:46:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [699/800][0/402] eta 0:34:37 lr 0.000025 time 5.1683 (5.1683) loss 0.6031 (0.6031) grad_norm 0.1946 (0.1946) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:47:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [699/800][100/402] eta 0:04:38 lr 0.000025 time 0.8783 (0.9216) loss 0.5700 (0.5752) grad_norm 0.2338 (0.2185) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:49:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [699/800][200/402] eta 0:03:01 lr 0.000025 time 0.8786 (0.9005) loss 0.5603 (0.5756) grad_norm 0.1914 (0.2203) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:50:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [699/800][300/402] eta 0:01:31 lr 0.000025 time 0.8783 (0.8934) loss 0.5537 (0.5758) grad_norm 0.2365 (0.2204) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:52:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [699/800][400/402] eta 0:00:01 lr 0.000025 time 0.8765 (0.8897) loss 0.5749 (0.5755) grad_norm 0.2200 (0.2213) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:52:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 699 training takes 0:05:57 [2024-03-08 00:52:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [700/800][0/402] eta 0:35:17 lr 0.000003 time 5.2667 (5.2667) loss 0.5914 (0.5914) grad_norm 0.1871 (0.1871) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:53:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [700/800][100/402] eta 0:04:38 lr 0.000003 time 0.8793 (0.9225) loss 0.5866 (0.5781) grad_norm 0.1611 (0.1893) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:55:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [700/800][200/402] eta 0:03:01 lr 0.000003 time 0.8789 (0.9008) loss 0.5542 (0.5760) grad_norm 0.1970 (0.1893) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:56:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [700/800][300/402] eta 0:01:31 lr 0.000003 time 0.8775 (0.8937) loss 0.5656 (0.5750) grad_norm 0.1773 (0.1884) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 00:58:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [700/800][400/402] eta 0:00:01 lr 0.000003 time 0.8762 (0.8900) loss 0.5882 (0.5752) grad_norm 0.1947 (nan) loss_scale 262144.0000 (514482.1147) mem 30609MB [2024-03-08 00:58:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 700 training takes 0:05:58 [2024-03-08 00:58:13 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_700.pth saving...... [2024-03-08 00:58:15 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_700.pth saved !!! [2024-03-08 00:58:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [701/800][0/402] eta 0:35:44 lr 0.000003 time 5.3356 (5.3356) loss 0.6001 (0.6001) grad_norm 0.1885 (0.1885) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 00:59:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [701/800][100/402] eta 0:04:38 lr 0.000003 time 0.8787 (0.9229) loss 0.5205 (0.5762) grad_norm 0.1892 (0.1924) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:01:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [701/800][200/402] eta 0:03:02 lr 0.000003 time 0.8788 (0.9010) loss 0.5639 (0.5766) grad_norm 0.2061 (0.1922) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:02:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [701/800][300/402] eta 0:01:31 lr 0.000003 time 0.8801 (0.8937) loss 0.5591 (0.5761) grad_norm 0.1918 (0.1957) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:04:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [701/800][400/402] eta 0:00:01 lr 0.000003 time 0.8772 (0.8900) loss 0.5717 (0.5758) grad_norm 0.1928 (0.1961) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:04:13 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 701 training takes 0:05:58 [2024-03-08 01:04:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [702/800][0/402] eta 0:32:18 lr 0.000003 time 4.8215 (4.8215) loss 0.5933 (0.5933) grad_norm 0.2101 (0.2101) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:05:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [702/800][100/402] eta 0:04:37 lr 0.000003 time 0.8791 (0.9181) loss 0.5615 (0.5738) grad_norm 0.1799 (0.1926) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:07:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [702/800][200/402] eta 0:03:01 lr 0.000003 time 0.8794 (0.8986) loss 0.5962 (0.5759) grad_norm 0.1814 (0.1947) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:08:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [702/800][300/402] eta 0:01:31 lr 0.000003 time 0.8782 (0.8923) loss 0.5422 (0.5750) grad_norm 0.1898 (0.1946) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:10:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [702/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8889) loss 0.5718 (0.5761) grad_norm 0.1961 (0.1948) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:10:11 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 702 training takes 0:05:57 [2024-03-08 01:10:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [703/800][0/402] eta 0:35:09 lr 0.000003 time 5.2483 (5.2483) loss 0.5861 (0.5861) grad_norm 0.1979 (0.1979) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:11:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [703/800][100/402] eta 0:04:38 lr 0.000003 time 0.8794 (0.9220) loss 0.6138 (0.5768) grad_norm 0.1866 (0.1965) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:13:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [703/800][200/402] eta 0:03:01 lr 0.000003 time 0.8786 (0.9006) loss 0.5919 (0.5761) grad_norm 0.1731 (0.1955) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:14:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [703/800][300/402] eta 0:01:31 lr 0.000003 time 0.8793 (0.8934) loss 0.5825 (0.5760) grad_norm 0.1898 (0.1968) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:16:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [703/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8897) loss 0.6086 (0.5762) grad_norm 0.1996 (0.1976) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:16:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 703 training takes 0:05:57 [2024-03-08 01:16:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [704/800][0/402] eta 0:35:10 lr 0.000003 time 5.2507 (5.2507) loss 0.5762 (0.5762) grad_norm 0.2087 (0.2087) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:17:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [704/800][100/402] eta 0:04:38 lr 0.000003 time 0.8803 (0.9226) loss 0.5774 (0.5748) grad_norm 0.1949 (0.2005) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:19:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [704/800][200/402] eta 0:03:01 lr 0.000003 time 0.8789 (0.9009) loss 0.5628 (0.5749) grad_norm 0.1955 (0.1993) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:20:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [704/800][300/402] eta 0:01:31 lr 0.000003 time 0.8785 (0.8936) loss 0.5564 (0.5745) grad_norm 0.1988 (0.1996) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:22:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [704/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8898) loss 0.5884 (0.5740) grad_norm 0.2011 (0.2003) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:22:07 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 704 training takes 0:05:57 [2024-03-08 01:22:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [705/800][0/402] eta 0:33:45 lr 0.000003 time 5.0389 (5.0389) loss 0.5550 (0.5550) grad_norm 0.2325 (0.2325) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:23:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [705/800][100/402] eta 0:04:37 lr 0.000003 time 0.8788 (0.9205) loss 0.5380 (0.5742) grad_norm 0.2063 (0.2043) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:25:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [705/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.8998) loss 0.5416 (0.5734) grad_norm 0.2299 (0.2029) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:26:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [705/800][300/402] eta 0:01:31 lr 0.000003 time 0.8783 (0.8929) loss 0.5656 (0.5742) grad_norm 0.2040 (0.2024) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:28:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [705/800][400/402] eta 0:00:01 lr 0.000003 time 0.8763 (0.8893) loss 0.6215 (0.5751) grad_norm 0.1921 (0.2026) loss_scale 524288.0000 (278487.1421) mem 30609MB [2024-03-08 01:28:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 705 training takes 0:05:57 [2024-03-08 01:28:04 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_705.pth saving...... [2024-03-08 01:28:07 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_705.pth saved !!! [2024-03-08 01:28:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [706/800][0/402] eta 0:35:31 lr 0.000003 time 5.3019 (5.3019) loss 0.5681 (0.5681) grad_norm 0.2103 (0.2103) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 01:29:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [706/800][100/402] eta 0:04:38 lr 0.000003 time 0.8774 (0.9224) loss 0.6034 (0.5756) grad_norm 0.1837 (0.2061) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 01:31:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [706/800][200/402] eta 0:03:01 lr 0.000003 time 0.8792 (0.9009) loss 0.6224 (0.5754) grad_norm 0.2192 (0.2048) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 01:32:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [706/800][300/402] eta 0:01:31 lr 0.000003 time 0.8784 (0.8935) loss 0.5961 (0.5752) grad_norm 0.1641 (nan) loss_scale 262144.0000 (483355.2159) mem 30609MB [2024-03-08 01:34:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [706/800][400/402] eta 0:00:01 lr 0.000003 time 0.8762 (0.8898) loss 0.5768 (0.5751) grad_norm 0.2013 (nan) loss_scale 262144.0000 (428190.3242) mem 30609MB [2024-03-08 01:34:05 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 706 training takes 0:05:57 [2024-03-08 01:34:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [707/800][0/402] eta 0:33:45 lr 0.000003 time 5.0385 (5.0385) loss 0.5736 (0.5736) grad_norm 0.2073 (0.2073) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:35:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [707/800][100/402] eta 0:04:38 lr 0.000003 time 0.8788 (0.9209) loss 0.5783 (0.5779) grad_norm 0.2048 (0.2061) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:37:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [707/800][200/402] eta 0:03:01 lr 0.000003 time 0.8797 (0.9000) loss 0.5826 (0.5773) grad_norm 0.1946 (0.2044) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:38:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [707/800][300/402] eta 0:01:31 lr 0.000003 time 0.8776 (0.8930) loss 0.5547 (0.5762) grad_norm 0.1862 (0.2047) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:40:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [707/800][400/402] eta 0:00:01 lr 0.000003 time 0.8774 (0.8894) loss 0.6003 (0.5754) grad_norm 0.1915 (0.2052) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:40:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 707 training takes 0:05:57 [2024-03-08 01:40:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [708/800][0/402] eta 0:33:06 lr 0.000003 time 4.9409 (4.9409) loss 0.6011 (0.6011) grad_norm 0.1968 (0.1968) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:41:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [708/800][100/402] eta 0:04:37 lr 0.000003 time 0.8786 (0.9191) loss 0.5730 (0.5735) grad_norm 0.2037 (0.2042) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:43:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [708/800][200/402] eta 0:03:01 lr 0.000003 time 0.8776 (0.8990) loss 0.5496 (0.5752) grad_norm 0.1902 (0.2050) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:44:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [708/800][300/402] eta 0:01:31 lr 0.000003 time 0.8792 (0.8924) loss 0.5657 (0.5751) grad_norm 0.2165 (0.2058) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:45:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [708/800][400/402] eta 0:00:01 lr 0.000003 time 0.8765 (0.8890) loss 0.5729 (0.5746) grad_norm 0.1849 (0.2061) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:46:00 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 708 training takes 0:05:57 [2024-03-08 01:46:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [709/800][0/402] eta 0:34:45 lr 0.000003 time 5.1889 (5.1889) loss 0.5770 (0.5770) grad_norm 0.2414 (0.2414) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:47:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [709/800][100/402] eta 0:04:38 lr 0.000003 time 0.8794 (0.9213) loss 0.5438 (0.5756) grad_norm 0.2233 (0.2068) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:49:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [709/800][200/402] eta 0:03:01 lr 0.000003 time 0.8791 (0.9003) loss 0.5818 (0.5748) grad_norm 0.1857 (0.2060) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:50:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [709/800][300/402] eta 0:01:31 lr 0.000003 time 0.8791 (0.8932) loss 0.5754 (0.5755) grad_norm 0.2248 (0.2074) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:51:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [709/800][400/402] eta 0:00:01 lr 0.000003 time 0.8773 (0.8896) loss 0.5974 (0.5757) grad_norm 0.1967 (0.2076) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:51:58 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 709 training takes 0:05:57 [2024-03-08 01:52:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [710/800][0/402] eta 0:36:42 lr 0.000003 time 5.4789 (5.4789) loss 0.5947 (0.5947) grad_norm 0.1903 (0.1903) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:53:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [710/800][100/402] eta 0:04:39 lr 0.000003 time 0.8780 (0.9244) loss 0.5663 (0.5739) grad_norm 0.1978 (0.2109) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:54:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [710/800][200/402] eta 0:03:02 lr 0.000003 time 0.8782 (0.9016) loss 0.6166 (0.5751) grad_norm 0.2258 (0.2102) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:56:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [710/800][300/402] eta 0:01:31 lr 0.000003 time 0.8786 (0.8941) loss 0.5545 (0.5758) grad_norm 0.2343 (0.2099) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:57:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [710/800][400/402] eta 0:00:01 lr 0.000003 time 0.8775 (0.8903) loss 0.5432 (0.5744) grad_norm 0.2052 (0.2094) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:57:56 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 710 training takes 0:05:58 [2024-03-08 01:57:56 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_710.pth saving...... [2024-03-08 01:57:58 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_710.pth saved !!! [2024-03-08 01:58:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [711/800][0/402] eta 0:32:44 lr 0.000003 time 4.8864 (4.8864) loss 0.5584 (0.5584) grad_norm 0.2168 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 01:59:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [711/800][100/402] eta 0:04:37 lr 0.000003 time 0.8786 (0.9185) loss 0.5790 (0.5727) grad_norm 0.1808 (0.2270) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:00:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [711/800][200/402] eta 0:03:01 lr 0.000003 time 0.8818 (0.8990) loss 0.5773 (0.5742) grad_norm 0.1972 (0.2181) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:02:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [711/800][300/402] eta 0:01:31 lr 0.000003 time 0.8780 (0.8923) loss 0.5802 (0.5743) grad_norm 0.1924 (0.2157) loss_scale 524288.0000 (311785.8870) mem 30609MB [2024-03-08 02:03:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [711/800][400/402] eta 0:00:01 lr 0.000003 time 0.8789 (0.8888) loss 0.5776 (0.5740) grad_norm 0.2197 (0.2141) loss_scale 524288.0000 (364778.9327) mem 30609MB [2024-03-08 02:03:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 711 training takes 0:05:57 [2024-03-08 02:04:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [712/800][0/402] eta 0:36:38 lr 0.000003 time 5.4694 (5.4694) loss 0.5753 (0.5753) grad_norm 0.2438 (0.2438) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:05:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [712/800][100/402] eta 0:04:39 lr 0.000003 time 0.8779 (0.9242) loss 0.5304 (0.5740) grad_norm 0.2030 (0.2089) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:06:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [712/800][200/402] eta 0:03:02 lr 0.000003 time 0.8790 (0.9016) loss 0.5640 (0.5750) grad_norm 0.2122 (0.2090) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:08:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [712/800][300/402] eta 0:01:31 lr 0.000003 time 0.8791 (0.8940) loss 0.5805 (0.5744) grad_norm 0.2356 (0.2099) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:09:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [712/800][400/402] eta 0:00:01 lr 0.000003 time 0.8772 (0.8902) loss 0.5255 (0.5744) grad_norm 0.2087 (0.2102) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:09:54 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 712 training takes 0:05:58 [2024-03-08 02:09:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [713/800][0/402] eta 0:34:28 lr 0.000003 time 5.1461 (5.1461) loss 0.5876 (0.5876) grad_norm 0.2355 (0.2355) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:11:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [713/800][100/402] eta 0:04:38 lr 0.000003 time 0.8786 (0.9214) loss 0.5929 (0.5740) grad_norm 0.2407 (0.2150) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:12:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [713/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.9002) loss 0.5640 (0.5752) grad_norm 0.1908 (0.2131) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:14:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [713/800][300/402] eta 0:01:31 lr 0.000003 time 0.8789 (0.8934) loss 0.5284 (0.5750) grad_norm 0.2241 (0.2133) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:15:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [713/800][400/402] eta 0:00:01 lr 0.000003 time 0.8776 (0.8896) loss 0.6066 (0.5746) grad_norm 0.2047 (0.2136) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:15:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 713 training takes 0:05:57 [2024-03-08 02:15:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [714/800][0/402] eta 0:32:24 lr 0.000003 time 4.8375 (4.8375) loss 0.5623 (0.5623) grad_norm 0.2208 (0.2208) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:17:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [714/800][100/402] eta 0:04:37 lr 0.000003 time 0.8789 (0.9180) loss 0.5585 (0.5747) grad_norm 0.2219 (0.2168) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:18:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [714/800][200/402] eta 0:03:01 lr 0.000003 time 0.8787 (0.8985) loss 0.5633 (0.5757) grad_norm 0.2373 (0.2140) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:20:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [714/800][300/402] eta 0:01:30 lr 0.000003 time 0.8796 (0.8919) loss 0.5994 (0.5759) grad_norm 0.1995 (0.2135) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:21:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [714/800][400/402] eta 0:00:01 lr 0.000003 time 0.8777 (0.8885) loss 0.5653 (0.5761) grad_norm 0.1914 (0.2137) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:21:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 714 training takes 0:05:57 [2024-03-08 02:21:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [715/800][0/402] eta 0:33:26 lr 0.000003 time 4.9909 (4.9909) loss 0.5791 (0.5791) grad_norm 0.2130 (0.2130) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:23:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [715/800][100/402] eta 0:04:37 lr 0.000003 time 0.8777 (0.9196) loss 0.5985 (0.5768) grad_norm 0.2356 (0.2157) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:24:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [715/800][200/402] eta 0:03:01 lr 0.000003 time 0.8782 (0.8991) loss 0.5631 (0.5761) grad_norm 0.2058 (0.2145) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:26:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [715/800][300/402] eta 0:01:31 lr 0.000003 time 0.8786 (0.8925) loss 0.6046 (0.5760) grad_norm 0.2069 (nan) loss_scale 262144.0000 (517320.7176) mem 30609MB [2024-03-08 02:27:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [715/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8891) loss 0.5564 (0.5757) grad_norm 0.2144 (nan) loss_scale 262144.0000 (453685.6259) mem 30609MB [2024-03-08 02:27:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 715 training takes 0:05:57 [2024-03-08 02:27:47 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_715.pth saving...... [2024-03-08 02:27:49 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_715.pth saved !!! [2024-03-08 02:27:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [716/800][0/402] eta 0:30:46 lr 0.000003 time 4.5940 (4.5940) loss 0.5469 (0.5469) grad_norm 0.2068 (0.2068) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:29:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [716/800][100/402] eta 0:04:36 lr 0.000003 time 0.8787 (0.9156) loss 0.5680 (0.5759) grad_norm 0.2245 (0.2131) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:30:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [716/800][200/402] eta 0:03:01 lr 0.000003 time 0.8785 (0.8973) loss 0.5718 (0.5754) grad_norm 0.2431 (0.2134) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:32:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [716/800][300/402] eta 0:01:30 lr 0.000003 time 0.8780 (0.8912) loss 0.5420 (0.5751) grad_norm 0.2362 (0.2141) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:33:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [716/800][400/402] eta 0:00:01 lr 0.000003 time 0.8767 (0.8880) loss 0.5597 (0.5747) grad_norm 0.1977 (0.2143) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:33:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 716 training takes 0:05:57 [2024-03-08 02:33:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [717/800][0/402] eta 0:35:40 lr 0.000003 time 5.3241 (5.3241) loss 0.5589 (0.5589) grad_norm 0.2328 (0.2328) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:35:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [717/800][100/402] eta 0:04:38 lr 0.000003 time 0.8788 (0.9229) loss 0.5664 (0.5730) grad_norm 0.2153 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:36:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [717/800][200/402] eta 0:03:01 lr 0.000003 time 0.8799 (0.9010) loss 0.5591 (0.5723) grad_norm 0.2549 (0.2171) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:38:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [717/800][300/402] eta 0:01:31 lr 0.000003 time 0.8790 (0.8937) loss 0.6027 (0.5741) grad_norm 0.2082 (0.2157) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:39:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [717/800][400/402] eta 0:00:01 lr 0.000003 time 0.8769 (0.8900) loss 0.5493 (0.5744) grad_norm 0.2261 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:39:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 717 training takes 0:05:58 [2024-03-08 02:39:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [718/800][0/402] eta 0:33:44 lr 0.000003 time 5.0352 (5.0352) loss 0.5929 (0.5929) grad_norm 0.2183 (0.2183) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:41:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [718/800][100/402] eta 0:04:37 lr 0.000003 time 0.8781 (0.9201) loss 0.5889 (0.5714) grad_norm 0.2130 (0.2194) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:42:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [718/800][200/402] eta 0:03:01 lr 0.000003 time 0.8793 (0.8997) loss 0.5591 (0.5729) grad_norm 0.1888 (0.2199) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:44:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [718/800][300/402] eta 0:01:31 lr 0.000003 time 0.8779 (0.8927) loss 0.5798 (0.5743) grad_norm 0.1900 (0.2190) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:45:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [718/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8891) loss 0.6113 (0.5751) grad_norm 0.2158 (0.2177) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:45:42 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 718 training takes 0:05:57 [2024-03-08 02:45:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [719/800][0/402] eta 0:32:23 lr 0.000003 time 4.8348 (4.8348) loss 0.5669 (0.5669) grad_norm 0.2444 (0.2444) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:47:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [719/800][100/402] eta 0:04:37 lr 0.000003 time 0.8791 (0.9181) loss 0.5656 (0.5768) grad_norm 0.2035 (0.2159) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:48:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [719/800][200/402] eta 0:03:01 lr 0.000003 time 0.8782 (0.8985) loss 0.5339 (0.5766) grad_norm 0.2175 (0.2165) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:50:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [719/800][300/402] eta 0:01:30 lr 0.000003 time 0.8790 (0.8920) loss 0.5778 (0.5771) grad_norm 0.1895 (0.2172) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:51:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [719/800][400/402] eta 0:00:01 lr 0.000003 time 0.8772 (0.8887) loss 0.6058 (0.5763) grad_norm 0.2165 (0.2168) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:51:39 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 719 training takes 0:05:57 [2024-03-08 02:51:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [720/800][0/402] eta 0:34:38 lr 0.000003 time 5.1704 (5.1704) loss 0.5594 (0.5594) grad_norm 0.2149 (0.2149) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:53:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [720/800][100/402] eta 0:04:38 lr 0.000003 time 0.8783 (0.9217) loss 0.6042 (0.5737) grad_norm 0.1862 (0.2139) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:54:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [720/800][200/402] eta 0:03:01 lr 0.000003 time 0.8793 (0.9004) loss 0.5889 (0.5750) grad_norm 0.2096 (0.2151) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 02:56:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [720/800][300/402] eta 0:01:31 lr 0.000003 time 0.8787 (0.8933) loss 0.5684 (0.5764) grad_norm 0.2322 (0.2165) loss_scale 524288.0000 (277820.3854) mem 30609MB [2024-03-08 02:57:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [720/800][400/402] eta 0:00:01 lr 0.000003 time 0.8763 (0.8897) loss 0.5261 (0.5753) grad_norm 0.2184 (0.2178) loss_scale 524288.0000 (339283.6309) mem 30609MB [2024-03-08 02:57:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 720 training takes 0:05:57 [2024-03-08 02:57:37 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_720.pth saving...... [2024-03-08 02:57:39 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_720.pth saved !!! [2024-03-08 02:57:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [721/800][0/402] eta 0:34:24 lr 0.000003 time 5.1347 (5.1347) loss 0.5638 (0.5638) grad_norm 0.2079 (0.2079) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 02:59:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [721/800][100/402] eta 0:04:38 lr 0.000003 time 0.8791 (0.9209) loss 0.5591 (0.5718) grad_norm 0.2254 (0.2189) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:00:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [721/800][200/402] eta 0:03:01 lr 0.000003 time 0.8789 (0.9000) loss 0.5633 (0.5741) grad_norm 0.2140 (0.2203) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:02:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [721/800][300/402] eta 0:01:31 lr 0.000003 time 0.8786 (0.8930) loss 0.5896 (0.5746) grad_norm 0.2199 (0.2195) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:03:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [721/800][400/402] eta 0:00:01 lr 0.000003 time 0.8768 (0.8894) loss 0.5459 (0.5733) grad_norm 0.2044 (0.2197) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:03:37 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 721 training takes 0:05:57 [2024-03-08 03:03:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [722/800][0/402] eta 0:33:49 lr 0.000003 time 5.0481 (5.0481) loss 0.5617 (0.5617) grad_norm 0.2721 (0.2721) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:05:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [722/800][100/402] eta 0:04:37 lr 0.000003 time 0.8791 (0.9204) loss 0.5571 (0.5746) grad_norm 0.2037 (0.2210) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:06:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [722/800][200/402] eta 0:03:01 lr 0.000003 time 0.8781 (0.9005) loss 0.6069 (0.5752) grad_norm 0.1894 (0.2188) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:08:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [722/800][300/402] eta 0:01:31 lr 0.000003 time 0.8781 (0.8932) loss 0.5693 (0.5742) grad_norm 0.2079 (0.2187) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:09:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [722/800][400/402] eta 0:00:01 lr 0.000003 time 0.8780 (0.8896) loss 0.5581 (0.5744) grad_norm 0.2222 (nan) loss_scale 262144.0000 (518404.4688) mem 30609MB [2024-03-08 03:09:35 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 722 training takes 0:05:57 [2024-03-08 03:09:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [723/800][0/402] eta 0:34:25 lr 0.000003 time 5.1373 (5.1373) loss 0.5693 (0.5693) grad_norm 0.1975 (0.1975) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:11:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [723/800][100/402] eta 0:04:38 lr 0.000003 time 0.8785 (0.9209) loss 0.6195 (0.5721) grad_norm 0.2266 (0.2219) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:12:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [723/800][200/402] eta 0:03:01 lr 0.000003 time 0.8795 (0.8999) loss 0.5512 (0.5747) grad_norm 0.2129 (0.2211) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:14:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [723/800][300/402] eta 0:01:31 lr 0.000003 time 0.8782 (0.8928) loss 0.5985 (0.5754) grad_norm 0.2070 (0.2209) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:15:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [723/800][400/402] eta 0:00:01 lr 0.000003 time 0.8766 (0.8892) loss 0.5469 (0.5747) grad_norm 0.2047 (0.2201) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:15:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 723 training takes 0:05:57 [2024-03-08 03:15:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [724/800][0/402] eta 0:34:26 lr 0.000003 time 5.1399 (5.1399) loss 0.5277 (0.5277) grad_norm 0.2182 (0.2182) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:17:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [724/800][100/402] eta 0:04:38 lr 0.000003 time 0.8786 (0.9208) loss 0.5798 (0.5728) grad_norm 0.2179 (0.2225) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:18:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [724/800][200/402] eta 0:03:01 lr 0.000003 time 0.8785 (0.9000) loss 0.5908 (0.5749) grad_norm 0.2159 (0.2204) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:20:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [724/800][300/402] eta 0:01:31 lr 0.000003 time 0.8784 (0.8931) loss 0.5387 (0.5749) grad_norm 0.2330 (0.2205) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:21:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [724/800][400/402] eta 0:00:01 lr 0.000003 time 0.8771 (0.8895) loss 0.6224 (0.5745) grad_norm 0.2367 (0.2204) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:21:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 724 training takes 0:05:57 [2024-03-08 03:21:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [725/800][0/402] eta 0:35:33 lr 0.000003 time 5.3083 (5.3083) loss 0.5912 (0.5912) grad_norm 0.2101 (0.2101) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:23:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [725/800][100/402] eta 0:04:38 lr 0.000003 time 0.8799 (0.9228) loss 0.5748 (0.5741) grad_norm 0.2197 (0.2209) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:24:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [725/800][200/402] eta 0:03:01 lr 0.000003 time 0.8782 (0.9009) loss 0.5570 (0.5739) grad_norm 0.2139 (0.2204) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:26:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [725/800][300/402] eta 0:01:31 lr 0.000003 time 0.8785 (0.8936) loss 0.5752 (0.5746) grad_norm 0.2041 (0.2206) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:27:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [725/800][400/402] eta 0:00:01 lr 0.000003 time 0.8781 (0.8899) loss 0.5523 (0.5752) grad_norm 0.2387 (0.2205) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:27:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 725 training takes 0:05:57 [2024-03-08 03:27:29 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_725.pth saving...... [2024-03-08 03:27:30 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_725.pth saved !!! [2024-03-08 03:27:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [726/800][0/402] eta 0:32:04 lr 0.000003 time 4.7879 (4.7879) loss 0.5743 (0.5743) grad_norm 0.2215 (0.2215) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:29:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [726/800][100/402] eta 0:04:37 lr 0.000003 time 0.8788 (0.9174) loss 0.5734 (0.5714) grad_norm 0.2138 (0.2211) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:30:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [726/800][200/402] eta 0:03:01 lr 0.000003 time 0.8790 (0.8983) loss 0.5642 (0.5747) grad_norm 0.2380 (0.2231) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:31:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [726/800][300/402] eta 0:01:30 lr 0.000003 time 0.8784 (0.8921) loss 0.5363 (0.5750) grad_norm 0.2227 (0.2235) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:33:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [726/800][400/402] eta 0:00:01 lr 0.000003 time 0.8769 (0.8888) loss 0.5526 (0.5741) grad_norm 0.2629 (0.2237) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:33:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 726 training takes 0:05:57 [2024-03-08 03:33:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [727/800][0/402] eta 0:34:46 lr 0.000003 time 5.1914 (5.1914) loss 0.5831 (0.5831) grad_norm 0.2431 (0.2431) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:35:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [727/800][100/402] eta 0:04:38 lr 0.000003 time 0.8784 (0.9223) loss 0.6107 (0.5751) grad_norm 0.2011 (0.2226) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:36:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [727/800][200/402] eta 0:03:01 lr 0.000003 time 0.8791 (0.9007) loss 0.6163 (0.5749) grad_norm 0.1908 (0.2220) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:37:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [727/800][300/402] eta 0:01:31 lr 0.000003 time 0.8794 (0.8934) loss 0.5649 (0.5749) grad_norm 0.2251 (0.2242) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:39:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [727/800][400/402] eta 0:00:01 lr 0.000003 time 0.8776 (0.8897) loss 0.5682 (0.5753) grad_norm 0.2126 (0.2249) loss_scale 524288.0000 (274564.7880) mem 30609MB [2024-03-08 03:39:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 727 training takes 0:05:57 [2024-03-08 03:39:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [728/800][0/402] eta 0:34:56 lr 0.000003 time 5.2153 (5.2153) loss 0.5538 (0.5538) grad_norm 0.2239 (0.2239) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:40:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [728/800][100/402] eta 0:04:38 lr 0.000003 time 0.8780 (0.9218) loss 0.5496 (0.5721) grad_norm 0.2332 (0.2263) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:42:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [728/800][200/402] eta 0:03:01 lr 0.000003 time 0.8774 (0.9004) loss 0.6177 (0.5739) grad_norm 0.1920 (0.2266) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:43:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [728/800][300/402] eta 0:01:31 lr 0.000003 time 0.8791 (0.8931) loss 0.5636 (0.5742) grad_norm 0.2387 (0.2250) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:45:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [728/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8896) loss 0.5368 (0.5745) grad_norm 0.2382 (0.2251) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:45:24 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 728 training takes 0:05:57 [2024-03-08 03:45:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [729/800][0/402] eta 0:33:38 lr 0.000003 time 5.0218 (5.0218) loss 0.5661 (0.5661) grad_norm 0.2399 (0.2399) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 03:46:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [729/800][100/402] eta 0:04:37 lr 0.000003 time 0.8785 (0.9200) loss 0.5563 (0.5759) grad_norm 0.2297 (nan) loss_scale 262144.0000 (360772.4356) mem 30609MB [2024-03-08 03:48:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [729/800][200/402] eta 0:03:01 lr 0.000003 time 0.8780 (0.8997) loss 0.5768 (0.5762) grad_norm 0.2501 (nan) loss_scale 262144.0000 (311703.5622) mem 30609MB [2024-03-08 03:49:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [729/800][300/402] eta 0:01:31 lr 0.000003 time 0.8789 (0.8927) loss 0.5684 (0.5753) grad_norm 0.2329 (nan) loss_scale 262144.0000 (295238.5914) mem 30609MB [2024-03-08 03:51:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [729/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8893) loss 0.5809 (0.5753) grad_norm 0.2181 (nan) loss_scale 262144.0000 (286985.5761) mem 30609MB [2024-03-08 03:51:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 729 training takes 0:05:57 [2024-03-08 03:51:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [730/800][0/402] eta 0:35:00 lr 0.000003 time 5.2249 (5.2249) loss 0.5588 (0.5588) grad_norm 0.2405 (0.2405) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:52:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [730/800][100/402] eta 0:04:38 lr 0.000003 time 0.8788 (0.9217) loss 0.5668 (0.5742) grad_norm 0.2221 (0.2271) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:54:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [730/800][200/402] eta 0:03:01 lr 0.000003 time 0.8796 (0.9004) loss 0.5839 (0.5731) grad_norm 0.1961 (0.2256) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:55:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [730/800][300/402] eta 0:01:31 lr 0.000003 time 0.8789 (0.8932) loss 0.6039 (0.5730) grad_norm 0.2156 (0.2256) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:57:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [730/800][400/402] eta 0:00:01 lr 0.000003 time 0.8779 (0.8895) loss 0.5794 (0.5737) grad_norm 0.2158 (0.2257) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:57:19 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 730 training takes 0:05:57 [2024-03-08 03:57:19 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_730.pth saving...... [2024-03-08 03:57:21 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_730.pth saved !!! [2024-03-08 03:57:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [731/800][0/402] eta 0:30:29 lr 0.000003 time 4.5516 (4.5516) loss 0.6094 (0.6094) grad_norm 0.2088 (0.2088) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 03:58:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [731/800][100/402] eta 0:04:36 lr 0.000003 time 0.8786 (0.9153) loss 0.5453 (0.5758) grad_norm 0.2345 (0.2267) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:00:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [731/800][200/402] eta 0:03:01 lr 0.000003 time 0.8788 (0.8972) loss 0.5704 (0.5743) grad_norm 0.1986 (0.2272) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:01:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [731/800][300/402] eta 0:01:30 lr 0.000003 time 0.8791 (0.8912) loss 0.5984 (0.5747) grad_norm 0.1991 (nan) loss_scale 131072.0000 (240806.6977) mem 30609MB [2024-03-08 04:03:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [731/800][400/402] eta 0:00:01 lr 0.000003 time 0.8768 (0.8881) loss 0.6022 (0.5741) grad_norm 0.2363 (nan) loss_scale 131072.0000 (213441.4364) mem 30609MB [2024-03-08 04:03:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 731 training takes 0:05:57 [2024-03-08 04:03:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [732/800][0/402] eta 0:35:08 lr 0.000003 time 5.2456 (5.2456) loss 0.6106 (0.6106) grad_norm 0.1962 (0.1962) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:04:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [732/800][100/402] eta 0:04:38 lr 0.000003 time 0.8791 (0.9222) loss 0.5484 (0.5742) grad_norm 0.2696 (0.2254) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:06:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [732/800][200/402] eta 0:03:01 lr 0.000003 time 0.8775 (0.9007) loss 0.5914 (0.5747) grad_norm 0.2290 (0.2246) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:07:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [732/800][300/402] eta 0:01:31 lr 0.000003 time 0.8787 (0.8933) loss 0.5617 (0.5745) grad_norm 0.2227 (0.2251) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:09:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [732/800][400/402] eta 0:00:01 lr 0.000003 time 0.8767 (0.8896) loss 0.5813 (0.5750) grad_norm 0.2461 (0.2252) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:09:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 732 training takes 0:05:57 [2024-03-08 04:09:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [733/800][0/402] eta 0:35:02 lr 0.000003 time 5.2303 (5.2303) loss 0.5487 (0.5487) grad_norm 0.2549 (0.2549) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:10:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [733/800][100/402] eta 0:04:38 lr 0.000003 time 0.8790 (0.9220) loss 0.5196 (0.5745) grad_norm 0.2251 (0.2236) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:12:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [733/800][200/402] eta 0:03:02 lr 0.000003 time 0.8791 (0.9011) loss 0.5894 (0.5747) grad_norm 0.1949 (0.2253) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:13:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [733/800][300/402] eta 0:01:31 lr 0.000003 time 0.8792 (0.8937) loss 0.5725 (0.5753) grad_norm 0.2255 (0.2278) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:15:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [733/800][400/402] eta 0:00:01 lr 0.000003 time 0.8766 (0.8900) loss 0.5698 (0.5753) grad_norm 0.2035 (0.2281) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:15:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 733 training takes 0:05:58 [2024-03-08 04:15:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [734/800][0/402] eta 0:33:21 lr 0.000003 time 4.9794 (4.9794) loss 0.5831 (0.5831) grad_norm 0.2148 (0.2148) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:16:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [734/800][100/402] eta 0:04:37 lr 0.000003 time 0.8793 (0.9193) loss 0.5865 (0.5768) grad_norm 0.2026 (0.2258) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:18:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [734/800][200/402] eta 0:03:01 lr 0.000003 time 0.8782 (0.8992) loss 0.6057 (0.5748) grad_norm 0.2091 (0.2279) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:19:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [734/800][300/402] eta 0:01:31 lr 0.000003 time 0.8788 (0.8924) loss 0.5980 (0.5747) grad_norm 0.2196 (0.2267) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:21:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [734/800][400/402] eta 0:00:01 lr 0.000003 time 0.8766 (0.8890) loss 0.5669 (0.5749) grad_norm 0.2628 (0.2270) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:21:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 734 training takes 0:05:57 [2024-03-08 04:21:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [735/800][0/402] eta 0:34:02 lr 0.000003 time 5.0803 (5.0803) loss 0.5819 (0.5819) grad_norm 0.2071 (0.2071) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:22:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [735/800][100/402] eta 0:04:37 lr 0.000003 time 0.8788 (0.9204) loss 0.6131 (0.5765) grad_norm 0.2353 (0.2265) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:24:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [735/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.8997) loss 0.6007 (0.5749) grad_norm 0.2202 (0.2276) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:25:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [735/800][300/402] eta 0:01:31 lr 0.000003 time 0.8786 (0.8929) loss 0.5372 (0.5735) grad_norm 0.2203 (0.2276) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:27:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [735/800][400/402] eta 0:00:01 lr 0.000003 time 0.8771 (0.8894) loss 0.5773 (0.5735) grad_norm 0.2276 (0.2263) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:27:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 735 training takes 0:05:57 [2024-03-08 04:27:10 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_735.pth saving...... [2024-03-08 04:27:12 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_735.pth saved !!! [2024-03-08 04:27:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [736/800][0/402] eta 0:34:33 lr 0.000003 time 5.1578 (5.1578) loss 0.5568 (0.5568) grad_norm 0.2078 (0.2078) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:28:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [736/800][100/402] eta 0:04:38 lr 0.000003 time 0.8791 (0.9217) loss 0.5726 (0.5733) grad_norm 0.2090 (0.2317) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:30:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [736/800][200/402] eta 0:03:01 lr 0.000003 time 0.8787 (0.9004) loss 0.5693 (0.5746) grad_norm 0.2423 (0.2296) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 04:31:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [736/800][300/402] eta 0:01:31 lr 0.000003 time 0.8777 (0.8932) loss 0.5646 (0.5749) grad_norm 0.2271 (0.2288) loss_scale 262144.0000 (156763.8538) mem 30609MB [2024-03-08 04:33:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [736/800][400/402] eta 0:00:01 lr 0.000003 time 0.8769 (0.8896) loss 0.6043 (0.5749) grad_norm 0.1963 (0.2290) loss_scale 262144.0000 (183043.1920) mem 30609MB [2024-03-08 04:33:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 736 training takes 0:05:57 [2024-03-08 04:33:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [737/800][0/402] eta 0:36:20 lr 0.000003 time 5.4237 (5.4237) loss 0.5545 (0.5545) grad_norm 0.2232 (0.2232) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:34:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [737/800][100/402] eta 0:04:38 lr 0.000003 time 0.8784 (0.9238) loss 0.5827 (0.5769) grad_norm 0.2376 (0.2295) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:36:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [737/800][200/402] eta 0:03:02 lr 0.000003 time 0.8777 (0.9015) loss 0.5533 (0.5767) grad_norm 0.2373 (0.2286) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:37:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [737/800][300/402] eta 0:01:31 lr 0.000003 time 0.8792 (0.8940) loss 0.5845 (0.5744) grad_norm 0.2191 (0.2302) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:39:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [737/800][400/402] eta 0:00:01 lr 0.000003 time 0.8778 (0.8904) loss 0.5661 (0.5737) grad_norm 0.2468 (0.2302) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:39:08 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 737 training takes 0:05:58 [2024-03-08 04:39:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [738/800][0/402] eta 0:34:58 lr 0.000003 time 5.2205 (5.2205) loss 0.6026 (0.6026) grad_norm 0.2619 (0.2619) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:40:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [738/800][100/402] eta 0:04:38 lr 0.000003 time 0.8787 (0.9216) loss 0.5567 (0.5741) grad_norm 0.2263 (0.2267) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:42:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [738/800][200/402] eta 0:03:01 lr 0.000003 time 0.8774 (0.9005) loss 0.5986 (0.5756) grad_norm 0.2278 (0.2288) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:43:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [738/800][300/402] eta 0:01:31 lr 0.000003 time 0.8787 (0.8933) loss 0.5903 (0.5752) grad_norm 0.2588 (0.2301) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:45:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [738/800][400/402] eta 0:00:01 lr 0.000003 time 0.8782 (0.8897) loss 0.6503 (0.5751) grad_norm 0.2259 (0.2305) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:45:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 738 training takes 0:05:57 [2024-03-08 04:45:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [739/800][0/402] eta 0:35:09 lr 0.000003 time 5.2484 (5.2484) loss 0.6098 (0.6098) grad_norm 0.2345 (0.2345) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:46:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [739/800][100/402] eta 0:04:38 lr 0.000003 time 0.8794 (0.9222) loss 0.5902 (0.5768) grad_norm 0.2384 (0.2343) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:48:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [739/800][200/402] eta 0:03:01 lr 0.000003 time 0.8790 (0.9004) loss 0.5639 (0.5759) grad_norm 0.2395 (0.2321) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:49:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [739/800][300/402] eta 0:01:31 lr 0.000003 time 0.8790 (0.8932) loss 0.5839 (0.5747) grad_norm 0.2436 (0.2325) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:51:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [739/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8896) loss 0.5872 (0.5737) grad_norm 0.2291 (0.2325) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:51:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 739 training takes 0:05:57 [2024-03-08 04:51:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [740/800][0/402] eta 0:34:25 lr 0.000003 time 5.1384 (5.1384) loss 0.5714 (0.5714) grad_norm 0.2090 (0.2090) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:52:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [740/800][100/402] eta 0:04:38 lr 0.000003 time 0.8789 (0.9216) loss 0.5563 (0.5752) grad_norm 0.2264 (0.2299) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:54:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [740/800][200/402] eta 0:03:01 lr 0.000003 time 0.8782 (0.9003) loss 0.5852 (0.5735) grad_norm 0.2600 (0.2312) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:55:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [740/800][300/402] eta 0:01:31 lr 0.000003 time 0.8792 (0.8931) loss 0.5396 (0.5736) grad_norm 0.2365 (0.2322) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:57:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [740/800][400/402] eta 0:00:01 lr 0.000003 time 0.8765 (0.8897) loss 0.5301 (0.5736) grad_norm 0.2557 (0.2319) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:57:02 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 740 training takes 0:05:57 [2024-03-08 04:57:02 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_740.pth saving...... [2024-03-08 04:57:04 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_740.pth saved !!! [2024-03-08 04:57:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [741/800][0/402] eta 0:32:29 lr 0.000003 time 4.8493 (4.8493) loss 0.5753 (0.5753) grad_norm 0.2375 (0.2375) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 04:58:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [741/800][100/402] eta 0:04:37 lr 0.000003 time 0.8779 (0.9179) loss 0.5763 (0.5736) grad_norm 0.2567 (0.2317) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:00:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [741/800][200/402] eta 0:03:01 lr 0.000003 time 0.8786 (0.8984) loss 0.5868 (0.5737) grad_norm 0.2442 (0.2304) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:01:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [741/800][300/402] eta 0:01:30 lr 0.000003 time 0.8805 (0.8919) loss 0.5545 (0.5729) grad_norm 0.2554 (0.2313) loss_scale 524288.0000 (322236.8106) mem 30609MB [2024-03-08 05:03:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [741/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8886) loss 0.5926 (0.5730) grad_norm 0.2634 (0.2311) loss_scale 524288.0000 (372623.6409) mem 30609MB [2024-03-08 05:03:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 741 training takes 0:05:57 [2024-03-08 05:03:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [742/800][0/402] eta 0:36:36 lr 0.000003 time 5.4640 (5.4640) loss 0.5602 (0.5602) grad_norm 0.2322 (0.2322) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 05:04:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [742/800][100/402] eta 0:04:39 lr 0.000003 time 0.8788 (0.9251) loss 0.5808 (0.5731) grad_norm 0.2015 (nan) loss_scale 262144.0000 (412682.1386) mem 30609MB [2024-03-08 05:06:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [742/800][200/402] eta 0:03:02 lr 0.000003 time 0.8782 (0.9021) loss 0.5887 (0.5721) grad_norm 0.2158 (nan) loss_scale 262144.0000 (337787.5423) mem 30609MB [2024-03-08 05:07:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [742/800][300/402] eta 0:01:31 lr 0.000003 time 0.8785 (0.8943) loss 0.5628 (0.5728) grad_norm 0.2234 (nan) loss_scale 262144.0000 (312656.7973) mem 30609MB [2024-03-08 05:08:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [742/800][400/402] eta 0:00:01 lr 0.000003 time 0.8761 (0.8904) loss 0.5984 (0.5737) grad_norm 0.2215 (nan) loss_scale 262144.0000 (300060.0898) mem 30609MB [2024-03-08 05:08:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 742 training takes 0:05:58 [2024-03-08 05:09:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [743/800][0/402] eta 0:35:19 lr 0.000003 time 5.2726 (5.2726) loss 0.5550 (0.5550) grad_norm 0.2356 (0.2356) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:10:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [743/800][100/402] eta 0:04:38 lr 0.000003 time 0.8799 (0.9227) loss 0.5776 (0.5731) grad_norm 0.2228 (0.2316) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:12:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [743/800][200/402] eta 0:03:01 lr 0.000003 time 0.8785 (0.9009) loss 0.5576 (0.5733) grad_norm 0.2734 (0.2318) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:13:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [743/800][300/402] eta 0:01:31 lr 0.000003 time 0.8794 (0.8936) loss 0.5523 (0.5736) grad_norm 0.2352 (0.2320) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:14:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [743/800][400/402] eta 0:00:01 lr 0.000003 time 0.8768 (0.8899) loss 0.5891 (0.5739) grad_norm 0.2329 (0.2315) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:14:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 743 training takes 0:05:57 [2024-03-08 05:15:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [744/800][0/402] eta 0:36:22 lr 0.000003 time 5.4289 (5.4289) loss 0.5459 (0.5459) grad_norm 0.2004 (0.2004) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:16:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [744/800][100/402] eta 0:04:38 lr 0.000003 time 0.8791 (0.9238) loss 0.5918 (0.5753) grad_norm 0.2299 (0.2314) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:17:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [744/800][200/402] eta 0:03:02 lr 0.000003 time 0.8788 (0.9013) loss 0.5624 (0.5749) grad_norm 0.2200 (0.2319) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:19:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [744/800][300/402] eta 0:01:31 lr 0.000003 time 0.8783 (0.8940) loss 0.5884 (0.5754) grad_norm 0.2377 (0.2326) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:20:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [744/800][400/402] eta 0:00:01 lr 0.000003 time 0.8777 (0.8901) loss 0.5686 (0.5748) grad_norm 0.2493 (0.2330) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:20:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 744 training takes 0:05:58 [2024-03-08 05:21:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [745/800][0/402] eta 0:34:55 lr 0.000003 time 5.2127 (5.2127) loss 0.5680 (0.5680) grad_norm 0.2170 (0.2170) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:22:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [745/800][100/402] eta 0:04:38 lr 0.000003 time 0.8789 (0.9218) loss 0.5984 (0.5721) grad_norm 0.2696 (0.2365) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:23:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [745/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.9005) loss 0.5888 (0.5735) grad_norm 0.2237 (0.2361) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:25:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [745/800][300/402] eta 0:01:31 lr 0.000003 time 0.8788 (0.8933) loss 0.6101 (0.5749) grad_norm 0.2355 (0.2346) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:26:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [745/800][400/402] eta 0:00:01 lr 0.000003 time 0.8772 (0.8896) loss 0.5690 (0.5751) grad_norm 0.2014 (0.2341) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:26:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 745 training takes 0:05:57 [2024-03-08 05:26:53 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_745.pth saving...... [2024-03-08 05:26:55 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_745.pth saved !!! [2024-03-08 05:27:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [746/800][0/402] eta 0:35:12 lr 0.000003 time 5.2552 (5.2552) loss 0.5917 (0.5917) grad_norm 0.2513 (0.2513) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:28:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [746/800][100/402] eta 0:04:38 lr 0.000003 time 0.8786 (0.9222) loss 0.6117 (0.5756) grad_norm 0.2102 (0.2334) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:29:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [746/800][200/402] eta 0:03:01 lr 0.000003 time 0.8790 (0.9007) loss 0.5768 (0.5742) grad_norm 0.2174 (0.2341) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:31:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [746/800][300/402] eta 0:01:31 lr 0.000003 time 0.8788 (0.8935) loss 0.5487 (0.5745) grad_norm 0.2299 (0.2341) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:32:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [746/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8897) loss 0.5384 (0.5741) grad_norm 0.2187 (0.2342) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:32:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 746 training takes 0:05:57 [2024-03-08 05:32:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [747/800][0/402] eta 0:31:17 lr 0.000003 time 4.6697 (4.6697) loss 0.5769 (0.5769) grad_norm 0.2397 (0.2397) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:34:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [747/800][100/402] eta 0:04:36 lr 0.000003 time 0.8793 (0.9165) loss 0.5743 (0.5759) grad_norm 0.2630 (0.2396) loss_scale 524288.0000 (399704.7129) mem 30609MB [2024-03-08 05:35:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [747/800][200/402] eta 0:03:01 lr 0.000003 time 0.8780 (0.8977) loss 0.6218 (0.5748) grad_norm 0.2239 (nan) loss_scale 262144.0000 (449948.6567) mem 30609MB [2024-03-08 05:37:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [747/800][300/402] eta 0:01:30 lr 0.000003 time 0.8782 (0.8916) loss 0.5823 (0.5755) grad_norm 0.2249 (nan) loss_scale 262144.0000 (387555.0831) mem 30609MB [2024-03-08 05:38:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [747/800][400/402] eta 0:00:01 lr 0.000003 time 0.8773 (0.8883) loss 0.5943 (0.5752) grad_norm 0.2578 (nan) loss_scale 262144.0000 (356280.4988) mem 30609MB [2024-03-08 05:38:51 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 747 training takes 0:05:57 [2024-03-08 05:38:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [748/800][0/402] eta 0:36:25 lr 0.000003 time 5.4356 (5.4356) loss 0.5860 (0.5860) grad_norm 0.2620 (0.2620) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:40:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [748/800][100/402] eta 0:04:38 lr 0.000003 time 0.8786 (0.9238) loss 0.5711 (0.5771) grad_norm 0.2263 (0.2322) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:41:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [748/800][200/402] eta 0:03:02 lr 0.000003 time 0.8788 (0.9015) loss 0.6036 (0.5758) grad_norm 0.2367 (0.2339) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:43:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [748/800][300/402] eta 0:01:31 lr 0.000003 time 0.8792 (0.8940) loss 0.5266 (0.5759) grad_norm 0.2328 (0.2357) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:44:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [748/800][400/402] eta 0:00:01 lr 0.000003 time 0.8776 (0.8902) loss 0.5780 (0.5745) grad_norm 0.2291 (0.2361) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:44:49 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 748 training takes 0:05:58 [2024-03-08 05:44:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [749/800][0/402] eta 0:32:50 lr 0.000003 time 4.9011 (4.9011) loss 0.5955 (0.5955) grad_norm 0.2521 (0.2521) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:46:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [749/800][100/402] eta 0:04:37 lr 0.000003 time 0.8783 (0.9190) loss 0.5756 (0.5760) grad_norm 0.2424 (0.2341) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:47:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [749/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.8991) loss 0.5903 (0.5753) grad_norm 0.2367 (0.2358) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:49:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [749/800][300/402] eta 0:01:31 lr 0.000003 time 0.8780 (0.8924) loss 0.5812 (0.5753) grad_norm 0.2406 (0.2362) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 05:50:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [749/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8889) loss 0.5550 (0.5741) grad_norm 0.2598 (nan) loss_scale 131072.0000 (233380.0698) mem 30609MB [2024-03-08 05:50:46 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 749 training takes 0:05:57 [2024-03-08 05:50:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [750/800][0/402] eta 0:32:39 lr 0.000003 time 4.8744 (4.8744) loss 0.5875 (0.5875) grad_norm 0.2281 (0.2281) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 05:52:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [750/800][100/402] eta 0:04:37 lr 0.000003 time 0.8776 (0.9182) loss 0.5642 (0.5772) grad_norm 0.2093 (0.2343) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 05:53:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [750/800][200/402] eta 0:03:01 lr 0.000003 time 0.8787 (0.8986) loss 0.5775 (0.5750) grad_norm 0.2378 (0.2360) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 05:55:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [750/800][300/402] eta 0:01:30 lr 0.000003 time 0.8816 (0.8921) loss 0.5553 (0.5745) grad_norm 0.2306 (0.2362) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 05:56:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [750/800][400/402] eta 0:00:01 lr 0.000003 time 0.8782 (0.8887) loss 0.5959 (0.5744) grad_norm 0.2368 (0.2363) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 05:56:44 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 750 training takes 0:05:57 [2024-03-08 05:56:44 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_750.pth saving...... [2024-03-08 05:56:46 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_750.pth saved !!! [2024-03-08 05:56:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [751/800][0/402] eta 0:32:01 lr 0.000003 time 4.7789 (4.7789) loss 0.5929 (0.5929) grad_norm 0.2502 (0.2502) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 05:58:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [751/800][100/402] eta 0:04:37 lr 0.000003 time 0.8785 (0.9182) loss 0.5704 (0.5723) grad_norm 0.2100 (0.2337) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 05:59:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [751/800][200/402] eta 0:03:01 lr 0.000003 time 0.8792 (0.8986) loss 0.6113 (0.5746) grad_norm 0.2191 (0.2364) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:01:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [751/800][300/402] eta 0:01:30 lr 0.000003 time 0.8789 (0.8920) loss 0.6361 (0.5740) grad_norm 0.2257 (0.2365) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:02:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [751/800][400/402] eta 0:00:01 lr 0.000003 time 0.8790 (0.8887) loss 0.5947 (0.5746) grad_norm 0.2349 (0.2359) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:02:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 751 training takes 0:05:57 [2024-03-08 06:02:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [752/800][0/402] eta 0:32:20 lr 0.000003 time 4.8271 (4.8271) loss 0.5915 (0.5915) grad_norm 0.2381 (0.2381) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:04:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [752/800][100/402] eta 0:04:37 lr 0.000003 time 0.8804 (0.9179) loss 0.5957 (0.5731) grad_norm 0.2268 (0.2376) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:05:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [752/800][200/402] eta 0:03:01 lr 0.000003 time 0.8786 (0.8988) loss 0.6009 (0.5727) grad_norm 0.2236 (0.2368) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:07:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [752/800][300/402] eta 0:01:31 lr 0.000003 time 0.8810 (0.8922) loss 0.5810 (0.5736) grad_norm 0.2128 (0.2373) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:08:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [752/800][400/402] eta 0:00:01 lr 0.000003 time 0.8769 (0.8888) loss 0.5562 (0.5737) grad_norm 0.2264 (0.2377) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:08:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 752 training takes 0:05:57 [2024-03-08 06:08:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [753/800][0/402] eta 0:32:08 lr 0.000003 time 4.7970 (4.7970) loss 0.5884 (0.5884) grad_norm 0.2294 (0.2294) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:10:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [753/800][100/402] eta 0:04:37 lr 0.000003 time 0.8789 (0.9177) loss 0.5651 (0.5745) grad_norm 0.2647 (0.2369) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:11:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [753/800][200/402] eta 0:03:01 lr 0.000003 time 0.8837 (0.8984) loss 0.5761 (0.5736) grad_norm 0.2500 (0.2391) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:13:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [753/800][300/402] eta 0:01:30 lr 0.000003 time 0.8787 (0.8919) loss 0.5657 (0.5742) grad_norm 0.2508 (0.2390) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:14:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [753/800][400/402] eta 0:00:01 lr 0.000003 time 0.8772 (0.8886) loss 0.5748 (0.5740) grad_norm 0.2446 (0.2389) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:14:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 753 training takes 0:05:57 [2024-03-08 06:14:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [754/800][0/402] eta 0:32:44 lr 0.000003 time 4.8862 (4.8862) loss 0.5811 (0.5811) grad_norm 0.2130 (0.2130) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:16:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [754/800][100/402] eta 0:04:37 lr 0.000003 time 0.8800 (0.9184) loss 0.5895 (0.5746) grad_norm 0.2419 (0.2360) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:17:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [754/800][200/402] eta 0:03:01 lr 0.000003 time 0.8783 (0.8987) loss 0.5636 (0.5750) grad_norm 0.2477 (0.2383) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:19:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [754/800][300/402] eta 0:01:31 lr 0.000003 time 0.8791 (0.8922) loss 0.6023 (0.5750) grad_norm 0.2618 (0.2394) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:20:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [754/800][400/402] eta 0:00:01 lr 0.000003 time 0.8766 (0.8888) loss 0.5881 (0.5753) grad_norm 0.2359 (0.2388) loss_scale 262144.0000 (163104.5586) mem 30609MB [2024-03-08 06:20:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 754 training takes 0:05:57 [2024-03-08 06:20:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [755/800][0/402] eta 0:30:53 lr 0.000003 time 4.6098 (4.6098) loss 0.5523 (0.5523) grad_norm 0.2608 (0.2608) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:22:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [755/800][100/402] eta 0:04:36 lr 0.000003 time 0.8781 (0.9158) loss 0.6033 (0.5747) grad_norm 0.2008 (0.2391) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:23:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [755/800][200/402] eta 0:03:01 lr 0.000003 time 0.8786 (0.8974) loss 0.5663 (0.5738) grad_norm 0.2123 (0.2386) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:25:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [755/800][300/402] eta 0:01:30 lr 0.000003 time 0.8786 (0.8913) loss 0.5814 (0.5745) grad_norm 0.2116 (0.2385) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:26:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [755/800][400/402] eta 0:00:01 lr 0.000003 time 0.8769 (0.8883) loss 0.5757 (0.5744) grad_norm 0.2192 (0.2389) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:26:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 755 training takes 0:05:57 [2024-03-08 06:26:33 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_755.pth saving...... [2024-03-08 06:26:35 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_755.pth saved !!! [2024-03-08 06:26:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [756/800][0/402] eta 0:35:33 lr 0.000003 time 5.3071 (5.3071) loss 0.5963 (0.5963) grad_norm 0.2235 (0.2235) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:28:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [756/800][100/402] eta 0:04:38 lr 0.000003 time 0.8789 (0.9225) loss 0.6038 (0.5724) grad_norm 0.2179 (0.2396) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:29:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [756/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.9007) loss 0.5686 (0.5725) grad_norm 0.2551 (0.2391) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:31:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [756/800][300/402] eta 0:01:31 lr 0.000003 time 0.8788 (0.8934) loss 0.5767 (0.5725) grad_norm 0.2433 (0.2401) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:32:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [756/800][400/402] eta 0:00:01 lr 0.000003 time 0.8774 (0.8898) loss 0.5872 (0.5737) grad_norm 0.2614 (0.2400) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:32:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 756 training takes 0:05:57 [2024-03-08 06:32:38 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [757/800][0/402] eta 0:32:10 lr 0.000003 time 4.8029 (4.8029) loss 0.5766 (0.5766) grad_norm 0.2268 (0.2268) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 06:34:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [757/800][100/402] eta 0:04:37 lr 0.000003 time 0.8778 (0.9173) loss 0.5659 (0.5749) grad_norm 0.2234 (inf) loss_scale 131072.0000 (144049.4257) mem 30609MB [2024-03-08 06:35:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [757/800][200/402] eta 0:03:01 lr 0.000003 time 0.8789 (0.8982) loss 0.5777 (0.5737) grad_norm 0.2632 (inf) loss_scale 131072.0000 (137592.9950) mem 30609MB [2024-03-08 06:37:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [757/800][300/402] eta 0:01:30 lr 0.000003 time 0.8787 (0.8918) loss 0.5370 (0.5735) grad_norm 0.2985 (inf) loss_scale 131072.0000 (135426.5515) mem 30609MB [2024-03-08 06:38:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [757/800][400/402] eta 0:00:01 lr 0.000003 time 0.8773 (0.8887) loss 0.5863 (0.5740) grad_norm 0.2178 (inf) loss_scale 131072.0000 (134340.6284) mem 30609MB [2024-03-08 06:38:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 757 training takes 0:05:57 [2024-03-08 06:38:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [758/800][0/402] eta 0:34:08 lr 0.000003 time 5.0958 (5.0958) loss 0.5841 (0.5841) grad_norm 0.2381 (0.2381) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:40:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [758/800][100/402] eta 0:04:37 lr 0.000003 time 0.8789 (0.9205) loss 0.5558 (0.5740) grad_norm 0.2548 (0.2410) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:41:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [758/800][200/402] eta 0:03:01 lr 0.000003 time 0.8787 (0.8997) loss 0.5673 (0.5751) grad_norm 0.2341 (0.2408) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:42:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [758/800][300/402] eta 0:01:31 lr 0.000003 time 0.8801 (0.8927) loss 0.5537 (0.5746) grad_norm 0.2314 (0.2407) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:44:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [758/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8891) loss 0.5777 (0.5750) grad_norm 0.2616 (0.2404) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:44:28 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 758 training takes 0:05:57 [2024-03-08 06:44:33 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [759/800][0/402] eta 0:32:11 lr 0.000003 time 4.8049 (4.8049) loss 0.5444 (0.5444) grad_norm 0.2447 (0.2447) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:46:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [759/800][100/402] eta 0:04:37 lr 0.000003 time 0.8789 (0.9182) loss 0.5556 (0.5739) grad_norm 0.2274 (0.2365) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:47:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [759/800][200/402] eta 0:03:01 lr 0.000003 time 0.8787 (0.8987) loss 0.5699 (0.5752) grad_norm 0.2330 (0.2378) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:48:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [759/800][300/402] eta 0:01:30 lr 0.000003 time 0.8782 (0.8921) loss 0.5819 (0.5745) grad_norm 0.2542 (0.2397) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:50:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [759/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8887) loss 0.6110 (0.5746) grad_norm 0.2249 (0.2407) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:50:26 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 759 training takes 0:05:57 [2024-03-08 06:50:31 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [760/800][0/402] eta 0:30:59 lr 0.000003 time 4.6259 (4.6259) loss 0.5345 (0.5345) grad_norm 0.2975 (0.2975) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:51:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [760/800][100/402] eta 0:04:36 lr 0.000003 time 0.8788 (0.9163) loss 0.5723 (0.5761) grad_norm 0.2112 (0.2401) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:53:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [760/800][200/402] eta 0:03:01 lr 0.000003 time 0.8785 (0.8976) loss 0.5185 (0.5735) grad_norm 0.2525 (0.2396) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:54:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [760/800][300/402] eta 0:01:30 lr 0.000003 time 0.8799 (0.8913) loss 0.5840 (0.5740) grad_norm 0.2469 (0.2399) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:56:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [760/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8881) loss 0.5744 (0.5743) grad_norm 0.2220 (0.2402) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:56:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 760 training takes 0:05:57 [2024-03-08 06:56:23 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_760.pth saving...... [2024-03-08 06:56:25 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_760.pth saved !!! [2024-03-08 06:56:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [761/800][0/402] eta 0:34:24 lr 0.000003 time 5.1365 (5.1365) loss 0.5775 (0.5775) grad_norm 0.2545 (0.2545) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:57:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [761/800][100/402] eta 0:04:38 lr 0.000003 time 0.8781 (0.9210) loss 0.5602 (0.5755) grad_norm 0.2330 (0.2405) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 06:59:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [761/800][200/402] eta 0:03:01 lr 0.000003 time 0.8781 (0.9001) loss 0.5576 (0.5756) grad_norm 0.2337 (0.2407) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 07:00:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [761/800][300/402] eta 0:01:31 lr 0.000003 time 0.8785 (0.8930) loss 0.5847 (0.5748) grad_norm 0.2656 (0.2401) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 07:02:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [761/800][400/402] eta 0:00:01 lr 0.000003 time 0.8765 (0.8894) loss 0.5892 (0.5750) grad_norm 0.2239 (0.2402) loss_scale 131072.0000 (131072.0000) mem 30609MB [2024-03-08 07:02:23 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 761 training takes 0:05:57 [2024-03-08 07:02:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [762/800][0/402] eta 0:31:24 lr 0.000003 time 4.6880 (4.6880) loss 0.5677 (0.5677) grad_norm 0.2702 (0.2702) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:03:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [762/800][100/402] eta 0:04:36 lr 0.000003 time 0.8790 (0.9170) loss 0.5937 (0.5772) grad_norm 0.2217 (0.2448) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:05:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [762/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.8979) loss 0.5664 (0.5752) grad_norm 0.2581 (0.2461) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:06:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [762/800][300/402] eta 0:01:30 lr 0.000003 time 0.8798 (0.8916) loss 0.5576 (0.5750) grad_norm 0.2708 (0.2459) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:08:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [762/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8884) loss 0.5745 (0.5742) grad_norm 0.2553 (0.2454) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:08:21 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 762 training takes 0:05:57 [2024-03-08 07:08:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [763/800][0/402] eta 0:33:15 lr 0.000003 time 4.9644 (4.9644) loss 0.5846 (0.5846) grad_norm 0.2216 (0.2216) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:09:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [763/800][100/402] eta 0:04:37 lr 0.000003 time 0.8796 (0.9192) loss 0.5637 (0.5734) grad_norm 0.2318 (0.2391) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:11:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [763/800][200/402] eta 0:03:01 lr 0.000003 time 0.8794 (0.8991) loss 0.5788 (0.5742) grad_norm 0.2337 (0.2417) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:12:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [763/800][300/402] eta 0:01:31 lr 0.000003 time 0.8777 (0.8923) loss 0.5953 (0.5741) grad_norm 0.2642 (0.2434) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:14:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [763/800][400/402] eta 0:00:01 lr 0.000003 time 0.8769 (0.8891) loss 0.5612 (0.5745) grad_norm 0.2239 (0.2430) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:14:18 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 763 training takes 0:05:57 [2024-03-08 07:14:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [764/800][0/402] eta 0:31:55 lr 0.000003 time 4.7660 (4.7660) loss 0.5799 (0.5799) grad_norm 0.2507 (0.2507) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:15:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [764/800][100/402] eta 0:04:37 lr 0.000003 time 0.8778 (0.9173) loss 0.5705 (0.5740) grad_norm 0.2612 (0.2439) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:17:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [764/800][200/402] eta 0:03:01 lr 0.000003 time 0.8787 (0.8983) loss 0.5446 (0.5749) grad_norm 0.2356 (0.2435) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:18:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [764/800][300/402] eta 0:01:30 lr 0.000003 time 0.8782 (0.8918) loss 0.5961 (0.5741) grad_norm 0.2415 (0.2431) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:20:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [764/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8884) loss 0.6044 (0.5745) grad_norm 0.2190 (0.2424) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:20:16 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 764 training takes 0:05:57 [2024-03-08 07:20:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [765/800][0/402] eta 0:36:10 lr 0.000003 time 5.3997 (5.3997) loss 0.5924 (0.5924) grad_norm 0.2364 (0.2364) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:21:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [765/800][100/402] eta 0:04:38 lr 0.000003 time 0.8790 (0.9236) loss 0.6243 (0.5726) grad_norm 0.2547 (0.2486) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:23:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [765/800][200/402] eta 0:03:02 lr 0.000003 time 0.8789 (0.9012) loss 0.5818 (0.5737) grad_norm 0.2441 (0.2464) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:24:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [765/800][300/402] eta 0:01:31 lr 0.000003 time 0.8788 (0.8938) loss 0.5803 (0.5741) grad_norm 0.2406 (0.2458) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:26:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [765/800][400/402] eta 0:00:01 lr 0.000003 time 0.8768 (0.8899) loss 0.5431 (0.5743) grad_norm 0.2243 (0.2458) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:26:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 765 training takes 0:05:58 [2024-03-08 07:26:14 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_765.pth saving...... [2024-03-08 07:26:16 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_765.pth saved !!! [2024-03-08 07:26:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [766/800][0/402] eta 0:33:18 lr 0.000003 time 4.9717 (4.9717) loss 0.5750 (0.5750) grad_norm 0.2756 (0.2756) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:27:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [766/800][100/402] eta 0:04:37 lr 0.000003 time 0.8787 (0.9202) loss 0.5764 (0.5729) grad_norm 0.2464 (0.2462) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:29:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [766/800][200/402] eta 0:03:01 lr 0.000003 time 0.8782 (0.8997) loss 0.5742 (0.5730) grad_norm 0.2893 (0.2449) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:30:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [766/800][300/402] eta 0:01:31 lr 0.000003 time 0.8793 (0.8927) loss 0.5574 (0.5740) grad_norm 0.2664 (0.2443) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:32:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [766/800][400/402] eta 0:00:01 lr 0.000003 time 0.8772 (0.8892) loss 0.5389 (0.5742) grad_norm 0.2371 (0.2442) loss_scale 524288.0000 (268027.5312) mem 30609MB [2024-03-08 07:32:14 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 766 training takes 0:05:57 [2024-03-08 07:32:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [767/800][0/402] eta 0:33:31 lr 0.000003 time 5.0025 (5.0025) loss 0.5752 (0.5752) grad_norm 0.2367 (0.2367) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 07:33:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [767/800][100/402] eta 0:04:37 lr 0.000003 time 0.8792 (0.9198) loss 0.5882 (0.5771) grad_norm 0.2400 (0.2462) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 07:35:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [767/800][200/402] eta 0:03:01 lr 0.000003 time 0.8790 (0.8996) loss 0.5964 (0.5749) grad_norm 0.2589 (0.2456) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 07:36:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [767/800][300/402] eta 0:01:31 lr 0.000003 time 0.8800 (0.8928) loss 0.5768 (0.5749) grad_norm 0.2534 (nan) loss_scale 262144.0000 (480742.4850) mem 30609MB [2024-03-08 07:38:10 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [767/800][400/402] eta 0:00:01 lr 0.000003 time 0.8780 (0.8893) loss 0.5739 (0.5752) grad_norm 0.2602 (nan) loss_scale 262144.0000 (426229.1471) mem 30609MB [2024-03-08 07:38:11 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 767 training takes 0:05:57 [2024-03-08 07:38:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [768/800][0/402] eta 0:32:21 lr 0.000003 time 4.8302 (4.8302) loss 0.5804 (0.5804) grad_norm 0.2471 (0.2471) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:39:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [768/800][100/402] eta 0:04:37 lr 0.000003 time 0.8784 (0.9179) loss 0.5978 (0.5754) grad_norm 0.2533 (0.2447) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:41:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [768/800][200/402] eta 0:03:01 lr 0.000003 time 0.8779 (0.8987) loss 0.5769 (0.5763) grad_norm 0.2044 (0.2452) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:42:40 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [768/800][300/402] eta 0:01:30 lr 0.000003 time 0.8781 (0.8921) loss 0.5768 (0.5763) grad_norm 0.2396 (0.2456) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:44:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [768/800][400/402] eta 0:00:01 lr 0.000003 time 0.8765 (0.8887) loss 0.5745 (0.5758) grad_norm 0.2247 (0.2453) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:44:09 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 768 training takes 0:05:57 [2024-03-08 07:44:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [769/800][0/402] eta 0:32:42 lr 0.000003 time 4.8828 (4.8828) loss 0.6212 (0.6212) grad_norm 0.2354 (0.2354) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:45:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [769/800][100/402] eta 0:04:37 lr 0.000003 time 0.8792 (0.9186) loss 0.6021 (0.5736) grad_norm 0.2497 (0.2445) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:47:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [769/800][200/402] eta 0:03:01 lr 0.000003 time 0.8787 (0.8988) loss 0.5810 (0.5745) grad_norm 0.2929 (0.2458) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:48:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [769/800][300/402] eta 0:01:30 lr 0.000003 time 0.8809 (0.8922) loss 0.5442 (0.5745) grad_norm 0.2595 (0.2451) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:50:05 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [769/800][400/402] eta 0:00:01 lr 0.000003 time 0.8769 (0.8887) loss 0.5795 (0.5744) grad_norm 0.2414 (0.2456) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:50:06 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 769 training takes 0:05:57 [2024-03-08 07:50:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [770/800][0/402] eta 0:34:21 lr 0.000003 time 5.1291 (5.1291) loss 0.5892 (0.5892) grad_norm 0.2499 (0.2499) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:51:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [770/800][100/402] eta 0:04:38 lr 0.000003 time 0.8778 (0.9210) loss 0.5967 (0.5735) grad_norm 0.2469 (0.2489) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:53:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [770/800][200/402] eta 0:03:01 lr 0.000003 time 0.8781 (0.8999) loss 0.5989 (0.5742) grad_norm 0.2371 (0.2470) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:54:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [770/800][300/402] eta 0:01:31 lr 0.000003 time 0.8787 (0.8930) loss 0.5517 (0.5745) grad_norm 0.2611 (0.2451) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:56:03 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [770/800][400/402] eta 0:00:01 lr 0.000003 time 0.8775 (0.8894) loss 0.5880 (0.5744) grad_norm 0.2500 (0.2454) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:56:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 770 training takes 0:05:57 [2024-03-08 07:56:04 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_770.pth saving...... [2024-03-08 07:56:06 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_770.pth saved !!! [2024-03-08 07:56:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [771/800][0/402] eta 0:33:04 lr 0.000003 time 4.9361 (4.9361) loss 0.5700 (0.5700) grad_norm 0.2333 (0.2333) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:57:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [771/800][100/402] eta 0:04:37 lr 0.000003 time 0.8790 (0.9188) loss 0.5713 (0.5752) grad_norm 0.2148 (0.2445) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 07:59:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [771/800][200/402] eta 0:03:01 lr 0.000003 time 0.8781 (0.8989) loss 0.5485 (0.5735) grad_norm 0.2176 (0.2436) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:00:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [771/800][300/402] eta 0:01:30 lr 0.000003 time 0.8784 (0.8921) loss 0.5501 (0.5740) grad_norm 0.2646 (0.2438) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:02:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [771/800][400/402] eta 0:00:01 lr 0.000003 time 0.8800 (0.8888) loss 0.5927 (0.5732) grad_norm 0.2265 (0.2445) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:02:04 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 771 training takes 0:05:57 [2024-03-08 08:02:08 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [772/800][0/402] eta 0:32:58 lr 0.000003 time 4.9205 (4.9205) loss 0.5774 (0.5774) grad_norm 0.2558 (0.2558) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:03:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [772/800][100/402] eta 0:04:37 lr 0.000003 time 0.8785 (0.9189) loss 0.5654 (0.5756) grad_norm 0.2530 (0.2479) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:05:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [772/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.8989) loss 0.5744 (0.5745) grad_norm 0.2237 (0.2469) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:06:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [772/800][300/402] eta 0:01:31 lr 0.000003 time 0.8848 (0.8922) loss 0.5649 (0.5751) grad_norm 0.2382 (0.2458) loss_scale 524288.0000 (314398.6179) mem 30609MB [2024-03-08 08:08:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [772/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8890) loss 0.5623 (0.5747) grad_norm 0.2381 (0.2448) loss_scale 524288.0000 (366740.1097) mem 30609MB [2024-03-08 08:08:01 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 772 training takes 0:05:57 [2024-03-08 08:08:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [773/800][0/402] eta 0:34:29 lr 0.000003 time 5.1475 (5.1475) loss 0.5546 (0.5546) grad_norm 0.2338 (0.2338) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 08:09:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [773/800][100/402] eta 0:04:38 lr 0.000003 time 0.8781 (0.9214) loss 0.5892 (0.5767) grad_norm 0.2294 (nan) loss_scale 262144.0000 (485355.7228) mem 30609MB [2024-03-08 08:11:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [773/800][200/402] eta 0:03:01 lr 0.000003 time 0.8778 (0.9002) loss 0.5859 (0.5728) grad_norm 0.2280 (nan) loss_scale 262144.0000 (374305.1144) mem 30609MB [2024-03-08 08:12:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [773/800][300/402] eta 0:01:31 lr 0.000003 time 0.8783 (0.8931) loss 0.5624 (0.5730) grad_norm 0.2200 (nan) loss_scale 262144.0000 (337042.2857) mem 30609MB [2024-03-08 08:13:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [773/800][400/402] eta 0:00:01 lr 0.000003 time 0.8777 (0.8895) loss 0.5820 (0.5729) grad_norm 0.2374 (nan) loss_scale 262144.0000 (318364.4090) mem 30609MB [2024-03-08 08:13:59 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 773 training takes 0:05:57 [2024-03-08 08:14:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [774/800][0/402] eta 0:33:52 lr 0.000003 time 5.0557 (5.0557) loss 0.5724 (0.5724) grad_norm 0.2571 (0.2571) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:15:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [774/800][100/402] eta 0:04:37 lr 0.000003 time 0.8786 (0.9202) loss 0.5899 (0.5769) grad_norm 0.3000 (0.2463) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:17:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [774/800][200/402] eta 0:03:01 lr 0.000003 time 0.8780 (0.8996) loss 0.6207 (0.5754) grad_norm 0.2175 (0.2485) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:18:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [774/800][300/402] eta 0:01:31 lr 0.000003 time 0.8786 (0.8927) loss 0.5816 (0.5749) grad_norm 0.2489 (0.2482) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:19:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [774/800][400/402] eta 0:00:01 lr 0.000003 time 0.8768 (0.8892) loss 0.5999 (0.5752) grad_norm 0.2655 (0.2471) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:19:57 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 774 training takes 0:05:57 [2024-03-08 08:20:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [775/800][0/402] eta 0:35:38 lr 0.000003 time 5.3189 (5.3189) loss 0.5908 (0.5908) grad_norm 0.2249 (0.2249) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:21:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [775/800][100/402] eta 0:04:38 lr 0.000003 time 0.8789 (0.9229) loss 0.5826 (0.5741) grad_norm 0.2680 (0.2510) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:22:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [775/800][200/402] eta 0:03:02 lr 0.000003 time 0.8783 (0.9011) loss 0.6004 (0.5746) grad_norm 0.2326 (0.2515) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:24:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [775/800][300/402] eta 0:01:31 lr 0.000003 time 0.8787 (0.8937) loss 0.5816 (0.5750) grad_norm 0.2351 (0.2508) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:25:54 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [775/800][400/402] eta 0:00:01 lr 0.000003 time 0.8767 (0.8899) loss 0.5757 (0.5746) grad_norm 0.2620 (0.2500) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:25:55 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 775 training takes 0:05:58 [2024-03-08 08:25:55 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_775.pth saving...... [2024-03-08 08:25:56 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_775.pth saved !!! [2024-03-08 08:26:01 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [776/800][0/402] eta 0:29:46 lr 0.000003 time 4.4439 (4.4439) loss 0.5683 (0.5683) grad_norm 0.2421 (0.2421) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:27:29 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [776/800][100/402] eta 0:04:36 lr 0.000003 time 0.8786 (0.9140) loss 0.5994 (0.5756) grad_norm 0.2389 (0.2490) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:28:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [776/800][200/402] eta 0:03:01 lr 0.000003 time 0.8789 (0.8964) loss 0.5391 (0.5741) grad_norm 0.2664 (0.2468) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:30:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [776/800][300/402] eta 0:01:30 lr 0.000003 time 0.8786 (0.8906) loss 0.5899 (0.5741) grad_norm 0.2464 (0.2472) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:31:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [776/800][400/402] eta 0:00:01 lr 0.000003 time 0.8774 (0.8876) loss 0.5845 (0.5743) grad_norm 0.2555 (0.2475) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:31:53 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 776 training takes 0:05:57 [2024-03-08 08:31:59 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [777/800][0/402] eta 0:36:19 lr 0.000003 time 5.4226 (5.4226) loss 0.5783 (0.5783) grad_norm 0.2542 (0.2542) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:33:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [777/800][100/402] eta 0:04:38 lr 0.000003 time 0.8790 (0.9235) loss 0.5469 (0.5738) grad_norm 0.2516 (0.2466) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:34:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [777/800][200/402] eta 0:03:02 lr 0.000003 time 0.8781 (0.9014) loss 0.5734 (0.5745) grad_norm 0.2388 (0.2474) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:36:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [777/800][300/402] eta 0:01:31 lr 0.000003 time 0.8787 (0.8940) loss 0.5511 (0.5742) grad_norm 0.2262 (0.2492) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:37:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [777/800][400/402] eta 0:00:01 lr 0.000003 time 0.8767 (0.8902) loss 0.5714 (0.5739) grad_norm 0.2614 (0.2495) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:37:52 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 777 training takes 0:05:58 [2024-03-08 08:37:57 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [778/800][0/402] eta 0:36:31 lr 0.000003 time 5.4506 (5.4506) loss 0.5662 (0.5662) grad_norm 0.2566 (0.2566) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:39:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [778/800][100/402] eta 0:04:39 lr 0.000003 time 0.8790 (0.9241) loss 0.5793 (0.5736) grad_norm 0.2158 (0.2456) loss_scale 524288.0000 (327031.1287) mem 30609MB [2024-03-08 08:40:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [778/800][200/402] eta 0:03:02 lr 0.000003 time 0.8800 (0.9016) loss 0.5659 (0.5738) grad_norm 0.2665 (0.2492) loss_scale 524288.0000 (425168.8756) mem 30609MB [2024-03-08 08:42:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [778/800][300/402] eta 0:01:31 lr 0.000003 time 0.8780 (0.8939) loss 0.5705 (0.5748) grad_norm 0.2774 (0.2488) loss_scale 524288.0000 (458098.8173) mem 30609MB [2024-03-08 08:43:49 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [778/800][400/402] eta 0:00:01 lr 0.000003 time 0.8779 (0.8901) loss 0.6106 (0.5750) grad_norm 0.2563 (0.2492) loss_scale 524288.0000 (474604.8479) mem 30609MB [2024-03-08 08:43:50 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 778 training takes 0:05:58 [2024-03-08 08:43:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [779/800][0/402] eta 0:32:44 lr 0.000003 time 4.8872 (4.8872) loss 0.5663 (0.5663) grad_norm 0.2703 (0.2703) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 08:45:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [779/800][100/402] eta 0:04:37 lr 0.000003 time 0.8781 (0.9184) loss 0.5447 (0.5742) grad_norm 0.2619 (inf) loss_scale 262144.0000 (506119.6040) mem 30609MB [2024-03-08 08:46:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [779/800][200/402] eta 0:03:01 lr 0.000003 time 0.8780 (0.8987) loss 0.6008 (0.5750) grad_norm 0.2328 (inf) loss_scale 262144.0000 (384738.7065) mem 30609MB [2024-03-08 08:48:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [779/800][300/402] eta 0:01:31 lr 0.000003 time 0.8776 (0.8923) loss 0.5666 (0.5744) grad_norm 0.2575 (inf) loss_scale 262144.0000 (344009.5681) mem 30609MB [2024-03-08 08:49:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [779/800][400/402] eta 0:00:01 lr 0.000003 time 0.8762 (0.8890) loss 0.5657 (0.5751) grad_norm 0.2804 (inf) loss_scale 262144.0000 (323594.2145) mem 30609MB [2024-03-08 08:49:47 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 779 training takes 0:05:57 [2024-03-08 08:49:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [780/800][0/402] eta 0:35:22 lr 0.000003 time 5.2799 (5.2799) loss 0.5636 (0.5636) grad_norm 0.2417 (0.2417) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:51:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [780/800][100/402] eta 0:04:38 lr 0.000003 time 0.8786 (0.9223) loss 0.5750 (0.5725) grad_norm 0.2394 (0.2539) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:52:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [780/800][200/402] eta 0:03:01 lr 0.000003 time 0.8794 (0.9006) loss 0.5629 (0.5725) grad_norm 0.2183 (0.2504) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:54:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [780/800][300/402] eta 0:01:31 lr 0.000003 time 0.8774 (0.8934) loss 0.5500 (0.5730) grad_norm 0.2668 (0.2503) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:55:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [780/800][400/402] eta 0:00:01 lr 0.000003 time 0.8766 (0.8898) loss 0.5977 (0.5730) grad_norm 0.2941 (0.2505) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:55:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 780 training takes 0:05:57 [2024-03-08 08:55:45 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_780.pth saving...... [2024-03-08 08:55:47 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_780.pth saved !!! [2024-03-08 08:55:52 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [781/800][0/402] eta 0:31:26 lr 0.000003 time 4.6936 (4.6936) loss 0.5885 (0.5885) grad_norm 0.2771 (0.2771) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:57:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [781/800][100/402] eta 0:04:36 lr 0.000003 time 0.8791 (0.9165) loss 0.5972 (0.5762) grad_norm 0.2595 (0.2482) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 08:58:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [781/800][200/402] eta 0:03:01 lr 0.000003 time 0.8786 (0.8976) loss 0.5710 (0.5755) grad_norm 0.2581 (0.2512) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:00:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [781/800][300/402] eta 0:01:30 lr 0.000003 time 0.8798 (0.8914) loss 0.5945 (0.5746) grad_norm 0.2228 (0.2521) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:01:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [781/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8885) loss 0.5582 (0.5744) grad_norm 0.2701 (0.2524) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:01:45 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 781 training takes 0:05:57 [2024-03-08 09:01:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [782/800][0/402] eta 0:36:04 lr 0.000003 time 5.3851 (5.3851) loss 0.5573 (0.5573) grad_norm 0.2457 (0.2457) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:03:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [782/800][100/402] eta 0:04:39 lr 0.000003 time 0.8785 (0.9240) loss 0.5664 (0.5743) grad_norm 0.2472 (0.2481) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:04:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [782/800][200/402] eta 0:03:02 lr 0.000003 time 0.8792 (0.9016) loss 0.5698 (0.5741) grad_norm 0.2692 (0.2497) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:06:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [782/800][300/402] eta 0:01:31 lr 0.000003 time 0.8790 (0.8940) loss 0.5619 (0.5742) grad_norm 0.2576 (0.2507) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:07:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [782/800][400/402] eta 0:00:01 lr 0.000003 time 0.8771 (0.8902) loss 0.5774 (0.5747) grad_norm 0.2446 (0.2513) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:07:43 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 782 training takes 0:05:58 [2024-03-08 09:07:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [783/800][0/402] eta 0:33:07 lr 0.000003 time 4.9428 (4.9428) loss 0.5576 (0.5576) grad_norm 0.2340 (0.2340) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:09:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [783/800][100/402] eta 0:04:37 lr 0.000003 time 0.8807 (0.9191) loss 0.5673 (0.5758) grad_norm 0.2317 (0.2544) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:10:44 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [783/800][200/402] eta 0:03:01 lr 0.000003 time 0.8778 (0.8990) loss 0.5892 (0.5752) grad_norm 0.2767 (0.2535) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:12:12 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [783/800][300/402] eta 0:01:31 lr 0.000003 time 0.8789 (0.8923) loss 0.6072 (0.5752) grad_norm 0.2452 (0.2528) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:13:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [783/800][400/402] eta 0:00:01 lr 0.000003 time 0.8797 (0.8889) loss 0.6027 (0.5755) grad_norm 0.2404 (0.2524) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:13:41 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 783 training takes 0:05:57 [2024-03-08 09:13:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [784/800][0/402] eta 0:34:48 lr 0.000003 time 5.1963 (5.1963) loss 0.6364 (0.6364) grad_norm 0.2396 (0.2396) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:15:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [784/800][100/402] eta 0:04:38 lr 0.000003 time 0.8788 (0.9217) loss 0.5420 (0.5737) grad_norm 0.2706 (0.2529) loss_scale 524288.0000 (306267.2475) mem 30609MB [2024-03-08 09:16:42 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [784/800][200/402] eta 0:03:01 lr 0.000003 time 0.8791 (0.9006) loss 0.5896 (0.5745) grad_norm 0.2417 (0.2504) loss_scale 524288.0000 (414735.2836) mem 30609MB [2024-03-08 09:18:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [784/800][300/402] eta 0:01:31 lr 0.000003 time 0.8804 (0.8934) loss 0.5853 (0.5733) grad_norm 0.2534 (0.2523) loss_scale 524288.0000 (451131.5349) mem 30609MB [2024-03-08 09:19:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [784/800][400/402] eta 0:00:01 lr 0.000003 time 0.8764 (0.8897) loss 0.5991 (0.5737) grad_norm 0.2105 (0.2522) loss_scale 524288.0000 (469375.0424) mem 30609MB [2024-03-08 09:19:38 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 784 training takes 0:05:57 [2024-03-08 09:19:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [785/800][0/402] eta 0:33:47 lr 0.000003 time 5.0437 (5.0437) loss 0.5902 (0.5902) grad_norm 0.2437 (0.2437) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 09:21:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [785/800][100/402] eta 0:04:37 lr 0.000003 time 0.8797 (0.9199) loss 0.5537 (0.5739) grad_norm 0.2717 (nan) loss_scale 262144.0000 (329626.6139) mem 30609MB [2024-03-08 09:22:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [785/800][200/402] eta 0:03:01 lr 0.000003 time 0.8789 (0.8995) loss 0.5573 (0.5747) grad_norm 0.2431 (nan) loss_scale 262144.0000 (296053.1741) mem 30609MB [2024-03-08 09:24:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [785/800][300/402] eta 0:01:31 lr 0.000003 time 0.8790 (0.8925) loss 0.5726 (0.5747) grad_norm 0.2760 (nan) loss_scale 262144.0000 (284787.6678) mem 30609MB [2024-03-08 09:25:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [785/800][400/402] eta 0:00:01 lr 0.000003 time 0.8766 (0.8891) loss 0.5873 (0.5754) grad_norm 0.2392 (nan) loss_scale 262144.0000 (279140.8678) mem 30609MB [2024-03-08 09:25:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 785 training takes 0:05:57 [2024-03-08 09:25:36 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_785.pth saving...... [2024-03-08 09:25:38 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_785.pth saved !!! [2024-03-08 09:25:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [786/800][0/402] eta 0:33:18 lr 0.000003 time 4.9706 (4.9706) loss 0.5880 (0.5880) grad_norm 0.2439 (0.2439) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:27:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [786/800][100/402] eta 0:04:37 lr 0.000003 time 0.8785 (0.9197) loss 0.5682 (0.5753) grad_norm 0.7973 (0.2567) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:28:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [786/800][200/402] eta 0:03:01 lr 0.000003 time 0.8782 (0.8994) loss 0.5629 (0.5745) grad_norm 0.2612 (0.2570) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:30:07 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [786/800][300/402] eta 0:01:31 lr 0.000003 time 0.8788 (0.8927) loss 0.5385 (0.5742) grad_norm 0.2441 (0.2550) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:31:35 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [786/800][400/402] eta 0:00:01 lr 0.000003 time 0.8765 (0.8891) loss 0.5639 (0.5743) grad_norm 0.2531 (0.2546) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:31:36 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 786 training takes 0:05:57 [2024-03-08 09:31:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [787/800][0/402] eta 0:32:44 lr 0.000003 time 4.8873 (4.8873) loss 0.5639 (0.5639) grad_norm 0.2653 (0.2653) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:33:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [787/800][100/402] eta 0:04:37 lr 0.000003 time 0.8788 (0.9187) loss 0.5743 (0.5748) grad_norm 0.2744 (0.2569) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:34:37 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [787/800][200/402] eta 0:03:01 lr 0.000003 time 0.8780 (0.8986) loss 0.6107 (0.5725) grad_norm 0.2086 (0.2554) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:36:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [787/800][300/402] eta 0:01:30 lr 0.000003 time 0.8804 (0.8921) loss 0.5615 (0.5736) grad_norm 0.2411 (0.2544) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:37:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [787/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8887) loss 0.6071 (0.5739) grad_norm 0.2458 (0.2548) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:37:33 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 787 training takes 0:05:57 [2024-03-08 09:37:39 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [788/800][0/402] eta 0:33:55 lr 0.000003 time 5.0641 (5.0641) loss 0.5792 (0.5792) grad_norm 0.2240 (0.2240) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:39:06 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [788/800][100/402] eta 0:04:37 lr 0.000003 time 0.8789 (0.9204) loss 0.5667 (0.5767) grad_norm 0.2822 (0.2542) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:40:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [788/800][200/402] eta 0:03:01 lr 0.000003 time 0.8782 (0.9002) loss 0.5727 (0.5752) grad_norm 0.2709 (0.2540) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:42:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [788/800][300/402] eta 0:01:31 lr 0.000003 time 0.8791 (0.8932) loss 0.5860 (0.5734) grad_norm 0.2576 (0.2550) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:43:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [788/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8895) loss 0.5695 (0.5733) grad_norm 0.2464 (0.2559) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:43:31 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 788 training takes 0:05:57 [2024-03-08 09:43:36 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [789/800][0/402] eta 0:33:11 lr 0.000003 time 4.9538 (4.9538) loss 0.5660 (0.5660) grad_norm 0.2577 (0.2577) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:45:04 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [789/800][100/402] eta 0:04:37 lr 0.000003 time 0.8790 (0.9193) loss 0.5567 (0.5749) grad_norm 0.2355 (0.2563) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:46:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [789/800][200/402] eta 0:03:01 lr 0.000003 time 0.8784 (0.8993) loss 0.5460 (0.5744) grad_norm 0.2265 (0.2533) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:48:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [789/800][300/402] eta 0:01:31 lr 0.000003 time 0.8802 (0.8925) loss 0.5697 (0.5745) grad_norm 0.2390 (0.2534) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:49:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [789/800][400/402] eta 0:00:01 lr 0.000003 time 0.8771 (0.8890) loss 0.6113 (0.5743) grad_norm 0.2675 (0.2529) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:49:29 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 789 training takes 0:05:57 [2024-03-08 09:49:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [790/800][0/402] eta 0:34:45 lr 0.000003 time 5.1876 (5.1876) loss 0.5596 (0.5596) grad_norm 0.2346 (0.2346) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 09:51:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [790/800][100/402] eta 0:04:38 lr 0.000003 time 0.8805 (0.9215) loss 0.5523 (0.5726) grad_norm 0.2534 (0.2521) loss_scale 524288.0000 (482760.2376) mem 30609MB [2024-03-08 09:52:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [790/800][200/402] eta 0:03:01 lr 0.000003 time 0.8786 (0.9003) loss 0.5863 (0.5748) grad_norm 0.2751 (0.2550) loss_scale 524288.0000 (503420.8159) mem 30609MB [2024-03-08 09:53:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [790/800][300/402] eta 0:01:31 lr 0.000003 time 0.8783 (0.8931) loss 0.5995 (0.5730) grad_norm 0.2346 (0.2558) loss_scale 524288.0000 (510353.4352) mem 30609MB [2024-03-08 09:55:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [790/800][400/402] eta 0:00:01 lr 0.000003 time 0.8776 (0.8898) loss 0.5864 (0.5739) grad_norm 0.2417 (0.2544) loss_scale 524288.0000 (513828.3890) mem 30609MB [2024-03-08 09:55:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 790 training takes 0:05:57 [2024-03-08 09:55:27 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_790.pth saving...... [2024-03-08 09:55:29 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_790.pth saved !!! [2024-03-08 09:55:34 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [791/800][0/402] eta 0:35:23 lr 0.000003 time 5.2824 (5.2824) loss 0.6219 (0.6219) grad_norm 0.2265 (0.2265) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 09:57:02 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [791/800][100/402] eta 0:04:38 lr 0.000003 time 0.8791 (0.9223) loss 0.5764 (0.5747) grad_norm 0.2355 (0.2518) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 09:58:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [791/800][200/402] eta 0:03:01 lr 0.000003 time 0.8781 (0.9009) loss 0.5725 (0.5737) grad_norm 0.3523 (0.2512) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 09:59:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [791/800][300/402] eta 0:01:31 lr 0.000003 time 0.8796 (0.8936) loss 0.6023 (0.5740) grad_norm 0.2390 (0.2521) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:01:26 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [791/800][400/402] eta 0:00:01 lr 0.000003 time 0.8774 (0.8899) loss 0.5789 (0.5744) grad_norm 0.2708 (0.2533) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:01:27 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 791 training takes 0:05:58 [2024-03-08 10:01:32 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [792/800][0/402] eta 0:33:27 lr 0.000003 time 4.9936 (4.9936) loss 0.6083 (0.6083) grad_norm 0.2301 (0.2301) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:03:00 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [792/800][100/402] eta 0:04:37 lr 0.000003 time 0.8790 (0.9197) loss 0.5814 (0.5778) grad_norm 0.2488 (0.2528) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:04:28 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [792/800][200/402] eta 0:03:01 lr 0.000003 time 0.8805 (0.8993) loss 0.5878 (0.5766) grad_norm 0.2410 (0.2524) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:05:56 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [792/800][300/402] eta 0:01:31 lr 0.000003 time 0.8789 (0.8926) loss 0.5829 (0.5760) grad_norm 0.3060 (0.2546) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:07:24 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [792/800][400/402] eta 0:00:01 lr 0.000003 time 0.8769 (0.8892) loss 0.6180 (0.5749) grad_norm 0.2380 (0.2553) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:07:25 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 792 training takes 0:05:57 [2024-03-08 10:07:30 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [793/800][0/402] eta 0:32:23 lr 0.000003 time 4.8357 (4.8357) loss 0.5900 (0.5900) grad_norm 0.2424 (0.2424) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:08:58 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [793/800][100/402] eta 0:04:37 lr 0.000003 time 0.8785 (0.9180) loss 0.5969 (0.5757) grad_norm 0.2607 (0.2569) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:10:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [793/800][200/402] eta 0:03:01 lr 0.000003 time 0.8774 (0.8985) loss 0.5709 (0.5759) grad_norm 0.2645 (0.2547) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:11:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [793/800][300/402] eta 0:01:30 lr 0.000003 time 0.8787 (0.8920) loss 0.5379 (0.5746) grad_norm 0.2722 (nan) loss_scale 262144.0000 (492935.2292) mem 30609MB [2024-03-08 10:13:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [793/800][400/402] eta 0:00:01 lr 0.000003 time 0.8768 (0.8887) loss 0.5968 (0.5744) grad_norm 0.2389 (nan) loss_scale 262144.0000 (435381.3067) mem 30609MB [2024-03-08 10:13:22 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 793 training takes 0:05:57 [2024-03-08 10:13:27 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [794/800][0/402] eta 0:33:43 lr 0.000003 time 5.0331 (5.0331) loss 0.5886 (0.5886) grad_norm 0.2369 (0.2369) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:14:55 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [794/800][100/402] eta 0:04:37 lr 0.000003 time 0.8784 (0.9198) loss 0.5456 (0.5729) grad_norm 0.2601 (0.2609) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:16:23 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [794/800][200/402] eta 0:03:01 lr 0.000003 time 0.8788 (0.8995) loss 0.5716 (0.5735) grad_norm 0.2832 (0.2585) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:17:51 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [794/800][300/402] eta 0:01:31 lr 0.000003 time 0.8776 (0.8926) loss 0.5911 (0.5743) grad_norm 0.2818 (0.2579) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:19:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [794/800][400/402] eta 0:00:01 lr 0.000003 time 0.8774 (0.8891) loss 0.5754 (0.5742) grad_norm 0.2467 (0.2575) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:19:20 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 794 training takes 0:05:57 [2024-03-08 10:19:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [795/800][0/402] eta 0:31:10 lr 0.000003 time 4.6521 (4.6521) loss 0.5890 (0.5890) grad_norm 0.2590 (0.2590) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:20:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [795/800][100/402] eta 0:04:36 lr 0.000003 time 0.8785 (0.9172) loss 0.5473 (0.5766) grad_norm 0.2541 (0.2569) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:22:21 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [795/800][200/402] eta 0:03:01 lr 0.000003 time 0.8786 (0.8981) loss 0.5470 (0.5756) grad_norm 0.2586 (0.2571) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:23:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [795/800][300/402] eta 0:01:30 lr 0.000003 time 0.8790 (0.8918) loss 0.5596 (0.5756) grad_norm 0.2923 (0.2569) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:25:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [795/800][400/402] eta 0:00:01 lr 0.000003 time 0.8778 (0.8887) loss 0.5719 (0.5750) grad_norm 0.2432 (0.2569) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:25:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 795 training takes 0:05:57 [2024-03-08 10:25:18 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_795.pth saving...... [2024-03-08 10:25:20 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_795.pth saved !!! [2024-03-08 10:25:25 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [796/800][0/402] eta 0:33:38 lr 0.000003 time 5.0216 (5.0216) loss 0.5774 (0.5774) grad_norm 0.2568 (0.2568) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:26:53 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [796/800][100/402] eta 0:04:37 lr 0.000003 time 0.8793 (0.9199) loss 0.5669 (0.5735) grad_norm 0.2280 (0.2546) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:28:20 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [796/800][200/402] eta 0:03:01 lr 0.000003 time 0.8796 (0.8995) loss 0.5814 (0.5728) grad_norm 0.2660 (0.2560) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:29:48 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [796/800][300/402] eta 0:01:31 lr 0.000003 time 0.8786 (0.8927) loss 0.5795 (0.5733) grad_norm 0.2696 (0.2563) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:31:16 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [796/800][400/402] eta 0:00:01 lr 0.000003 time 0.8772 (0.8892) loss 0.5823 (0.5741) grad_norm 0.2932 (0.2565) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:31:17 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 796 training takes 0:05:57 [2024-03-08 10:31:22 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [797/800][0/402] eta 0:31:55 lr 0.000003 time 4.7657 (4.7657) loss 0.5564 (0.5564) grad_norm 0.2686 (0.2686) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:32:50 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [797/800][100/402] eta 0:04:37 lr 0.000003 time 0.8803 (0.9173) loss 0.5784 (0.5736) grad_norm 0.2554 (0.2584) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:34:18 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [797/800][200/402] eta 0:03:01 lr 0.000003 time 0.8787 (0.8982) loss 0.5585 (0.5739) grad_norm 0.2831 (0.2563) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:35:46 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [797/800][300/402] eta 0:01:30 lr 0.000003 time 0.8781 (0.8918) loss 0.5838 (0.5744) grad_norm 0.2599 (0.2561) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:37:14 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [797/800][400/402] eta 0:00:01 lr 0.000003 time 0.8777 (0.8885) loss 0.5804 (0.5738) grad_norm 0.2665 (0.2564) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:37:15 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 797 training takes 0:05:57 [2024-03-08 10:37:19 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [798/800][0/402] eta 0:31:09 lr 0.000003 time 4.6516 (4.6516) loss 0.6135 (0.6135) grad_norm 0.2387 (0.2387) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:38:47 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [798/800][100/402] eta 0:04:36 lr 0.000003 time 0.8784 (0.9166) loss 0.5903 (0.5756) grad_norm 0.2404 (0.2581) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:40:15 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [798/800][200/402] eta 0:03:01 lr 0.000003 time 0.8789 (0.8979) loss 0.6007 (0.5725) grad_norm 0.2792 (0.2586) loss_scale 262144.0000 (262144.0000) mem 30609MB [2024-03-08 10:41:43 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [798/800][300/402] eta 0:01:30 lr 0.000003 time 0.8791 (0.8916) loss 0.5593 (0.5727) grad_norm 0.2313 (0.2577) loss_scale 524288.0000 (302205.8738) mem 30609MB [2024-03-08 10:43:11 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [798/800][400/402] eta 0:00:01 lr 0.000003 time 0.8770 (0.8883) loss 0.5655 (0.5732) grad_norm 0.2613 (0.2559) loss_scale 524288.0000 (357587.9501) mem 30609MB [2024-03-08 10:43:12 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 798 training takes 0:05:57 [2024-03-08 10:43:17 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [799/800][0/402] eta 0:32:49 lr 0.000003 time 4.8984 (4.8984) loss 0.5641 (0.5641) grad_norm 0.2887 (0.2887) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:44:45 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [799/800][100/402] eta 0:04:37 lr 0.000003 time 0.8795 (0.9185) loss 0.5702 (0.5734) grad_norm 0.2271 (0.2545) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:46:13 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [799/800][200/402] eta 0:03:01 lr 0.000003 time 0.8788 (0.8987) loss 0.5756 (0.5740) grad_norm 0.2219 (0.2549) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:47:41 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [799/800][300/402] eta 0:01:31 lr 0.000003 time 0.8790 (0.8924) loss 0.5772 (0.5747) grad_norm 0.2370 (0.2554) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:49:09 hydro_simmim_pretrain] (main_simmim_pt.py 176): INFO Train: [799/800][400/402] eta 0:00:01 lr 0.000003 time 0.8786 (0.8890) loss 0.5857 (0.5740) grad_norm 0.2431 (0.2557) loss_scale 524288.0000 (524288.0000) mem 30609MB [2024-03-08 10:49:10 hydro_simmim_pretrain] (main_simmim_pt.py 185): INFO EPOCH 799 training takes 0:05:57 [2024-03-08 10:49:10 hydro_simmim_pretrain] (utils_simmim.py 63): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_799.pth saving...... [2024-03-08 10:49:11 hydro_simmim_pretrain] (utils_simmim.py 65): INFO output/hydro_simmim_pretrain/hydro_simmim_pretrain_swinv2_base_img256_window16_800ep/ckpt_epoch_799.pth saved !!! [2024-03-08 10:49:11 hydro_simmim_pretrain] (main_simmim_pt.py 117): INFO Training time 3 days, 7:26:40