HCQ_MSRVTT_full_M16.txt

Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 735.8524401187897 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 38.14291334152222 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 240.30010342597961 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 152.73178005218506 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch0.pth ...
Done in 8.854s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch0.pth ...
Done in 10.458s
 epoch          : 0
 loss           : 0
 learning_rate  : 5e-05
 n_samples      : 0
 n_steps        : 0
 MSRVTT_full_val/t2v_metrics/R1: 0.0
 MSRVTT_full_val/t2v_metrics/R5: 1.0060362173038229
 MSRVTT_full_val/t2v_metrics/R10: 1.8108651911468814
 MSRVTT_full_val/t2v_metrics/R50: 8.853118712273641
 MSRVTT_full_val/t2v_metrics/MedR: 260.0
 MSRVTT_full_val/t2v_metrics/MeanR: 255.90342052313883
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
 MSRVTT_full_val/v2t_metrics/R1: 0.4024144869215292
 MSRVTT_full_val/v2t_metrics/R5: 1.2072434607645874
 MSRVTT_full_val/v2t_metrics/R10: 2.2132796780684103
 MSRVTT_full_val/v2t_metrics/R50: 11.066398390342052
 MSRVTT_full_val/v2t_metrics/MedR: 261.0
 MSRVTT_full_val/v2t_metrics/MeanR: 253.1871227364185
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 1.0244755270944645
 MSRVTT_full_test/t2v_metrics/R1: 0.06688963210702341
 MSRVTT_full_test/t2v_metrics/R5: 0.16722408026755853
 MSRVTT_full_test/t2v_metrics/R10: 0.4013377926421405
 MSRVTT_full_test/t2v_metrics/R50: 1.7725752508361203
 MSRVTT_full_test/t2v_metrics/MedR: 1505.0
 MSRVTT_full_test/t2v_metrics/MeanR: 1494.771906354515
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.16496401834986424
 MSRVTT_full_test/v2t_metrics/R1: 0.0
 MSRVTT_full_test/v2t_metrics/R5: 0.20066889632107024
 MSRVTT_full_test/v2t_metrics/R10: 0.43478260869565216
 MSRVTT_full_test/v2t_metrics/R50: 1.6387959866220736
 MSRVTT_full_test/v2t_metrics/MedR: 1515.5
 MSRVTT_full_test/v2t_metrics/MeanR: 1503.9270903010033
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
 mnt_best       : 0.16496401834986424
 not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.84016 (QuantReg: 16.73999) QuantErr: 16.73999 batch_time=30.56429 
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.69228 (QuantReg: 16.76402) QuantErr: 16.76402 batch_time=0.47377 
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.05664 (QuantReg: 16.69961) QuantErr: 16.69961 batch_time=0.48518 
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.86357 (QuantReg: 16.66694) QuantErr: 16.66694 batch_time=0.44848 
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.41189 (QuantReg: 16.71029) QuantErr: 16.71029 batch_time=0.45538 
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.04232 (QuantReg: 16.70889) QuantErr: 16.70889 batch_time=0.46087 
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.10253 (QuantReg: 16.70435) QuantErr: 16.70435 batch_time=0.58004 
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.79978 (QuantReg: 16.72273) QuantErr: 16.72273 batch_time=0.45761 
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.27905 (QuantReg: 16.70428) QuantErr: 16.70428 batch_time=0.50152 
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.46560 (QuantReg: 16.71296) QuantErr: 16.71296 batch_time=0.48241 
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.03153 (QuantReg: 16.68560) QuantErr: 16.68560 batch_time=0.46171 
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.78815 (QuantReg: 16.68539) QuantErr: 16.68539 batch_time=0.57397 
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.91429 (QuantReg: 16.70953) QuantErr: 16.70953 batch_time=0.44974 
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.63115 (QuantReg: 16.71470) QuantErr: 16.71470 batch_time=0.46273 
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.77717 (QuantReg: 16.72078) QuantErr: 16.72078 batch_time=3.05546 
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.99268 (QuantReg: 16.70894) QuantErr: 16.70894 batch_time=0.44207 
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.72166 (QuantReg: 16.69893) QuantErr: 16.69893 batch_time=0.46582 
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.35827 (QuantReg: 16.70883) QuantErr: 16.70883 batch_time=0.43931 
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.44274 (QuantReg: 16.71587) QuantErr: 16.71587 batch_time=0.44363 
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.08258 (QuantReg: 16.71847) QuantErr: 16.71847 batch_time=0.48828 
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.43849 (QuantReg: 16.70953) QuantErr: 16.70953 batch_time=0.46083 
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.15347 (QuantReg: 16.69563) QuantErr: 16.69563 batch_time=0.45648 
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.94227 (QuantReg: 16.70367) QuantErr: 16.70367 batch_time=0.44743 
Train Epoch: 1 codebook_update_time=1.03458
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch1.pth ...
Done in 4.683s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch1.pth ...
Done in 9.176s
 epoch          : 1
 loss           : 5.31894676399231
 quant_reg      : 16.707304183959963
 quant_err      : 16.707304183959963
 learning_rate  : 5e-05
 n_samples      : 32000
 n_steps        : 250
 MSRVTT_full_val/t2v_metrics/R1: 18.309859154929576
 MSRVTT_full_val/t2v_metrics/R5: 49.094567404426556
 MSRVTT_full_val/t2v_metrics/R10: 66.19718309859155
 MSRVTT_full_val/t2v_metrics/R50: 95.37223340040241
 MSRVTT_full_val/t2v_metrics/MedR: 6.0
 MSRVTT_full_val/t2v_metrics/MeanR: 14.050301810865191
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 39.040854694053984
 MSRVTT_full_val/v2t_metrics/R1: 18.91348088531187
 MSRVTT_full_val/v2t_metrics/R5: 51.50905432595574
 MSRVTT_full_val/v2t_metrics/R10: 68.41046277665995
 MSRVTT_full_val/v2t_metrics/R50: 93.76257545271629
 MSRVTT_full_val/v2t_metrics/MedR: 5.0
 MSRVTT_full_val/v2t_metrics/MeanR: 14.140845070422536
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 40.54393123227139
 MSRVTT_full_test/t2v_metrics/R1: 5.48494983277592
 MSRVTT_full_test/t2v_metrics/R5: 18.662207357859533
 MSRVTT_full_test/t2v_metrics/R10: 28.762541806020067
 MSRVTT_full_test/t2v_metrics/R50: 64.41471571906355
 MSRVTT_full_test/t2v_metrics/MedR: 28.0
 MSRVTT_full_test/t2v_metrics/MeanR: 76.99531772575251
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 14.332467899353855
 MSRVTT_full_test/v2t_metrics/R1: 5.68561872909699
 MSRVTT_full_test/v2t_metrics/R5: 20.93645484949833
 MSRVTT_full_test/v2t_metrics/R10: 31.97324414715719
 MSRVTT_full_test/v2t_metrics/R50: 67.52508361204013
 MSRVTT_full_test/v2t_metrics/MedR: 24.0
 MSRVTT_full_test/v2t_metrics/MeanR: 73.33779264214047
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 15.613101892507107
 mnt_best       : 14.332467899353855
 not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 3.59599 (QuantReg: 7.30109) QuantErr: 7.30109 batch_time=39.88664 
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.91338 (QuantReg: 7.46262) QuantErr: 7.46262 batch_time=2.24806 
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 3.84971 (QuantReg: 7.41927) QuantErr: 7.41927 batch_time=0.48409 
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.67504 (QuantReg: 7.52744) QuantErr: 7.52744 batch_time=0.44481 
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.65810 (QuantReg: 7.70300) QuantErr: 7.70300 batch_time=0.44407 
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.79013 (QuantReg: 7.59337) QuantErr: 7.59337 batch_time=0.44897 
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.92198 (QuantReg: 8.02096) QuantErr: 8.02096 batch_time=0.47103 
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.49982 (QuantReg: 8.10039) QuantErr: 8.10039 batch_time=0.46901 
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.06219 (QuantReg: 8.17074) QuantErr: 8.17074 batch_time=0.65653 
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.46992 (QuantReg: 8.16039) QuantErr: 8.16039 batch_time=0.45975 
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.56887 (QuantReg: 8.20473) QuantErr: 8.20473 batch_time=0.46294 
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.31151 (QuantReg: 8.52287) QuantErr: 8.52287 batch_time=0.49062 
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.22814 (QuantReg: 8.33879) QuantErr: 8.33879 batch_time=0.47888 
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.40891 (QuantReg: 8.51748) QuantErr: 8.51748 batch_time=0.44918 
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.27765 (QuantReg: 8.50100) QuantErr: 8.50100 batch_time=0.46915 
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.17155 (QuantReg: 8.99632) QuantErr: 8.99632 batch_time=0.44911 
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.46749 (QuantReg: 8.83380) QuantErr: 8.83380 batch_time=0.44359 
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.49086 (QuantReg: 8.95881) QuantErr: 8.95881 batch_time=0.47855 
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.15729 (QuantReg: 8.66644) QuantErr: 8.66644 batch_time=0.49070 
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.02822 (QuantReg: 8.93359) QuantErr: 8.93359 batch_time=0.44483 
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 2.79454 (QuantReg: 9.04696) QuantErr: 9.04696 batch_time=1.55515 
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.86574 (QuantReg: 9.01290) QuantErr: 9.01290 batch_time=0.45120 
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.46722 (QuantReg: 9.27474) QuantErr: 9.27474 batch_time=0.45251 
Train Epoch: 2 codebook_update_time=0.95445
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch2.pth ...
Done in 21.574s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch2.pth ...
Done in 25.501s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.00s]
 epoch          : 2
 loss           : 3.5179560661315916
 quant_reg      : 8.325530506134033
 quant_err      : 8.325530506134033
 learning_rate  : 4.75e-05
 n_samples      : 64000
 n_steps        : 500
 MSRVTT_full_val/t2v_metrics/R1: 21.327967806841045
 MSRVTT_full_val/t2v_metrics/R5: 57.74647887323944
 MSRVTT_full_val/t2v_metrics/R10: 72.63581488933602
 MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 11.352112676056338
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 44.72413323334615
 MSRVTT_full_val/v2t_metrics/R1: 24.346076458752513
 MSRVTT_full_val/v2t_metrics/R5: 59.758551307847085
 MSRVTT_full_val/v2t_metrics/R10: 76.86116700201207
 MSRVTT_full_val/v2t_metrics/R50: 96.579476861167
 MSRVTT_full_val/v2t_metrics/MedR: 4.0
 MSRVTT_full_val/v2t_metrics/MeanR: 11.112676056338028
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.17761966359318
 MSRVTT_full_test/t2v_metrics/R1: 7.157190635451505
 MSRVTT_full_test/t2v_metrics/R5: 23.745819397993312
 MSRVTT_full_test/t2v_metrics/R10: 35.4180602006689
 MSRVTT_full_test/t2v_metrics/R50: 71.30434782608695
 MSRVTT_full_test/t2v_metrics/MedR: 19.0
 MSRVTT_full_test/t2v_metrics/MeanR: 61.86622073578595
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.190787716079434
 MSRVTT_full_test/v2t_metrics/R1: 8.02675585284281
 MSRVTT_full_test/v2t_metrics/R5: 26.220735785953178
 MSRVTT_full_test/v2t_metrics/R10: 38.59531772575251
 MSRVTT_full_test/v2t_metrics/R50: 74.78260869565217
 MSRVTT_full_test/v2t_metrics/MedR: 17.0
 MSRVTT_full_test/v2t_metrics/MeanR: 54.60200668896321
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.102026881503765
 mnt_best       : 18.190787716079434
 not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 2.98773 (QuantReg: 7.42341) QuantErr: 7.42341 batch_time=30.21973 
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.18499 (QuantReg: 7.56751) QuantErr: 7.56751 batch_time=0.58662 
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.50044 (QuantReg: 7.86508) QuantErr: 7.86508 batch_time=0.47744 
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.62197 (QuantReg: 7.28161) QuantErr: 7.28161 batch_time=0.43181 
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.23049 (QuantReg: 7.56710) QuantErr: 7.56710 batch_time=0.44196 
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.63186 (QuantReg: 7.78139) QuantErr: 7.78139 batch_time=0.44114 
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.85709 (QuantReg: 7.71804) QuantErr: 7.71804 batch_time=0.90366 
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.02143 (QuantReg: 7.65518) QuantErr: 7.65518 batch_time=0.69424 
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.07048 (QuantReg: 7.73555) QuantErr: 7.73555 batch_time=0.47174 
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.82475 (QuantReg: 7.62398) QuantErr: 7.62398 batch_time=0.47576 
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.99884 (QuantReg: 7.61039) QuantErr: 7.61039 batch_time=0.46138 
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 2.54372 (QuantReg: 7.80277) QuantErr: 7.80277 batch_time=0.45417 
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.10043 (QuantReg: 8.02083) QuantErr: 8.02083 batch_time=0.49673 
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 2.76505 (QuantReg: 8.04368) QuantErr: 8.04368 batch_time=0.46630 
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.57136 (QuantReg: 7.78552) QuantErr: 7.78552 batch_time=0.44392 
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.27337 (QuantReg: 8.45651) QuantErr: 8.45651 batch_time=0.44440 
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.55169 (QuantReg: 8.11333) QuantErr: 8.11333 batch_time=0.53954 
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.13125 (QuantReg: 7.94663) QuantErr: 7.94663 batch_time=0.44720 
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.67915 (QuantReg: 8.01175) QuantErr: 8.01175 batch_time=0.49306 
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.78901 (QuantReg: 8.26646) QuantErr: 8.26646 batch_time=0.45635 
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 2.60713 (QuantReg: 8.26630) QuantErr: 8.26630 batch_time=1.05931 
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.35910 (QuantReg: 8.48475) QuantErr: 8.48475 batch_time=0.44838 
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.69812 (QuantReg: 8.31561) QuantErr: 8.31561 batch_time=0.45018 
Train Epoch: 3 codebook_update_time=0.88930
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch3.pth ...
Done in 5.316s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch3.pth ...
Done in 10.659s
removing stale ckpt [epoch 2] [took 0.49s]
 epoch          : 3
 loss           : 2.909803550720215
 quant_reg      : 7.928305335998536
 quant_err      : 7.928305335998536
 learning_rate  : 4.5125e-05
 n_samples      : 96000
 n_steps        : 750
 MSRVTT_full_val/t2v_metrics/R1: 22.736418511066397
 MSRVTT_full_val/t2v_metrics/R5: 60.76458752515091
 MSRVTT_full_val/t2v_metrics/R10: 74.04426559356136
 MSRVTT_full_val/t2v_metrics/R50: 96.17706237424547
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 10.501006036217303
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.76863338056698
 MSRVTT_full_val/v2t_metrics/R1: 23.541247484909455
 MSRVTT_full_val/v2t_metrics/R5: 61.77062374245473
 MSRVTT_full_val/v2t_metrics/R10: 78.26961770623743
 MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 9.733400402414487
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.46202501630321
 MSRVTT_full_test/t2v_metrics/R1: 7.926421404682274
 MSRVTT_full_test/t2v_metrics/R5: 24.94983277591973
 MSRVTT_full_test/t2v_metrics/R10: 38.929765886287626
 MSRVTT_full_test/t2v_metrics/R50: 73.61204013377926
 MSRVTT_full_test/t2v_metrics/MedR: 17.0
 MSRVTT_full_test/t2v_metrics/MeanR: 56.95936454849498
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.74583618044286
 MSRVTT_full_test/v2t_metrics/R1: 8.26086956521739
 MSRVTT_full_test/v2t_metrics/R5: 26.120401337792643
 MSRVTT_full_test/v2t_metrics/R10: 39.096989966555185
 MSRVTT_full_test/v2t_metrics/R50: 76.38795986622074
 MSRVTT_full_test/v2t_metrics/MedR: 15.0
 MSRVTT_full_test/v2t_metrics/MeanR: 53.79214046822742
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.35711880037244
 mnt_best       : 19.74583618044286
 not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.19706 (QuantReg: 7.35212) QuantErr: 7.35212 batch_time=32.67095 
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.72323 (QuantReg: 7.50227) QuantErr: 7.50227 batch_time=0.43636 
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.16819 (QuantReg: 7.61940) QuantErr: 7.61940 batch_time=0.45365 
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.42007 (QuantReg: 7.60854) QuantErr: 7.60854 batch_time=0.51583 
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.70793 (QuantReg: 7.51873) QuantErr: 7.51873 batch_time=0.45279 
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 2.97458 (QuantReg: 7.56108) QuantErr: 7.56108 batch_time=0.46951 
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 2.99625 (QuantReg: 7.61918) QuantErr: 7.61918 batch_time=0.50493 
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.75820 (QuantReg: 7.84297) QuantErr: 7.84297 batch_time=0.49085 
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.57696 (QuantReg: 7.64832) QuantErr: 7.64832 batch_time=0.44487 
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.44666 (QuantReg: 7.69211) QuantErr: 7.69211 batch_time=0.46579 
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.71818 (QuantReg: 7.90377) QuantErr: 7.90377 batch_time=0.45587 
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.64605 (QuantReg: 7.81533) QuantErr: 7.81533 batch_time=0.45321 
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.47454 (QuantReg: 7.78972) QuantErr: 7.78972 batch_time=3.29962 
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.66189 (QuantReg: 7.97533) QuantErr: 7.97533 batch_time=0.44832 
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.81895 (QuantReg: 8.07495) QuantErr: 8.07495 batch_time=0.48306 
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.91051 (QuantReg: 8.30396) QuantErr: 8.30396 batch_time=0.50846 
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.96161 (QuantReg: 7.95596) QuantErr: 7.95596 batch_time=0.48728 
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.69241 (QuantReg: 8.28374) QuantErr: 8.28374 batch_time=0.48621 
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.73560 (QuantReg: 7.98351) QuantErr: 7.98351 batch_time=0.49806 
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.37398 (QuantReg: 8.09957) QuantErr: 8.09957 batch_time=0.92931 
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.37267 (QuantReg: 8.39126) QuantErr: 8.39126 batch_time=0.45946 
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.69829 (QuantReg: 7.89078) QuantErr: 7.89078 batch_time=0.45829 
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.46215 (QuantReg: 8.01395) QuantErr: 8.01395 batch_time=0.49857 
Train Epoch: 4 codebook_update_time=0.91529
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch4.pth ...
Done in 5.459s
removing stale ckpt [epoch 3] [took 0.01s]
 epoch          : 4
 loss           : 2.6015873069763185
 quant_reg      : 7.880937025070191
 quant_err      : 7.880937025070191
 learning_rate  : 4.2868749999999995e-05
 n_samples      : 128000
 n_steps        : 1000
 MSRVTT_full_val/t2v_metrics/R1: 22.736418511066397
 MSRVTT_full_val/t2v_metrics/R5: 57.947686116700204
 MSRVTT_full_val/t2v_metrics/R10: 71.83098591549296
 MSRVTT_full_val/t2v_metrics/R50: 95.97585513078471
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 11.227364185110664
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 45.571150573099196
 MSRVTT_full_val/v2t_metrics/R1: 23.138832997987926
 MSRVTT_full_val/v2t_metrics/R5: 63.17907444668008
 MSRVTT_full_val/v2t_metrics/R10: 77.8672032193159
 MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 10.095573440643863
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 48.46442849022805
 MSRVTT_full_test/t2v_metrics/R1: 7.926421404682274
 MSRVTT_full_test/t2v_metrics/R5: 25.51839464882943
 MSRVTT_full_test/t2v_metrics/R10: 37.05685618729097
 MSRVTT_full_test/t2v_metrics/R50: 72.17391304347827
 MSRVTT_full_test/t2v_metrics/MedR: 18.0
 MSRVTT_full_test/t2v_metrics/MeanR: 61.235284280936455
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.570399586380805
 MSRVTT_full_test/v2t_metrics/R1: 8.294314381270903
 MSRVTT_full_test/v2t_metrics/R5: 26.421404682274247
 MSRVTT_full_test/v2t_metrics/R10: 40.23411371237458
 MSRVTT_full_test/v2t_metrics/R50: 77.19063545150502
 MSRVTT_full_test/v2t_metrics/MedR: 15.0
 MSRVTT_full_test/v2t_metrics/MeanR: 51.869565217391305
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.659046737475528
 mnt_best       : 19.74583618044286
 not_improved_count: 1
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.59429 (QuantReg: 7.77318) QuantErr: 7.77318 batch_time=45.74012 
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.69768 (QuantReg: 7.80708) QuantErr: 7.80708 batch_time=0.52217 
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.78110 (QuantReg: 7.69037) QuantErr: 7.69037 batch_time=0.46112 
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.14629 (QuantReg: 7.90836) QuantErr: 7.90836 batch_time=0.50327 
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.26641 (QuantReg: 7.59773) QuantErr: 7.59773 batch_time=0.53496 
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.55007 (QuantReg: 8.32857) QuantErr: 8.32857 batch_time=0.49464 
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.36226 (QuantReg: 7.81751) QuantErr: 7.81751 batch_time=0.49734 
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.63137 (QuantReg: 7.79102) QuantErr: 7.79102 batch_time=0.50802 
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.12600 (QuantReg: 7.91347) QuantErr: 7.91347 batch_time=0.44691 
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.55870 (QuantReg: 7.95387) QuantErr: 7.95387 batch_time=0.51328 
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.27325 (QuantReg: 8.07391) QuantErr: 8.07391 batch_time=0.51770 
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.26087 (QuantReg: 8.09440) QuantErr: 8.09440 batch_time=0.45084 
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.46019 (QuantReg: 8.16368) QuantErr: 8.16368 batch_time=0.45748 
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 1.69798 (QuantReg: 8.14197) QuantErr: 8.14197 batch_time=0.45085 
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.27959 (QuantReg: 8.18234) QuantErr: 8.18234 batch_time=0.87570 
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.30978 (QuantReg: 8.14336) QuantErr: 8.14336 batch_time=0.51653 
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.14477 (QuantReg: 8.49021) QuantErr: 8.49021 batch_time=0.45215 
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.51935 (QuantReg: 7.99435) QuantErr: 7.99435 batch_time=0.47457 
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.12465 (QuantReg: 8.16798) QuantErr: 8.16798 batch_time=0.44747 
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.30027 (QuantReg: 7.95562) QuantErr: 7.95562 batch_time=0.47739 
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.27225 (QuantReg: 8.16034) QuantErr: 8.16034 batch_time=0.49339 
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 1.87790 (QuantReg: 8.24682) QuantErr: 8.24682 batch_time=0.51520 
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.19844 (QuantReg: 8.18047) QuantErr: 8.18047 batch_time=0.45833 
Train Epoch: 5 codebook_update_time=0.87966
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch5.pth ...
Done in 4.299s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch5.pth ...
Done in 9.264s
removing stale ckpt [epoch 4] [took 0.01s]
 epoch          : 5
 loss           : 2.312036591529846
 quant_reg      : 7.996879131317138
 quant_err      : 7.996879131317138
 learning_rate  : 4.072531249999999e-05
 n_samples      : 160000
 n_steps        : 1250
 MSRVTT_full_val/t2v_metrics/R1: 26.156941649899398
 MSRVTT_full_val/t2v_metrics/R5: 62.374245472837025
 MSRVTT_full_val/t2v_metrics/R10: 76.86116700201207
 MSRVTT_full_val/t2v_metrics/R50: 95.97585513078471
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 10.462776659959758
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.05334209909608
 MSRVTT_full_val/v2t_metrics/R1: 25.955734406438633
 MSRVTT_full_val/v2t_metrics/R5: 65.19114688128772
 MSRVTT_full_val/v2t_metrics/R10: 78.06841046277665
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 9.174044265593562
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 50.92907166935899
 MSRVTT_full_test/t2v_metrics/R1: 8.963210702341136
 MSRVTT_full_test/t2v_metrics/R5: 27.65886287625418
 MSRVTT_full_test/t2v_metrics/R10: 40.635451505016725
 MSRVTT_full_test/t2v_metrics/R50: 76.12040133779264
 MSRVTT_full_test/t2v_metrics/MedR: 15.0
 MSRVTT_full_test/t2v_metrics/MeanR: 52.84916387959866
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.597376810850314
 MSRVTT_full_test/v2t_metrics/R1: 9.431438127090301
 MSRVTT_full_test/v2t_metrics/R5: 29.130434782608695
 MSRVTT_full_test/v2t_metrics/R10: 44.381270903010034
 MSRVTT_full_test/v2t_metrics/R50: 79.36454849498328
 MSRVTT_full_test/v2t_metrics/MedR: 13.0
 MSRVTT_full_test/v2t_metrics/MeanR: 46.115551839464885
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.016619615824776
 mnt_best       : 21.597376810850314
 not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.40799 (QuantReg: 7.76801) QuantErr: 7.76801 batch_time=33.30566 
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.43668 (QuantReg: 7.53778) QuantErr: 7.53778 batch_time=0.44853 
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 1.92265 (QuantReg: 7.81433) QuantErr: 7.81433 batch_time=0.43515 
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.29354 (QuantReg: 7.75794) QuantErr: 7.75794 batch_time=0.42998 
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 1.68761 (QuantReg: 7.81035) QuantErr: 7.81035 batch_time=0.47132 
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.01432 (QuantReg: 8.14798) QuantErr: 8.14798 batch_time=0.50373 
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.10787 (QuantReg: 8.27241) QuantErr: 8.27241 batch_time=0.43220 
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 1.71009 (QuantReg: 7.93464) QuantErr: 7.93464 batch_time=0.43150 
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.26776 (QuantReg: 8.03418) QuantErr: 8.03418 batch_time=0.43128 
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.05410 (QuantReg: 8.06084) QuantErr: 8.06084 batch_time=0.48680 
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.15353 (QuantReg: 7.76849) QuantErr: 7.76849 batch_time=0.47791 
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.12229 (QuantReg: 8.17722) QuantErr: 8.17722 batch_time=0.45040 
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.11962 (QuantReg: 7.91460) QuantErr: 7.91460 batch_time=0.43997 
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.26068 (QuantReg: 7.83080) QuantErr: 7.83080 batch_time=4.22353 
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 1.78132 (QuantReg: 8.11680) QuantErr: 8.11680 batch_time=0.46097 
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 1.85049 (QuantReg: 8.04171) QuantErr: 8.04171 batch_time=0.42815 
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.25253 (QuantReg: 8.29697) QuantErr: 8.29697 batch_time=0.44659 
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 1.93146 (QuantReg: 8.44028) QuantErr: 8.44028 batch_time=0.47024 
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.14228 (QuantReg: 8.12896) QuantErr: 8.12896 batch_time=0.49277 
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 1.81141 (QuantReg: 8.08717) QuantErr: 8.08717 batch_time=2.67750 
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 1.90865 (QuantReg: 8.21623) QuantErr: 8.21623 batch_time=0.50273 
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.09249 (QuantReg: 8.11153) QuantErr: 8.11153 batch_time=0.49498 
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.09657 (QuantReg: 8.07226) QuantErr: 8.07226 batch_time=0.47479 
Train Epoch: 6 codebook_update_time=0.96991
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch6.pth ...
Done in 4.298s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch6.pth ...
Done in 9.037s
removing stale ckpt [epoch 5] [took 1.06s]
 epoch          : 6
 loss           : 2.133497720718384
 quant_reg      : 8.016869579315186
 quant_err      : 8.016869579315186
 learning_rate  : 3.868904687499999e-05
 n_samples      : 192000
 n_steps        : 1500
 MSRVTT_full_val/t2v_metrics/R1: 23.74245472837022
 MSRVTT_full_val/t2v_metrics/R5: 62.374245472837025
 MSRVTT_full_val/t2v_metrics/R10: 76.05633802816901
 MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.83702213279678
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.29351077169958
 MSRVTT_full_val/v2t_metrics/R1: 29.77867203219316
 MSRVTT_full_val/v2t_metrics/R5: 66.19718309859155
 MSRVTT_full_val/v2t_metrics/R10: 80.48289738430583
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 8.189134808853119
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.13558233374701
 MSRVTT_full_test/t2v_metrics/R1: 8.896321070234114
 MSRVTT_full_test/t2v_metrics/R5: 28.82943143812709
 MSRVTT_full_test/t2v_metrics/R10: 42.207357859531776
 MSRVTT_full_test/t2v_metrics/R50: 76.78929765886288
 MSRVTT_full_test/t2v_metrics/MedR: 15.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.45284280936455
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.121346790611877
 MSRVTT_full_test/v2t_metrics/R1: 8.862876254180602
 MSRVTT_full_test/v2t_metrics/R5: 30.03344481605351
 MSRVTT_full_test/v2t_metrics/R10: 44.080267558528426
 MSRVTT_full_test/v2t_metrics/R50: 80.03344481605352
 MSRVTT_full_test/v2t_metrics/MedR: 13.0
 MSRVTT_full_test/v2t_metrics/MeanR: 44.783277591973246
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.723471779762715
 mnt_best       : 22.121346790611877
 not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.18876 (QuantReg: 7.46789) QuantErr: 7.46789 batch_time=44.08198 
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 1.81282 (QuantReg: 8.17941) QuantErr: 8.17941 batch_time=0.49757 
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.24164 (QuantReg: 8.10194) QuantErr: 8.10194 batch_time=0.44167 
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.11386 (QuantReg: 8.17007) QuantErr: 8.17007 batch_time=0.47386 
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 1.93450 (QuantReg: 7.89564) QuantErr: 7.89564 batch_time=0.49979 
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.12224 (QuantReg: 7.94424) QuantErr: 7.94424 batch_time=0.46980 
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.06363 (QuantReg: 8.22287) QuantErr: 8.22287 batch_time=0.45608 
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.20511 (QuantReg: 8.05877) QuantErr: 8.05877 batch_time=0.52327 
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.22483 (QuantReg: 8.07802) QuantErr: 8.07802 batch_time=0.51710 
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.97522 (QuantReg: 8.04437) QuantErr: 8.04437 batch_time=0.46817 
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.29586 (QuantReg: 8.09972) QuantErr: 8.09972 batch_time=0.48629 
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 1.57235 (QuantReg: 8.32914) QuantErr: 8.32914 batch_time=0.60985 
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.77084 (QuantReg: 7.99232) QuantErr: 7.99232 batch_time=0.50919 
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.13260 (QuantReg: 8.00529) QuantErr: 8.00529 batch_time=0.50002 
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.04336 (QuantReg: 7.86586) QuantErr: 7.86586 batch_time=0.47593 
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 1.77714 (QuantReg: 8.13755) QuantErr: 8.13755 batch_time=0.46512 
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.67405 (QuantReg: 8.34757) QuantErr: 8.34757 batch_time=0.50241 
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.04017 (QuantReg: 8.07919) QuantErr: 8.07919 batch_time=0.47082 
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.48966 (QuantReg: 8.24991) QuantErr: 8.24991 batch_time=0.48077 
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.74120 (QuantReg: 8.25739) QuantErr: 8.25739 batch_time=0.48162 
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.41778 (QuantReg: 8.18477) QuantErr: 8.18477 batch_time=0.57665 
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 1.89425 (QuantReg: 8.08080) QuantErr: 8.08080 batch_time=0.51147 
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.11163 (QuantReg: 8.01719) QuantErr: 8.01719 batch_time=0.51989 
Train Epoch: 7 codebook_update_time=0.95321
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch7.pth ...
Done in 6.589s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch7.pth ...
Done in 11.448s
removing stale ckpt [epoch 6] [took 0.00s]
 epoch          : 7
 loss           : 1.976937479019165
 quant_reg      : 8.117966808319093
 quant_err      : 8.117966808319093
 learning_rate  : 3.675459453124999e-05
 n_samples      : 224000
 n_steps        : 1750
 MSRVTT_full_val/t2v_metrics/R1: 27.16297786720322
 MSRVTT_full_val/t2v_metrics/R5: 62.57545271629779
 MSRVTT_full_val/t2v_metrics/R10: 78.47082494969818
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.398390342052314
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 51.09321126728928
 MSRVTT_full_val/v2t_metrics/R1: 29.979879275653925
 MSRVTT_full_val/v2t_metrics/R5: 67.40442655935614
 MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.98692152917505
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 55.39187468475721
 MSRVTT_full_test/t2v_metrics/R1: 8.695652173913043
 MSRVTT_full_test/t2v_metrics/R5: 29.130434782608695
 MSRVTT_full_test/t2v_metrics/R10: 43.37792642140468
 MSRVTT_full_test/t2v_metrics/R50: 77.52508361204013
 MSRVTT_full_test/t2v_metrics/MedR: 14.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.38160535117057
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.231698192821643
 MSRVTT_full_test/v2t_metrics/R1: 9.531772575250836
 MSRVTT_full_test/v2t_metrics/R5: 31.438127090301002
 MSRVTT_full_test/v2t_metrics/R10: 45.618729096989966
 MSRVTT_full_test/v2t_metrics/R50: 80.40133779264214
 MSRVTT_full_test/v2t_metrics/MedR: 13.0
 MSRVTT_full_test/v2t_metrics/MeanR: 43.63561872909699
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.91063847195828
 mnt_best       : 22.231698192821643
 not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.88702 (QuantReg: 8.14470) QuantErr: 8.14470 batch_time=40.08937 
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.08971 (QuantReg: 8.20032) QuantErr: 8.20032 batch_time=3.12022 
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.52665 (QuantReg: 7.87798) QuantErr: 7.87798 batch_time=0.46869 
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 1.89420 (QuantReg: 8.29531) QuantErr: 8.29531 batch_time=0.43821 
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.01943 (QuantReg: 8.12687) QuantErr: 8.12687 batch_time=0.49690 
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.01499 (QuantReg: 8.03370) QuantErr: 8.03370 batch_time=0.42801 
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.73081 (QuantReg: 8.37523) QuantErr: 8.37523 batch_time=0.50019 
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 1.68981 (QuantReg: 8.41403) QuantErr: 8.41403 batch_time=1.68078 
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 1.99073 (QuantReg: 8.26980) QuantErr: 8.26980 batch_time=0.43335 
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.88347 (QuantReg: 8.09546) QuantErr: 8.09546 batch_time=0.43459 
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 1.73520 (QuantReg: 8.33964) QuantErr: 8.33964 batch_time=0.48333 
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.48633 (QuantReg: 8.27896) QuantErr: 8.27896 batch_time=0.45105 
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.74082 (QuantReg: 8.19931) QuantErr: 8.19931 batch_time=0.47158 
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.68332 (QuantReg: 8.22073) QuantErr: 8.22073 batch_time=0.45215 
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.22019 (QuantReg: 8.38120) QuantErr: 8.38120 batch_time=0.46288 
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.61268 (QuantReg: 8.15263) QuantErr: 8.15263 batch_time=0.45565 
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.14731 (QuantReg: 8.30403) QuantErr: 8.30403 batch_time=0.48789 
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.05999 (QuantReg: 8.44931) QuantErr: 8.44931 batch_time=0.44231 
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.50176 (QuantReg: 8.36141) QuantErr: 8.36141 batch_time=0.44910 
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.38081 (QuantReg: 8.18626) QuantErr: 8.18626 batch_time=0.43418 
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.52283 (QuantReg: 8.60197) QuantErr: 8.60197 batch_time=0.48328 
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.14452 (QuantReg: 8.16210) QuantErr: 8.16210 batch_time=0.43185 
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.50274 (QuantReg: 8.26057) QuantErr: 8.26057 batch_time=0.50373 
Train Epoch: 8 codebook_update_time=0.89796
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch8.pth ...
Done in 5.040s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch8.pth ...
Done in 9.422s
removing stale ckpt [epoch 7] [took 0.00s]
 epoch          : 8
 loss           : 1.8778138318061828
 quant_reg      : 8.191797365188599
 quant_err      : 8.191797365188599
 learning_rate  : 3.4916864804687486e-05
 n_samples      : 256000
 n_steps        : 2000
 MSRVTT_full_val/t2v_metrics/R1: 24.14486921529175
 MSRVTT_full_val/t2v_metrics/R5: 61.16700201207244
 MSRVTT_full_val/t2v_metrics/R10: 77.2635814889336
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.412474849094567
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.50341262241395
 MSRVTT_full_val/v2t_metrics/R1: 25.35211267605634
 MSRVTT_full_val/v2t_metrics/R5: 67.80684104627767
 MSRVTT_full_val/v2t_metrics/R10: 82.69617706237425
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.869215291750503
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 52.19044908833167
 MSRVTT_full_test/t2v_metrics/R1: 9.23076923076923
 MSRVTT_full_test/t2v_metrics/R5: 28.963210702341136
 MSRVTT_full_test/t2v_metrics/R10: 43.24414715719063
 MSRVTT_full_test/t2v_metrics/R50: 76.68896321070234
 MSRVTT_full_test/t2v_metrics/MedR: 14.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.59816053511706
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.61191328632993
 MSRVTT_full_test/v2t_metrics/R1: 9.59866220735786
 MSRVTT_full_test/v2t_metrics/R5: 30.76923076923077
 MSRVTT_full_test/v2t_metrics/R10: 44.8494983277592
 MSRVTT_full_test/v2t_metrics/R50: 80.93645484949833
 MSRVTT_full_test/v2t_metrics/MedR: 13.0
 MSRVTT_full_test/v2t_metrics/MeanR: 43.37324414715719
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.660739404249046
 mnt_best       : 22.61191328632993
 not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.02476 (QuantReg: 7.83587) QuantErr: 7.83587 batch_time=41.14074 
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.75270 (QuantReg: 7.98267) QuantErr: 7.98267 batch_time=0.45686 
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.07921 (QuantReg: 8.10087) QuantErr: 8.10087 batch_time=0.45364 
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.00579 (QuantReg: 8.13948) QuantErr: 8.13948 batch_time=0.44179 
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.17367 (QuantReg: 8.02826) QuantErr: 8.02826 batch_time=0.45217 
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.00952 (QuantReg: 8.06099) QuantErr: 8.06099 batch_time=0.44589 
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.66982 (QuantReg: 8.14208) QuantErr: 8.14208 batch_time=0.52457 
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.43036 (QuantReg: 8.22685) QuantErr: 8.22685 batch_time=0.48465 
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.01516 (QuantReg: 8.04826) QuantErr: 8.04826 batch_time=0.43995 
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.74141 (QuantReg: 8.20966) QuantErr: 8.20966 batch_time=0.44378 
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.85509 (QuantReg: 8.09303) QuantErr: 8.09303 batch_time=0.44102 
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.65568 (QuantReg: 8.33539) QuantErr: 8.33539 batch_time=0.47196 
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.84780 (QuantReg: 8.29652) QuantErr: 8.29652 batch_time=0.45664 
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.84090 (QuantReg: 8.42270) QuantErr: 8.42270 batch_time=0.47980 
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.93296 (QuantReg: 8.39156) QuantErr: 8.39156 batch_time=0.43898 
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.83867 (QuantReg: 8.20962) QuantErr: 8.20962 batch_time=0.46970 
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.34603 (QuantReg: 8.37478) QuantErr: 8.37478 batch_time=0.43579 
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.51462 (QuantReg: 8.35482) QuantErr: 8.35482 batch_time=0.44872 
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.58590 (QuantReg: 8.42138) QuantErr: 8.42138 batch_time=0.48556 
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.62254 (QuantReg: 8.13542) QuantErr: 8.13542 batch_time=0.48743 
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.69803 (QuantReg: 8.27116) QuantErr: 8.27116 batch_time=0.46856 
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.71547 (QuantReg: 8.30029) QuantErr: 8.30029 batch_time=0.49051 
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.18706 (QuantReg: 8.42819) QuantErr: 8.42819 batch_time=0.45851 
Train Epoch: 9 codebook_update_time=0.97185
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch9.pth ...
Done in 4.274s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch9.pth ...
Done in 8.889s
removing stale ckpt [epoch 8] [took 0.00s]
 epoch          : 9
 loss           : 1.7472841029167174
 quant_reg      : 8.235431802749634
 quant_err      : 8.235431802749634
 learning_rate  : 3.317102156445311e-05
 n_samples      : 288000
 n_steps        : 2250
 MSRVTT_full_val/t2v_metrics/R1: 25.955734406438633
 MSRVTT_full_val/t2v_metrics/R5: 64.38631790744466
 MSRVTT_full_val/t2v_metrics/R10: 78.87323943661971
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.156941649899396
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.89231381425333
 MSRVTT_full_val/v2t_metrics/R1: 28.973843058350102
 MSRVTT_full_val/v2t_metrics/R5: 70.02012072434607
 MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.920523138832998
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 55.37604907436509
 MSRVTT_full_test/t2v_metrics/R1: 9.531772575250836
 MSRVTT_full_test/t2v_metrics/R5: 29.698996655518396
 MSRVTT_full_test/t2v_metrics/R10: 43.41137123745819
 MSRVTT_full_test/t2v_metrics/R50: 77.59197324414716
 MSRVTT_full_test/t2v_metrics/MedR: 13.25
 MSRVTT_full_test/t2v_metrics/MeanR: 48.31722408026756
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.076661753409937
 MSRVTT_full_test/v2t_metrics/R1: 10.334448160535118
 MSRVTT_full_test/v2t_metrics/R5: 31.806020066889634
 MSRVTT_full_test/v2t_metrics/R10: 46.32107023411371
 MSRVTT_full_test/v2t_metrics/R50: 81.10367892976589
 MSRVTT_full_test/v2t_metrics/MedR: 12.0
 MSRVTT_full_test/v2t_metrics/MeanR: 42.08311036789298
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.785160462385342
 mnt_best       : 23.076661753409937
 not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.42137 (QuantReg: 7.95502) QuantErr: 7.95502 batch_time=38.46035 
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.65185 (QuantReg: 8.25987) QuantErr: 8.25987 batch_time=0.45795 
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.47496 (QuantReg: 8.05108) QuantErr: 8.05108 batch_time=0.48701 
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.31118 (QuantReg: 7.95652) QuantErr: 7.95652 batch_time=0.44212 
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.53654 (QuantReg: 8.48740) QuantErr: 8.48740 batch_time=0.45902 
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.26949 (QuantReg: 8.27512) QuantErr: 8.27512 batch_time=0.43605 
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.47260 (QuantReg: 8.33287) QuantErr: 8.33287 batch_time=0.45799 
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.66278 (QuantReg: 8.09018) QuantErr: 8.09018 batch_time=0.56195 
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.67497 (QuantReg: 8.20064) QuantErr: 8.20064 batch_time=0.50875 
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.19055 (QuantReg: 8.03707) QuantErr: 8.03707 batch_time=0.50070 
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.46324 (QuantReg: 8.24740) QuantErr: 8.24740 batch_time=0.50197 
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.84688 (QuantReg: 8.37060) QuantErr: 8.37060 batch_time=0.70997 
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.97607 (QuantReg: 8.28537) QuantErr: 8.28537 batch_time=1.03811 
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.15120 (QuantReg: 8.42051) QuantErr: 8.42051 batch_time=0.49317 
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.37354 (QuantReg: 8.49020) QuantErr: 8.49020 batch_time=0.45622 
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.33163 (QuantReg: 8.41267) QuantErr: 8.41267 batch_time=0.48668 
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.53381 (QuantReg: 8.28936) QuantErr: 8.28936 batch_time=0.46552 
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.48198 (QuantReg: 8.68512) QuantErr: 8.68512 batch_time=0.45434 
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.37676 (QuantReg: 8.67065) QuantErr: 8.67065 batch_time=0.54113 
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.56342 (QuantReg: 8.29136) QuantErr: 8.29136 batch_time=0.49592 
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.53651 (QuantReg: 8.14687) QuantErr: 8.14687 batch_time=0.44258 
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.08562 (QuantReg: 8.33201) QuantErr: 8.33201 batch_time=0.45122 
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.87838 (QuantReg: 8.20865) QuantErr: 8.20865 batch_time=0.45626 
Train Epoch: 10 codebook_update_time=0.88879
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch10.pth ...
Done in 4.227s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch10.pth ...
Done in 10.223s
removing stale ckpt [epoch 9] [took 0.01s]
 epoch          : 10
 loss           : 1.6549950947761536
 quant_reg      : 8.282406482696533
 quant_err      : 8.282406482696533
 learning_rate  : 3.151247048623045e-05
 n_samples      : 320000
 n_steps        : 2500
 MSRVTT_full_val/t2v_metrics/R1: 27.56539235412475
 MSRVTT_full_val/t2v_metrics/R5: 62.97786720321932
 MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.364185110663984
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 51.58571205424756
 MSRVTT_full_val/v2t_metrics/R1: 29.175050301810867
 MSRVTT_full_val/v2t_metrics/R5: 68.00804828973843
 MSRVTT_full_val/v2t_metrics/R10: 82.897384305835
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.865191146881288
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.79037221304807
 MSRVTT_full_test/t2v_metrics/R1: 10.100334448160535
 MSRVTT_full_test/t2v_metrics/R5: 30.501672240802677
 MSRVTT_full_test/t2v_metrics/R10: 44.280936454849495
 MSRVTT_full_test/t2v_metrics/R50: 78.22742474916387
 MSRVTT_full_test/t2v_metrics/MedR: 13.0
 MSRVTT_full_test/t2v_metrics/MeanR: 46.36020066889632
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.894176486284685
 MSRVTT_full_test/v2t_metrics/R1: 10.501672240802675
 MSRVTT_full_test/v2t_metrics/R5: 32.94314381270903
 MSRVTT_full_test/v2t_metrics/R10: 48.22742474916388
 MSRVTT_full_test/v2t_metrics/R50: 82.20735785953177
 MSRVTT_full_test/v2t_metrics/MedR: 11.0
 MSRVTT_full_test/v2t_metrics/MeanR: 40.77859531772575
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.552840924342828
 mnt_best       : 23.894176486284685
 not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.54825 (QuantReg: 8.19456) QuantErr: 8.19456 batch_time=42.00426 
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.35264 (QuantReg: 8.34101) QuantErr: 8.34101 batch_time=0.51874 
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.59073 (QuantReg: 8.00216) QuantErr: 8.00216 batch_time=0.51159 
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.60471 (QuantReg: 8.31672) QuantErr: 8.31672 batch_time=0.52294 
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.71122 (QuantReg: 8.46028) QuantErr: 8.46028 batch_time=0.43072 
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.40994 (QuantReg: 8.40146) QuantErr: 8.40146 batch_time=0.47197 
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.83638 (QuantReg: 8.14286) QuantErr: 8.14286 batch_time=0.44816 
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.49060 (QuantReg: 8.47997) QuantErr: 8.47997 batch_time=0.45307 
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.52489 (QuantReg: 8.20913) QuantErr: 8.20913 batch_time=1.11839 
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.82774 (QuantReg: 8.24558) QuantErr: 8.24558 batch_time=0.45612 
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.88623 (QuantReg: 8.10766) QuantErr: 8.10766 batch_time=0.47654 
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.61005 (QuantReg: 8.57516) QuantErr: 8.57516 batch_time=0.50896 
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.99752 (QuantReg: 8.18654) QuantErr: 8.18654 batch_time=0.49879 
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.77971 (QuantReg: 8.21146) QuantErr: 8.21146 batch_time=0.43638 
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.59976 (QuantReg: 8.22147) QuantErr: 8.22147 batch_time=0.50918 
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.60412 (QuantReg: 8.17077) QuantErr: 8.17077 batch_time=0.45580 
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.59179 (QuantReg: 8.11076) QuantErr: 8.11076 batch_time=0.79430 
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.21101 (QuantReg: 8.29494) QuantErr: 8.29494 batch_time=0.50926 
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.41081 (QuantReg: 8.66389) QuantErr: 8.66389 batch_time=0.48804 
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.69623 (QuantReg: 8.51255) QuantErr: 8.51255 batch_time=0.49917 
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.66119 (QuantReg: 8.69075) QuantErr: 8.69075 batch_time=0.46573 
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.36089 (QuantReg: 8.47152) QuantErr: 8.47152 batch_time=0.51592 
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.22945 (QuantReg: 8.73256) QuantErr: 8.73256 batch_time=0.47680 
Train Epoch: 11 codebook_update_time=0.94405
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch11.pth ...
Done in 22.883s
removing stale ckpt [epoch 10] [took 0.00s]
 epoch          : 11
 loss           : 1.5657899136543274
 quant_reg      : 8.371494171142578
 quant_err      : 8.371494171142578
 learning_rate  : 2.993684696191893e-05
 n_samples      : 352000
 n_steps        : 2750
 MSRVTT_full_val/t2v_metrics/R1: 27.364185110663986
 MSRVTT_full_val/t2v_metrics/R5: 62.17303822937626
 MSRVTT_full_val/t2v_metrics/R10: 78.26961770623743
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.5
 MSRVTT_full_val/t2v_metrics/MeanR: 10.100603621730382
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 51.06530626898277
 MSRVTT_full_val/v2t_metrics/R1: 29.577464788732396
 MSRVTT_full_val/v2t_metrics/R5: 68.61167002012073
 MSRVTT_full_val/v2t_metrics/R10: 83.29979879275653
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 8.148893360160965
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 55.29267861680702
 MSRVTT_full_test/t2v_metrics/R1: 9.297658862876254
 MSRVTT_full_test/t2v_metrics/R5: 29.464882943143813
 MSRVTT_full_test/t2v_metrics/R10: 44.54849498327759
 MSRVTT_full_test/t2v_metrics/R50: 79.09698996655518
 MSRVTT_full_test/t2v_metrics/MedR: 13.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.5809364548495
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.023452795879013
 MSRVTT_full_test/v2t_metrics/R1: 11.77257525083612
 MSRVTT_full_test/v2t_metrics/R5: 33.87959866220736
 MSRVTT_full_test/v2t_metrics/R10: 48.72909698996656
 MSRVTT_full_test/v2t_metrics/R50: 82.3076923076923
 MSRVTT_full_test/v2t_metrics/MedR: 11.0
 MSRVTT_full_test/v2t_metrics/MeanR: 40.29598662207358
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.88640266348746
 mnt_best       : 23.894176486284685
 not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.55551 (QuantReg: 8.29226) QuantErr: 8.29226 batch_time=36.13115 
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.79523 (QuantReg: 8.38973) QuantErr: 8.38973 batch_time=0.44346 
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.31757 (QuantReg: 8.22722) QuantErr: 8.22722 batch_time=0.45529 
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.38968 (QuantReg: 8.37793) QuantErr: 8.37793 batch_time=0.47899 
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.25068 (QuantReg: 8.56202) QuantErr: 8.56202 batch_time=0.44544 
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.55167 (QuantReg: 8.35994) QuantErr: 8.35994 batch_time=0.45901 
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.31569 (QuantReg: 8.43706) QuantErr: 8.43706 batch_time=0.71085 
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.53443 (QuantReg: 8.34418) QuantErr: 8.34418 batch_time=0.47720 
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.38056 (QuantReg: 8.31768) QuantErr: 8.31768 batch_time=0.43589 
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.71183 (QuantReg: 8.37311) QuantErr: 8.37311 batch_time=0.72186 
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.18538 (QuantReg: 8.35047) QuantErr: 8.35047 batch_time=0.44567 
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.60226 (QuantReg: 8.31562) QuantErr: 8.31562 batch_time=0.44244 
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.33658 (QuantReg: 8.51434) QuantErr: 8.51434 batch_time=0.43446 
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.24546 (QuantReg: 8.74878) QuantErr: 8.74878 batch_time=0.45148 
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.73824 (QuantReg: 8.32606) QuantErr: 8.32606 batch_time=0.46037 
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.15702 (QuantReg: 8.40479) QuantErr: 8.40479 batch_time=0.47581 
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.18307 (QuantReg: 8.42032) QuantErr: 8.42032 batch_time=0.56758 
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.82137 (QuantReg: 8.24154) QuantErr: 8.24154 batch_time=0.46884 
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.56777 (QuantReg: 8.30400) QuantErr: 8.30400 batch_time=0.46423 
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.93165 (QuantReg: 8.38916) QuantErr: 8.38916 batch_time=0.43006 
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.53756 (QuantReg: 8.15477) QuantErr: 8.15477 batch_time=0.43950 
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.54785 (QuantReg: 8.51947) QuantErr: 8.51947 batch_time=0.42936 
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.54525 (QuantReg: 8.65172) QuantErr: 8.65172 batch_time=0.49093 
Train Epoch: 12 codebook_update_time=1.01123
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch12.pth ...
Done in 4.519s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch12.pth ...
Done in 9.285s
removing stale ckpt [epoch 11] [took 0.01s]
 epoch          : 12
 loss           : 1.498270281791687
 quant_reg      : 8.412913555145265
 quant_err      : 8.412913555145265
 learning_rate  : 2.844000461382298e-05
 n_samples      : 384000
 n_steps        : 3000
 MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
 MSRVTT_full_val/t2v_metrics/R5: 62.776659959758554
 MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.462776659959758
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.76887351665544
 MSRVTT_full_val/v2t_metrics/R1: 31.388329979879277
 MSRVTT_full_val/v2t_metrics/R5: 71.42857142857143
 MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.4044265593561365
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.4806883548893
 MSRVTT_full_test/t2v_metrics/R1: 9.832775919732441
 MSRVTT_full_test/t2v_metrics/R5: 31.103678929765888
 MSRVTT_full_test/t2v_metrics/R10: 44.8494983277592
 MSRVTT_full_test/t2v_metrics/R50: 79.59866220735786
 MSRVTT_full_test/t2v_metrics/MedR: 13.0
 MSRVTT_full_test/t2v_metrics/MeanR: 44.663879598662206
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.93766752123661
 MSRVTT_full_test/v2t_metrics/R1: 10.76923076923077
 MSRVTT_full_test/v2t_metrics/R5: 33.74581939799331
 MSRVTT_full_test/v2t_metrics/R10: 49.063545150501675
 MSRVTT_full_test/v2t_metrics/R50: 83.01003344481606
 MSRVTT_full_test/v2t_metrics/MedR: 11.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.00501672240803
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.124893514683496
 mnt_best       : 23.93766752123661
 not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.36834 (QuantReg: 8.40863) QuantErr: 8.40863 batch_time=37.89405 
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.94994 (QuantReg: 8.04239) QuantErr: 8.04239 batch_time=0.56406 
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.52175 (QuantReg: 8.58910) QuantErr: 8.58910 batch_time=0.46560 
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.28665 (QuantReg: 8.50205) QuantErr: 8.50205 batch_time=0.43852 
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.37741 (QuantReg: 8.40883) QuantErr: 8.40883 batch_time=0.52222 
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.42336 (QuantReg: 8.55419) QuantErr: 8.55419 batch_time=0.49656 
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.49197 (QuantReg: 8.63760) QuantErr: 8.63760 batch_time=0.48350 
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.81958 (QuantReg: 8.60985) QuantErr: 8.60985 batch_time=0.54250 
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.19853 (QuantReg: 8.13733) QuantErr: 8.13733 batch_time=0.45280 
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.57721 (QuantReg: 8.44085) QuantErr: 8.44085 batch_time=0.49658 
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.54021 (QuantReg: 8.37190) QuantErr: 8.37190 batch_time=0.44265 
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.02915 (QuantReg: 8.59293) QuantErr: 8.59293 batch_time=1.18136 
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.33551 (QuantReg: 8.48987) QuantErr: 8.48987 batch_time=4.51503 
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.40770 (QuantReg: 8.61752) QuantErr: 8.61752 batch_time=0.46168 
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.70159 (QuantReg: 8.39969) QuantErr: 8.39969 batch_time=0.44868 
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.26749 (QuantReg: 8.37543) QuantErr: 8.37543 batch_time=0.46568 
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.16584 (QuantReg: 8.63921) QuantErr: 8.63921 batch_time=0.46506 
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.76950 (QuantReg: 8.40182) QuantErr: 8.40182 batch_time=0.53677 
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.24544 (QuantReg: 8.69702) QuantErr: 8.69702 batch_time=0.53242 
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.82514 (QuantReg: 8.55359) QuantErr: 8.55359 batch_time=0.45656 
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.67045 (QuantReg: 8.58233) QuantErr: 8.58233 batch_time=0.44479 
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.30476 (QuantReg: 8.50051) QuantErr: 8.50051 batch_time=0.45957 
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.37235 (QuantReg: 8.57842) QuantErr: 8.57842 batch_time=0.52691 
Train Epoch: 13 codebook_update_time=1.52312
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch13.pth ...
Done in 3.948s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch13.pth ...
Done in 8.369s
removing stale ckpt [epoch 12] [took 0.00s]
 epoch          : 13
 loss           : 1.4319962499141694
 quant_reg      : 8.47877261352539
 quant_err      : 8.47877261352539
 learning_rate  : 2.7018004383131832e-05
 n_samples      : 416000
 n_steps        : 3250
 MSRVTT_full_val/t2v_metrics/R1: 26.156941649899398
 MSRVTT_full_val/t2v_metrics/R5: 63.17907444668008
 MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.907444668008047
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.74564709092073
 MSRVTT_full_val/v2t_metrics/R1: 34.40643863179074
 MSRVTT_full_val/v2t_metrics/R5: 70.22132796780684
 MSRVTT_full_val/v2t_metrics/R10: 83.90342052313883
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.400402414486922
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.74390443789089
 MSRVTT_full_test/t2v_metrics/R1: 9.698996655518394
 MSRVTT_full_test/t2v_metrics/R5: 31.37123745819398
 MSRVTT_full_test/t2v_metrics/R10: 45.986622073578594
 MSRVTT_full_test/t2v_metrics/R50: 78.79598662207358
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 45.00284280936455
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.097019170266726
 MSRVTT_full_test/v2t_metrics/R1: 11.638795986622073
 MSRVTT_full_test/v2t_metrics/R5: 34.71571906354515
 MSRVTT_full_test/v2t_metrics/R10: 50.30100334448161
 MSRVTT_full_test/v2t_metrics/R50: 83.01003344481606
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.2675585284281
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.290005482192498
 mnt_best       : 24.097019170266726
 not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.29100 (QuantReg: 8.48368) QuantErr: 8.48368 batch_time=37.27454 
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.87480 (QuantReg: 8.63382) QuantErr: 8.63382 batch_time=0.52387 
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.36418 (QuantReg: 8.52572) QuantErr: 8.52572 batch_time=0.48570 
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.35827 (QuantReg: 8.41562) QuantErr: 8.41562 batch_time=0.47269 
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.74584 (QuantReg: 8.40684) QuantErr: 8.40684 batch_time=0.47960 
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.42114 (QuantReg: 8.35642) QuantErr: 8.35642 batch_time=0.56214 
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.61128 (QuantReg: 8.48650) QuantErr: 8.48650 batch_time=0.44487 
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.63346 (QuantReg: 8.19270) QuantErr: 8.19270 batch_time=0.48219 
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.03081 (QuantReg: 8.35285) QuantErr: 8.35285 batch_time=0.47766 
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.38252 (QuantReg: 8.34219) QuantErr: 8.34219 batch_time=0.47605 
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.24638 (QuantReg: 8.59967) QuantErr: 8.59967 batch_time=0.44505 
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.62632 (QuantReg: 8.23174) QuantErr: 8.23174 batch_time=0.51179 
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.20980 (QuantReg: 8.69773) QuantErr: 8.69773 batch_time=0.53989 
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.08810 (QuantReg: 8.54517) QuantErr: 8.54517 batch_time=0.54534 
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.38310 (QuantReg: 8.57786) QuantErr: 8.57786 batch_time=0.45974 
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.17304 (QuantReg: 8.57750) QuantErr: 8.57750 batch_time=0.46963 
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.10948 (QuantReg: 8.43885) QuantErr: 8.43885 batch_time=0.55185 
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.70476 (QuantReg: 8.73166) QuantErr: 8.73166 batch_time=0.42826 
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.34841 (QuantReg: 8.41808) QuantErr: 8.41808 batch_time=0.52042 
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.66983 (QuantReg: 8.40590) QuantErr: 8.40590 batch_time=0.47435 
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.41365 (QuantReg: 8.62997) QuantErr: 8.62997 batch_time=0.43701 
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.31312 (QuantReg: 8.70792) QuantErr: 8.70792 batch_time=0.55807 
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.34421 (QuantReg: 8.53538) QuantErr: 8.53538 batch_time=0.51108 
Train Epoch: 14 codebook_update_time=1.00927
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch14.pth ...
Done in 3.999s
removing stale ckpt [epoch 13] [took 0.00s]
 epoch          : 14
 loss           : 1.3980245838165284
 quant_reg      : 8.509741901397705
 quant_err      : 8.509741901397705
 learning_rate  : 2.566710416397524e-05
 n_samples      : 448000
 n_steps        : 3500
 MSRVTT_full_val/t2v_metrics/R1: 25.553319919517104
 MSRVTT_full_val/t2v_metrics/R5: 62.776659959758554
 MSRVTT_full_val/t2v_metrics/R10: 76.65995975855131
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.720321931589536
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 49.72842542210175
 MSRVTT_full_val/v2t_metrics/R1: 30.58350100603622
 MSRVTT_full_val/v2t_metrics/R5: 67.40442655935614
 MSRVTT_full_val/v2t_metrics/R10: 82.49496981891348
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.877263581488934
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 55.40313777467415
 MSRVTT_full_test/t2v_metrics/R1: 9.732441471571907
 MSRVTT_full_test/t2v_metrics/R5: 29.665551839464882
 MSRVTT_full_test/t2v_metrics/R10: 43.84615384615385
 MSRVTT_full_test/t2v_metrics/R50: 77.4247491638796
 MSRVTT_full_test/t2v_metrics/MedR: 14.0
 MSRVTT_full_test/t2v_metrics/MeanR: 51.56789297658863
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.306044884304463
 MSRVTT_full_test/v2t_metrics/R1: 11.17056856187291
 MSRVTT_full_test/v2t_metrics/R5: 33.27759197324415
 MSRVTT_full_test/v2t_metrics/R10: 47.792642140468224
 MSRVTT_full_test/v2t_metrics/R50: 81.30434782608695
 MSRVTT_full_test/v2t_metrics/MedR: 11.5
 MSRVTT_full_test/v2t_metrics/MeanR: 42.215384615384615
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.093323811842435
 mnt_best       : 24.097019170266726
 not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.76880 (QuantReg: 8.29853) QuantErr: 8.29853 batch_time=38.03712 
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.27545 (QuantReg: 8.42686) QuantErr: 8.42686 batch_time=0.44555 
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.29186 (QuantReg: 8.51319) QuantErr: 8.51319 batch_time=0.48866 
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.44510 (QuantReg: 8.41016) QuantErr: 8.41016 batch_time=0.48289 
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.42260 (QuantReg: 8.66090) QuantErr: 8.66090 batch_time=0.51075 
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.57303 (QuantReg: 8.27209) QuantErr: 8.27209 batch_time=0.46661 
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.71030 (QuantReg: 8.42282) QuantErr: 8.42282 batch_time=0.46336 
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.18040 (QuantReg: 8.57352) QuantErr: 8.57352 batch_time=0.90123 
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.67017 (QuantReg: 8.53064) QuantErr: 8.53064 batch_time=0.49688 
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.34609 (QuantReg: 8.58758) QuantErr: 8.58758 batch_time=0.48161 
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.58289 (QuantReg: 8.49699) QuantErr: 8.49699 batch_time=0.51449 
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.12934 (QuantReg: 8.36041) QuantErr: 8.36041 batch_time=0.46157 
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.30384 (QuantReg: 8.42676) QuantErr: 8.42676 batch_time=1.58395 
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.56022 (QuantReg: 8.57522) QuantErr: 8.57522 batch_time=0.43784 
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.43859 (QuantReg: 8.44091) QuantErr: 8.44091 batch_time=0.43127 
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.44832 (QuantReg: 8.68060) QuantErr: 8.68060 batch_time=0.44780 
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.57859 (QuantReg: 8.73213) QuantErr: 8.73213 batch_time=0.43491 
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.23223 (QuantReg: 8.66219) QuantErr: 8.66219 batch_time=0.47474 
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.26382 (QuantReg: 8.59745) QuantErr: 8.59745 batch_time=0.43368 
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.51083 (QuantReg: 8.33848) QuantErr: 8.33848 batch_time=0.48410 
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.62551 (QuantReg: 8.64365) QuantErr: 8.64365 batch_time=0.44852 
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.53733 (QuantReg: 8.65686) QuantErr: 8.65686 batch_time=0.44697 
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.43212 (QuantReg: 8.39636) QuantErr: 8.39636 batch_time=0.45332 
Train Epoch: 15 codebook_update_time=0.92178
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch15.pth ...
Done in 18.585s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch15.pth ...
Done in 22.606s
removing stale ckpt [epoch 14] [took 0.08s]
 epoch          : 15
 loss           : 1.3389672248363496
 quant_reg      : 8.552904838562013
 quant_err      : 8.552904838562013
 learning_rate  : 2.4383748955776477e-05
 n_samples      : 480000
 n_steps        : 3750
 MSRVTT_full_val/t2v_metrics/R1: 28.571428571428573
 MSRVTT_full_val/t2v_metrics/R5: 64.58752515090544
 MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
 MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.95774647887324
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.646833312424384
 MSRVTT_full_val/v2t_metrics/R1: 35.61368209255533
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
 MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.917505030181086
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.04841440597589
 MSRVTT_full_test/t2v_metrics/R1: 9.832775919732441
 MSRVTT_full_test/t2v_metrics/R5: 31.40468227424749
 MSRVTT_full_test/t2v_metrics/R10: 46.65551839464883
 MSRVTT_full_test/t2v_metrics/R50: 79.29765886287625
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.10969899665552
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.33274968239196
 MSRVTT_full_test/v2t_metrics/R1: 11.538461538461538
 MSRVTT_full_test/v2t_metrics/R5: 36.05351170568562
 MSRVTT_full_test/v2t_metrics/R10: 51.03678929765886
 MSRVTT_full_test/v2t_metrics/R50: 83.17725752508362
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.74531772575251
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.69021162261362
 mnt_best       : 24.33274968239196
 not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.23047 (QuantReg: 8.24341) QuantErr: 8.24341 batch_time=31.35755 
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.26376 (QuantReg: 8.67214) QuantErr: 8.67214 batch_time=0.45334 
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 1.55718 (QuantReg: 8.58204) QuantErr: 8.58204 batch_time=0.51259 
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.27673 (QuantReg: 8.70040) QuantErr: 8.70040 batch_time=0.47212 
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.24699 (QuantReg: 8.56890) QuantErr: 8.56890 batch_time=0.52848 
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.39903 (QuantReg: 8.38432) QuantErr: 8.38432 batch_time=0.47524 
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 1.41294 (QuantReg: 8.77750) QuantErr: 8.77750 batch_time=1.87292 
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.23959 (QuantReg: 8.54809) QuantErr: 8.54809 batch_time=0.44396 
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.21065 (QuantReg: 8.80941) QuantErr: 8.80941 batch_time=0.54609 
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.14738 (QuantReg: 8.64345) QuantErr: 8.64345 batch_time=0.44223 
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 1.40208 (QuantReg: 8.98514) QuantErr: 8.98514 batch_time=0.50020 
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 1.34501 (QuantReg: 8.67928) QuantErr: 8.67928 batch_time=0.93311 
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.43242 (QuantReg: 8.61736) QuantErr: 8.61736 batch_time=0.51940 
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.16467 (QuantReg: 8.70598) QuantErr: 8.70598 batch_time=0.47358 
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.36041 (QuantReg: 8.70490) QuantErr: 8.70490 batch_time=0.44110 
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.26326 (QuantReg: 8.62556) QuantErr: 8.62556 batch_time=0.43963 
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.46314 (QuantReg: 8.42347) QuantErr: 8.42347 batch_time=0.45402 
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.02163 (QuantReg: 8.62024) QuantErr: 8.62024 batch_time=0.47548 
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.37881 (QuantReg: 8.76947) QuantErr: 8.76947 batch_time=0.43964 
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.26051 (QuantReg: 8.70336) QuantErr: 8.70336 batch_time=0.44393 
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.10854 (QuantReg: 8.73724) QuantErr: 8.73724 batch_time=0.53218 
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.29022 (QuantReg: 8.49542) QuantErr: 8.49542 batch_time=0.44327 
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.29741 (QuantReg: 8.86617) QuantErr: 8.86617 batch_time=0.60756 
Train Epoch: 16 codebook_update_time=0.94660
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch16.pth ...
Done in 3.945s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch16.pth ...
Done in 25.838s
removing stale ckpt [epoch 15] [took 0.00s]
 epoch          : 16
 loss           : 1.3182028224468232
 quant_reg      : 8.608148330688477
 quant_err      : 8.608148330688477
 learning_rate  : 2.3164561507987653e-05
 n_samples      : 512000
 n_steps        : 4000
 MSRVTT_full_val/t2v_metrics/R1: 28.37022132796781
 MSRVTT_full_val/t2v_metrics/R5: 64.58752515090544
 MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.901408450704224
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.74476361617637
 MSRVTT_full_val/v2t_metrics/R1: 34.60764587525151
 MSRVTT_full_val/v2t_metrics/R5: 73.03822937625755
 MSRVTT_full_val/v2t_metrics/R10: 85.91549295774648
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.570422535211268
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.10785917695611
 MSRVTT_full_test/t2v_metrics/R1: 11.20401337792642
 MSRVTT_full_test/t2v_metrics/R5: 30.93645484949833
 MSRVTT_full_test/t2v_metrics/R10: 46.52173913043478
 MSRVTT_full_test/t2v_metrics/R50: 80.13377926421404
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 44.73729096989967
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.263879133364796
 MSRVTT_full_test/v2t_metrics/R1: 11.638795986622073
 MSRVTT_full_test/v2t_metrics/R5: 36.55518394648829
 MSRVTT_full_test/v2t_metrics/R10: 50.702341137123746
 MSRVTT_full_test/v2t_metrics/R50: 83.37792642140468
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.798996655518394
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.837379326675045
 mnt_best       : 25.263879133364796
 not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.27115 (QuantReg: 8.44224) QuantErr: 8.44224 batch_time=43.75622 
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 1.48451 (QuantReg: 8.57573) QuantErr: 8.57573 batch_time=0.52714 
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.15460 (QuantReg: 8.55690) QuantErr: 8.55690 batch_time=0.54122 
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.06745 (QuantReg: 8.37192) QuantErr: 8.37192 batch_time=0.46636 
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 1.46759 (QuantReg: 8.48297) QuantErr: 8.48297 batch_time=0.46677 
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.14297 (QuantReg: 8.38984) QuantErr: 8.38984 batch_time=0.49195 
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 0.99617 (QuantReg: 8.87456) QuantErr: 8.87456 batch_time=0.49019 
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.35427 (QuantReg: 8.54870) QuantErr: 8.54870 batch_time=0.52715 
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.12587 (QuantReg: 8.84552) QuantErr: 8.84552 batch_time=1.39371 
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.30255 (QuantReg: 8.73670) QuantErr: 8.73670 batch_time=0.49776 
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.25100 (QuantReg: 8.76017) QuantErr: 8.76017 batch_time=0.49307 
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.31695 (QuantReg: 8.45290) QuantErr: 8.45290 batch_time=0.52590 
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.11292 (QuantReg: 8.55925) QuantErr: 8.55925 batch_time=0.43598 
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 1.67595 (QuantReg: 8.50317) QuantErr: 8.50317 batch_time=0.44694 
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.29818 (QuantReg: 8.94987) QuantErr: 8.94987 batch_time=0.50107 
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.12425 (QuantReg: 8.42895) QuantErr: 8.42895 batch_time=0.52807 
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.12843 (QuantReg: 8.58111) QuantErr: 8.58111 batch_time=0.46767 
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.13564 (QuantReg: 8.72340) QuantErr: 8.72340 batch_time=0.51473 
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 1.50370 (QuantReg: 8.91552) QuantErr: 8.91552 batch_time=0.46228 
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.27310 (QuantReg: 8.81724) QuantErr: 8.81724 batch_time=0.47211 
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.08420 (QuantReg: 8.68959) QuantErr: 8.68959 batch_time=0.46347 
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.38677 (QuantReg: 8.90173) QuantErr: 8.90173 batch_time=0.49091 
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.22963 (QuantReg: 8.73224) QuantErr: 8.73224 batch_time=0.53422 
Train Epoch: 17 codebook_update_time=0.91749
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch17.pth ...
Done in 4.289s
removing stale ckpt [epoch 16] [took 0.00s]
 epoch          : 17
 loss           : 1.2652524721622467
 quant_reg      : 8.657865341186524
 quant_err      : 8.657865341186524
 learning_rate  : 2.2006333432588268e-05
 n_samples      : 544000
 n_steps        : 4250
 MSRVTT_full_val/t2v_metrics/R1: 29.577464788732396
 MSRVTT_full_val/t2v_metrics/R5: 65.3923541247485
 MSRVTT_full_val/t2v_metrics/R10: 81.08651911468813
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.690140845070424
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.92773647385243
 MSRVTT_full_val/v2t_metrics/R1: 33.80281690140845
 MSRVTT_full_val/v2t_metrics/R5: 73.03822937625755
 MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.400402414486922
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.263448772167266
 MSRVTT_full_test/t2v_metrics/R1: 10.869565217391305
 MSRVTT_full_test/t2v_metrics/R5: 31.00334448160535
 MSRVTT_full_test/t2v_metrics/R10: 45.68561872909699
 MSRVTT_full_test/t2v_metrics/R50: 79.76588628762542
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 45.50501672240803
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.87711859199731
 MSRVTT_full_test/v2t_metrics/R1: 11.97324414715719
 MSRVTT_full_test/v2t_metrics/R5: 34.61538461538461
 MSRVTT_full_test/v2t_metrics/R10: 50.53511705685619
 MSRVTT_full_test/v2t_metrics/R50: 83.51170568561874
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.28628762541806
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.565006037268997
 mnt_best       : 25.263879133364796
 not_improved_count: 1
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 1.36229 (QuantReg: 8.64100) QuantErr: 8.64100 batch_time=33.31898 
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.28436 (QuantReg: 8.69621) QuantErr: 8.69621 batch_time=0.46508 
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.19963 (QuantReg: 8.54219) QuantErr: 8.54219 batch_time=0.48243 
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.19630 (QuantReg: 8.53339) QuantErr: 8.53339 batch_time=0.44902 
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.32297 (QuantReg: 8.57071) QuantErr: 8.57071 batch_time=0.45739 
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.30137 (QuantReg: 8.66813) QuantErr: 8.66813 batch_time=0.45250 
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 1.59795 (QuantReg: 8.59006) QuantErr: 8.59006 batch_time=0.46132 
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 1.34083 (QuantReg: 8.74612) QuantErr: 8.74612 batch_time=4.29766 
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.13121 (QuantReg: 8.52375) QuantErr: 8.52375 batch_time=0.43672 
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.23457 (QuantReg: 8.63341) QuantErr: 8.63341 batch_time=0.45508 
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.08460 (QuantReg: 8.66978) QuantErr: 8.66978 batch_time=0.46332 
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 1.56464 (QuantReg: 8.58583) QuantErr: 8.58583 batch_time=0.48096 
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 1.70708 (QuantReg: 8.85059) QuantErr: 8.85059 batch_time=0.52338 
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.09584 (QuantReg: 8.61501) QuantErr: 8.61501 batch_time=1.75258 
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.30429 (QuantReg: 8.81639) QuantErr: 8.81639 batch_time=0.45936 
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 1.57080 (QuantReg: 8.76459) QuantErr: 8.76459 batch_time=0.52096 
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.17335 (QuantReg: 8.72175) QuantErr: 8.72175 batch_time=0.48584 
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 0.98811 (QuantReg: 8.72619) QuantErr: 8.72619 batch_time=0.44524 
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.21947 (QuantReg: 8.59677) QuantErr: 8.59677 batch_time=0.47777 
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 1.57590 (QuantReg: 8.75057) QuantErr: 8.75057 batch_time=0.47313 
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.26141 (QuantReg: 8.69238) QuantErr: 8.69238 batch_time=0.44787 
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 0.84700 (QuantReg: 8.73680) QuantErr: 8.73680 batch_time=0.46957 
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.09927 (QuantReg: 8.84205) QuantErr: 8.84205 batch_time=0.45081 
Train Epoch: 18 codebook_update_time=0.94018
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch18.pth ...
Done in 5.880s
removing stale ckpt [epoch 17] [took 0.00s]
 epoch          : 18
 loss           : 1.2301620693206787
 quant_reg      : 8.670045093536377
 quant_err      : 8.670045093536377
 learning_rate  : 2.0906016760958855e-05
 n_samples      : 576000
 n_steps        : 4500
 MSRVTT_full_val/t2v_metrics/R1: 29.979879275653925
 MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
 MSRVTT_full_val/t2v_metrics/R10: 79.87927565392354
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.722334004024145
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.79022700636815
 MSRVTT_full_val/v2t_metrics/R1: 32.394366197183096
 MSRVTT_full_val/v2t_metrics/R5: 71.0261569416499
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.232394366197183
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.070726701945034
 MSRVTT_full_test/t2v_metrics/R1: 10.334448160535118
 MSRVTT_full_test/t2v_metrics/R5: 31.638795986622075
 MSRVTT_full_test/t2v_metrics/R10: 45.88628762541806
 MSRVTT_full_test/t2v_metrics/R50: 79.49832775919732
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 46.765551839464884
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.663992940292626
 MSRVTT_full_test/v2t_metrics/R1: 11.505016722408026
 MSRVTT_full_test/v2t_metrics/R5: 34.81605351170568
 MSRVTT_full_test/v2t_metrics/R10: 49.4314381270903
 MSRVTT_full_test/v2t_metrics/R50: 83.61204013377926
 MSRVTT_full_test/v2t_metrics/MedR: 11.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.69882943143813
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.053493015546195
 mnt_best       : 25.263879133364796
 not_improved_count: 2
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.13345 (QuantReg: 8.77547) QuantErr: 8.77547 batch_time=35.71180 
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.22365 (QuantReg: 8.73607) QuantErr: 8.73607 batch_time=0.47746 
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.34288 (QuantReg: 8.48270) QuantErr: 8.48270 batch_time=0.43549 
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.32799 (QuantReg: 8.80531) QuantErr: 8.80531 batch_time=0.45476 
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.19352 (QuantReg: 8.77804) QuantErr: 8.77804 batch_time=0.44150 
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.29207 (QuantReg: 8.79075) QuantErr: 8.79075 batch_time=0.54531 
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.22004 (QuantReg: 8.66687) QuantErr: 8.66687 batch_time=2.39429 
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.54258 (QuantReg: 8.83544) QuantErr: 8.83544 batch_time=0.43788 
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.26003 (QuantReg: 8.50540) QuantErr: 8.50540 batch_time=0.54597 
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.14324 (QuantReg: 8.75856) QuantErr: 8.75856 batch_time=0.44902 
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.04342 (QuantReg: 8.78110) QuantErr: 8.78110 batch_time=0.46009 
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.20436 (QuantReg: 8.80947) QuantErr: 8.80947 batch_time=0.57743 
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.25223 (QuantReg: 8.72697) QuantErr: 8.72697 batch_time=1.95281 
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 0.82881 (QuantReg: 8.55453) QuantErr: 8.55453 batch_time=0.45672 
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.31849 (QuantReg: 8.76754) QuantErr: 8.76754 batch_time=0.49216 
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 0.85378 (QuantReg: 8.54403) QuantErr: 8.54403 batch_time=0.44253 
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.32271 (QuantReg: 8.74001) QuantErr: 8.74001 batch_time=0.43984 
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.40996 (QuantReg: 8.78638) QuantErr: 8.78638 batch_time=0.90224 
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.14715 (QuantReg: 8.72192) QuantErr: 8.72192 batch_time=0.49188 
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 0.89961 (QuantReg: 8.85055) QuantErr: 8.85055 batch_time=0.44197 
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 0.86696 (QuantReg: 8.81271) QuantErr: 8.81271 batch_time=0.43366 
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 0.99901 (QuantReg: 8.72774) QuantErr: 8.72774 batch_time=0.52891 
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.10560 (QuantReg: 8.67653) QuantErr: 8.67653 batch_time=0.55418 
Train Epoch: 19 codebook_update_time=0.96438
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch19.pth ...
Done in 13.000s
removing stale ckpt [epoch 18] [took 0.00s]
 epoch          : 19
 loss           : 1.2000445659160615
 quant_reg      : 8.708576007843018
 quant_err      : 8.708576007843018
 learning_rate  : 1.986071592291091e-05
 n_samples      : 608000
 n_steps        : 4750
 MSRVTT_full_val/t2v_metrics/R1: 26.961770623742456
 MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
 MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
 MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.158953722334005
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 51.74601399386098
 MSRVTT_full_val/v2t_metrics/R1: 32.394366197183096
 MSRVTT_full_val/v2t_metrics/R5: 73.44064386317908
 MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.113682092555332
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.39571338319652
 MSRVTT_full_test/t2v_metrics/R1: 10.367892976588628
 MSRVTT_full_test/t2v_metrics/R5: 31.97324414715719
 MSRVTT_full_test/t2v_metrics/R10: 45.819397993311036
 MSRVTT_full_test/t2v_metrics/R50: 78.89632107023411
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.47976588628762
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.765220327278776
 MSRVTT_full_test/v2t_metrics/R1: 12.341137123745819
 MSRVTT_full_test/v2t_metrics/R5: 34.94983277591973
 MSRVTT_full_test/v2t_metrics/R10: 51.103678929765884
 MSRVTT_full_test/v2t_metrics/R50: 83.81270903010034
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.48879598662207
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.03824428809214
 mnt_best       : 25.263879133364796
 not_improved_count: 3
Train Epoch: 20 [1/250 128/32000 (0%)] Loss: 1.34388 (QuantReg: 8.61098) QuantErr: 8.61098 batch_time=38.73826 
Train Epoch: 20 [12/250 1536/32000 (5%)] Loss: 1.34332 (QuantReg: 8.62972) QuantErr: 8.62972 batch_time=0.45981 
Train Epoch: 20 [23/250 2944/32000 (9%)] Loss: 1.19009 (QuantReg: 8.61079) QuantErr: 8.61079 batch_time=0.43019 
Train Epoch: 20 [34/250 4352/32000 (14%)] Loss: 1.12890 (QuantReg: 8.87989) QuantErr: 8.87989 batch_time=0.46080 
Train Epoch: 20 [45/250 5760/32000 (18%)] Loss: 1.29451 (QuantReg: 8.83960) QuantErr: 8.83960 batch_time=0.47703 
Train Epoch: 20 [56/250 7168/32000 (22%)] Loss: 1.19173 (QuantReg: 8.57068) QuantErr: 8.57068 batch_time=0.42986 
Train Epoch: 20 [67/250 8576/32000 (27%)] Loss: 1.10434 (QuantReg: 8.82435) QuantErr: 8.82435 batch_time=0.44866 
Train Epoch: 20 [78/250 9984/32000 (31%)] Loss: 0.99721 (QuantReg: 8.70663) QuantErr: 8.70663 batch_time=0.51108 
Train Epoch: 20 [89/250 11392/32000 (36%)] Loss: 1.51674 (QuantReg: 8.57596) QuantErr: 8.57596 batch_time=0.43851 
Train Epoch: 20 [100/250 12800/32000 (40%)] Loss: 1.13475 (QuantReg: 8.89247) QuantErr: 8.89247 batch_time=0.49444 
Train Epoch: 20 [111/250 14208/32000 (44%)] Loss: 1.01217 (QuantReg: 8.93667) QuantErr: 8.93667 batch_time=0.45684 
Train Epoch: 20 [122/250 15616/32000 (49%)] Loss: 0.99008 (QuantReg: 8.71773) QuantErr: 8.71773 batch_time=0.48611 
Train Epoch: 20 [133/250 17024/32000 (53%)] Loss: 1.22652 (QuantReg: 8.70028) QuantErr: 8.70028 batch_time=0.43523 
Train Epoch: 20 [144/250 18432/32000 (58%)] Loss: 1.36484 (QuantReg: 8.65474) QuantErr: 8.65474 batch_time=0.59023 
Train Epoch: 20 [155/250 19840/32000 (62%)] Loss: 1.34010 (QuantReg: 8.60008) QuantErr: 8.60008 batch_time=0.47122 
Train Epoch: 20 [166/250 21248/32000 (66%)] Loss: 1.50771 (QuantReg: 8.80997) QuantErr: 8.80997 batch_time=0.44607 
Train Epoch: 20 [177/250 22656/32000 (71%)] Loss: 1.15415 (QuantReg: 9.09053) QuantErr: 9.09053 batch_time=0.44771 
Train Epoch: 20 [188/250 24064/32000 (75%)] Loss: 1.08045 (QuantReg: 8.98372) QuantErr: 8.98372 batch_time=0.46430 
Train Epoch: 20 [199/250 25472/32000 (80%)] Loss: 0.93603 (QuantReg: 8.64575) QuantErr: 8.64575 batch_time=0.44990 
Train Epoch: 20 [210/250 26880/32000 (84%)] Loss: 1.50350 (QuantReg: 8.58467) QuantErr: 8.58467 batch_time=0.47171 
Train Epoch: 20 [221/250 28288/32000 (88%)] Loss: 1.72772 (QuantReg: 8.58240) QuantErr: 8.58240 batch_time=0.49726 
Train Epoch: 20 [232/250 29696/32000 (93%)] Loss: 1.14704 (QuantReg: 8.73967) QuantErr: 8.73967 batch_time=0.46651 
Train Epoch: 20 [243/250 31104/32000 (97%)] Loss: 0.86571 (QuantReg: 9.07477) QuantErr: 9.07477 batch_time=0.43825 
Train Epoch: 20 codebook_update_time=0.87163
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch20.pth ...
Done in 4.409s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch20.pth ...
Done in 16.018s
removing stale ckpt [epoch 19] [took 0.19s]
 epoch          : 20
 loss           : 1.1876652874946594
 quant_reg      : 8.734018104553222
 quant_err      : 8.734018104553222
 learning_rate  : 1.8867680126765363e-05
 n_samples      : 640000
 n_steps        : 5000
 MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
 MSRVTT_full_val/t2v_metrics/R5: 68.81287726358148
 MSRVTT_full_val/t2v_metrics/R10: 82.29376257545272
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.364185110663984
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.74539327817189
 MSRVTT_full_val/v2t_metrics/R1: 34.60764587525151
 MSRVTT_full_val/v2t_metrics/R5: 74.04426559356136
 MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.405432595573441
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.523660291370206
 MSRVTT_full_test/t2v_metrics/R1: 10.668896321070234
 MSRVTT_full_test/t2v_metrics/R5: 32.374581939799334
 MSRVTT_full_test/t2v_metrics/R10: 47.759197324414714
 MSRVTT_full_test/t2v_metrics/R50: 79.7324414715719
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 45.0005016722408
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.45619921082064
 MSRVTT_full_test/v2t_metrics/R1: 11.538461538461538
 MSRVTT_full_test/v2t_metrics/R5: 35.819397993311036
 MSRVTT_full_test/v2t_metrics/R10: 51.40468227424749
 MSRVTT_full_test/v2t_metrics/R50: 84.18060200668896
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.17575250836121
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.696376617626743
 mnt_best       : 25.45619921082064
 not_improved_count: 0
Train Epoch: 21 [1/250 128/32000 (0%)] Loss: 1.06724 (QuantReg: 8.66253) QuantErr: 8.66253 batch_time=37.89829 
Train Epoch: 21 [12/250 1536/32000 (5%)] Loss: 1.14951 (QuantReg: 8.79792) QuantErr: 8.79792 batch_time=0.66107 
Train Epoch: 21 [23/250 2944/32000 (9%)] Loss: 1.17306 (QuantReg: 8.59265) QuantErr: 8.59265 batch_time=0.50212 
Train Epoch: 21 [34/250 4352/32000 (14%)] Loss: 1.04097 (QuantReg: 8.88953) QuantErr: 8.88953 batch_time=0.44033 
Train Epoch: 21 [45/250 5760/32000 (18%)] Loss: 0.92189 (QuantReg: 8.76045) QuantErr: 8.76045 batch_time=0.52618 
Train Epoch: 21 [56/250 7168/32000 (22%)] Loss: 0.89675 (QuantReg: 9.06977) QuantErr: 9.06977 batch_time=0.47188 
Train Epoch: 21 [67/250 8576/32000 (27%)] Loss: 1.19142 (QuantReg: 8.62306) QuantErr: 8.62306 batch_time=0.46289 
Train Epoch: 21 [78/250 9984/32000 (31%)] Loss: 1.31588 (QuantReg: 8.63650) QuantErr: 8.63650 batch_time=0.54931 
Train Epoch: 21 [89/250 11392/32000 (36%)] Loss: 1.33157 (QuantReg: 8.77721) QuantErr: 8.77721 batch_time=2.04577 
Train Epoch: 21 [100/250 12800/32000 (40%)] Loss: 1.26091 (QuantReg: 8.65148) QuantErr: 8.65148 batch_time=0.52951 
Train Epoch: 21 [111/250 14208/32000 (44%)] Loss: 1.11795 (QuantReg: 8.63829) QuantErr: 8.63829 batch_time=0.47704 
Train Epoch: 21 [122/250 15616/32000 (49%)] Loss: 1.13604 (QuantReg: 8.85990) QuantErr: 8.85990 batch_time=0.56204 
Train Epoch: 21 [133/250 17024/32000 (53%)] Loss: 0.95814 (QuantReg: 8.59581) QuantErr: 8.59581 batch_time=0.48009 
Train Epoch: 21 [144/250 18432/32000 (58%)] Loss: 1.26534 (QuantReg: 8.91080) QuantErr: 8.91080 batch_time=0.49936 
Train Epoch: 21 [155/250 19840/32000 (62%)] Loss: 1.14262 (QuantReg: 8.71587) QuantErr: 8.71587 batch_time=0.47977 
Train Epoch: 21 [166/250 21248/32000 (66%)] Loss: 1.01128 (QuantReg: 8.96426) QuantErr: 8.96426 batch_time=0.53324 
Train Epoch: 21 [177/250 22656/32000 (71%)] Loss: 1.34304 (QuantReg: 9.05865) QuantErr: 9.05865 batch_time=0.47892 
Train Epoch: 21 [188/250 24064/32000 (75%)] Loss: 1.11779 (QuantReg: 8.64809) QuantErr: 8.64809 batch_time=0.59510 
Train Epoch: 21 [199/250 25472/32000 (80%)] Loss: 1.17650 (QuantReg: 8.84744) QuantErr: 8.84744 batch_time=0.51183 
Train Epoch: 21 [210/250 26880/32000 (84%)] Loss: 1.45764 (QuantReg: 8.72653) QuantErr: 8.72653 batch_time=0.50557 
Train Epoch: 21 [221/250 28288/32000 (88%)] Loss: 1.13030 (QuantReg: 8.97361) QuantErr: 8.97361 batch_time=0.45988 
Train Epoch: 21 [232/250 29696/32000 (93%)] Loss: 1.07544 (QuantReg: 9.03804) QuantErr: 9.03804 batch_time=0.48817 
Train Epoch: 21 [243/250 31104/32000 (97%)] Loss: 0.93699 (QuantReg: 8.70116) QuantErr: 8.70116 batch_time=0.52743 
Train Epoch: 21 codebook_update_time=1.06651
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch21.pth ...
Done in 4.064s
removing stale ckpt [epoch 20] [took 0.00s]
 epoch          : 21
 loss           : 1.1517917494773864
 quant_reg      : 8.768441654205322
 quant_err      : 8.768441654205322
 learning_rate  : 1.7924296120427095e-05
 n_samples      : 672000
 n_steps        : 5250
 MSRVTT_full_val/t2v_metrics/R1: 27.56539235412475
 MSRVTT_full_val/t2v_metrics/R5: 65.3923541247485
 MSRVTT_full_val/t2v_metrics/R10: 78.47082494969818
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.754527162977867
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.10344706180215
 MSRVTT_full_val/v2t_metrics/R1: 32.99798792756539
 MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
 MSRVTT_full_val/v2t_metrics/R10: 82.69617706237425
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.2434607645875255
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.143322272722656
 MSRVTT_full_test/t2v_metrics/R1: 10.033444816053512
 MSRVTT_full_test/t2v_metrics/R5: 31.270903010033443
 MSRVTT_full_test/t2v_metrics/R10: 46.38795986622073
 MSRVTT_full_test/t2v_metrics/R50: 78.42809364548495
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.21622073578595
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.415479275458686
 MSRVTT_full_test/v2t_metrics/R1: 11.638795986622073
 MSRVTT_full_test/v2t_metrics/R5: 35.785953177257525
 MSRVTT_full_test/v2t_metrics/R10: 50.936454849498325
 MSRVTT_full_test/v2t_metrics/R50: 83.47826086956522
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 40.187290969899664
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.683210389727567
 mnt_best       : 25.45619921082064
 not_improved_count: 1
Train Epoch: 22 [1/250 128/32000 (0%)] Loss: 1.32436 (QuantReg: 8.64380) QuantErr: 8.64380 batch_time=38.50231 
Train Epoch: 22 [12/250 1536/32000 (5%)] Loss: 1.18192 (QuantReg: 8.83980) QuantErr: 8.83980 batch_time=3.29557 
Train Epoch: 22 [23/250 2944/32000 (9%)] Loss: 1.09797 (QuantReg: 8.79853) QuantErr: 8.79853 batch_time=0.47797 
Train Epoch: 22 [34/250 4352/32000 (14%)] Loss: 1.09296 (QuantReg: 8.50012) QuantErr: 8.50012 batch_time=0.45329 
Train Epoch: 22 [45/250 5760/32000 (18%)] Loss: 0.99768 (QuantReg: 8.82810) QuantErr: 8.82810 batch_time=0.43299 
Train Epoch: 22 [56/250 7168/32000 (22%)] Loss: 1.07916 (QuantReg: 8.72028) QuantErr: 8.72028 batch_time=0.49105 
Train Epoch: 22 [67/250 8576/32000 (27%)] Loss: 1.20449 (QuantReg: 8.55782) QuantErr: 8.55782 batch_time=4.91675 
Train Epoch: 22 [78/250 9984/32000 (31%)] Loss: 1.16419 (QuantReg: 8.60157) QuantErr: 8.60157 batch_time=0.46682 
Train Epoch: 22 [89/250 11392/32000 (36%)] Loss: 1.34252 (QuantReg: 8.44956) QuantErr: 8.44956 batch_time=0.52698 
Train Epoch: 22 [100/250 12800/32000 (40%)] Loss: 1.28094 (QuantReg: 8.77780) QuantErr: 8.77780 batch_time=0.48009 
Train Epoch: 22 [111/250 14208/32000 (44%)] Loss: 1.37806 (QuantReg: 8.70038) QuantErr: 8.70038 batch_time=0.44740 
Train Epoch: 22 [122/250 15616/32000 (49%)] Loss: 1.09520 (QuantReg: 8.81346) QuantErr: 8.81346 batch_time=0.51889 
Train Epoch: 22 [133/250 17024/32000 (53%)] Loss: 1.18879 (QuantReg: 8.79542) QuantErr: 8.79542 batch_time=0.61461 
Train Epoch: 22 [144/250 18432/32000 (58%)] Loss: 1.30192 (QuantReg: 8.79315) QuantErr: 8.79315 batch_time=0.47324 
Train Epoch: 22 [155/250 19840/32000 (62%)] Loss: 1.12946 (QuantReg: 8.83659) QuantErr: 8.83659 batch_time=0.50512 
Train Epoch: 22 [166/250 21248/32000 (66%)] Loss: 1.31255 (QuantReg: 8.71525) QuantErr: 8.71525 batch_time=0.45192 
Train Epoch: 22 [177/250 22656/32000 (71%)] Loss: 1.30284 (QuantReg: 8.77830) QuantErr: 8.77830 batch_time=0.45295 
Train Epoch: 22 [188/250 24064/32000 (75%)] Loss: 1.16064 (QuantReg: 8.82962) QuantErr: 8.82962 batch_time=0.43542 
Train Epoch: 22 [199/250 25472/32000 (80%)] Loss: 1.14291 (QuantReg: 8.63910) QuantErr: 8.63910 batch_time=0.90787 
Train Epoch: 22 [210/250 26880/32000 (84%)] Loss: 1.18419 (QuantReg: 8.78818) QuantErr: 8.78818 batch_time=0.50779 
Train Epoch: 22 [221/250 28288/32000 (88%)] Loss: 1.21537 (QuantReg: 8.77147) QuantErr: 8.77147 batch_time=0.45931 
Train Epoch: 22 [232/250 29696/32000 (93%)] Loss: 1.12447 (QuantReg: 8.72897) QuantErr: 8.72897 batch_time=0.50624 
Train Epoch: 22 [243/250 31104/32000 (97%)] Loss: 0.89014 (QuantReg: 8.90946) QuantErr: 8.90946 batch_time=0.69266 
Train Epoch: 22 codebook_update_time=0.92305
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch22.pth ...
Done in 4.003s
removing stale ckpt [epoch 21] [took 0.05s]
 epoch          : 22
 loss           : 1.1259275488853455
 quant_reg      : 8.791173404693604
 quant_err      : 8.791173404693604
 learning_rate  : 1.702808131440574e-05
 n_samples      : 704000
 n_steps        : 5500
 MSRVTT_full_val/t2v_metrics/R1: 27.16297786720322
 MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
 MSRVTT_full_val/t2v_metrics/R10: 78.26961770623743
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.285714285714286
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 51.6978214149819
 MSRVTT_full_val/v2t_metrics/R1: 31.99195171026157
 MSRVTT_full_val/v2t_metrics/R5: 72.23340040241449
 MSRVTT_full_val/v2t_metrics/R10: 82.29376257545272
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.309859154929577
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.506275183527734
 MSRVTT_full_test/t2v_metrics/R1: 10.602006688963211
 MSRVTT_full_test/t2v_metrics/R5: 32.709030100334445
 MSRVTT_full_test/t2v_metrics/R10: 46.05351170568562
 MSRVTT_full_test/t2v_metrics/R50: 78.66220735785953
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.16153846153846
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.182924538110523
 MSRVTT_full_test/v2t_metrics/R1: 12.441471571906355
 MSRVTT_full_test/v2t_metrics/R5: 35.585284280936456
 MSRVTT_full_test/v2t_metrics/R10: 51.27090301003344
 MSRVTT_full_test/v2t_metrics/R50: 83.54515050167224
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.77709030100335
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.31420589958766
 mnt_best       : 25.45619921082064
 not_improved_count: 2
Train Epoch: 23 [1/250 128/32000 (0%)] Loss: 1.20618 (QuantReg: 8.58880) QuantErr: 8.58880 batch_time=43.70552 
Train Epoch: 23 [12/250 1536/32000 (5%)] Loss: 1.26671 (QuantReg: 8.89089) QuantErr: 8.89089 batch_time=0.46530 
Train Epoch: 23 [23/250 2944/32000 (9%)] Loss: 0.86851 (QuantReg: 8.85663) QuantErr: 8.85663 batch_time=0.44389 
Train Epoch: 23 [34/250 4352/32000 (14%)] Loss: 1.18971 (QuantReg: 8.74155) QuantErr: 8.74155 batch_time=0.44398 
Train Epoch: 23 [45/250 5760/32000 (18%)] Loss: 1.11703 (QuantReg: 8.84905) QuantErr: 8.84905 batch_time=0.43521 
Train Epoch: 23 [56/250 7168/32000 (22%)] Loss: 1.13372 (QuantReg: 8.84206) QuantErr: 8.84206 batch_time=0.43052 
Train Epoch: 23 [67/250 8576/32000 (27%)] Loss: 1.01567 (QuantReg: 8.85686) QuantErr: 8.85686 batch_time=0.44085 
Train Epoch: 23 [78/250 9984/32000 (31%)] Loss: 1.23526 (QuantReg: 8.81939) QuantErr: 8.81939 batch_time=0.45282 
Train Epoch: 23 [89/250 11392/32000 (36%)] Loss: 1.20311 (QuantReg: 8.60166) QuantErr: 8.60166 batch_time=0.43319 
Train Epoch: 23 [100/250 12800/32000 (40%)] Loss: 0.94926 (QuantReg: 8.86362) QuantErr: 8.86362 batch_time=0.47429 
Train Epoch: 23 [111/250 14208/32000 (44%)] Loss: 1.33527 (QuantReg: 8.83322) QuantErr: 8.83322 batch_time=0.42632 
Train Epoch: 23 [122/250 15616/32000 (49%)] Loss: 1.12019 (QuantReg: 8.61791) QuantErr: 8.61791 batch_time=0.43304 
Train Epoch: 23 [133/250 17024/32000 (53%)] Loss: 1.05633 (QuantReg: 8.78428) QuantErr: 8.78428 batch_time=0.44142 
Train Epoch: 23 [144/250 18432/32000 (58%)] Loss: 1.27746 (QuantReg: 8.88458) QuantErr: 8.88458 batch_time=0.43566 
Train Epoch: 23 [155/250 19840/32000 (62%)] Loss: 1.38506 (QuantReg: 8.81744) QuantErr: 8.81744 batch_time=0.44151 
Train Epoch: 23 [166/250 21248/32000 (66%)] Loss: 1.12515 (QuantReg: 8.59926) QuantErr: 8.59926 batch_time=0.43758 
Train Epoch: 23 [177/250 22656/32000 (71%)] Loss: 0.87696 (QuantReg: 9.05049) QuantErr: 9.05049 batch_time=0.44932 
Train Epoch: 23 [188/250 24064/32000 (75%)] Loss: 1.14591 (QuantReg: 8.83354) QuantErr: 8.83354 batch_time=0.44618 
Train Epoch: 23 [199/250 25472/32000 (80%)] Loss: 0.89315 (QuantReg: 9.10403) QuantErr: 9.10403 batch_time=0.46553 
Train Epoch: 23 [210/250 26880/32000 (84%)] Loss: 0.92294 (QuantReg: 8.94475) QuantErr: 8.94475 batch_time=0.48957 
Train Epoch: 23 [221/250 28288/32000 (88%)] Loss: 0.88220 (QuantReg: 9.00857) QuantErr: 9.00857 batch_time=0.48868 
Train Epoch: 23 [232/250 29696/32000 (93%)] Loss: 1.04354 (QuantReg: 8.91830) QuantErr: 8.91830 batch_time=0.45500 
Train Epoch: 23 [243/250 31104/32000 (97%)] Loss: 1.29254 (QuantReg: 8.50315) QuantErr: 8.50315 batch_time=0.44195 
Train Epoch: 23 codebook_update_time=0.87110
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch23.pth ...
Done in 3.797s
removing stale ckpt [epoch 22] [took 0.00s]
 epoch          : 23
 loss           : 1.1407472681999207
 quant_reg      : 8.802459926605225
 quant_err      : 8.802459926605225
 learning_rate  : 1.6176677248685452e-05
 n_samples      : 736000
 n_steps        : 5750
 MSRVTT_full_val/t2v_metrics/R1: 27.56539235412475
 MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
 MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.279678068410464
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.97267144901895
 MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
 MSRVTT_full_val/v2t_metrics/R5: 72.83702213279678
 MSRVTT_full_val/v2t_metrics/R10: 83.09859154929578
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.23943661971831
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.618297255770564
 MSRVTT_full_test/t2v_metrics/R1: 10.869565217391305
 MSRVTT_full_test/t2v_metrics/R5: 32.441471571906355
 MSRVTT_full_test/t2v_metrics/R10: 46.52173913043478
 MSRVTT_full_test/t2v_metrics/R50: 79.76588628762542
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.269732441471575
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.409115853803435
 MSRVTT_full_test/v2t_metrics/R1: 12.508361204013378
 MSRVTT_full_test/v2t_metrics/R5: 35.953177257525084
 MSRVTT_full_test/v2t_metrics/R10: 51.83946488294314
 MSRVTT_full_test/v2t_metrics/R50: 84.41471571906355
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.59598662207358
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.56709417853361
 mnt_best       : 25.45619921082064
 not_improved_count: 3
Train Epoch: 24 [1/250 128/32000 (0%)] Loss: 1.39642 (QuantReg: 8.62329) QuantErr: 8.62329 batch_time=34.36365 
Train Epoch: 24 [12/250 1536/32000 (5%)] Loss: 1.02464 (QuantReg: 8.85102) QuantErr: 8.85102 batch_time=0.43786 
Train Epoch: 24 [23/250 2944/32000 (9%)] Loss: 0.97235 (QuantReg: 8.83476) QuantErr: 8.83476 batch_time=0.45862 
Train Epoch: 24 [34/250 4352/32000 (14%)] Loss: 1.26451 (QuantReg: 8.93535) QuantErr: 8.93535 batch_time=0.43046 
Train Epoch: 24 [45/250 5760/32000 (18%)] Loss: 1.31100 (QuantReg: 8.55857) QuantErr: 8.55857 batch_time=0.43857 
Train Epoch: 24 [56/250 7168/32000 (22%)] Loss: 0.76260 (QuantReg: 8.88997) QuantErr: 8.88997 batch_time=0.45311 
Train Epoch: 24 [67/250 8576/32000 (27%)] Loss: 0.78419 (QuantReg: 8.93751) QuantErr: 8.93751 batch_time=0.45005 
Train Epoch: 24 [78/250 9984/32000 (31%)] Loss: 1.07544 (QuantReg: 8.70848) QuantErr: 8.70848 batch_time=0.44173 
Train Epoch: 24 [89/250 11392/32000 (36%)] Loss: 0.97730 (QuantReg: 8.82807) QuantErr: 8.82807 batch_time=0.45759 
Train Epoch: 24 [100/250 12800/32000 (40%)] Loss: 1.09581 (QuantReg: 8.85731) QuantErr: 8.85731 batch_time=0.45581 
Train Epoch: 24 [111/250 14208/32000 (44%)] Loss: 1.23104 (QuantReg: 8.74799) QuantErr: 8.74799 batch_time=0.45943 
Train Epoch: 24 [122/250 15616/32000 (49%)] Loss: 1.27707 (QuantReg: 8.69029) QuantErr: 8.69029 batch_time=0.44839 
Train Epoch: 24 [133/250 17024/32000 (53%)] Loss: 0.95024 (QuantReg: 8.92227) QuantErr: 8.92227 batch_time=0.44500 
Train Epoch: 24 [144/250 18432/32000 (58%)] Loss: 1.41235 (QuantReg: 9.07333) QuantErr: 9.07333 batch_time=0.51099 
Train Epoch: 24 [155/250 19840/32000 (62%)] Loss: 1.17872 (QuantReg: 8.59931) QuantErr: 8.59931 batch_time=0.43698 
Train Epoch: 24 [166/250 21248/32000 (66%)] Loss: 0.91190 (QuantReg: 8.99865) QuantErr: 8.99865 batch_time=1.93524 
Train Epoch: 24 [177/250 22656/32000 (71%)] Loss: 0.93150 (QuantReg: 9.07625) QuantErr: 9.07625 batch_time=0.53562 
Train Epoch: 24 [188/250 24064/32000 (75%)] Loss: 0.95973 (QuantReg: 8.84778) QuantErr: 8.84778 batch_time=0.52179 
Train Epoch: 24 [199/250 25472/32000 (80%)] Loss: 1.11456 (QuantReg: 8.78374) QuantErr: 8.78374 batch_time=0.47737 
Train Epoch: 24 [210/250 26880/32000 (84%)] Loss: 0.90349 (QuantReg: 9.02382) QuantErr: 9.02382 batch_time=0.45104 
Train Epoch: 24 [221/250 28288/32000 (88%)] Loss: 1.29609 (QuantReg: 8.68662) QuantErr: 8.68662 batch_time=0.45366 
Train Epoch: 24 [232/250 29696/32000 (93%)] Loss: 1.29056 (QuantReg: 8.85795) QuantErr: 8.85795 batch_time=0.44834 
Train Epoch: 24 [243/250 31104/32000 (97%)] Loss: 1.12383 (QuantReg: 8.74538) QuantErr: 8.74538 batch_time=0.45108 
Train Epoch: 24 codebook_update_time=0.94355
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch24.pth ...
Done in 18.237s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch24.pth ...
Done in 22.000s
removing stale ckpt [epoch 23] [took 0.00s]
 epoch          : 24
 loss           : 1.0928616180419921
 quant_reg      : 8.850122539520264
 quant_err      : 8.850122539520264
 learning_rate  : 1.5367843386251178e-05
 n_samples      : 768000
 n_steps        : 6000
 MSRVTT_full_val/t2v_metrics/R1: 30.18108651911469
 MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
 MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.154929577464788
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.360710056638645
 MSRVTT_full_val/v2t_metrics/R1: 33.199195171026155
 MSRVTT_full_val/v2t_metrics/R5: 72.23340040241449
 MSRVTT_full_val/v2t_metrics/R10: 82.69617706237425
 MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.8199195171026155
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.31545692178897
 MSRVTT_full_test/t2v_metrics/R1: 10.80267558528428
 MSRVTT_full_test/t2v_metrics/R5: 32.441471571906355
 MSRVTT_full_test/t2v_metrics/R10: 47.22408026755853
 MSRVTT_full_test/t2v_metrics/R50: 78.96321070234114
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.2561872909699
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.48385524127274
 MSRVTT_full_test/v2t_metrics/R1: 12.408026755852843
 MSRVTT_full_test/v2t_metrics/R5: 36.2876254180602
 MSRVTT_full_test/v2t_metrics/R10: 51.17056856187291
 MSRVTT_full_test/v2t_metrics/R50: 83.87959866220736
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.95301003344482
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.45512544521861
 mnt_best       : 25.48385524127274
 not_improved_count: 0
Train Epoch: 25 [1/250 128/32000 (0%)] Loss: 1.29593 (QuantReg: 8.75605) QuantErr: 8.75605 batch_time=35.42604 
Train Epoch: 25 [12/250 1536/32000 (5%)] Loss: 1.03852 (QuantReg: 8.82737) QuantErr: 8.82737 batch_time=0.55978 
Train Epoch: 25 [23/250 2944/32000 (9%)] Loss: 0.94946 (QuantReg: 9.06319) QuantErr: 9.06319 batch_time=0.45257 
Train Epoch: 25 [34/250 4352/32000 (14%)] Loss: 1.25143 (QuantReg: 8.68715) QuantErr: 8.68715 batch_time=0.54930 
Train Epoch: 25 [45/250 5760/32000 (18%)] Loss: 1.05250 (QuantReg: 8.84033) QuantErr: 8.84033 batch_time=0.44233 
Train Epoch: 25 [56/250 7168/32000 (22%)] Loss: 1.04618 (QuantReg: 8.86853) QuantErr: 8.86853 batch_time=0.48362 
Train Epoch: 25 [67/250 8576/32000 (27%)] Loss: 1.18400 (QuantReg: 8.67106) QuantErr: 8.67106 batch_time=0.74133 
Train Epoch: 25 [78/250 9984/32000 (31%)] Loss: 1.12379 (QuantReg: 8.94420) QuantErr: 8.94420 batch_time=0.45250 
Train Epoch: 25 [89/250 11392/32000 (36%)] Loss: 0.89822 (QuantReg: 8.96597) QuantErr: 8.96597 batch_time=0.56924 
Train Epoch: 25 [100/250 12800/32000 (40%)] Loss: 1.19984 (QuantReg: 8.75847) QuantErr: 8.75847 batch_time=0.46526 
Train Epoch: 25 [111/250 14208/32000 (44%)] Loss: 1.27418 (QuantReg: 8.78703) QuantErr: 8.78703 batch_time=0.43322 
Train Epoch: 25 [122/250 15616/32000 (49%)] Loss: 0.86926 (QuantReg: 8.83217) QuantErr: 8.83217 batch_time=0.44405 
Train Epoch: 25 [133/250 17024/32000 (53%)] Loss: 1.04932 (QuantReg: 8.88610) QuantErr: 8.88610 batch_time=0.44997 
Train Epoch: 25 [144/250 18432/32000 (58%)] Loss: 1.07118 (QuantReg: 8.88633) QuantErr: 8.88633 batch_time=0.46733 
Train Epoch: 25 [155/250 19840/32000 (62%)] Loss: 1.21453 (QuantReg: 8.84540) QuantErr: 8.84540 batch_time=0.42466 
Train Epoch: 25 [166/250 21248/32000 (66%)] Loss: 1.33710 (QuantReg: 8.73003) QuantErr: 8.73003 batch_time=0.49726 
Train Epoch: 25 [177/250 22656/32000 (71%)] Loss: 0.98255 (QuantReg: 8.98432) QuantErr: 8.98432 batch_time=0.49583 
Train Epoch: 25 [188/250 24064/32000 (75%)] Loss: 1.06466 (QuantReg: 9.05955) QuantErr: 9.05955 batch_time=0.43158 
Train Epoch: 25 [199/250 25472/32000 (80%)] Loss: 0.98972 (QuantReg: 8.91266) QuantErr: 8.91266 batch_time=0.53708 
Train Epoch: 25 [210/250 26880/32000 (84%)] Loss: 1.13158 (QuantReg: 8.83899) QuantErr: 8.83899 batch_time=0.48045 
Train Epoch: 25 [221/250 28288/32000 (88%)] Loss: 1.14655 (QuantReg: 8.84596) QuantErr: 8.84596 batch_time=0.46460 
Train Epoch: 25 [232/250 29696/32000 (93%)] Loss: 1.09670 (QuantReg: 8.87621) QuantErr: 8.87621 batch_time=0.49758 
Train Epoch: 25 [243/250 31104/32000 (97%)] Loss: 1.09156 (QuantReg: 8.72426) QuantErr: 8.72426 batch_time=0.45248 
Train Epoch: 25 codebook_update_time=1.00202
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch25.pth ...
Done in 4.271s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch25.pth ...
Done in 8.386s
removing stale ckpt [epoch 24] [took 0.00s]
 epoch          : 25
 loss           : 1.1027993986606597
 quant_reg      : 8.858145931243897
 quant_err      : 8.858145931243897
 learning_rate  : 1.4599451216938618e-05
 n_samples      : 800000
 n_steps        : 6250
 MSRVTT_full_val/t2v_metrics/R1: 30.985915492957748
 MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
 MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.655935613682093
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.46216531608692
 MSRVTT_full_val/v2t_metrics/R1: 34.00402414486921
 MSRVTT_full_val/v2t_metrics/R5: 72.43460764587525
 MSRVTT_full_val/v2t_metrics/R10: 84.90945674044265
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.350100603621731
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.35775907381057
 MSRVTT_full_test/t2v_metrics/R1: 11.404682274247492
 MSRVTT_full_test/t2v_metrics/R5: 32.709030100334445
 MSRVTT_full_test/t2v_metrics/R10: 48.09364548494983
 MSRVTT_full_test/t2v_metrics/R50: 80.33444816053512
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 46.59749163879599
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.178585994987607
 MSRVTT_full_test/v2t_metrics/R1: 12.54180602006689
 MSRVTT_full_test/v2t_metrics/R5: 37.391304347826086
 MSRVTT_full_test/v2t_metrics/R10: 52.30769230769231
 MSRVTT_full_test/v2t_metrics/R50: 83.84615384615384
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.77792642140468
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.05574965791558
 mnt_best       : 26.178585994987607
 not_improved_count: 0
Train Epoch: 26 [1/250 128/32000 (0%)] Loss: 1.16619 (QuantReg: 8.77966) QuantErr: 8.77966 batch_time=35.81105 
Train Epoch: 26 [12/250 1536/32000 (5%)] Loss: 1.19158 (QuantReg: 8.58572) QuantErr: 8.58572 batch_time=0.54844 
Train Epoch: 26 [23/250 2944/32000 (9%)] Loss: 0.74167 (QuantReg: 8.88691) QuantErr: 8.88691 batch_time=0.48809 
Train Epoch: 26 [34/250 4352/32000 (14%)] Loss: 0.96498 (QuantReg: 8.79782) QuantErr: 8.79782 batch_time=0.50308 
Train Epoch: 26 [45/250 5760/32000 (18%)] Loss: 0.98253 (QuantReg: 8.84963) QuantErr: 8.84963 batch_time=0.51000 
Train Epoch: 26 [56/250 7168/32000 (22%)] Loss: 1.14858 (QuantReg: 8.87627) QuantErr: 8.87627 batch_time=0.50381 
Train Epoch: 26 [67/250 8576/32000 (27%)] Loss: 1.00925 (QuantReg: 9.05587) QuantErr: 9.05587 batch_time=0.52190 
Train Epoch: 26 [78/250 9984/32000 (31%)] Loss: 1.15017 (QuantReg: 9.02397) QuantErr: 9.02397 batch_time=2.21404 
Train Epoch: 26 [89/250 11392/32000 (36%)] Loss: 1.13100 (QuantReg: 8.79889) QuantErr: 8.79889 batch_time=0.45615 
Train Epoch: 26 [100/250 12800/32000 (40%)] Loss: 0.96029 (QuantReg: 8.75533) QuantErr: 8.75533 batch_time=0.46009 
Train Epoch: 26 [111/250 14208/32000 (44%)] Loss: 0.89334 (QuantReg: 8.95820) QuantErr: 8.95820 batch_time=0.55005 
Train Epoch: 26 [122/250 15616/32000 (49%)] Loss: 0.89973 (QuantReg: 8.86789) QuantErr: 8.86789 batch_time=0.51547 
Train Epoch: 26 [133/250 17024/32000 (53%)] Loss: 0.92754 (QuantReg: 8.62040) QuantErr: 8.62040 batch_time=1.95307 
Train Epoch: 26 [144/250 18432/32000 (58%)] Loss: 1.05968 (QuantReg: 8.81762) QuantErr: 8.81762 batch_time=1.57460 
Train Epoch: 26 [155/250 19840/32000 (62%)] Loss: 1.26242 (QuantReg: 8.70209) QuantErr: 8.70209 batch_time=0.45873 
Train Epoch: 26 [166/250 21248/32000 (66%)] Loss: 1.01721 (QuantReg: 8.92895) QuantErr: 8.92895 batch_time=0.44524 
Train Epoch: 26 [177/250 22656/32000 (71%)] Loss: 1.01856 (QuantReg: 8.78885) QuantErr: 8.78885 batch_time=0.50426 
Train Epoch: 26 [188/250 24064/32000 (75%)] Loss: 0.92553 (QuantReg: 8.94683) QuantErr: 8.94683 batch_time=0.56671 
Train Epoch: 26 [199/250 25472/32000 (80%)] Loss: 1.05084 (QuantReg: 9.01697) QuantErr: 9.01697 batch_time=0.43704 
Train Epoch: 26 [210/250 26880/32000 (84%)] Loss: 0.87542 (QuantReg: 8.95828) QuantErr: 8.95828 batch_time=0.45596 
Train Epoch: 26 [221/250 28288/32000 (88%)] Loss: 1.75168 (QuantReg: 8.82618) QuantErr: 8.82618 batch_time=0.56051 
Train Epoch: 26 [232/250 29696/32000 (93%)] Loss: 1.08812 (QuantReg: 8.93725) QuantErr: 8.93725 batch_time=0.53925 
Train Epoch: 26 [243/250 31104/32000 (97%)] Loss: 0.83742 (QuantReg: 8.93654) QuantErr: 8.93654 batch_time=0.44725 
Train Epoch: 26 codebook_update_time=0.91878
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch26.pth ...
Done in 21.672s
removing stale ckpt [epoch 25] [took 0.00s]
 epoch          : 26
 loss           : 1.0908293571472167
 quant_reg      : 8.890600914001466
 quant_err      : 8.890600914001466
 learning_rate  : 1.3869478656091687e-05
 n_samples      : 832000
 n_steps        : 6500
 MSRVTT_full_val/t2v_metrics/R1: 31.388329979879277
 MSRVTT_full_val/t2v_metrics/R5: 67.80684104627767
 MSRVTT_full_val/t2v_metrics/R10: 79.87927565392354
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.865191146881287
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.39772710842229
 MSRVTT_full_val/v2t_metrics/R1: 33.199195171026155
 MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
 MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.150905432595573
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.9626139019553
 MSRVTT_full_test/t2v_metrics/R1: 11.103678929765886
 MSRVTT_full_test/t2v_metrics/R5: 32.408026755852845
 MSRVTT_full_test/t2v_metrics/R10: 47.357859531772576
 MSRVTT_full_test/t2v_metrics/R50: 78.89632107023411
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 51.071571906354514
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.733795755050103
 MSRVTT_full_test/v2t_metrics/R1: 12.54180602006689
 MSRVTT_full_test/v2t_metrics/R5: 37.290969899665555
 MSRVTT_full_test/v2t_metrics/R10: 52.608695652173914
 MSRVTT_full_test/v2t_metrics/R50: 84.44816053511705
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.7309364548495
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.085314504056832
 mnt_best       : 26.178585994987607
 not_improved_count: 1
Train Epoch: 27 [1/250 128/32000 (0%)] Loss: 0.81776 (QuantReg: 9.04644) QuantErr: 9.04644 batch_time=34.52051 
Train Epoch: 27 [12/250 1536/32000 (5%)] Loss: 1.32900 (QuantReg: 8.99888) QuantErr: 8.99888 batch_time=0.48467 
Train Epoch: 27 [23/250 2944/32000 (9%)] Loss: 1.11502 (QuantReg: 9.03729) QuantErr: 9.03729 batch_time=0.45169 
Train Epoch: 27 [34/250 4352/32000 (14%)] Loss: 1.03282 (QuantReg: 9.01423) QuantErr: 9.01423 batch_time=0.46305 
Train Epoch: 27 [45/250 5760/32000 (18%)] Loss: 0.99952 (QuantReg: 8.88614) QuantErr: 8.88614 batch_time=0.44006 
Train Epoch: 27 [56/250 7168/32000 (22%)] Loss: 0.81537 (QuantReg: 9.10614) QuantErr: 9.10614 batch_time=0.44507 
Train Epoch: 27 [67/250 8576/32000 (27%)] Loss: 0.91585 (QuantReg: 8.91834) QuantErr: 8.91834 batch_time=0.49289 
Train Epoch: 27 [78/250 9984/32000 (31%)] Loss: 0.96931 (QuantReg: 8.81390) QuantErr: 8.81390 batch_time=0.47887 
Train Epoch: 27 [89/250 11392/32000 (36%)] Loss: 1.01549 (QuantReg: 9.00859) QuantErr: 9.00859 batch_time=0.43813 
Train Epoch: 27 [100/250 12800/32000 (40%)] Loss: 1.11062 (QuantReg: 8.78286) QuantErr: 8.78286 batch_time=0.44939 
Train Epoch: 27 [111/250 14208/32000 (44%)] Loss: 0.95464 (QuantReg: 9.04274) QuantErr: 9.04274 batch_time=0.44864 
Train Epoch: 27 [122/250 15616/32000 (49%)] Loss: 1.06851 (QuantReg: 8.87391) QuantErr: 8.87391 batch_time=0.45306 
Train Epoch: 27 [133/250 17024/32000 (53%)] Loss: 1.15192 (QuantReg: 8.90552) QuantErr: 8.90552 batch_time=0.44605 
Train Epoch: 27 [144/250 18432/32000 (58%)] Loss: 0.92587 (QuantReg: 8.87825) QuantErr: 8.87825 batch_time=0.45964 
Train Epoch: 27 [155/250 19840/32000 (62%)] Loss: 0.95738 (QuantReg: 9.08272) QuantErr: 9.08272 batch_time=0.48687 
Train Epoch: 27 [166/250 21248/32000 (66%)] Loss: 1.06204 (QuantReg: 8.91218) QuantErr: 8.91218 batch_time=0.44387 
Train Epoch: 27 [177/250 22656/32000 (71%)] Loss: 1.22036 (QuantReg: 8.82531) QuantErr: 8.82531 batch_time=0.46161 
Train Epoch: 27 [188/250 24064/32000 (75%)] Loss: 1.00924 (QuantReg: 8.82636) QuantErr: 8.82636 batch_time=0.43664 
Train Epoch: 27 [199/250 25472/32000 (80%)] Loss: 1.23956 (QuantReg: 8.96127) QuantErr: 8.96127 batch_time=0.44523 
Train Epoch: 27 [210/250 26880/32000 (84%)] Loss: 0.92837 (QuantReg: 9.22292) QuantErr: 9.22292 batch_time=0.47425 
Train Epoch: 27 [221/250 28288/32000 (88%)] Loss: 0.98401 (QuantReg: 8.88604) QuantErr: 8.88604 batch_time=0.44504 
Train Epoch: 27 [232/250 29696/32000 (93%)] Loss: 1.04949 (QuantReg: 8.88430) QuantErr: 8.88430 batch_time=0.45202 
Train Epoch: 27 [243/250 31104/32000 (97%)] Loss: 1.03218 (QuantReg: 8.89425) QuantErr: 8.89425 batch_time=0.46797 
Train Epoch: 27 codebook_update_time=1.01853
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch27.pth ...
Done in 6.626s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch27.pth ...
Done in 10.552s
removing stale ckpt [epoch 26] [took 0.00s]
 epoch          : 27
 loss           : 1.0568239960670471
 quant_reg      : 8.893590213775635
 quant_err      : 8.893590213775635
 learning_rate  : 1.3176004723287102e-05
 n_samples      : 864000
 n_steps        : 6750
 MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
 MSRVTT_full_val/t2v_metrics/R5: 69.01408450704226
 MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.45271629778672
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.524616092368234
 MSRVTT_full_val/v2t_metrics/R1: 35.2112676056338
 MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
 MSRVTT_full_val/v2t_metrics/R10: 85.31187122736418
 MSRVTT_full_val/v2t_metrics/R50: 98.79275653923541
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.487927565392354
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.698620512507745
 MSRVTT_full_test/t2v_metrics/R1: 11.237458193979933
 MSRVTT_full_test/t2v_metrics/R5: 34.414715719063544
 MSRVTT_full_test/t2v_metrics/R10: 49.096989966555185
 MSRVTT_full_test/t2v_metrics/R50: 79.56521739130434
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.10401337792642
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.678150256479153
 MSRVTT_full_test/v2t_metrics/R1: 13.578595317725753
 MSRVTT_full_test/v2t_metrics/R5: 38.22742474916388
 MSRVTT_full_test/v2t_metrics/R10: 53.61204013377927
 MSRVTT_full_test/v2t_metrics/R50: 84.61538461538461
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.1943143812709
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.303822053466426
 mnt_best       : 26.678150256479153
 not_improved_count: 0
Train Epoch: 28 [1/250 128/32000 (0%)] Loss: 0.87952 (QuantReg: 8.83198) QuantErr: 8.83198 batch_time=45.27622 
Train Epoch: 28 [12/250 1536/32000 (5%)] Loss: 1.26418 (QuantReg: 8.93548) QuantErr: 8.93548 batch_time=0.47529 
Train Epoch: 28 [23/250 2944/32000 (9%)] Loss: 1.06810 (QuantReg: 8.78128) QuantErr: 8.78128 batch_time=0.53931 
Train Epoch: 28 [34/250 4352/32000 (14%)] Loss: 1.10867 (QuantReg: 9.07847) QuantErr: 9.07847 batch_time=0.46840 
Train Epoch: 28 [45/250 5760/32000 (18%)] Loss: 1.36002 (QuantReg: 8.81629) QuantErr: 8.81629 batch_time=0.50267 
Train Epoch: 28 [56/250 7168/32000 (22%)] Loss: 1.06244 (QuantReg: 9.04755) QuantErr: 9.04755 batch_time=0.48320 
Train Epoch: 28 [67/250 8576/32000 (27%)] Loss: 1.36853 (QuantReg: 8.91874) QuantErr: 8.91874 batch_time=0.46738 
Train Epoch: 28 [78/250 9984/32000 (31%)] Loss: 0.88123 (QuantReg: 9.12968) QuantErr: 9.12968 batch_time=0.68703 
Train Epoch: 28 [89/250 11392/32000 (36%)] Loss: 1.14020 (QuantReg: 8.86441) QuantErr: 8.86441 batch_time=0.46431 
Train Epoch: 28 [100/250 12800/32000 (40%)] Loss: 1.24710 (QuantReg: 8.91610) QuantErr: 8.91610 batch_time=0.48436 
Train Epoch: 28 [111/250 14208/32000 (44%)] Loss: 1.00694 (QuantReg: 8.90620) QuantErr: 8.90620 batch_time=0.45099 
Train Epoch: 28 [122/250 15616/32000 (49%)] Loss: 1.06639 (QuantReg: 9.02200) QuantErr: 9.02200 batch_time=0.45642 
Train Epoch: 28 [133/250 17024/32000 (53%)] Loss: 1.18633 (QuantReg: 9.06215) QuantErr: 9.06215 batch_time=0.57464 
Train Epoch: 28 [144/250 18432/32000 (58%)] Loss: 1.23425 (QuantReg: 8.73286) QuantErr: 8.73286 batch_time=0.45544 
Train Epoch: 28 [155/250 19840/32000 (62%)] Loss: 1.25968 (QuantReg: 8.85326) QuantErr: 8.85326 batch_time=0.49745 
Train Epoch: 28 [166/250 21248/32000 (66%)] Loss: 0.81510 (QuantReg: 8.85935) QuantErr: 8.85935 batch_time=1.06492 
Train Epoch: 28 [177/250 22656/32000 (71%)] Loss: 1.29031 (QuantReg: 8.91745) QuantErr: 8.91745 batch_time=0.45719 
Train Epoch: 28 [188/250 24064/32000 (75%)] Loss: 1.08629 (QuantReg: 8.77507) QuantErr: 8.77507 batch_time=0.43196 
Train Epoch: 28 [199/250 25472/32000 (80%)] Loss: 0.90856 (QuantReg: 8.96291) QuantErr: 8.96291 batch_time=0.49149 
Train Epoch: 28 [210/250 26880/32000 (84%)] Loss: 1.04390 (QuantReg: 9.07114) QuantErr: 9.07114 batch_time=0.53547 
Train Epoch: 28 [221/250 28288/32000 (88%)] Loss: 1.04211 (QuantReg: 8.87506) QuantErr: 8.87506 batch_time=0.50210 
Train Epoch: 28 [232/250 29696/32000 (93%)] Loss: 1.31171 (QuantReg: 8.76566) QuantErr: 8.76566 batch_time=0.49712 
Train Epoch: 28 [243/250 31104/32000 (97%)] Loss: 0.99993 (QuantReg: 9.00876) QuantErr: 9.00876 batch_time=0.43293 
Train Epoch: 28 codebook_update_time=0.96690
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch28.pth ...
Done in 3.880s
removing stale ckpt [epoch 27] [took 0.00s]
 epoch          : 28
 loss           : 1.0553498713970184
 quant_reg      : 8.93080961227417
 quant_err      : 8.93080961227417
 learning_rate  : 1.2517204487122746e-05
 n_samples      : 896000
 n_steps        : 7000
 MSRVTT_full_val/t2v_metrics/R1: 30.58350100603622
 MSRVTT_full_val/t2v_metrics/R5: 67.6056338028169
 MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.710261569416499
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.09513489629973
 MSRVTT_full_val/v2t_metrics/R1: 34.80885311871227
 MSRVTT_full_val/v2t_metrics/R5: 75.85513078470825
 MSRVTT_full_val/v2t_metrics/R10: 85.31187122736418
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.629778672032193
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.845441665030094
 MSRVTT_full_test/t2v_metrics/R1: 11.103678929765886
 MSRVTT_full_test/t2v_metrics/R5: 32.675585284280935
 MSRVTT_full_test/t2v_metrics/R10: 47.69230769230769
 MSRVTT_full_test/t2v_metrics/R50: 79.33110367892976
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.885451505016725
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.86502302311195
 MSRVTT_full_test/v2t_metrics/R1: 13.444816053511706
 MSRVTT_full_test/v2t_metrics/R5: 39.096989966555185
 MSRVTT_full_test/v2t_metrics/R10: 52.90969899665552
 MSRVTT_full_test/v2t_metrics/R50: 84.44816053511705
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.99280936454849
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.297804538338887
 mnt_best       : 26.678150256479153
 not_improved_count: 1
Train Epoch: 29 [1/250 128/32000 (0%)] Loss: 1.12122 (QuantReg: 8.95524) QuantErr: 8.95524 batch_time=39.39948 
Train Epoch: 29 [12/250 1536/32000 (5%)] Loss: 1.17405 (QuantReg: 8.95604) QuantErr: 8.95604 batch_time=0.43891 
Train Epoch: 29 [23/250 2944/32000 (9%)] Loss: 1.24261 (QuantReg: 8.81756) QuantErr: 8.81756 batch_time=0.43724 
Train Epoch: 29 [34/250 4352/32000 (14%)] Loss: 1.10792 (QuantReg: 8.81258) QuantErr: 8.81258 batch_time=0.47887 
Train Epoch: 29 [45/250 5760/32000 (18%)] Loss: 1.08870 (QuantReg: 8.97873) QuantErr: 8.97873 batch_time=0.46961 
Train Epoch: 29 [56/250 7168/32000 (22%)] Loss: 1.13438 (QuantReg: 8.99931) QuantErr: 8.99931 batch_time=0.43760 
Train Epoch: 29 [67/250 8576/32000 (27%)] Loss: 1.04365 (QuantReg: 9.02832) QuantErr: 9.02832 batch_time=0.50179 
Train Epoch: 29 [78/250 9984/32000 (31%)] Loss: 1.16600 (QuantReg: 8.91961) QuantErr: 8.91961 batch_time=0.43584 
Train Epoch: 29 [89/250 11392/32000 (36%)] Loss: 0.88362 (QuantReg: 9.01499) QuantErr: 9.01499 batch_time=0.47078 
Train Epoch: 29 [100/250 12800/32000 (40%)] Loss: 0.83433 (QuantReg: 8.86642) QuantErr: 8.86642 batch_time=0.48832 
Train Epoch: 29 [111/250 14208/32000 (44%)] Loss: 1.28244 (QuantReg: 8.86614) QuantErr: 8.86614 batch_time=0.45271 
Train Epoch: 29 [122/250 15616/32000 (49%)] Loss: 0.87331 (QuantReg: 9.13753) QuantErr: 9.13753 batch_time=0.47888 
Train Epoch: 29 [133/250 17024/32000 (53%)] Loss: 1.25694 (QuantReg: 8.87094) QuantErr: 8.87094 batch_time=0.45470 
Train Epoch: 29 [144/250 18432/32000 (58%)] Loss: 1.04782 (QuantReg: 8.93492) QuantErr: 8.93492 batch_time=0.47468 
Train Epoch: 29 [155/250 19840/32000 (62%)] Loss: 1.03132 (QuantReg: 8.78978) QuantErr: 8.78978 batch_time=0.46940 
Train Epoch: 29 [166/250 21248/32000 (66%)] Loss: 0.97207 (QuantReg: 8.94798) QuantErr: 8.94798 batch_time=0.45503 
Train Epoch: 29 [177/250 22656/32000 (71%)] Loss: 1.12929 (QuantReg: 8.77212) QuantErr: 8.77212 batch_time=0.44699 
Train Epoch: 29 [188/250 24064/32000 (75%)] Loss: 0.99985 (QuantReg: 8.85433) QuantErr: 8.85433 batch_time=0.44788 
Train Epoch: 29 [199/250 25472/32000 (80%)] Loss: 1.18124 (QuantReg: 8.90159) QuantErr: 8.90159 batch_time=2.89806 
Train Epoch: 29 [210/250 26880/32000 (84%)] Loss: 0.85630 (QuantReg: 9.11271) QuantErr: 9.11271 batch_time=2.79357 
Train Epoch: 29 [221/250 28288/32000 (88%)] Loss: 0.94735 (QuantReg: 8.75330) QuantErr: 8.75330 batch_time=0.45860 
Train Epoch: 29 [232/250 29696/32000 (93%)] Loss: 1.06252 (QuantReg: 8.88311) QuantErr: 8.88311 batch_time=0.44053 
Train Epoch: 29 [243/250 31104/32000 (97%)] Loss: 0.83190 (QuantReg: 8.98930) QuantErr: 8.98930 batch_time=0.52192 
Train Epoch: 29 codebook_update_time=1.01423
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch29.pth ...
Done in 4.016s
removing stale ckpt [epoch 28] [took 0.00s]
 epoch          : 29
 loss           : 1.0567060089111329
 quant_reg      : 8.935617458343506
 quant_err      : 8.935617458343506
 learning_rate  : 1.1891344262766608e-05
 n_samples      : 928000
 n_steps        : 7250
 MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
 MSRVTT_full_val/t2v_metrics/R5: 69.41649899396378
 MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.575452716297786
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.841536326659465
 MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
 MSRVTT_full_val/v2t_metrics/R5: 74.64788732394366
 MSRVTT_full_val/v2t_metrics/R10: 84.90945674044265
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.358148893360161
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.54182116647987
 MSRVTT_full_test/t2v_metrics/R1: 10.568561872909699
 MSRVTT_full_test/t2v_metrics/R5: 33.11036789297659
 MSRVTT_full_test/t2v_metrics/R10: 47.357859531772576
 MSRVTT_full_test/t2v_metrics/R50: 78.8628762541806
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.93578595317726
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.495135347007924
 MSRVTT_full_test/v2t_metrics/R1: 12.809364548494983
 MSRVTT_full_test/v2t_metrics/R5: 37.558528428093645
 MSRVTT_full_test/v2t_metrics/R10: 53.27759197324415
 MSRVTT_full_test/v2t_metrics/R50: 83.64548494983278
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.509197324414714
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.484487302390644
 mnt_best       : 26.678150256479153
 not_improved_count: 2
Train Epoch: 30 [1/250 128/32000 (0%)] Loss: 0.94403 (QuantReg: 8.97326) QuantErr: 8.97326 batch_time=38.72887 
Train Epoch: 30 [12/250 1536/32000 (5%)] Loss: 1.14699 (QuantReg: 8.95151) QuantErr: 8.95151 batch_time=0.96494 
Train Epoch: 30 [23/250 2944/32000 (9%)] Loss: 1.03017 (QuantReg: 8.93809) QuantErr: 8.93809 batch_time=0.46012 
Train Epoch: 30 [34/250 4352/32000 (14%)] Loss: 1.14422 (QuantReg: 8.90379) QuantErr: 8.90379 batch_time=0.44736 
Train Epoch: 30 [45/250 5760/32000 (18%)] Loss: 0.99457 (QuantReg: 8.85017) QuantErr: 8.85017 batch_time=0.79085 
Train Epoch: 30 [56/250 7168/32000 (22%)] Loss: 1.03779 (QuantReg: 9.18603) QuantErr: 9.18603 batch_time=0.46758 
Train Epoch: 30 [67/250 8576/32000 (27%)] Loss: 1.05412 (QuantReg: 8.91890) QuantErr: 8.91890 batch_time=0.43857 
Train Epoch: 30 [78/250 9984/32000 (31%)] Loss: 0.85514 (QuantReg: 9.08693) QuantErr: 9.08693 batch_time=0.43793 
Train Epoch: 30 [89/250 11392/32000 (36%)] Loss: 1.22899 (QuantReg: 8.81434) QuantErr: 8.81434 batch_time=0.44196 
Train Epoch: 30 [100/250 12800/32000 (40%)] Loss: 0.96383 (QuantReg: 9.13943) QuantErr: 9.13943 batch_time=0.48135 
Train Epoch: 30 [111/250 14208/32000 (44%)] Loss: 0.97716 (QuantReg: 9.04363) QuantErr: 9.04363 batch_time=0.44672 
Train Epoch: 30 [122/250 15616/32000 (49%)] Loss: 0.96062 (QuantReg: 9.01356) QuantErr: 9.01356 batch_time=0.42933 
Train Epoch: 30 [133/250 17024/32000 (53%)] Loss: 1.06087 (QuantReg: 9.04232) QuantErr: 9.04232 batch_time=0.48690 
Train Epoch: 30 [144/250 18432/32000 (58%)] Loss: 1.15067 (QuantReg: 8.86556) QuantErr: 8.86556 batch_time=0.44335 
Train Epoch: 30 [155/250 19840/32000 (62%)] Loss: 0.99988 (QuantReg: 8.99231) QuantErr: 8.99231 batch_time=0.47680 
Train Epoch: 30 [166/250 21248/32000 (66%)] Loss: 0.98095 (QuantReg: 9.05989) QuantErr: 9.05989 batch_time=0.44130 
Train Epoch: 30 [177/250 22656/32000 (71%)] Loss: 0.97238 (QuantReg: 8.94249) QuantErr: 8.94249 batch_time=0.49826 
Train Epoch: 30 [188/250 24064/32000 (75%)] Loss: 0.80182 (QuantReg: 9.09707) QuantErr: 9.09707 batch_time=0.49288 
Train Epoch: 30 [199/250 25472/32000 (80%)] Loss: 1.41455 (QuantReg: 8.93696) QuantErr: 8.93696 batch_time=0.45317 
Train Epoch: 30 [210/250 26880/32000 (84%)] Loss: 1.01650 (QuantReg: 8.97579) QuantErr: 8.97579 batch_time=0.48119 
Train Epoch: 30 [221/250 28288/32000 (88%)] Loss: 1.00832 (QuantReg: 9.08645) QuantErr: 9.08645 batch_time=0.47210 
Train Epoch: 30 [232/250 29696/32000 (93%)] Loss: 1.29474 (QuantReg: 9.10767) QuantErr: 9.10767 batch_time=0.43216 
Train Epoch: 30 [243/250 31104/32000 (97%)] Loss: 1.25712 (QuantReg: 8.99415) QuantErr: 8.99415 batch_time=0.47828 
Train Epoch: 30 codebook_update_time=0.91588
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch30.pth ...
Done in 4.122s
removing stale ckpt [epoch 29] [took 0.03s]
 epoch          : 30
 loss           : 1.0189383323192596
 quant_reg      : 8.980804702758789
 quant_err      : 8.980804702758789
 learning_rate  : 1.1296777049628277e-05
 n_samples      : 960000
 n_steps        : 7500
 MSRVTT_full_val/t2v_metrics/R1: 28.772635814889338
 MSRVTT_full_val/t2v_metrics/R5: 67.6056338028169
 MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.201207243460765
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.07491406301992
 MSRVTT_full_val/v2t_metrics/R1: 34.20523138832998
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 85.31187122736418
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.789738430583501
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.62351898634007
 MSRVTT_full_test/t2v_metrics/R1: 11.036789297658864
 MSRVTT_full_test/t2v_metrics/R5: 32.80936454849498
 MSRVTT_full_test/t2v_metrics/R10: 47.85953177257525
 MSRVTT_full_test/t2v_metrics/R50: 78.96321070234114
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.42775919732441
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.878335482074824
 MSRVTT_full_test/v2t_metrics/R1: 13.076923076923077
 MSRVTT_full_test/v2t_metrics/R5: 38.2943143812709
 MSRVTT_full_test/v2t_metrics/R10: 53.01003344481605
 MSRVTT_full_test/v2t_metrics/R50: 84.51505016722408
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.647157190635454
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.830874285518
 mnt_best       : 26.678150256479153
 not_improved_count: 3
Train Epoch: 31 [1/250 128/32000 (0%)] Loss: 1.24389 (QuantReg: 8.87390) QuantErr: 8.87390 batch_time=45.87753 
Train Epoch: 31 [12/250 1536/32000 (5%)] Loss: 1.08514 (QuantReg: 8.97845) QuantErr: 8.97845 batch_time=0.49418 
Train Epoch: 31 [23/250 2944/32000 (9%)] Loss: 1.31063 (QuantReg: 8.84191) QuantErr: 8.84191 batch_time=0.45905 
Train Epoch: 31 [34/250 4352/32000 (14%)] Loss: 0.91312 (QuantReg: 8.81836) QuantErr: 8.81836 batch_time=0.45255 
Train Epoch: 31 [45/250 5760/32000 (18%)] Loss: 1.12789 (QuantReg: 8.81698) QuantErr: 8.81698 batch_time=0.51326 
Train Epoch: 31 [56/250 7168/32000 (22%)] Loss: 1.31625 (QuantReg: 8.50784) QuantErr: 8.50784 batch_time=0.47322 
Train Epoch: 31 [67/250 8576/32000 (27%)] Loss: 1.06647 (QuantReg: 8.97064) QuantErr: 8.97064 batch_time=0.45452 
Train Epoch: 31 [78/250 9984/32000 (31%)] Loss: 1.13796 (QuantReg: 9.04907) QuantErr: 9.04907 batch_time=0.52176 
Train Epoch: 31 [89/250 11392/32000 (36%)] Loss: 1.09297 (QuantReg: 8.95063) QuantErr: 8.95063 batch_time=0.48682 
Train Epoch: 31 [100/250 12800/32000 (40%)] Loss: 1.03837 (QuantReg: 8.87157) QuantErr: 8.87157 batch_time=0.48122 
Train Epoch: 31 [111/250 14208/32000 (44%)] Loss: 0.88668 (QuantReg: 9.09840) QuantErr: 9.09840 batch_time=0.52267 
Train Epoch: 31 [122/250 15616/32000 (49%)] Loss: 0.94360 (QuantReg: 8.94718) QuantErr: 8.94718 batch_time=0.47689 
Train Epoch: 31 [133/250 17024/32000 (53%)] Loss: 1.03611 (QuantReg: 9.08113) QuantErr: 9.08113 batch_time=0.44903 
Train Epoch: 31 [144/250 18432/32000 (58%)] Loss: 1.31301 (QuantReg: 9.16083) QuantErr: 9.16083 batch_time=0.48849 
Train Epoch: 31 [155/250 19840/32000 (62%)] Loss: 1.12157 (QuantReg: 8.90040) QuantErr: 8.90040 batch_time=0.57609 
Train Epoch: 31 [166/250 21248/32000 (66%)] Loss: 0.95006 (QuantReg: 8.77325) QuantErr: 8.77325 batch_time=0.53290 
Train Epoch: 31 [177/250 22656/32000 (71%)] Loss: 1.33277 (QuantReg: 8.76525) QuantErr: 8.76525 batch_time=0.45953 
Train Epoch: 31 [188/250 24064/32000 (75%)] Loss: 0.97331 (QuantReg: 9.10680) QuantErr: 9.10680 batch_time=0.49157 
Train Epoch: 31 [199/250 25472/32000 (80%)] Loss: 0.87794 (QuantReg: 8.96563) QuantErr: 8.96563 batch_time=0.43887 
Train Epoch: 31 [210/250 26880/32000 (84%)] Loss: 1.23176 (QuantReg: 8.88256) QuantErr: 8.88256 batch_time=0.52335 
Train Epoch: 31 [221/250 28288/32000 (88%)] Loss: 0.82116 (QuantReg: 9.24151) QuantErr: 9.24151 batch_time=0.49259 
Train Epoch: 31 [232/250 29696/32000 (93%)] Loss: 0.99686 (QuantReg: 8.95410) QuantErr: 8.95410 batch_time=0.45482 
Train Epoch: 31 [243/250 31104/32000 (97%)] Loss: 1.16315 (QuantReg: 8.95990) QuantErr: 8.95990 batch_time=0.80094 
Train Epoch: 31 codebook_update_time=0.89480
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch31.pth ...
Done in 3.992s
removing stale ckpt [epoch 30] [took 0.00s]
 epoch          : 31
 loss           : 1.0444082336425782
 quant_reg      : 8.98666237258911
 quant_err      : 8.98666237258911
 learning_rate  : 1.0731938197146863e-05
 n_samples      : 992000
 n_steps        : 7750
 MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.024144869215291
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.52175333500434
 MSRVTT_full_val/v2t_metrics/R1: 35.2112676056338
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.7002012072434605
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.15505810014524
 MSRVTT_full_test/t2v_metrics/R1: 11.237458193979933
 MSRVTT_full_test/t2v_metrics/R5: 33.979933110367895
 MSRVTT_full_test/t2v_metrics/R10: 48.32775919732441
 MSRVTT_full_test/t2v_metrics/R50: 79.76588628762542
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.717725752508365
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.425857524447984
 MSRVTT_full_test/v2t_metrics/R1: 13.110367892976589
 MSRVTT_full_test/v2t_metrics/R5: 38.39464882943144
 MSRVTT_full_test/v2t_metrics/R10: 53.812709030100336
 MSRVTT_full_test/v2t_metrics/R50: 84.94983277591973
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 35.705016722408025
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.032407266112582
 mnt_best       : 26.678150256479153
 not_improved_count: 4
Train Epoch: 32 [1/250 128/32000 (0%)] Loss: 0.79594 (QuantReg: 9.02945) QuantErr: 9.02945 batch_time=40.27875 
Train Epoch: 32 [12/250 1536/32000 (5%)] Loss: 1.01818 (QuantReg: 9.09702) QuantErr: 9.09702 batch_time=0.50233 
Train Epoch: 32 [23/250 2944/32000 (9%)] Loss: 0.98864 (QuantReg: 8.95494) QuantErr: 8.95494 batch_time=0.44434 
Train Epoch: 32 [34/250 4352/32000 (14%)] Loss: 0.96943 (QuantReg: 9.19144) QuantErr: 9.19144 batch_time=0.46159 
Train Epoch: 32 [45/250 5760/32000 (18%)] Loss: 0.78268 (QuantReg: 9.07714) QuantErr: 9.07714 batch_time=0.46544 
Train Epoch: 32 [56/250 7168/32000 (22%)] Loss: 1.06096 (QuantReg: 8.93642) QuantErr: 8.93642 batch_time=0.47371 
Train Epoch: 32 [67/250 8576/32000 (27%)] Loss: 0.72547 (QuantReg: 9.10376) QuantErr: 9.10376 batch_time=0.46174 
Train Epoch: 32 [78/250 9984/32000 (31%)] Loss: 0.98119 (QuantReg: 8.93449) QuantErr: 8.93449 batch_time=0.71921 
Train Epoch: 32 [89/250 11392/32000 (36%)] Loss: 1.21209 (QuantReg: 8.93475) QuantErr: 8.93475 batch_time=0.47520 
Train Epoch: 32 [100/250 12800/32000 (40%)] Loss: 1.01052 (QuantReg: 8.86682) QuantErr: 8.86682 batch_time=0.44217 
Train Epoch: 32 [111/250 14208/32000 (44%)] Loss: 1.08566 (QuantReg: 9.35552) QuantErr: 9.35552 batch_time=0.45673 
Train Epoch: 32 [122/250 15616/32000 (49%)] Loss: 0.85992 (QuantReg: 8.83418) QuantErr: 8.83418 batch_time=0.47065 
Train Epoch: 32 [133/250 17024/32000 (53%)] Loss: 1.10238 (QuantReg: 8.94493) QuantErr: 8.94493 batch_time=0.48278 
Train Epoch: 32 [144/250 18432/32000 (58%)] Loss: 1.04251 (QuantReg: 8.95116) QuantErr: 8.95116 batch_time=0.47101 
Train Epoch: 32 [155/250 19840/32000 (62%)] Loss: 0.88321 (QuantReg: 8.96163) QuantErr: 8.96163 batch_time=0.46879 
Train Epoch: 32 [166/250 21248/32000 (66%)] Loss: 1.03539 (QuantReg: 9.09415) QuantErr: 9.09415 batch_time=0.45366 
Train Epoch: 32 [177/250 22656/32000 (71%)] Loss: 1.15264 (QuantReg: 9.20422) QuantErr: 9.20422 batch_time=0.50124 
Train Epoch: 32 [188/250 24064/32000 (75%)] Loss: 1.07569 (QuantReg: 9.16306) QuantErr: 9.16306 batch_time=0.45762 
Train Epoch: 32 [199/250 25472/32000 (80%)] Loss: 0.95842 (QuantReg: 9.25529) QuantErr: 9.25529 batch_time=2.38083 
Train Epoch: 32 [210/250 26880/32000 (84%)] Loss: 1.20985 (QuantReg: 9.10604) QuantErr: 9.10604 batch_time=0.46711 
Train Epoch: 32 [221/250 28288/32000 (88%)] Loss: 0.91113 (QuantReg: 9.19496) QuantErr: 9.19496 batch_time=0.47854 
Train Epoch: 32 [232/250 29696/32000 (93%)] Loss: 0.98589 (QuantReg: 8.91524) QuantErr: 8.91524 batch_time=0.45875 
Train Epoch: 32 [243/250 31104/32000 (97%)] Loss: 0.79958 (QuantReg: 9.10063) QuantErr: 9.10063 batch_time=0.46070 
Train Epoch: 32 codebook_update_time=0.91218
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch32.pth ...
Done in 3.911s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch32.pth ...
Done in 7.772s
removing stale ckpt [epoch 31] [took 0.00s]
 epoch          : 32
 loss           : 1.0131052272319794
 quant_reg      : 9.013548164367675
 quant_err      : 9.013548164367675
 learning_rate  : 1.019534128728952e-05
 n_samples      : 1024000
 n_steps        : 8000
 MSRVTT_full_val/t2v_metrics/R1: 30.985915492957748
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 81.89134808853119
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 2.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.152917505030182
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.72908176602859
 MSRVTT_full_val/v2t_metrics/R1: 36.21730382293762
 MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.6579476861167
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.22307566184777
 MSRVTT_full_test/t2v_metrics/R1: 11.872909698996656
 MSRVTT_full_test/t2v_metrics/R5: 34.61538461538461
 MSRVTT_full_test/t2v_metrics/R10: 48.896321070234116
 MSRVTT_full_test/t2v_metrics/R50: 80.0
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.21371237458194
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.187389145442058
 MSRVTT_full_test/v2t_metrics/R1: 12.876254180602007
 MSRVTT_full_test/v2t_metrics/R5: 38.896321070234116
 MSRVTT_full_test/v2t_metrics/R10: 53.87959866220736
 MSRVTT_full_test/v2t_metrics/R50: 85.28428093645485
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 35.089297658862876
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.99444335273938
 mnt_best       : 27.187389145442058
 not_improved_count: 0
Train Epoch: 33 [1/250 128/32000 (0%)] Loss: 1.03307 (QuantReg: 9.24211) QuantErr: 9.24211 batch_time=32.14990 
Train Epoch: 33 [12/250 1536/32000 (5%)] Loss: 0.87840 (QuantReg: 8.95082) QuantErr: 8.95082 batch_time=0.46199 
Train Epoch: 33 [23/250 2944/32000 (9%)] Loss: 1.11092 (QuantReg: 9.17109) QuantErr: 9.17109 batch_time=0.46375 
Train Epoch: 33 [34/250 4352/32000 (14%)] Loss: 0.84317 (QuantReg: 8.66073) QuantErr: 8.66073 batch_time=0.45309 
Train Epoch: 33 [45/250 5760/32000 (18%)] Loss: 0.90740 (QuantReg: 9.14845) QuantErr: 9.14845 batch_time=0.52276 
Train Epoch: 33 [56/250 7168/32000 (22%)] Loss: 1.00763 (QuantReg: 9.05284) QuantErr: 9.05284 batch_time=0.45547 
Train Epoch: 33 [67/250 8576/32000 (27%)] Loss: 1.22781 (QuantReg: 8.98972) QuantErr: 8.98972 batch_time=5.66102 
Train Epoch: 33 [78/250 9984/32000 (31%)] Loss: 0.95495 (QuantReg: 8.83240) QuantErr: 8.83240 batch_time=1.60072 
Train Epoch: 33 [89/250 11392/32000 (36%)] Loss: 1.02829 (QuantReg: 8.81188) QuantErr: 8.81188 batch_time=0.48681 
Train Epoch: 33 [100/250 12800/32000 (40%)] Loss: 1.07222 (QuantReg: 8.93070) QuantErr: 8.93070 batch_time=0.50542 
Train Epoch: 33 [111/250 14208/32000 (44%)] Loss: 1.13670 (QuantReg: 8.94498) QuantErr: 8.94498 batch_time=0.47006 
Train Epoch: 33 [122/250 15616/32000 (49%)] Loss: 0.95818 (QuantReg: 8.90517) QuantErr: 8.90517 batch_time=0.48124 
Train Epoch: 33 [133/250 17024/32000 (53%)] Loss: 1.30467 (QuantReg: 8.88441) QuantErr: 8.88441 batch_time=0.44982 
Train Epoch: 33 [144/250 18432/32000 (58%)] Loss: 1.06458 (QuantReg: 9.04946) QuantErr: 9.04946 batch_time=0.45520 
Train Epoch: 33 [155/250 19840/32000 (62%)] Loss: 0.90790 (QuantReg: 8.84271) QuantErr: 8.84271 batch_time=0.44166 
Train Epoch: 33 [166/250 21248/32000 (66%)] Loss: 1.01655 (QuantReg: 9.19376) QuantErr: 9.19376 batch_time=0.50047 
Train Epoch: 33 [177/250 22656/32000 (71%)] Loss: 0.86296 (QuantReg: 9.07358) QuantErr: 9.07358 batch_time=0.44877 
Train Epoch: 33 [188/250 24064/32000 (75%)] Loss: 1.14243 (QuantReg: 9.09145) QuantErr: 9.09145 batch_time=0.45313 
Train Epoch: 33 [199/250 25472/32000 (80%)] Loss: 0.97149 (QuantReg: 8.90114) QuantErr: 8.90114 batch_time=0.44662 
Train Epoch: 33 [210/250 26880/32000 (84%)] Loss: 0.98423 (QuantReg: 9.19083) QuantErr: 9.19083 batch_time=0.46441 
Train Epoch: 33 [221/250 28288/32000 (88%)] Loss: 1.33623 (QuantReg: 8.93040) QuantErr: 8.93040 batch_time=0.43437 
Train Epoch: 33 [232/250 29696/32000 (93%)] Loss: 0.96424 (QuantReg: 8.90163) QuantErr: 8.90163 batch_time=0.48431 
Train Epoch: 33 [243/250 31104/32000 (97%)] Loss: 0.98581 (QuantReg: 9.03046) QuantErr: 9.03046 batch_time=0.50790 
Train Epoch: 33 codebook_update_time=0.92159
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch33.pth ...
Done in 15.037s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch33.pth ...
Done in 18.935s
removing stale ckpt [epoch 32] [took 0.00s]
 epoch          : 33
 loss           : 1.0076525197029114
 quant_reg      : 9.011994983673096
 quant_err      : 9.011994983673096
 learning_rate  : 9.685574222925043e-06
 n_samples      : 1056000
 n_steps        : 8250
 MSRVTT_full_val/t2v_metrics/R1: 32.394366197183096
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
 MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.845070422535212
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.234840341176174
 MSRVTT_full_val/v2t_metrics/R1: 37.02213279678068
 MSRVTT_full_val/v2t_metrics/R5: 73.8430583501006
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.907444668008048
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.36037481744393
 MSRVTT_full_test/t2v_metrics/R1: 11.73913043478261
 MSRVTT_full_test/t2v_metrics/R5: 34.91638795986622
 MSRVTT_full_test/t2v_metrics/R10: 49.096989966555185
 MSRVTT_full_test/t2v_metrics/R50: 79.76588628762542
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.242140468227426
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.20027961732058
 MSRVTT_full_test/v2t_metrics/R1: 13.043478260869565
 MSRVTT_full_test/v2t_metrics/R5: 39.23076923076923
 MSRVTT_full_test/v2t_metrics/R10: 53.1438127090301
 MSRVTT_full_test/v2t_metrics/R50: 85.01672240802675
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 34.37040133779264
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.07167717218024
 mnt_best       : 27.20027961732058
 not_improved_count: 0
Train Epoch: 34 [1/250 128/32000 (0%)] Loss: 0.92331 (QuantReg: 9.18447) QuantErr: 9.18447 batch_time=41.57141 
Train Epoch: 34 [12/250 1536/32000 (5%)] Loss: 0.94912 (QuantReg: 9.20599) QuantErr: 9.20599 batch_time=0.46431 
Train Epoch: 34 [23/250 2944/32000 (9%)] Loss: 0.94084 (QuantReg: 9.13111) QuantErr: 9.13111 batch_time=0.48044 
Train Epoch: 34 [34/250 4352/32000 (14%)] Loss: 0.95232 (QuantReg: 9.03726) QuantErr: 9.03726 batch_time=0.43411 
Train Epoch: 34 [45/250 5760/32000 (18%)] Loss: 1.10396 (QuantReg: 9.13901) QuantErr: 9.13901 batch_time=0.45869 
Train Epoch: 34 [56/250 7168/32000 (22%)] Loss: 1.07784 (QuantReg: 8.96675) QuantErr: 8.96675 batch_time=0.47254 
Train Epoch: 34 [67/250 8576/32000 (27%)] Loss: 1.05467 (QuantReg: 9.04553) QuantErr: 9.04553 batch_time=0.46246 
Train Epoch: 34 [78/250 9984/32000 (31%)] Loss: 1.31490 (QuantReg: 8.89335) QuantErr: 8.89335 batch_time=0.47067 
Train Epoch: 34 [89/250 11392/32000 (36%)] Loss: 1.11248 (QuantReg: 8.96154) QuantErr: 8.96154 batch_time=0.48321 
Train Epoch: 34 [100/250 12800/32000 (40%)] Loss: 1.38108 (QuantReg: 8.95666) QuantErr: 8.95666 batch_time=0.45502 
Train Epoch: 34 [111/250 14208/32000 (44%)] Loss: 1.13884 (QuantReg: 8.85026) QuantErr: 8.85026 batch_time=0.45956 
Train Epoch: 34 [122/250 15616/32000 (49%)] Loss: 1.00262 (QuantReg: 8.98493) QuantErr: 8.98493 batch_time=0.46981 
Train Epoch: 34 [133/250 17024/32000 (53%)] Loss: 1.11633 (QuantReg: 9.09636) QuantErr: 9.09636 batch_time=0.48683 
Train Epoch: 34 [144/250 18432/32000 (58%)] Loss: 1.06512 (QuantReg: 8.97498) QuantErr: 8.97498 batch_time=0.47714 
Train Epoch: 34 [155/250 19840/32000 (62%)] Loss: 0.94792 (QuantReg: 9.28553) QuantErr: 9.28553 batch_time=0.44981 
Train Epoch: 34 [166/250 21248/32000 (66%)] Loss: 1.17527 (QuantReg: 8.83055) QuantErr: 8.83055 batch_time=0.46519 
Train Epoch: 34 [177/250 22656/32000 (71%)] Loss: 0.94946 (QuantReg: 9.02452) QuantErr: 9.02452 batch_time=0.45052 
Train Epoch: 34 [188/250 24064/32000 (75%)] Loss: 1.23807 (QuantReg: 8.94143) QuantErr: 8.94143 batch_time=0.49950 
Train Epoch: 34 [199/250 25472/32000 (80%)] Loss: 0.83083 (QuantReg: 8.95852) QuantErr: 8.95852 batch_time=0.52702 
Train Epoch: 34 [210/250 26880/32000 (84%)] Loss: 1.12435 (QuantReg: 8.93236) QuantErr: 8.93236 batch_time=0.45649 
Train Epoch: 34 [221/250 28288/32000 (88%)] Loss: 0.91452 (QuantReg: 9.13090) QuantErr: 9.13090 batch_time=0.52657 
Train Epoch: 34 [232/250 29696/32000 (93%)] Loss: 0.82842 (QuantReg: 9.07923) QuantErr: 9.07923 batch_time=0.55697 
Train Epoch: 34 [243/250 31104/32000 (97%)] Loss: 1.24951 (QuantReg: 9.20921) QuantErr: 9.20921 batch_time=0.51340 
Train Epoch: 34 codebook_update_time=0.90755
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch34.pth ...
Done in 4.140s
removing stale ckpt [epoch 33] [took 0.00s]
 epoch          : 34
 loss           : 1.0386132056713104
 quant_reg      : 9.022835964202882
 quant_err      : 9.022835964202882
 learning_rate  : 9.20129551177879e-06
 n_samples      : 1088000
 n_steps        : 8500
 MSRVTT_full_val/t2v_metrics/R1: 29.77867203219316
 MSRVTT_full_val/t2v_metrics/R5: 65.59356136820925
 MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.26559356136821
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.88043257665481
 MSRVTT_full_val/v2t_metrics/R1: 33.80281690140845
 MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
 MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.934607645875252
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.27059135633378
 MSRVTT_full_test/t2v_metrics/R1: 10.40133779264214
 MSRVTT_full_test/t2v_metrics/R5: 33.1438127090301
 MSRVTT_full_test/t2v_metrics/R10: 47.02341137123746
 MSRVTT_full_test/t2v_metrics/R50: 78.69565217391305
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 52.56956521739131
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.308627847268234
 MSRVTT_full_test/v2t_metrics/R1: 13.91304347826087
 MSRVTT_full_test/v2t_metrics/R5: 38.72909698996656
 MSRVTT_full_test/v2t_metrics/R10: 53.812709030100336
 MSRVTT_full_test/v2t_metrics/R50: 84.34782608695652
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.759698996655516
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.721903668409652
 mnt_best       : 27.20027961732058
 not_improved_count: 1
Train Epoch: 35 [1/250 128/32000 (0%)] Loss: 0.94165 (QuantReg: 8.99585) QuantErr: 8.99585 batch_time=35.47227 
Train Epoch: 35 [12/250 1536/32000 (5%)] Loss: 1.12693 (QuantReg: 9.03320) QuantErr: 9.03320 batch_time=0.43416 
Train Epoch: 35 [23/250 2944/32000 (9%)] Loss: 1.14711 (QuantReg: 8.97809) QuantErr: 8.97809 batch_time=0.45271 
Train Epoch: 35 [34/250 4352/32000 (14%)] Loss: 1.29606 (QuantReg: 9.00242) QuantErr: 9.00242 batch_time=0.49365 
Train Epoch: 35 [45/250 5760/32000 (18%)] Loss: 1.01473 (QuantReg: 8.94806) QuantErr: 8.94806 batch_time=0.44889 
Train Epoch: 35 [56/250 7168/32000 (22%)] Loss: 0.89097 (QuantReg: 9.05538) QuantErr: 9.05538 batch_time=0.47431 
Train Epoch: 35 [67/250 8576/32000 (27%)] Loss: 0.88509 (QuantReg: 9.16314) QuantErr: 9.16314 batch_time=0.47873 
Train Epoch: 35 [78/250 9984/32000 (31%)] Loss: 1.32878 (QuantReg: 9.20346) QuantErr: 9.20346 batch_time=0.50932 
Train Epoch: 35 [89/250 11392/32000 (36%)] Loss: 0.99688 (QuantReg: 9.00641) QuantErr: 9.00641 batch_time=0.44001 
Train Epoch: 35 [100/250 12800/32000 (40%)] Loss: 1.10745 (QuantReg: 9.03082) QuantErr: 9.03082 batch_time=0.48148 
Train Epoch: 35 [111/250 14208/32000 (44%)] Loss: 1.06313 (QuantReg: 8.98230) QuantErr: 8.98230 batch_time=0.46262 
Train Epoch: 35 [122/250 15616/32000 (49%)] Loss: 0.99954 (QuantReg: 9.00409) QuantErr: 9.00409 batch_time=0.43781 
Train Epoch: 35 [133/250 17024/32000 (53%)] Loss: 1.33944 (QuantReg: 8.90123) QuantErr: 8.90123 batch_time=0.43813 
Train Epoch: 35 [144/250 18432/32000 (58%)] Loss: 0.91086 (QuantReg: 9.21556) QuantErr: 9.21556 batch_time=0.53653 
Train Epoch: 35 [155/250 19840/32000 (62%)] Loss: 0.92056 (QuantReg: 8.95365) QuantErr: 8.95365 batch_time=0.48289 
Train Epoch: 35 [166/250 21248/32000 (66%)] Loss: 1.13368 (QuantReg: 9.15838) QuantErr: 9.15838 batch_time=0.46341 
Train Epoch: 35 [177/250 22656/32000 (71%)] Loss: 0.90385 (QuantReg: 9.40901) QuantErr: 9.40901 batch_time=0.48059 
Train Epoch: 35 [188/250 24064/32000 (75%)] Loss: 0.91384 (QuantReg: 8.88925) QuantErr: 8.88925 batch_time=0.47602 
Train Epoch: 35 [199/250 25472/32000 (80%)] Loss: 0.93962 (QuantReg: 9.27868) QuantErr: 9.27868 batch_time=0.44305 
Train Epoch: 35 [210/250 26880/32000 (84%)] Loss: 0.87936 (QuantReg: 9.14911) QuantErr: 9.14911 batch_time=0.45977 
Train Epoch: 35 [221/250 28288/32000 (88%)] Loss: 0.97409 (QuantReg: 9.10223) QuantErr: 9.10223 batch_time=0.44806 
Train Epoch: 35 [232/250 29696/32000 (93%)] Loss: 0.95300 (QuantReg: 9.23629) QuantErr: 9.23629 batch_time=0.45839 
Train Epoch: 35 [243/250 31104/32000 (97%)] Loss: 0.88047 (QuantReg: 9.17017) QuantErr: 9.17017 batch_time=0.47376 
Train Epoch: 35 codebook_update_time=1.04557
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch35.pth ...
Done in 3.797s
removing stale ckpt [epoch 34] [took 0.00s]
 epoch          : 35
 loss           : 1.0018201265335083
 quant_reg      : 9.05860733795166
 quant_err      : 9.05860733795166
 learning_rate  : 8.74123073618985e-06
 n_samples      : 1120000
 n_steps        : 8750
 MSRVTT_full_val/t2v_metrics/R1: 30.18108651911469
 MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
 MSRVTT_full_val/t2v_metrics/R10: 79.47686116700201
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.575452716297786
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.4237611248838
 MSRVTT_full_val/v2t_metrics/R1: 36.01609657947686
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.940643863179075
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.609934036247935
 MSRVTT_full_test/t2v_metrics/R1: 11.237458193979933
 MSRVTT_full_test/t2v_metrics/R5: 34.11371237458194
 MSRVTT_full_test/t2v_metrics/R10: 48.19397993311037
 MSRVTT_full_test/t2v_metrics/R50: 79.56521739130434
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.214548494983276
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.436053473266025
 MSRVTT_full_test/v2t_metrics/R1: 13.779264214046822
 MSRVTT_full_test/v2t_metrics/R5: 39.4314381270903
 MSRVTT_full_test/v2t_metrics/R10: 54.214046822742475
 MSRVTT_full_test/v2t_metrics/R50: 85.31772575250837
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 35.36103678929766
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.88352211593281
 mnt_best       : 27.20027961732058
 not_improved_count: 2
Train Epoch: 36 [1/250 128/32000 (0%)] Loss: 0.93991 (QuantReg: 8.96974) QuantErr: 8.96974 batch_time=42.28686 
Train Epoch: 36 [12/250 1536/32000 (5%)] Loss: 1.12114 (QuantReg: 9.04320) QuantErr: 9.04320 batch_time=0.43045 
Train Epoch: 36 [23/250 2944/32000 (9%)] Loss: 0.71846 (QuantReg: 8.90218) QuantErr: 8.90218 batch_time=0.43277 
Train Epoch: 36 [34/250 4352/32000 (14%)] Loss: 0.91777 (QuantReg: 8.92563) QuantErr: 8.92563 batch_time=0.42982 
Train Epoch: 36 [45/250 5760/32000 (18%)] Loss: 1.05103 (QuantReg: 9.04042) QuantErr: 9.04042 batch_time=0.47606 
Train Epoch: 36 [56/250 7168/32000 (22%)] Loss: 0.73633 (QuantReg: 8.92055) QuantErr: 8.92055 batch_time=0.45196 
Train Epoch: 36 [67/250 8576/32000 (27%)] Loss: 0.97472 (QuantReg: 9.15101) QuantErr: 9.15101 batch_time=0.43623 
Train Epoch: 36 [78/250 9984/32000 (31%)] Loss: 1.17361 (QuantReg: 9.16186) QuantErr: 9.16186 batch_time=0.45888 
Train Epoch: 36 [89/250 11392/32000 (36%)] Loss: 1.16916 (QuantReg: 8.91600) QuantErr: 8.91600 batch_time=0.45282 
Train Epoch: 36 [100/250 12800/32000 (40%)] Loss: 1.04004 (QuantReg: 9.04735) QuantErr: 9.04735 batch_time=0.44923 
Train Epoch: 36 [111/250 14208/32000 (44%)] Loss: 1.21985 (QuantReg: 8.97369) QuantErr: 8.97369 batch_time=0.44723 
Train Epoch: 36 [122/250 15616/32000 (49%)] Loss: 1.13771 (QuantReg: 9.01427) QuantErr: 9.01427 batch_time=0.49160 
Train Epoch: 36 [133/250 17024/32000 (53%)] Loss: 0.96897 (QuantReg: 8.90307) QuantErr: 8.90307 batch_time=0.80032 
Train Epoch: 36 [144/250 18432/32000 (58%)] Loss: 1.07457 (QuantReg: 9.23968) QuantErr: 9.23968 batch_time=0.43878 
Train Epoch: 36 [155/250 19840/32000 (62%)] Loss: 0.80055 (QuantReg: 9.10122) QuantErr: 9.10122 batch_time=0.42747 
Train Epoch: 36 [166/250 21248/32000 (66%)] Loss: 1.13405 (QuantReg: 9.07596) QuantErr: 9.07596 batch_time=0.44042 
Train Epoch: 36 [177/250 22656/32000 (71%)] Loss: 1.16944 (QuantReg: 9.07417) QuantErr: 9.07417 batch_time=0.45646 
Train Epoch: 36 [188/250 24064/32000 (75%)] Loss: 0.84632 (QuantReg: 9.12602) QuantErr: 9.12602 batch_time=0.45862 
Train Epoch: 36 [199/250 25472/32000 (80%)] Loss: 0.83131 (QuantReg: 9.03835) QuantErr: 9.03835 batch_time=0.44383 
Train Epoch: 36 [210/250 26880/32000 (84%)] Loss: 1.01056 (QuantReg: 8.99810) QuantErr: 8.99810 batch_time=0.43947 
Train Epoch: 36 [221/250 28288/32000 (88%)] Loss: 1.18577 (QuantReg: 8.79784) QuantErr: 8.79784 batch_time=0.47186 
Train Epoch: 36 [232/250 29696/32000 (93%)] Loss: 1.10106 (QuantReg: 9.03236) QuantErr: 9.03236 batch_time=0.55594 
Train Epoch: 36 [243/250 31104/32000 (97%)] Loss: 0.90971 (QuantReg: 8.85775) QuantErr: 8.85775 batch_time=0.42971 
Train Epoch: 36 codebook_update_time=0.88306
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch36.pth ...
Done in 10.371s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch36.pth ...
Done in 14.396s
removing stale ckpt [epoch 35] [took 0.00s]
 epoch          : 36
 loss           : 1.0115836184024811
 quant_reg      : 9.06857816696167
 quant_err      : 9.06857816696167
 learning_rate  : 8.304169199380357e-06
 n_samples      : 1152000
 n_steps        : 9000
 MSRVTT_full_val/t2v_metrics/R1: 31.790744466800806
 MSRVTT_full_val/t2v_metrics/R5: 67.80684104627767
 MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
 MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.6579476861167
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.95854266431811
 MSRVTT_full_val/v2t_metrics/R1: 36.21730382293762
 MSRVTT_full_val/v2t_metrics/R5: 75.65392354124748
 MSRVTT_full_val/v2t_metrics/R10: 85.51307847082495
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.153923541247485
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.64909486232793
 MSRVTT_full_test/t2v_metrics/R1: 12.14046822742475
 MSRVTT_full_test/t2v_metrics/R5: 35.51839464882943
 MSRVTT_full_test/t2v_metrics/R10: 49.4314381270903
 MSRVTT_full_test/t2v_metrics/R50: 79.53177257525084
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.72692307692308
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.726645958967108
 MSRVTT_full_test/v2t_metrics/R1: 13.678929765886288
 MSRVTT_full_test/v2t_metrics/R5: 39.86622073578595
 MSRVTT_full_test/v2t_metrics/R10: 54.68227424749164
 MSRVTT_full_test/v2t_metrics/R50: 84.98327759197325
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 35.67073578595318
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.009963268392045
 mnt_best       : 27.726645958967108
 not_improved_count: 0
Train Epoch: 37 [1/250 128/32000 (0%)] Loss: 1.06990 (QuantReg: 9.03796) QuantErr: 9.03796 batch_time=42.37543 
Train Epoch: 37 [12/250 1536/32000 (5%)] Loss: 0.87967 (QuantReg: 9.00621) QuantErr: 9.00621 batch_time=0.48185 
Train Epoch: 37 [23/250 2944/32000 (9%)] Loss: 1.05182 (QuantReg: 8.93210) QuantErr: 8.93210 batch_time=0.46705 
Train Epoch: 37 [34/250 4352/32000 (14%)] Loss: 0.87847 (QuantReg: 9.07882) QuantErr: 9.07882 batch_time=0.45413 
Train Epoch: 37 [45/250 5760/32000 (18%)] Loss: 0.97353 (QuantReg: 9.10439) QuantErr: 9.10439 batch_time=0.45086 
Train Epoch: 37 [56/250 7168/32000 (22%)] Loss: 1.26676 (QuantReg: 8.92014) QuantErr: 8.92014 batch_time=0.45499 
Train Epoch: 37 [67/250 8576/32000 (27%)] Loss: 1.25339 (QuantReg: 9.05920) QuantErr: 9.05920 batch_time=2.49516 
Train Epoch: 37 [78/250 9984/32000 (31%)] Loss: 0.87237 (QuantReg: 9.05060) QuantErr: 9.05060 batch_time=0.44748 
Train Epoch: 37 [89/250 11392/32000 (36%)] Loss: 1.07834 (QuantReg: 9.16431) QuantErr: 9.16431 batch_time=0.43589 
Train Epoch: 37 [100/250 12800/32000 (40%)] Loss: 0.87230 (QuantReg: 8.95583) QuantErr: 8.95583 batch_time=0.44839 
Train Epoch: 37 [111/250 14208/32000 (44%)] Loss: 0.92140 (QuantReg: 9.08202) QuantErr: 9.08202 batch_time=0.43910 
Train Epoch: 37 [122/250 15616/32000 (49%)] Loss: 0.93715 (QuantReg: 9.10462) QuantErr: 9.10462 batch_time=0.48538 
Train Epoch: 37 [133/250 17024/32000 (53%)] Loss: 1.06431 (QuantReg: 8.98703) QuantErr: 8.98703 batch_time=0.48859 
Train Epoch: 37 [144/250 18432/32000 (58%)] Loss: 1.22234 (QuantReg: 9.02183) QuantErr: 9.02183 batch_time=1.22378 
Train Epoch: 37 [155/250 19840/32000 (62%)] Loss: 1.00372 (QuantReg: 8.99136) QuantErr: 8.99136 batch_time=0.47487 
Train Epoch: 37 [166/250 21248/32000 (66%)] Loss: 1.08368 (QuantReg: 9.13159) QuantErr: 9.13159 batch_time=0.46841 
Train Epoch: 37 [177/250 22656/32000 (71%)] Loss: 1.08903 (QuantReg: 9.05352) QuantErr: 9.05352 batch_time=0.44596 
Train Epoch: 37 [188/250 24064/32000 (75%)] Loss: 1.13299 (QuantReg: 9.00513) QuantErr: 9.00513 batch_time=0.44731 
Train Epoch: 37 [199/250 25472/32000 (80%)] Loss: 0.98351 (QuantReg: 9.26791) QuantErr: 9.26791 batch_time=0.46051 
Train Epoch: 37 [210/250 26880/32000 (84%)] Loss: 1.19639 (QuantReg: 9.03631) QuantErr: 9.03631 batch_time=1.93682 
Train Epoch: 37 [221/250 28288/32000 (88%)] Loss: 0.99557 (QuantReg: 9.39935) QuantErr: 9.39935 batch_time=0.47078 
Train Epoch: 37 [232/250 29696/32000 (93%)] Loss: 1.33980 (QuantReg: 8.84546) QuantErr: 8.84546 batch_time=0.45882 
Train Epoch: 37 [243/250 31104/32000 (97%)] Loss: 0.90954 (QuantReg: 9.12291) QuantErr: 9.12291 batch_time=0.44297 
Train Epoch: 37 codebook_update_time=0.92790
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch37.pth ...
Done in 17.913s
removing stale ckpt [epoch 36] [took 0.02s]
 epoch          : 37
 loss           : 1.022519877433777
 quant_reg      : 9.06248625946045
 quant_err      : 9.06248625946045
 learning_rate  : 7.888960739411339e-06
 n_samples      : 1184000
 n_steps        : 9250
 MSRVTT_full_val/t2v_metrics/R1: 31.187122736418512
 MSRVTT_full_val/t2v_metrics/R5: 67.40442655935614
 MSRVTT_full_val/t2v_metrics/R10: 81.08651911468813
 MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.490945674044266
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.446073335950075
 MSRVTT_full_val/v2t_metrics/R1: 37.625754527162975
 MSRVTT_full_val/v2t_metrics/R5: 73.44064386317908
 MSRVTT_full_val/v2t_metrics/R10: 85.51307847082495
 MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.907444668008048
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.823190178203944
 MSRVTT_full_test/t2v_metrics/R1: 11.906354515050166
 MSRVTT_full_test/t2v_metrics/R5: 35.01672240802676
 MSRVTT_full_test/t2v_metrics/R10: 49.86622073578595
 MSRVTT_full_test/t2v_metrics/R50: 79.49832775919732
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.13595317725753
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.497101664475895
 MSRVTT_full_test/v2t_metrics/R1: 13.812709030100335
 MSRVTT_full_test/v2t_metrics/R5: 40.30100334448161
 MSRVTT_full_test/v2t_metrics/R10: 55.28428093645485
 MSRVTT_full_test/v2t_metrics/R50: 85.21739130434783
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 35.716722408026754
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.337580485914113
 mnt_best       : 27.726645958967108
 not_improved_count: 1
Train Epoch: 38 [1/250 128/32000 (0%)] Loss: 1.08200 (QuantReg: 9.01517) QuantErr: 9.01517 batch_time=38.42698 
Train Epoch: 38 [12/250 1536/32000 (5%)] Loss: 1.06336 (QuantReg: 9.16173) QuantErr: 9.16173 batch_time=0.46894 
Train Epoch: 38 [23/250 2944/32000 (9%)] Loss: 0.82071 (QuantReg: 9.08736) QuantErr: 9.08736 batch_time=0.47985 
Train Epoch: 38 [34/250 4352/32000 (14%)] Loss: 1.05024 (QuantReg: 9.06079) QuantErr: 9.06079 batch_time=0.68177 
Train Epoch: 38 [45/250 5760/32000 (18%)] Loss: 1.07642 (QuantReg: 9.00395) QuantErr: 9.00395 batch_time=0.45398 
Train Epoch: 38 [56/250 7168/32000 (22%)] Loss: 1.20110 (QuantReg: 9.16766) QuantErr: 9.16766 batch_time=0.46335 
Train Epoch: 38 [67/250 8576/32000 (27%)] Loss: 0.95704 (QuantReg: 8.95071) QuantErr: 8.95071 batch_time=0.48420 
Train Epoch: 38 [78/250 9984/32000 (31%)] Loss: 0.79646 (QuantReg: 9.19958) QuantErr: 9.19958 batch_time=0.51035 
Train Epoch: 38 [89/250 11392/32000 (36%)] Loss: 0.80770 (QuantReg: 9.08867) QuantErr: 9.08867 batch_time=0.44612 
Train Epoch: 38 [100/250 12800/32000 (40%)] Loss: 0.86427 (QuantReg: 9.30446) QuantErr: 9.30446 batch_time=0.45505 
Train Epoch: 38 [111/250 14208/32000 (44%)] Loss: 1.07752 (QuantReg: 9.03649) QuantErr: 9.03649 batch_time=0.46223 
Train Epoch: 38 [122/250 15616/32000 (49%)] Loss: 1.10763 (QuantReg: 9.06860) QuantErr: 9.06860 batch_time=0.47625 
Train Epoch: 38 [133/250 17024/32000 (53%)] Loss: 1.02206 (QuantReg: 9.13975) QuantErr: 9.13975 batch_time=0.49247 
Train Epoch: 38 [144/250 18432/32000 (58%)] Loss: 1.02291 (QuantReg: 9.05458) QuantErr: 9.05458 batch_time=1.11611 
Train Epoch: 38 [155/250 19840/32000 (62%)] Loss: 0.80330 (QuantReg: 9.27298) QuantErr: 9.27298 batch_time=0.47421 
Train Epoch: 38 [166/250 21248/32000 (66%)] Loss: 1.12634 (QuantReg: 9.03066) QuantErr: 9.03066 batch_time=0.51254 
Train Epoch: 38 [177/250 22656/32000 (71%)] Loss: 0.96545 (QuantReg: 9.10211) QuantErr: 9.10211 batch_time=0.48356 
Train Epoch: 38 [188/250 24064/32000 (75%)] Loss: 0.90825 (QuantReg: 9.05697) QuantErr: 9.05697 batch_time=0.57183 
Train Epoch: 38 [199/250 25472/32000 (80%)] Loss: 0.99031 (QuantReg: 9.00997) QuantErr: 9.00997 batch_time=0.46890 
Train Epoch: 38 [210/250 26880/32000 (84%)] Loss: 1.02113 (QuantReg: 9.03557) QuantErr: 9.03557 batch_time=2.01506 
Train Epoch: 38 [221/250 28288/32000 (88%)] Loss: 1.05560 (QuantReg: 9.18001) QuantErr: 9.18001 batch_time=0.61519 
Train Epoch: 38 [232/250 29696/32000 (93%)] Loss: 1.00263 (QuantReg: 8.97447) QuantErr: 8.97447 batch_time=0.49357 
Train Epoch: 38 [243/250 31104/32000 (97%)] Loss: 0.91649 (QuantReg: 9.14716) QuantErr: 9.14716 batch_time=0.47233 
Train Epoch: 38 codebook_update_time=1.03638
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch38.pth ...
Done in 6.258s
removing stale ckpt [epoch 37] [took 0.01s]
 epoch          : 38
 loss           : 1.011225662469864
 quant_reg      : 9.07555604171753
 quant_err      : 9.07555604171753
 learning_rate  : 7.494512702440772e-06
 n_samples      : 1216000
 n_steps        : 9500
 MSRVTT_full_val/t2v_metrics/R1: 30.784708249496983
 MSRVTT_full_val/t2v_metrics/R5: 66.19718309859155
 MSRVTT_full_val/t2v_metrics/R10: 78.47082494969818
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.88128772635815
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.278469073294175
 MSRVTT_full_val/v2t_metrics/R1: 37.02213279678068
 MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
 MSRVTT_full_val/v2t_metrics/R10: 84.90945674044265
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.223340040241449
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.790728173582266
 MSRVTT_full_test/t2v_metrics/R1: 12.240802675585284
 MSRVTT_full_test/t2v_metrics/R5: 35.25083612040134
 MSRVTT_full_test/t2v_metrics/R10: 48.49498327759197
 MSRVTT_full_test/t2v_metrics/R50: 79.29765886287625
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 51.04782608695652
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.556583858437783
 MSRVTT_full_test/v2t_metrics/R1: 14.214046822742475
 MSRVTT_full_test/v2t_metrics/R5: 40.30100334448161
 MSRVTT_full_test/v2t_metrics/R10: 54.78260869565217
 MSRVTT_full_test/v2t_metrics/R50: 85.18394648829431
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 35.69264214046823
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.542208859999512
 mnt_best       : 27.726645958967108
 not_improved_count: 2
Train Epoch: 39 [1/250 128/32000 (0%)] Loss: 1.16600 (QuantReg: 9.09825) QuantErr: 9.09825 batch_time=34.06336 
Train Epoch: 39 [12/250 1536/32000 (5%)] Loss: 1.15631 (QuantReg: 8.98433) QuantErr: 8.98433 batch_time=0.45370 
Train Epoch: 39 [23/250 2944/32000 (9%)] Loss: 0.83826 (QuantReg: 9.00655) QuantErr: 9.00655 batch_time=0.47576 
Train Epoch: 39 [34/250 4352/32000 (14%)] Loss: 1.26516 (QuantReg: 9.12936) QuantErr: 9.12936 batch_time=0.60425 
Train Epoch: 39 [45/250 5760/32000 (18%)] Loss: 0.84050 (QuantReg: 8.97103) QuantErr: 8.97103 batch_time=0.67965 
Train Epoch: 39 [56/250 7168/32000 (22%)] Loss: 0.95134 (QuantReg: 9.15799) QuantErr: 9.15799 batch_time=0.46966 
Train Epoch: 39 [67/250 8576/32000 (27%)] Loss: 0.98133 (QuantReg: 8.97648) QuantErr: 8.97648 batch_time=0.87400 
Train Epoch: 39 [78/250 9984/32000 (31%)] Loss: 0.82090 (QuantReg: 9.24136) QuantErr: 9.24136 batch_time=0.47470 
Train Epoch: 39 [89/250 11392/32000 (36%)] Loss: 0.77449 (QuantReg: 9.00185) QuantErr: 9.00185 batch_time=0.51292 
Train Epoch: 39 [100/250 12800/32000 (40%)] Loss: 0.83465 (QuantReg: 9.18028) QuantErr: 9.18028 batch_time=0.44112 
Train Epoch: 39 [111/250 14208/32000 (44%)] Loss: 0.84229 (QuantReg: 8.95897) QuantErr: 8.95897 batch_time=0.44400 
Train Epoch: 39 [122/250 15616/32000 (49%)] Loss: 0.80733 (QuantReg: 9.26059) QuantErr: 9.26059 batch_time=0.46754 
Train Epoch: 39 [133/250 17024/32000 (53%)] Loss: 1.08257 (QuantReg: 9.10891) QuantErr: 9.10891 batch_time=0.46274 
Train Epoch: 39 [144/250 18432/32000 (58%)] Loss: 1.10879 (QuantReg: 8.91849) QuantErr: 8.91849 batch_time=0.49966 
Train Epoch: 39 [155/250 19840/32000 (62%)] Loss: 0.87308 (QuantReg: 8.97188) QuantErr: 8.97188 batch_time=0.49407 
Train Epoch: 39 [166/250 21248/32000 (66%)] Loss: 0.78635 (QuantReg: 9.16352) QuantErr: 9.16352 batch_time=0.44004 
Train Epoch: 39 [177/250 22656/32000 (71%)] Loss: 0.96777 (QuantReg: 9.24489) QuantErr: 9.24489 batch_time=0.50950 
Train Epoch: 39 [188/250 24064/32000 (75%)] Loss: 1.02465 (QuantReg: 9.03643) QuantErr: 9.03643 batch_time=0.44958 
Train Epoch: 39 [199/250 25472/32000 (80%)] Loss: 0.95648 (QuantReg: 9.21037) QuantErr: 9.21037 batch_time=5.08892 
Train Epoch: 39 [210/250 26880/32000 (84%)] Loss: 0.90324 (QuantReg: 9.39441) QuantErr: 9.39441 batch_time=0.47718 
Train Epoch: 39 [221/250 28288/32000 (88%)] Loss: 0.98541 (QuantReg: 9.20092) QuantErr: 9.20092 batch_time=0.45516 
Train Epoch: 39 [232/250 29696/32000 (93%)] Loss: 0.89240 (QuantReg: 9.32067) QuantErr: 9.32067 batch_time=0.44904 
Train Epoch: 39 [243/250 31104/32000 (97%)] Loss: 0.87950 (QuantReg: 9.08625) QuantErr: 9.08625 batch_time=0.46430 
Train Epoch: 39 codebook_update_time=1.08002
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch39.pth ...
Done in 4.513s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch39.pth ...
Done in 9.413s
removing stale ckpt [epoch 38] [took 0.00s]
 epoch          : 39
 loss           : 0.9919865148067475
 quant_reg      : 9.119152667999268
 quant_err      : 9.119152667999268
 learning_rate  : 7.119787067318733e-06
 n_samples      : 1248000
 n_steps        : 9750
 MSRVTT_full_val/t2v_metrics/R1: 32.19315895372233
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
 MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.04225352112676
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.93048262284751
 MSRVTT_full_val/v2t_metrics/R1: 38.028169014084504
 MSRVTT_full_val/v2t_metrics/R5: 75.45271629778672
 MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.855130784708249
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.2587020624744
 MSRVTT_full_test/t2v_metrics/R1: 12.54180602006689
 MSRVTT_full_test/t2v_metrics/R5: 35.752508361204015
 MSRVTT_full_test/t2v_metrics/R10: 48.52842809364549
 MSRVTT_full_test/t2v_metrics/R50: 79.4314381270903
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 51.69113712374582
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.918212386229982
 MSRVTT_full_test/v2t_metrics/R1: 14.013377926421406
 MSRVTT_full_test/v2t_metrics/R5: 40.16722408026756
 MSRVTT_full_test/v2t_metrics/R10: 54.91638795986622
 MSRVTT_full_test/v2t_metrics/R50: 84.68227424749163
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.34230769230769
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.38380085976607
 mnt_best       : 27.918212386229982
 not_improved_count: 0
Train Epoch: 40 [1/250 128/32000 (0%)] Loss: 1.02778 (QuantReg: 9.11184) QuantErr: 9.11184 batch_time=36.32937 
Train Epoch: 40 [12/250 1536/32000 (5%)] Loss: 0.93578 (QuantReg: 9.15800) QuantErr: 9.15800 batch_time=0.45208 
Train Epoch: 40 [23/250 2944/32000 (9%)] Loss: 1.00351 (QuantReg: 9.09822) QuantErr: 9.09822 batch_time=0.52370 
Train Epoch: 40 [34/250 4352/32000 (14%)] Loss: 0.91869 (QuantReg: 9.06121) QuantErr: 9.06121 batch_time=0.45909 
Train Epoch: 40 [45/250 5760/32000 (18%)] Loss: 1.20428 (QuantReg: 8.97962) QuantErr: 8.97962 batch_time=0.50222 
Train Epoch: 40 [56/250 7168/32000 (22%)] Loss: 1.19647 (QuantReg: 9.14206) QuantErr: 9.14206 batch_time=0.50516 
Train Epoch: 40 [67/250 8576/32000 (27%)] Loss: 1.01191 (QuantReg: 9.14435) QuantErr: 9.14435 batch_time=0.43482 
Train Epoch: 40 [78/250 9984/32000 (31%)] Loss: 0.91605 (QuantReg: 9.27273) QuantErr: 9.27273 batch_time=0.46319 
Train Epoch: 40 [89/250 11392/32000 (36%)] Loss: 1.31256 (QuantReg: 8.96439) QuantErr: 8.96439 batch_time=0.78431 
Train Epoch: 40 [100/250 12800/32000 (40%)] Loss: 0.78380 (QuantReg: 9.03781) QuantErr: 9.03781 batch_time=0.43939 
Train Epoch: 40 [111/250 14208/32000 (44%)] Loss: 1.00318 (QuantReg: 9.09278) QuantErr: 9.09278 batch_time=0.45015 
Train Epoch: 40 [122/250 15616/32000 (49%)] Loss: 0.84875 (QuantReg: 9.11146) QuantErr: 9.11146 batch_time=0.44166 
Train Epoch: 40 [133/250 17024/32000 (53%)] Loss: 1.03350 (QuantReg: 9.19460) QuantErr: 9.19460 batch_time=0.46806 
Train Epoch: 40 [144/250 18432/32000 (58%)] Loss: 0.88585 (QuantReg: 9.05431) QuantErr: 9.05431 batch_time=0.43690 
Train Epoch: 40 [155/250 19840/32000 (62%)] Loss: 0.98008 (QuantReg: 9.06107) QuantErr: 9.06107 batch_time=1.04921 
Train Epoch: 40 [166/250 21248/32000 (66%)] Loss: 1.15544 (QuantReg: 9.10765) QuantErr: 9.10765 batch_time=0.46600 
Train Epoch: 40 [177/250 22656/32000 (71%)] Loss: 1.05521 (QuantReg: 9.18531) QuantErr: 9.18531 batch_time=0.46229 
Train Epoch: 40 [188/250 24064/32000 (75%)] Loss: 1.29119 (QuantReg: 9.11617) QuantErr: 9.11617 batch_time=0.43156 
Train Epoch: 40 [199/250 25472/32000 (80%)] Loss: 1.11331 (QuantReg: 9.14528) QuantErr: 9.14528 batch_time=0.47488 
Train Epoch: 40 [210/250 26880/32000 (84%)] Loss: 0.93219 (QuantReg: 9.10757) QuantErr: 9.10757 batch_time=0.45430 
Train Epoch: 40 [221/250 28288/32000 (88%)] Loss: 0.98204 (QuantReg: 9.25586) QuantErr: 9.25586 batch_time=0.51405 
Train Epoch: 40 [232/250 29696/32000 (93%)] Loss: 1.17271 (QuantReg: 9.10085) QuantErr: 9.10085 batch_time=0.53038 
Train Epoch: 40 [243/250 31104/32000 (97%)] Loss: 0.89801 (QuantReg: 8.96004) QuantErr: 8.96004 batch_time=0.45107 
Train Epoch: 40 codebook_update_time=0.93251
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch40.pth ...
Done in 5.188s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch40.pth ...
Done in 9.330s
removing stale ckpt [epoch 39] [took 0.00s]
 epoch          : 40
 loss           : 1.008396326303482
 quant_reg      : 9.118275611877442
 quant_err      : 9.118275611877442
 learning_rate  : 6.763797713952796e-06
 n_samples      : 1280000
 n_steps        : 10000
 MSRVTT_full_val/t2v_metrics/R1: 30.58350100603622
 MSRVTT_full_val/t2v_metrics/R5: 69.01408450704226
 MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.748490945674044
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.19773314983828
 MSRVTT_full_val/v2t_metrics/R1: 37.42454728370221
 MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
 MSRVTT_full_val/v2t_metrics/R10: 85.31187122736418
 MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.774647887323944
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.11161505800582
 MSRVTT_full_test/t2v_metrics/R1: 12.307692307692308
 MSRVTT_full_test/t2v_metrics/R5: 36.08695652173913
 MSRVTT_full_test/t2v_metrics/R10: 49.096989966555185
 MSRVTT_full_test/t2v_metrics/R50: 79.09698996655518
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 52.60133779264214
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.93791025859014
 MSRVTT_full_test/v2t_metrics/R1: 13.879598662207357
 MSRVTT_full_test/v2t_metrics/R5: 39.79933110367893
 MSRVTT_full_test/v2t_metrics/R10: 54.381270903010034
 MSRVTT_full_test/v2t_metrics/R50: 84.78260869565217
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.184615384615384
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.08617909806986
 mnt_best       : 27.93791025859014
 not_improved_count: 0
Train Epoch: 41 [1/250 128/32000 (0%)] Loss: 1.07290 (QuantReg: 8.96534) QuantErr: 8.96534 batch_time=34.76258 
Train Epoch: 41 [12/250 1536/32000 (5%)] Loss: 1.12933 (QuantReg: 9.01997) QuantErr: 9.01997 batch_time=0.57411 
Train Epoch: 41 [23/250 2944/32000 (9%)] Loss: 0.93326 (QuantReg: 9.18076) QuantErr: 9.18076 batch_time=0.42877 
Train Epoch: 41 [34/250 4352/32000 (14%)] Loss: 1.10778 (QuantReg: 9.18580) QuantErr: 9.18580 batch_time=0.44050 
Train Epoch: 41 [45/250 5760/32000 (18%)] Loss: 0.61274 (QuantReg: 9.15997) QuantErr: 9.15997 batch_time=0.47913 
Train Epoch: 41 [56/250 7168/32000 (22%)] Loss: 1.13361 (QuantReg: 8.90361) QuantErr: 8.90361 batch_time=0.50258 
Train Epoch: 41 [67/250 8576/32000 (27%)] Loss: 1.23845 (QuantReg: 8.91298) QuantErr: 8.91298 batch_time=0.48953 
Train Epoch: 41 [78/250 9984/32000 (31%)] Loss: 1.05679 (QuantReg: 9.15566) QuantErr: 9.15566 batch_time=0.45296 
Train Epoch: 41 [89/250 11392/32000 (36%)] Loss: 1.05686 (QuantReg: 9.21696) QuantErr: 9.21696 batch_time=0.43854 
Train Epoch: 41 [100/250 12800/32000 (40%)] Loss: 0.86543 (QuantReg: 9.29383) QuantErr: 9.29383 batch_time=0.49729 
Train Epoch: 41 [111/250 14208/32000 (44%)] Loss: 1.17118 (QuantReg: 9.09667) QuantErr: 9.09667 batch_time=0.45532 
Train Epoch: 41 [122/250 15616/32000 (49%)] Loss: 0.94708 (QuantReg: 9.15178) QuantErr: 9.15178 batch_time=0.51038 
Train Epoch: 41 [133/250 17024/32000 (53%)] Loss: 0.89171 (QuantReg: 9.16054) QuantErr: 9.16054 batch_time=1.05591 
Train Epoch: 41 [144/250 18432/32000 (58%)] Loss: 0.81953 (QuantReg: 9.10156) QuantErr: 9.10156 batch_time=0.46891 
Train Epoch: 41 [155/250 19840/32000 (62%)] Loss: 1.06747 (QuantReg: 9.25834) QuantErr: 9.25834 batch_time=0.47199 
Train Epoch: 41 [166/250 21248/32000 (66%)] Loss: 0.96392 (QuantReg: 9.22620) QuantErr: 9.22620 batch_time=0.47808 
Train Epoch: 41 [177/250 22656/32000 (71%)] Loss: 1.10093 (QuantReg: 9.22092) QuantErr: 9.22092 batch_time=0.67880 
Train Epoch: 41 [188/250 24064/32000 (75%)] Loss: 1.14717 (QuantReg: 9.08788) QuantErr: 9.08788 batch_time=0.49870 
Train Epoch: 41 [199/250 25472/32000 (80%)] Loss: 1.00454 (QuantReg: 9.02510) QuantErr: 9.02510 batch_time=0.52042 
Train Epoch: 41 [210/250 26880/32000 (84%)] Loss: 0.90689 (QuantReg: 9.23651) QuantErr: 9.23651 batch_time=0.44181 
Train Epoch: 41 [221/250 28288/32000 (88%)] Loss: 1.04434 (QuantReg: 9.25846) QuantErr: 9.25846 batch_time=0.44213 
Train Epoch: 41 [232/250 29696/32000 (93%)] Loss: 1.01927 (QuantReg: 9.22390) QuantErr: 9.22390 batch_time=0.46167 
Train Epoch: 41 [243/250 31104/32000 (97%)] Loss: 1.14573 (QuantReg: 9.15606) QuantErr: 9.15606 batch_time=0.60940 
Train Epoch: 41 codebook_update_time=1.07179
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch41.pth ...
Done in 4.192s
removing stale ckpt [epoch 40] [took 0.01s]
 epoch          : 41
 loss           : 1.0207084805965423
 quant_reg      : 9.165190174102783
 quant_err      : 9.165190174102783
 learning_rate  : 6.425607828255156e-06
 n_samples      : 1312000
 n_steps        : 10250
 MSRVTT_full_val/t2v_metrics/R1: 29.979879275653925
 MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
 MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
 MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.911468812877263
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.29404191613553
 MSRVTT_full_val/v2t_metrics/R1: 35.61368209255533
 MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
 MSRVTT_full_val/v2t_metrics/R10: 85.31187122736418
 MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.704225352112676
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.09320786185992
 MSRVTT_full_test/t2v_metrics/R1: 12.408026755852843
 MSRVTT_full_test/t2v_metrics/R5: 35.31772575250836
 MSRVTT_full_test/t2v_metrics/R10: 48.82943143812709
 MSRVTT_full_test/t2v_metrics/R50: 79.53177257525084
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 51.00735785953177
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.76252991618711
 MSRVTT_full_test/v2t_metrics/R1: 14.080267558528428
 MSRVTT_full_test/v2t_metrics/R5: 40.36789297658863
 MSRVTT_full_test/v2t_metrics/R10: 55.18394648829432
 MSRVTT_full_test/v2t_metrics/R50: 84.51505016722408
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.38227424749164
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.536966375119313
 mnt_best       : 27.93791025859014
 not_improved_count: 1
Train Epoch: 42 [1/250 128/32000 (0%)] Loss: 1.00192 (QuantReg: 9.04348) QuantErr: 9.04348 batch_time=32.97017 
Train Epoch: 42 [12/250 1536/32000 (5%)] Loss: 0.97876 (QuantReg: 9.10073) QuantErr: 9.10073 batch_time=0.50531 
Train Epoch: 42 [23/250 2944/32000 (9%)] Loss: 1.01820 (QuantReg: 9.14224) QuantErr: 9.14224 batch_time=0.44362 
Train Epoch: 42 [34/250 4352/32000 (14%)] Loss: 0.89282 (QuantReg: 9.23695) QuantErr: 9.23695 batch_time=0.52025 
Train Epoch: 42 [45/250 5760/32000 (18%)] Loss: 1.18338 (QuantReg: 9.25661) QuantErr: 9.25661 batch_time=0.51093 
Train Epoch: 42 [56/250 7168/32000 (22%)] Loss: 1.01981 (QuantReg: 9.12787) QuantErr: 9.12787 batch_time=0.46743 
Train Epoch: 42 [67/250 8576/32000 (27%)] Loss: 0.94021 (QuantReg: 9.17449) QuantErr: 9.17449 batch_time=0.43259 
Train Epoch: 42 [78/250 9984/32000 (31%)] Loss: 1.15084 (QuantReg: 8.99861) QuantErr: 8.99861 batch_time=0.45818 
Train Epoch: 42 [89/250 11392/32000 (36%)] Loss: 1.03853 (QuantReg: 9.14018) QuantErr: 9.14018 batch_time=0.45670 
Train Epoch: 42 [100/250 12800/32000 (40%)] Loss: 0.87472 (QuantReg: 9.23622) QuantErr: 9.23622 batch_time=0.49937 
Train Epoch: 42 [111/250 14208/32000 (44%)] Loss: 1.20722 (QuantReg: 9.14275) QuantErr: 9.14275 batch_time=0.48827 
Train Epoch: 42 [122/250 15616/32000 (49%)] Loss: 1.13727 (QuantReg: 9.36779) QuantErr: 9.36779 batch_time=0.57933 
Train Epoch: 42 [133/250 17024/32000 (53%)] Loss: 0.94187 (QuantReg: 9.16936) QuantErr: 9.16936 batch_time=0.51007 
Train Epoch: 42 [144/250 18432/32000 (58%)] Loss: 0.96869 (QuantReg: 9.06094) QuantErr: 9.06094 batch_time=0.43091 
Train Epoch: 42 [155/250 19840/32000 (62%)] Loss: 1.20769 (QuantReg: 9.10320) QuantErr: 9.10320 batch_time=0.43663 
Train Epoch: 42 [166/250 21248/32000 (66%)] Loss: 0.93060 (QuantReg: 9.18419) QuantErr: 9.18419 batch_time=0.47714 
Train Epoch: 42 [177/250 22656/32000 (71%)] Loss: 1.28329 (QuantReg: 9.16092) QuantErr: 9.16092 batch_time=0.51386 
Train Epoch: 42 [188/250 24064/32000 (75%)] Loss: 0.90864 (QuantReg: 9.12879) QuantErr: 9.12879 batch_time=0.46282 
Train Epoch: 42 [199/250 25472/32000 (80%)] Loss: 0.99522 (QuantReg: 9.31187) QuantErr: 9.31187 batch_time=0.50936 
Train Epoch: 42 [210/250 26880/32000 (84%)] Loss: 1.05320 (QuantReg: 9.14883) QuantErr: 9.14883 batch_time=0.44538 
Train Epoch: 42 [221/250 28288/32000 (88%)] Loss: 0.81581 (QuantReg: 9.22781) QuantErr: 9.22781 batch_time=0.43700 
Train Epoch: 42 [232/250 29696/32000 (93%)] Loss: 1.08187 (QuantReg: 9.20323) QuantErr: 9.20323 batch_time=0.43582 
Train Epoch: 42 [243/250 31104/32000 (97%)] Loss: 0.82093 (QuantReg: 9.23107) QuantErr: 9.23107 batch_time=0.46518 
Train Epoch: 42 codebook_update_time=0.88485
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch42.pth ...
Done in 3.940s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch42.pth ...
Done in 7.851s
removing stale ckpt [epoch 41] [took 0.00s]
 epoch          : 42
 loss           : 1.0119447643756867
 quant_reg      : 9.163893898010254
 quant_err      : 9.163893898010254
 learning_rate  : 6.104327436842398e-06
 n_samples      : 1344000
 n_steps        : 10500
 MSRVTT_full_val/t2v_metrics/R1: 32.796780684104625
 MSRVTT_full_val/t2v_metrics/R5: 68.81287726358148
 MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
 MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.04225352112676
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.44340997912012
 MSRVTT_full_val/v2t_metrics/R1: 36.82092555331992
 MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
 MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.062374245472837
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.31781109224289
 MSRVTT_full_test/t2v_metrics/R1: 12.37458193979933
 MSRVTT_full_test/t2v_metrics/R5: 36.15384615384615
 MSRVTT_full_test/t2v_metrics/R10: 50.03344481605351
 MSRVTT_full_test/t2v_metrics/R50: 79.76588628762542
 MSRVTT_full_test/t2v_metrics/MedR: 10.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.311705685618726
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.182649243758004
 MSRVTT_full_test/v2t_metrics/R1: 13.879598662207357
 MSRVTT_full_test/v2t_metrics/R5: 41.23745819397993
 MSRVTT_full_test/v2t_metrics/R10: 56.25418060200669
 MSRVTT_full_test/v2t_metrics/R50: 84.54849498327759
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.5247491638796
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.8132374202372
 mnt_best       : 28.182649243758004
 not_improved_count: 0
Train Epoch: 43 [1/250 128/32000 (0%)] Loss: 1.03830 (QuantReg: 9.16408) QuantErr: 9.16408 batch_time=41.90753 
Train Epoch: 43 [12/250 1536/32000 (5%)] Loss: 1.12515 (QuantReg: 9.22907) QuantErr: 9.22907 batch_time=0.46664 
Train Epoch: 43 [23/250 2944/32000 (9%)] Loss: 0.98814 (QuantReg: 9.12829) QuantErr: 9.12829 batch_time=0.48846 
Train Epoch: 43 [34/250 4352/32000 (14%)] Loss: 0.90204 (QuantReg: 9.18506) QuantErr: 9.18506 batch_time=0.44974 
Train Epoch: 43 [45/250 5760/32000 (18%)] Loss: 0.95011 (QuantReg: 9.16769) QuantErr: 9.16769 batch_time=0.46266 
Train Epoch: 43 [56/250 7168/32000 (22%)] Loss: 1.03505 (QuantReg: 9.37173) QuantErr: 9.37173 batch_time=0.55227 
Train Epoch: 43 [67/250 8576/32000 (27%)] Loss: 0.94424 (QuantReg: 9.11081) QuantErr: 9.11081 batch_time=0.84712 
Train Epoch: 43 [78/250 9984/32000 (31%)] Loss: 0.98152 (QuantReg: 9.22435) QuantErr: 9.22435 batch_time=0.51769 
Train Epoch: 43 [89/250 11392/32000 (36%)] Loss: 0.99659 (QuantReg: 9.11517) QuantErr: 9.11517 batch_time=0.45826 
Train Epoch: 43 [100/250 12800/32000 (40%)] Loss: 1.02526 (QuantReg: 9.06020) QuantErr: 9.06020 batch_time=0.48779 
Train Epoch: 43 [111/250 14208/32000 (44%)] Loss: 1.23346 (QuantReg: 9.18477) QuantErr: 9.18477 batch_time=0.44804 
Train Epoch: 43 [122/250 15616/32000 (49%)] Loss: 1.35266 (QuantReg: 9.06765) QuantErr: 9.06765 batch_time=0.48131 
Train Epoch: 43 [133/250 17024/32000 (53%)] Loss: 1.07749 (QuantReg: 9.33014) QuantErr: 9.33014 batch_time=0.44595 
Train Epoch: 43 [144/250 18432/32000 (58%)] Loss: 0.84004 (QuantReg: 9.16342) QuantErr: 9.16342 batch_time=0.44921 
Train Epoch: 43 [155/250 19840/32000 (62%)] Loss: 1.15193 (QuantReg: 9.24234) QuantErr: 9.24234 batch_time=0.46155 
Train Epoch: 43 [166/250 21248/32000 (66%)] Loss: 1.08653 (QuantReg: 9.18618) QuantErr: 9.18618 batch_time=0.44978 
Train Epoch: 43 [177/250 22656/32000 (71%)] Loss: 0.91656 (QuantReg: 9.04385) QuantErr: 9.04385 batch_time=0.44752 
Train Epoch: 43 [188/250 24064/32000 (75%)] Loss: 1.02234 (QuantReg: 9.29624) QuantErr: 9.29624 batch_time=0.53366 
Train Epoch: 43 [199/250 25472/32000 (80%)] Loss: 0.78444 (QuantReg: 9.34814) QuantErr: 9.34814 batch_time=0.47526 
Train Epoch: 43 [210/250 26880/32000 (84%)] Loss: 1.04752 (QuantReg: 9.28515) QuantErr: 9.28515 batch_time=0.79055 
Train Epoch: 43 [221/250 28288/32000 (88%)] Loss: 1.08963 (QuantReg: 9.25718) QuantErr: 9.25718 batch_time=0.45668 
Train Epoch: 43 [232/250 29696/32000 (93%)] Loss: 0.89200 (QuantReg: 9.15814) QuantErr: 9.15814 batch_time=0.45958 
Train Epoch: 43 [243/250 31104/32000 (97%)] Loss: 1.11310 (QuantReg: 9.17736) QuantErr: 9.17736 batch_time=0.46466 
Train Epoch: 43 codebook_update_time=0.93185
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch43.pth ...
Done in 16.053s
removing stale ckpt [epoch 42] [took 0.00s]
 epoch          : 43
 loss           : 1.0078665728569032
 quant_reg      : 9.17196290588379
 quant_err      : 9.17196290588379
 learning_rate  : 5.799111065000278e-06
 n_samples      : 1376000
 n_steps        : 10750
 MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
 MSRVTT_full_val/t2v_metrics/R5: 67.80684104627767
 MSRVTT_full_val/t2v_metrics/R10: 79.47686116700201
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.694164989939637
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.656810937650185
 MSRVTT_full_val/v2t_metrics/R1: 37.625754527162975
 MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.7645875251509056
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.00663696082762
 MSRVTT_full_test/t2v_metrics/R1: 12.307692307692308
 MSRVTT_full_test/t2v_metrics/R5: 35.719063545150505
 MSRVTT_full_test/t2v_metrics/R10: 49.297658862876254
 MSRVTT_full_test/t2v_metrics/R50: 80.0
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 52.25484949832776
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.880528159740116
 MSRVTT_full_test/v2t_metrics/R1: 14.214046822742475
 MSRVTT_full_test/v2t_metrics/R5: 41.17056856187291
 MSRVTT_full_test/v2t_metrics/R10: 56.25418060200669
 MSRVTT_full_test/v2t_metrics/R50: 84.44816053511705
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.272742474916384
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.04939259791418
 mnt_best       : 28.182649243758004
 not_improved_count: 1
Train Epoch: 44 [1/250 128/32000 (0%)] Loss: 0.77280 (QuantReg: 9.36510) QuantErr: 9.36510 batch_time=39.05578 
Train Epoch: 44 [12/250 1536/32000 (5%)] Loss: 0.79930 (QuantReg: 9.10653) QuantErr: 9.10653 batch_time=0.47751 
Train Epoch: 44 [23/250 2944/32000 (9%)] Loss: 0.89500 (QuantReg: 9.20239) QuantErr: 9.20239 batch_time=0.46616 
Train Epoch: 44 [34/250 4352/32000 (14%)] Loss: 1.00180 (QuantReg: 9.15525) QuantErr: 9.15525 batch_time=0.44209 
Train Epoch: 44 [45/250 5760/32000 (18%)] Loss: 1.25268 (QuantReg: 9.15407) QuantErr: 9.15407 batch_time=0.45457 
Train Epoch: 44 [56/250 7168/32000 (22%)] Loss: 0.82163 (QuantReg: 8.93544) QuantErr: 8.93544 batch_time=0.45299 
Train Epoch: 44 [67/250 8576/32000 (27%)] Loss: 0.98331 (QuantReg: 9.28109) QuantErr: 9.28109 batch_time=0.51411 
Train Epoch: 44 [78/250 9984/32000 (31%)] Loss: 1.36051 (QuantReg: 9.26956) QuantErr: 9.26956 batch_time=0.44267 
Train Epoch: 44 [89/250 11392/32000 (36%)] Loss: 1.03093 (QuantReg: 9.29882) QuantErr: 9.29882 batch_time=1.64094 
Train Epoch: 44 [100/250 12800/32000 (40%)] Loss: 0.95989 (QuantReg: 9.15754) QuantErr: 9.15754 batch_time=0.54885 
Train Epoch: 44 [111/250 14208/32000 (44%)] Loss: 1.14105 (QuantReg: 9.46662) QuantErr: 9.46662 batch_time=0.48873 
Train Epoch: 44 [122/250 15616/32000 (49%)] Loss: 1.13858 (QuantReg: 9.16374) QuantErr: 9.16374 batch_time=0.56818 
Train Epoch: 44 [133/250 17024/32000 (53%)] Loss: 0.99285 (QuantReg: 9.20415) QuantErr: 9.20415 batch_time=0.43622 
Train Epoch: 44 [144/250 18432/32000 (58%)] Loss: 1.08211 (QuantReg: 9.06543) QuantErr: 9.06543 batch_time=1.59991 
Train Epoch: 44 [155/250 19840/32000 (62%)] Loss: 1.12003 (QuantReg: 9.27402) QuantErr: 9.27402 batch_time=0.42942 
Train Epoch: 44 [166/250 21248/32000 (66%)] Loss: 0.93993 (QuantReg: 9.31381) QuantErr: 9.31381 batch_time=0.47305 
Train Epoch: 44 [177/250 22656/32000 (71%)] Loss: 0.93823 (QuantReg: 8.95992) QuantErr: 8.95992 batch_time=0.51565 
Train Epoch: 44 [188/250 24064/32000 (75%)] Loss: 0.92030 (QuantReg: 8.95366) QuantErr: 8.95366 batch_time=0.50643 
Train Epoch: 44 [199/250 25472/32000 (80%)] Loss: 1.07948 (QuantReg: 9.09724) QuantErr: 9.09724 batch_time=0.43650 
Train Epoch: 44 [210/250 26880/32000 (84%)] Loss: 1.01081 (QuantReg: 9.28541) QuantErr: 9.28541 batch_time=0.44910 
Train Epoch: 44 [221/250 28288/32000 (88%)] Loss: 1.04683 (QuantReg: 9.16526) QuantErr: 9.16526 batch_time=0.44332 
Train Epoch: 44 [232/250 29696/32000 (93%)] Loss: 0.76835 (QuantReg: 9.11545) QuantErr: 9.11545 batch_time=0.47286 
Train Epoch: 44 [243/250 31104/32000 (97%)] Loss: 0.97019 (QuantReg: 9.30853) QuantErr: 9.30853 batch_time=0.43773 
Train Epoch: 44 codebook_update_time=1.03745
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch44.pth ...
Done in 3.927s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch44.pth ...
Done in 8.034s
removing stale ckpt [epoch 43] [took 0.00s]
 epoch          : 44
 loss           : 1.0124134533405305
 quant_reg      : 9.18615475845337
 quant_err      : 9.18615475845337
 learning_rate  : 5.5091555117502635e-06
 n_samples      : 1408000
 n_steps        : 11000
 MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
 MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
 MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 2.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.305835010060363
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.31221527900868
 MSRVTT_full_val/v2t_metrics/R1: 37.82696177062374
 MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
 MSRVTT_full_val/v2t_metrics/R10: 84.90945674044265
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.776659959758551
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.23528107101639
 MSRVTT_full_test/t2v_metrics/R1: 12.642140468227424
 MSRVTT_full_test/t2v_metrics/R5: 36.020066889632105
 MSRVTT_full_test/t2v_metrics/R10: 49.66555183946488
 MSRVTT_full_test/t2v_metrics/R50: 79.93311036789298
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.535451505016724
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.27961312977395
 MSRVTT_full_test/v2t_metrics/R1: 14.046822742474916
 MSRVTT_full_test/v2t_metrics/R5: 41.50501672240803
 MSRVTT_full_test/v2t_metrics/R10: 56.15384615384615
 MSRVTT_full_test/v2t_metrics/R50: 84.84949832775919
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.66538461538462
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.99038015526766
 mnt_best       : 28.27961312977395
 not_improved_count: 0
Train Epoch: 45 [1/250 128/32000 (0%)] Loss: 1.05622 (QuantReg: 9.09152) QuantErr: 9.09152 batch_time=40.89070 
Train Epoch: 45 [12/250 1536/32000 (5%)] Loss: 0.85972 (QuantReg: 9.31163) QuantErr: 9.31163 batch_time=0.48232 
Train Epoch: 45 [23/250 2944/32000 (9%)] Loss: 1.11343 (QuantReg: 9.20884) QuantErr: 9.20884 batch_time=0.52406 
Train Epoch: 45 [34/250 4352/32000 (14%)] Loss: 0.78971 (QuantReg: 9.30992) QuantErr: 9.30992 batch_time=0.53206 
Train Epoch: 45 [45/250 5760/32000 (18%)] Loss: 1.02223 (QuantReg: 9.20103) QuantErr: 9.20103 batch_time=0.45969 
Train Epoch: 45 [56/250 7168/32000 (22%)] Loss: 0.71594 (QuantReg: 9.19966) QuantErr: 9.19966 batch_time=0.44230 
Train Epoch: 45 [67/250 8576/32000 (27%)] Loss: 1.11766 (QuantReg: 9.09350) QuantErr: 9.09350 batch_time=0.45060 
Train Epoch: 45 [78/250 9984/32000 (31%)] Loss: 1.34334 (QuantReg: 9.09308) QuantErr: 9.09308 batch_time=0.46176 
Train Epoch: 45 [89/250 11392/32000 (36%)] Loss: 1.05872 (QuantReg: 9.17490) QuantErr: 9.17490 batch_time=0.55520 
Train Epoch: 45 [100/250 12800/32000 (40%)] Loss: 0.85612 (QuantReg: 9.06346) QuantErr: 9.06346 batch_time=0.49980 
Train Epoch: 45 [111/250 14208/32000 (44%)] Loss: 1.26523 (QuantReg: 9.28472) QuantErr: 9.28472 batch_time=0.48429 
Train Epoch: 45 [122/250 15616/32000 (49%)] Loss: 1.03970 (QuantReg: 9.18976) QuantErr: 9.18976 batch_time=0.44082 
Train Epoch: 45 [133/250 17024/32000 (53%)] Loss: 1.00070 (QuantReg: 9.28291) QuantErr: 9.28291 batch_time=0.58509 
Train Epoch: 45 [144/250 18432/32000 (58%)] Loss: 0.95541 (QuantReg: 9.26156) QuantErr: 9.26156 batch_time=0.43663 
Train Epoch: 45 [155/250 19840/32000 (62%)] Loss: 1.06687 (QuantReg: 9.25699) QuantErr: 9.25699 batch_time=0.47089 
Train Epoch: 45 [166/250 21248/32000 (66%)] Loss: 0.87155 (QuantReg: 9.15836) QuantErr: 9.15836 batch_time=0.45445 
Train Epoch: 45 [177/250 22656/32000 (71%)] Loss: 0.96701 (QuantReg: 9.22132) QuantErr: 9.22132 batch_time=0.49247 
Train Epoch: 45 [188/250 24064/32000 (75%)] Loss: 1.00440 (QuantReg: 9.35327) QuantErr: 9.35327 batch_time=0.47096 
Train Epoch: 45 [199/250 25472/32000 (80%)] Loss: 0.75649 (QuantReg: 9.39091) QuantErr: 9.39091 batch_time=0.44601 
Train Epoch: 45 [210/250 26880/32000 (84%)] Loss: 1.04535 (QuantReg: 9.13076) QuantErr: 9.13076 batch_time=0.45215 
Train Epoch: 45 [221/250 28288/32000 (88%)] Loss: 1.07334 (QuantReg: 9.19156) QuantErr: 9.19156 batch_time=0.47884 
Train Epoch: 45 [232/250 29696/32000 (93%)] Loss: 0.74661 (QuantReg: 9.26084) QuantErr: 9.26084 batch_time=0.46281 
Train Epoch: 45 [243/250 31104/32000 (97%)] Loss: 1.03993 (QuantReg: 9.07511) QuantErr: 9.07511 batch_time=0.50349 
Train Epoch: 45 codebook_update_time=0.89878
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch45.pth ...
Done in 4.236s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch45.pth ...
Done in 8.316s
removing stale ckpt [epoch 44] [took 0.01s]
 epoch          : 45
 loss           : 0.9981106514930725
 quant_reg      : 9.214768146514892
 quant_err      : 9.214768146514892
 learning_rate  : 5.23369773616275e-06
 n_samples      : 1440000
 n_steps        : 11250
 MSRVTT_full_val/t2v_metrics/R1: 33.40040241448692
 MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
 MSRVTT_full_val/t2v_metrics/R10: 81.48893360160966
 MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.028169014084508
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.70822574274383
 MSRVTT_full_val/v2t_metrics/R1: 37.625754527162975
 MSRVTT_full_val/v2t_metrics/R5: 75.65392354124748
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.0885311871227366
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.19228668228633
 MSRVTT_full_test/t2v_metrics/R1: 12.441471571906355
 MSRVTT_full_test/t2v_metrics/R5: 36.82274247491639
 MSRVTT_full_test/t2v_metrics/R10: 50.13377926421405
 MSRVTT_full_test/t2v_metrics/R50: 79.89966555183946
 MSRVTT_full_test/t2v_metrics/MedR: 10.0
 MSRVTT_full_test/t2v_metrics/MeanR: 52.34214046822743
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.425368854655897
 MSRVTT_full_test/v2t_metrics/R1: 14.381270903010034
 MSRVTT_full_test/v2t_metrics/R5: 40.50167224080268
 MSRVTT_full_test/v2t_metrics/R10: 56.55518394648829
 MSRVTT_full_test/v2t_metrics/R50: 85.08361204013377
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.866889632107025
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.05636052461655
 mnt_best       : 28.425368854655897
 not_improved_count: 0
Train Epoch: 46 [1/250 128/32000 (0%)] Loss: 1.00958 (QuantReg: 9.00410) QuantErr: 9.00410 batch_time=32.99497 
Train Epoch: 46 [12/250 1536/32000 (5%)] Loss: 0.95322 (QuantReg: 9.24702) QuantErr: 9.24702 batch_time=0.43621 
Train Epoch: 46 [23/250 2944/32000 (9%)] Loss: 0.76636 (QuantReg: 9.15665) QuantErr: 9.15665 batch_time=0.45374 
Train Epoch: 46 [34/250 4352/32000 (14%)] Loss: 0.76927 (QuantReg: 9.20652) QuantErr: 9.20652 batch_time=1.33430 
Train Epoch: 46 [45/250 5760/32000 (18%)] Loss: 1.17067 (QuantReg: 9.17765) QuantErr: 9.17765 batch_time=0.45301 
Train Epoch: 46 [56/250 7168/32000 (22%)] Loss: 0.91250 (QuantReg: 9.21777) QuantErr: 9.21777 batch_time=0.44993 
Train Epoch: 46 [67/250 8576/32000 (27%)] Loss: 1.06003 (QuantReg: 9.36395) QuantErr: 9.36395 batch_time=0.47381 
Train Epoch: 46 [78/250 9984/32000 (31%)] Loss: 1.12024 (QuantReg: 9.30069) QuantErr: 9.30069 batch_time=0.46718 
Train Epoch: 46 [89/250 11392/32000 (36%)] Loss: 1.07572 (QuantReg: 9.17721) QuantErr: 9.17721 batch_time=0.43474 
Train Epoch: 46 [100/250 12800/32000 (40%)] Loss: 1.08814 (QuantReg: 9.16302) QuantErr: 9.16302 batch_time=0.44843 
Train Epoch: 46 [111/250 14208/32000 (44%)] Loss: 0.82282 (QuantReg: 9.03750) QuantErr: 9.03750 batch_time=0.46782 
Train Epoch: 46 [122/250 15616/32000 (49%)] Loss: 0.97509 (QuantReg: 9.22095) QuantErr: 9.22095 batch_time=0.43493 
Train Epoch: 46 [133/250 17024/32000 (53%)] Loss: 1.24012 (QuantReg: 9.18087) QuantErr: 9.18087 batch_time=1.52273 
Train Epoch: 46 [144/250 18432/32000 (58%)] Loss: 1.03962 (QuantReg: 9.12592) QuantErr: 9.12592 batch_time=0.42997 
Train Epoch: 46 [155/250 19840/32000 (62%)] Loss: 0.90477 (QuantReg: 9.30225) QuantErr: 9.30225 batch_time=0.43632 
Train Epoch: 46 [166/250 21248/32000 (66%)] Loss: 0.86011 (QuantReg: 9.35922) QuantErr: 9.35922 batch_time=0.44194 
Train Epoch: 46 [177/250 22656/32000 (71%)] Loss: 1.06605 (QuantReg: 9.23131) QuantErr: 9.23131 batch_time=0.45772 
Train Epoch: 46 [188/250 24064/32000 (75%)] Loss: 0.93516 (QuantReg: 9.25604) QuantErr: 9.25604 batch_time=0.44129 
Train Epoch: 46 [199/250 25472/32000 (80%)] Loss: 0.90378 (QuantReg: 9.22864) QuantErr: 9.22864 batch_time=0.95479 
Train Epoch: 46 [210/250 26880/32000 (84%)] Loss: 0.83790 (QuantReg: 9.45299) QuantErr: 9.45299 batch_time=0.48679 
Train Epoch: 46 [221/250 28288/32000 (88%)] Loss: 0.73499 (QuantReg: 9.13873) QuantErr: 9.13873 batch_time=0.44390 
Train Epoch: 46 [232/250 29696/32000 (93%)] Loss: 0.91925 (QuantReg: 9.28317) QuantErr: 9.28317 batch_time=0.46188 
Train Epoch: 46 [243/250 31104/32000 (97%)] Loss: 0.82380 (QuantReg: 9.23661) QuantErr: 9.23661 batch_time=0.44203 
Train Epoch: 46 codebook_update_time=0.98526
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch46.pth ...
Done in 4.352s
removing stale ckpt [epoch 45] [took 0.00s]
 epoch          : 46
 loss           : 1.001091089963913
 quant_reg      : 9.207054256439209
 quant_err      : 9.207054256439209
 learning_rate  : 4.972012849354612e-06
 n_samples      : 1472000
 n_steps        : 11500
 MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
 MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.780684104627767
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.304029980750556
 MSRVTT_full_val/v2t_metrics/R1: 36.21730382293762
 MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
 MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.909456740442656
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.291194990907975
 MSRVTT_full_test/t2v_metrics/R1: 12.006688963210703
 MSRVTT_full_test/t2v_metrics/R5: 35.752508361204015
 MSRVTT_full_test/t2v_metrics/R10: 49.264214046822744
 MSRVTT_full_test/t2v_metrics/R50: 79.36454849498328
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 52.477090301003344
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.653733839704948
 MSRVTT_full_test/v2t_metrics/R1: 14.381270903010034
 MSRVTT_full_test/v2t_metrics/R5: 40.96989966555184
 MSRVTT_full_test/v2t_metrics/R10: 55.31772575250836
 MSRVTT_full_test/v2t_metrics/R50: 84.54849498327759
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.732608695652175
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.94298480149776
 mnt_best       : 28.425368854655897
 not_improved_count: 1
Train Epoch: 47 [1/250 128/32000 (0%)] Loss: 0.94817 (QuantReg: 9.35262) QuantErr: 9.35262 batch_time=34.48957 
Train Epoch: 47 [12/250 1536/32000 (5%)] Loss: 1.15628 (QuantReg: 9.11729) QuantErr: 9.11729 batch_time=0.46097 
Train Epoch: 47 [23/250 2944/32000 (9%)] Loss: 0.95371 (QuantReg: 9.10663) QuantErr: 9.10663 batch_time=0.50787 
Train Epoch: 47 [34/250 4352/32000 (14%)] Loss: 0.92875 (QuantReg: 9.24398) QuantErr: 9.24398 batch_time=0.57408 
Train Epoch: 47 [45/250 5760/32000 (18%)] Loss: 1.09731 (QuantReg: 9.30199) QuantErr: 9.30199 batch_time=0.60405 
Train Epoch: 47 [56/250 7168/32000 (22%)] Loss: 0.88320 (QuantReg: 9.07721) QuantErr: 9.07721 batch_time=0.48574 
Train Epoch: 47 [67/250 8576/32000 (27%)] Loss: 1.28250 (QuantReg: 9.18374) QuantErr: 9.18374 batch_time=0.87174 
Train Epoch: 47 [78/250 9984/32000 (31%)] Loss: 0.87344 (QuantReg: 9.08596) QuantErr: 9.08596 batch_time=0.44652 
Train Epoch: 47 [89/250 11392/32000 (36%)] Loss: 0.92486 (QuantReg: 9.28990) QuantErr: 9.28990 batch_time=0.57660 
Train Epoch: 47 [100/250 12800/32000 (40%)] Loss: 0.84303 (QuantReg: 9.19240) QuantErr: 9.19240 batch_time=0.46434 
Train Epoch: 47 [111/250 14208/32000 (44%)] Loss: 1.21818 (QuantReg: 9.10341) QuantErr: 9.10341 batch_time=0.56416 
Train Epoch: 47 [122/250 15616/32000 (49%)] Loss: 0.84478 (QuantReg: 9.28786) QuantErr: 9.28786 batch_time=0.47249 
Train Epoch: 47 [133/250 17024/32000 (53%)] Loss: 1.05662 (QuantReg: 9.39796) QuantErr: 9.39796 batch_time=0.71044 
Train Epoch: 47 [144/250 18432/32000 (58%)] Loss: 0.96171 (QuantReg: 9.18582) QuantErr: 9.18582 batch_time=0.49149 
Train Epoch: 47 [155/250 19840/32000 (62%)] Loss: 1.10095 (QuantReg: 9.12595) QuantErr: 9.12595 batch_time=0.49546 
Train Epoch: 47 [166/250 21248/32000 (66%)] Loss: 0.97647 (QuantReg: 9.30723) QuantErr: 9.30723 batch_time=0.47037 
Train Epoch: 47 [177/250 22656/32000 (71%)] Loss: 1.26617 (QuantReg: 9.11947) QuantErr: 9.11947 batch_time=0.44177 
Train Epoch: 47 [188/250 24064/32000 (75%)] Loss: 0.86107 (QuantReg: 9.28109) QuantErr: 9.28109 batch_time=0.46193 
Train Epoch: 47 [199/250 25472/32000 (80%)] Loss: 0.95632 (QuantReg: 9.20102) QuantErr: 9.20102 batch_time=5.06592 
Train Epoch: 47 [210/250 26880/32000 (84%)] Loss: 1.01858 (QuantReg: 9.27306) QuantErr: 9.27306 batch_time=0.44788 
Train Epoch: 47 [221/250 28288/32000 (88%)] Loss: 1.15524 (QuantReg: 9.31761) QuantErr: 9.31761 batch_time=0.44812 
Train Epoch: 47 [232/250 29696/32000 (93%)] Loss: 0.99041 (QuantReg: 9.21965) QuantErr: 9.21965 batch_time=0.46662 
Train Epoch: 47 [243/250 31104/32000 (97%)] Loss: 1.01328 (QuantReg: 9.24906) QuantErr: 9.24906 batch_time=0.44975 
Train Epoch: 47 codebook_update_time=0.88626
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch47.pth ...
Done in 4.178s
removing stale ckpt [epoch 46] [took 0.00s]
 epoch          : 47
 loss           : 1.019083097219467
 quant_reg      : 9.226915588378906
 quant_err      : 9.226915588378906
 learning_rate  : 4.723412206886882e-06
 n_samples      : 1504000
 n_steps        : 11750
 MSRVTT_full_val/t2v_metrics/R1: 32.394366197183096
 MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
 MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
 MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.352112676056338
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.76984714738561
 MSRVTT_full_val/v2t_metrics/R1: 35.2112676056338
 MSRVTT_full_val/v2t_metrics/R5: 74.84909456740442
 MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.210261569416499
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.66403674940346
 MSRVTT_full_test/t2v_metrics/R1: 12.307692307692308
 MSRVTT_full_test/t2v_metrics/R5: 36.220735785953174
 MSRVTT_full_test/t2v_metrics/R10: 48.66220735785953
 MSRVTT_full_test/t2v_metrics/R50: 79.19732441471572
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 53.016053511705685
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.889575319111252
 MSRVTT_full_test/v2t_metrics/R1: 14.414715719063546
 MSRVTT_full_test/v2t_metrics/R5: 40.60200668896321
 MSRVTT_full_test/v2t_metrics/R10: 55.919732441471574
 MSRVTT_full_test/v2t_metrics/R50: 85.08361204013377
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.4556856187291
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.986954188298295
 mnt_best       : 28.425368854655897
 not_improved_count: 2
Train Epoch: 48 [1/250 128/32000 (0%)] Loss: 1.15702 (QuantReg: 9.04238) QuantErr: 9.04238 batch_time=37.97103 
Train Epoch: 48 [12/250 1536/32000 (5%)] Loss: 1.04125 (QuantReg: 9.24184) QuantErr: 9.24184 batch_time=0.49331 
Train Epoch: 48 [23/250 2944/32000 (9%)] Loss: 0.79330 (QuantReg: 9.24836) QuantErr: 9.24836 batch_time=0.46557 
Train Epoch: 48 [34/250 4352/32000 (14%)] Loss: 0.94325 (QuantReg: 9.34030) QuantErr: 9.34030 batch_time=0.45552 
Train Epoch: 48 [45/250 5760/32000 (18%)] Loss: 0.96138 (QuantReg: 9.06893) QuantErr: 9.06893 batch_time=0.49995 
Train Epoch: 48 [56/250 7168/32000 (22%)] Loss: 0.93208 (QuantReg: 9.32775) QuantErr: 9.32775 batch_time=0.56047 
Train Epoch: 48 [67/250 8576/32000 (27%)] Loss: 1.06792 (QuantReg: 9.31823) QuantErr: 9.31823 batch_time=0.71754 
Train Epoch: 48 [78/250 9984/32000 (31%)] Loss: 0.89209 (QuantReg: 9.14194) QuantErr: 9.14194 batch_time=1.86407 
Train Epoch: 48 [89/250 11392/32000 (36%)] Loss: 0.92858 (QuantReg: 9.28527) QuantErr: 9.28527 batch_time=0.45056 
Train Epoch: 48 [100/250 12800/32000 (40%)] Loss: 0.90539 (QuantReg: 9.15907) QuantErr: 9.15907 batch_time=0.52613 
Train Epoch: 48 [111/250 14208/32000 (44%)] Loss: 0.89055 (QuantReg: 9.26182) QuantErr: 9.26182 batch_time=0.47020 
Train Epoch: 48 [122/250 15616/32000 (49%)] Loss: 1.08013 (QuantReg: 9.19350) QuantErr: 9.19350 batch_time=0.45710 
Train Epoch: 48 [133/250 17024/32000 (53%)] Loss: 1.04032 (QuantReg: 9.17850) QuantErr: 9.17850 batch_time=0.45756 
Train Epoch: 48 [144/250 18432/32000 (58%)] Loss: 0.98182 (QuantReg: 9.27351) QuantErr: 9.27351 batch_time=0.46049 
Train Epoch: 48 [155/250 19840/32000 (62%)] Loss: 0.90138 (QuantReg: 9.43812) QuantErr: 9.43812 batch_time=0.50308 
Train Epoch: 48 [166/250 21248/32000 (66%)] Loss: 1.24480 (QuantReg: 9.10846) QuantErr: 9.10846 batch_time=0.46850 
Train Epoch: 48 [177/250 22656/32000 (71%)] Loss: 0.82967 (QuantReg: 9.37062) QuantErr: 9.37062 batch_time=0.86905 
Train Epoch: 48 [188/250 24064/32000 (75%)] Loss: 1.06366 (QuantReg: 9.20793) QuantErr: 9.20793 batch_time=0.50165 
Train Epoch: 48 [199/250 25472/32000 (80%)] Loss: 1.18614 (QuantReg: 9.31968) QuantErr: 9.31968 batch_time=0.48373 
Train Epoch: 48 [210/250 26880/32000 (84%)] Loss: 1.07535 (QuantReg: 9.30924) QuantErr: 9.30924 batch_time=0.46335 
Train Epoch: 48 [221/250 28288/32000 (88%)] Loss: 0.95991 (QuantReg: 9.25432) QuantErr: 9.25432 batch_time=0.44409 
Train Epoch: 48 [232/250 29696/32000 (93%)] Loss: 1.29325 (QuantReg: 9.38805) QuantErr: 9.38805 batch_time=0.45852 
Train Epoch: 48 [243/250 31104/32000 (97%)] Loss: 1.20176 (QuantReg: 9.17579) QuantErr: 9.17579 batch_time=0.45129 
Train Epoch: 48 codebook_update_time=0.90079
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch48.pth ...
Done in 15.633s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch48.pth ...
Done in 19.393s
removing stale ckpt [epoch 47] [took 0.00s]
 epoch          : 48
 loss           : 1.0189792993068696
 quant_reg      : 9.240041751861572
 quant_err      : 9.240041751861572
 learning_rate  : 4.487241596542537e-06
 n_samples      : 1536000
 n_steps        : 12000
 MSRVTT_full_val/t2v_metrics/R1: 30.18108651911469
 MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
 MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
 MSRVTT_full_val/t2v_metrics/R50: 96.17706237424547
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.390342052313883
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.834005961445634
 MSRVTT_full_val/v2t_metrics/R1: 37.223340040241446
 MSRVTT_full_val/v2t_metrics/R5: 75.25150905432595
 MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.780684104627767
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.90875733960203
 MSRVTT_full_test/t2v_metrics/R1: 12.709030100334449
 MSRVTT_full_test/t2v_metrics/R5: 36.52173913043478
 MSRVTT_full_test/t2v_metrics/R10: 50.0
 MSRVTT_full_test/t2v_metrics/R50: 79.1638795986622
 MSRVTT_full_test/t2v_metrics/MedR: 10.5
 MSRVTT_full_test/t2v_metrics/MeanR: 52.4190635451505
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.524056469607952
 MSRVTT_full_test/v2t_metrics/R1: 14.816053511705686
 MSRVTT_full_test/v2t_metrics/R5: 41.23745819397993
 MSRVTT_full_test/v2t_metrics/R10: 56.45484949832776
 MSRVTT_full_test/v2t_metrics/R50: 84.38127090301003
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 36.28494983277592
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.55181635382337
 mnt_best       : 28.524056469607952
 not_improved_count: 0
Train Epoch: 49 [1/250 128/32000 (0%)] Loss: 0.83173 (QuantReg: 9.05780) QuantErr: 9.05780 batch_time=41.36377 
Train Epoch: 49 [12/250 1536/32000 (5%)] Loss: 0.88439 (QuantReg: 9.29618) QuantErr: 9.29618 batch_time=0.51797 
Train Epoch: 49 [23/250 2944/32000 (9%)] Loss: 0.95312 (QuantReg: 9.23872) QuantErr: 9.23872 batch_time=0.51433 
Train Epoch: 49 [34/250 4352/32000 (14%)] Loss: 0.93427 (QuantReg: 9.32638) QuantErr: 9.32638 batch_time=0.51798 
Train Epoch: 49 [45/250 5760/32000 (18%)] Loss: 0.83071 (QuantReg: 9.25922) QuantErr: 9.25922 batch_time=0.53450 
Train Epoch: 49 [56/250 7168/32000 (22%)] Loss: 1.19249 (QuantReg: 9.28619) QuantErr: 9.28619 batch_time=0.50169 
Train Epoch: 49 [67/250 8576/32000 (27%)] Loss: 1.06813 (QuantReg: 9.39978) QuantErr: 9.39978 batch_time=0.53446 
Train Epoch: 49 [78/250 9984/32000 (31%)] Loss: 0.95013 (QuantReg: 9.15623) QuantErr: 9.15623 batch_time=0.49509 
Train Epoch: 49 [89/250 11392/32000 (36%)] Loss: 1.11786 (QuantReg: 9.12387) QuantErr: 9.12387 batch_time=0.46488 
Train Epoch: 49 [100/250 12800/32000 (40%)] Loss: 0.91756 (QuantReg: 9.21535) QuantErr: 9.21535 batch_time=0.48614 
Train Epoch: 49 [111/250 14208/32000 (44%)] Loss: 1.04858 (QuantReg: 9.24152) QuantErr: 9.24152 batch_time=0.45901 
Train Epoch: 49 [122/250 15616/32000 (49%)] Loss: 1.20350 (QuantReg: 9.08748) QuantErr: 9.08748 batch_time=0.51067 
Train Epoch: 49 [133/250 17024/32000 (53%)] Loss: 1.20038 (QuantReg: 9.14033) QuantErr: 9.14033 batch_time=0.53007 
Train Epoch: 49 [144/250 18432/32000 (58%)] Loss: 1.00423 (QuantReg: 9.33381) QuantErr: 9.33381 batch_time=0.49761 
Train Epoch: 49 [155/250 19840/32000 (62%)] Loss: 1.01201 (QuantReg: 9.32199) QuantErr: 9.32199 batch_time=0.45346 
Train Epoch: 49 [166/250 21248/32000 (66%)] Loss: 0.98166 (QuantReg: 9.24660) QuantErr: 9.24660 batch_time=0.49094 
Train Epoch: 49 [177/250 22656/32000 (71%)] Loss: 0.85686 (QuantReg: 9.17015) QuantErr: 9.17015 batch_time=0.49814 
Train Epoch: 49 [188/250 24064/32000 (75%)] Loss: 1.05198 (QuantReg: 9.23161) QuantErr: 9.23161 batch_time=0.48802 
Train Epoch: 49 [199/250 25472/32000 (80%)] Loss: 1.03470 (QuantReg: 9.39864) QuantErr: 9.39864 batch_time=4.42239 
Train Epoch: 49 [210/250 26880/32000 (84%)] Loss: 1.10146 (QuantReg: 9.21782) QuantErr: 9.21782 batch_time=0.53747 
Train Epoch: 49 [221/250 28288/32000 (88%)] Loss: 1.08987 (QuantReg: 9.32038) QuantErr: 9.32038 batch_time=0.45811 
Train Epoch: 49 [232/250 29696/32000 (93%)] Loss: 0.73461 (QuantReg: 9.33701) QuantErr: 9.33701 batch_time=0.46061 
Train Epoch: 49 [243/250 31104/32000 (97%)] Loss: 0.94515 (QuantReg: 9.17491) QuantErr: 9.17491 batch_time=0.47707 
Train Epoch: 49 codebook_update_time=0.95092
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch49.pth ...
Done in 4.264s
removing stale ckpt [epoch 48] [took 0.01s]
 epoch          : 49
 loss           : 1.0138564803600312
 quant_reg      : 9.238303939819335
 quant_err      : 9.238303939819335
 learning_rate  : 4.26287951671541e-06
 n_samples      : 1568000
 n_steps        : 12250
 MSRVTT_full_val/t2v_metrics/R1: 32.99798792756539
 MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
 MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
 MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.295774647887324
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.86010812147764
 MSRVTT_full_val/v2t_metrics/R1: 36.82092555331992
 MSRVTT_full_val/v2t_metrics/R5: 74.84909456740442
 MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.951710261569416
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.47704187284462
 MSRVTT_full_test/t2v_metrics/R1: 12.74247491638796
 MSRVTT_full_test/t2v_metrics/R5: 35.919732441471574
 MSRVTT_full_test/t2v_metrics/R10: 49.46488294314381
 MSRVTT_full_test/t2v_metrics/R50: 78.36120401337793
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 53.97558528428094
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.289674815636285
 MSRVTT_full_test/v2t_metrics/R1: 14.013377926421406
 MSRVTT_full_test/v2t_metrics/R5: 41.070234113712374
 MSRVTT_full_test/v2t_metrics/R10: 55.28428093645485
 MSRVTT_full_test/v2t_metrics/R50: 84.21404682274247
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.36772575250836
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.68768844830002
 mnt_best       : 28.524056469607952
 not_improved_count: 1
Train Epoch: 50 [1/250 128/32000 (0%)] Loss: 0.93956 (QuantReg: 9.24461) QuantErr: 9.24461 batch_time=42.84308 
Train Epoch: 50 [12/250 1536/32000 (5%)] Loss: 0.91421 (QuantReg: 9.19634) QuantErr: 9.19634 batch_time=0.49819 
Train Epoch: 50 [23/250 2944/32000 (9%)] Loss: 1.02110 (QuantReg: 9.30737) QuantErr: 9.30737 batch_time=0.69952 
Train Epoch: 50 [34/250 4352/32000 (14%)] Loss: 0.94594 (QuantReg: 9.38431) QuantErr: 9.38431 batch_time=0.44970 
Train Epoch: 50 [45/250 5760/32000 (18%)] Loss: 1.20082 (QuantReg: 9.12655) QuantErr: 9.12655 batch_time=0.47411 
Train Epoch: 50 [56/250 7168/32000 (22%)] Loss: 0.83882 (QuantReg: 9.31012) QuantErr: 9.31012 batch_time=0.50721 
Train Epoch: 50 [67/250 8576/32000 (27%)] Loss: 0.95485 (QuantReg: 9.33434) QuantErr: 9.33434 batch_time=0.45473 
Train Epoch: 50 [78/250 9984/32000 (31%)] Loss: 1.16929 (QuantReg: 9.20540) QuantErr: 9.20540 batch_time=0.49698 
Train Epoch: 50 [89/250 11392/32000 (36%)] Loss: 1.03441 (QuantReg: 9.21520) QuantErr: 9.21520 batch_time=0.46187 
Train Epoch: 50 [100/250 12800/32000 (40%)] Loss: 0.89280 (QuantReg: 9.15619) QuantErr: 9.15619 batch_time=0.44534 
Train Epoch: 50 [111/250 14208/32000 (44%)] Loss: 0.98646 (QuantReg: 9.21636) QuantErr: 9.21636 batch_time=0.44950 
Train Epoch: 50 [122/250 15616/32000 (49%)] Loss: 1.03816 (QuantReg: 9.42493) QuantErr: 9.42493 batch_time=0.47158 
Train Epoch: 50 [133/250 17024/32000 (53%)] Loss: 1.08018 (QuantReg: 9.30183) QuantErr: 9.30183 batch_time=0.44129 
Train Epoch: 50 [144/250 18432/32000 (58%)] Loss: 1.01754 (QuantReg: 9.07010) QuantErr: 9.07010 batch_time=0.49947 
Train Epoch: 50 [155/250 19840/32000 (62%)] Loss: 1.23053 (QuantReg: 9.22077) QuantErr: 9.22077 batch_time=0.45274 
Train Epoch: 50 [166/250 21248/32000 (66%)] Loss: 0.82622 (QuantReg: 9.45850) QuantErr: 9.45850 batch_time=0.45948 
Train Epoch: 50 [177/250 22656/32000 (71%)] Loss: 1.18499 (QuantReg: 9.25737) QuantErr: 9.25737 batch_time=0.47765 
Train Epoch: 50 [188/250 24064/32000 (75%)] Loss: 1.21535 (QuantReg: 9.28868) QuantErr: 9.28868 batch_time=0.45855 
Train Epoch: 50 [199/250 25472/32000 (80%)] Loss: 0.70499 (QuantReg: 9.36482) QuantErr: 9.36482 batch_time=0.44170 
Train Epoch: 50 [210/250 26880/32000 (84%)] Loss: 1.09124 (QuantReg: 9.40586) QuantErr: 9.40586 batch_time=0.89920 
Train Epoch: 50 [221/250 28288/32000 (88%)] Loss: 0.96803 (QuantReg: 9.27297) QuantErr: 9.27297 batch_time=0.42983 
Train Epoch: 50 [232/250 29696/32000 (93%)] Loss: 1.09002 (QuantReg: 9.30131) QuantErr: 9.30131 batch_time=0.50070 
Train Epoch: 50 [243/250 31104/32000 (97%)] Loss: 1.09697 (QuantReg: 9.24830) QuantErr: 9.24830 batch_time=0.45419 
Train Epoch: 50 codebook_update_time=0.95057
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch50.pth ...
Done in 21.418s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/checkpoint-epoch50.pth ...
Done in 25.458s
removing stale ckpt [epoch 49] [took 0.03s]
 epoch          : 50
 loss           : 1.0118906447887421
 quant_reg      : 9.264628273010254
 quant_err      : 9.264628273010254
 learning_rate  : 4.04973554087964e-06
 n_samples      : 1600000
 n_steps        : 12500
 MSRVTT_full_val/t2v_metrics/R1: 33.80281690140845
 MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
 MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
 MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.997987927565394
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.47127605379415
 MSRVTT_full_val/v2t_metrics/R1: 39.03420523138833
 MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
 MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
 MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.334004024144869
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.691100324656325
 MSRVTT_full_test/t2v_metrics/R1: 12.809364548494983
 MSRVTT_full_test/t2v_metrics/R5: 36.45484949832776
 MSRVTT_full_test/t2v_metrics/R10: 50.16722408026756
 MSRVTT_full_test/t2v_metrics/R50: 79.06354515050167
 MSRVTT_full_test/t2v_metrics/MedR: 10.0
 MSRVTT_full_test/t2v_metrics/MeanR: 52.58227424749164
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.613280697640615
 MSRVTT_full_test/v2t_metrics/R1: 14.548494983277592
 MSRVTT_full_test/v2t_metrics/R5: 41.070234113712374
 MSRVTT_full_test/v2t_metrics/R10: 55.852842809364546
 MSRVTT_full_test/v2t_metrics/R50: 84.74916387959867
 MSRVTT_full_test/v2t_metrics/MedR: 8.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.39414715719064
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.19562374173631
 mnt_best       : 28.613280697640615
 not_improved_count: 0
Final evaluation ...
Loading checkpoint from: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/trained_model.pth ...
Ckpt loaded at epoch 50.
Saved similarity matrix (quantize videos) to /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/MSRVTT-test-qv-sims.npy
Saved v2t similarity matrix (quantize texts) to /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/MSRVTT-test-qt-sims.npy
MSRVTT_full_test:
 t2v_metrics/R1/final_eval: 12.809364548494983
 t2v_metrics/R5/final_eval: 36.45484949832776
 t2v_metrics/R10/final_eval: 50.16722408026756
 t2v_metrics/R50/final_eval: 79.06354515050167
 t2v_metrics/MedR/final_eval: 10.0
 t2v_metrics/MeanR/final_eval: 52.58227424749164
 t2v_metrics/geometric_mean_R1-R5-R10/final_eval: 28.613280697640615
 v2t_metrics/R1/final_eval: 14.548494983277592
 v2t_metrics/R5/final_eval: 41.070234113712374
 v2t_metrics/R10/final_eval: 55.852842809364546
 v2t_metrics/R50/final_eval: 84.74916387959867
 v2t_metrics/MedR/final_eval: 8.0
 v2t_metrics/MeanR/final_eval: 37.39414715719064
 v2t_metrics/geometric_mean_R1-R5-R10/final_eval: 32.19562374173631
Best epoch for the monitored metric: 50
Script took 04h17m08s
The best performing ckpt can be found at /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_M16/trained_model.pth