HCQ_MSRVTT_full_t0.1.txt

Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 825.5565114021301 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 56.00535583496094 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 280.3361213207245 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 166.00615072250366 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch0.pth ...
Done in 2.689s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch0.pth ...
Done in 4.603s
 epoch          : 0
 loss           : 0
 learning_rate  : 5e-05
 n_samples      : 0
 n_steps        : 0
 MSRVTT_full_val/t2v_metrics/R1: 0.0
 MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
 MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
 MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
 MSRVTT_full_val/t2v_metrics/MedR: 252.0
 MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
 MSRVTT_full_val/v2t_metrics/R1: 0.0
 MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
 MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
 MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
 MSRVTT_full_val/v2t_metrics/MedR: 243.0
 MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
 MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
 MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
 MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
 MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
 MSRVTT_full_test/t2v_metrics/MedR: 1515.0
 MSRVTT_full_test/t2v_metrics/MeanR: 1498.5565217391304
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
 MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
 MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
 MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
 MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
 MSRVTT_full_test/v2t_metrics/MedR: 1471.5
 MSRVTT_full_test/v2t_metrics/MeanR: 1495.3264214046824
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
 mnt_best       : 0.12154652794863813
 not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.73770 (QuantReg: 22.44554) QuantErr: 22.44554 batch_time=29.43825 
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.71574 (QuantReg: 22.52802) QuantErr: 22.52802 batch_time=1.04319 
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.12892 (QuantReg: 22.61871) QuantErr: 22.61871 batch_time=0.49833 
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 7.01682 (QuantReg: 22.65707) QuantErr: 22.65707 batch_time=1.98183 
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.45240 (QuantReg: 22.64140) QuantErr: 22.64140 batch_time=0.52782 
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.23941 (QuantReg: 22.62227) QuantErr: 22.62227 batch_time=0.50463 
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.21231 (QuantReg: 22.62445) QuantErr: 22.62445 batch_time=0.51893 
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.91538 (QuantReg: 22.62173) QuantErr: 22.62173 batch_time=2.18201 
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.59422 (QuantReg: 22.66101) QuantErr: 22.66101 batch_time=0.49413 
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.74538 (QuantReg: 22.64662) QuantErr: 22.64662 batch_time=0.49244 
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.42443 (QuantReg: 22.67731) QuantErr: 22.67731 batch_time=0.51193 
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.13228 (QuantReg: 22.66573) QuantErr: 22.66573 batch_time=0.49358 
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.11523 (QuantReg: 22.64967) QuantErr: 22.64967 batch_time=0.51159 
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.96043 (QuantReg: 22.66985) QuantErr: 22.66985 batch_time=0.49095 
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 5.00170 (QuantReg: 22.66203) QuantErr: 22.66203 batch_time=0.50808 
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 5.21806 (QuantReg: 22.64056) QuantErr: 22.64056 batch_time=0.48666 
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 5.00095 (QuantReg: 22.63998) QuantErr: 22.63998 batch_time=0.50429 
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.74884 (QuantReg: 22.61730) QuantErr: 22.61730 batch_time=0.48899 
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.80275 (QuantReg: 22.60291) QuantErr: 22.60291 batch_time=0.50597 
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.51238 (QuantReg: 22.61337) QuantErr: 22.61337 batch_time=0.48762 
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.74374 (QuantReg: 22.64421) QuantErr: 22.64421 batch_time=0.49285 
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.59383 (QuantReg: 22.65733) QuantErr: 22.65733 batch_time=1.13534 
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.27732 (QuantReg: 22.61850) QuantErr: 22.61850 batch_time=0.52516 
Train Epoch: 1 codebook_update_time=2.06193
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch1.pth ...
Done in 4.082s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch1.pth ...
Done in 8.051s
 epoch          : 1
 loss           : 5.572731321334839
 quant_reg      : 22.626732887268066
 quant_err      : 22.626732887268066
 learning_rate  : 5e-05
 n_samples      : 32000
 n_steps        : 250
 MSRVTT_full_val/t2v_metrics/R1: 17.10261569416499
 MSRVTT_full_val/t2v_metrics/R5: 47.88732394366197
 MSRVTT_full_val/t2v_metrics/R10: 64.98993963782696
 MSRVTT_full_val/t2v_metrics/R50: 94.56740442655935
 MSRVTT_full_val/t2v_metrics/MedR: 6.0
 MSRVTT_full_val/t2v_metrics/MeanR: 14.891348088531187
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 37.61632923755842
 MSRVTT_full_val/v2t_metrics/R1: 19.114688128772634
 MSRVTT_full_val/v2t_metrics/R5: 54.12474849094568
 MSRVTT_full_val/v2t_metrics/R10: 68.61167002012073
 MSRVTT_full_val/v2t_metrics/R50: 94.36619718309859
 MSRVTT_full_val/v2t_metrics/MedR: 5.0
 MSRVTT_full_val/v2t_metrics/MeanR: 13.03420523138833
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 41.40508679467761
 MSRVTT_full_test/t2v_metrics/R1: 5.48494983277592
 MSRVTT_full_test/t2v_metrics/R5: 18.327759197324415
 MSRVTT_full_test/t2v_metrics/R10: 28.896321070234112
 MSRVTT_full_test/t2v_metrics/R50: 64.31438127090301
 MSRVTT_full_test/t2v_metrics/MedR: 27.0
 MSRVTT_full_test/t2v_metrics/MeanR: 78.23979933110368
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 14.268386382178925
 MSRVTT_full_test/v2t_metrics/R1: 6.454849498327759
 MSRVTT_full_test/v2t_metrics/R5: 21.505016722408026
 MSRVTT_full_test/v2t_metrics/R10: 32.374581939799334
 MSRVTT_full_test/v2t_metrics/R50: 68.19397993311037
 MSRVTT_full_test/v2t_metrics/MedR: 23.0
 MSRVTT_full_test/v2t_metrics/MeanR: 69.4685618729097
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.5022574452145
 mnt_best       : 14.268386382178925
 not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.09026 (QuantReg: 11.12673) QuantErr: 11.12673 batch_time=39.91298 
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.29285 (QuantReg: 11.10795) QuantErr: 11.10795 batch_time=0.53410 
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.21585 (QuantReg: 11.39646) QuantErr: 11.39646 batch_time=0.52495 
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.09998 (QuantReg: 11.52046) QuantErr: 11.52046 batch_time=0.50162 
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.15817 (QuantReg: 11.67440) QuantErr: 11.67440 batch_time=0.49016 
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.08800 (QuantReg: 11.34377) QuantErr: 11.34377 batch_time=0.50613 
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.40115 (QuantReg: 11.81781) QuantErr: 11.81781 batch_time=0.51506 
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.01411 (QuantReg: 11.80146) QuantErr: 11.80146 batch_time=0.51150 
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.47492 (QuantReg: 11.98538) QuantErr: 11.98538 batch_time=0.48719 
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.94307 (QuantReg: 11.95199) QuantErr: 11.95199 batch_time=0.49832 
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.94128 (QuantReg: 11.86184) QuantErr: 11.86184 batch_time=0.48847 
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.81122 (QuantReg: 11.91259) QuantErr: 11.91259 batch_time=0.71582 
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.63881 (QuantReg: 11.97691) QuantErr: 11.97691 batch_time=0.53155 
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.85640 (QuantReg: 12.16196) QuantErr: 12.16196 batch_time=0.54048 
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.68790 (QuantReg: 12.32705) QuantErr: 12.32705 batch_time=0.53406 
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.77296 (QuantReg: 12.71255) QuantErr: 12.71255 batch_time=0.55122 
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.86028 (QuantReg: 12.35546) QuantErr: 12.35546 batch_time=0.52024 
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.89819 (QuantReg: 12.63291) QuantErr: 12.63291 batch_time=0.52697 
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.50343 (QuantReg: 12.29199) QuantErr: 12.29199 batch_time=0.49445 
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.61492 (QuantReg: 12.78368) QuantErr: 12.78368 batch_time=0.50620 
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.34665 (QuantReg: 12.64008) QuantErr: 12.64008 batch_time=0.54852 
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 4.23856 (QuantReg: 12.53615) QuantErr: 12.53615 batch_time=0.49521 
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.88702 (QuantReg: 13.02143) QuantErr: 13.02143 batch_time=0.50886 
Train Epoch: 2 codebook_update_time=1.77663
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch2.pth ...
Done in 21.386s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch2.pth ...
Done in 26.864s
removing stale ckpt [epoch 1] [took 0.36s]
removing stale ckpt [epoch 0] [took 0.12s]
 epoch          : 2
 loss           : 3.952354695320129
 quant_reg      : 12.031916660308838
 quant_err      : 12.031916660308838
 learning_rate  : 4.75e-05
 n_samples      : 64000
 n_steps        : 500
 MSRVTT_full_val/t2v_metrics/R1: 20.120724346076457
 MSRVTT_full_val/t2v_metrics/R5: 54.32595573440644
 MSRVTT_full_val/t2v_metrics/R10: 70.82494969818913
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 11.736418511066399
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 42.61989942043538
 MSRVTT_full_val/v2t_metrics/R1: 25.35211267605634
 MSRVTT_full_val/v2t_metrics/R5: 60.160965794768615
 MSRVTT_full_val/v2t_metrics/R10: 78.47082494969818
 MSRVTT_full_val/v2t_metrics/R50: 95.97585513078471
 MSRVTT_full_val/v2t_metrics/MedR: 4.0
 MSRVTT_full_val/v2t_metrics/MeanR: 10.362173038229376
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 49.28094850891742
 MSRVTT_full_test/t2v_metrics/R1: 7.3244147157190636
 MSRVTT_full_test/t2v_metrics/R5: 23.244147157190636
 MSRVTT_full_test/t2v_metrics/R10: 36.32107023411371
 MSRVTT_full_test/t2v_metrics/R50: 71.00334448160535
 MSRVTT_full_test/t2v_metrics/MedR: 20.0
 MSRVTT_full_test/t2v_metrics/MeanR: 61.6494983277592
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.354746603869867
 MSRVTT_full_test/v2t_metrics/R1: 8.729096989966555
 MSRVTT_full_test/v2t_metrics/R5: 26.923076923076923
 MSRVTT_full_test/v2t_metrics/R10: 40.0
 MSRVTT_full_test/v2t_metrics/R50: 75.01672240802675
 MSRVTT_full_test/v2t_metrics/MedR: 16.0
 MSRVTT_full_test/v2t_metrics/MeanR: 52.44046822742475
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.10496651594879
 mnt_best       : 18.354746603869867
 not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.37737 (QuantReg: 10.06279) QuantErr: 10.06279 batch_time=30.63429 
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.68980 (QuantReg: 10.37116) QuantErr: 10.37116 batch_time=0.52137 
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.97268 (QuantReg: 10.54015) QuantErr: 10.54015 batch_time=0.49641 
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.29520 (QuantReg: 9.73119) QuantErr: 9.73119 batch_time=0.57759 
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.64717 (QuantReg: 10.25964) QuantErr: 10.25964 batch_time=0.51855 
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.17138 (QuantReg: 10.34797) QuantErr: 10.34797 batch_time=0.52937 
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.32736 (QuantReg: 10.43727) QuantErr: 10.43727 batch_time=1.74391 
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.55277 (QuantReg: 10.26670) QuantErr: 10.26670 batch_time=0.50670 
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.41212 (QuantReg: 10.19809) QuantErr: 10.19809 batch_time=0.52065 
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.27094 (QuantReg: 10.26163) QuantErr: 10.26163 batch_time=0.50218 
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.44696 (QuantReg: 10.30338) QuantErr: 10.30338 batch_time=0.53134 
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.04033 (QuantReg: 10.47595) QuantErr: 10.47595 batch_time=0.51319 
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.55742 (QuantReg: 10.77750) QuantErr: 10.77750 batch_time=0.50156 
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.39430 (QuantReg: 10.77383) QuantErr: 10.77383 batch_time=0.50735 
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.01198 (QuantReg: 10.57118) QuantErr: 10.57118 batch_time=0.53221 
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.93611 (QuantReg: 11.03334) QuantErr: 11.03334 batch_time=0.49669 
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.97524 (QuantReg: 10.60830) QuantErr: 10.60830 batch_time=0.49548 
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.59445 (QuantReg: 10.55796) QuantErr: 10.55796 batch_time=0.52496 
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.09915 (QuantReg: 10.54907) QuantErr: 10.54907 batch_time=0.55437 
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.41932 (QuantReg: 10.78624) QuantErr: 10.78624 batch_time=0.49309 
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.08494 (QuantReg: 10.87619) QuantErr: 10.87619 batch_time=0.51717 
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.91045 (QuantReg: 10.87408) QuantErr: 10.87408 batch_time=0.52304 
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.16005 (QuantReg: 10.90548) QuantErr: 10.90548 batch_time=0.51729 
Train Epoch: 3 codebook_update_time=1.66114
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch3.pth ...
Done in 5.387s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch3.pth ...
Done in 10.381s
removing stale ckpt [epoch 2] [took 0.06s]
 epoch          : 3
 loss           : 3.4093528327941893
 quant_reg      : 10.546623703002929
 quant_err      : 10.546623703002929
 learning_rate  : 4.5125e-05
 n_samples      : 96000
 n_steps        : 750
 MSRVTT_full_val/t2v_metrics/R1: 23.541247484909455
 MSRVTT_full_val/t2v_metrics/R5: 58.95372233400403
 MSRVTT_full_val/t2v_metrics/R10: 73.8430583501006
 MSRVTT_full_val/t2v_metrics/R50: 95.97585513078471
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 10.79476861167002
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.79686860402423
 MSRVTT_full_val/v2t_metrics/R1: 25.150905432595575
 MSRVTT_full_val/v2t_metrics/R5: 65.59356136820925
 MSRVTT_full_val/v2t_metrics/R10: 80.28169014084507
 MSRVTT_full_val/v2t_metrics/R50: 96.78068410462777
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 9.112676056338028
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 50.973420918388776
 MSRVTT_full_test/t2v_metrics/R1: 8.82943143812709
 MSRVTT_full_test/t2v_metrics/R5: 26.287625418060202
 MSRVTT_full_test/t2v_metrics/R10: 38.896321070234116
 MSRVTT_full_test/t2v_metrics/R50: 73.87959866220736
 MSRVTT_full_test/t2v_metrics/MedR: 17.0
 MSRVTT_full_test/t2v_metrics/MeanR: 56.42809364548495
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.822404275177053
 MSRVTT_full_test/v2t_metrics/R1: 9.698996655518394
 MSRVTT_full_test/v2t_metrics/R5: 29.39799331103679
 MSRVTT_full_test/v2t_metrics/R10: 44.147157190635454
 MSRVTT_full_test/v2t_metrics/R50: 78.69565217391305
 MSRVTT_full_test/v2t_metrics/MedR: 14.0
 MSRVTT_full_test/v2t_metrics/MeanR: 49.760200668896324
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.262108643917717
 mnt_best       : 20.822404275177053
 not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.68439 (QuantReg: 9.69773) QuantErr: 9.69773 batch_time=32.54195 
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.26771 (QuantReg: 9.77025) QuantErr: 9.77025 batch_time=0.51274 
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.89415 (QuantReg: 9.81610) QuantErr: 9.81610 batch_time=0.49554 
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.08339 (QuantReg: 9.75113) QuantErr: 9.75113 batch_time=0.72861 
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.17119 (QuantReg: 9.83648) QuantErr: 9.83648 batch_time=0.49895 
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.44674 (QuantReg: 9.95036) QuantErr: 9.95036 batch_time=0.64911 
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.36930 (QuantReg: 9.99620) QuantErr: 9.99620 batch_time=0.48922 
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 3.24072 (QuantReg: 10.01899) QuantErr: 10.01899 batch_time=1.82407 
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.99840 (QuantReg: 9.85213) QuantErr: 9.85213 batch_time=0.49726 
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.01096 (QuantReg: 9.85825) QuantErr: 9.85825 batch_time=1.74516 
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.19425 (QuantReg: 10.04464) QuantErr: 10.04464 batch_time=0.48531 
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.16859 (QuantReg: 9.86044) QuantErr: 9.86044 batch_time=0.52387 
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.01676 (QuantReg: 9.85894) QuantErr: 9.85894 batch_time=0.53163 
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 3.25720 (QuantReg: 10.24677) QuantErr: 10.24677 batch_time=0.55261 
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.18467 (QuantReg: 10.33797) QuantErr: 10.33797 batch_time=0.51991 
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 3.37325 (QuantReg: 10.50442) QuantErr: 10.50442 batch_time=0.50148 
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.34100 (QuantReg: 10.18457) QuantErr: 10.18457 batch_time=0.51430 
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 3.17873 (QuantReg: 10.46721) QuantErr: 10.46721 batch_time=0.50082 
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 3.11876 (QuantReg: 10.16031) QuantErr: 10.16031 batch_time=0.51672 
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.90681 (QuantReg: 10.15817) QuantErr: 10.15817 batch_time=0.48763 
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.87286 (QuantReg: 10.54223) QuantErr: 10.54223 batch_time=1.40970 
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 3.12201 (QuantReg: 10.02069) QuantErr: 10.02069 batch_time=0.50345 
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.94202 (QuantReg: 10.15910) QuantErr: 10.15910 batch_time=0.53462 
Train Epoch: 4 codebook_update_time=1.97003
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch4.pth ...
Done in 4.858s
removing stale ckpt [epoch 3] [took 0.05s]
 epoch          : 4
 loss           : 3.1139764575958253
 quant_reg      : 10.087905838012695
 quant_err      : 10.087905838012695
 learning_rate  : 4.2868749999999995e-05
 n_samples      : 128000
 n_steps        : 1000
 MSRVTT_full_val/t2v_metrics/R1: 22.736418511066397
 MSRVTT_full_val/t2v_metrics/R5: 59.95975855130785
 MSRVTT_full_val/t2v_metrics/R10: 73.44064386317908
 MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 10.774647887323944
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.434360482866055
 MSRVTT_full_val/v2t_metrics/R1: 28.571428571428573
 MSRVTT_full_val/v2t_metrics/R5: 66.59959758551308
 MSRVTT_full_val/v2t_metrics/R10: 79.47686116700201
 MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 8.589537223340042
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 53.27802155244179
 MSRVTT_full_test/t2v_metrics/R1: 8.060200668896321
 MSRVTT_full_test/t2v_metrics/R5: 26.02006688963211
 MSRVTT_full_test/t2v_metrics/R10: 37.95986622073578
 MSRVTT_full_test/t2v_metrics/R50: 73.17725752508362
 MSRVTT_full_test/t2v_metrics/MedR: 18.0
 MSRVTT_full_test/t2v_metrics/MeanR: 58.610033444816054
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.9676204108203
 MSRVTT_full_test/v2t_metrics/R1: 9.498327759197325
 MSRVTT_full_test/v2t_metrics/R5: 30.334448160535118
 MSRVTT_full_test/v2t_metrics/R10: 44.381270903010034
 MSRVTT_full_test/v2t_metrics/R50: 79.66555183946488
 MSRVTT_full_test/v2t_metrics/MedR: 13.0
 MSRVTT_full_test/v2t_metrics/MeanR: 48.73177257525084
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.38447702281215
 mnt_best       : 20.822404275177053
 not_improved_count: 1
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.13990 (QuantReg: 9.88596) QuantErr: 9.88596 batch_time=32.15676 
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.10803 (QuantReg: 9.98583) QuantErr: 9.98583 batch_time=0.52725 
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 3.30469 (QuantReg: 9.74473) QuantErr: 9.74473 batch_time=0.50052 
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.57423 (QuantReg: 9.97383) QuantErr: 9.97383 batch_time=0.53020 
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.93654 (QuantReg: 9.62316) QuantErr: 9.62316 batch_time=0.90989 
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 3.03321 (QuantReg: 10.19886) QuantErr: 10.19886 batch_time=0.50889 
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.85704 (QuantReg: 9.70779) QuantErr: 9.70779 batch_time=1.23671 
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.09554 (QuantReg: 9.61354) QuantErr: 9.61354 batch_time=0.48864 
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.70750 (QuantReg: 9.91764) QuantErr: 9.91764 batch_time=0.50003 
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 3.09703 (QuantReg: 9.80161) QuantErr: 9.80161 batch_time=0.50581 
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.86978 (QuantReg: 9.90972) QuantErr: 9.90972 batch_time=0.53417 
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.87661 (QuantReg: 9.86893) QuantErr: 9.86893 batch_time=0.49897 
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.91789 (QuantReg: 10.18225) QuantErr: 10.18225 batch_time=0.66242 
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.36128 (QuantReg: 10.02870) QuantErr: 10.02870 batch_time=0.82336 
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.84728 (QuantReg: 10.15547) QuantErr: 10.15547 batch_time=0.50563 
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.81316 (QuantReg: 10.07198) QuantErr: 10.07198 batch_time=0.51897 
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.81374 (QuantReg: 10.36751) QuantErr: 10.36751 batch_time=0.49567 
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 3.03981 (QuantReg: 9.84819) QuantErr: 9.84819 batch_time=0.54878 
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.67214 (QuantReg: 10.04933) QuantErr: 10.04933 batch_time=0.50429 
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.96338 (QuantReg: 9.80186) QuantErr: 9.80186 batch_time=0.61354 
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.83187 (QuantReg: 10.02945) QuantErr: 10.02945 batch_time=0.49724 
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.47673 (QuantReg: 10.10045) QuantErr: 10.10045 batch_time=0.49132 
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.74514 (QuantReg: 9.80353) QuantErr: 9.80353 batch_time=0.67664 
Train Epoch: 5 codebook_update_time=1.66572
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch5.pth ...
Done in 4.582s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch5.pth ...
Done in 10.574s
removing stale ckpt [epoch 4] [took 0.01s]
 epoch          : 5
 loss           : 2.8581816635131836
 quant_reg      : 9.902129875183105
 quant_err      : 9.902129875183105
 learning_rate  : 4.072531249999999e-05
 n_samples      : 160000
 n_steps        : 1250
 MSRVTT_full_val/t2v_metrics/R1: 24.346076458752513
 MSRVTT_full_val/t2v_metrics/R5: 61.77062374245473
 MSRVTT_full_val/t2v_metrics/R10: 76.86116700201207
 MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
 MSRVTT_full_val/t2v_metrics/MedR: 4.0
 MSRVTT_full_val/t2v_metrics/MeanR: 10.269617706237424
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.71237609943349
 MSRVTT_full_val/v2t_metrics/R1: 30.18108651911469
 MSRVTT_full_val/v2t_metrics/R5: 67.20321931589537
 MSRVTT_full_val/v2t_metrics/R10: 81.89134808853119
 MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 8.327967806841047
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.969399118376025
 MSRVTT_full_test/t2v_metrics/R1: 9.063545150501673
 MSRVTT_full_test/t2v_metrics/R5: 28.42809364548495
 MSRVTT_full_test/t2v_metrics/R10: 41.070234113712374
 MSRVTT_full_test/t2v_metrics/R50: 76.08695652173913
 MSRVTT_full_test/t2v_metrics/MedR: 15.5
 MSRVTT_full_test/t2v_metrics/MeanR: 51.94531772575251
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.954539870762147
 MSRVTT_full_test/v2t_metrics/R1: 10.167224080267559
 MSRVTT_full_test/v2t_metrics/R5: 33.17725752508361
 MSRVTT_full_test/v2t_metrics/R10: 47.22408026755853
 MSRVTT_full_test/v2t_metrics/R50: 80.73578595317726
 MSRVTT_full_test/v2t_metrics/MedR: 12.0
 MSRVTT_full_test/v2t_metrics/MeanR: 44.60200668896321
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.16143827105787
 mnt_best       : 21.954539870762147
 not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.99657 (QuantReg: 9.47920) QuantErr: 9.47920 batch_time=38.54350 
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.89840 (QuantReg: 9.21438) QuantErr: 9.21438 batch_time=0.48793 
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.52199 (QuantReg: 9.54584) QuantErr: 9.54584 batch_time=1.16960 
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.63163 (QuantReg: 9.62455) QuantErr: 9.62455 batch_time=0.58165 
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.32571 (QuantReg: 9.48413) QuantErr: 9.48413 batch_time=0.52311 
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.46689 (QuantReg: 9.85377) QuantErr: 9.85377 batch_time=0.49950 
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.50733 (QuantReg: 9.97149) QuantErr: 9.97149 batch_time=0.52004 
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.34401 (QuantReg: 9.63919) QuantErr: 9.63919 batch_time=0.52736 
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 3.00869 (QuantReg: 9.82076) QuantErr: 9.82076 batch_time=0.49763 
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.57704 (QuantReg: 9.75625) QuantErr: 9.75625 batch_time=0.54543 
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.64799 (QuantReg: 9.21709) QuantErr: 9.21709 batch_time=0.50032 
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.53533 (QuantReg: 9.83111) QuantErr: 9.83111 batch_time=0.49501 
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.49441 (QuantReg: 9.65105) QuantErr: 9.65105 batch_time=0.51058 
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.73912 (QuantReg: 9.49064) QuantErr: 9.49064 batch_time=0.51246 
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.39400 (QuantReg: 9.98860) QuantErr: 9.98860 batch_time=0.74026 
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.43133 (QuantReg: 9.76527) QuantErr: 9.76527 batch_time=0.51600 
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.81697 (QuantReg: 10.02618) QuantErr: 10.02618 batch_time=0.49831 
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.58350 (QuantReg: 10.29776) QuantErr: 10.29776 batch_time=0.49505 
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.59959 (QuantReg: 9.70841) QuantErr: 9.70841 batch_time=0.49433 
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.48245 (QuantReg: 9.88706) QuantErr: 9.88706 batch_time=1.29983 
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.45225 (QuantReg: 9.72032) QuantErr: 9.72032 batch_time=0.49378 
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.69947 (QuantReg: 9.79434) QuantErr: 9.79434 batch_time=0.48545 
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.70364 (QuantReg: 9.82782) QuantErr: 9.82782 batch_time=0.53104 
Train Epoch: 6 codebook_update_time=1.76751
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch6.pth ...
Done in 5.788s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch6.pth ...
Done in 10.404s
removing stale ckpt [epoch 5] [took 0.00s]
 epoch          : 6
 loss           : 2.6766875925064086
 quant_reg      : 9.727334255218507
 quant_err      : 9.727334255218507
 learning_rate  : 3.868904687499999e-05
 n_samples      : 192000
 n_steps        : 1500
 MSRVTT_full_val/t2v_metrics/R1: 26.156941649899398
 MSRVTT_full_val/t2v_metrics/R5: 62.57545271629779
 MSRVTT_full_val/t2v_metrics/R10: 76.65995975855131
 MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.537223340040242
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.06334344258597
 MSRVTT_full_val/v2t_metrics/R1: 32.99798792756539
 MSRVTT_full_val/v2t_metrics/R5: 69.01408450704226
 MSRVTT_full_val/v2t_metrics/R10: 81.69014084507042
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.804828973843058
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.08626271287999
 MSRVTT_full_test/t2v_metrics/R1: 9.966555183946488
 MSRVTT_full_test/t2v_metrics/R5: 29.464882943143813
 MSRVTT_full_test/t2v_metrics/R10: 42.408026755852845
 MSRVTT_full_test/t2v_metrics/R50: 76.88963210702342
 MSRVTT_full_test/t2v_metrics/MedR: 14.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.54648829431438
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.17924505911893
 MSRVTT_full_test/v2t_metrics/R1: 11.638795986622073
 MSRVTT_full_test/v2t_metrics/R5: 33.84615384615385
 MSRVTT_full_test/v2t_metrics/R10: 47.92642140468227
 MSRVTT_full_test/v2t_metrics/R50: 81.53846153846153
 MSRVTT_full_test/v2t_metrics/MedR: 12.0
 MSRVTT_full_test/v2t_metrics/MeanR: 42.76454849498328
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.627524561701808
 mnt_best       : 23.17924505911893
 not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.59162 (QuantReg: 9.07832) QuantErr: 9.07832 batch_time=31.68241 
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.37440 (QuantReg: 9.81546) QuantErr: 9.81546 batch_time=0.50044 
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.71263 (QuantReg: 9.83595) QuantErr: 9.83595 batch_time=0.49382 
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.54326 (QuantReg: 9.66266) QuantErr: 9.66266 batch_time=0.49237 
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.43478 (QuantReg: 9.30049) QuantErr: 9.30049 batch_time=0.50629 
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.65263 (QuantReg: 9.40894) QuantErr: 9.40894 batch_time=0.50007 
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.64846 (QuantReg: 9.66359) QuantErr: 9.66359 batch_time=0.49264 
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.89611 (QuantReg: 9.65736) QuantErr: 9.65736 batch_time=0.55162 
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.66627 (QuantReg: 9.72351) QuantErr: 9.72351 batch_time=1.24448 
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.52217 (QuantReg: 9.71109) QuantErr: 9.71109 batch_time=0.54137 
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.74051 (QuantReg: 9.65954) QuantErr: 9.65954 batch_time=0.49147 
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.15028 (QuantReg: 9.95643) QuantErr: 9.95643 batch_time=0.50506 
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.36845 (QuantReg: 9.54147) QuantErr: 9.54147 batch_time=0.49604 
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.59382 (QuantReg: 9.58740) QuantErr: 9.58740 batch_time=0.52728 
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.54196 (QuantReg: 9.35888) QuantErr: 9.35888 batch_time=0.52925 
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.44788 (QuantReg: 9.62339) QuantErr: 9.62339 batch_time=0.50715 
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 2.24542 (QuantReg: 9.75147) QuantErr: 9.75147 batch_time=0.73058 
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.54757 (QuantReg: 9.32264) QuantErr: 9.32264 batch_time=0.52059 
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.05029 (QuantReg: 9.72981) QuantErr: 9.72981 batch_time=0.71408 
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.32787 (QuantReg: 9.66348) QuantErr: 9.66348 batch_time=0.51769 
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.93216 (QuantReg: 9.72155) QuantErr: 9.72155 batch_time=0.49941 
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.49091 (QuantReg: 9.55352) QuantErr: 9.55352 batch_time=0.54838 
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.58999 (QuantReg: 9.56586) QuantErr: 9.56586 batch_time=0.49734 
Train Epoch: 7 codebook_update_time=1.74190
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch7.pth ...
Done in 5.485s
removing stale ckpt [epoch 6] [took 0.00s]
 epoch          : 7
 loss           : 2.5298948454856873
 quant_reg      : 9.687481285095215
 quant_err      : 9.687481285095215
 learning_rate  : 3.675459453124999e-05
 n_samples      : 224000
 n_steps        : 1750
 MSRVTT_full_val/t2v_metrics/R1: 29.175050301810867
 MSRVTT_full_val/t2v_metrics/R5: 63.78269617706238
 MSRVTT_full_val/t2v_metrics/R10: 78.67203219315896
 MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.179074446680081
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.70416679036259
 MSRVTT_full_val/v2t_metrics/R1: 29.979879275653925
 MSRVTT_full_val/v2t_metrics/R5: 71.42857142857143
 MSRVTT_full_val/v2t_metrics/R10: 82.29376257545272
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.698189134808853
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.06471099313647
 MSRVTT_full_test/t2v_metrics/R1: 9.130434782608695
 MSRVTT_full_test/t2v_metrics/R5: 29.096989966555185
 MSRVTT_full_test/t2v_metrics/R10: 42.74247491638796
 MSRVTT_full_test/t2v_metrics/R50: 77.72575250836121
 MSRVTT_full_test/t2v_metrics/MedR: 15.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.48227424749164
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.476726592132223
 MSRVTT_full_test/v2t_metrics/R1: 12.240802675585284
 MSRVTT_full_test/v2t_metrics/R5: 35.986622073578594
 MSRVTT_full_test/v2t_metrics/R10: 49.93311036789298
 MSRVTT_full_test/v2t_metrics/R50: 82.10702341137124
 MSRVTT_full_test/v2t_metrics/MedR: 11.0
 MSRVTT_full_test/v2t_metrics/MeanR: 41.38010033444816
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.018606578993825
 mnt_best       : 23.17924505911893
 not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.46202 (QuantReg: 9.71058) QuantErr: 9.71058 batch_time=29.70550 
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.64801 (QuantReg: 9.81532) QuantErr: 9.81532 batch_time=0.49961 
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.83141 (QuantReg: 9.47721) QuantErr: 9.47721 batch_time=0.49697 
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.42982 (QuantReg: 9.79024) QuantErr: 9.79024 batch_time=0.71522 
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.71418 (QuantReg: 9.73182) QuantErr: 9.73182 batch_time=0.49351 
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.59098 (QuantReg: 9.63990) QuantErr: 9.63990 batch_time=0.49915 
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.37611 (QuantReg: 9.99559) QuantErr: 9.99559 batch_time=0.51581 
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.32934 (QuantReg: 9.91954) QuantErr: 9.91954 batch_time=0.49457 
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.71578 (QuantReg: 9.76595) QuantErr: 9.76595 batch_time=1.84108 
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.29398 (QuantReg: 9.40602) QuantErr: 9.40602 batch_time=0.55429 
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.40632 (QuantReg: 9.84028) QuantErr: 9.84028 batch_time=0.51900 
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.75755 (QuantReg: 9.68016) QuantErr: 9.68016 batch_time=0.50176 
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.27802 (QuantReg: 9.69058) QuantErr: 9.69058 batch_time=0.49031 
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.13260 (QuantReg: 9.57693) QuantErr: 9.57693 batch_time=0.74940 
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.73441 (QuantReg: 9.83162) QuantErr: 9.83162 batch_time=0.51683 
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.26931 (QuantReg: 9.55881) QuantErr: 9.55881 batch_time=0.50479 
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.62212 (QuantReg: 9.59399) QuantErr: 9.59399 batch_time=0.49800 
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.66893 (QuantReg: 9.85495) QuantErr: 9.85495 batch_time=0.53638 
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.01026 (QuantReg: 9.73357) QuantErr: 9.73357 batch_time=0.71179 
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.04058 (QuantReg: 9.52713) QuantErr: 9.52713 batch_time=1.36102 
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.05333 (QuantReg: 10.03412) QuantErr: 10.03412 batch_time=0.49152 
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.76531 (QuantReg: 9.52021) QuantErr: 9.52021 batch_time=0.49237 
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.99917 (QuantReg: 9.65403) QuantErr: 9.65403 batch_time=0.50284 
Train Epoch: 8 codebook_update_time=1.67370
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch8.pth ...
Done in 5.718s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch8.pth ...
Done in 10.341s
removing stale ckpt [epoch 7] [took 0.03s]
 epoch          : 8
 loss           : 2.4371680397987365
 quant_reg      : 9.656676395416259
 quant_err      : 9.656676395416259
 learning_rate  : 3.4916864804687486e-05
 n_samples      : 256000
 n_steps        : 2000
 MSRVTT_full_val/t2v_metrics/R1: 23.541247484909455
 MSRVTT_full_val/t2v_metrics/R5: 61.77062374245473
 MSRVTT_full_val/t2v_metrics/R10: 78.47082494969818
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.460764587525151
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.503516496344744
 MSRVTT_full_val/v2t_metrics/R1: 31.58953722334004
 MSRVTT_full_val/v2t_metrics/R5: 68.81287726358148
 MSRVTT_full_val/v2t_metrics/R10: 82.29376257545272
 MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.605633802816901
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.34559466667467
 MSRVTT_full_test/t2v_metrics/R1: 10.267558528428093
 MSRVTT_full_test/t2v_metrics/R5: 29.632107023411372
 MSRVTT_full_test/t2v_metrics/R10: 43.74581939799331
 MSRVTT_full_test/t2v_metrics/R50: 77.59197324414716
 MSRVTT_full_test/t2v_metrics/MedR: 14.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.053177257525086
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.698567335614317
 MSRVTT_full_test/v2t_metrics/R1: 11.605351170568563
 MSRVTT_full_test/v2t_metrics/R5: 34.31438127090301
 MSRVTT_full_test/v2t_metrics/R10: 49.264214046822744
 MSRVTT_full_test/v2t_metrics/R50: 82.876254180602
 MSRVTT_full_test/v2t_metrics/MedR: 11.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.991638795986624
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.970479796031082
 mnt_best       : 23.698567335614317
 not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.51206 (QuantReg: 9.17986) QuantErr: 9.17986 batch_time=33.97367 
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.22183 (QuantReg: 9.33017) QuantErr: 9.33017 batch_time=0.51623 
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.56076 (QuantReg: 9.47258) QuantErr: 9.47258 batch_time=0.48837 
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.57506 (QuantReg: 9.56305) QuantErr: 9.56305 batch_time=0.51932 
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.77858 (QuantReg: 9.45335) QuantErr: 9.45335 batch_time=0.61899 
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.60535 (QuantReg: 9.68318) QuantErr: 9.68318 batch_time=0.50933 
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.31978 (QuantReg: 9.60558) QuantErr: 9.60558 batch_time=1.32197 
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 2.04680 (QuantReg: 9.53398) QuantErr: 9.53398 batch_time=0.51296 
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.54745 (QuantReg: 9.45408) QuantErr: 9.45408 batch_time=0.49749 
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.37399 (QuantReg: 9.60892) QuantErr: 9.60892 batch_time=0.52215 
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.34031 (QuantReg: 9.51572) QuantErr: 9.51572 batch_time=0.73944 
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 2.19973 (QuantReg: 9.72665) QuantErr: 9.72665 batch_time=0.55105 
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.37502 (QuantReg: 9.75116) QuantErr: 9.75116 batch_time=0.52153 
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.45866 (QuantReg: 9.74775) QuantErr: 9.74775 batch_time=1.91571 
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.41220 (QuantReg: 9.99392) QuantErr: 9.99392 batch_time=0.54061 
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 2.49264 (QuantReg: 9.54187) QuantErr: 9.54187 batch_time=0.49666 
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.96751 (QuantReg: 9.72149) QuantErr: 9.72149 batch_time=0.51746 
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.09095 (QuantReg: 9.64550) QuantErr: 9.64550 batch_time=0.51599 
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.05409 (QuantReg: 9.94329) QuantErr: 9.94329 batch_time=0.85304 
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.21573 (QuantReg: 9.44708) QuantErr: 9.44708 batch_time=0.94558 
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.26482 (QuantReg: 9.59301) QuantErr: 9.59301 batch_time=0.51745 
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.30976 (QuantReg: 9.66478) QuantErr: 9.66478 batch_time=0.55678 
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.79804 (QuantReg: 9.63644) QuantErr: 9.63644 batch_time=0.49144 
Train Epoch: 9 codebook_update_time=1.63111
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch9.pth ...
Done in 5.032s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch9.pth ...
Done in 9.885s
removing stale ckpt [epoch 8] [took 0.00s]
 epoch          : 9
 loss           : 2.3159452924728394
 quant_reg      : 9.607510822296142
 quant_err      : 9.607510822296142
 learning_rate  : 3.317102156445311e-05
 n_samples      : 288000
 n_steps        : 2250
 MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
 MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
 MSRVTT_full_val/t2v_metrics/R10: 78.67203219315896
 MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.054325955734406
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.91240907730674
 MSRVTT_full_val/v2t_metrics/R1: 34.40643863179074
 MSRVTT_full_val/v2t_metrics/R5: 70.82494969818913
 MSRVTT_full_val/v2t_metrics/R10: 83.09859154929578
 MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.637826961770624
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.72277266369747
 MSRVTT_full_test/t2v_metrics/R1: 11.37123745819398
 MSRVTT_full_test/t2v_metrics/R5: 30.668896321070235
 MSRVTT_full_test/t2v_metrics/R10: 44.78260869565217
 MSRVTT_full_test/t2v_metrics/R50: 78.66220735785953
 MSRVTT_full_test/t2v_metrics/MedR: 13.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.3685618729097
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.996071306752615
 MSRVTT_full_test/v2t_metrics/R1: 12.675585284280936
 MSRVTT_full_test/v2t_metrics/R5: 36.08695652173913
 MSRVTT_full_test/v2t_metrics/R10: 51.20401337792642
 MSRVTT_full_test/v2t_metrics/R50: 82.47491638795987
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 41.1752508361204
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.611508774992938
 mnt_best       : 24.996071306752615
 not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.96644 (QuantReg: 9.16974) QuantErr: 9.16974 batch_time=29.97284 
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.18232 (QuantReg: 9.59371) QuantErr: 9.59371 batch_time=0.55908 
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.99576 (QuantReg: 9.22052) QuantErr: 9.22052 batch_time=0.51684 
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.00756 (QuantReg: 9.27909) QuantErr: 9.27909 batch_time=0.49169 
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 2.07798 (QuantReg: 9.88936) QuantErr: 9.88936 batch_time=0.55410 
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.60426 (QuantReg: 9.57611) QuantErr: 9.57611 batch_time=0.55188 
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.98219 (QuantReg: 9.61676) QuantErr: 9.61676 batch_time=0.57520 
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 2.11472 (QuantReg: 9.35296) QuantErr: 9.35296 batch_time=0.49493 
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.25513 (QuantReg: 9.55895) QuantErr: 9.55895 batch_time=0.50978 
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.70581 (QuantReg: 9.32677) QuantErr: 9.32677 batch_time=0.55462 
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.01430 (QuantReg: 9.45224) QuantErr: 9.45224 batch_time=0.49704 
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.30217 (QuantReg: 9.66920) QuantErr: 9.66920 batch_time=0.53874 
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.39563 (QuantReg: 9.56509) QuantErr: 9.56509 batch_time=0.51522 
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.64699 (QuantReg: 9.64866) QuantErr: 9.64866 batch_time=0.49500 
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.97913 (QuantReg: 9.77444) QuantErr: 9.77444 batch_time=0.52196 
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.83510 (QuantReg: 9.59297) QuantErr: 9.59297 batch_time=0.48997 
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.16964 (QuantReg: 9.64781) QuantErr: 9.64781 batch_time=0.48649 
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.99683 (QuantReg: 9.80363) QuantErr: 9.80363 batch_time=0.48100 
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.07168 (QuantReg: 9.89488) QuantErr: 9.89488 batch_time=0.98274 
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.00081 (QuantReg: 9.48889) QuantErr: 9.48889 batch_time=0.51529 
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.02262 (QuantReg: 9.31100) QuantErr: 9.31100 batch_time=0.54454 
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.71328 (QuantReg: 9.67448) QuantErr: 9.67448 batch_time=0.50020 
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.28821 (QuantReg: 9.37965) QuantErr: 9.37965 batch_time=0.50599 
Train Epoch: 10 codebook_update_time=1.63599
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch10.pth ...
Done in 14.933s
removing stale ckpt [epoch 9] [took 0.02s]
 epoch          : 10
 loss           : 2.217851896762848
 quant_reg      : 9.5602430267334
 quant_err      : 9.5602430267334
 learning_rate  : 3.151247048623045e-05
 n_samples      : 320000
 n_steps        : 2500
 MSRVTT_full_val/t2v_metrics/R1: 28.772635814889338
 MSRVTT_full_val/t2v_metrics/R5: 63.17907444668008
 MSRVTT_full_val/t2v_metrics/R10: 78.47082494969818
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.509054325955734
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.25008991242564
 MSRVTT_full_val/v2t_metrics/R1: 33.40040241448692
 MSRVTT_full_val/v2t_metrics/R5: 69.61770623742454
 MSRVTT_full_val/v2t_metrics/R10: 82.09255533199195
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.682092555331992
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.57824118704483
 MSRVTT_full_test/t2v_metrics/R1: 9.498327759197325
 MSRVTT_full_test/t2v_metrics/R5: 31.337792642140467
 MSRVTT_full_test/t2v_metrics/R10: 43.64548494983278
 MSRVTT_full_test/t2v_metrics/R50: 78.52842809364549
 MSRVTT_full_test/t2v_metrics/MedR: 14.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.300334448160534
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.50814130478168
 MSRVTT_full_test/v2t_metrics/R1: 11.97324414715719
 MSRVTT_full_test/v2t_metrics/R5: 34.68227424749164
 MSRVTT_full_test/v2t_metrics/R10: 50.06688963210702
 MSRVTT_full_test/v2t_metrics/R50: 82.77591973244147
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 41.23344481605351
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.49729710139384
 mnt_best       : 24.996071306752615
 not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.06770 (QuantReg: 9.47787) QuantErr: 9.47787 batch_time=32.66722 
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.94592 (QuantReg: 9.58068) QuantErr: 9.58068 batch_time=0.50407 
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.15909 (QuantReg: 9.19869) QuantErr: 9.19869 batch_time=0.51872 
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.23761 (QuantReg: 9.70990) QuantErr: 9.70990 batch_time=0.50641 
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.25193 (QuantReg: 9.85109) QuantErr: 9.85109 batch_time=0.51024 
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.08097 (QuantReg: 9.53024) QuantErr: 9.53024 batch_time=0.50658 
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 2.28118 (QuantReg: 9.21422) QuantErr: 9.21422 batch_time=0.49643 
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.93962 (QuantReg: 9.68282) QuantErr: 9.68282 batch_time=0.49809 
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.10501 (QuantReg: 9.49878) QuantErr: 9.49878 batch_time=0.49178 
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.57227 (QuantReg: 9.42266) QuantErr: 9.42266 batch_time=1.01204 
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.35502 (QuantReg: 9.27045) QuantErr: 9.27045 batch_time=0.49666 
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.17776 (QuantReg: 9.77425) QuantErr: 9.77425 batch_time=0.51091 
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.62638 (QuantReg: 9.33208) QuantErr: 9.33208 batch_time=0.51514 
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.35438 (QuantReg: 9.47332) QuantErr: 9.47332 batch_time=0.50555 
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 2.06156 (QuantReg: 9.23397) QuantErr: 9.23397 batch_time=0.49327 
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.09214 (QuantReg: 9.24219) QuantErr: 9.24219 batch_time=0.50331 
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.00233 (QuantReg: 9.27843) QuantErr: 9.27843 batch_time=0.49831 
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.88464 (QuantReg: 9.43280) QuantErr: 9.43280 batch_time=0.49637 
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 2.01603 (QuantReg: 9.95047) QuantErr: 9.95047 batch_time=0.49687 
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 2.35759 (QuantReg: 9.72434) QuantErr: 9.72434 batch_time=0.49294 
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 2.28620 (QuantReg: 9.90346) QuantErr: 9.90346 batch_time=0.49594 
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.99707 (QuantReg: 9.67170) QuantErr: 9.67170 batch_time=0.51454 
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.90150 (QuantReg: 10.02171) QuantErr: 10.02171 batch_time=0.51025 
Train Epoch: 11 codebook_update_time=1.81401
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch11.pth ...
Done in 4.668s
removing stale ckpt [epoch 10] [took 0.00s]
 epoch          : 11
 loss           : 2.139794557094574
 quant_reg      : 9.572949630737305
 quant_err      : 9.572949630737305
 learning_rate  : 2.993684696191893e-05
 n_samples      : 352000
 n_steps        : 2750
 MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
 MSRVTT_full_val/t2v_metrics/R5: 64.38631790744466
 MSRVTT_full_val/t2v_metrics/R10: 78.26961770623743
 MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.619718309859154
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.65799292794433
 MSRVTT_full_val/v2t_metrics/R1: 33.601609657947684
 MSRVTT_full_val/v2t_metrics/R5: 70.22132796780684
 MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.617706237424548
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.188917259404526
 MSRVTT_full_test/t2v_metrics/R1: 10.668896321070234
 MSRVTT_full_test/t2v_metrics/R5: 32.34113712374582
 MSRVTT_full_test/t2v_metrics/R10: 44.381270903010034
 MSRVTT_full_test/t2v_metrics/R50: 78.69565217391305
 MSRVTT_full_test/t2v_metrics/MedR: 13.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.34615384615385
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.83275125698871
 MSRVTT_full_test/v2t_metrics/R1: 11.806020066889632
 MSRVTT_full_test/v2t_metrics/R5: 36.82274247491639
 MSRVTT_full_test/v2t_metrics/R10: 50.468227424749166
 MSRVTT_full_test/v2t_metrics/R50: 83.01003344481606
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.92775919732441
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.99492014746828
 mnt_best       : 24.996071306752615
 not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 2.10512 (QuantReg: 9.47118) QuantErr: 9.47118 batch_time=31.88058 
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 2.25439 (QuantReg: 9.61046) QuantErr: 9.61046 batch_time=1.24664 
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.90563 (QuantReg: 9.32929) QuantErr: 9.32929 batch_time=0.49180 
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.96778 (QuantReg: 9.44614) QuantErr: 9.44614 batch_time=0.49880 
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.77077 (QuantReg: 9.57475) QuantErr: 9.57475 batch_time=0.49278 
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 2.11504 (QuantReg: 9.34071) QuantErr: 9.34071 batch_time=0.49764 
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.79708 (QuantReg: 9.51548) QuantErr: 9.51548 batch_time=0.50828 
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.98440 (QuantReg: 9.48151) QuantErr: 9.48151 batch_time=0.49767 
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.92475 (QuantReg: 9.48985) QuantErr: 9.48985 batch_time=0.50758 
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 2.36933 (QuantReg: 9.49559) QuantErr: 9.49559 batch_time=0.49533 
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.75533 (QuantReg: 9.45672) QuantErr: 9.45672 batch_time=0.49122 
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 2.22941 (QuantReg: 9.46260) QuantErr: 9.46260 batch_time=0.49892 
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.88048 (QuantReg: 9.60844) QuantErr: 9.60844 batch_time=0.50459 
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.83956 (QuantReg: 9.83655) QuantErr: 9.83655 batch_time=2.87316 
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.12114 (QuantReg: 9.49096) QuantErr: 9.49096 batch_time=0.49490 
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.68622 (QuantReg: 9.54963) QuantErr: 9.54963 batch_time=1.46688 
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.81448 (QuantReg: 9.46148) QuantErr: 9.46148 batch_time=0.49026 
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 2.44194 (QuantReg: 9.31620) QuantErr: 9.31620 batch_time=0.50052 
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.30803 (QuantReg: 9.45235) QuantErr: 9.45235 batch_time=0.49535 
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 2.27734 (QuantReg: 9.52746) QuantErr: 9.52746 batch_time=0.49055 
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 2.06120 (QuantReg: 9.09724) QuantErr: 9.09724 batch_time=0.49435 
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.00586 (QuantReg: 9.64898) QuantErr: 9.64898 batch_time=0.50630 
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 2.16629 (QuantReg: 9.91232) QuantErr: 9.91232 batch_time=0.49167 
Train Epoch: 12 codebook_update_time=1.73578
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch12.pth ...
Done in 5.707s
removing stale ckpt [epoch 11] [took 0.01s]
 epoch          : 12
 loss           : 2.072114281654358
 quant_reg      : 9.519542114257813
 quant_err      : 9.519542114257813
 learning_rate  : 2.844000461382298e-05
 n_samples      : 384000
 n_steps        : 3000
 MSRVTT_full_val/t2v_metrics/R1: 27.96780684104628
 MSRVTT_full_val/t2v_metrics/R5: 65.59356136820925
 MSRVTT_full_val/t2v_metrics/R10: 78.06841046277665
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.716297786720322
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.3196688564011
 MSRVTT_full_val/v2t_metrics/R1: 32.99798792756539
 MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
 MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
 MSRVTT_full_val/v2t_metrics/R50: 96.579476861167
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.7364185110663986
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.331337557881824
 MSRVTT_full_test/t2v_metrics/R1: 10.668896321070234
 MSRVTT_full_test/t2v_metrics/R5: 31.17056856187291
 MSRVTT_full_test/t2v_metrics/R10: 43.91304347826087
 MSRVTT_full_test/t2v_metrics/R50: 77.9933110367893
 MSRVTT_full_test/t2v_metrics/MedR: 13.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.94648829431438
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.442891811447314
 MSRVTT_full_test/v2t_metrics/R1: 12.073578595317725
 MSRVTT_full_test/v2t_metrics/R5: 36.25418060200669
 MSRVTT_full_test/v2t_metrics/R10: 51.53846153846154
 MSRVTT_full_test/v2t_metrics/R50: 82.876254180602
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.45535117056856
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.255859296754053
 mnt_best       : 24.996071306752615
 not_improved_count: 3
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.88787 (QuantReg: 9.42522) QuantErr: 9.42522 batch_time=40.06229 
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 2.58840 (QuantReg: 9.02827) QuantErr: 9.02827 batch_time=0.48536 
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 2.26518 (QuantReg: 9.55122) QuantErr: 9.55122 batch_time=0.48332 
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.86520 (QuantReg: 9.52706) QuantErr: 9.52706 batch_time=0.48673 
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.06043 (QuantReg: 9.55785) QuantErr: 9.55785 batch_time=0.48600 
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.02428 (QuantReg: 9.76405) QuantErr: 9.76405 batch_time=0.49660 
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 2.04610 (QuantReg: 9.83495) QuantErr: 9.83495 batch_time=0.49687 
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 2.44406 (QuantReg: 9.75840) QuantErr: 9.75840 batch_time=0.48772 
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.75288 (QuantReg: 9.10587) QuantErr: 9.10587 batch_time=0.48741 
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 2.12261 (QuantReg: 9.53263) QuantErr: 9.53263 batch_time=0.51636 
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 2.06262 (QuantReg: 9.41004) QuantErr: 9.41004 batch_time=0.48536 
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.66601 (QuantReg: 9.45994) QuantErr: 9.45994 batch_time=0.49041 
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.92650 (QuantReg: 9.53346) QuantErr: 9.53346 batch_time=0.49639 
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.96565 (QuantReg: 9.76510) QuantErr: 9.76510 batch_time=1.92621 
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 2.17498 (QuantReg: 9.33095) QuantErr: 9.33095 batch_time=0.51195 
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.85902 (QuantReg: 9.43775) QuantErr: 9.43775 batch_time=0.49742 
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.69002 (QuantReg: 9.68123) QuantErr: 9.68123 batch_time=0.51605 
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 2.27813 (QuantReg: 9.40952) QuantErr: 9.40952 batch_time=0.48867 
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.73845 (QuantReg: 9.82372) QuantErr: 9.82372 batch_time=0.51271 
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.36710 (QuantReg: 9.60610) QuantErr: 9.60610 batch_time=0.70397 
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 2.32362 (QuantReg: 9.67703) QuantErr: 9.67703 batch_time=0.51331 
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.89714 (QuantReg: 9.63847) QuantErr: 9.63847 batch_time=0.50799 
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.97510 (QuantReg: 9.67541) QuantErr: 9.67541 batch_time=0.49075 
Train Epoch: 13 codebook_update_time=1.59810
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch13.pth ...
Done in 4.548s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch13.pth ...
Done in 8.607s
removing stale ckpt [epoch 12] [took 0.65s]
 epoch          : 13
 loss           : 2.0058848280906676
 quant_reg      : 9.545579822540283
 quant_err      : 9.545579822540283
 learning_rate  : 2.7018004383131832e-05
 n_samples      : 416000
 n_steps        : 3250
 MSRVTT_full_val/t2v_metrics/R1: 29.77867203219316
 MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
 MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
 MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.788732394366196
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.08095285315748
 MSRVTT_full_val/v2t_metrics/R1: 34.20523138832998
 MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
 MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
 MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.4607645875251505
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.0815713597829
 MSRVTT_full_test/t2v_metrics/R1: 11.538461538461538
 MSRVTT_full_test/t2v_metrics/R5: 30.869565217391305
 MSRVTT_full_test/t2v_metrics/R10: 45.852842809364546
 MSRVTT_full_test/t2v_metrics/R50: 79.1638795986622
 MSRVTT_full_test/t2v_metrics/MedR: 13.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.51270903010033
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.37162177896157
 MSRVTT_full_test/v2t_metrics/R1: 11.839464882943144
 MSRVTT_full_test/v2t_metrics/R5: 36.92307692307692
 MSRVTT_full_test/v2t_metrics/R10: 52.0066889632107
 MSRVTT_full_test/v2t_metrics/R50: 83.1438127090301
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.60033444816054
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.32890064609451
 mnt_best       : 25.37162177896157
 not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.82533 (QuantReg: 9.67546) QuantErr: 9.67546 batch_time=31.06962 
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.42672 (QuantReg: 9.75568) QuantErr: 9.75568 batch_time=0.48927 
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.87704 (QuantReg: 9.68481) QuantErr: 9.68481 batch_time=0.48556 
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.89257 (QuantReg: 9.53909) QuantErr: 9.53909 batch_time=0.50941 
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.21130 (QuantReg: 9.47822) QuantErr: 9.47822 batch_time=0.47662 
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.94275 (QuantReg: 9.35827) QuantErr: 9.35827 batch_time=0.48947 
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.23821 (QuantReg: 9.54591) QuantErr: 9.54591 batch_time=1.93385 
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 2.11043 (QuantReg: 9.08395) QuantErr: 9.08395 batch_time=0.51096 
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.71229 (QuantReg: 9.20695) QuantErr: 9.20695 batch_time=0.50121 
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.85489 (QuantReg: 9.25117) QuantErr: 9.25117 batch_time=0.48751 
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.87429 (QuantReg: 9.54137) QuantErr: 9.54137 batch_time=0.48229 
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 2.12534 (QuantReg: 9.27463) QuantErr: 9.27463 batch_time=0.48722 
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.85656 (QuantReg: 9.88797) QuantErr: 9.88797 batch_time=1.31964 
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.69104 (QuantReg: 9.54278) QuantErr: 9.54278 batch_time=1.81757 
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.79037 (QuantReg: 9.48609) QuantErr: 9.48609 batch_time=0.49646 
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.72930 (QuantReg: 9.56291) QuantErr: 9.56291 batch_time=0.48718 
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.73860 (QuantReg: 9.28780) QuantErr: 9.28780 batch_time=0.49636 
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 2.09216 (QuantReg: 9.79860) QuantErr: 9.79860 batch_time=0.50285 
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.78240 (QuantReg: 9.30983) QuantErr: 9.30983 batch_time=0.48246 
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 2.19040 (QuantReg: 9.24648) QuantErr: 9.24648 batch_time=0.49347 
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.07599 (QuantReg: 9.64071) QuantErr: 9.64071 batch_time=0.48438 
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.89952 (QuantReg: 9.76194) QuantErr: 9.76194 batch_time=0.49498 
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.85480 (QuantReg: 9.53834) QuantErr: 9.53834 batch_time=0.51759 
Train Epoch: 14 codebook_update_time=1.72912
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch14.pth ...
Done in 23.000s
removing stale ckpt [epoch 13] [took 0.00s]
 epoch          : 14
 loss           : 1.960813542842865
 quant_reg      : 9.510387943267823
 quant_err      : 9.510387943267823
 learning_rate  : 2.566710416397524e-05
 n_samples      : 448000
 n_steps        : 3500
 MSRVTT_full_val/t2v_metrics/R1: 28.571428571428573
 MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
 MSRVTT_full_val/t2v_metrics/R10: 77.06237424547284
 MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.37625754527163
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.304632877580566
 MSRVTT_full_val/v2t_metrics/R1: 34.60764587525151
 MSRVTT_full_val/v2t_metrics/R5: 70.02012072434607
 MSRVTT_full_val/v2t_metrics/R10: 81.89134808853119
 MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.458752515090543
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.32807513025551
 MSRVTT_full_test/t2v_metrics/R1: 10.167224080267559
 MSRVTT_full_test/t2v_metrics/R5: 30.80267558528428
 MSRVTT_full_test/t2v_metrics/R10: 44.71571906354515
 MSRVTT_full_test/t2v_metrics/R50: 77.45819397993311
 MSRVTT_full_test/t2v_metrics/MedR: 13.5
 MSRVTT_full_test/t2v_metrics/MeanR: 53.709030100334445
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.103698445496494
 MSRVTT_full_test/v2t_metrics/R1: 12.073578595317725
 MSRVTT_full_test/v2t_metrics/R5: 36.38795986622073
 MSRVTT_full_test/v2t_metrics/R10: 50.36789297658863
 MSRVTT_full_test/v2t_metrics/R50: 82.17391304347827
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 41.602341137123744
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.074745932180612
 mnt_best       : 25.37162177896157
 not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.36010 (QuantReg: 9.21051) QuantErr: 9.21051 batch_time=37.28978 
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.93706 (QuantReg: 9.29362) QuantErr: 9.29362 batch_time=1.70647 
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.01339 (QuantReg: 9.29539) QuantErr: 9.29539 batch_time=0.49440 
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.02460 (QuantReg: 9.30553) QuantErr: 9.30553 batch_time=0.49966 
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.03756 (QuantReg: 9.73160) QuantErr: 9.73160 batch_time=0.50255 
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.97864 (QuantReg: 9.17741) QuantErr: 9.17741 batch_time=0.49869 
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.25368 (QuantReg: 9.32194) QuantErr: 9.32194 batch_time=0.49896 
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.78992 (QuantReg: 9.48733) QuantErr: 9.48733 batch_time=2.14365 
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.15175 (QuantReg: 9.48558) QuantErr: 9.48558 batch_time=0.53302 
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.83628 (QuantReg: 9.54686) QuantErr: 9.54686 batch_time=0.49695 
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.09321 (QuantReg: 9.36822) QuantErr: 9.36822 batch_time=0.74092 
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.67429 (QuantReg: 9.24765) QuantErr: 9.24765 batch_time=0.49484 
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.78697 (QuantReg: 9.43836) QuantErr: 9.43836 batch_time=0.51219 
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 2.30752 (QuantReg: 9.67488) QuantErr: 9.67488 batch_time=0.51052 
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.89413 (QuantReg: 9.29821) QuantErr: 9.29821 batch_time=0.49340 
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.02241 (QuantReg: 9.72334) QuantErr: 9.72334 batch_time=0.50351 
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 2.28120 (QuantReg: 9.72381) QuantErr: 9.72381 batch_time=0.51449 
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.81703 (QuantReg: 9.48864) QuantErr: 9.48864 batch_time=0.52049 
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.72446 (QuantReg: 9.63053) QuantErr: 9.63053 batch_time=0.50604 
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.09094 (QuantReg: 9.21898) QuantErr: 9.21898 batch_time=0.49425 
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.14681 (QuantReg: 9.67914) QuantErr: 9.67914 batch_time=0.53563 
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.09959 (QuantReg: 9.73994) QuantErr: 9.73994 batch_time=0.49978 
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.07152 (QuantReg: 9.38213) QuantErr: 9.38213 batch_time=0.51385 
Train Epoch: 15 codebook_update_time=1.66108
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch15.pth ...
Done in 20.338s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch15.pth ...
Done in 24.186s
removing stale ckpt [epoch 14] [took 0.00s]
 epoch          : 15
 loss           : 1.9088825998306274
 quant_reg      : 9.507624305725098
 quant_err      : 9.507624305725098
 learning_rate  : 2.4383748955776477e-05
 n_samples      : 480000
 n_steps        : 3750
 MSRVTT_full_val/t2v_metrics/R1: 30.382293762575454
 MSRVTT_full_val/t2v_metrics/R5: 69.01408450704226
 MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.368209255533198
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.9835412794789
 MSRVTT_full_val/v2t_metrics/R1: 32.99798792756539
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.325955734406438
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.72779900735495
 MSRVTT_full_test/t2v_metrics/R1: 11.20401337792642
 MSRVTT_full_test/t2v_metrics/R5: 32.90969899665552
 MSRVTT_full_test/t2v_metrics/R10: 46.45484949832776
 MSRVTT_full_test/t2v_metrics/R50: 78.39464882943143
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.82508361204013
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.777622660049722
 MSRVTT_full_test/v2t_metrics/R1: 12.073578595317725
 MSRVTT_full_test/v2t_metrics/R5: 36.78929765886288
 MSRVTT_full_test/v2t_metrics/R10: 52.274247491638796
 MSRVTT_full_test/v2t_metrics/R50: 83.57859531772576
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.08411371237458
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.52868584978328
 mnt_best       : 25.777622660049722
 not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 1.75426 (QuantReg: 8.99031) QuantErr: 8.99031 batch_time=34.72141 
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 1.80818 (QuantReg: 9.62088) QuantErr: 9.62088 batch_time=0.49522 
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 2.18750 (QuantReg: 9.50370) QuantErr: 9.50370 batch_time=0.86522 
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 1.80136 (QuantReg: 9.64519) QuantErr: 9.64519 batch_time=0.58379 
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 1.75047 (QuantReg: 9.23210) QuantErr: 9.23210 batch_time=0.48795 
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 1.84844 (QuantReg: 9.39919) QuantErr: 9.39919 batch_time=0.52163 
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.02338 (QuantReg: 9.86209) QuantErr: 9.86209 batch_time=0.54499 
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 1.75927 (QuantReg: 9.47820) QuantErr: 9.47820 batch_time=0.50279 
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 1.82105 (QuantReg: 9.80011) QuantErr: 9.80011 batch_time=0.51039 
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 1.67447 (QuantReg: 9.46044) QuantErr: 9.46044 batch_time=0.52126 
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 2.05843 (QuantReg: 10.03994) QuantErr: 10.03994 batch_time=0.49912 
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 2.03087 (QuantReg: 9.74309) QuantErr: 9.74309 batch_time=0.49735 
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 1.91752 (QuantReg: 9.45516) QuantErr: 9.45516 batch_time=0.49031 
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 1.70306 (QuantReg: 9.69277) QuantErr: 9.69277 batch_time=3.11226 
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 1.98449 (QuantReg: 9.52983) QuantErr: 9.52983 batch_time=0.82021 
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 1.76097 (QuantReg: 9.52037) QuantErr: 9.52037 batch_time=0.49498 
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 1.98704 (QuantReg: 9.11706) QuantErr: 9.11706 batch_time=0.49163 
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 1.65074 (QuantReg: 9.46402) QuantErr: 9.46402 batch_time=0.49623 
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 1.97519 (QuantReg: 9.68357) QuantErr: 9.68357 batch_time=0.49094 
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 1.77292 (QuantReg: 9.47809) QuantErr: 9.47809 batch_time=0.61203 
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 1.67542 (QuantReg: 9.63364) QuantErr: 9.63364 batch_time=0.49179 
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 1.74076 (QuantReg: 9.34164) QuantErr: 9.34164 batch_time=0.49802 
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 1.93302 (QuantReg: 9.84244) QuantErr: 9.84244 batch_time=0.50556 
Train Epoch: 16 codebook_update_time=1.64891
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch16.pth ...
Done in 4.356s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch16.pth ...
Done in 8.377s
removing stale ckpt [epoch 15] [took 0.00s]
 epoch          : 16
 loss           : 1.8791129927635193
 quant_reg      : 9.520309608459472
 quant_err      : 9.520309608459472
 learning_rate  : 2.3164561507987653e-05
 n_samples      : 512000
 n_steps        : 4000
 MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
 MSRVTT_full_val/t2v_metrics/R5: 68.61167002012073
 MSRVTT_full_val/t2v_metrics/R10: 79.87927565392354
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.52112676056338
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.32012085023618
 MSRVTT_full_val/v2t_metrics/R1: 32.394366197183096
 MSRVTT_full_val/v2t_metrics/R5: 70.22132796780684
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.338028169014085
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.85055121648768
 MSRVTT_full_test/t2v_metrics/R1: 11.638795986622073
 MSRVTT_full_test/t2v_metrics/R5: 32.90969899665552
 MSRVTT_full_test/t2v_metrics/R10: 47.391304347826086
 MSRVTT_full_test/t2v_metrics/R50: 80.10033444816054
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 45.944147157190635
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.281100230659863
 MSRVTT_full_test/v2t_metrics/R1: 12.876254180602007
 MSRVTT_full_test/v2t_metrics/R5: 37.625418060200666
 MSRVTT_full_test/v2t_metrics/R10: 52.642140468227424
 MSRVTT_full_test/v2t_metrics/R50: 83.74581939799332
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.3185618729097
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.43527776246134
 mnt_best       : 26.281100230659863
 not_improved_count: 0
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 1.95813 (QuantReg: 9.34626) QuantErr: 9.34626 batch_time=34.61501 
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 2.04149 (QuantReg: 9.39290) QuantErr: 9.39290 batch_time=0.49379 
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 1.72469 (QuantReg: 9.40053) QuantErr: 9.40053 batch_time=0.51683 
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 1.59605 (QuantReg: 9.33825) QuantErr: 9.33825 batch_time=0.48617 
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 2.05722 (QuantReg: 9.35642) QuantErr: 9.35642 batch_time=0.49367 
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 1.66803 (QuantReg: 9.23627) QuantErr: 9.23627 batch_time=0.48977 
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 1.69276 (QuantReg: 9.91179) QuantErr: 9.91179 batch_time=0.47868 
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 1.90210 (QuantReg: 9.45105) QuantErr: 9.45105 batch_time=0.50101 
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 1.75021 (QuantReg: 9.80458) QuantErr: 9.80458 batch_time=2.23369 
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 1.77394 (QuantReg: 9.64889) QuantErr: 9.64889 batch_time=0.53238 
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 1.91446 (QuantReg: 9.67779) QuantErr: 9.67779 batch_time=0.49447 
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 1.88873 (QuantReg: 9.36861) QuantErr: 9.36861 batch_time=1.07778 
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 1.66646 (QuantReg: 9.36809) QuantErr: 9.36809 batch_time=0.49833 
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.24066 (QuantReg: 9.33623) QuantErr: 9.33623 batch_time=1.16269 
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 1.89206 (QuantReg: 9.75270) QuantErr: 9.75270 batch_time=0.49521 
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 1.62622 (QuantReg: 9.22056) QuantErr: 9.22056 batch_time=0.49618 
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 1.74966 (QuantReg: 9.26269) QuantErr: 9.26269 batch_time=0.48795 
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 1.64791 (QuantReg: 9.51490) QuantErr: 9.51490 batch_time=0.49218 
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.10010 (QuantReg: 9.99932) QuantErr: 9.99932 batch_time=1.42015 
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 1.78647 (QuantReg: 9.65647) QuantErr: 9.65647 batch_time=0.51637 
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 1.55821 (QuantReg: 9.39260) QuantErr: 9.39260 batch_time=0.48806 
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 1.94704 (QuantReg: 9.94757) QuantErr: 9.94757 batch_time=0.48408 
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 1.77555 (QuantReg: 9.50289) QuantErr: 9.50289 batch_time=0.48499 
Train Epoch: 17 codebook_update_time=1.73854
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch17.pth ...
Done in 18.934s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch17.pth ...
Done in 22.881s
removing stale ckpt [epoch 16] [took 0.00s]
 epoch          : 17
 loss           : 1.8316268496513366
 quant_reg      : 9.533726100921632
 quant_err      : 9.533726100921632
 learning_rate  : 2.2006333432588268e-05
 n_samples      : 544000
 n_steps        : 4250
 MSRVTT_full_val/t2v_metrics/R1: 28.973843058350102
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 80.6841046277666
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 9.13682092555332
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.22661129809902
 MSRVTT_full_val/v2t_metrics/R1: 33.601609657947684
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
 MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.746478873239437
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.895567720166895
 MSRVTT_full_test/t2v_metrics/R1: 11.77257525083612
 MSRVTT_full_test/t2v_metrics/R5: 33.51170568561873
 MSRVTT_full_test/t2v_metrics/R10: 46.72240802675585
 MSRVTT_full_test/t2v_metrics/R50: 78.89632107023411
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 46.8056856187291
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.415838984198537
 MSRVTT_full_test/v2t_metrics/R1: 13.010033444816054
 MSRVTT_full_test/v2t_metrics/R5: 36.62207357859532
 MSRVTT_full_test/v2t_metrics/R10: 51.070234113712374
 MSRVTT_full_test/v2t_metrics/R50: 83.64548494983278
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.98377926421405
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.9776434346862
 mnt_best       : 26.415838984198537
 not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.05802 (QuantReg: 9.62333) QuantErr: 9.62333 batch_time=32.96514 
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 1.74629 (QuantReg: 9.59029) QuantErr: 9.59029 batch_time=0.73574 
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 1.86531 (QuantReg: 9.46898) QuantErr: 9.46898 batch_time=0.49632 
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 1.78531 (QuantReg: 9.40103) QuantErr: 9.40103 batch_time=0.50870 
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 1.89961 (QuantReg: 9.44938) QuantErr: 9.44938 batch_time=0.49557 
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 1.86347 (QuantReg: 9.52056) QuantErr: 9.52056 batch_time=0.48898 
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 2.08093 (QuantReg: 9.42471) QuantErr: 9.42471 batch_time=0.49607 
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 2.05998 (QuantReg: 9.52875) QuantErr: 9.52875 batch_time=0.49577 
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 1.67843 (QuantReg: 9.47397) QuantErr: 9.47397 batch_time=0.49113 
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 1.71030 (QuantReg: 9.51346) QuantErr: 9.51346 batch_time=0.49263 
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 1.56196 (QuantReg: 9.43665) QuantErr: 9.43665 batch_time=0.50332 
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 2.00555 (QuantReg: 9.46582) QuantErr: 9.46582 batch_time=0.48939 
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.22445 (QuantReg: 9.68588) QuantErr: 9.68588 batch_time=2.68251 
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 1.61767 (QuantReg: 9.46181) QuantErr: 9.46181 batch_time=0.49329 
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 1.78110 (QuantReg: 9.75218) QuantErr: 9.75218 batch_time=0.49811 
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 2.04926 (QuantReg: 9.61158) QuantErr: 9.61158 batch_time=0.49135 
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 1.70585 (QuantReg: 9.54279) QuantErr: 9.54279 batch_time=0.48829 
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 1.62394 (QuantReg: 9.50879) QuantErr: 9.50879 batch_time=0.62664 
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 1.73038 (QuantReg: 9.31925) QuantErr: 9.31925 batch_time=0.55159 
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 2.23469 (QuantReg: 9.71026) QuantErr: 9.71026 batch_time=0.49489 
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 1.90001 (QuantReg: 9.54532) QuantErr: 9.54532 batch_time=0.49205 
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 1.38215 (QuantReg: 9.50268) QuantErr: 9.50268 batch_time=0.60221 
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 1.73830 (QuantReg: 9.63191) QuantErr: 9.63191 batch_time=0.49114 
Train Epoch: 18 codebook_update_time=1.99723
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch18.pth ...
Done in 4.072s
removing stale ckpt [epoch 17] [took 0.02s]
 epoch          : 18
 loss           : 1.789527262210846
 quant_reg      : 9.524178833007813
 quant_err      : 9.524178833007813
 learning_rate  : 2.0906016760958855e-05
 n_samples      : 576000
 n_steps        : 4500
 MSRVTT_full_val/t2v_metrics/R1: 29.979879275653925
 MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
 MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.599597585513077
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.29404191613553
 MSRVTT_full_val/v2t_metrics/R1: 32.59557344064386
 MSRVTT_full_val/v2t_metrics/R5: 73.03822937625755
 MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
 MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.090543259557344
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.54936178823143
 MSRVTT_full_test/t2v_metrics/R1: 11.97324414715719
 MSRVTT_full_test/t2v_metrics/R5: 32.675585284280935
 MSRVTT_full_test/t2v_metrics/R10: 46.08695652173913
 MSRVTT_full_test/t2v_metrics/R50: 78.29431438127091
 MSRVTT_full_test/t2v_metrics/MedR: 13.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.78695652173913
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.222318071253774
 MSRVTT_full_test/v2t_metrics/R1: 13.545150501672241
 MSRVTT_full_test/v2t_metrics/R5: 37.424749163879596
 MSRVTT_full_test/v2t_metrics/R10: 52.07357859531773
 MSRVTT_full_test/v2t_metrics/R50: 83.4448160535117
 MSRVTT_full_test/v2t_metrics/MedR: 9.75
 MSRVTT_full_test/v2t_metrics/MeanR: 38.6061872909699
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.775110766565124
 mnt_best       : 26.415838984198537
 not_improved_count: 1
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 1.71641 (QuantReg: 9.52732) QuantErr: 9.52732 batch_time=30.82531 
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 1.77048 (QuantReg: 9.64170) QuantErr: 9.64170 batch_time=0.53405 
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 1.88089 (QuantReg: 9.36380) QuantErr: 9.36380 batch_time=0.49502 
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 1.87618 (QuantReg: 9.61906) QuantErr: 9.61906 batch_time=0.74152 
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 1.79599 (QuantReg: 9.61406) QuantErr: 9.61406 batch_time=0.52410 
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 1.78803 (QuantReg: 9.69718) QuantErr: 9.69718 batch_time=0.48958 
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 1.74082 (QuantReg: 9.39055) QuantErr: 9.39055 batch_time=0.50477 
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 1.96293 (QuantReg: 9.77095) QuantErr: 9.77095 batch_time=0.50339 
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 1.86247 (QuantReg: 9.32090) QuantErr: 9.32090 batch_time=0.55059 
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 1.70224 (QuantReg: 9.58550) QuantErr: 9.58550 batch_time=0.50774 
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 1.68059 (QuantReg: 9.52907) QuantErr: 9.52907 batch_time=0.52082 
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 1.80783 (QuantReg: 9.67827) QuantErr: 9.67827 batch_time=0.51822 
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 1.78878 (QuantReg: 9.54752) QuantErr: 9.54752 batch_time=0.65155 
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 1.40826 (QuantReg: 9.29713) QuantErr: 9.29713 batch_time=0.54643 
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 1.83062 (QuantReg: 9.67524) QuantErr: 9.67524 batch_time=0.49769 
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 1.33455 (QuantReg: 9.29501) QuantErr: 9.29501 batch_time=0.77149 
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 1.91756 (QuantReg: 9.55946) QuantErr: 9.55946 batch_time=0.54392 
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 1.97288 (QuantReg: 9.67645) QuantErr: 9.67645 batch_time=0.49600 
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 1.69402 (QuantReg: 9.48849) QuantErr: 9.48849 batch_time=0.49335 
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 1.42871 (QuantReg: 9.55738) QuantErr: 9.55738 batch_time=0.50519 
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 1.51840 (QuantReg: 9.69844) QuantErr: 9.69844 batch_time=0.49175 
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 1.52710 (QuantReg: 9.70265) QuantErr: 9.70265 batch_time=0.47581 
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 1.74974 (QuantReg: 9.45254) QuantErr: 9.45254 batch_time=0.48973 
Train Epoch: 19 codebook_update_time=1.70654
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch19.pth ...
Done in 3.993s
removing stale ckpt [epoch 18] [took 0.00s]
 epoch          : 19
 loss           : 1.7443925795555115
 quant_reg      : 9.517262279510499
 quant_err      : 9.517262279510499
 learning_rate  : 1.986071592291091e-05
 n_samples      : 608000
 n_steps        : 4750
 MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
 MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
 MSRVTT_full_val/t2v_metrics/R10: 79.47686116700201
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.732394366197184
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.82775780257941
 MSRVTT_full_val/v2t_metrics/R1: 32.394366197183096
 MSRVTT_full_val/v2t_metrics/R5: 71.83098591549296
 MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.046277665995976
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.05866818029883
 MSRVTT_full_test/t2v_metrics/R1: 11.37123745819398
 MSRVTT_full_test/t2v_metrics/R5: 33.04347826086956
 MSRVTT_full_test/t2v_metrics/R10: 47.05685618729097
 MSRVTT_full_test/t2v_metrics/R50: 78.66220735785953
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.2
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.05186374171031
 MSRVTT_full_test/v2t_metrics/R1: 13.210702341137123
 MSRVTT_full_test/v2t_metrics/R5: 37.290969899665555
 MSRVTT_full_test/v2t_metrics/R10: 51.50501672240803
 MSRVTT_full_test/v2t_metrics/R50: 84.24749163879599
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.63461538461539
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.385045868250376
 mnt_best       : 26.415838984198537
 not_improved_count: 2
Train Epoch: 20 [1/250 128/32000 (0%)] Loss: 1.79883 (QuantReg: 9.39865) QuantErr: 9.39865 batch_time=37.70236 
Train Epoch: 20 [12/250 1536/32000 (5%)] Loss: 1.88303 (QuantReg: 9.24618) QuantErr: 9.24618 batch_time=0.50923 
Train Epoch: 20 [23/250 2944/32000 (9%)] Loss: 1.66747 (QuantReg: 9.38556) QuantErr: 9.38556 batch_time=0.50564 
Train Epoch: 20 [34/250 4352/32000 (14%)] Loss: 1.76239 (QuantReg: 9.69280) QuantErr: 9.69280 batch_time=0.50095 
Train Epoch: 20 [45/250 5760/32000 (18%)] Loss: 1.90069 (QuantReg: 9.76253) QuantErr: 9.76253 batch_time=0.48762 
Train Epoch: 20 [56/250 7168/32000 (22%)] Loss: 1.64909 (QuantReg: 9.29867) QuantErr: 9.29867 batch_time=0.51478 
Train Epoch: 20 [67/250 8576/32000 (27%)] Loss: 1.60715 (QuantReg: 9.62043) QuantErr: 9.62043 batch_time=0.52761 
Train Epoch: 20 [78/250 9984/32000 (31%)] Loss: 1.51895 (QuantReg: 9.43768) QuantErr: 9.43768 batch_time=0.57131 
Train Epoch: 20 [89/250 11392/32000 (36%)] Loss: 2.02734 (QuantReg: 9.34900) QuantErr: 9.34900 batch_time=0.50247 
Train Epoch: 20 [100/250 12800/32000 (40%)] Loss: 1.66415 (QuantReg: 9.64132) QuantErr: 9.64132 batch_time=0.48495 
Train Epoch: 20 [111/250 14208/32000 (44%)] Loss: 1.49996 (QuantReg: 9.62969) QuantErr: 9.62969 batch_time=0.48449 
Train Epoch: 20 [122/250 15616/32000 (49%)] Loss: 1.48322 (QuantReg: 9.42546) QuantErr: 9.42546 batch_time=0.49056 
Train Epoch: 20 [133/250 17024/32000 (53%)] Loss: 1.76042 (QuantReg: 9.30418) QuantErr: 9.30418 batch_time=0.48854 
Train Epoch: 20 [144/250 18432/32000 (58%)] Loss: 1.81354 (QuantReg: 9.55243) QuantErr: 9.55243 batch_time=1.30804 
Train Epoch: 20 [155/250 19840/32000 (62%)] Loss: 1.90719 (QuantReg: 9.28745) QuantErr: 9.28745 batch_time=0.73394 
Train Epoch: 20 [166/250 21248/32000 (66%)] Loss: 2.04554 (QuantReg: 9.72399) QuantErr: 9.72399 batch_time=0.49540 
Train Epoch: 20 [177/250 22656/32000 (71%)] Loss: 1.85257 (QuantReg: 9.82662) QuantErr: 9.82662 batch_time=0.53457 
Train Epoch: 20 [188/250 24064/32000 (75%)] Loss: 1.74535 (QuantReg: 9.71889) QuantErr: 9.71889 batch_time=0.50302 
Train Epoch: 20 [199/250 25472/32000 (80%)] Loss: 1.57912 (QuantReg: 9.43680) QuantErr: 9.43680 batch_time=0.48824 
Train Epoch: 20 [210/250 26880/32000 (84%)] Loss: 2.14244 (QuantReg: 9.28006) QuantErr: 9.28006 batch_time=0.51256 
Train Epoch: 20 [221/250 28288/32000 (88%)] Loss: 2.24001 (QuantReg: 9.28749) QuantErr: 9.28749 batch_time=0.48861 
Train Epoch: 20 [232/250 29696/32000 (93%)] Loss: 1.65395 (QuantReg: 9.56929) QuantErr: 9.56929 batch_time=0.49382 
Train Epoch: 20 [243/250 31104/32000 (97%)] Loss: 1.43329 (QuantReg: 9.84917) QuantErr: 9.84917 batch_time=0.50024 
Train Epoch: 20 codebook_update_time=1.64211
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch20.pth ...
Done in 20.125s
removing stale ckpt [epoch 19] [took 0.08s]
 epoch          : 20
 loss           : 1.7376019611358642
 quant_reg      : 9.491741008758545
 quant_err      : 9.491741008758545
 learning_rate  : 1.8867680126765363e-05
 n_samples      : 640000
 n_steps        : 5000
 MSRVTT_full_val/t2v_metrics/R1: 31.187122736418512
 MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
 MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.267605633802816
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.160817037432196
 MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
 MSRVTT_full_val/v2t_metrics/R5: 71.83098591549296
 MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.802816901408451
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.723254357635454
 MSRVTT_full_test/t2v_metrics/R1: 11.672240802675585
 MSRVTT_full_test/t2v_metrics/R5: 33.07692307692308
 MSRVTT_full_test/t2v_metrics/R10: 47.591973244147155
 MSRVTT_full_test/t2v_metrics/R50: 79.1304347826087
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.0371237458194
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.3878709334692
 MSRVTT_full_test/v2t_metrics/R1: 13.143812709030101
 MSRVTT_full_test/v2t_metrics/R5: 38.02675585284281
 MSRVTT_full_test/v2t_metrics/R10: 52.74247491638796
 MSRVTT_full_test/v2t_metrics/R50: 83.67892976588628
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.542809364548496
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.761652931164438
 mnt_best       : 26.415838984198537
 not_improved_count: 3
Train Epoch: 21 [1/250 128/32000 (0%)] Loss: 1.75288 (QuantReg: 9.47837) QuantErr: 9.47837 batch_time=32.07543 
Train Epoch: 21 [12/250 1536/32000 (5%)] Loss: 1.75875 (QuantReg: 9.56156) QuantErr: 9.56156 batch_time=0.49910 
Train Epoch: 21 [23/250 2944/32000 (9%)] Loss: 1.68684 (QuantReg: 9.28677) QuantErr: 9.28677 batch_time=0.51147 
Train Epoch: 21 [34/250 4352/32000 (14%)] Loss: 1.52328 (QuantReg: 9.83942) QuantErr: 9.83942 batch_time=0.52901 
Train Epoch: 21 [45/250 5760/32000 (18%)] Loss: 1.40829 (QuantReg: 9.49316) QuantErr: 9.49316 batch_time=0.51987 
Train Epoch: 21 [56/250 7168/32000 (22%)] Loss: 1.50284 (QuantReg: 9.92679) QuantErr: 9.92679 batch_time=0.50262 
Train Epoch: 21 [67/250 8576/32000 (27%)] Loss: 1.80224 (QuantReg: 9.54291) QuantErr: 9.54291 batch_time=0.52150 
Train Epoch: 21 [78/250 9984/32000 (31%)] Loss: 1.75690 (QuantReg: 9.24975) QuantErr: 9.24975 batch_time=0.52977 
Train Epoch: 21 [89/250 11392/32000 (36%)] Loss: 1.91019 (QuantReg: 9.59717) QuantErr: 9.59717 batch_time=0.54727 
Train Epoch: 21 [100/250 12800/32000 (40%)] Loss: 1.83689 (QuantReg: 9.24201) QuantErr: 9.24201 batch_time=0.48839 
Train Epoch: 21 [111/250 14208/32000 (44%)] Loss: 1.62738 (QuantReg: 9.35217) QuantErr: 9.35217 batch_time=0.50146 
Train Epoch: 21 [122/250 15616/32000 (49%)] Loss: 1.65080 (QuantReg: 9.77177) QuantErr: 9.77177 batch_time=0.48978 
Train Epoch: 21 [133/250 17024/32000 (53%)] Loss: 1.40909 (QuantReg: 9.05827) QuantErr: 9.05827 batch_time=0.50813 
Train Epoch: 21 [144/250 18432/32000 (58%)] Loss: 1.86217 (QuantReg: 9.77983) QuantErr: 9.77983 batch_time=1.46485 
Train Epoch: 21 [155/250 19840/32000 (62%)] Loss: 1.67633 (QuantReg: 9.30448) QuantErr: 9.30448 batch_time=0.48620 
Train Epoch: 21 [166/250 21248/32000 (66%)] Loss: 1.63142 (QuantReg: 9.77353) QuantErr: 9.77353 batch_time=0.49957 
Train Epoch: 21 [177/250 22656/32000 (71%)] Loss: 1.89068 (QuantReg: 9.88630) QuantErr: 9.88630 batch_time=0.52257 
Train Epoch: 21 [188/250 24064/32000 (75%)] Loss: 1.65009 (QuantReg: 9.40972) QuantErr: 9.40972 batch_time=0.51598 
Train Epoch: 21 [199/250 25472/32000 (80%)] Loss: 1.62105 (QuantReg: 9.47817) QuantErr: 9.47817 batch_time=0.60487 
Train Epoch: 21 [210/250 26880/32000 (84%)] Loss: 1.96484 (QuantReg: 9.34007) QuantErr: 9.34007 batch_time=0.63777 
Train Epoch: 21 [221/250 28288/32000 (88%)] Loss: 1.66704 (QuantReg: 9.64561) QuantErr: 9.64561 batch_time=0.48977 
Train Epoch: 21 [232/250 29696/32000 (93%)] Loss: 1.69443 (QuantReg: 9.81507) QuantErr: 9.81507 batch_time=0.50850 
Train Epoch: 21 [243/250 31104/32000 (97%)] Loss: 1.48258 (QuantReg: 9.41990) QuantErr: 9.41990 batch_time=1.00712 
Train Epoch: 21 codebook_update_time=1.95048
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch21.pth ...
Done in 21.126s
removing stale ckpt [epoch 20] [took 0.00s]
 epoch          : 21
 loss           : 1.7017106328010558
 quant_reg      : 9.507885272979737
 quant_err      : 9.507885272979737
 learning_rate  : 1.7924296120427095e-05
 n_samples      : 672000
 n_steps        : 5250
 MSRVTT_full_val/t2v_metrics/R1: 27.766599597585515
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
 MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.941649899396378
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.373723898276374
 MSRVTT_full_val/v2t_metrics/R1: 31.388329979879277
 MSRVTT_full_val/v2t_metrics/R5: 70.62374245472837
 MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
 MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.561368209255533
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.03638078104901
 MSRVTT_full_test/t2v_metrics/R1: 11.872909698996656
 MSRVTT_full_test/t2v_metrics/R5: 32.642140468227424
 MSRVTT_full_test/t2v_metrics/R10: 46.55518394648829
 MSRVTT_full_test/t2v_metrics/R50: 78.52842809364549
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.99866220735786
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.228167408515024
 MSRVTT_full_test/v2t_metrics/R1: 13.010033444816054
 MSRVTT_full_test/v2t_metrics/R5: 36.75585284280937
 MSRVTT_full_test/v2t_metrics/R10: 52.17391304347826
 MSRVTT_full_test/v2t_metrics/R50: 83.67892976588628
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 40.20301003344482
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.220396791866666
 mnt_best       : 26.415838984198537
 not_improved_count: 4
Train Epoch: 22 [1/250 128/32000 (0%)] Loss: 2.00361 (QuantReg: 9.35647) QuantErr: 9.35647 batch_time=30.71519 
Train Epoch: 22 [12/250 1536/32000 (5%)] Loss: 1.71583 (QuantReg: 9.66001) QuantErr: 9.66001 batch_time=0.75328 
Train Epoch: 22 [23/250 2944/32000 (9%)] Loss: 1.67093 (QuantReg: 9.57218) QuantErr: 9.57218 batch_time=0.58969 
Train Epoch: 22 [34/250 4352/32000 (14%)] Loss: 1.57832 (QuantReg: 9.08066) QuantErr: 9.08066 batch_time=0.49472 
Train Epoch: 22 [45/250 5760/32000 (18%)] Loss: 1.69889 (QuantReg: 9.60046) QuantErr: 9.60046 batch_time=0.49482 
Train Epoch: 22 [56/250 7168/32000 (22%)] Loss: 1.52572 (QuantReg: 9.37136) QuantErr: 9.37136 batch_time=0.52669 
Train Epoch: 22 [67/250 8576/32000 (27%)] Loss: 1.68469 (QuantReg: 9.20136) QuantErr: 9.20136 batch_time=0.49759 
Train Epoch: 22 [78/250 9984/32000 (31%)] Loss: 1.52738 (QuantReg: 9.35476) QuantErr: 9.35476 batch_time=0.49694 
Train Epoch: 22 [89/250 11392/32000 (36%)] Loss: 1.92782 (QuantReg: 9.08238) QuantErr: 9.08238 batch_time=0.50914 
Train Epoch: 22 [100/250 12800/32000 (40%)] Loss: 1.78364 (QuantReg: 9.52058) QuantErr: 9.52058 batch_time=0.49024 
Train Epoch: 22 [111/250 14208/32000 (44%)] Loss: 1.94604 (QuantReg: 9.41859) QuantErr: 9.41859 batch_time=0.49769 
Train Epoch: 22 [122/250 15616/32000 (49%)] Loss: 1.61008 (QuantReg: 9.55817) QuantErr: 9.55817 batch_time=0.52136 
Train Epoch: 22 [133/250 17024/32000 (53%)] Loss: 1.77247 (QuantReg: 9.54689) QuantErr: 9.54689 batch_time=0.48834 
Train Epoch: 22 [144/250 18432/32000 (58%)] Loss: 1.82145 (QuantReg: 9.45368) QuantErr: 9.45368 batch_time=2.66939 
Train Epoch: 22 [155/250 19840/32000 (62%)] Loss: 1.78356 (QuantReg: 9.63218) QuantErr: 9.63218 batch_time=1.36264 
Train Epoch: 22 [166/250 21248/32000 (66%)] Loss: 1.84525 (QuantReg: 9.39621) QuantErr: 9.39621 batch_time=0.49579 
Train Epoch: 22 [177/250 22656/32000 (71%)] Loss: 1.83356 (QuantReg: 9.50737) QuantErr: 9.50737 batch_time=0.51562 
Train Epoch: 22 [188/250 24064/32000 (75%)] Loss: 1.80060 (QuantReg: 9.70302) QuantErr: 9.70302 batch_time=0.48971 
Train Epoch: 22 [199/250 25472/32000 (80%)] Loss: 1.70041 (QuantReg: 9.17151) QuantErr: 9.17151 batch_time=0.49416 
Train Epoch: 22 [210/250 26880/32000 (84%)] Loss: 1.66103 (QuantReg: 9.52474) QuantErr: 9.52474 batch_time=1.17874 
Train Epoch: 22 [221/250 28288/32000 (88%)] Loss: 1.82730 (QuantReg: 9.46200) QuantErr: 9.46200 batch_time=0.49440 
Train Epoch: 22 [232/250 29696/32000 (93%)] Loss: 1.63053 (QuantReg: 9.39317) QuantErr: 9.39317 batch_time=0.49566 
Train Epoch: 22 [243/250 31104/32000 (97%)] Loss: 1.40221 (QuantReg: 9.61899) QuantErr: 9.61899 batch_time=0.61727 
Train Epoch: 22 codebook_update_time=1.66073
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch22.pth ...
Done in 3.916s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch22.pth ...
Done in 7.802s
removing stale ckpt [epoch 21] [took 0.00s]
 epoch          : 22
 loss           : 1.665872045993805
 quant_reg      : 9.492067199707032
 quant_err      : 9.492067199707032
 learning_rate  : 1.702808131440574e-05
 n_samples      : 704000
 n_steps        : 5500
 MSRVTT_full_val/t2v_metrics/R1: 29.979879275653925
 MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
 MSRVTT_full_val/t2v_metrics/R10: 81.08651911468813
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.515090543259557
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.557368014109436
 MSRVTT_full_val/v2t_metrics/R1: 33.40040241448692
 MSRVTT_full_val/v2t_metrics/R5: 71.83098591549296
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.231388329979879
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.886512597956994
 MSRVTT_full_test/t2v_metrics/R1: 12.842809364548495
 MSRVTT_full_test/t2v_metrics/R5: 33.74581939799331
 MSRVTT_full_test/t2v_metrics/R10: 47.85953177257525
 MSRVTT_full_test/t2v_metrics/R50: 78.8628762541806
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.6695652173913
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.475745594051897
 MSRVTT_full_test/v2t_metrics/R1: 13.377926421404682
 MSRVTT_full_test/v2t_metrics/R5: 38.06020066889632
 MSRVTT_full_test/v2t_metrics/R10: 51.83946488294314
 MSRVTT_full_test/v2t_metrics/R50: 83.34448160535118
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.59816053511706
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.774201889772584
 mnt_best       : 27.475745594051897
 not_improved_count: 0
Train Epoch: 23 [1/250 128/32000 (0%)] Loss: 1.69277 (QuantReg: 9.21201) QuantErr: 9.21201 batch_time=31.49403 
Train Epoch: 23 [12/250 1536/32000 (5%)] Loss: 1.79821 (QuantReg: 9.63877) QuantErr: 9.63877 batch_time=0.49951 
Train Epoch: 23 [23/250 2944/32000 (9%)] Loss: 1.36465 (QuantReg: 9.56429) QuantErr: 9.56429 batch_time=0.51912 
Train Epoch: 23 [34/250 4352/32000 (14%)] Loss: 1.76675 (QuantReg: 9.54721) QuantErr: 9.54721 batch_time=0.49145 
Train Epoch: 23 [45/250 5760/32000 (18%)] Loss: 1.74777 (QuantReg: 9.63632) QuantErr: 9.63632 batch_time=0.48031 
Train Epoch: 23 [56/250 7168/32000 (22%)] Loss: 1.71182 (QuantReg: 9.46025) QuantErr: 9.46025 batch_time=0.61454 
Train Epoch: 23 [67/250 8576/32000 (27%)] Loss: 1.63028 (QuantReg: 9.56918) QuantErr: 9.56918 batch_time=0.49923 
Train Epoch: 23 [78/250 9984/32000 (31%)] Loss: 1.80037 (QuantReg: 9.66157) QuantErr: 9.66157 batch_time=0.50429 
Train Epoch: 23 [89/250 11392/32000 (36%)] Loss: 1.79949 (QuantReg: 9.20115) QuantErr: 9.20115 batch_time=0.49155 
Train Epoch: 23 [100/250 12800/32000 (40%)] Loss: 1.43532 (QuantReg: 9.46479) QuantErr: 9.46479 batch_time=0.56925 
Train Epoch: 23 [111/250 14208/32000 (44%)] Loss: 1.79231 (QuantReg: 9.24489) QuantErr: 9.24489 batch_time=0.52685 
Train Epoch: 23 [122/250 15616/32000 (49%)] Loss: 1.71836 (QuantReg: 9.22925) QuantErr: 9.22925 batch_time=0.49906 
Train Epoch: 23 [133/250 17024/32000 (53%)] Loss: 1.54245 (QuantReg: 9.49170) QuantErr: 9.49170 batch_time=0.61712 
Train Epoch: 23 [144/250 18432/32000 (58%)] Loss: 1.79652 (QuantReg: 9.63761) QuantErr: 9.63761 batch_time=0.51132 
Train Epoch: 23 [155/250 19840/32000 (62%)] Loss: 1.78787 (QuantReg: 9.52760) QuantErr: 9.52760 batch_time=0.49869 
Train Epoch: 23 [166/250 21248/32000 (66%)] Loss: 1.77917 (QuantReg: 9.20745) QuantErr: 9.20745 batch_time=0.49918 
Train Epoch: 23 [177/250 22656/32000 (71%)] Loss: 1.44500 (QuantReg: 9.82187) QuantErr: 9.82187 batch_time=0.50781 
Train Epoch: 23 [188/250 24064/32000 (75%)] Loss: 1.71001 (QuantReg: 9.50904) QuantErr: 9.50904 batch_time=0.49965 
Train Epoch: 23 [199/250 25472/32000 (80%)] Loss: 1.34594 (QuantReg: 9.83181) QuantErr: 9.83181 batch_time=0.49870 
Train Epoch: 23 [210/250 26880/32000 (84%)] Loss: 1.55983 (QuantReg: 9.58395) QuantErr: 9.58395 batch_time=2.83351 
Train Epoch: 23 [221/250 28288/32000 (88%)] Loss: 1.43832 (QuantReg: 9.78165) QuantErr: 9.78165 batch_time=0.50993 
Train Epoch: 23 [232/250 29696/32000 (93%)] Loss: 1.62672 (QuantReg: 9.62873) QuantErr: 9.62873 batch_time=2.11240 
Train Epoch: 23 [243/250 31104/32000 (97%)] Loss: 1.65008 (QuantReg: 9.06921) QuantErr: 9.06921 batch_time=0.49212 
Train Epoch: 23 codebook_update_time=1.98694
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch23.pth ...
Done in 4.291s
removing stale ckpt [epoch 22] [took 0.05s]
 epoch          : 23
 loss           : 1.67966024684906
 quant_reg      : 9.49744633102417
 quant_err      : 9.49744633102417
 learning_rate  : 1.6176677248685452e-05
 n_samples      : 736000
 n_steps        : 5750
 MSRVTT_full_val/t2v_metrics/R1: 31.187122736418512
 MSRVTT_full_val/t2v_metrics/R5: 67.40442655935614
 MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.969818913480886
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.40017424764127
 MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
 MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
 MSRVTT_full_val/v2t_metrics/R10: 85.31187122736418
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.126760563380282
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.920623229057554
 MSRVTT_full_test/t2v_metrics/R1: 12.240802675585284
 MSRVTT_full_test/t2v_metrics/R5: 34.214046822742475
 MSRVTT_full_test/t2v_metrics/R10: 46.88963210702341
 MSRVTT_full_test/t2v_metrics/R50: 78.59531772575251
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.91939799331104
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.979282102497464
 MSRVTT_full_test/v2t_metrics/R1: 13.34448160535117
 MSRVTT_full_test/v2t_metrics/R5: 38.16053511705686
 MSRVTT_full_test/v2t_metrics/R10: 53.01003344481605
 MSRVTT_full_test/v2t_metrics/R50: 83.87959866220736
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.59448160535117
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.9979387396858
 mnt_best       : 27.475745594051897
 not_improved_count: 1
Train Epoch: 24 [1/250 128/32000 (0%)] Loss: 1.91618 (QuantReg: 9.26833) QuantErr: 9.26833 batch_time=31.83665 
Train Epoch: 24 [12/250 1536/32000 (5%)] Loss: 1.57888 (QuantReg: 9.50268) QuantErr: 9.50268 batch_time=0.47874 
Train Epoch: 24 [23/250 2944/32000 (9%)] Loss: 1.45025 (QuantReg: 9.61970) QuantErr: 9.61970 batch_time=0.48594 
Train Epoch: 24 [34/250 4352/32000 (14%)] Loss: 1.81748 (QuantReg: 9.55445) QuantErr: 9.55445 batch_time=0.48780 
Train Epoch: 24 [45/250 5760/32000 (18%)] Loss: 1.84313 (QuantReg: 9.30814) QuantErr: 9.30814 batch_time=0.48713 
Train Epoch: 24 [56/250 7168/32000 (22%)] Loss: 1.25377 (QuantReg: 9.44784) QuantErr: 9.44784 batch_time=0.49925 
Train Epoch: 24 [67/250 8576/32000 (27%)] Loss: 1.33659 (QuantReg: 9.72631) QuantErr: 9.72631 batch_time=0.50159 
Train Epoch: 24 [78/250 9984/32000 (31%)] Loss: 1.76843 (QuantReg: 9.33851) QuantErr: 9.33851 batch_time=0.49374 
Train Epoch: 24 [89/250 11392/32000 (36%)] Loss: 1.52208 (QuantReg: 9.50790) QuantErr: 9.50790 batch_time=0.50217 
Train Epoch: 24 [100/250 12800/32000 (40%)] Loss: 1.65457 (QuantReg: 9.60672) QuantErr: 9.60672 batch_time=0.51001 
Train Epoch: 24 [111/250 14208/32000 (44%)] Loss: 1.73807 (QuantReg: 9.27615) QuantErr: 9.27615 batch_time=0.51477 
Train Epoch: 24 [122/250 15616/32000 (49%)] Loss: 1.82449 (QuantReg: 9.42568) QuantErr: 9.42568 batch_time=0.50090 
Train Epoch: 24 [133/250 17024/32000 (53%)] Loss: 1.56370 (QuantReg: 9.57693) QuantErr: 9.57693 batch_time=0.50409 
Train Epoch: 24 [144/250 18432/32000 (58%)] Loss: 2.06597 (QuantReg: 9.81899) QuantErr: 9.81899 batch_time=0.50740 
Train Epoch: 24 [155/250 19840/32000 (62%)] Loss: 1.74676 (QuantReg: 9.02961) QuantErr: 9.02961 batch_time=2.68024 
Train Epoch: 24 [166/250 21248/32000 (66%)] Loss: 1.48410 (QuantReg: 9.59378) QuantErr: 9.59378 batch_time=0.50931 
Train Epoch: 24 [177/250 22656/32000 (71%)] Loss: 1.50822 (QuantReg: 9.66948) QuantErr: 9.66948 batch_time=0.49541 
Train Epoch: 24 [188/250 24064/32000 (75%)] Loss: 1.45644 (QuantReg: 9.48560) QuantErr: 9.48560 batch_time=0.50988 
Train Epoch: 24 [199/250 25472/32000 (80%)] Loss: 1.71076 (QuantReg: 9.44734) QuantErr: 9.44734 batch_time=1.09073 
Train Epoch: 24 [210/250 26880/32000 (84%)] Loss: 1.51810 (QuantReg: 9.66264) QuantErr: 9.66264 batch_time=0.49060 
Train Epoch: 24 [221/250 28288/32000 (88%)] Loss: 1.80349 (QuantReg: 9.25891) QuantErr: 9.25891 batch_time=0.49585 
Train Epoch: 24 [232/250 29696/32000 (93%)] Loss: 1.74140 (QuantReg: 9.30144) QuantErr: 9.30144 batch_time=0.50023 
Train Epoch: 24 [243/250 31104/32000 (97%)] Loss: 1.59775 (QuantReg: 9.36683) QuantErr: 9.36683 batch_time=0.49100 
Train Epoch: 24 codebook_update_time=1.73152
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch24.pth ...
Done in 10.311s
removing stale ckpt [epoch 23] [took 0.00s]
 epoch          : 24
 loss           : 1.6344546637535096
 quant_reg      : 9.497603416442871
 quant_err      : 9.497603416442871
 learning_rate  : 1.5367843386251178e-05
 n_samples      : 768000
 n_steps        : 6000
 MSRVTT_full_val/t2v_metrics/R1: 31.58953722334004
 MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
 MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.678068410462776
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.77344015433668
 MSRVTT_full_val/v2t_metrics/R1: 33.601609657947684
 MSRVTT_full_val/v2t_metrics/R5: 71.62977867203219
 MSRVTT_full_val/v2t_metrics/R10: 86.11670020120724
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.462776659959759
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.18073536753132
 MSRVTT_full_test/t2v_metrics/R1: 11.906354515050166
 MSRVTT_full_test/t2v_metrics/R5: 33.7123745819398
 MSRVTT_full_test/t2v_metrics/R10: 46.55518394648829
 MSRVTT_full_test/t2v_metrics/R50: 78.66220735785953
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.197324414715716
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.53660745361662
 MSRVTT_full_test/v2t_metrics/R1: 13.076923076923077
 MSRVTT_full_test/v2t_metrics/R5: 37.19063545150502
 MSRVTT_full_test/v2t_metrics/R10: 51.97324414715719
 MSRVTT_full_test/v2t_metrics/R50: 83.4448160535117
 MSRVTT_full_test/v2t_metrics/MedR: 10.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.087959866220736
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.347627497770603
 mnt_best       : 27.475745594051897
 not_improved_count: 2
Train Epoch: 25 [1/250 128/32000 (0%)] Loss: 1.87631 (QuantReg: 9.38880) QuantErr: 9.38880 batch_time=36.57174 
Train Epoch: 25 [12/250 1536/32000 (5%)] Loss: 1.59211 (QuantReg: 9.58848) QuantErr: 9.58848 batch_time=0.47984 
Train Epoch: 25 [23/250 2944/32000 (9%)] Loss: 1.47157 (QuantReg: 9.80046) QuantErr: 9.80046 batch_time=0.48760 
Train Epoch: 25 [34/250 4352/32000 (14%)] Loss: 1.74339 (QuantReg: 9.24611) QuantErr: 9.24611 batch_time=0.48759 
Train Epoch: 25 [45/250 5760/32000 (18%)] Loss: 1.57300 (QuantReg: 9.52405) QuantErr: 9.52405 batch_time=0.52430 
Train Epoch: 25 [56/250 7168/32000 (22%)] Loss: 1.61576 (QuantReg: 9.53586) QuantErr: 9.53586 batch_time=0.52552 
Train Epoch: 25 [67/250 8576/32000 (27%)] Loss: 1.69465 (QuantReg: 9.14407) QuantErr: 9.14407 batch_time=0.51360 
Train Epoch: 25 [78/250 9984/32000 (31%)] Loss: 1.72454 (QuantReg: 9.70256) QuantErr: 9.70256 batch_time=0.49966 
Train Epoch: 25 [89/250 11392/32000 (36%)] Loss: 1.51487 (QuantReg: 9.55976) QuantErr: 9.55976 batch_time=0.51408 
Train Epoch: 25 [100/250 12800/32000 (40%)] Loss: 1.67492 (QuantReg: 9.37941) QuantErr: 9.37941 batch_time=0.49836 
Train Epoch: 25 [111/250 14208/32000 (44%)] Loss: 1.77498 (QuantReg: 9.37268) QuantErr: 9.37268 batch_time=0.48960 
Train Epoch: 25 [122/250 15616/32000 (49%)] Loss: 1.44852 (QuantReg: 9.31125) QuantErr: 9.31125 batch_time=0.50905 
Train Epoch: 25 [133/250 17024/32000 (53%)] Loss: 1.64768 (QuantReg: 9.66872) QuantErr: 9.66872 batch_time=0.50258 
Train Epoch: 25 [144/250 18432/32000 (58%)] Loss: 1.58332 (QuantReg: 9.68471) QuantErr: 9.68471 batch_time=0.50378 
Train Epoch: 25 [155/250 19840/32000 (62%)] Loss: 1.73904 (QuantReg: 9.51041) QuantErr: 9.51041 batch_time=0.49644 
Train Epoch: 25 [166/250 21248/32000 (66%)] Loss: 1.84669 (QuantReg: 9.20688) QuantErr: 9.20688 batch_time=0.49500 
Train Epoch: 25 [177/250 22656/32000 (71%)] Loss: 1.59643 (QuantReg: 9.74953) QuantErr: 9.74953 batch_time=0.49585 
Train Epoch: 25 [188/250 24064/32000 (75%)] Loss: 1.57243 (QuantReg: 9.74500) QuantErr: 9.74500 batch_time=0.49556 
Train Epoch: 25 [199/250 25472/32000 (80%)] Loss: 1.52286 (QuantReg: 9.55844) QuantErr: 9.55844 batch_time=0.55741 
Train Epoch: 25 [210/250 26880/32000 (84%)] Loss: 1.62697 (QuantReg: 9.58928) QuantErr: 9.58928 batch_time=1.90238 
Train Epoch: 25 [221/250 28288/32000 (88%)] Loss: 1.66391 (QuantReg: 9.55161) QuantErr: 9.55161 batch_time=0.63621 
Train Epoch: 25 [232/250 29696/32000 (93%)] Loss: 1.71058 (QuantReg: 9.41155) QuantErr: 9.41155 batch_time=0.50971 
Train Epoch: 25 [243/250 31104/32000 (97%)] Loss: 1.57841 (QuantReg: 9.31143) QuantErr: 9.31143 batch_time=0.51377 
Train Epoch: 25 codebook_update_time=1.78514
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch25.pth ...
Done in 4.215s
removing stale ckpt [epoch 24] [took 0.00s]
 epoch          : 25
 loss           : 1.6320488305091858
 quant_reg      : 9.50203385925293
 quant_err      : 9.50203385925293
 learning_rate  : 1.4599451216938618e-05
 n_samples      : 800000
 n_steps        : 6250
 MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
 MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
 MSRVTT_full_val/t2v_metrics/R10: 81.08651911468813
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.585513078470825
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.86328755580814
 MSRVTT_full_val/v2t_metrics/R1: 33.199195171026155
 MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
 MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.350100603621731
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.9626139019553
 MSRVTT_full_test/t2v_metrics/R1: 11.97324414715719
 MSRVTT_full_test/t2v_metrics/R5: 33.57859531772575
 MSRVTT_full_test/t2v_metrics/R10: 47.391304347826086
 MSRVTT_full_test/t2v_metrics/R50: 79.26421404682274
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.50133779264214
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.70900287900066
 MSRVTT_full_test/v2t_metrics/R1: 13.511705685618729
 MSRVTT_full_test/v2t_metrics/R5: 38.12709030100334
 MSRVTT_full_test/v2t_metrics/R10: 53.01003344481605
 MSRVTT_full_test/v2t_metrics/R50: 83.81270903010034
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.86220735785953
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.11392094853091
 mnt_best       : 27.475745594051897
 not_improved_count: 3
Train Epoch: 26 [1/250 128/32000 (0%)] Loss: 1.75686 (QuantReg: 9.39587) QuantErr: 9.39587 batch_time=35.28262 
Train Epoch: 26 [12/250 1536/32000 (5%)] Loss: 1.55873 (QuantReg: 9.24529) QuantErr: 9.24529 batch_time=0.50240 
Train Epoch: 26 [23/250 2944/32000 (9%)] Loss: 1.22223 (QuantReg: 9.52730) QuantErr: 9.52730 batch_time=0.49498 
Train Epoch: 26 [34/250 4352/32000 (14%)] Loss: 1.51600 (QuantReg: 9.48476) QuantErr: 9.48476 batch_time=0.49742 
Train Epoch: 26 [45/250 5760/32000 (18%)] Loss: 1.54608 (QuantReg: 9.45098) QuantErr: 9.45098 batch_time=0.53504 
Train Epoch: 26 [56/250 7168/32000 (22%)] Loss: 1.71246 (QuantReg: 9.30539) QuantErr: 9.30539 batch_time=0.49643 
Train Epoch: 26 [67/250 8576/32000 (27%)] Loss: 1.53860 (QuantReg: 9.74234) QuantErr: 9.74234 batch_time=1.53346 
Train Epoch: 26 [78/250 9984/32000 (31%)] Loss: 1.70033 (QuantReg: 9.58926) QuantErr: 9.58926 batch_time=0.49203 
Train Epoch: 26 [89/250 11392/32000 (36%)] Loss: 1.55923 (QuantReg: 9.55369) QuantErr: 9.55369 batch_time=0.49036 
Train Epoch: 26 [100/250 12800/32000 (40%)] Loss: 1.51909 (QuantReg: 9.31560) QuantErr: 9.31560 batch_time=0.49108 
Train Epoch: 26 [111/250 14208/32000 (44%)] Loss: 1.39040 (QuantReg: 9.53403) QuantErr: 9.53403 batch_time=0.48683 
Train Epoch: 26 [122/250 15616/32000 (49%)] Loss: 1.44758 (QuantReg: 9.47975) QuantErr: 9.47975 batch_time=0.48893 
Train Epoch: 26 [133/250 17024/32000 (53%)] Loss: 1.44450 (QuantReg: 9.03290) QuantErr: 9.03290 batch_time=0.50049 
Train Epoch: 26 [144/250 18432/32000 (58%)] Loss: 1.52246 (QuantReg: 9.34006) QuantErr: 9.34006 batch_time=0.50614 
Train Epoch: 26 [155/250 19840/32000 (62%)] Loss: 1.77438 (QuantReg: 9.29950) QuantErr: 9.29950 batch_time=0.62451 
Train Epoch: 26 [166/250 21248/32000 (66%)] Loss: 1.69911 (QuantReg: 9.70765) QuantErr: 9.70765 batch_time=0.49775 
Train Epoch: 26 [177/250 22656/32000 (71%)] Loss: 1.58739 (QuantReg: 9.26162) QuantErr: 9.26162 batch_time=0.53536 
Train Epoch: 26 [188/250 24064/32000 (75%)] Loss: 1.40858 (QuantReg: 9.56770) QuantErr: 9.56770 batch_time=0.50011 
Train Epoch: 26 [199/250 25472/32000 (80%)] Loss: 1.55671 (QuantReg: 9.66152) QuantErr: 9.66152 batch_time=0.49671 
Train Epoch: 26 [210/250 26880/32000 (84%)] Loss: 1.39430 (QuantReg: 9.71688) QuantErr: 9.71688 batch_time=0.69092 
Train Epoch: 26 [221/250 28288/32000 (88%)] Loss: 2.11364 (QuantReg: 9.52971) QuantErr: 9.52971 batch_time=0.50234 
Train Epoch: 26 [232/250 29696/32000 (93%)] Loss: 1.51318 (QuantReg: 9.48959) QuantErr: 9.48959 batch_time=0.52010 
Train Epoch: 26 [243/250 31104/32000 (97%)] Loss: 1.43173 (QuantReg: 9.40472) QuantErr: 9.40472 batch_time=0.50660 
Train Epoch: 26 codebook_update_time=1.69006
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch26.pth ...
Done in 3.853s
removing stale ckpt [epoch 25] [took 0.01s]
 epoch          : 26
 loss           : 1.6292129535675048
 quant_reg      : 9.496189002990723
 quant_err      : 9.496189002990723
 learning_rate  : 1.3869478656091687e-05
 n_samples      : 832000
 n_steps        : 6500
 MSRVTT_full_val/t2v_metrics/R1: 30.382293762575454
 MSRVTT_full_val/t2v_metrics/R5: 67.6056338028169
 MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
 MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.509054325955734
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.88272711546331
 MSRVTT_full_val/v2t_metrics/R1: 34.00402414486921
 MSRVTT_full_val/v2t_metrics/R5: 70.82494969818913
 MSRVTT_full_val/v2t_metrics/R10: 85.31187122736418
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 3.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.181086519114688
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.00769987870178
 MSRVTT_full_test/t2v_metrics/R1: 11.505016722408026
 MSRVTT_full_test/t2v_metrics/R5: 34.147157190635454
 MSRVTT_full_test/t2v_metrics/R10: 47.525083612040135
 MSRVTT_full_test/t2v_metrics/R50: 79.36454849498328
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.431103678929766
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.52904191284754
 MSRVTT_full_test/v2t_metrics/R1: 13.812709030100335
 MSRVTT_full_test/v2t_metrics/R5: 37.22408026755853
 MSRVTT_full_test/v2t_metrics/R10: 53.1438127090301
 MSRVTT_full_test/v2t_metrics/R50: 83.84615384615384
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.45284280936455
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.11978374709078
 mnt_best       : 27.475745594051897
 not_improved_count: 4
Train Epoch: 27 [1/250 128/32000 (0%)] Loss: 1.35202 (QuantReg: 9.73993) QuantErr: 9.73993 batch_time=36.50492 
Train Epoch: 27 [12/250 1536/32000 (5%)] Loss: 1.72841 (QuantReg: 9.65887) QuantErr: 9.65887 batch_time=0.48723 
Train Epoch: 27 [23/250 2944/32000 (9%)] Loss: 1.66040 (QuantReg: 9.65072) QuantErr: 9.65072 batch_time=0.51217 
Train Epoch: 27 [34/250 4352/32000 (14%)] Loss: 1.63189 (QuantReg: 9.50932) QuantErr: 9.50932 batch_time=0.50930 
Train Epoch: 27 [45/250 5760/32000 (18%)] Loss: 1.53765 (QuantReg: 9.37343) QuantErr: 9.37343 batch_time=0.48914 
Train Epoch: 27 [56/250 7168/32000 (22%)] Loss: 1.28375 (QuantReg: 9.76724) QuantErr: 9.76724 batch_time=0.53004 
Train Epoch: 27 [67/250 8576/32000 (27%)] Loss: 1.45150 (QuantReg: 9.53555) QuantErr: 9.53555 batch_time=0.50506 
Train Epoch: 27 [78/250 9984/32000 (31%)] Loss: 1.46407 (QuantReg: 9.33321) QuantErr: 9.33321 batch_time=0.54087 
Train Epoch: 27 [89/250 11392/32000 (36%)] Loss: 1.66074 (QuantReg: 9.47112) QuantErr: 9.47112 batch_time=0.49795 
Train Epoch: 27 [100/250 12800/32000 (40%)] Loss: 1.70379 (QuantReg: 9.17705) QuantErr: 9.17705 batch_time=0.51775 
Train Epoch: 27 [111/250 14208/32000 (44%)] Loss: 1.39828 (QuantReg: 9.58154) QuantErr: 9.58154 batch_time=0.49190 
Train Epoch: 27 [122/250 15616/32000 (49%)] Loss: 1.65335 (QuantReg: 9.45369) QuantErr: 9.45369 batch_time=0.51153 
Train Epoch: 27 [133/250 17024/32000 (53%)] Loss: 1.59697 (QuantReg: 9.32389) QuantErr: 9.32389 batch_time=0.80067 
Train Epoch: 27 [144/250 18432/32000 (58%)] Loss: 1.42975 (QuantReg: 9.43487) QuantErr: 9.43487 batch_time=1.27916 
Train Epoch: 27 [155/250 19840/32000 (62%)] Loss: 1.51754 (QuantReg: 9.70921) QuantErr: 9.70921 batch_time=0.50871 
Train Epoch: 27 [166/250 21248/32000 (66%)] Loss: 1.61253 (QuantReg: 9.53670) QuantErr: 9.53670 batch_time=0.50245 
Train Epoch: 27 [177/250 22656/32000 (71%)] Loss: 1.69436 (QuantReg: 9.38847) QuantErr: 9.38847 batch_time=0.53783 
Train Epoch: 27 [188/250 24064/32000 (75%)] Loss: 1.56096 (QuantReg: 9.32370) QuantErr: 9.32370 batch_time=0.52317 
Train Epoch: 27 [199/250 25472/32000 (80%)] Loss: 1.69969 (QuantReg: 9.43791) QuantErr: 9.43791 batch_time=0.49864 
Train Epoch: 27 [210/250 26880/32000 (84%)] Loss: 1.49543 (QuantReg: 9.90478) QuantErr: 9.90478 batch_time=0.49228 
Train Epoch: 27 [221/250 28288/32000 (88%)] Loss: 1.48855 (QuantReg: 9.34812) QuantErr: 9.34812 batch_time=0.51210 
Train Epoch: 27 [232/250 29696/32000 (93%)] Loss: 1.50845 (QuantReg: 9.48376) QuantErr: 9.48376 batch_time=0.49931 
Train Epoch: 27 [243/250 31104/32000 (97%)] Loss: 1.56630 (QuantReg: 9.54984) QuantErr: 9.54984 batch_time=0.52387 
Train Epoch: 27 codebook_update_time=1.73034
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch27.pth ...
Done in 4.085s
removing stale ckpt [epoch 26] [took 0.00s]
 epoch          : 27
 loss           : 1.5742400102615357
 quant_reg      : 9.457121089935303
 quant_err      : 9.457121089935303
 learning_rate  : 1.3176004723287102e-05
 n_samples      : 864000
 n_steps        : 6750
 MSRVTT_full_val/t2v_metrics/R1: 32.394366197183096
 MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
 MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.847082494969818
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.290080887835586
 MSRVTT_full_val/v2t_metrics/R1: 35.61368209255533
 MSRVTT_full_val/v2t_metrics/R5: 71.42857142857143
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.020120724346077
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.04694824030683
 MSRVTT_full_test/t2v_metrics/R1: 12.307692307692308
 MSRVTT_full_test/t2v_metrics/R5: 33.94648829431438
 MSRVTT_full_test/t2v_metrics/R10: 47.391304347826086
 MSRVTT_full_test/t2v_metrics/R50: 78.66220735785953
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 51.738127090301006
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.05349519640939
 MSRVTT_full_test/v2t_metrics/R1: 13.010033444816054
 MSRVTT_full_test/v2t_metrics/R5: 38.42809364548495
 MSRVTT_full_test/v2t_metrics/R10: 52.97658862876254
 MSRVTT_full_test/v2t_metrics/R50: 83.87959866220736
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.23311036789298
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.80829123625354
 mnt_best       : 27.475745594051897
 not_improved_count: 5
Train Epoch: 28 [1/250 128/32000 (0%)] Loss: 1.44572 (QuantReg: 9.47980) QuantErr: 9.47980 batch_time=32.28837 
Train Epoch: 28 [12/250 1536/32000 (5%)] Loss: 1.81166 (QuantReg: 9.58827) QuantErr: 9.58827 batch_time=0.50637 
Train Epoch: 28 [23/250 2944/32000 (9%)] Loss: 1.50172 (QuantReg: 9.41248) QuantErr: 9.41248 batch_time=0.49704 
Train Epoch: 28 [34/250 4352/32000 (14%)] Loss: 1.64953 (QuantReg: 9.62468) QuantErr: 9.62468 batch_time=0.50112 
Train Epoch: 28 [45/250 5760/32000 (18%)] Loss: 1.92405 (QuantReg: 9.64126) QuantErr: 9.64126 batch_time=0.49001 
Train Epoch: 28 [56/250 7168/32000 (22%)] Loss: 1.73037 (QuantReg: 9.85366) QuantErr: 9.85366 batch_time=0.52172 
Train Epoch: 28 [67/250 8576/32000 (27%)] Loss: 2.00953 (QuantReg: 9.60544) QuantErr: 9.60544 batch_time=2.45375 
Train Epoch: 28 [78/250 9984/32000 (31%)] Loss: 1.48583 (QuantReg: 9.74305) QuantErr: 9.74305 batch_time=0.49213 
Train Epoch: 28 [89/250 11392/32000 (36%)] Loss: 1.70971 (QuantReg: 9.58978) QuantErr: 9.58978 batch_time=0.50483 
Train Epoch: 28 [100/250 12800/32000 (40%)] Loss: 1.63367 (QuantReg: 9.46907) QuantErr: 9.46907 batch_time=0.49314 
Train Epoch: 28 [111/250 14208/32000 (44%)] Loss: 1.61470 (QuantReg: 9.43194) QuantErr: 9.43194 batch_time=0.49420 
Train Epoch: 28 [122/250 15616/32000 (49%)] Loss: 1.58738 (QuantReg: 9.51909) QuantErr: 9.51909 batch_time=0.54365 
Train Epoch: 28 [133/250 17024/32000 (53%)] Loss: 1.64596 (QuantReg: 9.68825) QuantErr: 9.68825 batch_time=0.49810 
Train Epoch: 28 [144/250 18432/32000 (58%)] Loss: 1.75821 (QuantReg: 9.11137) QuantErr: 9.11137 batch_time=0.61688 
Train Epoch: 28 [155/250 19840/32000 (62%)] Loss: 1.82263 (QuantReg: 9.45008) QuantErr: 9.45008 batch_time=0.50204 
Train Epoch: 28 [166/250 21248/32000 (66%)] Loss: 1.31767 (QuantReg: 9.30167) QuantErr: 9.30167 batch_time=0.49834 
Train Epoch: 28 [177/250 22656/32000 (71%)] Loss: 1.78022 (QuantReg: 9.38097) QuantErr: 9.38097 batch_time=0.58064 
Train Epoch: 28 [188/250 24064/32000 (75%)] Loss: 1.59257 (QuantReg: 9.26343) QuantErr: 9.26343 batch_time=0.50623 
Train Epoch: 28 [199/250 25472/32000 (80%)] Loss: 1.41919 (QuantReg: 9.47310) QuantErr: 9.47310 batch_time=0.52121 
Train Epoch: 28 [210/250 26880/32000 (84%)] Loss: 1.71251 (QuantReg: 9.79108) QuantErr: 9.79108 batch_time=3.07936 
Train Epoch: 28 [221/250 28288/32000 (88%)] Loss: 1.69946 (QuantReg: 9.41596) QuantErr: 9.41596 batch_time=1.01494 
Train Epoch: 28 [232/250 29696/32000 (93%)] Loss: 1.88543 (QuantReg: 9.18791) QuantErr: 9.18791 batch_time=0.51931 
Train Epoch: 28 [243/250 31104/32000 (97%)] Loss: 1.60576 (QuantReg: 9.65066) QuantErr: 9.65066 batch_time=0.53516 
Train Epoch: 28 codebook_update_time=1.64684
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch28.pth ...
Done in 3.853s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch28.pth ...
Done in 7.749s
removing stale ckpt [epoch 27] [took 0.00s]
 epoch          : 28
 loss           : 1.5870485367774962
 quant_reg      : 9.491376190185546
 quant_err      : 9.491376190185546
 learning_rate  : 1.2517204487122746e-05
 n_samples      : 896000
 n_steps        : 7000
 MSRVTT_full_val/t2v_metrics/R1: 33.199195171026155
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 81.89134808853119
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.056338028169014
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.02556914820245
 MSRVTT_full_val/v2t_metrics/R1: 35.2112676056338
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 85.51307847082495
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.8631790744466805
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.24971609750558
 MSRVTT_full_test/t2v_metrics/R1: 12.876254180602007
 MSRVTT_full_test/t2v_metrics/R5: 34.414715719063544
 MSRVTT_full_test/t2v_metrics/R10: 48.09364548494983
 MSRVTT_full_test/t2v_metrics/R50: 79.7324414715719
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.940133779264215
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.72514387005249
 MSRVTT_full_test/v2t_metrics/R1: 14.448160535117056
 MSRVTT_full_test/v2t_metrics/R5: 38.72909698996656
 MSRVTT_full_test/v2t_metrics/R10: 53.04347826086956
 MSRVTT_full_test/v2t_metrics/R50: 84.68227424749163
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.46889632107023
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.96187892877575
 mnt_best       : 27.72514387005249
 not_improved_count: 0
Train Epoch: 29 [1/250 128/32000 (0%)] Loss: 1.65278 (QuantReg: 9.49584) QuantErr: 9.49584 batch_time=32.13316 
Train Epoch: 29 [12/250 1536/32000 (5%)] Loss: 1.76951 (QuantReg: 9.44203) QuantErr: 9.44203 batch_time=0.51602 
Train Epoch: 29 [23/250 2944/32000 (9%)] Loss: 1.77260 (QuantReg: 9.06223) QuantErr: 9.06223 batch_time=0.87166 
Train Epoch: 29 [34/250 4352/32000 (14%)] Loss: 1.52656 (QuantReg: 9.23860) QuantErr: 9.23860 batch_time=0.51415 
Train Epoch: 29 [45/250 5760/32000 (18%)] Loss: 1.60518 (QuantReg: 9.40908) QuantErr: 9.40908 batch_time=0.49984 
Train Epoch: 29 [56/250 7168/32000 (22%)] Loss: 1.78731 (QuantReg: 9.59069) QuantErr: 9.59069 batch_time=0.48864 
Train Epoch: 29 [67/250 8576/32000 (27%)] Loss: 1.57707 (QuantReg: 9.69454) QuantErr: 9.69454 batch_time=0.49747 
Train Epoch: 29 [78/250 9984/32000 (31%)] Loss: 1.63376 (QuantReg: 9.49892) QuantErr: 9.49892 batch_time=0.52031 
Train Epoch: 29 [89/250 11392/32000 (36%)] Loss: 1.47980 (QuantReg: 9.79396) QuantErr: 9.79396 batch_time=0.50218 
Train Epoch: 29 [100/250 12800/32000 (40%)] Loss: 1.32773 (QuantReg: 9.48690) QuantErr: 9.48690 batch_time=0.51821 
Train Epoch: 29 [111/250 14208/32000 (44%)] Loss: 1.74837 (QuantReg: 9.25016) QuantErr: 9.25016 batch_time=0.51086 
Train Epoch: 29 [122/250 15616/32000 (49%)] Loss: 1.52327 (QuantReg: 9.68509) QuantErr: 9.68509 batch_time=0.53145 
Train Epoch: 29 [133/250 17024/32000 (53%)] Loss: 1.69610 (QuantReg: 9.45139) QuantErr: 9.45139 batch_time=0.50720 
Train Epoch: 29 [144/250 18432/32000 (58%)] Loss: 1.67076 (QuantReg: 9.45663) QuantErr: 9.45663 batch_time=4.12859 
Train Epoch: 29 [155/250 19840/32000 (62%)] Loss: 1.50218 (QuantReg: 9.31887) QuantErr: 9.31887 batch_time=0.50426 
Train Epoch: 29 [166/250 21248/32000 (66%)] Loss: 1.63604 (QuantReg: 9.51132) QuantErr: 9.51132 batch_time=0.62019 
Train Epoch: 29 [177/250 22656/32000 (71%)] Loss: 1.69469 (QuantReg: 9.28178) QuantErr: 9.28178 batch_time=0.49957 
Train Epoch: 29 [188/250 24064/32000 (75%)] Loss: 1.45011 (QuantReg: 9.23339) QuantErr: 9.23339 batch_time=0.50801 
Train Epoch: 29 [199/250 25472/32000 (80%)] Loss: 1.69601 (QuantReg: 9.41935) QuantErr: 9.41935 batch_time=0.85790 
Train Epoch: 29 [210/250 26880/32000 (84%)] Loss: 1.36293 (QuantReg: 9.59768) QuantErr: 9.59768 batch_time=0.51740 
Train Epoch: 29 [221/250 28288/32000 (88%)] Loss: 1.44775 (QuantReg: 9.25959) QuantErr: 9.25959 batch_time=0.51160 
Train Epoch: 29 [232/250 29696/32000 (93%)] Loss: 1.54525 (QuantReg: 9.42095) QuantErr: 9.42095 batch_time=0.49924 
Train Epoch: 29 [243/250 31104/32000 (97%)] Loss: 1.36491 (QuantReg: 9.49924) QuantErr: 9.49924 batch_time=0.84647 
Train Epoch: 29 codebook_update_time=1.66380
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch29.pth ...
Done in 4.182s
removing stale ckpt [epoch 28] [took 0.00s]
 epoch          : 29
 loss           : 1.5719065680503845
 quant_reg      : 9.458993873596192
 quant_err      : 9.458993873596192
 learning_rate  : 1.1891344262766608e-05
 n_samples      : 928000
 n_steps        : 7250
 MSRVTT_full_val/t2v_metrics/R1: 31.790744466800806
 MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
 MSRVTT_full_val/t2v_metrics/R10: 81.69014084507042
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.253521126760564
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.21656306684067
 MSRVTT_full_val/v2t_metrics/R1: 34.80885311871227
 MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
 MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.3420523138833
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.995701930452135
 MSRVTT_full_test/t2v_metrics/R1: 12.073578595317725
 MSRVTT_full_test/t2v_metrics/R5: 34.48160535117057
 MSRVTT_full_test/t2v_metrics/R10: 48.06020066889632
 MSRVTT_full_test/t2v_metrics/R50: 79.6989966555184
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.5876254180602
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.14790725095668
 MSRVTT_full_test/v2t_metrics/R1: 13.7123745819398
 MSRVTT_full_test/v2t_metrics/R5: 37.725752508361204
 MSRVTT_full_test/v2t_metrics/R10: 52.541806020066886
 MSRVTT_full_test/v2t_metrics/R50: 83.9799331103679
 MSRVTT_full_test/v2t_metrics/MedR: 9.25
 MSRVTT_full_test/v2t_metrics/MeanR: 39.19130434782609
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.066660388954304
 mnt_best       : 27.72514387005249
 not_improved_count: 1
Train Epoch: 30 [1/250 128/32000 (0%)] Loss: 1.39840 (QuantReg: 9.33711) QuantErr: 9.33711 batch_time=34.70718 
Train Epoch: 30 [12/250 1536/32000 (5%)] Loss: 1.63738 (QuantReg: 9.41542) QuantErr: 9.41542 batch_time=0.49634 
Train Epoch: 30 [23/250 2944/32000 (9%)] Loss: 1.58396 (QuantReg: 9.45449) QuantErr: 9.45449 batch_time=0.50897 
Train Epoch: 30 [34/250 4352/32000 (14%)] Loss: 1.68526 (QuantReg: 9.52879) QuantErr: 9.52879 batch_time=0.49522 
Train Epoch: 30 [45/250 5760/32000 (18%)] Loss: 1.53688 (QuantReg: 9.43845) QuantErr: 9.43845 batch_time=0.49363 
Train Epoch: 30 [56/250 7168/32000 (22%)] Loss: 1.54860 (QuantReg: 9.73524) QuantErr: 9.73524 batch_time=0.52573 
Train Epoch: 30 [67/250 8576/32000 (27%)] Loss: 1.58817 (QuantReg: 9.54157) QuantErr: 9.54157 batch_time=0.51070 
Train Epoch: 30 [78/250 9984/32000 (31%)] Loss: 1.46060 (QuantReg: 9.73549) QuantErr: 9.73549 batch_time=1.17062 
Train Epoch: 30 [89/250 11392/32000 (36%)] Loss: 1.66629 (QuantReg: 9.26822) QuantErr: 9.26822 batch_time=0.50992 
Train Epoch: 30 [100/250 12800/32000 (40%)] Loss: 1.39507 (QuantReg: 9.64842) QuantErr: 9.64842 batch_time=0.54362 
Train Epoch: 30 [111/250 14208/32000 (44%)] Loss: 1.47957 (QuantReg: 9.54216) QuantErr: 9.54216 batch_time=0.49814 
Train Epoch: 30 [122/250 15616/32000 (49%)] Loss: 1.51786 (QuantReg: 9.39154) QuantErr: 9.39154 batch_time=0.53732 
Train Epoch: 30 [133/250 17024/32000 (53%)] Loss: 1.47342 (QuantReg: 9.50413) QuantErr: 9.50413 batch_time=0.50743 
Train Epoch: 30 [144/250 18432/32000 (58%)] Loss: 1.57058 (QuantReg: 9.35701) QuantErr: 9.35701 batch_time=0.51266 
Train Epoch: 30 [155/250 19840/32000 (62%)] Loss: 1.57317 (QuantReg: 9.53713) QuantErr: 9.53713 batch_time=0.48587 
Train Epoch: 30 [166/250 21248/32000 (66%)] Loss: 1.58199 (QuantReg: 9.54922) QuantErr: 9.54922 batch_time=0.51391 
Train Epoch: 30 [177/250 22656/32000 (71%)] Loss: 1.46866 (QuantReg: 9.33984) QuantErr: 9.33984 batch_time=0.50660 
Train Epoch: 30 [188/250 24064/32000 (75%)] Loss: 1.30525 (QuantReg: 9.57175) QuantErr: 9.57175 batch_time=0.50130 
Train Epoch: 30 [199/250 25472/32000 (80%)] Loss: 1.80020 (QuantReg: 9.33133) QuantErr: 9.33133 batch_time=0.50318 
Train Epoch: 30 [210/250 26880/32000 (84%)] Loss: 1.60609 (QuantReg: 9.40399) QuantErr: 9.40399 batch_time=1.97934 
Train Epoch: 30 [221/250 28288/32000 (88%)] Loss: 1.56539 (QuantReg: 9.59635) QuantErr: 9.59635 batch_time=0.50664 
Train Epoch: 30 [232/250 29696/32000 (93%)] Loss: 1.88948 (QuantReg: 9.62592) QuantErr: 9.62592 batch_time=0.50553 
Train Epoch: 30 [243/250 31104/32000 (97%)] Loss: 1.93958 (QuantReg: 9.42411) QuantErr: 9.42411 batch_time=0.49313 
Train Epoch: 30 codebook_update_time=1.64080
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch30.pth ...
Done in 4.260s
removing stale ckpt [epoch 29] [took 0.00s]
 epoch          : 30
 loss           : 1.5409939980506897
 quant_reg      : 9.47495951461792
 quant_err      : 9.47495951461792
 learning_rate  : 1.1296777049628277e-05
 n_samples      : 960000
 n_steps        : 7500
 MSRVTT_full_val/t2v_metrics/R1: 31.790744466800806
 MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
 MSRVTT_full_val/t2v_metrics/R10: 80.6841046277666
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.106639839034205
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.98483511474458
 MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.905432595573441
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.496855538462945
 MSRVTT_full_test/t2v_metrics/R1: 12.74247491638796
 MSRVTT_full_test/t2v_metrics/R5: 34.280936454849495
 MSRVTT_full_test/t2v_metrics/R10: 47.82608695652174
 MSRVTT_full_test/t2v_metrics/R50: 79.59866220735786
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.122408026755856
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.541680623428316
 MSRVTT_full_test/v2t_metrics/R1: 13.411371237458194
 MSRVTT_full_test/v2t_metrics/R5: 37.625418060200666
 MSRVTT_full_test/v2t_metrics/R10: 53.11036789297659
 MSRVTT_full_test/v2t_metrics/R50: 84.21404682274247
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.48478260869565
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.92572008625922
 mnt_best       : 27.72514387005249
 not_improved_count: 2
Train Epoch: 31 [1/250 128/32000 (0%)] Loss: 1.70824 (QuantReg: 9.41495) QuantErr: 9.41495 batch_time=37.11484 
Train Epoch: 31 [12/250 1536/32000 (5%)] Loss: 1.63295 (QuantReg: 9.49536) QuantErr: 9.49536 batch_time=0.48787 
Train Epoch: 31 [23/250 2944/32000 (9%)] Loss: 1.76452 (QuantReg: 9.39657) QuantErr: 9.39657 batch_time=0.48995 
Train Epoch: 31 [34/250 4352/32000 (14%)] Loss: 1.39103 (QuantReg: 9.36294) QuantErr: 9.36294 batch_time=0.54738 
Train Epoch: 31 [45/250 5760/32000 (18%)] Loss: 1.69889 (QuantReg: 9.43865) QuantErr: 9.43865 batch_time=0.78603 
Train Epoch: 31 [56/250 7168/32000 (22%)] Loss: 1.77662 (QuantReg: 8.79921) QuantErr: 8.79921 batch_time=0.48495 
Train Epoch: 31 [67/250 8576/32000 (27%)] Loss: 1.59746 (QuantReg: 9.42052) QuantErr: 9.42052 batch_time=0.53549 
Train Epoch: 31 [78/250 9984/32000 (31%)] Loss: 1.63067 (QuantReg: 9.51882) QuantErr: 9.51882 batch_time=0.50545 
Train Epoch: 31 [89/250 11392/32000 (36%)] Loss: 1.60793 (QuantReg: 9.34359) QuantErr: 9.34359 batch_time=0.50902 
Train Epoch: 31 [100/250 12800/32000 (40%)] Loss: 1.49398 (QuantReg: 9.33097) QuantErr: 9.33097 batch_time=0.57190 
Train Epoch: 31 [111/250 14208/32000 (44%)] Loss: 1.44100 (QuantReg: 9.66949) QuantErr: 9.66949 batch_time=0.50950 
Train Epoch: 31 [122/250 15616/32000 (49%)] Loss: 1.33364 (QuantReg: 9.43745) QuantErr: 9.43745 batch_time=0.50313 
Train Epoch: 31 [133/250 17024/32000 (53%)] Loss: 1.52496 (QuantReg: 9.42975) QuantErr: 9.42975 batch_time=0.50130 
Train Epoch: 31 [144/250 18432/32000 (58%)] Loss: 1.92911 (QuantReg: 9.64878) QuantErr: 9.64878 batch_time=0.51655 
Train Epoch: 31 [155/250 19840/32000 (62%)] Loss: 1.51659 (QuantReg: 9.32009) QuantErr: 9.32009 batch_time=0.49175 
Train Epoch: 31 [166/250 21248/32000 (66%)] Loss: 1.48736 (QuantReg: 9.18293) QuantErr: 9.18293 batch_time=0.49608 
Train Epoch: 31 [177/250 22656/32000 (71%)] Loss: 1.91127 (QuantReg: 9.15382) QuantErr: 9.15382 batch_time=0.48926 
Train Epoch: 31 [188/250 24064/32000 (75%)] Loss: 1.54492 (QuantReg: 9.64904) QuantErr: 9.64904 batch_time=0.49345 
Train Epoch: 31 [199/250 25472/32000 (80%)] Loss: 1.45273 (QuantReg: 9.40959) QuantErr: 9.40959 batch_time=0.51520 
Train Epoch: 31 [210/250 26880/32000 (84%)] Loss: 1.75632 (QuantReg: 9.40705) QuantErr: 9.40705 batch_time=0.49192 
Train Epoch: 31 [221/250 28288/32000 (88%)] Loss: 1.43313 (QuantReg: 9.80571) QuantErr: 9.80571 batch_time=0.49655 
Train Epoch: 31 [232/250 29696/32000 (93%)] Loss: 1.50898 (QuantReg: 9.52484) QuantErr: 9.52484 batch_time=0.50350 
Train Epoch: 31 [243/250 31104/32000 (97%)] Loss: 1.58780 (QuantReg: 9.50312) QuantErr: 9.50312 batch_time=0.50047 
Train Epoch: 31 codebook_update_time=1.65839
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch31.pth ...
Done in 3.965s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch31.pth ...
Done in 7.694s
removing stale ckpt [epoch 30] [took 0.00s]
 epoch          : 31
 loss           : 1.5639903964996338
 quant_reg      : 9.47365818786621
 quant_err      : 9.47365818786621
 learning_rate  : 1.0731938197146863e-05
 n_samples      : 992000
 n_steps        : 7750
 MSRVTT_full_val/t2v_metrics/R1: 33.601609657947684
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 82.897384305835
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.056338028169014
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.48855682586208
 MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
 MSRVTT_full_val/v2t_metrics/R5: 74.04426559356136
 MSRVTT_full_val/v2t_metrics/R10: 83.90342052313883
 MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.802816901408451
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.596132039756476
 MSRVTT_full_test/t2v_metrics/R1: 13.277591973244148
 MSRVTT_full_test/t2v_metrics/R5: 34.94983277591973
 MSRVTT_full_test/t2v_metrics/R10: 48.52842809364549
 MSRVTT_full_test/t2v_metrics/R50: 79.6989966555184
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.26488294314381
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.239275360591982
 MSRVTT_full_test/v2t_metrics/R1: 13.511705685618729
 MSRVTT_full_test/v2t_metrics/R5: 38.19397993311037
 MSRVTT_full_test/v2t_metrics/R10: 52.84280936454849
 MSRVTT_full_test/v2t_metrics/R50: 84.0133779264214
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.06103678929766
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.09980372600178
 mnt_best       : 28.239275360591982
 not_improved_count: 0
Train Epoch: 32 [1/250 128/32000 (0%)] Loss: 1.34760 (QuantReg: 9.48017) QuantErr: 9.48017 batch_time=30.44557 
Train Epoch: 32 [12/250 1536/32000 (5%)] Loss: 1.60582 (QuantReg: 9.48541) QuantErr: 9.48541 batch_time=0.48587 
Train Epoch: 32 [23/250 2944/32000 (9%)] Loss: 1.46838 (QuantReg: 9.19872) QuantErr: 9.19872 batch_time=0.50274 
Train Epoch: 32 [34/250 4352/32000 (14%)] Loss: 1.47344 (QuantReg: 9.75964) QuantErr: 9.75964 batch_time=0.47794 
Train Epoch: 32 [45/250 5760/32000 (18%)] Loss: 1.26874 (QuantReg: 9.60306) QuantErr: 9.60306 batch_time=0.48039 
Train Epoch: 32 [56/250 7168/32000 (22%)] Loss: 1.53715 (QuantReg: 9.36158) QuantErr: 9.36158 batch_time=0.50565 
Train Epoch: 32 [67/250 8576/32000 (27%)] Loss: 1.24862 (QuantReg: 9.63874) QuantErr: 9.63874 batch_time=0.48792 
Train Epoch: 32 [78/250 9984/32000 (31%)] Loss: 1.58194 (QuantReg: 9.37201) QuantErr: 9.37201 batch_time=0.49349 
Train Epoch: 32 [89/250 11392/32000 (36%)] Loss: 1.60264 (QuantReg: 9.48078) QuantErr: 9.48078 batch_time=1.42070 
Train Epoch: 32 [100/250 12800/32000 (40%)] Loss: 1.52180 (QuantReg: 9.36432) QuantErr: 9.36432 batch_time=1.81871 
Train Epoch: 32 [111/250 14208/32000 (44%)] Loss: 1.52143 (QuantReg: 10.04296) QuantErr: 10.04296 batch_time=0.51811 
Train Epoch: 32 [122/250 15616/32000 (49%)] Loss: 1.34874 (QuantReg: 9.24604) QuantErr: 9.24604 batch_time=0.49185 
Train Epoch: 32 [133/250 17024/32000 (53%)] Loss: 1.67835 (QuantReg: 9.40105) QuantErr: 9.40105 batch_time=1.22526 
Train Epoch: 32 [144/250 18432/32000 (58%)] Loss: 1.48737 (QuantReg: 9.43097) QuantErr: 9.43097 batch_time=0.50058 
Train Epoch: 32 [155/250 19840/32000 (62%)] Loss: 1.38713 (QuantReg: 9.56392) QuantErr: 9.56392 batch_time=0.49532 
Train Epoch: 32 [166/250 21248/32000 (66%)] Loss: 1.56188 (QuantReg: 9.49724) QuantErr: 9.49724 batch_time=0.49070 
Train Epoch: 32 [177/250 22656/32000 (71%)] Loss: 1.61948 (QuantReg: 9.76560) QuantErr: 9.76560 batch_time=0.49406 
Train Epoch: 32 [188/250 24064/32000 (75%)] Loss: 1.61472 (QuantReg: 9.59048) QuantErr: 9.59048 batch_time=0.49266 
Train Epoch: 32 [199/250 25472/32000 (80%)] Loss: 1.45401 (QuantReg: 9.81258) QuantErr: 9.81258 batch_time=3.25315 
Train Epoch: 32 [210/250 26880/32000 (84%)] Loss: 1.87066 (QuantReg: 9.55040) QuantErr: 9.55040 batch_time=0.51465 
Train Epoch: 32 [221/250 28288/32000 (88%)] Loss: 1.42736 (QuantReg: 9.62111) QuantErr: 9.62111 batch_time=0.48282 
Train Epoch: 32 [232/250 29696/32000 (93%)] Loss: 1.60147 (QuantReg: 9.33473) QuantErr: 9.33473 batch_time=0.48908 
Train Epoch: 32 [243/250 31104/32000 (97%)] Loss: 1.26233 (QuantReg: 9.76259) QuantErr: 9.76259 batch_time=0.48401 
Train Epoch: 32 codebook_update_time=1.66191
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch32.pth ...
Done in 3.892s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch32.pth ...
Done in 7.625s
removing stale ckpt [epoch 31] [took 0.01s]
 epoch          : 32
 loss           : 1.5313026194572448
 quant_reg      : 9.478361492156983
 quant_err      : 9.478361492156983
 learning_rate  : 1.019534128728952e-05
 n_samples      : 1024000
 n_steps        : 8000
 MSRVTT_full_val/t2v_metrics/R1: 33.199195171026155
 MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
 MSRVTT_full_val/t2v_metrics/R10: 82.09255533199195
 MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.164989939637827
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.1282980727618
 MSRVTT_full_val/v2t_metrics/R1: 35.61368209255533
 MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
 MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.889336016096579
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.45457789839057
 MSRVTT_full_test/t2v_metrics/R1: 13.511705685618729
 MSRVTT_full_test/t2v_metrics/R5: 35.25083612040134
 MSRVTT_full_test/t2v_metrics/R10: 49.69899665551839
 MSRVTT_full_test/t2v_metrics/R50: 80.43478260869566
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 46.092976588628765
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.712812844534504
 MSRVTT_full_test/v2t_metrics/R1: 13.91304347826087
 MSRVTT_full_test/v2t_metrics/R5: 39.096989966555185
 MSRVTT_full_test/v2t_metrics/R10: 53.54515050167224
 MSRVTT_full_test/v2t_metrics/R50: 84.84949832775919
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 37.42541806020067
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.767712084667842
 mnt_best       : 28.712812844534504
 not_improved_count: 0
Train Epoch: 33 [1/250 128/32000 (0%)] Loss: 1.51455 (QuantReg: 9.78585) QuantErr: 9.78585 batch_time=31.99565 
Train Epoch: 33 [12/250 1536/32000 (5%)] Loss: 1.48214 (QuantReg: 9.47606) QuantErr: 9.47606 batch_time=0.50003 
Train Epoch: 33 [23/250 2944/32000 (9%)] Loss: 1.68499 (QuantReg: 9.63124) QuantErr: 9.63124 batch_time=0.51558 
Train Epoch: 33 [34/250 4352/32000 (14%)] Loss: 1.30981 (QuantReg: 9.04922) QuantErr: 9.04922 batch_time=0.53564 
Train Epoch: 33 [45/250 5760/32000 (18%)] Loss: 1.47485 (QuantReg: 9.66112) QuantErr: 9.66112 batch_time=0.52964 
Train Epoch: 33 [56/250 7168/32000 (22%)] Loss: 1.58166 (QuantReg: 9.49010) QuantErr: 9.49010 batch_time=0.50028 
Train Epoch: 33 [67/250 8576/32000 (27%)] Loss: 1.77912 (QuantReg: 9.47044) QuantErr: 9.47044 batch_time=0.50742 
Train Epoch: 33 [78/250 9984/32000 (31%)] Loss: 1.47089 (QuantReg: 9.24075) QuantErr: 9.24075 batch_time=0.73094 
Train Epoch: 33 [89/250 11392/32000 (36%)] Loss: 1.64947 (QuantReg: 9.29901) QuantErr: 9.29901 batch_time=0.50161 
Train Epoch: 33 [100/250 12800/32000 (40%)] Loss: 1.59674 (QuantReg: 9.50126) QuantErr: 9.50126 batch_time=0.49619 
Train Epoch: 33 [111/250 14208/32000 (44%)] Loss: 1.73307 (QuantReg: 9.45003) QuantErr: 9.45003 batch_time=0.49683 
Train Epoch: 33 [122/250 15616/32000 (49%)] Loss: 1.46995 (QuantReg: 9.48441) QuantErr: 9.48441 batch_time=0.63180 
Train Epoch: 33 [133/250 17024/32000 (53%)] Loss: 1.90495 (QuantReg: 9.42791) QuantErr: 9.42791 batch_time=1.23506 
Train Epoch: 33 [144/250 18432/32000 (58%)] Loss: 1.56353 (QuantReg: 9.55567) QuantErr: 9.55567 batch_time=0.49099 
Train Epoch: 33 [155/250 19840/32000 (62%)] Loss: 1.51685 (QuantReg: 9.16382) QuantErr: 9.16382 batch_time=0.49348 
Train Epoch: 33 [166/250 21248/32000 (66%)] Loss: 1.63900 (QuantReg: 9.61014) QuantErr: 9.61014 batch_time=0.49887 
Train Epoch: 33 [177/250 22656/32000 (71%)] Loss: 1.42625 (QuantReg: 9.46447) QuantErr: 9.46447 batch_time=0.49352 
Train Epoch: 33 [188/250 24064/32000 (75%)] Loss: 1.61683 (QuantReg: 9.51720) QuantErr: 9.51720 batch_time=0.50515 
Train Epoch: 33 [199/250 25472/32000 (80%)] Loss: 1.53126 (QuantReg: 9.44112) QuantErr: 9.44112 batch_time=0.49722 
Train Epoch: 33 [210/250 26880/32000 (84%)] Loss: 1.46452 (QuantReg: 9.77122) QuantErr: 9.77122 batch_time=0.49014 
Train Epoch: 33 [221/250 28288/32000 (88%)] Loss: 1.75203 (QuantReg: 9.39156) QuantErr: 9.39156 batch_time=0.55061 
Train Epoch: 33 [232/250 29696/32000 (93%)] Loss: 1.43105 (QuantReg: 9.45205) QuantErr: 9.45205 batch_time=0.51236 
Train Epoch: 33 [243/250 31104/32000 (97%)] Loss: 1.55423 (QuantReg: 9.40999) QuantErr: 9.40999 batch_time=0.49615 
Train Epoch: 33 codebook_update_time=1.62442
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch33.pth ...
Done in 3.971s
removing stale ckpt [epoch 32] [took 0.00s]
 epoch          : 33
 loss           : 1.523693199157715
 quant_reg      : 9.462786594390868
 quant_err      : 9.462786594390868
 learning_rate  : 9.685574222925043e-06
 n_samples      : 1056000
 n_steps        : 8250
 MSRVTT_full_val/t2v_metrics/R1: 33.601609657947684
 MSRVTT_full_val/t2v_metrics/R5: 68.2092555331992
 MSRVTT_full_val/t2v_metrics/R10: 82.69617706237425
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 7.927565392354125
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.44200736678198
 MSRVTT_full_val/v2t_metrics/R1: 36.01609657947686
 MSRVTT_full_val/v2t_metrics/R5: 72.83702213279678
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.26559356136821
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.52208882045913
 MSRVTT_full_test/t2v_metrics/R1: 12.809364548494983
 MSRVTT_full_test/t2v_metrics/R5: 35.28428093645485
 MSRVTT_full_test/t2v_metrics/R10: 49.331103678929765
 MSRVTT_full_test/t2v_metrics/R50: 80.16722408026756
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.25451505016722
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.145560146677635
 MSRVTT_full_test/v2t_metrics/R1: 14.013377926421406
 MSRVTT_full_test/v2t_metrics/R5: 38.2943143812709
 MSRVTT_full_test/v2t_metrics/R10: 52.74247491638796
 MSRVTT_full_test/v2t_metrics/R50: 83.87959866220736
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.71036789297659
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.47515192416164
 mnt_best       : 28.712812844534504
 not_improved_count: 1
Train Epoch: 34 [1/250 128/32000 (0%)] Loss: 1.57747 (QuantReg: 9.54038) QuantErr: 9.54038 batch_time=31.51871 
Train Epoch: 34 [12/250 1536/32000 (5%)] Loss: 1.57496 (QuantReg: 9.72984) QuantErr: 9.72984 batch_time=0.50808 
Train Epoch: 34 [23/250 2944/32000 (9%)] Loss: 1.47244 (QuantReg: 9.49988) QuantErr: 9.49988 batch_time=0.56932 
Train Epoch: 34 [34/250 4352/32000 (14%)] Loss: 1.65624 (QuantReg: 9.50012) QuantErr: 9.50012 batch_time=0.49792 
Train Epoch: 34 [45/250 5760/32000 (18%)] Loss: 1.68199 (QuantReg: 9.64612) QuantErr: 9.64612 batch_time=0.51481 
Train Epoch: 34 [56/250 7168/32000 (22%)] Loss: 1.68281 (QuantReg: 9.49935) QuantErr: 9.49935 batch_time=0.66205 
Train Epoch: 34 [67/250 8576/32000 (27%)] Loss: 1.48081 (QuantReg: 9.52109) QuantErr: 9.52109 batch_time=0.50932 
Train Epoch: 34 [78/250 9984/32000 (31%)] Loss: 1.82180 (QuantReg: 9.34176) QuantErr: 9.34176 batch_time=0.50252 
Train Epoch: 34 [89/250 11392/32000 (36%)] Loss: 1.70996 (QuantReg: 9.42708) QuantErr: 9.42708 batch_time=0.49943 
Train Epoch: 34 [100/250 12800/32000 (40%)] Loss: 1.80061 (QuantReg: 9.41537) QuantErr: 9.41537 batch_time=0.52149 
Train Epoch: 34 [111/250 14208/32000 (44%)] Loss: 1.57444 (QuantReg: 9.23439) QuantErr: 9.23439 batch_time=0.50643 
Train Epoch: 34 [122/250 15616/32000 (49%)] Loss: 1.45817 (QuantReg: 9.41690) QuantErr: 9.41690 batch_time=0.51238 
Train Epoch: 34 [133/250 17024/32000 (53%)] Loss: 1.60588 (QuantReg: 9.52721) QuantErr: 9.52721 batch_time=0.51240 
Train Epoch: 34 [144/250 18432/32000 (58%)] Loss: 1.60646 (QuantReg: 9.32806) QuantErr: 9.32806 batch_time=3.18948 
Train Epoch: 34 [155/250 19840/32000 (62%)] Loss: 1.61771 (QuantReg: 9.91513) QuantErr: 9.91513 batch_time=0.52393 
Train Epoch: 34 [166/250 21248/32000 (66%)] Loss: 1.71822 (QuantReg: 9.29160) QuantErr: 9.29160 batch_time=0.51196 
Train Epoch: 34 [177/250 22656/32000 (71%)] Loss: 1.37493 (QuantReg: 9.44621) QuantErr: 9.44621 batch_time=0.49929 
Train Epoch: 34 [188/250 24064/32000 (75%)] Loss: 1.71287 (QuantReg: 9.22710) QuantErr: 9.22710 batch_time=0.73465 
Train Epoch: 34 [199/250 25472/32000 (80%)] Loss: 1.38114 (QuantReg: 9.53947) QuantErr: 9.53947 batch_time=1.10167 
Train Epoch: 34 [210/250 26880/32000 (84%)] Loss: 1.56227 (QuantReg: 9.32870) QuantErr: 9.32870 batch_time=0.50954 
Train Epoch: 34 [221/250 28288/32000 (88%)] Loss: 1.40542 (QuantReg: 9.60448) QuantErr: 9.60448 batch_time=0.49841 
Train Epoch: 34 [232/250 29696/32000 (93%)] Loss: 1.48025 (QuantReg: 9.55309) QuantErr: 9.55309 batch_time=0.51131 
Train Epoch: 34 [243/250 31104/32000 (97%)] Loss: 1.76548 (QuantReg: 9.61094) QuantErr: 9.61094 batch_time=0.52003 
Train Epoch: 34 codebook_update_time=1.69951
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch34.pth ...
Done in 4.067s
removing stale ckpt [epoch 33] [took 0.00s]
 epoch          : 34
 loss           : 1.5546002106666565
 quant_reg      : 9.46555680847168
 quant_err      : 9.46555680847168
 learning_rate  : 9.20129551177879e-06
 n_samples      : 1088000
 n_steps        : 8500
 MSRVTT_full_val/t2v_metrics/R1: 32.394366197183096
 MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
 MSRVTT_full_val/t2v_metrics/R10: 81.69014084507042
 MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 7.915492957746479
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.179214032555116
 MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
 MSRVTT_full_val/v2t_metrics/R5: 71.42857142857143
 MSRVTT_full_val/v2t_metrics/R10: 83.90342052313883
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.158953722334004
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.42212291763899
 MSRVTT_full_test/t2v_metrics/R1: 12.341137123745819
 MSRVTT_full_test/t2v_metrics/R5: 34.280936454849495
 MSRVTT_full_test/t2v_metrics/R10: 47.491638795986624
 MSRVTT_full_test/t2v_metrics/R50: 79.23076923076923
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.92642140468227
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.18577200356663
 MSRVTT_full_test/v2t_metrics/R1: 13.444816053511706
 MSRVTT_full_test/v2t_metrics/R5: 38.02675585284281
 MSRVTT_full_test/v2t_metrics/R10: 53.27759197324415
 MSRVTT_full_test/v2t_metrics/R50: 83.54515050167224
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 40.40351170568562
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.088202468706182
 mnt_best       : 28.712812844534504
 not_improved_count: 2
Train Epoch: 35 [1/250 128/32000 (0%)] Loss: 1.48954 (QuantReg: 9.60636) QuantErr: 9.60636 batch_time=37.50462 
Train Epoch: 35 [12/250 1536/32000 (5%)] Loss: 1.69671 (QuantReg: 9.54128) QuantErr: 9.54128 batch_time=0.50484 
Train Epoch: 35 [23/250 2944/32000 (9%)] Loss: 1.55554 (QuantReg: 9.54733) QuantErr: 9.54733 batch_time=0.74237 
Train Epoch: 35 [34/250 4352/32000 (14%)] Loss: 1.73405 (QuantReg: 9.47464) QuantErr: 9.47464 batch_time=0.48135 
Train Epoch: 35 [45/250 5760/32000 (18%)] Loss: 1.54911 (QuantReg: 9.44086) QuantErr: 9.44086 batch_time=1.70940 
Train Epoch: 35 [56/250 7168/32000 (22%)] Loss: 1.43835 (QuantReg: 9.38467) QuantErr: 9.38467 batch_time=0.50600 
Train Epoch: 35 [67/250 8576/32000 (27%)] Loss: 1.44668 (QuantReg: 9.69128) QuantErr: 9.69128 batch_time=0.99557 
Train Epoch: 35 [78/250 9984/32000 (31%)] Loss: 1.91112 (QuantReg: 9.73558) QuantErr: 9.73558 batch_time=0.50363 
Train Epoch: 35 [89/250 11392/32000 (36%)] Loss: 1.49617 (QuantReg: 9.31176) QuantErr: 9.31176 batch_time=0.49667 
Train Epoch: 35 [100/250 12800/32000 (40%)] Loss: 1.70132 (QuantReg: 9.49545) QuantErr: 9.49545 batch_time=0.56250 
Train Epoch: 35 [111/250 14208/32000 (44%)] Loss: 1.51015 (QuantReg: 9.31157) QuantErr: 9.31157 batch_time=0.51056 
Train Epoch: 35 [122/250 15616/32000 (49%)] Loss: 1.49379 (QuantReg: 9.44508) QuantErr: 9.44508 batch_time=0.49159 
Train Epoch: 35 [133/250 17024/32000 (53%)] Loss: 1.82037 (QuantReg: 9.31278) QuantErr: 9.31278 batch_time=0.50600 
Train Epoch: 35 [144/250 18432/32000 (58%)] Loss: 1.45960 (QuantReg: 9.71564) QuantErr: 9.71564 batch_time=1.45242 
Train Epoch: 35 [155/250 19840/32000 (62%)] Loss: 1.41158 (QuantReg: 9.28474) QuantErr: 9.28474 batch_time=0.50766 
Train Epoch: 35 [166/250 21248/32000 (66%)] Loss: 1.70089 (QuantReg: 9.65021) QuantErr: 9.65021 batch_time=0.48532 
Train Epoch: 35 [177/250 22656/32000 (71%)] Loss: 1.56595 (QuantReg: 10.03207) QuantErr: 10.03207 batch_time=0.49806 
Train Epoch: 35 [188/250 24064/32000 (75%)] Loss: 1.40062 (QuantReg: 9.28658) QuantErr: 9.28658 batch_time=0.48886 
Train Epoch: 35 [199/250 25472/32000 (80%)] Loss: 1.46927 (QuantReg: 9.79768) QuantErr: 9.79768 batch_time=0.48898 
Train Epoch: 35 [210/250 26880/32000 (84%)] Loss: 1.55181 (QuantReg: 9.56108) QuantErr: 9.56108 batch_time=0.48879 
Train Epoch: 35 [221/250 28288/32000 (88%)] Loss: 1.54037 (QuantReg: 9.63134) QuantErr: 9.63134 batch_time=0.49378 
Train Epoch: 35 [232/250 29696/32000 (93%)] Loss: 1.54398 (QuantReg: 9.72152) QuantErr: 9.72152 batch_time=0.49235 
Train Epoch: 35 [243/250 31104/32000 (97%)] Loss: 1.35685 (QuantReg: 9.59343) QuantErr: 9.59343 batch_time=0.48987 
Train Epoch: 35 codebook_update_time=1.73571
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch35.pth ...
Done in 4.007s
removing stale ckpt [epoch 34] [took 0.00s]
 epoch          : 35
 loss           : 1.5211439251899719
 quant_reg      : 9.480939052581787
 quant_err      : 9.480939052581787
 learning_rate  : 8.74123073618985e-06
 n_samples      : 1120000
 n_steps        : 8750
 MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
 MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
 MSRVTT_full_val/t2v_metrics/R10: 80.6841046277666
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.388329979879277
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.7707218724692
 MSRVTT_full_val/v2t_metrics/R1: 36.61971830985915
 MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
 MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.181086519114688
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.55999632455685
 MSRVTT_full_test/t2v_metrics/R1: 12.876254180602007
 MSRVTT_full_test/t2v_metrics/R5: 34.414715719063544
 MSRVTT_full_test/t2v_metrics/R10: 48.26086956521739
 MSRVTT_full_test/t2v_metrics/R50: 79.36454849498328
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.711371237458195
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.75724061539573
 MSRVTT_full_test/v2t_metrics/R1: 14.11371237458194
 MSRVTT_full_test/v2t_metrics/R5: 38.99665551839465
 MSRVTT_full_test/v2t_metrics/R10: 52.97658862876254
 MSRVTT_full_test/v2t_metrics/R50: 84.18060200668896
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.5994983277592
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.778742425671634
 mnt_best       : 28.712812844534504
 not_improved_count: 3
Train Epoch: 36 [1/250 128/32000 (0%)] Loss: 1.40498 (QuantReg: 9.28892) QuantErr: 9.28892 batch_time=30.26924 
Train Epoch: 36 [12/250 1536/32000 (5%)] Loss: 1.66437 (QuantReg: 9.59803) QuantErr: 9.59803 batch_time=0.48440 
Train Epoch: 36 [23/250 2944/32000 (9%)] Loss: 1.23444 (QuantReg: 9.14650) QuantErr: 9.14650 batch_time=0.50401 
Train Epoch: 36 [34/250 4352/32000 (14%)] Loss: 1.45496 (QuantReg: 9.32443) QuantErr: 9.32443 batch_time=0.53516 
Train Epoch: 36 [45/250 5760/32000 (18%)] Loss: 1.55077 (QuantReg: 9.38801) QuantErr: 9.38801 batch_time=0.73935 
Train Epoch: 36 [56/250 7168/32000 (22%)] Loss: 1.24664 (QuantReg: 9.21995) QuantErr: 9.21995 batch_time=0.50651 
Train Epoch: 36 [67/250 8576/32000 (27%)] Loss: 1.45380 (QuantReg: 9.39791) QuantErr: 9.39791 batch_time=0.53161 
Train Epoch: 36 [78/250 9984/32000 (31%)] Loss: 1.69246 (QuantReg: 9.44276) QuantErr: 9.44276 batch_time=0.49909 
Train Epoch: 36 [89/250 11392/32000 (36%)] Loss: 1.64162 (QuantReg: 9.17035) QuantErr: 9.17035 batch_time=0.51061 
Train Epoch: 36 [100/250 12800/32000 (40%)] Loss: 1.54627 (QuantReg: 9.42693) QuantErr: 9.42693 batch_time=0.49383 
Train Epoch: 36 [111/250 14208/32000 (44%)] Loss: 1.80806 (QuantReg: 9.43518) QuantErr: 9.43518 batch_time=0.50314 
Train Epoch: 36 [122/250 15616/32000 (49%)] Loss: 1.72822 (QuantReg: 9.33965) QuantErr: 9.33965 batch_time=0.50031 
Train Epoch: 36 [133/250 17024/32000 (53%)] Loss: 1.44962 (QuantReg: 9.19234) QuantErr: 9.19234 batch_time=1.32518 
Train Epoch: 36 [144/250 18432/32000 (58%)] Loss: 1.65499 (QuantReg: 9.74937) QuantErr: 9.74937 batch_time=0.51221 
Train Epoch: 36 [155/250 19840/32000 (62%)] Loss: 1.32954 (QuantReg: 9.36991) QuantErr: 9.36991 batch_time=0.50366 
Train Epoch: 36 [166/250 21248/32000 (66%)] Loss: 1.68573 (QuantReg: 9.60999) QuantErr: 9.60999 batch_time=0.49825 
Train Epoch: 36 [177/250 22656/32000 (71%)] Loss: 1.68710 (QuantReg: 9.46933) QuantErr: 9.46933 batch_time=0.50365 
Train Epoch: 36 [188/250 24064/32000 (75%)] Loss: 1.33213 (QuantReg: 9.53777) QuantErr: 9.53777 batch_time=0.50507 
Train Epoch: 36 [199/250 25472/32000 (80%)] Loss: 1.41627 (QuantReg: 9.36195) QuantErr: 9.36195 batch_time=0.50411 
Train Epoch: 36 [210/250 26880/32000 (84%)] Loss: 1.61173 (QuantReg: 9.50714) QuantErr: 9.50714 batch_time=0.49538 
Train Epoch: 36 [221/250 28288/32000 (88%)] Loss: 1.64896 (QuantReg: 9.15265) QuantErr: 9.15265 batch_time=0.52044 
Train Epoch: 36 [232/250 29696/32000 (93%)] Loss: 1.57826 (QuantReg: 9.44562) QuantErr: 9.44562 batch_time=0.51339 
Train Epoch: 36 [243/250 31104/32000 (97%)] Loss: 1.32438 (QuantReg: 9.13254) QuantErr: 9.13254 batch_time=0.61717 
Train Epoch: 36 codebook_update_time=1.65052
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch36.pth ...
Done in 12.444s
removing stale ckpt [epoch 35] [took 0.00s]
 epoch          : 36
 loss           : 1.528282608985901
 quant_reg      : 9.480553466796875
 quant_err      : 9.480553466796875
 learning_rate  : 8.304169199380357e-06
 n_samples      : 1152000
 n_steps        : 9000
 MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
 MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
 MSRVTT_full_val/t2v_metrics/R10: 79.47686116700201
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.273641851106639
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.78221122234577
 MSRVTT_full_val/v2t_metrics/R1: 36.61971830985915
 MSRVTT_full_val/v2t_metrics/R5: 72.83702213279678
 MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.072434607645875
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.61586351151941
 MSRVTT_full_test/t2v_metrics/R1: 13.244147157190636
 MSRVTT_full_test/t2v_metrics/R5: 35.28428093645485
 MSRVTT_full_test/t2v_metrics/R10: 49.130434782608695
 MSRVTT_full_test/t2v_metrics/R50: 79.93311036789298
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.59799331103679
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.42182485128718
 MSRVTT_full_test/v2t_metrics/R1: 14.013377926421406
 MSRVTT_full_test/v2t_metrics/R5: 38.02675585284281
 MSRVTT_full_test/v2t_metrics/R10: 53.1438127090301
 MSRVTT_full_test/v2t_metrics/R50: 84.24749163879599
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.92909698996655
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.480934191407442
 mnt_best       : 28.712812844534504
 not_improved_count: 4
Train Epoch: 37 [1/250 128/32000 (0%)] Loss: 1.57419 (QuantReg: 9.40212) QuantErr: 9.40212 batch_time=32.20789 
Train Epoch: 37 [12/250 1536/32000 (5%)] Loss: 1.37674 (QuantReg: 9.43255) QuantErr: 9.43255 batch_time=0.48509 
Train Epoch: 37 [23/250 2944/32000 (9%)] Loss: 1.54067 (QuantReg: 9.27275) QuantErr: 9.27275 batch_time=1.23153 
Train Epoch: 37 [34/250 4352/32000 (14%)] Loss: 1.42006 (QuantReg: 9.46906) QuantErr: 9.46906 batch_time=0.50803 
Train Epoch: 37 [45/250 5760/32000 (18%)] Loss: 1.62836 (QuantReg: 9.57577) QuantErr: 9.57577 batch_time=0.50265 
Train Epoch: 37 [56/250 7168/32000 (22%)] Loss: 1.78557 (QuantReg: 9.27277) QuantErr: 9.27277 batch_time=0.50793 
Train Epoch: 37 [67/250 8576/32000 (27%)] Loss: 1.76450 (QuantReg: 9.47644) QuantErr: 9.47644 batch_time=0.50248 
Train Epoch: 37 [78/250 9984/32000 (31%)] Loss: 1.50249 (QuantReg: 9.50041) QuantErr: 9.50041 batch_time=0.50484 
Train Epoch: 37 [89/250 11392/32000 (36%)] Loss: 1.63172 (QuantReg: 9.58359) QuantErr: 9.58359 batch_time=0.63886 
Train Epoch: 37 [100/250 12800/32000 (40%)] Loss: 1.41148 (QuantReg: 9.33031) QuantErr: 9.33031 batch_time=0.60117 
Train Epoch: 37 [111/250 14208/32000 (44%)] Loss: 1.45161 (QuantReg: 9.42001) QuantErr: 9.42001 batch_time=0.48758 
Train Epoch: 37 [122/250 15616/32000 (49%)] Loss: 1.39243 (QuantReg: 9.66461) QuantErr: 9.66461 batch_time=0.48688 
Train Epoch: 37 [133/250 17024/32000 (53%)] Loss: 1.53972 (QuantReg: 9.21142) QuantErr: 9.21142 batch_time=0.74790 
Train Epoch: 37 [144/250 18432/32000 (58%)] Loss: 1.71253 (QuantReg: 9.42285) QuantErr: 9.42285 batch_time=0.52614 
Train Epoch: 37 [155/250 19840/32000 (62%)] Loss: 1.41476 (QuantReg: 9.11718) QuantErr: 9.11718 batch_time=0.49721 
Train Epoch: 37 [166/250 21248/32000 (66%)] Loss: 1.68087 (QuantReg: 9.81830) QuantErr: 9.81830 batch_time=0.54141 
Train Epoch: 37 [177/250 22656/32000 (71%)] Loss: 1.53343 (QuantReg: 9.53336) QuantErr: 9.53336 batch_time=0.50918 
Train Epoch: 37 [188/250 24064/32000 (75%)] Loss: 1.55213 (QuantReg: 9.39351) QuantErr: 9.39351 batch_time=0.50385 
Train Epoch: 37 [199/250 25472/32000 (80%)] Loss: 1.57547 (QuantReg: 9.43193) QuantErr: 9.43193 batch_time=0.53188 
Train Epoch: 37 [210/250 26880/32000 (84%)] Loss: 1.63864 (QuantReg: 9.49578) QuantErr: 9.49578 batch_time=0.53309 
Train Epoch: 37 [221/250 28288/32000 (88%)] Loss: 1.62602 (QuantReg: 9.91539) QuantErr: 9.91539 batch_time=0.51930 
Train Epoch: 37 [232/250 29696/32000 (93%)] Loss: 1.76510 (QuantReg: 9.18492) QuantErr: 9.18492 batch_time=1.24227 
Train Epoch: 37 [243/250 31104/32000 (97%)] Loss: 1.34571 (QuantReg: 9.39617) QuantErr: 9.39617 batch_time=0.49427 
Train Epoch: 37 codebook_update_time=1.65168
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch37.pth ...
Done in 11.086s
removing stale ckpt [epoch 36] [took 0.01s]
 epoch          : 37
 loss           : 1.5271483488082886
 quant_reg      : 9.435237857818603
 quant_err      : 9.435237857818603
 learning_rate  : 7.888960739411339e-06
 n_samples      : 1184000
 n_steps        : 9250
 MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
 MSRVTT_full_val/t2v_metrics/R5: 68.61167002012073
 MSRVTT_full_val/t2v_metrics/R10: 82.29376257545272
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 7.949698189134809
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.5286414715235
 MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
 MSRVTT_full_val/v2t_metrics/R5: 73.64185110663983
 MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
 MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.957746478873239
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.534468614267396
 MSRVTT_full_test/t2v_metrics/R1: 12.909698996655518
 MSRVTT_full_test/t2v_metrics/R5: 35.719063545150505
 MSRVTT_full_test/t2v_metrics/R10: 50.03344481605351
 MSRVTT_full_test/t2v_metrics/R50: 80.2675585284281
 MSRVTT_full_test/t2v_metrics/MedR: 10.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.40351170568562
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.468124861096403
 MSRVTT_full_test/v2t_metrics/R1: 13.946488294314381
 MSRVTT_full_test/v2t_metrics/R5: 38.561872909699
 MSRVTT_full_test/v2t_metrics/R10: 53.94648829431438
 MSRVTT_full_test/v2t_metrics/R50: 84.38127090301003
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.26220735785953
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.72760572001128
 mnt_best       : 28.712812844534504
 not_improved_count: 5
Train Epoch: 38 [1/250 128/32000 (0%)] Loss: 1.63231 (QuantReg: 9.31833) QuantErr: 9.31833 batch_time=31.49777 
Train Epoch: 38 [12/250 1536/32000 (5%)] Loss: 1.58694 (QuantReg: 9.54103) QuantErr: 9.54103 batch_time=0.49096 
Train Epoch: 38 [23/250 2944/32000 (9%)] Loss: 1.25418 (QuantReg: 9.64517) QuantErr: 9.64517 batch_time=0.51558 
Train Epoch: 38 [34/250 4352/32000 (14%)] Loss: 1.56558 (QuantReg: 9.42299) QuantErr: 9.42299 batch_time=0.50329 
Train Epoch: 38 [45/250 5760/32000 (18%)] Loss: 1.54373 (QuantReg: 9.41807) QuantErr: 9.41807 batch_time=0.52205 
Train Epoch: 38 [56/250 7168/32000 (22%)] Loss: 1.68972 (QuantReg: 9.53505) QuantErr: 9.53505 batch_time=0.51354 
Train Epoch: 38 [67/250 8576/32000 (27%)] Loss: 1.43663 (QuantReg: 9.22910) QuantErr: 9.22910 batch_time=0.60226 
Train Epoch: 38 [78/250 9984/32000 (31%)] Loss: 1.37565 (QuantReg: 9.56456) QuantErr: 9.56456 batch_time=0.89561 
Train Epoch: 38 [89/250 11392/32000 (36%)] Loss: 1.29720 (QuantReg: 9.35437) QuantErr: 9.35437 batch_time=0.48054 
Train Epoch: 38 [100/250 12800/32000 (40%)] Loss: 1.44884 (QuantReg: 9.77022) QuantErr: 9.77022 batch_time=1.38627 
Train Epoch: 38 [111/250 14208/32000 (44%)] Loss: 1.55022 (QuantReg: 9.33897) QuantErr: 9.33897 batch_time=0.49939 
Train Epoch: 38 [122/250 15616/32000 (49%)] Loss: 1.64242 (QuantReg: 9.41420) QuantErr: 9.41420 batch_time=0.51365 
Train Epoch: 38 [133/250 17024/32000 (53%)] Loss: 1.49632 (QuantReg: 9.47045) QuantErr: 9.47045 batch_time=0.60625 
Train Epoch: 38 [144/250 18432/32000 (58%)] Loss: 1.54156 (QuantReg: 9.38428) QuantErr: 9.38428 batch_time=2.58821 
Train Epoch: 38 [155/250 19840/32000 (62%)] Loss: 1.38251 (QuantReg: 9.53942) QuantErr: 9.53942 batch_time=0.49493 
Train Epoch: 38 [166/250 21248/32000 (66%)] Loss: 1.54082 (QuantReg: 9.34563) QuantErr: 9.34563 batch_time=0.51490 
Train Epoch: 38 [177/250 22656/32000 (71%)] Loss: 1.56966 (QuantReg: 9.51385) QuantErr: 9.51385 batch_time=0.51112 
Train Epoch: 38 [188/250 24064/32000 (75%)] Loss: 1.29535 (QuantReg: 9.37306) QuantErr: 9.37306 batch_time=0.59735 
Train Epoch: 38 [199/250 25472/32000 (80%)] Loss: 1.43016 (QuantReg: 9.41294) QuantErr: 9.41294 batch_time=0.50753 
Train Epoch: 38 [210/250 26880/32000 (84%)] Loss: 1.43485 (QuantReg: 9.46555) QuantErr: 9.46555 batch_time=0.55055 
Train Epoch: 38 [221/250 28288/32000 (88%)] Loss: 1.55784 (QuantReg: 9.49799) QuantErr: 9.49799 batch_time=0.51017 
Train Epoch: 38 [232/250 29696/32000 (93%)] Loss: 1.51096 (QuantReg: 9.29013) QuantErr: 9.29013 batch_time=0.48949 
Train Epoch: 38 [243/250 31104/32000 (97%)] Loss: 1.48379 (QuantReg: 9.46203) QuantErr: 9.46203 batch_time=0.49192 
Train Epoch: 38 codebook_update_time=1.65705
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch38.pth ...
Done in 5.787s
removing stale ckpt [epoch 37] [took 0.01s]
 epoch          : 38
 loss           : 1.5208532757759095
 quant_reg      : 9.458342937469482
 quant_err      : 9.458342937469482
 learning_rate  : 7.494512702440772e-06
 n_samples      : 1216000
 n_steps        : 9500
 MSRVTT_full_val/t2v_metrics/R1: 32.19315895372233
 MSRVTT_full_val/t2v_metrics/R5: 67.6056338028169
 MSRVTT_full_val/t2v_metrics/R10: 81.89134808853119
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.323943661971832
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.27664100171816
 MSRVTT_full_val/v2t_metrics/R1: 37.223340040241446
 MSRVTT_full_val/v2t_metrics/R5: 73.8430583501006
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.374245472837022
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.471334145587505
 MSRVTT_full_test/t2v_metrics/R1: 13.177257525083611
 MSRVTT_full_test/t2v_metrics/R5: 34.78260869565217
 MSRVTT_full_test/t2v_metrics/R10: 49.03010033444816
 MSRVTT_full_test/t2v_metrics/R50: 79.96655518394648
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.29665551839465
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.219543812487316
 MSRVTT_full_test/v2t_metrics/R1: 13.612040133779264
 MSRVTT_full_test/v2t_metrics/R5: 38.42809364548495
 MSRVTT_full_test/v2t_metrics/R10: 53.779264214046826
 MSRVTT_full_test/v2t_metrics/R50: 84.44816053511705
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.09849498327759
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.41321315625841
 mnt_best       : 28.712812844534504
 not_improved_count: 6
Train Epoch: 39 [1/250 128/32000 (0%)] Loss: 1.60703 (QuantReg: 9.51729) QuantErr: 9.51729 batch_time=34.87449 
Train Epoch: 39 [12/250 1536/32000 (5%)] Loss: 1.63557 (QuantReg: 9.29070) QuantErr: 9.29070 batch_time=0.54698 
Train Epoch: 39 [23/250 2944/32000 (9%)] Loss: 1.30303 (QuantReg: 9.37673) QuantErr: 9.37673 batch_time=0.49992 
Train Epoch: 39 [34/250 4352/32000 (14%)] Loss: 1.67183 (QuantReg: 9.51457) QuantErr: 9.51457 batch_time=0.49638 
Train Epoch: 39 [45/250 5760/32000 (18%)] Loss: 1.32864 (QuantReg: 9.28228) QuantErr: 9.28228 batch_time=0.62123 
Train Epoch: 39 [56/250 7168/32000 (22%)] Loss: 1.48871 (QuantReg: 9.67890) QuantErr: 9.67890 batch_time=0.50057 
Train Epoch: 39 [67/250 8576/32000 (27%)] Loss: 1.43981 (QuantReg: 9.27093) QuantErr: 9.27093 batch_time=0.48993 
Train Epoch: 39 [78/250 9984/32000 (31%)] Loss: 1.40670 (QuantReg: 9.61410) QuantErr: 9.61410 batch_time=0.92753 
Train Epoch: 39 [89/250 11392/32000 (36%)] Loss: 1.24664 (QuantReg: 9.23118) QuantErr: 9.23118 batch_time=0.49068 
Train Epoch: 39 [100/250 12800/32000 (40%)] Loss: 1.32731 (QuantReg: 9.55207) QuantErr: 9.55207 batch_time=0.49381 
Train Epoch: 39 [111/250 14208/32000 (44%)] Loss: 1.37627 (QuantReg: 9.23398) QuantErr: 9.23398 batch_time=0.49134 
Train Epoch: 39 [122/250 15616/32000 (49%)] Loss: 1.35753 (QuantReg: 9.60479) QuantErr: 9.60479 batch_time=0.50616 
Train Epoch: 39 [133/250 17024/32000 (53%)] Loss: 1.61647 (QuantReg: 9.48693) QuantErr: 9.48693 batch_time=0.74652 
Train Epoch: 39 [144/250 18432/32000 (58%)] Loss: 1.49417 (QuantReg: 9.30360) QuantErr: 9.30360 batch_time=1.24313 
Train Epoch: 39 [155/250 19840/32000 (62%)] Loss: 1.39671 (QuantReg: 9.25000) QuantErr: 9.25000 batch_time=0.52976 
Train Epoch: 39 [166/250 21248/32000 (66%)] Loss: 1.29623 (QuantReg: 9.53081) QuantErr: 9.53081 batch_time=0.49297 
Train Epoch: 39 [177/250 22656/32000 (71%)] Loss: 1.45841 (QuantReg: 9.52019) QuantErr: 9.52019 batch_time=0.51659 
Train Epoch: 39 [188/250 24064/32000 (75%)] Loss: 1.50321 (QuantReg: 9.45744) QuantErr: 9.45744 batch_time=0.50478 
Train Epoch: 39 [199/250 25472/32000 (80%)] Loss: 1.53671 (QuantReg: 9.39245) QuantErr: 9.39245 batch_time=0.54270 
Train Epoch: 39 [210/250 26880/32000 (84%)] Loss: 1.45145 (QuantReg: 9.77350) QuantErr: 9.77350 batch_time=0.51547 
Train Epoch: 39 [221/250 28288/32000 (88%)] Loss: 1.45136 (QuantReg: 9.34759) QuantErr: 9.34759 batch_time=0.50249 
Train Epoch: 39 [232/250 29696/32000 (93%)] Loss: 1.43317 (QuantReg: 9.72595) QuantErr: 9.72595 batch_time=0.50609 
Train Epoch: 39 [243/250 31104/32000 (97%)] Loss: 1.36207 (QuantReg: 9.29631) QuantErr: 9.29631 batch_time=0.54929 
Train Epoch: 39 codebook_update_time=1.62315
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch39.pth ...
Done in 4.093s
removing stale ckpt [epoch 38] [took 0.00s]
 epoch          : 39
 loss           : 1.5035850381851197
 quant_reg      : 9.476790237426759
 quant_err      : 9.476790237426759
 learning_rate  : 7.119787067318733e-06
 n_samples      : 1248000
 n_steps        : 9750
 MSRVTT_full_val/t2v_metrics/R1: 32.796780684104625
 MSRVTT_full_val/t2v_metrics/R5: 65.59356136820925
 MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
 MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.585513078470825
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.64246613011147
 MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
 MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
 MSRVTT_full_val/v2t_metrics/R10: 83.90342052313883
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.458752515090543
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.92009688467775
 MSRVTT_full_test/t2v_metrics/R1: 12.54180602006689
 MSRVTT_full_test/t2v_metrics/R5: 34.81605351170568
 MSRVTT_full_test/t2v_metrics/R10: 47.89297658862876
 MSRVTT_full_test/t2v_metrics/R50: 79.29765886287625
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 51.68026755852843
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.550986253648222
 MSRVTT_full_test/v2t_metrics/R1: 13.244147157190636
 MSRVTT_full_test/v2t_metrics/R5: 37.92642140468227
 MSRVTT_full_test/v2t_metrics/R10: 52.74247491638796
 MSRVTT_full_test/v2t_metrics/R50: 83.87959866220736
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 40.56053511705686
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.810925299324968
 mnt_best       : 28.712812844534504
 not_improved_count: 7
Train Epoch: 40 [1/250 128/32000 (0%)] Loss: 1.56714 (QuantReg: 9.61850) QuantErr: 9.61850 batch_time=33.85530 
Train Epoch: 40 [12/250 1536/32000 (5%)] Loss: 1.55380 (QuantReg: 9.52097) QuantErr: 9.52097 batch_time=0.54206 
Train Epoch: 40 [23/250 2944/32000 (9%)] Loss: 1.54584 (QuantReg: 9.37131) QuantErr: 9.37131 batch_time=0.50716 
Train Epoch: 40 [34/250 4352/32000 (14%)] Loss: 1.39287 (QuantReg: 9.19448) QuantErr: 9.19448 batch_time=0.51812 
Train Epoch: 40 [45/250 5760/32000 (18%)] Loss: 1.70897 (QuantReg: 9.41411) QuantErr: 9.41411 batch_time=0.52493 
Train Epoch: 40 [56/250 7168/32000 (22%)] Loss: 1.72168 (QuantReg: 9.50319) QuantErr: 9.50319 batch_time=0.53677 
Train Epoch: 40 [67/250 8576/32000 (27%)] Loss: 1.53107 (QuantReg: 9.55077) QuantErr: 9.55077 batch_time=0.54060 
Train Epoch: 40 [78/250 9984/32000 (31%)] Loss: 1.53181 (QuantReg: 9.66083) QuantErr: 9.66083 batch_time=0.49213 
Train Epoch: 40 [89/250 11392/32000 (36%)] Loss: 1.73235 (QuantReg: 9.29358) QuantErr: 9.29358 batch_time=0.53677 
Train Epoch: 40 [100/250 12800/32000 (40%)] Loss: 1.32656 (QuantReg: 9.44202) QuantErr: 9.44202 batch_time=0.49660 
Train Epoch: 40 [111/250 14208/32000 (44%)] Loss: 1.48671 (QuantReg: 9.56827) QuantErr: 9.56827 batch_time=0.50360 
Train Epoch: 40 [122/250 15616/32000 (49%)] Loss: 1.37821 (QuantReg: 9.43344) QuantErr: 9.43344 batch_time=0.75835 
Train Epoch: 40 [133/250 17024/32000 (53%)] Loss: 1.51132 (QuantReg: 9.56477) QuantErr: 9.56477 batch_time=0.49801 
Train Epoch: 40 [144/250 18432/32000 (58%)] Loss: 1.36920 (QuantReg: 9.43087) QuantErr: 9.43087 batch_time=2.83300 
Train Epoch: 40 [155/250 19840/32000 (62%)] Loss: 1.45621 (QuantReg: 9.35017) QuantErr: 9.35017 batch_time=0.52198 
Train Epoch: 40 [166/250 21248/32000 (66%)] Loss: 1.53584 (QuantReg: 9.25504) QuantErr: 9.25504 batch_time=0.50381 
Train Epoch: 40 [177/250 22656/32000 (71%)] Loss: 1.62954 (QuantReg: 9.41303) QuantErr: 9.41303 batch_time=0.49473 
Train Epoch: 40 [188/250 24064/32000 (75%)] Loss: 1.75004 (QuantReg: 9.54137) QuantErr: 9.54137 batch_time=0.48788 
Train Epoch: 40 [199/250 25472/32000 (80%)] Loss: 1.61949 (QuantReg: 9.47357) QuantErr: 9.47357 batch_time=0.54466 
Train Epoch: 40 [210/250 26880/32000 (84%)] Loss: 1.48162 (QuantReg: 9.44203) QuantErr: 9.44203 batch_time=0.95120 
Train Epoch: 40 [221/250 28288/32000 (88%)] Loss: 1.54509 (QuantReg: 9.70423) QuantErr: 9.70423 batch_time=0.49363 
Train Epoch: 40 [232/250 29696/32000 (93%)] Loss: 1.69100 (QuantReg: 9.43260) QuantErr: 9.43260 batch_time=0.70690 
Train Epoch: 40 [243/250 31104/32000 (97%)] Loss: 1.35565 (QuantReg: 9.11448) QuantErr: 9.11448 batch_time=0.49679 
Train Epoch: 40 codebook_update_time=1.65343
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch40.pth ...
Done in 4.239s
removing stale ckpt [epoch 39] [took 0.00s]
 epoch          : 40
 loss           : 1.5175247006416321
 quant_reg      : 9.456102207183838
 quant_err      : 9.456102207183838
 learning_rate  : 6.763797713952796e-06
 n_samples      : 1280000
 n_steps        : 10000
 MSRVTT_full_val/t2v_metrics/R1: 31.790744466800806
 MSRVTT_full_val/t2v_metrics/R5: 67.40442655935614
 MSRVTT_full_val/t2v_metrics/R10: 81.48893360160966
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.199195171026158
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.89366448814086
 MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
 MSRVTT_full_val/v2t_metrics/R5: 74.24547283702213
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.197183098591549
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.79606876602506
 MSRVTT_full_test/t2v_metrics/R1: 12.94314381270903
 MSRVTT_full_test/t2v_metrics/R5: 35.18394648829432
 MSRVTT_full_test/t2v_metrics/R10: 48.42809364548495
 MSRVTT_full_test/t2v_metrics/R50: 79.6989966555184
 MSRVTT_full_test/t2v_metrics/MedR: 12.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.49632107023412
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.043178341991243
 MSRVTT_full_test/v2t_metrics/R1: 13.7123745819398
 MSRVTT_full_test/v2t_metrics/R5: 38.39464882943144
 MSRVTT_full_test/v2t_metrics/R10: 53.64548494983278
 MSRVTT_full_test/v2t_metrics/R50: 84.08026755852843
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.86755852842809
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.453614643194488
 mnt_best       : 28.712812844534504
 not_improved_count: 8
Train Epoch: 41 [1/250 128/32000 (0%)] Loss: 1.55894 (QuantReg: 9.20165) QuantErr: 9.20165 batch_time=33.04003 
Train Epoch: 41 [12/250 1536/32000 (5%)] Loss: 1.60225 (QuantReg: 9.31721) QuantErr: 9.31721 batch_time=0.49354 
Train Epoch: 41 [23/250 2944/32000 (9%)] Loss: 1.47507 (QuantReg: 9.46834) QuantErr: 9.46834 batch_time=0.84729 
Train Epoch: 41 [34/250 4352/32000 (14%)] Loss: 1.61960 (QuantReg: 9.68321) QuantErr: 9.68321 batch_time=0.49804 
Train Epoch: 41 [45/250 5760/32000 (18%)] Loss: 1.13530 (QuantReg: 9.54570) QuantErr: 9.54570 batch_time=0.49811 
Train Epoch: 41 [56/250 7168/32000 (22%)] Loss: 1.64864 (QuantReg: 9.07949) QuantErr: 9.07949 batch_time=0.50388 
Train Epoch: 41 [67/250 8576/32000 (27%)] Loss: 1.82246 (QuantReg: 9.22184) QuantErr: 9.22184 batch_time=1.49310 
Train Epoch: 41 [78/250 9984/32000 (31%)] Loss: 1.55607 (QuantReg: 9.36751) QuantErr: 9.36751 batch_time=0.60519 
Train Epoch: 41 [89/250 11392/32000 (36%)] Loss: 1.58620 (QuantReg: 9.68403) QuantErr: 9.68403 batch_time=0.50889 
Train Epoch: 41 [100/250 12800/32000 (40%)] Loss: 1.39660 (QuantReg: 9.60744) QuantErr: 9.60744 batch_time=0.50639 
Train Epoch: 41 [111/250 14208/32000 (44%)] Loss: 1.60366 (QuantReg: 9.44090) QuantErr: 9.44090 batch_time=0.77290 
Train Epoch: 41 [122/250 15616/32000 (49%)] Loss: 1.50779 (QuantReg: 9.44687) QuantErr: 9.44687 batch_time=0.48898 
Train Epoch: 41 [133/250 17024/32000 (53%)] Loss: 1.35553 (QuantReg: 9.38547) QuantErr: 9.38547 batch_time=0.49000 
Train Epoch: 41 [144/250 18432/32000 (58%)] Loss: 1.28784 (QuantReg: 9.35162) QuantErr: 9.35162 batch_time=0.51105 
Train Epoch: 41 [155/250 19840/32000 (62%)] Loss: 1.50474 (QuantReg: 9.41885) QuantErr: 9.41885 batch_time=0.49330 
Train Epoch: 41 [166/250 21248/32000 (66%)] Loss: 1.48993 (QuantReg: 9.54524) QuantErr: 9.54524 batch_time=0.49214 
Train Epoch: 41 [177/250 22656/32000 (71%)] Loss: 1.53613 (QuantReg: 9.41517) QuantErr: 9.41517 batch_time=0.51095 
Train Epoch: 41 [188/250 24064/32000 (75%)] Loss: 1.65443 (QuantReg: 9.48078) QuantErr: 9.48078 batch_time=0.63892 
Train Epoch: 41 [199/250 25472/32000 (80%)] Loss: 1.43910 (QuantReg: 9.40983) QuantErr: 9.40983 batch_time=0.50858 
Train Epoch: 41 [210/250 26880/32000 (84%)] Loss: 1.38705 (QuantReg: 9.61277) QuantErr: 9.61277 batch_time=0.51038 
Train Epoch: 41 [221/250 28288/32000 (88%)] Loss: 1.57115 (QuantReg: 9.50770) QuantErr: 9.50770 batch_time=0.50834 
Train Epoch: 41 [232/250 29696/32000 (93%)] Loss: 1.57249 (QuantReg: 9.44282) QuantErr: 9.44282 batch_time=0.48762 
Train Epoch: 41 [243/250 31104/32000 (97%)] Loss: 1.62681 (QuantReg: 9.47906) QuantErr: 9.47906 batch_time=0.51005 
Train Epoch: 41 codebook_update_time=1.61253
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch41.pth ...
Done in 4.011s
removing stale ckpt [epoch 40] [took 0.00s]
 epoch          : 41
 loss           : 1.5233461389541625
 quant_reg      : 9.500748115539551
 quant_err      : 9.500748115539551
 learning_rate  : 6.425607828255156e-06
 n_samples      : 1312000
 n_steps        : 10250
 MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
 MSRVTT_full_val/t2v_metrics/R5: 65.99597585513078
 MSRVTT_full_val/t2v_metrics/R10: 80.6841046277666
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.462776659959758
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.43474551535854
 MSRVTT_full_val/v2t_metrics/R1: 37.02213279678068
 MSRVTT_full_val/v2t_metrics/R5: 74.04426559356136
 MSRVTT_full_val/v2t_metrics/R10: 83.09859154929578
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.207243460764587
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.07294215289279
 MSRVTT_full_test/t2v_metrics/R1: 12.94314381270903
 MSRVTT_full_test/t2v_metrics/R5: 35.21739130434783
 MSRVTT_full_test/t2v_metrics/R10: 48.896321070234116
 MSRVTT_full_test/t2v_metrics/R50: 79.93311036789298
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.87073578595318
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.14217858205683
 MSRVTT_full_test/v2t_metrics/R1: 13.946488294314381
 MSRVTT_full_test/v2t_metrics/R5: 39.063545150501675
 MSRVTT_full_test/v2t_metrics/R10: 54.046822742474916
 MSRVTT_full_test/v2t_metrics/R50: 84.61538461538461
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.3752508361204
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.87940307075169
 mnt_best       : 28.712812844534504
 not_improved_count: 9
Train Epoch: 42 [1/250 128/32000 (0%)] Loss: 1.48382 (QuantReg: 9.27070) QuantErr: 9.27070 batch_time=34.00523 
Train Epoch: 42 [12/250 1536/32000 (5%)] Loss: 1.55015 (QuantReg: 9.27906) QuantErr: 9.27906 batch_time=0.56761 
Train Epoch: 42 [23/250 2944/32000 (9%)] Loss: 1.53861 (QuantReg: 9.38589) QuantErr: 9.38589 batch_time=0.50449 
Train Epoch: 42 [34/250 4352/32000 (14%)] Loss: 1.35486 (QuantReg: 9.42496) QuantErr: 9.42496 batch_time=0.50784 
Train Epoch: 42 [45/250 5760/32000 (18%)] Loss: 1.76481 (QuantReg: 9.70301) QuantErr: 9.70301 batch_time=0.50592 
Train Epoch: 42 [56/250 7168/32000 (22%)] Loss: 1.55132 (QuantReg: 9.48581) QuantErr: 9.48581 batch_time=0.51617 
Train Epoch: 42 [67/250 8576/32000 (27%)] Loss: 1.45160 (QuantReg: 9.62514) QuantErr: 9.62514 batch_time=0.51038 
Train Epoch: 42 [78/250 9984/32000 (31%)] Loss: 1.57750 (QuantReg: 9.17365) QuantErr: 9.17365 batch_time=0.50803 
Train Epoch: 42 [89/250 11392/32000 (36%)] Loss: 1.66380 (QuantReg: 9.47662) QuantErr: 9.47662 batch_time=0.49130 
Train Epoch: 42 [100/250 12800/32000 (40%)] Loss: 1.35928 (QuantReg: 9.55243) QuantErr: 9.55243 batch_time=0.50608 
Train Epoch: 42 [111/250 14208/32000 (44%)] Loss: 1.77272 (QuantReg: 9.47831) QuantErr: 9.47831 batch_time=0.52350 
Train Epoch: 42 [122/250 15616/32000 (49%)] Loss: 1.70673 (QuantReg: 9.82141) QuantErr: 9.82141 batch_time=0.50461 
Train Epoch: 42 [133/250 17024/32000 (53%)] Loss: 1.45925 (QuantReg: 9.63569) QuantErr: 9.63569 batch_time=0.51340 
Train Epoch: 42 [144/250 18432/32000 (58%)] Loss: 1.41891 (QuantReg: 9.44215) QuantErr: 9.44215 batch_time=5.03080 
Train Epoch: 42 [155/250 19840/32000 (62%)] Loss: 1.69246 (QuantReg: 9.29122) QuantErr: 9.29122 batch_time=0.49752 
Train Epoch: 42 [166/250 21248/32000 (66%)] Loss: 1.41739 (QuantReg: 9.57486) QuantErr: 9.57486 batch_time=0.49082 
Train Epoch: 42 [177/250 22656/32000 (71%)] Loss: 1.80292 (QuantReg: 9.44934) QuantErr: 9.44934 batch_time=0.52878 
Train Epoch: 42 [188/250 24064/32000 (75%)] Loss: 1.49133 (QuantReg: 9.28933) QuantErr: 9.28933 batch_time=0.50207 
Train Epoch: 42 [199/250 25472/32000 (80%)] Loss: 1.62740 (QuantReg: 9.57483) QuantErr: 9.57483 batch_time=0.50361 
Train Epoch: 42 [210/250 26880/32000 (84%)] Loss: 1.58400 (QuantReg: 9.47194) QuantErr: 9.47194 batch_time=0.52123 
Train Epoch: 42 [221/250 28288/32000 (88%)] Loss: 1.37859 (QuantReg: 9.38231) QuantErr: 9.38231 batch_time=0.52604 
Train Epoch: 42 [232/250 29696/32000 (93%)] Loss: 1.70355 (QuantReg: 9.58195) QuantErr: 9.58195 batch_time=0.49669 
Train Epoch: 42 [243/250 31104/32000 (97%)] Loss: 1.39815 (QuantReg: 9.54462) QuantErr: 9.54462 batch_time=0.51707 
Train Epoch: 42 codebook_update_time=1.62777
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch42.pth ...
Done in 22.751s
removing stale ckpt [epoch 41] [took 0.00s]
 epoch          : 42
 loss           : 1.522733112335205
 quant_reg      : 9.4723673248291
 quant_err      : 9.4723673248291
 learning_rate  : 6.104327436842398e-06
 n_samples      : 1344000
 n_steps        : 10500
 MSRVTT_full_val/t2v_metrics/R1: 32.796780684104625
 MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
 MSRVTT_full_val/t2v_metrics/R10: 81.48893360160966
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.217303822937627
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.308053560326876
 MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
 MSRVTT_full_val/v2t_metrics/R5: 72.83702213279678
 MSRVTT_full_val/v2t_metrics/R10: 84.10462776659959
 MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.366197183098592
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.31313402882649
 MSRVTT_full_test/t2v_metrics/R1: 12.675585284280936
 MSRVTT_full_test/t2v_metrics/R5: 35.585284280936456
 MSRVTT_full_test/t2v_metrics/R10: 49.79933110367893
 MSRVTT_full_test/t2v_metrics/R50: 80.13377926421404
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.12173913043478
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.215472249832242
 MSRVTT_full_test/v2t_metrics/R1: 14.147157190635452
 MSRVTT_full_test/v2t_metrics/R5: 38.79598662207358
 MSRVTT_full_test/v2t_metrics/R10: 54.11371237458194
 MSRVTT_full_test/v2t_metrics/R50: 83.94648829431438
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.33913043478261
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.968566510538995
 mnt_best       : 28.712812844534504
 not_improved_count: 10
Train Epoch: 43 [1/250 128/32000 (0%)] Loss: 1.64006 (QuantReg: 9.49926) QuantErr: 9.49926 batch_time=37.62183 
Train Epoch: 43 [12/250 1536/32000 (5%)] Loss: 1.59950 (QuantReg: 9.50320) QuantErr: 9.50320 batch_time=0.51442 
Train Epoch: 43 [23/250 2944/32000 (9%)] Loss: 1.45504 (QuantReg: 9.31168) QuantErr: 9.31168 batch_time=0.54694 
Train Epoch: 43 [34/250 4352/32000 (14%)] Loss: 1.34491 (QuantReg: 9.61225) QuantErr: 9.61225 batch_time=0.49676 
Train Epoch: 43 [45/250 5760/32000 (18%)] Loss: 1.37286 (QuantReg: 9.41312) QuantErr: 9.41312 batch_time=0.84897 
Train Epoch: 43 [56/250 7168/32000 (22%)] Loss: 1.64391 (QuantReg: 9.76804) QuantErr: 9.76804 batch_time=0.49447 
Train Epoch: 43 [67/250 8576/32000 (27%)] Loss: 1.39803 (QuantReg: 9.32878) QuantErr: 9.32878 batch_time=0.53402 
Train Epoch: 43 [78/250 9984/32000 (31%)] Loss: 1.43847 (QuantReg: 9.41079) QuantErr: 9.41079 batch_time=0.50183 
Train Epoch: 43 [89/250 11392/32000 (36%)] Loss: 1.55024 (QuantReg: 9.44634) QuantErr: 9.44634 batch_time=0.49741 
Train Epoch: 43 [100/250 12800/32000 (40%)] Loss: 1.53235 (QuantReg: 9.41701) QuantErr: 9.41701 batch_time=0.51567 
Train Epoch: 43 [111/250 14208/32000 (44%)] Loss: 1.63697 (QuantReg: 9.51121) QuantErr: 9.51121 batch_time=0.47821 
Train Epoch: 43 [122/250 15616/32000 (49%)] Loss: 1.85334 (QuantReg: 9.25890) QuantErr: 9.25890 batch_time=0.49335 
Train Epoch: 43 [133/250 17024/32000 (53%)] Loss: 1.56626 (QuantReg: 9.72848) QuantErr: 9.72848 batch_time=0.52154 
Train Epoch: 43 [144/250 18432/32000 (58%)] Loss: 1.27474 (QuantReg: 9.49780) QuantErr: 9.49780 batch_time=3.41171 
Train Epoch: 43 [155/250 19840/32000 (62%)] Loss: 1.70979 (QuantReg: 9.44254) QuantErr: 9.44254 batch_time=0.49244 
Train Epoch: 43 [166/250 21248/32000 (66%)] Loss: 1.63078 (QuantReg: 9.51546) QuantErr: 9.51546 batch_time=0.49767 
Train Epoch: 43 [177/250 22656/32000 (71%)] Loss: 1.35610 (QuantReg: 9.39738) QuantErr: 9.39738 batch_time=0.52914 
Train Epoch: 43 [188/250 24064/32000 (75%)] Loss: 1.43988 (QuantReg: 9.62458) QuantErr: 9.62458 batch_time=0.51814 
Train Epoch: 43 [199/250 25472/32000 (80%)] Loss: 1.32132 (QuantReg: 9.72799) QuantErr: 9.72799 batch_time=0.50823 
Train Epoch: 43 [210/250 26880/32000 (84%)] Loss: 1.69136 (QuantReg: 9.65145) QuantErr: 9.65145 batch_time=0.51240 
Train Epoch: 43 [221/250 28288/32000 (88%)] Loss: 1.67672 (QuantReg: 9.70382) QuantErr: 9.70382 batch_time=0.50903 
Train Epoch: 43 [232/250 29696/32000 (93%)] Loss: 1.36068 (QuantReg: 9.49177) QuantErr: 9.49177 batch_time=0.50097 
Train Epoch: 43 [243/250 31104/32000 (97%)] Loss: 1.62585 (QuantReg: 9.48573) QuantErr: 9.48573 batch_time=0.55707 
Train Epoch: 43 codebook_update_time=1.62293
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch43.pth ...
Done in 3.782s
removing stale ckpt [epoch 42] [took 0.00s]
 epoch          : 43
 loss           : 1.5108926692008973
 quant_reg      : 9.478757190704346
 quant_err      : 9.478757190704346
 learning_rate  : 5.799111065000278e-06
 n_samples      : 1376000
 n_steps        : 10750
 MSRVTT_full_val/t2v_metrics/R1: 34.40643863179074
 MSRVTT_full_val/t2v_metrics/R5: 68.41046277665995
 MSRVTT_full_val/t2v_metrics/R10: 80.48289738430583
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.120724346076459
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.43217771541855
 MSRVTT_full_val/v2t_metrics/R1: 36.21730382293762
 MSRVTT_full_val/v2t_metrics/R5: 73.64185110663983
 MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.072434607645875
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.61463676774198
 MSRVTT_full_test/t2v_metrics/R1: 12.976588628762542
 MSRVTT_full_test/t2v_metrics/R5: 35.819397993311036
 MSRVTT_full_test/t2v_metrics/R10: 48.862876254180605
 MSRVTT_full_test/t2v_metrics/R50: 79.79933110367892
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.27190635451505
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.319523984026077
 MSRVTT_full_test/v2t_metrics/R1: 14.214046822742475
 MSRVTT_full_test/v2t_metrics/R5: 38.79598662207358
 MSRVTT_full_test/v2t_metrics/R10: 53.812709030100336
 MSRVTT_full_test/v2t_metrics/R50: 83.74581939799332
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 40.160702341137124
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.95968021233071
 mnt_best       : 28.712812844534504
 not_improved_count: 11
Train Epoch: 44 [1/250 128/32000 (0%)] Loss: 1.28390 (QuantReg: 9.56989) QuantErr: 9.56989 batch_time=36.68348 
Train Epoch: 44 [12/250 1536/32000 (5%)] Loss: 1.35403 (QuantReg: 9.51347) QuantErr: 9.51347 batch_time=0.51459 
Train Epoch: 44 [23/250 2944/32000 (9%)] Loss: 1.42741 (QuantReg: 9.54801) QuantErr: 9.54801 batch_time=0.48996 
Train Epoch: 44 [34/250 4352/32000 (14%)] Loss: 1.47301 (QuantReg: 9.44478) QuantErr: 9.44478 batch_time=0.50227 
Train Epoch: 44 [45/250 5760/32000 (18%)] Loss: 1.73699 (QuantReg: 9.43584) QuantErr: 9.43584 batch_time=0.49548 
Train Epoch: 44 [56/250 7168/32000 (22%)] Loss: 1.25859 (QuantReg: 9.12693) QuantErr: 9.12693 batch_time=0.48444 
Train Epoch: 44 [67/250 8576/32000 (27%)] Loss: 1.40175 (QuantReg: 9.53998) QuantErr: 9.53998 batch_time=0.49235 
Train Epoch: 44 [78/250 9984/32000 (31%)] Loss: 1.89789 (QuantReg: 9.46335) QuantErr: 9.46335 batch_time=0.48452 
Train Epoch: 44 [89/250 11392/32000 (36%)] Loss: 1.46762 (QuantReg: 9.60995) QuantErr: 9.60995 batch_time=0.50261 
Train Epoch: 44 [100/250 12800/32000 (40%)] Loss: 1.58276 (QuantReg: 9.37015) QuantErr: 9.37015 batch_time=0.48948 
Train Epoch: 44 [111/250 14208/32000 (44%)] Loss: 1.75498 (QuantReg: 9.83059) QuantErr: 9.83059 batch_time=0.60152 
Train Epoch: 44 [122/250 15616/32000 (49%)] Loss: 1.63180 (QuantReg: 9.44882) QuantErr: 9.44882 batch_time=0.48307 
Train Epoch: 44 [133/250 17024/32000 (53%)] Loss: 1.46343 (QuantReg: 9.61099) QuantErr: 9.61099 batch_time=1.28602 
Train Epoch: 44 [144/250 18432/32000 (58%)] Loss: 1.54297 (QuantReg: 9.31493) QuantErr: 9.31493 batch_time=0.50548 
Train Epoch: 44 [155/250 19840/32000 (62%)] Loss: 1.58848 (QuantReg: 9.47394) QuantErr: 9.47394 batch_time=0.49509 
Train Epoch: 44 [166/250 21248/32000 (66%)] Loss: 1.39865 (QuantReg: 9.69779) QuantErr: 9.69779 batch_time=0.49358 
Train Epoch: 44 [177/250 22656/32000 (71%)] Loss: 1.41761 (QuantReg: 9.20802) QuantErr: 9.20802 batch_time=0.49328 
Train Epoch: 44 [188/250 24064/32000 (75%)] Loss: 1.36865 (QuantReg: 9.17427) QuantErr: 9.17427 batch_time=0.50106 
Train Epoch: 44 [199/250 25472/32000 (80%)] Loss: 1.57817 (QuantReg: 9.27779) QuantErr: 9.27779 batch_time=0.50408 
Train Epoch: 44 [210/250 26880/32000 (84%)] Loss: 1.51093 (QuantReg: 9.71532) QuantErr: 9.71532 batch_time=0.50614 
Train Epoch: 44 [221/250 28288/32000 (88%)] Loss: 1.52643 (QuantReg: 9.50951) QuantErr: 9.50951 batch_time=0.51366 
Train Epoch: 44 [232/250 29696/32000 (93%)] Loss: 1.23479 (QuantReg: 9.33315) QuantErr: 9.33315 batch_time=0.95432 
Train Epoch: 44 [243/250 31104/32000 (97%)] Loss: 1.36779 (QuantReg: 9.48165) QuantErr: 9.48165 batch_time=0.48942 
Train Epoch: 44 codebook_update_time=1.87114
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch44.pth ...
Done in 3.998s
removing stale ckpt [epoch 43] [took 0.00s]
 epoch          : 44
 loss           : 1.5220510869026185
 quant_reg      : 9.47707689666748
 quant_err      : 9.47707689666748
 learning_rate  : 5.5091555117502635e-06
 n_samples      : 1408000
 n_steps        : 11000
 MSRVTT_full_val/t2v_metrics/R1: 32.19315895372233
 MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
 MSRVTT_full_val/t2v_metrics/R10: 81.08651911468813
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.08450704225352
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.868196086610496
 MSRVTT_full_val/v2t_metrics/R1: 37.625754527162975
 MSRVTT_full_val/v2t_metrics/R5: 73.44064386317908
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.929577464788732
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.579789271544136
 MSRVTT_full_test/t2v_metrics/R1: 13.277591973244148
 MSRVTT_full_test/t2v_metrics/R5: 35.11705685618729
 MSRVTT_full_test/t2v_metrics/R10: 49.23076923076923
 MSRVTT_full_test/t2v_metrics/R50: 79.63210702341136
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.31638795986622
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.420039919999102
 MSRVTT_full_test/v2t_metrics/R1: 14.31438127090301
 MSRVTT_full_test/v2t_metrics/R5: 39.23076923076923
 MSRVTT_full_test/v2t_metrics/R10: 53.67892976588629
 MSRVTT_full_test/v2t_metrics/R50: 84.21404682274247
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.99230769230769
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.12201810074213
 mnt_best       : 28.712812844534504
 not_improved_count: 12
Train Epoch: 45 [1/250 128/32000 (0%)] Loss: 1.55881 (QuantReg: 9.37475) QuantErr: 9.37475 batch_time=37.02791 
Train Epoch: 45 [12/250 1536/32000 (5%)] Loss: 1.50334 (QuantReg: 9.50648) QuantErr: 9.50648 batch_time=0.48539 
Train Epoch: 45 [23/250 2944/32000 (9%)] Loss: 1.61625 (QuantReg: 9.52200) QuantErr: 9.52200 batch_time=0.48467 
Train Epoch: 45 [34/250 4352/32000 (14%)] Loss: 1.31557 (QuantReg: 9.50097) QuantErr: 9.50097 batch_time=0.50864 
Train Epoch: 45 [45/250 5760/32000 (18%)] Loss: 1.53752 (QuantReg: 9.37968) QuantErr: 9.37968 batch_time=0.51870 
Train Epoch: 45 [56/250 7168/32000 (22%)] Loss: 1.22961 (QuantReg: 9.29997) QuantErr: 9.29997 batch_time=0.76887 
Train Epoch: 45 [67/250 8576/32000 (27%)] Loss: 1.57816 (QuantReg: 9.22387) QuantErr: 9.22387 batch_time=0.50439 
Train Epoch: 45 [78/250 9984/32000 (31%)] Loss: 1.95076 (QuantReg: 9.39998) QuantErr: 9.39998 batch_time=0.48746 
Train Epoch: 45 [89/250 11392/32000 (36%)] Loss: 1.53622 (QuantReg: 9.53074) QuantErr: 9.53074 batch_time=0.52420 
Train Epoch: 45 [100/250 12800/32000 (40%)] Loss: 1.30349 (QuantReg: 9.33874) QuantErr: 9.33874 batch_time=0.49352 
Train Epoch: 45 [111/250 14208/32000 (44%)] Loss: 1.64615 (QuantReg: 9.59106) QuantErr: 9.59106 batch_time=0.50090 
Train Epoch: 45 [122/250 15616/32000 (49%)] Loss: 1.44261 (QuantReg: 9.21469) QuantErr: 9.21469 batch_time=0.49090 
Train Epoch: 45 [133/250 17024/32000 (53%)] Loss: 1.47728 (QuantReg: 9.43978) QuantErr: 9.43978 batch_time=0.49293 
Train Epoch: 45 [144/250 18432/32000 (58%)] Loss: 1.54752 (QuantReg: 9.55948) QuantErr: 9.55948 batch_time=0.98951 
Train Epoch: 45 [155/250 19840/32000 (62%)] Loss: 1.52905 (QuantReg: 9.61188) QuantErr: 9.61188 batch_time=0.51632 
Train Epoch: 45 [166/250 21248/32000 (66%)] Loss: 1.37370 (QuantReg: 9.27694) QuantErr: 9.27694 batch_time=0.48755 
Train Epoch: 45 [177/250 22656/32000 (71%)] Loss: 1.49718 (QuantReg: 9.41270) QuantErr: 9.41270 batch_time=0.73369 
Train Epoch: 45 [188/250 24064/32000 (75%)] Loss: 1.50246 (QuantReg: 9.72505) QuantErr: 9.72505 batch_time=0.49444 
Train Epoch: 45 [199/250 25472/32000 (80%)] Loss: 1.26556 (QuantReg: 9.62471) QuantErr: 9.62471 batch_time=0.50117 
Train Epoch: 45 [210/250 26880/32000 (84%)] Loss: 1.59379 (QuantReg: 9.43329) QuantErr: 9.43329 batch_time=0.50547 
Train Epoch: 45 [221/250 28288/32000 (88%)] Loss: 1.55173 (QuantReg: 9.35785) QuantErr: 9.35785 batch_time=0.50496 
Train Epoch: 45 [232/250 29696/32000 (93%)] Loss: 1.29914 (QuantReg: 9.42374) QuantErr: 9.42374 batch_time=0.51839 
Train Epoch: 45 [243/250 31104/32000 (97%)] Loss: 1.62791 (QuantReg: 9.21881) QuantErr: 9.21881 batch_time=0.50132 
Train Epoch: 45 codebook_update_time=1.65778
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch45.pth ...
Done in 3.965s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch45.pth ...
Done in 7.864s
removing stale ckpt [epoch 44] [took 0.00s]
 epoch          : 45
 loss           : 1.5075517749786378
 quant_reg      : 9.481056987762452
 quant_err      : 9.481056987762452
 learning_rate  : 5.23369773616275e-06
 n_samples      : 1440000
 n_steps        : 11250
 MSRVTT_full_val/t2v_metrics/R1: 33.199195171026155
 MSRVTT_full_val/t2v_metrics/R5: 65.99597585513078
 MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 7.961770623742455
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.26305745279515
 MSRVTT_full_val/v2t_metrics/R1: 36.21730382293762
 MSRVTT_full_val/v2t_metrics/R5: 73.8430583501006
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 6.835010060362173
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.91247290181563
 MSRVTT_full_test/t2v_metrics/R1: 13.478260869565217
 MSRVTT_full_test/t2v_metrics/R5: 35.785953177257525
 MSRVTT_full_test/t2v_metrics/R10: 49.59866220735786
 MSRVTT_full_test/t2v_metrics/R50: 80.2675585284281
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.98227424749164
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.814127261468936
 MSRVTT_full_test/v2t_metrics/R1: 14.046822742474916
 MSRVTT_full_test/v2t_metrics/R5: 39.63210702341137
 MSRVTT_full_test/v2t_metrics/R10: 54.61538461538461
 MSRVTT_full_test/v2t_metrics/R50: 83.9799331103679
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.45852842809364
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.21141226134715
 mnt_best       : 28.814127261468936
 not_improved_count: 0
Train Epoch: 46 [1/250 128/32000 (0%)] Loss: 1.47206 (QuantReg: 9.26955) QuantErr: 9.26955 batch_time=31.62664 
Train Epoch: 46 [12/250 1536/32000 (5%)] Loss: 1.48852 (QuantReg: 9.48969) QuantErr: 9.48969 batch_time=0.48842 
Train Epoch: 46 [23/250 2944/32000 (9%)] Loss: 1.21354 (QuantReg: 9.46548) QuantErr: 9.46548 batch_time=0.48572 
Train Epoch: 46 [34/250 4352/32000 (14%)] Loss: 1.36820 (QuantReg: 9.59521) QuantErr: 9.59521 batch_time=0.48898 
Train Epoch: 46 [45/250 5760/32000 (18%)] Loss: 1.66850 (QuantReg: 9.60164) QuantErr: 9.60164 batch_time=0.48925 
Train Epoch: 46 [56/250 7168/32000 (22%)] Loss: 1.26146 (QuantReg: 9.32588) QuantErr: 9.32588 batch_time=0.80832 
Train Epoch: 46 [67/250 8576/32000 (27%)] Loss: 1.53512 (QuantReg: 9.43691) QuantErr: 9.43691 batch_time=0.48723 
Train Epoch: 46 [78/250 9984/32000 (31%)] Loss: 1.56344 (QuantReg: 9.61324) QuantErr: 9.61324 batch_time=0.49047 
Train Epoch: 46 [89/250 11392/32000 (36%)] Loss: 1.53786 (QuantReg: 9.47754) QuantErr: 9.47754 batch_time=0.48823 
Train Epoch: 46 [100/250 12800/32000 (40%)] Loss: 1.52212 (QuantReg: 9.40713) QuantErr: 9.40713 batch_time=0.49705 
Train Epoch: 46 [111/250 14208/32000 (44%)] Loss: 1.33281 (QuantReg: 9.21268) QuantErr: 9.21268 batch_time=0.52670 
Train Epoch: 46 [122/250 15616/32000 (49%)] Loss: 1.51756 (QuantReg: 9.51104) QuantErr: 9.51104 batch_time=0.49288 
Train Epoch: 46 [133/250 17024/32000 (53%)] Loss: 1.80396 (QuantReg: 9.38201) QuantErr: 9.38201 batch_time=0.49686 
Train Epoch: 46 [144/250 18432/32000 (58%)] Loss: 1.50614 (QuantReg: 9.31961) QuantErr: 9.31961 batch_time=0.49385 
Train Epoch: 46 [155/250 19840/32000 (62%)] Loss: 1.48060 (QuantReg: 9.62952) QuantErr: 9.62952 batch_time=0.49609 
Train Epoch: 46 [166/250 21248/32000 (66%)] Loss: 1.42431 (QuantReg: 9.83809) QuantErr: 9.83809 batch_time=0.50183 
Train Epoch: 46 [177/250 22656/32000 (71%)] Loss: 1.57670 (QuantReg: 9.46948) QuantErr: 9.46948 batch_time=0.64504 
Train Epoch: 46 [188/250 24064/32000 (75%)] Loss: 1.38779 (QuantReg: 9.39229) QuantErr: 9.39229 batch_time=0.49710 
Train Epoch: 46 [199/250 25472/32000 (80%)] Loss: 1.41829 (QuantReg: 9.53138) QuantErr: 9.53138 batch_time=0.49168 
Train Epoch: 46 [210/250 26880/32000 (84%)] Loss: 1.35993 (QuantReg: 9.71729) QuantErr: 9.71729 batch_time=0.49981 
Train Epoch: 46 [221/250 28288/32000 (88%)] Loss: 1.28355 (QuantReg: 9.38439) QuantErr: 9.38439 batch_time=0.49381 
Train Epoch: 46 [232/250 29696/32000 (93%)] Loss: 1.34975 (QuantReg: 9.51690) QuantErr: 9.51690 batch_time=0.49542 
Train Epoch: 46 [243/250 31104/32000 (97%)] Loss: 1.43327 (QuantReg: 9.43896) QuantErr: 9.43896 batch_time=0.53172 
Train Epoch: 46 codebook_update_time=1.63482
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch46.pth ...
Done in 4.095s
removing stale ckpt [epoch 45] [took 0.02s]
 epoch          : 46
 loss           : 1.5075752882957458
 quant_reg      : 9.461283302307129
 quant_err      : 9.461283302307129
 learning_rate  : 4.972012849354612e-06
 n_samples      : 1472000
 n_steps        : 11500
 MSRVTT_full_val/t2v_metrics/R1: 31.99195171026157
 MSRVTT_full_val/t2v_metrics/R5: 66.59959758551308
 MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.498993963782695
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.510646141184445
 MSRVTT_full_val/v2t_metrics/R1: 37.625754527162975
 MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
 MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.122736418511066
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.47463360560591
 MSRVTT_full_test/t2v_metrics/R1: 12.876254180602007
 MSRVTT_full_test/t2v_metrics/R5: 34.94983277591973
 MSRVTT_full_test/t2v_metrics/R10: 48.52842809364549
 MSRVTT_full_test/t2v_metrics/R50: 79.6989966555184
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 51.18494983277592
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.951832756285068
 MSRVTT_full_test/v2t_metrics/R1: 13.846153846153847
 MSRVTT_full_test/v2t_metrics/R5: 38.46153846153846
 MSRVTT_full_test/v2t_metrics/R10: 53.779264214046826
 MSRVTT_full_test/v2t_metrics/R50: 83.74581939799332
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.624916387959864
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.595452941809604
 mnt_best       : 28.814127261468936
 not_improved_count: 1
Train Epoch: 47 [1/250 128/32000 (0%)] Loss: 1.47871 (QuantReg: 9.73051) QuantErr: 9.73051 batch_time=30.60000 
Train Epoch: 47 [12/250 1536/32000 (5%)] Loss: 1.62865 (QuantReg: 9.42380) QuantErr: 9.42380 batch_time=0.51206 
Train Epoch: 47 [23/250 2944/32000 (9%)] Loss: 1.46197 (QuantReg: 9.25947) QuantErr: 9.25947 batch_time=0.74232 
Train Epoch: 47 [34/250 4352/32000 (14%)] Loss: 1.43817 (QuantReg: 9.51270) QuantErr: 9.51270 batch_time=0.49421 
Train Epoch: 47 [45/250 5760/32000 (18%)] Loss: 1.56169 (QuantReg: 9.56592) QuantErr: 9.56592 batch_time=0.51917 
Train Epoch: 47 [56/250 7168/32000 (22%)] Loss: 1.30665 (QuantReg: 9.42777) QuantErr: 9.42777 batch_time=0.50965 
Train Epoch: 47 [67/250 8576/32000 (27%)] Loss: 1.71516 (QuantReg: 9.42411) QuantErr: 9.42411 batch_time=0.49575 
Train Epoch: 47 [78/250 9984/32000 (31%)] Loss: 1.30933 (QuantReg: 9.13834) QuantErr: 9.13834 batch_time=0.49156 
Train Epoch: 47 [89/250 11392/32000 (36%)] Loss: 1.39936 (QuantReg: 9.66839) QuantErr: 9.66839 batch_time=0.49483 
Train Epoch: 47 [100/250 12800/32000 (40%)] Loss: 1.39289 (QuantReg: 9.51623) QuantErr: 9.51623 batch_time=0.48811 
Train Epoch: 47 [111/250 14208/32000 (44%)] Loss: 1.71902 (QuantReg: 9.23594) QuantErr: 9.23594 batch_time=0.48392 
Train Epoch: 47 [122/250 15616/32000 (49%)] Loss: 1.44174 (QuantReg: 9.51369) QuantErr: 9.51369 batch_time=0.60436 
Train Epoch: 47 [133/250 17024/32000 (53%)] Loss: 1.63741 (QuantReg: 9.76719) QuantErr: 9.76719 batch_time=0.48915 
Train Epoch: 47 [144/250 18432/32000 (58%)] Loss: 1.54528 (QuantReg: 9.42062) QuantErr: 9.42062 batch_time=0.48645 
Train Epoch: 47 [155/250 19840/32000 (62%)] Loss: 1.58600 (QuantReg: 9.45987) QuantErr: 9.45987 batch_time=0.50685 
Train Epoch: 47 [166/250 21248/32000 (66%)] Loss: 1.52405 (QuantReg: 9.59878) QuantErr: 9.59878 batch_time=0.49500 
Train Epoch: 47 [177/250 22656/32000 (71%)] Loss: 1.79537 (QuantReg: 9.34644) QuantErr: 9.34644 batch_time=0.49404 
Train Epoch: 47 [188/250 24064/32000 (75%)] Loss: 1.43428 (QuantReg: 9.66184) QuantErr: 9.66184 batch_time=0.50346 
Train Epoch: 47 [199/250 25472/32000 (80%)] Loss: 1.50690 (QuantReg: 9.50735) QuantErr: 9.50735 batch_time=0.51047 
Train Epoch: 47 [210/250 26880/32000 (84%)] Loss: 1.50701 (QuantReg: 9.58549) QuantErr: 9.58549 batch_time=0.48785 
Train Epoch: 47 [221/250 28288/32000 (88%)] Loss: 1.69745 (QuantReg: 9.36290) QuantErr: 9.36290 batch_time=0.49926 
Train Epoch: 47 [232/250 29696/32000 (93%)] Loss: 1.34922 (QuantReg: 9.54176) QuantErr: 9.54176 batch_time=0.52037 
Train Epoch: 47 [243/250 31104/32000 (97%)] Loss: 1.54031 (QuantReg: 9.50594) QuantErr: 9.50594 batch_time=0.50831 
Train Epoch: 47 codebook_update_time=1.68850
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch47.pth ...
Done in 4.137s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch47.pth ...
Done in 8.008s
removing stale ckpt [epoch 46] [took 0.00s]
 epoch          : 47
 loss           : 1.5264765391349793
 quant_reg      : 9.472005313873291
 quant_err      : 9.472005313873291
 learning_rate  : 4.723412206886882e-06
 n_samples      : 1504000
 n_steps        : 11750
 MSRVTT_full_val/t2v_metrics/R1: 33.80281690140845
 MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
 MSRVTT_full_val/t2v_metrics/R10: 82.29376257545272
 MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.048289738430583
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.06465146745117
 MSRVTT_full_val/v2t_metrics/R1: 37.02213279678068
 MSRVTT_full_val/v2t_metrics/R5: 73.64185110663983
 MSRVTT_full_val/v2t_metrics/R10: 84.50704225352112
 MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.140845070422535
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.30459262777103
 MSRVTT_full_test/t2v_metrics/R1: 13.377926421404682
 MSRVTT_full_test/t2v_metrics/R5: 36.187290969899664
 MSRVTT_full_test/t2v_metrics/R10: 49.93311036789298
 MSRVTT_full_test/t2v_metrics/R50: 80.66889632107024
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 47.57424749163879
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.91419895025553
 MSRVTT_full_test/v2t_metrics/R1: 14.481605351170568
 MSRVTT_full_test/v2t_metrics/R5: 39.59866220735786
 MSRVTT_full_test/v2t_metrics/R10: 54.147157190635454
 MSRVTT_full_test/v2t_metrics/R50: 84.18060200668896
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 38.240635451505014
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.430958666348467
 mnt_best       : 28.91419895025553
 not_improved_count: 0
Train Epoch: 48 [1/250 128/32000 (0%)] Loss: 1.68240 (QuantReg: 9.20157) QuantErr: 9.20157 batch_time=37.18823 
Train Epoch: 48 [12/250 1536/32000 (5%)] Loss: 1.49160 (QuantReg: 9.52228) QuantErr: 9.52228 batch_time=0.49295 
Train Epoch: 48 [23/250 2944/32000 (9%)] Loss: 1.41564 (QuantReg: 9.44518) QuantErr: 9.44518 batch_time=0.49119 
Train Epoch: 48 [34/250 4352/32000 (14%)] Loss: 1.44839 (QuantReg: 9.43309) QuantErr: 9.43309 batch_time=0.78982 
Train Epoch: 48 [45/250 5760/32000 (18%)] Loss: 1.47048 (QuantReg: 9.27577) QuantErr: 9.27577 batch_time=0.48665 
Train Epoch: 48 [56/250 7168/32000 (22%)] Loss: 1.45676 (QuantReg: 9.56008) QuantErr: 9.56008 batch_time=0.47912 
Train Epoch: 48 [67/250 8576/32000 (27%)] Loss: 1.56361 (QuantReg: 9.53611) QuantErr: 9.53611 batch_time=0.49192 
Train Epoch: 48 [78/250 9984/32000 (31%)] Loss: 1.45280 (QuantReg: 9.21843) QuantErr: 9.21843 batch_time=0.49299 
Train Epoch: 48 [89/250 11392/32000 (36%)] Loss: 1.45882 (QuantReg: 9.53280) QuantErr: 9.53280 batch_time=0.93002 
Train Epoch: 48 [100/250 12800/32000 (40%)] Loss: 1.32287 (QuantReg: 9.41652) QuantErr: 9.41652 batch_time=0.50536 
Train Epoch: 48 [111/250 14208/32000 (44%)] Loss: 1.42691 (QuantReg: 9.59514) QuantErr: 9.59514 batch_time=0.51136 
Train Epoch: 48 [122/250 15616/32000 (49%)] Loss: 1.64581 (QuantReg: 9.43554) QuantErr: 9.43554 batch_time=0.48842 
Train Epoch: 48 [133/250 17024/32000 (53%)] Loss: 1.45928 (QuantReg: 9.38230) QuantErr: 9.38230 batch_time=0.72696 
Train Epoch: 48 [144/250 18432/32000 (58%)] Loss: 1.43155 (QuantReg: 9.46937) QuantErr: 9.46937 batch_time=2.96792 
Train Epoch: 48 [155/250 19840/32000 (62%)] Loss: 1.50581 (QuantReg: 9.67483) QuantErr: 9.67483 batch_time=0.49681 
Train Epoch: 48 [166/250 21248/32000 (66%)] Loss: 1.57390 (QuantReg: 9.33010) QuantErr: 9.33010 batch_time=0.49506 
Train Epoch: 48 [177/250 22656/32000 (71%)] Loss: 1.36142 (QuantReg: 9.69042) QuantErr: 9.69042 batch_time=0.49337 
Train Epoch: 48 [188/250 24064/32000 (75%)] Loss: 1.70226 (QuantReg: 9.58259) QuantErr: 9.58259 batch_time=0.50003 
Train Epoch: 48 [199/250 25472/32000 (80%)] Loss: 1.64479 (QuantReg: 9.59400) QuantErr: 9.59400 batch_time=0.49144 
Train Epoch: 48 [210/250 26880/32000 (84%)] Loss: 1.58454 (QuantReg: 9.53990) QuantErr: 9.53990 batch_time=0.50885 
Train Epoch: 48 [221/250 28288/32000 (88%)] Loss: 1.43394 (QuantReg: 9.45001) QuantErr: 9.45001 batch_time=0.49432 
Train Epoch: 48 [232/250 29696/32000 (93%)] Loss: 1.65255 (QuantReg: 9.67623) QuantErr: 9.67623 batch_time=0.51186 
Train Epoch: 48 [243/250 31104/32000 (97%)] Loss: 1.75303 (QuantReg: 9.37603) QuantErr: 9.37603 batch_time=0.50922 
Train Epoch: 48 codebook_update_time=1.77122
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch48.pth ...
Done in 3.969s
removing stale ckpt [epoch 47] [took 0.00s]
 epoch          : 48
 loss           : 1.5284766383171082
 quant_reg      : 9.475387145996093
 quant_err      : 9.475387145996093
 learning_rate  : 4.487241596542537e-06
 n_samples      : 1536000
 n_steps        : 12000
 MSRVTT_full_val/t2v_metrics/R1: 35.61368209255533
 MSRVTT_full_val/t2v_metrics/R5: 66.19718309859155
 MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
 MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.078470824949697
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.414858380699215
 MSRVTT_full_val/v2t_metrics/R1: 35.61368209255533
 MSRVTT_full_val/v2t_metrics/R5: 73.64185110663983
 MSRVTT_full_val/v2t_metrics/R10: 85.71428571428571
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.175050301810865
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.80392223322613
 MSRVTT_full_test/t2v_metrics/R1: 13.311036789297658
 MSRVTT_full_test/t2v_metrics/R5: 36.45484949832776
 MSRVTT_full_test/t2v_metrics/R10: 49.39799331103679
 MSRVTT_full_test/t2v_metrics/R50: 80.5351170568562
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 48.46220735785953
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.83315500971282
 MSRVTT_full_test/v2t_metrics/R1: 14.548494983277592
 MSRVTT_full_test/v2t_metrics/R5: 39.297658862876254
 MSRVTT_full_test/v2t_metrics/R10: 54.214046822742475
 MSRVTT_full_test/v2t_metrics/R50: 84.0133779264214
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.4938127090301
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.412236446079632
 mnt_best       : 28.91419895025553
 not_improved_count: 1
Train Epoch: 49 [1/250 128/32000 (0%)] Loss: 1.31009 (QuantReg: 9.26079) QuantErr: 9.26079 batch_time=32.01596 
Train Epoch: 49 [12/250 1536/32000 (5%)] Loss: 1.40391 (QuantReg: 9.33600) QuantErr: 9.33600 batch_time=1.02734 
Train Epoch: 49 [23/250 2944/32000 (9%)] Loss: 1.40054 (QuantReg: 9.44669) QuantErr: 9.44669 batch_time=0.49551 
Train Epoch: 49 [34/250 4352/32000 (14%)] Loss: 1.55031 (QuantReg: 9.66387) QuantErr: 9.66387 batch_time=0.48530 
Train Epoch: 49 [45/250 5760/32000 (18%)] Loss: 1.33638 (QuantReg: 9.52297) QuantErr: 9.52297 batch_time=0.66910 
Train Epoch: 49 [56/250 7168/32000 (22%)] Loss: 1.80610 (QuantReg: 9.58459) QuantErr: 9.58459 batch_time=0.47542 
Train Epoch: 49 [67/250 8576/32000 (27%)] Loss: 1.61333 (QuantReg: 9.66402) QuantErr: 9.66402 batch_time=0.61070 
Train Epoch: 49 [78/250 9984/32000 (31%)] Loss: 1.47913 (QuantReg: 9.37160) QuantErr: 9.37160 batch_time=0.48376 
Train Epoch: 49 [89/250 11392/32000 (36%)] Loss: 1.63889 (QuantReg: 9.44981) QuantErr: 9.44981 batch_time=0.49778 
Train Epoch: 49 [100/250 12800/32000 (40%)] Loss: 1.33491 (QuantReg: 9.36910) QuantErr: 9.36910 batch_time=0.48676 
Train Epoch: 49 [111/250 14208/32000 (44%)] Loss: 1.47150 (QuantReg: 9.59024) QuantErr: 9.59024 batch_time=1.42043 
Train Epoch: 49 [122/250 15616/32000 (49%)] Loss: 1.63538 (QuantReg: 9.26803) QuantErr: 9.26803 batch_time=0.49255 
Train Epoch: 49 [133/250 17024/32000 (53%)] Loss: 1.66697 (QuantReg: 9.22471) QuantErr: 9.22471 batch_time=0.48753 
Train Epoch: 49 [144/250 18432/32000 (58%)] Loss: 1.46003 (QuantReg: 9.50345) QuantErr: 9.50345 batch_time=1.72638 
Train Epoch: 49 [155/250 19840/32000 (62%)] Loss: 1.56339 (QuantReg: 9.60189) QuantErr: 9.60189 batch_time=0.47618 
Train Epoch: 49 [166/250 21248/32000 (66%)] Loss: 1.50908 (QuantReg: 9.52833) QuantErr: 9.52833 batch_time=0.50613 
Train Epoch: 49 [177/250 22656/32000 (71%)] Loss: 1.39133 (QuantReg: 9.42415) QuantErr: 9.42415 batch_time=0.51218 
Train Epoch: 49 [188/250 24064/32000 (75%)] Loss: 1.50620 (QuantReg: 9.39944) QuantErr: 9.39944 batch_time=0.48793 
Train Epoch: 49 [199/250 25472/32000 (80%)] Loss: 1.46060 (QuantReg: 9.59057) QuantErr: 9.59057 batch_time=0.48782 
Train Epoch: 49 [210/250 26880/32000 (84%)] Loss: 1.68572 (QuantReg: 9.47055) QuantErr: 9.47055 batch_time=0.48106 
Train Epoch: 49 [221/250 28288/32000 (88%)] Loss: 1.47717 (QuantReg: 9.50227) QuantErr: 9.50227 batch_time=1.05788 
Train Epoch: 49 [232/250 29696/32000 (93%)] Loss: 1.20556 (QuantReg: 9.58174) QuantErr: 9.58174 batch_time=0.60571 
Train Epoch: 49 [243/250 31104/32000 (97%)] Loss: 1.41176 (QuantReg: 9.36058) QuantErr: 9.36058 batch_time=0.49566 
Train Epoch: 49 codebook_update_time=1.59748
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch49.pth ...
Done in 4.133s
removing stale ckpt [epoch 48] [took 0.01s]
 epoch          : 49
 loss           : 1.5185086607933045
 quant_reg      : 9.479478122711182
 quant_err      : 9.479478122711182
 learning_rate  : 4.26287951671541e-06
 n_samples      : 1568000
 n_steps        : 12250
 MSRVTT_full_val/t2v_metrics/R1: 31.58953722334004
 MSRVTT_full_val/t2v_metrics/R5: 67.40442655935614
 MSRVTT_full_val/t2v_metrics/R10: 79.87927565392354
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 8.064386317907445
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.40580467796156
 MSRVTT_full_val/v2t_metrics/R1: 35.814889336016094
 MSRVTT_full_val/v2t_metrics/R5: 74.04426559356136
 MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
 MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.0885311871227366
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.69285388314163
 MSRVTT_full_test/t2v_metrics/R1: 13.110367892976589
 MSRVTT_full_test/t2v_metrics/R5: 35.752508361204015
 MSRVTT_full_test/t2v_metrics/R10: 48.52842809364549
 MSRVTT_full_test/t2v_metrics/R50: 79.96655518394648
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 50.90886287625418
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.333868451571878
 MSRVTT_full_test/v2t_metrics/R1: 14.046822742474916
 MSRVTT_full_test/v2t_metrics/R5: 38.46153846153846
 MSRVTT_full_test/v2t_metrics/R10: 54.54849498327759
 MSRVTT_full_test/v2t_metrics/R50: 83.77926421404682
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.40518394648829
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.88843055268599
 mnt_best       : 28.91419895025553
 not_improved_count: 2
Train Epoch: 50 [1/250 128/32000 (0%)] Loss: 1.50569 (QuantReg: 9.53514) QuantErr: 9.53514 batch_time=35.38312 
Train Epoch: 50 [12/250 1536/32000 (5%)] Loss: 1.38772 (QuantReg: 9.40860) QuantErr: 9.40860 batch_time=0.51552 
Train Epoch: 50 [23/250 2944/32000 (9%)] Loss: 1.51057 (QuantReg: 9.49381) QuantErr: 9.49381 batch_time=0.48981 
Train Epoch: 50 [34/250 4352/32000 (14%)] Loss: 1.49526 (QuantReg: 9.68480) QuantErr: 9.68480 batch_time=0.53566 
Train Epoch: 50 [45/250 5760/32000 (18%)] Loss: 1.71112 (QuantReg: 9.36571) QuantErr: 9.36571 batch_time=0.49223 
Train Epoch: 50 [56/250 7168/32000 (22%)] Loss: 1.47547 (QuantReg: 9.53427) QuantErr: 9.53427 batch_time=0.49969 
Train Epoch: 50 [67/250 8576/32000 (27%)] Loss: 1.46760 (QuantReg: 9.58861) QuantErr: 9.58861 batch_time=0.48813 
Train Epoch: 50 [78/250 9984/32000 (31%)] Loss: 1.66694 (QuantReg: 9.38085) QuantErr: 9.38085 batch_time=0.50735 
Train Epoch: 50 [89/250 11392/32000 (36%)] Loss: 1.50223 (QuantReg: 9.33335) QuantErr: 9.33335 batch_time=0.52539 
Train Epoch: 50 [100/250 12800/32000 (40%)] Loss: 1.32509 (QuantReg: 9.37892) QuantErr: 9.37892 batch_time=0.49436 
Train Epoch: 50 [111/250 14208/32000 (44%)] Loss: 1.51739 (QuantReg: 9.53167) QuantErr: 9.53167 batch_time=0.49060 
Train Epoch: 50 [122/250 15616/32000 (49%)] Loss: 1.63084 (QuantReg: 9.79192) QuantErr: 9.79192 batch_time=0.49826 
Train Epoch: 50 [133/250 17024/32000 (53%)] Loss: 1.62378 (QuantReg: 9.39314) QuantErr: 9.39314 batch_time=1.19297 
Train Epoch: 50 [144/250 18432/32000 (58%)] Loss: 1.52456 (QuantReg: 9.18322) QuantErr: 9.18322 batch_time=1.23589 
Train Epoch: 50 [155/250 19840/32000 (62%)] Loss: 1.69071 (QuantReg: 9.32410) QuantErr: 9.32410 batch_time=0.50813 
Train Epoch: 50 [166/250 21248/32000 (66%)] Loss: 1.38248 (QuantReg: 9.75749) QuantErr: 9.75749 batch_time=0.54525 
Train Epoch: 50 [177/250 22656/32000 (71%)] Loss: 1.78123 (QuantReg: 9.59569) QuantErr: 9.59569 batch_time=0.57322 
Train Epoch: 50 [188/250 24064/32000 (75%)] Loss: 1.78815 (QuantReg: 9.49948) QuantErr: 9.49948 batch_time=0.49201 
Train Epoch: 50 [199/250 25472/32000 (80%)] Loss: 1.26881 (QuantReg: 9.75693) QuantErr: 9.75693 batch_time=0.50695 
Train Epoch: 50 [210/250 26880/32000 (84%)] Loss: 1.52798 (QuantReg: 9.59736) QuantErr: 9.59736 batch_time=1.43885 
Train Epoch: 50 [221/250 28288/32000 (88%)] Loss: 1.48424 (QuantReg: 9.35078) QuantErr: 9.35078 batch_time=0.49633 
Train Epoch: 50 [232/250 29696/32000 (93%)] Loss: 1.59476 (QuantReg: 9.51394) QuantErr: 9.51394 batch_time=0.50598 
Train Epoch: 50 [243/250 31104/32000 (97%)] Loss: 1.52277 (QuantReg: 9.55389) QuantErr: 9.55389 batch_time=0.49711 
Train Epoch: 50 codebook_update_time=1.64989
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch50.pth ...
Done in 4.206s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/checkpoint-epoch50.pth ...
Done in 8.042s
removing stale ckpt [epoch 49] [took 0.00s]
 epoch          : 50
 loss           : 1.5200352306365967
 quant_reg      : 9.493198120117187
 quant_err      : 9.493198120117187
 learning_rate  : 4.04973554087964e-06
 n_samples      : 1600000
 n_steps        : 12500
 MSRVTT_full_val/t2v_metrics/R1: 33.40040241448692
 MSRVTT_full_val/t2v_metrics/R5: 67.40442655935614
 MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
 MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
 MSRVTT_full_val/t2v_metrics/MedR: 3.0
 MSRVTT_full_val/t2v_metrics/MeanR: 7.923541247484909
 MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.68088126763415
 MSRVTT_full_val/v2t_metrics/R1: 36.61971830985915
 MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
 MSRVTT_full_val/v2t_metrics/R10: 85.11066398390342
 MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
 MSRVTT_full_val/v2t_metrics/MedR: 2.0
 MSRVTT_full_val/v2t_metrics/MeanR: 7.084507042253521
 MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.11502460381455
 MSRVTT_full_test/t2v_metrics/R1: 13.578595317725753
 MSRVTT_full_test/t2v_metrics/R5: 36.55518394648829
 MSRVTT_full_test/t2v_metrics/R10: 49.063545150501675
 MSRVTT_full_test/t2v_metrics/R50: 80.43478260869566
 MSRVTT_full_test/t2v_metrics/MedR: 11.0
 MSRVTT_full_test/t2v_metrics/MeanR: 49.79598662207358
 MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.98595285267025
 MSRVTT_full_test/v2t_metrics/R1: 14.347826086956522
 MSRVTT_full_test/v2t_metrics/R5: 39.130434782608695
 MSRVTT_full_test/v2t_metrics/R10: 53.64548494983278
 MSRVTT_full_test/v2t_metrics/R50: 84.14715719063545
 MSRVTT_full_test/v2t_metrics/MedR: 9.0
 MSRVTT_full_test/v2t_metrics/MeanR: 39.69581939799331
 MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.113197870278498
 mnt_best       : 28.98595285267025
 not_improved_count: 0
Final evaluation ...
Loading checkpoint from: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/trained_model.pth ...
Ckpt loaded at epoch 50.
Saved similarity matrix (quantize videos) to /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/MSRVTT-test-qv-sims.npy
Saved v2t similarity matrix (quantize texts) to /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/MSRVTT-test-qt-sims.npy
MSRVTT_full_test:
 t2v_metrics/R1/final_eval: 13.578595317725753
 t2v_metrics/R5/final_eval: 36.55518394648829
 t2v_metrics/R10/final_eval: 49.063545150501675
 t2v_metrics/R50/final_eval: 80.43478260869566
 t2v_metrics/MedR/final_eval: 11.0
 t2v_metrics/MeanR/final_eval: 49.79598662207358
 t2v_metrics/geometric_mean_R1-R5-R10/final_eval: 28.98595285267025
 v2t_metrics/R1/final_eval: 14.347826086956522
 v2t_metrics/R5/final_eval: 39.130434782608695
 v2t_metrics/R10/final_eval: 53.64548494983278
 v2t_metrics/R50/final_eval: 84.14715719063545
 v2t_metrics/MedR/final_eval: 9.0
 v2t_metrics/MeanR/final_eval: 39.69565217391305
 v2t_metrics/geometric_mean_R1-R5-R10/final_eval: 31.113197870278498
Best epoch for the monitored metric: 50
Script took 04h12m58s
The best performing ckpt can be found at /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.1/trained_model.pth