-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full_t0.12.txt
3313 lines (3313 loc) · 233 KB
/
HCQ_MSRVTT_full_t0.12.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 1072.132703781128 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 50.190322399139404 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 366.5508248806 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 61.49386525154114 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch0.pth ...
Done in 1.670s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch0.pth ...
Done in 3.096s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
MSRVTT_full_val/t2v_metrics/MedR: 252.0
MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
MSRVTT_full_val/v2t_metrics/MedR: 243.0
MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
MSRVTT_full_test/t2v_metrics/MedR: 1515.0
MSRVTT_full_test/t2v_metrics/MeanR: 1498.5565217391304
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1471.5
MSRVTT_full_test/v2t_metrics/MeanR: 1495.3264214046824
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
mnt_best : 0.12154652794863813
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.72885 (QuantReg: 22.44570) QuantErr: 22.44570 batch_time=41.90310
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.80065 (QuantReg: 22.54847) QuantErr: 22.54847 batch_time=0.51393
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.26229 (QuantReg: 22.61779) QuantErr: 22.61779 batch_time=0.50631
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 7.14518 (QuantReg: 22.61533) QuantErr: 22.61533 batch_time=0.51276
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.63854 (QuantReg: 22.60823) QuantErr: 22.60823 batch_time=0.53270
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.36732 (QuantReg: 22.62352) QuantErr: 22.62352 batch_time=0.49190
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.30730 (QuantReg: 22.61246) QuantErr: 22.61246 batch_time=0.49865
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 6.04092 (QuantReg: 22.61831) QuantErr: 22.61831 batch_time=0.49554
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.76108 (QuantReg: 22.65146) QuantErr: 22.65146 batch_time=0.50425
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.85885 (QuantReg: 22.62796) QuantErr: 22.62796 batch_time=0.49862
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.56884 (QuantReg: 22.66053) QuantErr: 22.66053 batch_time=0.88913
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 5.28902 (QuantReg: 22.64396) QuantErr: 22.64396 batch_time=0.48869
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 5.29891 (QuantReg: 22.63265) QuantErr: 22.63265 batch_time=0.50761
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 5.19045 (QuantReg: 22.65735) QuantErr: 22.65735 batch_time=0.50199
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 5.21407 (QuantReg: 22.63473) QuantErr: 22.63473 batch_time=0.49038
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 5.36224 (QuantReg: 22.62296) QuantErr: 22.62296 batch_time=0.49774
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 5.20357 (QuantReg: 22.62401) QuantErr: 22.62401 batch_time=0.50316
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.92567 (QuantReg: 22.60976) QuantErr: 22.60976 batch_time=0.56567
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.96669 (QuantReg: 22.58120) QuantErr: 22.58120 batch_time=0.49954
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.74352 (QuantReg: 22.60469) QuantErr: 22.60469 batch_time=0.49476
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.90642 (QuantReg: 22.64165) QuantErr: 22.64165 batch_time=0.49095
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.80301 (QuantReg: 22.62801) QuantErr: 22.62801 batch_time=0.50725
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.52817 (QuantReg: 22.58497) QuantErr: 22.58497 batch_time=0.53663
Train Epoch: 1 codebook_update_time=1.96650
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch1.pth ...
Done in 4.088s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch1.pth ...
Done in 8.241s
epoch : 1
loss : 5.745349178314209
quant_reg : 22.61379605102539
quant_err : 22.61379605102539
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 16.096579476861166
MSRVTT_full_val/t2v_metrics/R5: 45.67404426559356
MSRVTT_full_val/t2v_metrics/R10: 65.19114688128772
MSRVTT_full_val/t2v_metrics/R50: 93.36016096579476
MSRVTT_full_val/t2v_metrics/MedR: 7.0
MSRVTT_full_val/t2v_metrics/MeanR: 15.826961770623743
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 36.324297956572124
MSRVTT_full_val/v2t_metrics/R1: 18.91348088531187
MSRVTT_full_val/v2t_metrics/R5: 51.7102615694165
MSRVTT_full_val/v2t_metrics/R10: 67.00201207243461
MSRVTT_full_val/v2t_metrics/R50: 94.36619718309859
MSRVTT_full_val/v2t_metrics/MedR: 5.0
MSRVTT_full_val/v2t_metrics/MeanR: 13.943661971830986
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 40.31611556385849
MSRVTT_full_test/t2v_metrics/R1: 5.585284280936455
MSRVTT_full_test/t2v_metrics/R5: 17.692307692307693
MSRVTT_full_test/t2v_metrics/R10: 28.160535117056856
MSRVTT_full_test/t2v_metrics/R50: 62.876254180602004
MSRVTT_full_test/t2v_metrics/MedR: 30.0
MSRVTT_full_test/t2v_metrics/MeanR: 82.2494983277592
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 14.065555538629564
MSRVTT_full_test/v2t_metrics/R1: 5.986622073578595
MSRVTT_full_test/v2t_metrics/R5: 19.464882943143813
MSRVTT_full_test/v2t_metrics/R10: 29.632107023411372
MSRVTT_full_test/v2t_metrics/R50: 65.91973244147157
MSRVTT_full_test/v2t_metrics/MedR: 26.0
MSRVTT_full_test/v2t_metrics/MeanR: 75.3923076923077
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 15.114671898694434
mnt_best : 14.065555538629564
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 4.42823 (QuantReg: 10.13707) QuantErr: 10.13707 batch_time=37.86120
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.51439 (QuantReg: 10.03518) QuantErr: 10.03518 batch_time=0.48860
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.43345 (QuantReg: 10.34135) QuantErr: 10.34135 batch_time=0.50624
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 4.33597 (QuantReg: 10.48574) QuantErr: 10.48574 batch_time=0.48766
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 4.39777 (QuantReg: 10.63858) QuantErr: 10.63858 batch_time=0.52402
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 4.30148 (QuantReg: 10.34891) QuantErr: 10.34891 batch_time=0.51884
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.65588 (QuantReg: 10.84698) QuantErr: 10.84698 batch_time=0.53789
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 4.23408 (QuantReg: 10.68344) QuantErr: 10.68344 batch_time=0.49405
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.68635 (QuantReg: 10.95356) QuantErr: 10.95356 batch_time=0.49778
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.21802 (QuantReg: 10.82187) QuantErr: 10.82187 batch_time=0.52688
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 4.19688 (QuantReg: 10.71955) QuantErr: 10.71955 batch_time=0.48711
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 4.09379 (QuantReg: 10.81696) QuantErr: 10.81696 batch_time=0.48616
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.86894 (QuantReg: 10.96618) QuantErr: 10.96618 batch_time=4.08531
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 4.07488 (QuantReg: 11.07791) QuantErr: 11.07791 batch_time=0.51400
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.94696 (QuantReg: 11.37475) QuantErr: 11.37475 batch_time=1.41296
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 4.03905 (QuantReg: 11.62150) QuantErr: 11.62150 batch_time=0.50572
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 4.10564 (QuantReg: 11.34908) QuantErr: 11.34908 batch_time=0.55678
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 4.14432 (QuantReg: 11.62245) QuantErr: 11.62245 batch_time=0.74021
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.72152 (QuantReg: 11.36531) QuantErr: 11.36531 batch_time=0.51642
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.92543 (QuantReg: 11.73821) QuantErr: 11.73821 batch_time=0.52920
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 3.68051 (QuantReg: 11.61061) QuantErr: 11.61061 batch_time=0.53746
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 4.42551 (QuantReg: 11.52544) QuantErr: 11.52544 batch_time=0.50459
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 4.18806 (QuantReg: 12.12099) QuantErr: 12.12099 batch_time=0.49746
Train Epoch: 2 codebook_update_time=1.69222
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch2.pth ...
Done in 4.335s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch2.pth ...
Done in 9.447s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 4.205911855697632
quant_reg : 10.98818658065796
quant_err : 10.98818658065796
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 17.706237424547282
MSRVTT_full_val/t2v_metrics/R5: 52.11267605633803
MSRVTT_full_val/t2v_metrics/R10: 68.41046277665995
MSRVTT_full_val/t2v_metrics/R50: 95.57344064386318
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 12.62374245472837
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 39.81658992875864
MSRVTT_full_val/v2t_metrics/R1: 23.34004024144869
MSRVTT_full_val/v2t_metrics/R5: 58.5513078470825
MSRVTT_full_val/v2t_metrics/R10: 75.45271629778672
MSRVTT_full_val/v2t_metrics/R50: 95.77464788732394
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 11.311871227364184
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 46.8926050403427
MSRVTT_full_test/t2v_metrics/R1: 6.8561872909699
MSRVTT_full_test/t2v_metrics/R5: 21.672240802675585
MSRVTT_full_test/t2v_metrics/R10: 33.24414715719063
MSRVTT_full_test/t2v_metrics/R50: 69.93311036789298
MSRVTT_full_test/t2v_metrics/MedR: 22.0
MSRVTT_full_test/t2v_metrics/MeanR: 66.57190635451505
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 17.030754768917923
MSRVTT_full_test/v2t_metrics/R1: 7.8595317725752505
MSRVTT_full_test/v2t_metrics/R5: 24.91638795986622
MSRVTT_full_test/v2t_metrics/R10: 37.69230769230769
MSRVTT_full_test/v2t_metrics/R50: 73.37792642140468
MSRVTT_full_test/v2t_metrics/MedR: 18.0
MSRVTT_full_test/v2t_metrics/MeanR: 58.08896321070234
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.47054742116143
mnt_best : 17.030754768917923
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.68878 (QuantReg: 9.00932) QuantErr: 9.00932 batch_time=37.90547
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.99709 (QuantReg: 9.32326) QuantErr: 9.32326 batch_time=1.57230
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 4.22183 (QuantReg: 9.53270) QuantErr: 9.53270 batch_time=0.49876
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 3.63387 (QuantReg: 8.66118) QuantErr: 8.66118 batch_time=0.49927
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.92758 (QuantReg: 9.20904) QuantErr: 9.20904 batch_time=0.54292
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 3.50306 (QuantReg: 9.26976) QuantErr: 9.26976 batch_time=0.52824
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 3.62468 (QuantReg: 9.26461) QuantErr: 9.26461 batch_time=0.49264
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.83491 (QuantReg: 9.15611) QuantErr: 9.15611 batch_time=0.55917
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.71609 (QuantReg: 9.06057) QuantErr: 9.06057 batch_time=0.50007
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.54888 (QuantReg: 9.22374) QuantErr: 9.22374 batch_time=0.50157
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.74488 (QuantReg: 9.33908) QuantErr: 9.33908 batch_time=0.52350
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 3.33279 (QuantReg: 9.37748) QuantErr: 9.37748 batch_time=0.49186
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.87366 (QuantReg: 9.71930) QuantErr: 9.71930 batch_time=0.50440
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.67538 (QuantReg: 9.76959) QuantErr: 9.76959 batch_time=0.49435
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.29164 (QuantReg: 9.55890) QuantErr: 9.55890 batch_time=0.50897
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 3.27991 (QuantReg: 9.97247) QuantErr: 9.97247 batch_time=0.50686
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 3.32245 (QuantReg: 9.59984) QuantErr: 9.59984 batch_time=0.50562
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.84129 (QuantReg: 9.57587) QuantErr: 9.57587 batch_time=0.50933
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 3.35186 (QuantReg: 9.52397) QuantErr: 9.52397 batch_time=0.49908
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.73876 (QuantReg: 9.70943) QuantErr: 9.70943 batch_time=0.50008
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 3.38270 (QuantReg: 9.85117) QuantErr: 9.85117 batch_time=0.51021
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 3.23603 (QuantReg: 9.80564) QuantErr: 9.80564 batch_time=1.05461
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 3.46898 (QuantReg: 9.96078) QuantErr: 9.96078 batch_time=0.50394
Train Epoch: 3 codebook_update_time=1.64500
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch3.pth ...
Done in 7.001s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch3.pth ...
Done in 11.517s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 3.6997866382598876
quant_reg : 9.498856716156006
quant_err : 9.498856716156006
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 23.74245472837022
MSRVTT_full_val/t2v_metrics/R5: 57.947686116700204
MSRVTT_full_val/t2v_metrics/R10: 72.83702213279678
MSRVTT_full_val/t2v_metrics/R50: 96.17706237424547
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.362173038229376
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.44845789058825
MSRVTT_full_val/v2t_metrics/R1: 27.364185110663986
MSRVTT_full_val/v2t_metrics/R5: 63.58148893360161
MSRVTT_full_val/v2t_metrics/R10: 78.06841046277665
MSRVTT_full_val/v2t_metrics/R50: 96.78068410462777
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.714285714285714
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 51.40391342350176
MSRVTT_full_test/t2v_metrics/R1: 8.160535117056856
MSRVTT_full_test/t2v_metrics/R5: 25.317725752508363
MSRVTT_full_test/t2v_metrics/R10: 37.65886287625418
MSRVTT_full_test/t2v_metrics/R50: 72.97658862876254
MSRVTT_full_test/t2v_metrics/MedR: 17.0
MSRVTT_full_test/t2v_metrics/MeanR: 58.067558528428094
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 19.815430428173375
MSRVTT_full_test/v2t_metrics/R1: 9.23076923076923
MSRVTT_full_test/v2t_metrics/R5: 28.42809364548495
MSRVTT_full_test/v2t_metrics/R10: 41.90635451505017
MSRVTT_full_test/v2t_metrics/R50: 77.05685618729098
MSRVTT_full_test/v2t_metrics/MedR: 15.0
MSRVTT_full_test/v2t_metrics/MeanR: 52.93913043478261
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.23763023750083
mnt_best : 19.815430428173375
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.94949 (QuantReg: 8.63872) QuantErr: 8.63872 batch_time=36.54790
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 3.57177 (QuantReg: 8.68550) QuantErr: 8.68550 batch_time=0.49238
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 3.22390 (QuantReg: 8.77107) QuantErr: 8.77107 batch_time=0.49906
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 3.40838 (QuantReg: 8.62546) QuantErr: 8.62546 batch_time=0.52039
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 3.52442 (QuantReg: 8.75300) QuantErr: 8.75300 batch_time=0.51478
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.71559 (QuantReg: 8.81952) QuantErr: 8.81952 batch_time=0.52137
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.63731 (QuantReg: 8.84395) QuantErr: 8.84395 batch_time=0.50568
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 3.51851 (QuantReg: 8.83222) QuantErr: 8.83222 batch_time=0.51245
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 3.32339 (QuantReg: 8.75166) QuantErr: 8.75166 batch_time=0.50096
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 3.30373 (QuantReg: 8.76719) QuantErr: 8.76719 batch_time=0.51722
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 3.45772 (QuantReg: 8.95716) QuantErr: 8.95716 batch_time=0.50596
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 3.49605 (QuantReg: 8.65892) QuantErr: 8.65892 batch_time=0.49731
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 3.31828 (QuantReg: 8.71518) QuantErr: 8.71518 batch_time=0.49803
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 3.49837 (QuantReg: 9.22557) QuantErr: 9.22557 batch_time=4.13184
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.47697 (QuantReg: 9.22894) QuantErr: 9.22894 batch_time=0.51915
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 3.68503 (QuantReg: 9.45893) QuantErr: 9.45893 batch_time=0.50779
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.61491 (QuantReg: 9.14130) QuantErr: 9.14130 batch_time=0.50802
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 3.48902 (QuantReg: 9.40144) QuantErr: 9.40144 batch_time=0.51286
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 3.40744 (QuantReg: 9.09771) QuantErr: 9.09771 batch_time=0.50198
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 3.22204 (QuantReg: 9.10086) QuantErr: 9.10086 batch_time=0.51048
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 3.21448 (QuantReg: 9.49034) QuantErr: 9.49034 batch_time=0.49585
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 3.46248 (QuantReg: 9.01759) QuantErr: 9.01759 batch_time=0.50678
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 3.28304 (QuantReg: 9.10120) QuantErr: 9.10120 batch_time=0.49695
Train Epoch: 4 codebook_update_time=1.70443
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch4.pth ...
Done in 4.905s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 3.413597466468811
quant_reg : 9.000101329803467
quant_err : 9.000101329803467
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 21.93158953722334
MSRVTT_full_val/t2v_metrics/R5: 56.13682092555332
MSRVTT_full_val/t2v_metrics/R10: 71.62977867203219
MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.327967806841047
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 44.51132382859376
MSRVTT_full_val/v2t_metrics/R1: 26.559356136820927
MSRVTT_full_val/v2t_metrics/R5: 63.78269617706238
MSRVTT_full_val/v2t_metrics/R10: 76.86116700201207
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.142857142857142
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 50.68457354228465
MSRVTT_full_test/t2v_metrics/R1: 7.190635451505017
MSRVTT_full_test/t2v_metrics/R5: 25.016722408026755
MSRVTT_full_test/t2v_metrics/R10: 36.72240802675585
MSRVTT_full_test/t2v_metrics/R50: 71.90635451505017
MSRVTT_full_test/t2v_metrics/MedR: 19.0
MSRVTT_full_test/t2v_metrics/MeanR: 61.86220735785953
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.763316787571974
MSRVTT_full_test/v2t_metrics/R1: 9.933110367892976
MSRVTT_full_test/v2t_metrics/R5: 28.82943143812709
MSRVTT_full_test/v2t_metrics/R10: 42.84280936454849
MSRVTT_full_test/v2t_metrics/R50: 77.22408026755853
MSRVTT_full_test/v2t_metrics/MedR: 15.0
MSRVTT_full_test/v2t_metrics/MeanR: 51.8943143812709
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.063918424044036
mnt_best : 19.815430428173375
not_improved_count: 1
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 3.43253 (QuantReg: 8.74314) QuantErr: 8.74314 batch_time=40.82144
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 3.44422 (QuantReg: 8.86721) QuantErr: 8.86721 batch_time=0.50524
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 3.56666 (QuantReg: 8.71335) QuantErr: 8.71335 batch_time=0.50006
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.89464 (QuantReg: 8.99664) QuantErr: 8.99664 batch_time=0.82335
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 3.27848 (QuantReg: 8.62022) QuantErr: 8.62022 batch_time=0.50899
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 3.33936 (QuantReg: 9.10406) QuantErr: 9.10406 batch_time=0.51261
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 3.19051 (QuantReg: 8.70761) QuantErr: 8.70761 batch_time=0.52499
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 3.32305 (QuantReg: 8.49810) QuantErr: 8.49810 batch_time=0.50262
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.99806 (QuantReg: 8.84956) QuantErr: 8.84956 batch_time=0.53637
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 3.37835 (QuantReg: 8.66274) QuantErr: 8.66274 batch_time=0.50686
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 3.21384 (QuantReg: 8.85074) QuantErr: 8.85074 batch_time=0.49094
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 3.17471 (QuantReg: 8.68341) QuantErr: 8.68341 batch_time=0.50912
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 3.20557 (QuantReg: 9.06376) QuantErr: 9.06376 batch_time=0.55197
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 2.74672 (QuantReg: 8.97294) QuantErr: 8.97294 batch_time=0.55444
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 3.20401 (QuantReg: 9.12493) QuantErr: 9.12493 batch_time=0.50233
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 3.13510 (QuantReg: 9.03919) QuantErr: 9.03919 batch_time=0.54085
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 3.09642 (QuantReg: 9.28384) QuantErr: 9.28384 batch_time=0.51052
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 3.26592 (QuantReg: 8.73754) QuantErr: 8.73754 batch_time=0.50164
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.99676 (QuantReg: 8.86870) QuantErr: 8.86870 batch_time=0.53244
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 3.25919 (QuantReg: 8.75532) QuantErr: 8.75532 batch_time=0.51902
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 3.20123 (QuantReg: 8.90953) QuantErr: 8.90953 batch_time=0.51291
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.82786 (QuantReg: 8.92601) QuantErr: 8.92601 batch_time=0.54189
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.03959 (QuantReg: 8.71608) QuantErr: 8.71608 batch_time=0.50398
Train Epoch: 5 codebook_update_time=1.65685
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch5.pth ...
Done in 4.643s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch5.pth ...
Done in 9.325s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 3.170547881126404
quant_reg : 8.805496494293212
quant_err : 8.805496494293212
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 26.358148893360163
MSRVTT_full_val/t2v_metrics/R5: 59.758551307847085
MSRVTT_full_val/t2v_metrics/R10: 75.85513078470825
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.474849094567404
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 49.25307044367255
MSRVTT_full_val/v2t_metrics/R1: 29.175050301810867
MSRVTT_full_val/v2t_metrics/R5: 64.98993963782696
MSRVTT_full_val/v2t_metrics/R10: 79.87927565392354
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.895372233400401
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 53.30450994802107
MSRVTT_full_test/t2v_metrics/R1: 9.063545150501673
MSRVTT_full_test/t2v_metrics/R5: 27.357859531772576
MSRVTT_full_test/t2v_metrics/R10: 40.0
MSRVTT_full_test/t2v_metrics/R50: 74.98327759197325
MSRVTT_full_test/t2v_metrics/MedR: 17.0
MSRVTT_full_test/t2v_metrics/MeanR: 54.55819397993311
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.485562908087655
MSRVTT_full_test/v2t_metrics/R1: 11.20401337792642
MSRVTT_full_test/v2t_metrics/R5: 31.337792642140467
MSRVTT_full_test/v2t_metrics/R10: 45.18394648829432
MSRVTT_full_test/v2t_metrics/R50: 79.76588628762542
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 47.94882943143813
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.127082209230114
mnt_best : 21.485562908087655
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 3.31503 (QuantReg: 8.28360) QuantErr: 8.28360 batch_time=40.11799
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 3.21019 (QuantReg: 8.05114) QuantErr: 8.05114 batch_time=0.57322
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.80760 (QuantReg: 8.45016) QuantErr: 8.45016 batch_time=0.51917
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.93085 (QuantReg: 8.53140) QuantErr: 8.53140 batch_time=0.49706
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 2.65074 (QuantReg: 8.44987) QuantErr: 8.44987 batch_time=0.52131
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.76866 (QuantReg: 8.75597) QuantErr: 8.75597 batch_time=0.53019
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.85682 (QuantReg: 8.91278) QuantErr: 8.91278 batch_time=0.49560
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 2.73101 (QuantReg: 8.49535) QuantErr: 8.49535 batch_time=0.51624
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 3.32742 (QuantReg: 8.77588) QuantErr: 8.77588 batch_time=0.49521
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.90812 (QuantReg: 8.60736) QuantErr: 8.60736 batch_time=0.49891
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.99106 (QuantReg: 8.11824) QuantErr: 8.11824 batch_time=0.49603
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.83778 (QuantReg: 8.69330) QuantErr: 8.69330 batch_time=0.52381
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.83509 (QuantReg: 8.60323) QuantErr: 8.60323 batch_time=0.50403
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 3.01724 (QuantReg: 8.33820) QuantErr: 8.33820 batch_time=3.40245
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 2.74247 (QuantReg: 8.83310) QuantErr: 8.83310 batch_time=0.50157
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.76306 (QuantReg: 8.58801) QuantErr: 8.58801 batch_time=0.48775
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 3.16142 (QuantReg: 8.94202) QuantErr: 8.94202 batch_time=0.95752
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.89222 (QuantReg: 9.16814) QuantErr: 9.16814 batch_time=0.50689
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.93823 (QuantReg: 8.55306) QuantErr: 8.55306 batch_time=0.49030
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.87368 (QuantReg: 8.81452) QuantErr: 8.81452 batch_time=0.49766
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.80878 (QuantReg: 8.62155) QuantErr: 8.62155 batch_time=1.22003
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 3.07463 (QuantReg: 8.70056) QuantErr: 8.70056 batch_time=0.90938
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 3.02184 (QuantReg: 8.73673) QuantErr: 8.73673 batch_time=0.51062
Train Epoch: 6 codebook_update_time=1.78495
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch6.pth ...
Done in 4.322s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch6.pth ...
Done in 15.311s
removing stale ckpt [epoch 5] [took 0.10s]
epoch : 6
loss : 2.993827298164368
quant_reg : 8.620305374145508
quant_err : 8.620305374145508
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 25.75452716297787
MSRVTT_full_val/t2v_metrics/R5: 61.77062374245473
MSRVTT_full_val/t2v_metrics/R10: 74.84909456740442
MSRVTT_full_val/t2v_metrics/R50: 96.37826961770624
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.181086519114688
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 49.19723626696388
MSRVTT_full_val/v2t_metrics/R1: 30.58350100603622
MSRVTT_full_val/v2t_metrics/R5: 65.59356136820925
MSRVTT_full_val/v2t_metrics/R10: 78.26961770623743
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.6841046277666
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 53.948644979924524
MSRVTT_full_test/t2v_metrics/R1: 10.0
MSRVTT_full_test/t2v_metrics/R5: 29.36454849498328
MSRVTT_full_test/t2v_metrics/R10: 41.20401337792642
MSRVTT_full_test/t2v_metrics/R50: 75.81939799331104
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 54.161204013377926
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.957307375236496
MSRVTT_full_test/v2t_metrics/R1: 11.070234113712374
MSRVTT_full_test/v2t_metrics/R5: 32.04013377926422
MSRVTT_full_test/v2t_metrics/R10: 45.852842809364546
MSRVTT_full_test/v2t_metrics/R50: 80.5685618729097
MSRVTT_full_test/v2t_metrics/MedR: 13.0
MSRVTT_full_test/v2t_metrics/MeanR: 46.38662207357859
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.336063036141834
mnt_best : 22.957307375236496
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.89788 (QuantReg: 7.95394) QuantErr: 7.95394 batch_time=31.83483
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.72664 (QuantReg: 8.78274) QuantErr: 8.78274 batch_time=0.53346
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.96576 (QuantReg: 8.75802) QuantErr: 8.75802 batch_time=2.84469
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.83202 (QuantReg: 8.57943) QuantErr: 8.57943 batch_time=1.13634
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.74901 (QuantReg: 8.15250) QuantErr: 8.15250 batch_time=0.50866
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.98635 (QuantReg: 8.23004) QuantErr: 8.23004 batch_time=0.49271
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.95213 (QuantReg: 8.61137) QuantErr: 8.61137 batch_time=1.20148
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 3.21553 (QuantReg: 8.55984) QuantErr: 8.55984 batch_time=0.50005
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.95393 (QuantReg: 8.60965) QuantErr: 8.60965 batch_time=0.51028
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.86588 (QuantReg: 8.64419) QuantErr: 8.64419 batch_time=0.49846
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 3.02767 (QuantReg: 8.57058) QuantErr: 8.57058 batch_time=0.50558
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 2.46697 (QuantReg: 8.88451) QuantErr: 8.88451 batch_time=0.50803
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 2.71746 (QuantReg: 8.47041) QuantErr: 8.47041 batch_time=0.49333
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.92989 (QuantReg: 8.55650) QuantErr: 8.55650 batch_time=0.49078
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.86575 (QuantReg: 8.20452) QuantErr: 8.20452 batch_time=0.49894
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.76897 (QuantReg: 8.40073) QuantErr: 8.40073 batch_time=1.21776
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 2.56974 (QuantReg: 8.64886) QuantErr: 8.64886 batch_time=0.50852
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.83508 (QuantReg: 8.20711) QuantErr: 8.20711 batch_time=0.50196
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 2.35994 (QuantReg: 8.68453) QuantErr: 8.68453 batch_time=0.52538
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.63585 (QuantReg: 8.56564) QuantErr: 8.56564 batch_time=1.10044
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 3.23311 (QuantReg: 8.62993) QuantErr: 8.62993 batch_time=0.49165
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.79244 (QuantReg: 8.36901) QuantErr: 8.36901 batch_time=0.49279
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.84946 (QuantReg: 8.48457) QuantErr: 8.48457 batch_time=0.51970
Train Epoch: 7 codebook_update_time=1.74308
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch7.pth ...
Done in 4.795s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.849834909439087
quant_reg : 8.562337953567505
quant_err : 8.562337953567505
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 27.766599597585515
MSRVTT_full_val/t2v_metrics/R5: 61.3682092555332
MSRVTT_full_val/t2v_metrics/R10: 75.0503018108652
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.661971830985916
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.3817053634723
MSRVTT_full_val/v2t_metrics/R1: 29.577464788732396
MSRVTT_full_val/v2t_metrics/R5: 67.80684104627767
MSRVTT_full_val/v2t_metrics/R10: 79.87927565392354
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.368209255533198
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.3112158579596
MSRVTT_full_test/t2v_metrics/R1: 8.695652173913043
MSRVTT_full_test/t2v_metrics/R5: 28.662207357859533
MSRVTT_full_test/t2v_metrics/R10: 40.903010033444815
MSRVTT_full_test/t2v_metrics/R50: 75.68561872909699
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 54.02608695652174
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.68314899137463
MSRVTT_full_test/v2t_metrics/R1: 11.705685618729097
MSRVTT_full_test/v2t_metrics/R5: 32.34113712374582
MSRVTT_full_test/v2t_metrics/R10: 47.9933110367893
MSRVTT_full_test/v2t_metrics/R50: 80.60200668896321
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 46.151672240802675
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.28921499858743
mnt_best : 22.957307375236496
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.80567 (QuantReg: 8.68004) QuantErr: 8.68004 batch_time=36.36306
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.94535 (QuantReg: 8.70411) QuantErr: 8.70411 batch_time=0.50313
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 3.09446 (QuantReg: 8.45464) QuantErr: 8.45464 batch_time=0.52369
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.77800 (QuantReg: 8.66173) QuantErr: 8.66173 batch_time=0.48584
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 3.11377 (QuantReg: 8.63901) QuantErr: 8.63901 batch_time=0.50894
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.95604 (QuantReg: 8.46328) QuantErr: 8.46328 batch_time=0.54693
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 2.68484 (QuantReg: 8.83052) QuantErr: 8.83052 batch_time=0.49167
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 2.63118 (QuantReg: 8.77955) QuantErr: 8.77955 batch_time=2.66374
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 3.06515 (QuantReg: 8.73549) QuantErr: 8.73549 batch_time=0.52484
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.64398 (QuantReg: 8.28878) QuantErr: 8.28878 batch_time=0.52282
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.74270 (QuantReg: 8.71348) QuantErr: 8.71348 batch_time=0.50267
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 3.05650 (QuantReg: 8.52277) QuantErr: 8.52277 batch_time=1.38361
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 2.64098 (QuantReg: 8.50592) QuantErr: 8.50592 batch_time=0.49030
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 2.49062 (QuantReg: 8.41309) QuantErr: 8.41309 batch_time=0.48818
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 3.01232 (QuantReg: 8.69260) QuantErr: 8.69260 batch_time=0.59636
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 2.61416 (QuantReg: 8.42805) QuantErr: 8.42805 batch_time=0.52507
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.89965 (QuantReg: 8.49700) QuantErr: 8.49700 batch_time=0.49385
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.99872 (QuantReg: 8.75843) QuantErr: 8.75843 batch_time=0.50730
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 2.34878 (QuantReg: 8.52160) QuantErr: 8.52160 batch_time=0.52408
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 2.41431 (QuantReg: 8.40542) QuantErr: 8.40542 batch_time=0.59466
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 2.41700 (QuantReg: 8.87407) QuantErr: 8.87407 batch_time=0.50061
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 2.12503 (QuantReg: 8.43999) QuantErr: 8.43999 batch_time=0.51078
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 2.29575 (QuantReg: 8.54765) QuantErr: 8.54765 batch_time=0.50360
Train Epoch: 8 codebook_update_time=1.76059
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch8.pth ...
Done in 11.701s
removing stale ckpt [epoch 7] [took 0.06s]
epoch : 8
loss : 2.761152189254761
quant_reg : 8.518430122375488
quant_err : 8.518430122375488
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 23.74245472837022
MSRVTT_full_val/t2v_metrics/R5: 61.3682092555332
MSRVTT_full_val/t2v_metrics/R10: 73.44064386317908
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.859154929577464
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 47.47538714809595
MSRVTT_full_val/v2t_metrics/R1: 28.973843058350102
MSRVTT_full_val/v2t_metrics/R5: 66.59959758551308
MSRVTT_full_val/v2t_metrics/R10: 79.07444668008048
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.318913480885312
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 53.436494073570955
MSRVTT_full_test/t2v_metrics/R1: 8.929765886287626
MSRVTT_full_test/t2v_metrics/R5: 29.39799331103679
MSRVTT_full_test/t2v_metrics/R10: 41.63879598662207
MSRVTT_full_test/t2v_metrics/R50: 75.65217391304348
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 52.739297658862874
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.19313428486773
MSRVTT_full_test/v2t_metrics/R1: 10.501672240802675
MSRVTT_full_test/v2t_metrics/R5: 32.30769230769231
MSRVTT_full_test/v2t_metrics/R10: 46.65551839464883
MSRVTT_full_test/v2t_metrics/R50: 80.73578595317726
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 43.75769230769231
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.108598513396124
mnt_best : 22.957307375236496
not_improved_count: 2
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.80035 (QuantReg: 8.04115) QuantErr: 8.04115 batch_time=39.47127
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 2.55357 (QuantReg: 8.16975) QuantErr: 8.16975 batch_time=0.95697
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.82430 (QuantReg: 8.31321) QuantErr: 8.31321 batch_time=0.77479
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.91637 (QuantReg: 8.37815) QuantErr: 8.37815 batch_time=0.49730
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 3.01461 (QuantReg: 8.21257) QuantErr: 8.21257 batch_time=0.49760
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.93659 (QuantReg: 8.63647) QuantErr: 8.63647 batch_time=0.49542
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.63062 (QuantReg: 8.49904) QuantErr: 8.49904 batch_time=0.48732
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 2.42460 (QuantReg: 8.45849) QuantErr: 8.45849 batch_time=0.52979
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.84755 (QuantReg: 8.27382) QuantErr: 8.27382 batch_time=0.49425
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 2.67829 (QuantReg: 8.44625) QuantErr: 8.44625 batch_time=0.52110
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.65476 (QuantReg: 8.34321) QuantErr: 8.34321 batch_time=0.51302
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 2.61293 (QuantReg: 8.55029) QuantErr: 8.55029 batch_time=0.56518
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.71509 (QuantReg: 8.58084) QuantErr: 8.58084 batch_time=0.48754
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.75491 (QuantReg: 8.56202) QuantErr: 8.56202 batch_time=1.21507
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.71948 (QuantReg: 8.87803) QuantErr: 8.87803 batch_time=0.49084
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 2.84826 (QuantReg: 8.33460) QuantErr: 8.33460 batch_time=0.48833
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 2.30022 (QuantReg: 8.52070) QuantErr: 8.52070 batch_time=0.51309
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 2.43517 (QuantReg: 8.36849) QuantErr: 8.36849 batch_time=0.51032
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 2.42498 (QuantReg: 8.74361) QuantErr: 8.74361 batch_time=0.69233
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 2.58857 (QuantReg: 8.30020) QuantErr: 8.30020 batch_time=0.48397
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 2.61504 (QuantReg: 8.42797) QuantErr: 8.42797 batch_time=0.49394
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 2.70220 (QuantReg: 8.48230) QuantErr: 8.48230 batch_time=0.50530
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 2.21096 (QuantReg: 8.46750) QuantErr: 8.46750 batch_time=0.52993
Train Epoch: 9 codebook_update_time=1.84452
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch9.pth ...
Done in 5.247s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch9.pth ...
Done in 10.279s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 2.6465283918380735
quant_reg : 8.450729997634888
quant_err : 8.450729997634888
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 27.96780684104628
MSRVTT_full_val/t2v_metrics/R5: 62.57545271629779
MSRVTT_full_val/t2v_metrics/R10: 75.45271629778672
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.736418511066399
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.92281815859953
MSRVTT_full_val/v2t_metrics/R1: 29.979879275653925
MSRVTT_full_val/v2t_metrics/R5: 67.80684104627767
MSRVTT_full_val/v2t_metrics/R10: 79.07444668008048
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.404426559356136
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.372568759875506
MSRVTT_full_test/t2v_metrics/R1: 9.732441471571907
MSRVTT_full_test/t2v_metrics/R5: 29.431438127090303
MSRVTT_full_test/t2v_metrics/R10: 42.30769230769231
MSRVTT_full_test/t2v_metrics/R50: 76.75585284280936
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 51.7489966555184
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.969464686654593
MSRVTT_full_test/v2t_metrics/R1: 11.036789297658864
MSRVTT_full_test/v2t_metrics/R5: 33.21070234113712
MSRVTT_full_test/v2t_metrics/R10: 47.491638795986624
MSRVTT_full_test/v2t_metrics/R50: 80.8695652173913
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 45.052675585284284
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.916677385261263
mnt_best : 22.969464686654593
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 2.34362 (QuantReg: 7.95790) QuantErr: 7.95790 batch_time=45.71053
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 2.52518 (QuantReg: 8.43314) QuantErr: 8.43314 batch_time=0.55126
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 2.36883 (QuantReg: 8.05186) QuantErr: 8.05186 batch_time=0.52713
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 2.41048 (QuantReg: 8.13811) QuantErr: 8.13811 batch_time=0.54194
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 2.45655 (QuantReg: 8.68238) QuantErr: 8.68238 batch_time=0.50607
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.86205 (QuantReg: 8.40415) QuantErr: 8.40415 batch_time=0.51352
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 2.33468 (QuantReg: 8.37192) QuantErr: 8.37192 batch_time=0.53918
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 2.46542 (QuantReg: 8.19285) QuantErr: 8.19285 batch_time=0.50881
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 2.59175 (QuantReg: 8.45004) QuantErr: 8.45004 batch_time=0.50309
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.97586 (QuantReg: 8.16409) QuantErr: 8.16409 batch_time=0.49664
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 2.30969 (QuantReg: 8.31900) QuantErr: 8.31900 batch_time=0.52926
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.66274 (QuantReg: 8.45964) QuantErr: 8.45964 batch_time=0.50358
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.69607 (QuantReg: 8.46297) QuantErr: 8.46297 batch_time=0.56507
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.93534 (QuantReg: 8.42823) QuantErr: 8.42823 batch_time=0.52192
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 2.28561 (QuantReg: 8.56437) QuantErr: 8.56437 batch_time=0.50705
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 2.22382 (QuantReg: 8.39795) QuantErr: 8.39795 batch_time=0.53508
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.44669 (QuantReg: 8.51436) QuantErr: 8.51436 batch_time=0.51363
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 2.34576 (QuantReg: 8.61557) QuantErr: 8.61557 batch_time=0.50592
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 2.43312 (QuantReg: 8.80298) QuantErr: 8.80298 batch_time=0.50247
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 2.32439 (QuantReg: 8.39297) QuantErr: 8.39297 batch_time=0.51196
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 2.31481 (QuantReg: 8.05573) QuantErr: 8.05573 batch_time=0.49569
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 3.00685 (QuantReg: 8.54558) QuantErr: 8.54558 batch_time=0.52047
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.62618 (QuantReg: 8.26210) QuantErr: 8.26210 batch_time=0.56183
Train Epoch: 10 codebook_update_time=1.71910
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch10.pth ...
Done in 4.540s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 2.5496216144561767
quant_reg : 8.395909038543701
quant_err : 8.395909038543701
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 26.559356136820927
MSRVTT_full_val/t2v_metrics/R5: 60.160965794768615
MSRVTT_full_val/t2v_metrics/R10: 76.25754527162978
MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.235412474849095
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 49.576027163597686
MSRVTT_full_val/v2t_metrics/R1: 31.388329979879277
MSRVTT_full_val/v2t_metrics/R5: 65.3923541247485
MSRVTT_full_val/v2t_metrics/R10: 79.67806841046277
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.515090543259557
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.68623104879141
MSRVTT_full_test/t2v_metrics/R1: 9.264214046822742
MSRVTT_full_test/t2v_metrics/R5: 28.963210702341136
MSRVTT_full_test/t2v_metrics/R10: 42.30769230769231
MSRVTT_full_test/t2v_metrics/R50: 76.2876254180602
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 52.99598662207358
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.474577726049763
MSRVTT_full_test/v2t_metrics/R1: 11.070234113712374
MSRVTT_full_test/v2t_metrics/R5: 32.675585284280935
MSRVTT_full_test/v2t_metrics/R10: 47.290969899665555
MSRVTT_full_test/v2t_metrics/R50: 81.1371237458194
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 45.9994983277592
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.76634400054173
mnt_best : 22.969464686654593
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 2.40435 (QuantReg: 8.34536) QuantErr: 8.34536 batch_time=37.14388
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 2.26512 (QuantReg: 8.46232) QuantErr: 8.46232 batch_time=0.50523
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 2.48361 (QuantReg: 8.04511) QuantErr: 8.04511 batch_time=0.50649
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 2.55206 (QuantReg: 8.47365) QuantErr: 8.47365 batch_time=0.93702
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 2.61865 (QuantReg: 8.72427) QuantErr: 8.72427 batch_time=0.50259
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 2.41728 (QuantReg: 8.31251) QuantErr: 8.31251 batch_time=0.56359
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 2.54850 (QuantReg: 8.06738) QuantErr: 8.06738 batch_time=0.52932
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 2.29276 (QuantReg: 8.45452) QuantErr: 8.45452 batch_time=0.49566
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 2.44797 (QuantReg: 8.19229) QuantErr: 8.19229 batch_time=0.51530
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.90980 (QuantReg: 8.21423) QuantErr: 8.21423 batch_time=0.50154
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.64593 (QuantReg: 8.10622) QuantErr: 8.10622 batch_time=0.50295
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.50654 (QuantReg: 8.63940) QuantErr: 8.63940 batch_time=0.51333
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.92842 (QuantReg: 8.13276) QuantErr: 8.13276 batch_time=0.50280
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.65718 (QuantReg: 8.31246) QuantErr: 8.31246 batch_time=0.50803
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 2.37344 (QuantReg: 7.99287) QuantErr: 7.99287 batch_time=0.51213
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 2.43538 (QuantReg: 8.02904) QuantErr: 8.02904 batch_time=0.97609
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 2.32829 (QuantReg: 8.15233) QuantErr: 8.15233 batch_time=0.51383
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 2.25557 (QuantReg: 8.30895) QuantErr: 8.30895 batch_time=0.49857
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 2.37821 (QuantReg: 8.68974) QuantErr: 8.68974 batch_time=0.51405
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 2.74183 (QuantReg: 8.54065) QuantErr: 8.54065 batch_time=0.49684
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 2.59616 (QuantReg: 8.69942) QuantErr: 8.69942 batch_time=0.51921
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 2.38087 (QuantReg: 8.49596) QuantErr: 8.49596 batch_time=0.51911
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 2.23865 (QuantReg: 8.73281) QuantErr: 8.73281 batch_time=0.52507
Train Epoch: 11 codebook_update_time=1.86263
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch11.pth ...
Done in 4.335s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch11.pth ...
Done in 8.262s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 2.4736286954879763
quant_reg : 8.3900371799469
quant_err : 8.3900371799469
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 27.96780684104628
MSRVTT_full_val/t2v_metrics/R5: 60.563380281690144
MSRVTT_full_val/t2v_metrics/R10: 77.46478873239437
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.154929577464788
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.81488570148408
MSRVTT_full_val/v2t_metrics/R1: 31.58953722334004
MSRVTT_full_val/v2t_metrics/R5: 66.80080482897384
MSRVTT_full_val/v2t_metrics/R10: 79.67806841046277
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.418511066398391
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 55.193497921313245
MSRVTT_full_test/t2v_metrics/R1: 10.0
MSRVTT_full_test/t2v_metrics/R5: 29.665551839464882
MSRVTT_full_test/t2v_metrics/R10: 43.24414715719063
MSRVTT_full_test/t2v_metrics/R50: 76.25418060200668
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 52.01939799331104
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.40956027672761
MSRVTT_full_test/v2t_metrics/R1: 11.538461538461538
MSRVTT_full_test/v2t_metrics/R5: 33.37792642140468
MSRVTT_full_test/v2t_metrics/R10: 48.62876254180602
MSRVTT_full_test/v2t_metrics/R50: 81.40468227424749
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 44.40886287625418
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.556254741206455
mnt_best : 23.40956027672761
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 2.45388 (QuantReg: 8.29667) QuantErr: 8.29667 batch_time=35.53156
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 2.62395 (QuantReg: 8.43632) QuantErr: 8.43632 batch_time=0.50977
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 2.24141 (QuantReg: 8.20150) QuantErr: 8.20150 batch_time=0.52380
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.34304 (QuantReg: 8.19348) QuantErr: 8.19348 batch_time=0.50636
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.14762 (QuantReg: 8.30533) QuantErr: 8.30533 batch_time=0.75435
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 2.45715 (QuantReg: 8.12950) QuantErr: 8.12950 batch_time=0.52046
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 2.15424 (QuantReg: 8.30710) QuantErr: 8.30710 batch_time=5.51192
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 2.35431 (QuantReg: 8.34807) QuantErr: 8.34807 batch_time=0.51813
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 2.24414 (QuantReg: 8.29351) QuantErr: 8.29351 batch_time=0.81656
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 2.70382 (QuantReg: 8.20411) QuantErr: 8.20411 batch_time=0.51788
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 2.10203 (QuantReg: 8.29691) QuantErr: 8.29691 batch_time=0.55286
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 2.59033 (QuantReg: 8.24210) QuantErr: 8.24210 batch_time=0.53808
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 2.15598 (QuantReg: 8.33104) QuantErr: 8.33104 batch_time=0.52642
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 2.18896 (QuantReg: 8.64405) QuantErr: 8.64405 batch_time=0.50654
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.45406 (QuantReg: 8.37643) QuantErr: 8.37643 batch_time=0.55036
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 2.07867 (QuantReg: 8.39933) QuantErr: 8.39933 batch_time=0.54908
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.18007 (QuantReg: 8.23509) QuantErr: 8.23509 batch_time=0.51813
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 2.73328 (QuantReg: 8.03338) QuantErr: 8.03338 batch_time=0.49323
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.69458 (QuantReg: 8.24255) QuantErr: 8.24255 batch_time=0.59723
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 2.59786 (QuantReg: 8.27096) QuantErr: 8.27096 batch_time=0.49242
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 2.40526 (QuantReg: 7.85552) QuantErr: 7.85552 batch_time=0.53348
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 2.30664 (QuantReg: 8.52204) QuantErr: 8.52204 batch_time=0.50609
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 2.56673 (QuantReg: 8.76486) QuantErr: 8.76486 batch_time=0.76609
Train Epoch: 12 codebook_update_time=1.63634
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch12.pth ...
Done in 5.048s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 2.4059512128829956
quant_reg : 8.329371377944947
quant_err : 8.329371377944947
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 27.56539235412475
MSRVTT_full_val/t2v_metrics/R5: 61.16700201207244
MSRVTT_full_val/t2v_metrics/R10: 75.65392354124748
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.269617706237424
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.3389643688051
MSRVTT_full_val/v2t_metrics/R1: 29.77867203219316
MSRVTT_full_val/v2t_metrics/R5: 67.6056338028169
MSRVTT_full_val/v2t_metrics/R10: 80.0804828973843
MSRVTT_full_val/v2t_metrics/R50: 96.579476861167
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.517102615694165
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 54.42581814677459
MSRVTT_full_test/t2v_metrics/R1: 9.866220735785953
MSRVTT_full_test/t2v_metrics/R5: 29.36454849498328
MSRVTT_full_test/t2v_metrics/R10: 42.84280936454849
MSRVTT_full_test/t2v_metrics/R50: 76.5551839464883
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 52.414715719063544
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.15353812345255
MSRVTT_full_test/v2t_metrics/R1: 10.936454849498327
MSRVTT_full_test/v2t_metrics/R5: 33.61204013377927
MSRVTT_full_test/v2t_metrics/R10: 48.22742474916388
MSRVTT_full_test/v2t_metrics/R50: 81.33779264214047
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 44.478595317725755
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.07485107206233
mnt_best : 23.40956027672761
not_improved_count: 1
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.19682 (QuantReg: 8.23326) QuantErr: 8.23326 batch_time=36.77957
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 2.87841 (QuantReg: 7.87070) QuantErr: 7.87070 batch_time=0.50201
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 2.57110 (QuantReg: 8.40661) QuantErr: 8.40661 batch_time=0.51085
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 2.20812 (QuantReg: 8.30681) QuantErr: 8.30681 batch_time=0.48960
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.44596 (QuantReg: 8.39647) QuantErr: 8.39647 batch_time=0.49357
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.30643 (QuantReg: 8.52170) QuantErr: 8.52170 batch_time=0.86570
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 2.36457 (QuantReg: 8.58799) QuantErr: 8.58799 batch_time=0.51762
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 2.66467 (QuantReg: 8.63814) QuantErr: 8.63814 batch_time=1.36658
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 2.10103 (QuantReg: 7.93274) QuantErr: 7.93274 batch_time=0.49294
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 2.47558 (QuantReg: 8.34057) QuantErr: 8.34057 batch_time=0.51029
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 2.45706 (QuantReg: 8.11498) QuantErr: 8.11498 batch_time=0.54783
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 2.09987 (QuantReg: 8.17969) QuantErr: 8.17969 batch_time=0.49793
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 2.25942 (QuantReg: 8.26654) QuantErr: 8.26654 batch_time=0.49511
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 2.32816 (QuantReg: 8.63454) QuantErr: 8.63454 batch_time=2.74454
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 2.47364 (QuantReg: 8.10776) QuantErr: 8.10776 batch_time=0.50447
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 2.18718 (QuantReg: 8.24839) QuantErr: 8.24839 batch_time=0.50135
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 2.07547 (QuantReg: 8.56392) QuantErr: 8.56392 batch_time=0.49255
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 2.59760 (QuantReg: 8.19182) QuantErr: 8.19182 batch_time=0.51118
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 2.09503 (QuantReg: 8.53197) QuantErr: 8.53197 batch_time=0.49273
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.70988 (QuantReg: 8.41025) QuantErr: 8.41025 batch_time=0.50129
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 2.64687 (QuantReg: 8.42998) QuantErr: 8.42998 batch_time=0.49455
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 2.27886 (QuantReg: 8.45477) QuantErr: 8.45477 batch_time=0.50505
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 2.24998 (QuantReg: 8.41097) QuantErr: 8.41097 batch_time=0.49513
Train Epoch: 13 codebook_update_time=1.88010
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch13.pth ...
Done in 4.113s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch13.pth ...
Done in 26.205s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 2.3409380464553835
quant_reg : 8.337442255020141
quant_err : 8.337442255020141
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 62.97786720321932
MSRVTT_full_val/t2v_metrics/R10: 76.25754527162978
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.432595573440643
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.0581917846743
MSRVTT_full_val/v2t_metrics/R1: 32.394366197183096
MSRVTT_full_val/v2t_metrics/R5: 68.41046277665995
MSRVTT_full_val/v2t_metrics/R10: 81.28772635814889
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.780684104627767
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.47709249715213
MSRVTT_full_test/t2v_metrics/R1: 9.966555183946488
MSRVTT_full_test/t2v_metrics/R5: 30.100334448160535
MSRVTT_full_test/t2v_metrics/R10: 42.97658862876254
MSRVTT_full_test/t2v_metrics/R50: 77.59197324414716
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 51.77759197324415
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.448556472532992
MSRVTT_full_test/v2t_metrics/R1: 11.137123745819398
MSRVTT_full_test/v2t_metrics/R5: 33.07692307692308
MSRVTT_full_test/v2t_metrics/R10: 49.163879598662206
MSRVTT_full_test/v2t_metrics/R50: 81.70568561872909
MSRVTT_full_test/v2t_metrics/MedR: 11.0
MSRVTT_full_test/v2t_metrics/MeanR: 44.2933110367893
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.261211958146387
mnt_best : 23.448556472532992
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 2.20810 (QuantReg: 8.40740) QuantErr: 8.40740 batch_time=45.10745
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.69733 (QuantReg: 8.52735) QuantErr: 8.52735 batch_time=0.50671
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 2.23480 (QuantReg: 8.46786) QuantErr: 8.46786 batch_time=0.52277
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.22840 (QuantReg: 8.30952) QuantErr: 8.30952 batch_time=0.54345
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.49872 (QuantReg: 8.25100) QuantErr: 8.25100 batch_time=0.59524
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.22117 (QuantReg: 8.04845) QuantErr: 8.04845 batch_time=0.53772
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.56784 (QuantReg: 8.29965) QuantErr: 8.29965 batch_time=0.56459
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 2.46421 (QuantReg: 7.80682) QuantErr: 7.80682 batch_time=0.52826
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.06409 (QuantReg: 7.96910) QuantErr: 7.96910 batch_time=0.55383
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 2.22004 (QuantReg: 8.01488) QuantErr: 8.01488 batch_time=0.50244
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.28248 (QuantReg: 8.33156) QuantErr: 8.33156 batch_time=0.55640
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 2.38675 (QuantReg: 8.07142) QuantErr: 8.07142 batch_time=0.50793
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 2.26099 (QuantReg: 8.72960) QuantErr: 8.72960 batch_time=0.53531
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 2.06610 (QuantReg: 8.44509) QuantErr: 8.44509 batch_time=1.53106
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 2.10701 (QuantReg: 8.30491) QuantErr: 8.30491 batch_time=0.50323
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 2.10505 (QuantReg: 8.29241) QuantErr: 8.29241 batch_time=0.52019
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 2.10090 (QuantReg: 8.14166) QuantErr: 8.14166 batch_time=0.49580
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 2.38889 (QuantReg: 8.63279) QuantErr: 8.63279 batch_time=0.54777
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 2.09940 (QuantReg: 8.00165) QuantErr: 8.00165 batch_time=0.52831
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 2.51046 (QuantReg: 8.06996) QuantErr: 8.06996 batch_time=2.75614
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.43833 (QuantReg: 8.46265) QuantErr: 8.46265 batch_time=0.51132
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.23987 (QuantReg: 8.48841) QuantErr: 8.48841 batch_time=0.50793
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.17521 (QuantReg: 8.33518) QuantErr: 8.33518 batch_time=0.52778
Train Epoch: 14 codebook_update_time=1.64188
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch14.pth ...
Done in 7.950s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 2.2927296481132506
quant_reg : 8.301300720214844
quant_err : 8.301300720214844
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 27.16297786720322
MSRVTT_full_val/t2v_metrics/R5: 62.17303822937626
MSRVTT_full_val/t2v_metrics/R10: 75.85513078470825
MSRVTT_full_val/t2v_metrics/R50: 95.97585513078471
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.267605633802816
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 50.410554818453015
MSRVTT_full_val/v2t_metrics/R1: 31.790744466800806
MSRVTT_full_val/v2t_metrics/R5: 67.40442655935614
MSRVTT_full_val/v2t_metrics/R10: 80.48289738430583
MSRVTT_full_val/v2t_metrics/R50: 96.98189134808852
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.283702213279678
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 55.66269629769511
MSRVTT_full_test/t2v_metrics/R1: 9.933110367892976
MSRVTT_full_test/t2v_metrics/R5: 29.096989966555185
MSRVTT_full_test/t2v_metrics/R10: 42.77591973244147
MSRVTT_full_test/t2v_metrics/R50: 75.95317725752508
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 56.580267558528426
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.12300255374094
MSRVTT_full_test/v2t_metrics/R1: 10.468227424749164
MSRVTT_full_test/v2t_metrics/R5: 33.37792642140468
MSRVTT_full_test/v2t_metrics/R10: 47.591973244147155
MSRVTT_full_test/v2t_metrics/R50: 80.8695652173913
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 46.11254180602007
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.524391930267537
mnt_best : 23.448556472532992
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.64414 (QuantReg: 7.98986) QuantErr: 7.98986 batch_time=41.78895
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.24169 (QuantReg: 8.10284) QuantErr: 8.10284 batch_time=0.49005
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.30993 (QuantReg: 8.07542) QuantErr: 8.07542 batch_time=0.60356
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.37312 (QuantReg: 8.00307) QuantErr: 8.00307 batch_time=0.51035
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.37297 (QuantReg: 8.52014) QuantErr: 8.52014 batch_time=0.55553
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 2.31369 (QuantReg: 7.87999) QuantErr: 7.87999 batch_time=0.50707
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.60621 (QuantReg: 8.08046) QuantErr: 8.08046 batch_time=0.77056
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.17252 (QuantReg: 8.19789) QuantErr: 8.19789 batch_time=1.20455
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.42175 (QuantReg: 8.27815) QuantErr: 8.27815 batch_time=0.50794
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 2.15627 (QuantReg: 8.34803) QuantErr: 8.34803 batch_time=0.50873
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.31701 (QuantReg: 8.13134) QuantErr: 8.13134 batch_time=0.50200
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.98566 (QuantReg: 8.04528) QuantErr: 8.04528 batch_time=0.50700
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.10056 (QuantReg: 8.02490) QuantErr: 8.02490 batch_time=0.49193
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 2.67935 (QuantReg: 8.43626) QuantErr: 8.43626 batch_time=2.14953
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.28326 (QuantReg: 8.03010) QuantErr: 8.03010 batch_time=0.50906
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.38019 (QuantReg: 8.45613) QuantErr: 8.45613 batch_time=0.50277
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 2.66768 (QuantReg: 8.45640) QuantErr: 8.45640 batch_time=0.49377
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 2.15048 (QuantReg: 8.21418) QuantErr: 8.21418 batch_time=0.49305
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 2.08096 (QuantReg: 8.38048) QuantErr: 8.38048 batch_time=0.85124
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.40818 (QuantReg: 8.06625) QuantErr: 8.06625 batch_time=0.50681
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.56884 (QuantReg: 8.42414) QuantErr: 8.42414 batch_time=0.51725
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.41879 (QuantReg: 8.56157) QuantErr: 8.56157 batch_time=0.50121
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 2.44882 (QuantReg: 8.05519) QuantErr: 8.05519 batch_time=0.57392
Train Epoch: 15 codebook_update_time=1.70485
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch15.pth ...
Done in 3.914s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.12/checkpoint-epoch15.pth ...
Done in 15.800s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 2.250026999950409
quant_reg : 8.265110805511474
quant_err : 8.265110805511474
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
MSRVTT_full_val/t2v_metrics/R1: 28.571428571428573
MSRVTT_full_val/t2v_metrics/R5: 65.19114688128772