-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full_t0.07.txt
3311 lines (3311 loc) · 235 KB
/
HCQ_MSRVTT_full_t0.07.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 1020.7898154258728 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 66.1305000782013 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 354.94786858558655 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 100.95828795433044 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch0.pth ...
Done in 1.648s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch0.pth ...
Done in 3.185s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
MSRVTT_full_val/t2v_metrics/MedR: 252.0
MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
MSRVTT_full_val/v2t_metrics/MedR: 243.0
MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
MSRVTT_full_test/t2v_metrics/MedR: 1515.0
MSRVTT_full_test/t2v_metrics/MeanR: 1498.5565217391304
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1471.5
MSRVTT_full_test/v2t_metrics/MeanR: 1495.3264214046824
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
mnt_best : 0.12154652794863813
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.77391 (QuantReg: 22.44805) QuantErr: 22.44805 batch_time=47.01011
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.58264 (QuantReg: 22.55249) QuantErr: 22.55249 batch_time=0.57426
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.11818 (QuantReg: 22.59467) QuantErr: 22.59467 batch_time=0.52202
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.95232 (QuantReg: 22.61903) QuantErr: 22.61903 batch_time=0.51765
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.37375 (QuantReg: 22.62500) QuantErr: 22.62500 batch_time=0.52267
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.12799 (QuantReg: 22.66029) QuantErr: 22.66029 batch_time=0.51865
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.07767 (QuantReg: 22.62787) QuantErr: 22.62787 batch_time=0.52308
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.79228 (QuantReg: 22.61881) QuantErr: 22.61881 batch_time=0.57342
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.43413 (QuantReg: 22.65380) QuantErr: 22.65380 batch_time=0.55716
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.68314 (QuantReg: 22.64840) QuantErr: 22.64840 batch_time=0.55230
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.19744 (QuantReg: 22.65308) QuantErr: 22.65308 batch_time=0.53924
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.93246 (QuantReg: 22.62785) QuantErr: 22.62785 batch_time=0.52654
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.95023 (QuantReg: 22.60018) QuantErr: 22.60018 batch_time=0.52566
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.78616 (QuantReg: 22.64834) QuantErr: 22.64834 batch_time=0.52936
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.82764 (QuantReg: 22.64358) QuantErr: 22.64358 batch_time=0.56887
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 5.11477 (QuantReg: 22.62901) QuantErr: 22.62901 batch_time=0.61115
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.82254 (QuantReg: 22.62661) QuantErr: 22.62661 batch_time=0.49754
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.53074 (QuantReg: 22.61680) QuantErr: 22.61680 batch_time=0.53478
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.60178 (QuantReg: 22.57908) QuantErr: 22.57908 batch_time=0.63469
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.24386 (QuantReg: 22.59327) QuantErr: 22.59327 batch_time=0.56167
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.51081 (QuantReg: 22.60149) QuantErr: 22.60149 batch_time=0.55768
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.32607 (QuantReg: 22.63106) QuantErr: 22.63106 batch_time=0.50867
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 4.11226 (QuantReg: 22.57971) QuantErr: 22.57971 batch_time=0.54619
Train Epoch: 1 codebook_update_time=1.93654
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch1.pth ...
Done in 4.736s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch1.pth ...
Done in 9.889s
epoch : 1
loss : 5.412819831848145
quant_reg : 22.618492904663086
quant_err : 22.618492904663086
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 17.10261569416499
MSRVTT_full_val/t2v_metrics/R5: 51.7102615694165
MSRVTT_full_val/t2v_metrics/R10: 66.39839034205231
MSRVTT_full_val/t2v_metrics/R50: 95.37223340040241
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 13.89738430583501
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 38.8686038266997
MSRVTT_full_val/v2t_metrics/R1: 20.120724346076457
MSRVTT_full_val/v2t_metrics/R5: 52.91750503018109
MSRVTT_full_val/v2t_metrics/R10: 69.81891348088531
MSRVTT_full_val/v2t_metrics/R50: 93.96378269617706
MSRVTT_full_val/v2t_metrics/MedR: 5.0
MSRVTT_full_val/v2t_metrics/MeanR: 12.798792756539235
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 42.04735539789754
MSRVTT_full_test/t2v_metrics/R1: 5.852842809364549
MSRVTT_full_test/t2v_metrics/R5: 19.531772575250837
MSRVTT_full_test/t2v_metrics/R10: 31.103678929765888
MSRVTT_full_test/t2v_metrics/R50: 65.58528428093645
MSRVTT_full_test/t2v_metrics/MedR: 25.0
MSRVTT_full_test/t2v_metrics/MeanR: 74.11371237458194
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 15.263006638949394
MSRVTT_full_test/v2t_metrics/R1: 6.354515050167224
MSRVTT_full_test/v2t_metrics/R5: 22.107023411371237
MSRVTT_full_test/v2t_metrics/R10: 33.84615384615385
MSRVTT_full_test/v2t_metrics/R50: 69.66555183946488
MSRVTT_full_test/v2t_metrics/MedR: 22.0
MSRVTT_full_test/v2t_metrics/MeanR: 67.39498327759198
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.815405162827243
mnt_best : 15.263006638949394
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 3.88770 (QuantReg: 12.20144) QuantErr: 12.20144 batch_time=37.17690
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 4.04224 (QuantReg: 12.18251) QuantErr: 12.18251 batch_time=0.54047
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 4.00097 (QuantReg: 12.61269) QuantErr: 12.61269 batch_time=0.51413
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.92518 (QuantReg: 12.70034) QuantErr: 12.70034 batch_time=0.80676
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.84010 (QuantReg: 12.84592) QuantErr: 12.84592 batch_time=0.51040
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.82373 (QuantReg: 12.56346) QuantErr: 12.56346 batch_time=0.51869
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 4.06285 (QuantReg: 13.02727) QuantErr: 13.02727 batch_time=0.51798
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.60486 (QuantReg: 13.12265) QuantErr: 13.12265 batch_time=0.52420
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.26026 (QuantReg: 13.39605) QuantErr: 13.39605 batch_time=0.50165
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.68304 (QuantReg: 13.30996) QuantErr: 13.30996 batch_time=0.57867
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.73385 (QuantReg: 13.30649) QuantErr: 13.30649 batch_time=0.59075
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.50423 (QuantReg: 13.67502) QuantErr: 13.67502 batch_time=0.49957
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.37078 (QuantReg: 13.50175) QuantErr: 13.50175 batch_time=0.59874
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.50311 (QuantReg: 13.73877) QuantErr: 13.73877 batch_time=0.92752
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.43155 (QuantReg: 13.62914) QuantErr: 13.62914 batch_time=0.51909
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.36669 (QuantReg: 14.21064) QuantErr: 14.21064 batch_time=0.53329
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.52532 (QuantReg: 13.93039) QuantErr: 13.93039 batch_time=0.51698
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.60874 (QuantReg: 14.14732) QuantErr: 14.14732 batch_time=0.61615
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.29943 (QuantReg: 13.94801) QuantErr: 13.94801 batch_time=0.51125
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 3.26050 (QuantReg: 14.08821) QuantErr: 14.08821 batch_time=0.50686
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 2.96536 (QuantReg: 14.25895) QuantErr: 14.25895 batch_time=0.54116
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.96756 (QuantReg: 14.25811) QuantErr: 14.25811 batch_time=0.56252
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.56164 (QuantReg: 14.60070) QuantErr: 14.60070 batch_time=0.52196
Train Epoch: 2 codebook_update_time=1.71778
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch2.pth ...
Done in 22.146s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch2.pth ...
Done in 26.949s
removing stale ckpt [epoch 1] [took 0.00s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.6862801761627195
quant_reg : 13.442541538238526
quant_err : 13.442541538238526
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 23.34004024144869
MSRVTT_full_val/t2v_metrics/R5: 59.356136820925556
MSRVTT_full_val/t2v_metrics/R10: 72.83702213279678
MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 11.26559356136821
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.55572953691834
MSRVTT_full_val/v2t_metrics/R1: 26.961770623742456
MSRVTT_full_val/v2t_metrics/R5: 63.38028169014085
MSRVTT_full_val/v2t_metrics/R10: 80.6841046277666
MSRVTT_full_val/v2t_metrics/R50: 96.78068410462777
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 9.619718309859154
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 51.661084574415966
MSRVTT_full_test/t2v_metrics/R1: 8.193979933110368
MSRVTT_full_test/t2v_metrics/R5: 26.287625418060202
MSRVTT_full_test/t2v_metrics/R10: 37.85953177257525
MSRVTT_full_test/t2v_metrics/R50: 71.60535117056857
MSRVTT_full_test/t2v_metrics/MedR: 18.0
MSRVTT_full_test/t2v_metrics/MeanR: 60.189632107023414
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.128303123206496
MSRVTT_full_test/v2t_metrics/R1: 9.531772575250836
MSRVTT_full_test/v2t_metrics/R5: 29.899665551839465
MSRVTT_full_test/v2t_metrics/R10: 42.77591973244147
MSRVTT_full_test/v2t_metrics/R50: 77.82608695652173
MSRVTT_full_test/v2t_metrics/MedR: 15.0
MSRVTT_full_test/v2t_metrics/MeanR: 48.60702341137124
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.015113418379524
mnt_best : 20.128303123206496
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 3.13781 (QuantReg: 11.65062) QuantErr: 11.65062 batch_time=33.49385
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.44036 (QuantReg: 11.78328) QuantErr: 11.78328 batch_time=0.56207
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.58301 (QuantReg: 12.14386) QuantErr: 12.14386 batch_time=0.50526
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.87982 (QuantReg: 11.41515) QuantErr: 11.41515 batch_time=0.51140
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.28991 (QuantReg: 11.79025) QuantErr: 11.79025 batch_time=0.56818
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.86980 (QuantReg: 12.08874) QuantErr: 12.08874 batch_time=0.50715
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.94241 (QuantReg: 12.07496) QuantErr: 12.07496 batch_time=5.03373
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.26625 (QuantReg: 11.93898) QuantErr: 11.93898 batch_time=0.53173
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.20390 (QuantReg: 11.84624) QuantErr: 11.84624 batch_time=0.51965
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 3.04278 (QuantReg: 11.79964) QuantErr: 11.79964 batch_time=0.54556
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 3.14330 (QuantReg: 11.81054) QuantErr: 11.81054 batch_time=0.51118
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 2.74002 (QuantReg: 12.04723) QuantErr: 12.04723 batch_time=0.55221
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.24083 (QuantReg: 12.24439) QuantErr: 12.24439 batch_time=0.51721
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 3.01740 (QuantReg: 12.40362) QuantErr: 12.40362 batch_time=0.51486
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.75673 (QuantReg: 12.04416) QuantErr: 12.04416 batch_time=0.69472
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.52386 (QuantReg: 12.66013) QuantErr: 12.66013 batch_time=0.52123
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.68481 (QuantReg: 12.23236) QuantErr: 12.23236 batch_time=0.55950
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.24790 (QuantReg: 12.17422) QuantErr: 12.17422 batch_time=0.56592
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.80571 (QuantReg: 12.17754) QuantErr: 12.17754 batch_time=0.52969
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 3.05248 (QuantReg: 12.53262) QuantErr: 12.53262 batch_time=0.56546
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 2.74809 (QuantReg: 12.57468) QuantErr: 12.57468 batch_time=0.52147
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.52702 (QuantReg: 12.72495) QuantErr: 12.72495 batch_time=0.54532
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.81293 (QuantReg: 12.58868) QuantErr: 12.58868 batch_time=0.55076
Train Epoch: 3 codebook_update_time=1.84101
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch3.pth ...
Done in 5.628s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch3.pth ...
Done in 18.520s
removing stale ckpt [epoch 2] [took 0.20s]
epoch : 3
loss : 3.099186414718628
quant_reg : 12.171798236846923
quant_err : 12.171798236846923
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 23.943661971830984
MSRVTT_full_val/t2v_metrics/R5: 61.3682092555332
MSRVTT_full_val/t2v_metrics/R10: 75.45271629778672
MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.651911468812877
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.03999653677077
MSRVTT_full_val/v2t_metrics/R1: 27.96780684104628
MSRVTT_full_val/v2t_metrics/R5: 66.19718309859155
MSRVTT_full_val/v2t_metrics/R10: 80.6841046277666
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.935613682092555
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 53.05935523203142
MSRVTT_full_test/t2v_metrics/R1: 8.896321070234114
MSRVTT_full_test/t2v_metrics/R5: 27.692307692307693
MSRVTT_full_test/t2v_metrics/R10: 40.53511705685619
MSRVTT_full_test/t2v_metrics/R50: 74.71571906354515
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 56.203678929765886
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.534444641678324
MSRVTT_full_test/v2t_metrics/R1: 10.334448160535118
MSRVTT_full_test/v2t_metrics/R5: 31.471571906354516
MSRVTT_full_test/v2t_metrics/R10: 46.15384615384615
MSRVTT_full_test/v2t_metrics/R50: 80.16722408026756
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 46.300334448160534
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.668223510004257
mnt_best : 21.534444641678324
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.35870 (QuantReg: 11.40896) QuantErr: 11.40896 batch_time=41.38342
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.92728 (QuantReg: 11.45858) QuantErr: 11.45858 batch_time=0.75363
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.44317 (QuantReg: 11.50343) QuantErr: 11.50343 batch_time=0.52056
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.70272 (QuantReg: 11.45753) QuantErr: 11.45753 batch_time=0.55409
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.83132 (QuantReg: 11.53209) QuantErr: 11.53209 batch_time=0.55282
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.14459 (QuantReg: 11.58107) QuantErr: 11.58107 batch_time=0.53815
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.17282 (QuantReg: 11.62832) QuantErr: 11.62832 batch_time=0.56101
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.99388 (QuantReg: 11.69040) QuantErr: 11.69040 batch_time=0.49911
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.70363 (QuantReg: 11.56804) QuantErr: 11.56804 batch_time=0.57363
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.66612 (QuantReg: 11.49324) QuantErr: 11.49324 batch_time=0.51852
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.87745 (QuantReg: 11.86169) QuantErr: 11.86169 batch_time=0.52468
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.77251 (QuantReg: 11.65112) QuantErr: 11.65112 batch_time=0.53124
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.65643 (QuantReg: 11.66679) QuantErr: 11.66679 batch_time=0.57443
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.83904 (QuantReg: 11.78616) QuantErr: 11.78616 batch_time=0.52772
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 3.01342 (QuantReg: 12.01978) QuantErr: 12.01978 batch_time=0.51490
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 3.20527 (QuantReg: 12.32074) QuantErr: 12.32074 batch_time=0.52205
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 3.06108 (QuantReg: 11.85444) QuantErr: 11.85444 batch_time=0.60050
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.91079 (QuantReg: 12.22481) QuantErr: 12.22481 batch_time=0.51295
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.85279 (QuantReg: 11.88989) QuantErr: 11.88989 batch_time=0.53867
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.52997 (QuantReg: 11.95353) QuantErr: 11.95353 batch_time=0.61798
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.45242 (QuantReg: 12.22916) QuantErr: 12.22916 batch_time=0.56432
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.85980 (QuantReg: 11.60314) QuantErr: 11.60314 batch_time=0.55492
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.65181 (QuantReg: 11.76749) QuantErr: 11.76749 batch_time=0.54694
Train Epoch: 4 codebook_update_time=1.76299
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch4.pth ...
Done in 19.853s
removing stale ckpt [epoch 3] [took 0.00s]
epoch : 4
loss : 2.7865691423416137
quant_reg : 11.793484714508057
quant_err : 11.793484714508057
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 24.949698189134807
MSRVTT_full_val/t2v_metrics/R5: 61.16700201207244
MSRVTT_full_val/t2v_metrics/R10: 73.8430583501006
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.193158953722333
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.301882332188484
MSRVTT_full_val/v2t_metrics/R1: 28.973843058350102
MSRVTT_full_val/v2t_metrics/R5: 69.21529175050301
MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
MSRVTT_full_val/v2t_metrics/R50: 96.78068410462777
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.175050301810865
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 55.163062165704936
MSRVTT_full_test/t2v_metrics/R1: 9.163879598662207
MSRVTT_full_test/t2v_metrics/R5: 26.956521739130434
MSRVTT_full_test/t2v_metrics/R10: 39.86622073578595
MSRVTT_full_test/t2v_metrics/R50: 74.24749163879599
MSRVTT_full_test/t2v_metrics/MedR: 17.0
MSRVTT_full_test/t2v_metrics/MeanR: 55.63846153846154
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.43463528786568
MSRVTT_full_test/v2t_metrics/R1: 10.668896321070234
MSRVTT_full_test/v2t_metrics/R5: 32.74247491638796
MSRVTT_full_test/v2t_metrics/R10: 47.625418060200666
MSRVTT_full_test/v2t_metrics/R50: 80.96989966555184
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 43.916053511705684
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.52838061825068
mnt_best : 21.534444641678324
not_improved_count: 1
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.70300 (QuantReg: 11.49697) QuantErr: 11.49697 batch_time=43.69156
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.85977 (QuantReg: 11.61374) QuantErr: 11.61374 batch_time=0.54188
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 3.02633 (QuantReg: 11.47749) QuantErr: 11.47749 batch_time=0.50933
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.27905 (QuantReg: 11.70121) QuantErr: 11.70121 batch_time=0.52437
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.49064 (QuantReg: 11.30599) QuantErr: 11.30599 batch_time=0.50953
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.63152 (QuantReg: 12.04444) QuantErr: 12.04444 batch_time=0.55109
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.51641 (QuantReg: 11.48363) QuantErr: 11.48363 batch_time=0.52173
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.85428 (QuantReg: 11.38153) QuantErr: 11.38153 batch_time=0.58248
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.35424 (QuantReg: 11.63548) QuantErr: 11.63548 batch_time=0.59841
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.76929 (QuantReg: 11.70386) QuantErr: 11.70386 batch_time=0.52125
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.52469 (QuantReg: 11.78701) QuantErr: 11.78701 batch_time=0.52129
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.58646 (QuantReg: 11.73017) QuantErr: 11.73017 batch_time=0.56573
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.57594 (QuantReg: 11.82214) QuantErr: 11.82214 batch_time=0.51905
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 1.94951 (QuantReg: 11.86550) QuantErr: 11.86550 batch_time=0.51793
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.48398 (QuantReg: 11.92948) QuantErr: 11.92948 batch_time=0.58810
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.56561 (QuantReg: 11.93442) QuantErr: 11.93442 batch_time=0.52003
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.42811 (QuantReg: 12.21174) QuantErr: 12.21174 batch_time=0.51306
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.76697 (QuantReg: 11.71444) QuantErr: 11.71444 batch_time=0.52858
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.39045 (QuantReg: 11.81081) QuantErr: 11.81081 batch_time=0.58086
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.55335 (QuantReg: 11.58211) QuantErr: 11.58211 batch_time=0.55310
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.62877 (QuantReg: 11.98074) QuantErr: 11.98074 batch_time=1.40256
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 2.06765 (QuantReg: 11.95234) QuantErr: 11.95234 batch_time=0.50256
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.32088 (QuantReg: 11.72560) QuantErr: 11.72560 batch_time=0.59435
Train Epoch: 5 codebook_update_time=1.79114
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch5.pth ...
Done in 5.365s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch5.pth ...
Done in 10.695s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.516263931274414
quant_reg : 11.703462162017821
quant_err : 11.703462162017821
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 63.78269617706238
MSRVTT_full_val/t2v_metrics/R10: 77.66599597585513
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.780684104627767
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.59891093693319
MSRVTT_full_val/v2t_metrics/R1: 30.18108651911469
MSRVTT_full_val/v2t_metrics/R5: 71.0261569416499
MSRVTT_full_val/v2t_metrics/R10: 84.70824949698189
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.762575452716298
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.627357781521646
MSRVTT_full_test/t2v_metrics/R1: 10.167224080267559
MSRVTT_full_test/t2v_metrics/R5: 29.464882943143813
MSRVTT_full_test/t2v_metrics/R10: 42.441471571906355
MSRVTT_full_test/t2v_metrics/R50: 77.29096989966555
MSRVTT_full_test/t2v_metrics/MedR: 15.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.92943143812709
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.339910322381222
MSRVTT_full_test/v2t_metrics/R1: 11.939799331103679
MSRVTT_full_test/v2t_metrics/R5: 34.71571906354515
MSRVTT_full_test/v2t_metrics/R10: 50.13377926421405
MSRVTT_full_test/v2t_metrics/R50: 83.1438127090301
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 38.7376254180602
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.49273077126585
mnt_best : 23.339910322381222
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.59335 (QuantReg: 11.24922) QuantErr: 11.24922 batch_time=40.28222
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.54465 (QuantReg: 11.12907) QuantErr: 11.12907 batch_time=0.50393
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 2.11548 (QuantReg: 11.45550) QuantErr: 11.45550 batch_time=0.54189
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.38774 (QuantReg: 11.46848) QuantErr: 11.46848 batch_time=0.51004
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 1.94801 (QuantReg: 11.43239) QuantErr: 11.43239 batch_time=0.54848
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.19195 (QuantReg: 11.68954) QuantErr: 11.68954 batch_time=0.51024
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.21751 (QuantReg: 11.96029) QuantErr: 11.96029 batch_time=0.82264
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 1.99474 (QuantReg: 11.60527) QuantErr: 11.60527 batch_time=0.55292
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.62937 (QuantReg: 11.65399) QuantErr: 11.65399 batch_time=0.58037
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.22124 (QuantReg: 11.60020) QuantErr: 11.60020 batch_time=0.52090
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.30212 (QuantReg: 11.20187) QuantErr: 11.20187 batch_time=0.50994
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.17442 (QuantReg: 11.78745) QuantErr: 11.78745 batch_time=0.51065
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.28933 (QuantReg: 11.55613) QuantErr: 11.55613 batch_time=0.50987
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.40699 (QuantReg: 11.40170) QuantErr: 11.40170 batch_time=1.14983
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 1.94824 (QuantReg: 11.82324) QuantErr: 11.82324 batch_time=0.55997
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 2.08924 (QuantReg: 11.66867) QuantErr: 11.66867 batch_time=0.61838
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.51469 (QuantReg: 11.95981) QuantErr: 11.95981 batch_time=0.56255
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 2.27581 (QuantReg: 12.19253) QuantErr: 12.19253 batch_time=0.52008
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.28193 (QuantReg: 11.71165) QuantErr: 11.71165 batch_time=0.56838
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 2.13461 (QuantReg: 11.67261) QuantErr: 11.67261 batch_time=0.51778
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 2.13761 (QuantReg: 11.85642) QuantErr: 11.85642 batch_time=0.54045
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.30751 (QuantReg: 11.67204) QuantErr: 11.67204 batch_time=0.51481
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.35966 (QuantReg: 11.77254) QuantErr: 11.77254 batch_time=0.55472
Train Epoch: 6 codebook_update_time=1.82769
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch6.pth ...
Done in 5.446s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch6.pth ...
Done in 10.584s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.340284837245941
quant_reg : 11.62506506729126
quant_err : 11.62506506729126
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 28.571428571428573
MSRVTT_full_val/t2v_metrics/R5: 64.98993963782696
MSRVTT_full_val/t2v_metrics/R10: 78.06841046277665
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.098591549295774
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.531258900456436
MSRVTT_full_val/v2t_metrics/R1: 33.80281690140845
MSRVTT_full_val/v2t_metrics/R5: 70.22132796780684
MSRVTT_full_val/v2t_metrics/R10: 83.5010060362173
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.253521126760563
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.30483174355487
MSRVTT_full_test/t2v_metrics/R1: 11.33779264214047
MSRVTT_full_test/t2v_metrics/R5: 30.468227424749163
MSRVTT_full_test/t2v_metrics/R10: 43.779264214046826
MSRVTT_full_test/t2v_metrics/R50: 77.55852842809365
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.29197324414716
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.729464887831178
MSRVTT_full_test/v2t_metrics/R1: 12.54180602006689
MSRVTT_full_test/v2t_metrics/R5: 36.220735785953174
MSRVTT_full_test/v2t_metrics/R10: 50.836120401337794
MSRVTT_full_test/v2t_metrics/R50: 82.876254180602
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 38.61153846153846
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.477153907888326
mnt_best : 24.729464887831178
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.40661 (QuantReg: 11.05707) QuantErr: 11.05707 batch_time=37.95921
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 2.09211 (QuantReg: 11.71162) QuantErr: 11.71162 batch_time=0.57302
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.38495 (QuantReg: 11.74564) QuantErr: 11.74564 batch_time=0.50994
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.19113 (QuantReg: 11.68282) QuantErr: 11.68282 batch_time=0.51298
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 2.06068 (QuantReg: 11.36801) QuantErr: 11.36801 batch_time=0.58407
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.30894 (QuantReg: 11.47174) QuantErr: 11.47174 batch_time=0.60897
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.37163 (QuantReg: 11.65036) QuantErr: 11.65036 batch_time=0.53639
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.40344 (QuantReg: 11.53265) QuantErr: 11.53265 batch_time=0.51026
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.37457 (QuantReg: 11.52960) QuantErr: 11.52960 batch_time=0.53480
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 2.14176 (QuantReg: 11.60627) QuantErr: 11.60627 batch_time=0.51494
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.41357 (QuantReg: 11.49462) QuantErr: 11.49462 batch_time=0.52106
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 1.76194 (QuantReg: 11.83174) QuantErr: 11.83174 batch_time=0.58945
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.92903 (QuantReg: 11.49373) QuantErr: 11.49373 batch_time=0.58696
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.19344 (QuantReg: 11.50199) QuantErr: 11.50199 batch_time=0.57280
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.23140 (QuantReg: 11.29274) QuantErr: 11.29274 batch_time=0.54816
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 2.01688 (QuantReg: 11.65135) QuantErr: 11.65135 batch_time=0.51560
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.93831 (QuantReg: 11.81394) QuantErr: 11.81394 batch_time=0.55879
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 2.17347 (QuantReg: 11.41331) QuantErr: 11.41331 batch_time=0.60742
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.67495 (QuantReg: 11.68602) QuantErr: 11.68602 batch_time=0.52103
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 2.00729 (QuantReg: 11.72759) QuantErr: 11.72759 batch_time=4.34511
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.59461 (QuantReg: 11.63562) QuantErr: 11.63562 batch_time=0.58016
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 2.12326 (QuantReg: 11.59038) QuantErr: 11.59038 batch_time=0.53123
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.27963 (QuantReg: 11.57055) QuantErr: 11.57055 batch_time=0.66934
Train Epoch: 7 codebook_update_time=1.79426
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch7.pth ...
Done in 6.666s
removing stale ckpt [epoch 6] [took 0.00s]
epoch : 7
loss : 2.179917585849762
quant_reg : 11.621173519134521
quant_err : 11.621173519134521
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 27.766599597585515
MSRVTT_full_val/t2v_metrics/R5: 64.78873239436619
MSRVTT_full_val/t2v_metrics/R10: 79.07444668008048
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.164989939637827
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.20187640760736
MSRVTT_full_val/v2t_metrics/R1: 34.80885311871227
MSRVTT_full_val/v2t_metrics/R5: 71.83098591549296
MSRVTT_full_val/v2t_metrics/R10: 85.71428571428571
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.255533199195171
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.843674313186014
MSRVTT_full_test/t2v_metrics/R1: 10.735785953177258
MSRVTT_full_test/t2v_metrics/R5: 30.401337792642142
MSRVTT_full_test/t2v_metrics/R10: 44.280936454849495
MSRVTT_full_test/t2v_metrics/R50: 78.39464882943143
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 47.08277591973244
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.358345071023688
MSRVTT_full_test/v2t_metrics/R1: 12.575250836120402
MSRVTT_full_test/v2t_metrics/R5: 36.98996655518395
MSRVTT_full_test/v2t_metrics/R10: 52.04013377926422
MSRVTT_full_test/v2t_metrics/R50: 83.64548494983278
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 37.42675585284281
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.92763971741934
mnt_best : 24.729464887831178
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 2.19907 (QuantReg: 11.65493) QuantErr: 11.65493 batch_time=40.49276
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.30966 (QuantReg: 11.70714) QuantErr: 11.70714 batch_time=0.55615
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.60621 (QuantReg: 11.38847) QuantErr: 11.38847 batch_time=0.59805
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 2.18723 (QuantReg: 11.78175) QuantErr: 11.78175 batch_time=0.59635
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.30704 (QuantReg: 11.69215) QuantErr: 11.69215 batch_time=0.54663
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 2.22150 (QuantReg: 11.57495) QuantErr: 11.57495 batch_time=0.54189
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.93148 (QuantReg: 11.93077) QuantErr: 11.93077 batch_time=2.09630
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 1.95852 (QuantReg: 11.95084) QuantErr: 11.95084 batch_time=0.61754
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.32653 (QuantReg: 11.73135) QuantErr: 11.73135 batch_time=0.51594
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 2.05029 (QuantReg: 11.48753) QuantErr: 11.48753 batch_time=0.51642
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 2.02202 (QuantReg: 11.86049) QuantErr: 11.86049 batch_time=0.60723
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.50456 (QuantReg: 11.76438) QuantErr: 11.76438 batch_time=0.50318
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.97397 (QuantReg: 11.57116) QuantErr: 11.57116 batch_time=0.57570
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.84985 (QuantReg: 11.57576) QuantErr: 11.57576 batch_time=0.52209
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.43257 (QuantReg: 11.89294) QuantErr: 11.89294 batch_time=0.53008
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.75947 (QuantReg: 11.62036) QuantErr: 11.62036 batch_time=1.69112
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.30339 (QuantReg: 11.66820) QuantErr: 11.66820 batch_time=0.51382
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.33217 (QuantReg: 11.92832) QuantErr: 11.92832 batch_time=0.61069
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.66953 (QuantReg: 11.77697) QuantErr: 11.77697 batch_time=0.54617
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.66003 (QuantReg: 11.64679) QuantErr: 11.64679 batch_time=0.50075
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.69062 (QuantReg: 11.99554) QuantErr: 11.99554 batch_time=0.52610
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.34522 (QuantReg: 11.52314) QuantErr: 11.52314 batch_time=0.53232
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.67697 (QuantReg: 11.66985) QuantErr: 11.66985 batch_time=0.54570
Train Epoch: 8 codebook_update_time=1.73764
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch8.pth ...
Done in 5.635s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch8.pth ...
Done in 10.762s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 2.093755865573883
quant_reg : 11.635057960510254
quant_err : 11.635057960510254
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 27.16297786720322
MSRVTT_full_val/t2v_metrics/R5: 65.3923541247485
MSRVTT_full_val/t2v_metrics/R10: 78.87323943661971
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.945674044265594
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 51.93713751882282
MSRVTT_full_val/v2t_metrics/R1: 32.59557344064386
MSRVTT_full_val/v2t_metrics/R5: 70.62374245472837
MSRVTT_full_val/v2t_metrics/R10: 83.70221327967806
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.03420523138833
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 57.758436936976274
MSRVTT_full_test/t2v_metrics/R1: 11.003344481605351
MSRVTT_full_test/t2v_metrics/R5: 31.103678929765888
MSRVTT_full_test/t2v_metrics/R10: 44.81605351170568
MSRVTT_full_test/t2v_metrics/R50: 78.09364548494983
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.64448160535117
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.84601215212569
MSRVTT_full_test/v2t_metrics/R1: 13.377926421404682
MSRVTT_full_test/v2t_metrics/R5: 37.558528428093645
MSRVTT_full_test/v2t_metrics/R10: 52.207357859531776
MSRVTT_full_test/v2t_metrics/R50: 84.04682274247492
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.40200668896321
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.712762294322726
mnt_best : 24.84601215212569
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.20690 (QuantReg: 11.19340) QuantErr: 11.19340 batch_time=34.10919
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.92060 (QuantReg: 11.29930) QuantErr: 11.29930 batch_time=0.52632
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.27796 (QuantReg: 11.43930) QuantErr: 11.43930 batch_time=0.58279
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.20597 (QuantReg: 11.48434) QuantErr: 11.48434 batch_time=0.58015
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.50008 (QuantReg: 11.32133) QuantErr: 11.32133 batch_time=0.51108
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.25727 (QuantReg: 11.47133) QuantErr: 11.47133 batch_time=0.52125
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 2.00697 (QuantReg: 11.40488) QuantErr: 11.40488 batch_time=1.99377
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.67452 (QuantReg: 11.54734) QuantErr: 11.54734 batch_time=0.60118
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.21268 (QuantReg: 11.37899) QuantErr: 11.37899 batch_time=0.51152
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.99844 (QuantReg: 11.60079) QuantErr: 11.60079 batch_time=0.58359
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 2.05236 (QuantReg: 11.50801) QuantErr: 11.50801 batch_time=0.60021
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.81980 (QuantReg: 11.75173) QuantErr: 11.75173 batch_time=0.50084
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 2.03514 (QuantReg: 11.66919) QuantErr: 11.66919 batch_time=0.61991
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 2.07255 (QuantReg: 11.77496) QuantErr: 11.77496 batch_time=0.52930
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 2.14344 (QuantReg: 11.77888) QuantErr: 11.77888 batch_time=0.56192
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 2.15623 (QuantReg: 11.49101) QuantErr: 11.49101 batch_time=0.55401
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.60514 (QuantReg: 11.71663) QuantErr: 11.71663 batch_time=0.69276
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.66495 (QuantReg: 11.75121) QuantErr: 11.75121 batch_time=0.57874
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.71295 (QuantReg: 11.95378) QuantErr: 11.95378 batch_time=0.57642
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.83185 (QuantReg: 11.50524) QuantErr: 11.50524 batch_time=0.52228
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.87430 (QuantReg: 11.67666) QuantErr: 11.67666 batch_time=0.62233
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.97577 (QuantReg: 11.67441) QuantErr: 11.67441 batch_time=0.56674
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.38473 (QuantReg: 11.74274) QuantErr: 11.74274 batch_time=0.57726
Train Epoch: 9 codebook_update_time=1.71423
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch9.pth ...
Done in 22.899s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch9.pth ...
Done in 28.001s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 1.9617829852104187
quant_reg : 11.606198234558105
quant_err : 11.606198234558105
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 68.00804828973843
MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.724346076458753
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.19574515597089
MSRVTT_full_val/v2t_metrics/R1: 35.010060362173036
MSRVTT_full_val/v2t_metrics/R5: 73.2394366197183
MSRVTT_full_val/v2t_metrics/R10: 83.29979879275653
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.2716297786720325
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 59.7760576346889
MSRVTT_full_test/t2v_metrics/R1: 11.57190635451505
MSRVTT_full_test/t2v_metrics/R5: 31.839464882943144
MSRVTT_full_test/t2v_metrics/R10: 46.98996655518395
MSRVTT_full_test/t2v_metrics/R50: 80.03344481605352
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.63511705685619
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.86973234892621
MSRVTT_full_test/v2t_metrics/R1: 13.678929765886288
MSRVTT_full_test/v2t_metrics/R5: 38.32775919732441
MSRVTT_full_test/v2t_metrics/R10: 53.27759197324415
MSRVTT_full_test/v2t_metrics/R50: 83.91304347826087
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.54013377926422
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.341476585392563
mnt_best : 25.86973234892621
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.61102 (QuantReg: 11.21761) QuantErr: 11.21761 batch_time=33.05160
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.85247 (QuantReg: 11.52294) QuantErr: 11.52294 batch_time=0.54785
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.60490 (QuantReg: 11.27076) QuantErr: 11.27076 batch_time=0.52569
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.55053 (QuantReg: 11.17906) QuantErr: 11.17906 batch_time=0.51907
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.70939 (QuantReg: 11.79498) QuantErr: 11.79498 batch_time=0.92530
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.35118 (QuantReg: 11.56734) QuantErr: 11.56734 batch_time=0.61147
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.69214 (QuantReg: 11.72706) QuantErr: 11.72706 batch_time=0.68005
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.81977 (QuantReg: 11.30351) QuantErr: 11.30351 batch_time=0.50060
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.88532 (QuantReg: 11.57197) QuantErr: 11.57197 batch_time=0.55531
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.40111 (QuantReg: 11.40131) QuantErr: 11.40131 batch_time=0.53897
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.67090 (QuantReg: 11.52390) QuantErr: 11.52390 batch_time=0.53221
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 2.04337 (QuantReg: 11.66258) QuantErr: 11.66258 batch_time=0.51648
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 2.21206 (QuantReg: 11.57865) QuantErr: 11.57865 batch_time=0.52171
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.34095 (QuantReg: 11.71697) QuantErr: 11.71697 batch_time=0.51092
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.61739 (QuantReg: 11.83615) QuantErr: 11.83615 batch_time=0.51268
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.50930 (QuantReg: 11.71652) QuantErr: 11.71652 batch_time=0.51993
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.79344 (QuantReg: 11.55478) QuantErr: 11.55478 batch_time=0.57464
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.68820 (QuantReg: 11.85641) QuantErr: 11.85641 batch_time=0.54505
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.67413 (QuantReg: 11.95765) QuantErr: 11.95765 batch_time=2.26941
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.65237 (QuantReg: 11.52663) QuantErr: 11.52663 batch_time=1.48524
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.64814 (QuantReg: 11.48932) QuantErr: 11.48932 batch_time=0.59003
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.34367 (QuantReg: 11.61227) QuantErr: 11.61227 batch_time=0.53725
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 2.10852 (QuantReg: 11.44310) QuantErr: 11.44310 batch_time=0.50585
Train Epoch: 10 codebook_update_time=1.70695
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch10.pth ...
Done in 23.035s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 1.8629838514328003
quant_reg : 11.587866710662842
quant_err : 11.587866710662842
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 29.37625754527163
MSRVTT_full_val/t2v_metrics/R5: 66.39839034205231
MSRVTT_full_val/t2v_metrics/R10: 81.89134808853119
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.028169014084508
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.25802894351553
MSRVTT_full_val/v2t_metrics/R1: 35.2112676056338
MSRVTT_full_val/v2t_metrics/R5: 73.64185110663983
MSRVTT_full_val/v2t_metrics/R10: 85.91549295774648
MSRVTT_full_val/v2t_metrics/R50: 97.78672032193158
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.160965794768612
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 60.62139802120216
MSRVTT_full_test/t2v_metrics/R1: 11.404682274247492
MSRVTT_full_test/t2v_metrics/R5: 31.93979933110368
MSRVTT_full_test/t2v_metrics/R10: 46.15384615384615
MSRVTT_full_test/t2v_metrics/R50: 79.79933110367892
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.37157190635452
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.617756344080327
MSRVTT_full_test/v2t_metrics/R1: 13.545150501672241
MSRVTT_full_test/v2t_metrics/R5: 38.96321070234114
MSRVTT_full_test/v2t_metrics/R10: 53.779264214046826
MSRVTT_full_test/v2t_metrics/R50: 85.15050167224081
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.496989966555184
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.50360318610667
mnt_best : 25.86973234892621
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.72795 (QuantReg: 11.40042) QuantErr: 11.40042 batch_time=36.16041
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.48746 (QuantReg: 11.60921) QuantErr: 11.60921 batch_time=0.85130
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.83301 (QuantReg: 11.26380) QuantErr: 11.26380 batch_time=0.51712
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.90048 (QuantReg: 11.61469) QuantErr: 11.61469 batch_time=0.61636
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.85194 (QuantReg: 11.90867) QuantErr: 11.90867 batch_time=0.57019
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.58865 (QuantReg: 11.68125) QuantErr: 11.68125 batch_time=0.52841
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 2.00478 (QuantReg: 11.40169) QuantErr: 11.40169 batch_time=0.52727
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.63320 (QuantReg: 11.82310) QuantErr: 11.82310 batch_time=0.52199
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.66200 (QuantReg: 11.47217) QuantErr: 11.47217 batch_time=0.52191
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 2.04585 (QuantReg: 11.43656) QuantErr: 11.43656 batch_time=0.51302
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 2.07096 (QuantReg: 11.40209) QuantErr: 11.40209 batch_time=0.63289
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.80540 (QuantReg: 11.90274) QuantErr: 11.90274 batch_time=1.07458
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 2.30232 (QuantReg: 11.37877) QuantErr: 11.37877 batch_time=0.56912
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.00121 (QuantReg: 11.47518) QuantErr: 11.47518 batch_time=0.52806
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.71700 (QuantReg: 11.36317) QuantErr: 11.36317 batch_time=0.50673
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.75484 (QuantReg: 11.45068) QuantErr: 11.45068 batch_time=0.51403
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.74273 (QuantReg: 11.30688) QuantErr: 11.30688 batch_time=0.51988
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.49201 (QuantReg: 11.52070) QuantErr: 11.52070 batch_time=0.53791
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.56417 (QuantReg: 11.99835) QuantErr: 11.99835 batch_time=5.17426
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.87757 (QuantReg: 11.74767) QuantErr: 11.74767 batch_time=0.60105
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.91059 (QuantReg: 12.07938) QuantErr: 12.07938 batch_time=0.53966
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.61798 (QuantReg: 11.73509) QuantErr: 11.73509 batch_time=0.53427
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.45647 (QuantReg: 12.17063) QuantErr: 12.17063 batch_time=0.59776
Train Epoch: 11 codebook_update_time=2.29293
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch11.pth ...
Done in 4.569s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch11.pth ...
Done in 8.770s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 1.7771258697509766
quant_reg : 11.6400193901062
quant_err : 11.6400193901062
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 27.96780684104628
MSRVTT_full_val/t2v_metrics/R5: 65.79476861167002
MSRVTT_full_val/t2v_metrics/R10: 78.67203219315896
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.853118712273641
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.50774668213979
MSRVTT_full_val/v2t_metrics/R1: 37.02213279678068
MSRVTT_full_val/v2t_metrics/R5: 72.63581488933602
MSRVTT_full_val/v2t_metrics/R10: 85.31187122736418
MSRVTT_full_val/v2t_metrics/R50: 97.58551307847083
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.2434607645875255
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.217262512921465
MSRVTT_full_test/t2v_metrics/R1: 11.605351170568563
MSRVTT_full_test/t2v_metrics/R5: 33.64548494983278
MSRVTT_full_test/t2v_metrics/R10: 45.48494983277592
MSRVTT_full_test/t2v_metrics/R50: 78.72909698996655
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 46.78829431438127
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.09061199770579
MSRVTT_full_test/v2t_metrics/R1: 13.812709030100335
MSRVTT_full_test/v2t_metrics/R5: 39.264214046822744
MSRVTT_full_test/v2t_metrics/R10: 53.37792642140468
MSRVTT_full_test/v2t_metrics/R50: 84.44816053511705
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 35.09548494983277
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.705239569786624
mnt_best : 26.09061199770579
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.72793 (QuantReg: 11.52650) QuantErr: 11.52650 batch_time=37.75331
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.99751 (QuantReg: 11.66045) QuantErr: 11.66045 batch_time=0.55933
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.53140 (QuantReg: 11.41864) QuantErr: 11.41864 batch_time=0.50598
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.57125 (QuantReg: 11.52532) QuantErr: 11.52532 batch_time=0.55336
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.44005 (QuantReg: 11.76022) QuantErr: 11.76022 batch_time=0.60451
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.83135 (QuantReg: 11.51352) QuantErr: 11.51352 batch_time=0.58667
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.48991 (QuantReg: 11.56724) QuantErr: 11.56724 batch_time=0.49573
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.70737 (QuantReg: 11.53901) QuantErr: 11.53901 batch_time=3.57606
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.54656 (QuantReg: 11.47422) QuantErr: 11.47422 batch_time=0.50763
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.94682 (QuantReg: 11.50401) QuantErr: 11.50401 batch_time=0.58889
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.39847 (QuantReg: 11.51711) QuantErr: 11.51711 batch_time=0.52796
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.83876 (QuantReg: 11.58532) QuantErr: 11.58532 batch_time=0.62877
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.52568 (QuantReg: 11.69342) QuantErr: 11.69342 batch_time=0.51515
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.45302 (QuantReg: 11.98818) QuantErr: 11.98818 batch_time=0.50574
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.80402 (QuantReg: 11.46289) QuantErr: 11.46289 batch_time=0.56898
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.28027 (QuantReg: 11.75695) QuantErr: 11.75695 batch_time=0.52403
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.42436 (QuantReg: 11.62524) QuantErr: 11.62524 batch_time=0.56984
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 2.17225 (QuantReg: 11.40473) QuantErr: 11.40473 batch_time=0.51355
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.82361 (QuantReg: 11.45098) QuantErr: 11.45098 batch_time=0.65373
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 2.03339 (QuantReg: 11.49326) QuantErr: 11.49326 batch_time=0.52173
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.75164 (QuantReg: 11.19897) QuantErr: 11.19897 batch_time=0.59891
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.65719 (QuantReg: 11.78705) QuantErr: 11.78705 batch_time=0.51029
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.83213 (QuantReg: 11.99819) QuantErr: 11.99819 batch_time=0.58788
Train Epoch: 12 codebook_update_time=2.07048
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch12.pth ...
Done in 4.426s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch12.pth ...
Done in 8.628s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 1.7092913761138917
quant_reg : 11.60410792541504
quant_err : 11.60410792541504
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 31.388329979879277
MSRVTT_full_val/t2v_metrics/R5: 65.79476861167002
MSRVTT_full_val/t2v_metrics/R10: 78.87323943661971
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.251509054325956
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.61304821720724
MSRVTT_full_val/v2t_metrics/R1: 35.412474849094565
MSRVTT_full_val/v2t_metrics/R5: 75.65392354124748
MSRVTT_full_val/v2t_metrics/R10: 85.71428571428571
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.442655935613682
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.23696363283803
MSRVTT_full_test/t2v_metrics/R1: 12.006688963210703
MSRVTT_full_test/t2v_metrics/R5: 33.47826086956522
MSRVTT_full_test/t2v_metrics/R10: 46.78929765886288
MSRVTT_full_test/t2v_metrics/R50: 79.6989966555184
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.39163879598662
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.593626108500633
MSRVTT_full_test/v2t_metrics/R1: 13.210702341137123
MSRVTT_full_test/v2t_metrics/R5: 40.03344481605351
MSRVTT_full_test/v2t_metrics/R10: 55.1505016722408
MSRVTT_full_test/v2t_metrics/R50: 85.35117056856187
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.436120401337796
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.782185114839322
mnt_best : 26.593626108500633
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.59158 (QuantReg: 11.61705) QuantErr: 11.61705 batch_time=36.25209
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 2.28670 (QuantReg: 11.02643) QuantErr: 11.02643 batch_time=0.50242
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.81709 (QuantReg: 11.71812) QuantErr: 11.71812 batch_time=0.54697
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.46106 (QuantReg: 11.67252) QuantErr: 11.67252 batch_time=0.53171
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.57473 (QuantReg: 11.65146) QuantErr: 11.65146 batch_time=0.54950
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.63882 (QuantReg: 11.89000) QuantErr: 11.89000 batch_time=0.55349
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.75302 (QuantReg: 11.79594) QuantErr: 11.79594 batch_time=0.72011
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 2.03178 (QuantReg: 11.92260) QuantErr: 11.92260 batch_time=1.35448
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.36340 (QuantReg: 11.31789) QuantErr: 11.31789 batch_time=0.50800
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.75147 (QuantReg: 11.54757) QuantErr: 11.54757 batch_time=0.53378
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.69149 (QuantReg: 11.45167) QuantErr: 11.45167 batch_time=0.51272
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.32511 (QuantReg: 11.65325) QuantErr: 11.65325 batch_time=0.52319
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.60727 (QuantReg: 11.65968) QuantErr: 11.65968 batch_time=0.96482
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.55614 (QuantReg: 11.80621) QuantErr: 11.80621 batch_time=0.61875
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.86800 (QuantReg: 11.36281) QuantErr: 11.36281 batch_time=0.51603
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.45794 (QuantReg: 11.49675) QuantErr: 11.49675 batch_time=0.51974
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.30106 (QuantReg: 11.84064) QuantErr: 11.84064 batch_time=0.54628
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.97968 (QuantReg: 11.45084) QuantErr: 11.45084 batch_time=0.51940
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.39861 (QuantReg: 12.00076) QuantErr: 12.00076 batch_time=0.52664
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.93260 (QuantReg: 11.79449) QuantErr: 11.79449 batch_time=2.51369
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.97482 (QuantReg: 11.84524) QuantErr: 11.84524 batch_time=0.56618
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.62955 (QuantReg: 11.66097) QuantErr: 11.66097 batch_time=0.92027
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.67232 (QuantReg: 11.76225) QuantErr: 11.76225 batch_time=0.53727
Train Epoch: 13 codebook_update_time=1.79681
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch13.pth ...
Done in 4.248s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch13.pth ...
Done in 8.613s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 1.6453655591011047
quant_reg : 11.64262022781372
quant_err : 11.64262022781372
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 30.58350100603622
MSRVTT_full_val/t2v_metrics/R5: 67.00201207243461
MSRVTT_full_val/t2v_metrics/R10: 79.27565392354124
MSRVTT_full_val/t2v_metrics/R50: 96.78068410462777
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.046277665995976
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 54.56384405460885
MSRVTT_full_val/v2t_metrics/R1: 36.82092555331992
MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
MSRVTT_full_val/v2t_metrics/R10: 84.30583501006036
MSRVTT_full_val/v2t_metrics/R50: 97.38430583501005
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.3762575452716295
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.366669926534655
MSRVTT_full_test/t2v_metrics/R1: 11.438127090301004
MSRVTT_full_test/t2v_metrics/R5: 34.54849498327759
MSRVTT_full_test/t2v_metrics/R10: 47.65886287625418
MSRVTT_full_test/t2v_metrics/R50: 80.50167224080268
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 43.04113712374582
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.605774691678516
MSRVTT_full_test/v2t_metrics/R1: 14.347826086956522
MSRVTT_full_test/v2t_metrics/R5: 39.83277591973244
MSRVTT_full_test/v2t_metrics/R10: 54.98327759197324
MSRVTT_full_test/v2t_metrics/R50: 85.75250836120401
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.15351170568562
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.556277557393592
mnt_best : 26.605774691678516
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.47511 (QuantReg: 11.62300) QuantErr: 11.62300 batch_time=39.65243
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 2.11674 (QuantReg: 11.89652) QuantErr: 11.89652 batch_time=0.52499
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.52221 (QuantReg: 11.76492) QuantErr: 11.76492 batch_time=0.60097
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.54415 (QuantReg: 11.61810) QuantErr: 11.61810 batch_time=1.54147
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.88749 (QuantReg: 11.58026) QuantErr: 11.58026 batch_time=1.03392
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.54973 (QuantReg: 11.58943) QuantErr: 11.58943 batch_time=0.51481
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.94067 (QuantReg: 11.60915) QuantErr: 11.60915 batch_time=0.58672
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.73766 (QuantReg: 11.31845) QuantErr: 11.31845 batch_time=0.51110
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.27546 (QuantReg: 11.38786) QuantErr: 11.38786 batch_time=0.53649
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.56218 (QuantReg: 11.39961) QuantErr: 11.39961 batch_time=0.52510
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.44905 (QuantReg: 11.67322) QuantErr: 11.67322 batch_time=0.58031
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.73960 (QuantReg: 11.33343) QuantErr: 11.33343 batch_time=0.54800
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.44818 (QuantReg: 11.93923) QuantErr: 11.93923 batch_time=0.50962
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.32034 (QuantReg: 11.71851) QuantErr: 11.71851 batch_time=0.50672
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.53741 (QuantReg: 11.72583) QuantErr: 11.72583 batch_time=0.51615
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.30552 (QuantReg: 11.73163) QuantErr: 11.73163 batch_time=0.59152
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.30808 (QuantReg: 11.38637) QuantErr: 11.38637 batch_time=0.51731
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.77235 (QuantReg: 11.86670) QuantErr: 11.86670 batch_time=0.51240
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.44524 (QuantReg: 11.57985) QuantErr: 11.57985 batch_time=0.58448
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.85285 (QuantReg: 11.32318) QuantErr: 11.32318 batch_time=0.78479
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.61649 (QuantReg: 11.72989) QuantErr: 11.72989 batch_time=0.55199
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.45945 (QuantReg: 11.84936) QuantErr: 11.84936 batch_time=0.52450
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.48618 (QuantReg: 11.66719) QuantErr: 11.66719 batch_time=0.56006
Train Epoch: 14 codebook_update_time=2.10078
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch14.pth ...
Done in 4.882s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 1.5956147027015686
quant_reg : 11.635172561645508
quant_err : 11.635172561645508
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 29.77867203219316
MSRVTT_full_val/t2v_metrics/R5: 65.99597585513078
MSRVTT_full_val/t2v_metrics/R10: 77.8672032193159
MSRVTT_full_val/t2v_metrics/R50: 96.579476861167
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.768611670020121
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.488339931925964
MSRVTT_full_val/v2t_metrics/R1: 38.83299798792756
MSRVTT_full_val/v2t_metrics/R5: 71.0261569416499
MSRVTT_full_val/v2t_metrics/R10: 82.897384305835
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.301810865191147
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.14862390531215
MSRVTT_full_test/t2v_metrics/R1: 11.605351170568563
MSRVTT_full_test/t2v_metrics/R5: 31.57190635451505
MSRVTT_full_test/t2v_metrics/R10: 45.752508361204015
MSRVTT_full_test/t2v_metrics/R50: 78.72909698996655
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 49.73076923076923
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.593204517047404
MSRVTT_full_test/v2t_metrics/R1: 13.779264214046822
MSRVTT_full_test/v2t_metrics/R5: 38.09364548494983
MSRVTT_full_test/v2t_metrics/R10: 52.90969899665552
MSRVTT_full_test/v2t_metrics/R50: 84.21404682274247
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 37.28494983277592
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.28339896154844
mnt_best : 26.605774691678516
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.95249 (QuantReg: 11.33286) QuantErr: 11.33286 batch_time=35.52925
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.56161 (QuantReg: 11.50145) QuantErr: 11.50145 batch_time=0.56127
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.58614 (QuantReg: 11.60115) QuantErr: 11.60115 batch_time=0.55623
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.63981 (QuantReg: 11.47667) QuantErr: 11.47667 batch_time=0.51336
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.59922 (QuantReg: 11.81721) QuantErr: 11.81721 batch_time=0.50904
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.71815 (QuantReg: 11.21412) QuantErr: 11.21412 batch_time=0.60266
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.89546 (QuantReg: 11.46809) QuantErr: 11.46809 batch_time=2.35477
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.40888 (QuantReg: 11.62913) QuantErr: 11.62913 batch_time=0.50947
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.85109 (QuantReg: 11.64462) QuantErr: 11.64462 batch_time=0.57524
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.51165 (QuantReg: 11.70871) QuantErr: 11.70871 batch_time=0.52838
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.76771 (QuantReg: 11.51316) QuantErr: 11.51316 batch_time=0.50157
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.30209 (QuantReg: 11.43328) QuantErr: 11.43328 batch_time=0.56658
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.47371 (QuantReg: 11.55328) QuantErr: 11.55328 batch_time=0.54371
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.85570 (QuantReg: 11.71335) QuantErr: 11.71335 batch_time=0.54578
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.58508 (QuantReg: 11.38414) QuantErr: 11.38414 batch_time=0.52603
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.60457 (QuantReg: 11.81451) QuantErr: 11.81451 batch_time=0.83264
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.82673 (QuantReg: 11.87243) QuantErr: 11.87243 batch_time=0.54491
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.44265 (QuantReg: 11.70019) QuantErr: 11.70019 batch_time=0.50604
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.42991 (QuantReg: 11.71539) QuantErr: 11.71539 batch_time=0.52293
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.71918 (QuantReg: 11.37422) QuantErr: 11.37422 batch_time=0.58122
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.82907 (QuantReg: 11.75482) QuantErr: 11.75482 batch_time=0.61090
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.71728 (QuantReg: 11.71305) QuantErr: 11.71305 batch_time=0.72839
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.67209 (QuantReg: 11.51295) QuantErr: 11.51295 batch_time=0.54545
Train Epoch: 15 codebook_update_time=1.75584
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch15.pth ...
Done in 8.110s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full_t0.07/checkpoint-epoch15.pth ...
Done in 12.466s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 1.5406653852462768
quant_reg : 11.637196418762207
quant_err : 11.637196418762207
learning_rate : 2.4383748955776477e-05