Some Confusion about Prediction Results？

[dunzane]

🙏🙏🙏🙏🙏🙏
Thank you for your outstanding work!!!, details about this issue as follows:

• Description:
  I’m using the Autoformer implementation under models/Autoformer to predict the ETT-small/ETTh1.csv dataset. Since there’s no hyperparameter search provided, I directly ran the bash script under the scripts folder. However, I noticed a significant gap in the results when using an input length of 96 to predict 720 steps (MSE: 0.587), compared to the reported result in the original paper (MSE: 0.498).

• Questions:

1. Are there specific configurations or hyperparameter adjustments needed to reproduce the paper’s results?
2. Do you plan to include standard hyperparameter searches in the future to help bridge performance gaps, or do you have suggestions for tuning the library to achieve better results?

• Result Description:

long_term_forecast_ETTh1_96_96_Autoformer_ETTh1_ftM_sl96_ll48_pl24_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0  
mse:0.37013882398605347, mae:0.42097944021224976, dtw:Not calculated

long_term_forecast_ETTh1_96_336_Autoformer_ETTh1_ftM_sl96_ll48_pl168_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0  
mse:0.5342859625816345, mae:0.49671873450279236, dtw:Not calculated

long_term_forecast_ETTh1_96_24_Autoformer_ETTh1_ftM_sl96_ll48_pl24_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0  
mse:0.37013882398605347, mae:0.42097944021224976, dtw:Not calculated

long_term_forecast_ETTh1_96_48_Autoformer_ETTh1_ftM_sl96_ll48_pl48_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0  
mse:0.3968711197376251, mae:0.4279240369796753, dtw:Not calculated

long_term_forecast_ETTh1_96_168_Autoformer_ETTh1_ftM_sl96_ll48_pl168_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0  
mse:0.5342862010002136, mae:0.4967191219329834, dtw:Not calculated

long_term_forecast_ETTh1_96_336_Autoformer_ETTh1_ftM_sl96_ll48_pl336_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0  
mse:0.5114971995353699, mae:0.49485111236572266, dtw:Not calculated

long_term_forecast_ETTh1_96_720_Autoformer_ETTh1_ftM_sl96_ll48_pl720_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0  
mse:0.586983859539032, mae:0.5547620058059692, dtw:Not calculated`

• Logs:

Using GPU
Args in experiment:
Namespace(task_name='long_term_forecast', is_training=1, model_id='ETTh1_96_720', model='Autoformer', data='ETTh1', root_path='./dataset/ETT-small/', data_path='ETTh1.csv', features='M', target='OT', freq='h', checkpoints='./checkpoints/', seq_len=96, label_len=48, pred_len=720, seasonal_patterns='Monthly', inverse=False, expand=2, d_conv=4, top_k=5, num_kernels=6, enc_in=7, dec_in=7, c_out=7, d_model=512, n_heads=8, e_layers=2, d_layers=1, d_ff=2048, moving_avg=25, factor=3, distil=True, dropout=0.1, embed='timeF', activation='gelu', channel_independence=1, decomp_method='moving_avg', use_norm=1, down_sampling_layers=0, down_sampling_window=1, down_sampling_method=None, seg_len=96, num_workers=10, itr=1, train_epochs=10, batch_size=32, patience=3, learning_rate=0.0001, des='Exp', loss='MSE', lradj='type1', use_amp=False, use_gpu=True, gpu=0, gpu_type='cuda', use_multi_gpu=False, devices='0,1,2,3', p_hidden_dims=[128, 128], p_hidden_layers=2, use_dtw=False, augmentation_ratio=0, seed=2, jitter=False, scaling=
Use GPU: cuda:0
>>>>>>>start training : long_term_forecast_ETTh1_96_720_Autoformer_ETTh1_ftM_sl96_ll48_pl720_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0>>>>>>>>>>>>>>>>>>>>>>>>>>
train 7825
val 2161
test 2161
iters: 100, epoch: 1 | loss: 0.5048556
speed: 0.2038s/iter; left time: 479.0261s
iters: 200, epoch: 1 | loss: 0.5594337
speed: 0.1837s/iter; left time: 413.4543s
Epoch: 1 cost time: 45.83665156364441
Epoch: 1, Steps: 245 | Train Loss: 0.6220744 Vali Loss: 1.7238371 Test Loss: 0.5491471 Peak Memory: 6193.65 MB
Validation loss decreased (inf --> 1.723837). Saving model ...
Updating learning rate to 0.0001
iters: 100, epoch: 2 | loss: 0.5470211
speed: 0.5258s/iter; left time: 1107.2467s
iters: 200, epoch: 2 | loss: 0.5807275
speed: 0.1828s/iter; left time: 366.7448s
Epoch: 2 cost time: 45.278279304504395
Epoch: 2, Steps: 245 | Train Loss: 0.5600557 Vali Loss: 1.7011523 Test Loss: 0.5873765 Peak Memory: 6193.65 MB
Validation loss decreased (1.723837 --> 1.701152). Saving model ...
Updating learning rate to 5e-05
iters: 100, epoch: 3 | loss: 0.4777997
speed: 0.5231s/iter; left time: 973.5513s
iters: 200, epoch: 3 | loss: 0.4350199
speed: 0.1828s/iter; left time: 321.9951s
Epoch: 3 cost time: 45.362247705459595
Epoch: 3, Steps: 245 | Train Loss: 0.4881297 Vali Loss: 1.8302903 Test Loss: 0.6512775 Peak Memory: 6193.65 MB
EarlyStopping counter: 1 out of 3
Updating learning rate to 2.5e-05
iters: 100, epoch: 4 | loss: 0.4568125
speed: 0.5268s/iter; left time: 851.3388s
iters: 200, epoch: 4 | loss: 0.4054065
speed: 0.1822s/iter; left time: 276.2789s
Epoch: 4 cost time: 45.10278296470642
Epoch: 4, Steps: 245 | Train Loss: 0.4523836 Vali Loss: 1.8280600 Test Loss: 0.6942503 Peak Memory: 6193.65 MB
EarlyStopping counter: 2 out of 3
Updating learning rate to 1.25e-05
iters: 100, epoch: 5 | loss: 0.4479555
speed: 0.5277s/iter; left time: 723.5420s
iters: 200, epoch: 5 | loss: 0.4255082
speed: 0.1838s/iter; left time: 233.5750s
Epoch: 5 cost time: 45.26950550079346
Epoch: 5, Steps: 245 | Train Loss: 0.4344996 Vali Loss: 1.8271523 Test Loss: 0.6992639 Peak Memory: 6193.65 MB
EarlyStopping counter: 3 out of 3
Early stopping
>>>>>>>testing : long_term_forecast_ETTh1_96_720_Autoformer_ETTh1_ftM_sl96_ll48_pl720_dm512_nh8_el2_dl1_df2048_expand2_dc4_fc3_ebtimeF_dtTrue_Exp_0<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
test 2161
test shape: (2161, 720, 7) (2161, 720, 7)
test shape: (2161, 720, 7) (2161, 720, 7)
mse:0.586983859539032, mae:0.5547620058059692, dtw:Not calculated
swanlab: Experiment Autoformer/ETTh1_96_720 has completed

[erikalien5595]

Hello, do you change the code yourself? Because I found that the command line did not print "Peak Memory" when long term forecasting is performed using the code of this repo.

[dunzane]

Yes, I added some codes for recording peak reveals during training, the pytorch framework has a corresponding section provided, this change shouldn't have any effect on the prediction results

[erikalien5595]

Yes, I added some codes for recording peak reveals during training, the pytorch framework has a corresponding section provided, this change shouldn't have any effect on the prediction results

Thank you for your reply. My doubts have been cleared up. Hope your question can be answered by the author as soon as possible!

[wuhaixu2016]

Hi, thanks for your detailed question and experiment.

I think the model and script in TSLib are correct, but due to the torch version, the final performance may be unstable. I would suggest you try our official code for Autoformer: https://github.com/thuml/Autoformer