fit_gpytorch_mll multi-start support

[AlexanderMouton]

Hi, I am trying different kernel configurations for the multi-fidelity use-case with a noisy 3-dimensional toy problem and have run into issues when using fit_gpytorch_mll.

Per my understanding, by default fit_gpytorch_mll uses scipy's LFBFGS optimizer to fit the kernel hyperparameters to the data. When using the default implementation of fit_gpytorch_mll, the average MSE (mean-squared error) betwen the true function and the model's predicted function would jump up and down between iterations.

For example, the average MSE for a couple of subsequent iterations would look something like:

...
iteration 30 : average MSE = 9.4e-4
iteration 31 : average MSE = 1.14e-1
iteration 32 : average MSE = 1.12e-1
iteration 33 : average MSE = 1.07e-3
iteration 34 : average MSE = 1.06e-1
iteration 35 : average MSE = 9.53e-4
...

After instead using fit_gpytorch_mll_torch, which uses the Adam optimiser by default, this behaviour ceased and I observed a mostly stable decline in average MSE.

fit_gpytorch_mll generally took a couple of seconds to fit to <200 points, while fit_gpytorch_mll_torch takes 40-80 seconds.

Is it possible that LFBFGS was just starting from a bad initial state and terminating almost immediately?
If so, could adding multi-start support for fitting the kernel hyperparameters possibly solve this problem?

Thanks!

Alex

[saitcakmak]

Hi @AlexanderMouton. Thanks for reporting. Do you see any warnings related to optimization termination during model fitting? We sometimes see numerical issues leading to LBFGS terminating early, which could be one possibility here. We can take a closer look at this if you share the code for reproducing these results.

If so, could adding multi-start support for fitting the kernel hyperparameters possibly solve this problem?

It could definitely help. There's a feature request open for this though it hasn't been high on the priority list so far: #1724. It shouldn't be difficult to test this out to see if it helps with the issue though

[AlexanderMouton]

Hi @saitcakmak

Thank you for your prompt response!

I had some warning filters up, so I will rerun the tests without them and report back. I will also set up a notebook to share the results in the next day or two.

[Balandat]

Another thing to check is the actual MLL value the optimizer achieves. It is possible that a "better" MLL results in a worse fit if the model isn't well specified / the priors are off. It would be concerning if the L-BFGS-B optimizer performed worse than Adam here on the MLL though, so it would be great to get a repro for this.

[AlexanderMouton]

Hi @saitcakmak and @Balandat

Unfortunately I was not able to find the kernel configuration (of the many I tested) for which I had as erratic fluctuations in MSE as I mentioned previously, but I did find one that is quite erratic with the L-BFGS-B optimiser:

The toy problem is 3 dimensional, with the latter two dimensions being fidelity dimensions. The first fidelity dimension represents how correlated the 'data source' is with the target function, and the second fidelity dimensions is related to how many bernoulli sample evaluations were taken to produce an observation's value, where x_3 = 0 corresponds to 10 evaluations and x_3 = 1 corresponds to 50 evaluations.

I have attached a notebook and other required Python/json files where I made plots with two runs, one with fewer and one with more observations.

src.zip

In the run with fewer points, Adam seems to be more stable than L-BFGS-B.

Thanks!

Alex

[AlexanderMouton]

Oh and I did receive this OptimizationWarning message quite often:

scipy_minimize terminated with status 3, displaying original message from scipy.optimize.minimize: ABNORMAL_TERMINATION_IN_LNSRCH