Does Botorch support noisy binary search?

[thomasahle]

I'm trying to optimize a model over three real, positive parameters.
I have a function score(x, y, z) and time(x, y, z).
I want to find argmax_{x,y,z} score(x, y, z) given time(x, y, z) <= 1ms.
Is something like this possible with botorch?
I also know that score and time are both monotone in all three parameters.

I'm imagining fitting two gaussian processes with monotone priors. One for score and one for time.
As my accusation function I want to pick something with a high LCB on score, and a reasonably large probability for time being below the limit.

If I had just one parameter, this would correspond to a noisy binary search.
With three parameters I guess it's slightly different, but not too much.

[Balandat]

Do you always observe the numerical value of time(x, y, z) and score(x, y, z), or just the censored versions1(time(x, y, z) <= 1ms) time(x, y, z) and 1(time(x, y, z) <= 1ms) score(x, y, z), i.e. only if the constraint is satisfied?

If you always observe both numerical values, then this is a pretty standard constrained optimization problem that BoTorch supports out of the box - the tutorial https://github.com/pytorch/botorch/blob/main/tutorials/closed_loop_botorch_only.ipynb shows how to do that with a feasibiltiy-weighted expected improvement formulation.

[Balandat]

As this is a pretty standard setup: If you want a simpler interface to this problem without having to worry about some of the details such as normalizing data etc. I would also recommend you take a look at Ax, which makes specifying and running this kind of optimization a lot easier / more convenient: https://github.com/facebook/Ax

[thomasahle]

I didn't think botorch would work well with non-differentiable functions like 1(...), but that's good to know.
Regarding monotonicity, I haven't found an option for that in Ax?

[thomasahle]

Using the 1(...) method also means that the (gaussian) noise in the time function gets turned into (basically) categorical noise. Like, the same parameters would sometimes give 0 and sometimes give score(x, y, z). I don't know if botorch handles this kind of heavy noise well?

[Balandat]

Oh I am not suggesting that you use the 1(...) "method", my question was about whether in your setting you can actually observe the data (i.e. you may not have a choice what to do...). But from your response it sounds like you do.

What's your plan for the monotonic modeling? You mentioned using monotone priors, how exactly were you planning to go about that? @bletham has played around with monotonic GPs in the past, though at this point we don't have anything canned hooked up in BoTorch.

Using the 1(...) method also means that the (gaussian) noise in the time function gets turned into (basically) categorical noise. Like, the same parameters would sometimes give 0 and sometimes give score(x, y, z). I don't know if botorch handles this kind of heavy noise well?

Not in the standard models. What we would do instead in such situation is fit both a regression and a classification model, and then use the predictions from the classification model in our acquisition function. @j-wilson has done some work on this - we haven't open sourced these models and associated acquisition functions yet, but will do so in the future.

[thomasahle]

What's your plan for the monotonic modeling? You mentioned using monotone priors, how exactly were you planning to go about that?

Oh, I didn't really have a plan. I just realized that adding a prior that the function must be monotone would in principle solve my problem. And so I thought "I wonder if botorch can already do this" :-) Maybe I will talk to Ben about his experience with this! Thank you!