Benchmarking Meta-Black-Box Optimization under
Diverse Optimization Scenarios with Efficiency and Flexibility
Diverse Optimization Scenarios with Efficiency and Flexibility
we propose MetaBox 2.0 version (MetaBox-v2) as a major upgradation of MetaBox-v1. MetaBox-v2 now supports plentiful optimization scenarios to embrace users from single-objective optimization, multi-objective optimization, multi-modal optimization, multi-task optimization and etc. Correspondingly, 18 optimization problem sets (synthetic + realistic), 1900+ problem instances and 36 baseline methods (traditional optimizers + up-to-date MetaBBOs) are reproduced within MetaBox-v2 to assist various research ideas and comprehensive comparison. To address MetaBBO's inherent efficiency issue, we have optimized low-level implementation of MetaBox-v2 to support parallel meta-training and evaluation, which reduces the running cost from days to hours. More importantly, we have optimized MetaBox-v2's sourcecode to support sufficient development flexbility, with clear and sound tutotials correspondingly. Enjoy your journey of learning and using MetaBBO from here!
Important
Below we install a cpu-version torch for you, if you need install any other versions,
see torch and replace the corresponding installation instruction below.
## create a venv
conda create -n metaevobox_env python=3.11.5 -y
conda activate metaevobox_env
## install pytorch
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cpu
## install metabox
pip install metaevoboxcreate your_dir, then create a your_train.py file in your_dir, write following codes into your_train.py.
from metaevobox import Config, Trainer
# import meta-level agent of MetaBBO you want to meta-train
from metaevobox.baseline.metabbo import GLEET
# import low-level BBO optimizer of MetaBBO you want to meta-train
from metaevobox.environment.optimizer import GLEET_Optimizer
from metaevobox.environment.problem.utils import construct_problem_set
# put user-specific configuration
config = {'train_problem': 'bbob-10D', # specify the problem set you want to train your MetaBBO
'train_batch_size': 16,
'train_parallel_mode':'subproc', # choose parallel training mode
}
config = Config(config)
# construct dataset
config, datasets = construct_problem_set(config)
# initialize your MetaBBO's meta-level agent & low-level optimizer
gleet = GLEET(config)
gleet_opt = GLEET_Optimizer(config)
trainer = Trainer(config, gleet, gleet_opt, datasets)
trainer.train()If you want to check out the visualized information of the training progress, run following code to start training logger.
cd your_dir/output/tensorboard
tensorboard --logdir=./from metaevobox import Config, Tester, get_baseline
# import meta-level agent of MetaBBO you want to test
from metaevobox.baseline.metabbo import GLEET
# import low-level BBO optimizer of MetaBBO you want to test
from metaevobox.environment.optimizer import GLEET_Optimizer
# import other baselines you want to compare with your MetaBBO
from metaevobox.baseline.bbo import CMAES, SHADE
from metaevobox.environment.problem.utils import construct_problem_set
# specify your configuration
config = {
'test_problem':'bbob-10D', # specify the problem set you want to benchmark
'test_batch_size':16,
'test_difficulty':'difficult', # this is a train-test split mode
'baselines':{
# your MetaBBO
'GLEET':{
'agent': 'GLEET',
'optimizer': GLEET_Optimizer,
'model_load_path': None, # by default is None, we will load a built-in pre-trained checkpoint for you.
},
# Other baselines to compare
'SHADE':{'optimizer': SHADE},
'CMAES':{'optimizer':CMAES},
},
}
config = Config(config)
# load test dataset
config, datasets = construct_problem_set(config)
# initialize all baselines to compare (yours + others)
baselines, config = get_baseline(config)
# initialize tester
tester = Tester(config, baselines, datasets)
# test
tester.test()By default, MetaBox would automatically generate various visualized experimental results in your_dir/output/test/, enjoy these useful analysis results!
We sincerely suggest researchers with interests to check out Online Documentation for further flexible usege of MetaBox-v2, such as implementing your own MetaBBO, customized experimental design & analysis, using pre-collected metadata and seamless API calling with other famous optimization repos.
| Type | Problem Set | Description | ||
|---|---|---|---|---|
| Name | Paper | Code | ||
| Single-Objective Optimization | bbob | Paper | Code | bbob is based on CoCo platform, which includes 96 representative single-objective synthetic problem instances. These instances all originate from the same group of 24 objective functions (CoCo-BBOB), which have been used in many papers and widely accepted as golden standard for evaluating the robustbess of an optimizer. In MetaBox-v2, bbob includes 4 subsets: bbob-10D, bbob-30D, bbob-noisy-10D and bbob-noisy-30D, each of them contains the 24 functions. "noisy" here indicates that the function's objective value is added with a gaussian noise before it is output, which significantly increase the solving difficulty. |
| bbob-surrogate | Paper | Code | bbob-surrogate includes 72 problem instances, each of which is a surrogate model. In specific, it can be divided into 3 subsets: bbob-surrogate-2D, bbob-surrogate-5D and bbob-surrogate-10D, each of which corresponds to 24 bbob problems. We first train KAN or MLP networks to fit 24 black box functions from bbob, then use the one with more accuracy as the surrogate model. This set is mainly developed for users who aims at exploring the potential of surrogate model in MetaBBO. | |
| hpo-b | Paper | Code | hpo-b is an autoML hyper-parameter optimization benchmark which includes a wide range of hyperparameter optimization tasks for 16 different model types (e.g., SVM, XGBoost, etc.), resulting in a total of 935 problem instances. The dimension of these problem instances range from 2 to 16. We also note that HPO-B represents problems with ill-conditioned landscape such as huge flattern. | |
| uav | Paper | Code | uav provides 56 terrain-based landscapes as realistic Unmanned Aerial Vehicle(UAV) path planning problems, each of which is 30D. The objective is to select given number of path nodes (x,y,z coordinates) from the 3D space, so the the UAV could fly as shortly as possible in a collision-free way. | |
| ne (large-scale) |
Paper | Code | This problem set is based on the neuroevolution interfaces in EvoX. The goal is to optimize the parameters of neural network-based RL agents for a series of Robotic Control tasks. We pre-define 11 control tasks (e.g., swimmer, ant, walker2D etc.), and 6 MLP structures with 0~5 hidden layers. The combinations of task & network structure result in 66 problem instances, which feature extremely high-dimensional problems (>=1000D). | |
| protein | Paper | Code | protein-docking benchmark, where the objective is to minimize the Gibbs free energy resulting from protein-protein interaction between a given complex and any other conformation. We select 28 protein complexes and randomly initialize 10 starting points for each complex, resulting in 280 problem instances. To simplify the problem structure, we only optimize 12 interaction points in a complex instance (12D problem). | |
| lsgo (large-scale) |
Paper | Code |
lsgo contains 20 large-scale problems instances (>=905D. <=1000D):
|
|
| Multi-Objective Optimization | moo-synthetic |
ZDT UF DTLZ WFG |
Code | moo-synthetic is constructed by mixing 4 well-known multi-objective problem sets: ZDT, UF, DTLZ and WFG. In total, we have constructed 187 problem instances. Their objective numbers range from 2~10, dimensions range from 6D~38D. |
| moo-uav |
paper |
Code | We decompose the objective value of instances in uav into 5 separate objectives, which results in 56 30D realistic 5-objective problem instances. | |
| Multi-Model Optimization | mmo | Paper | Code | mmo is based on CEC2013LSGO benchmark and specially crafeted for multi-modal optimization, which includes 20 synthetic problem instances covering various dimensions (1D~20D), each with varied number of (1 ~ 216) global optima. Among them, F1 to F5 are simple uni-modal functions, F6 to F10 are dimension-scalable functions with multiple global optima, and F11 to F20 are complex composition functions with challenging landscapes. |
| Multi-Task Optimization | cec2017mto | Paper | Code | cec2017mto comprises 9 multi-task problem instances, each of which contains two basic problems. Optional basic problems include Shpere, Rosenbrock, Ackley, Rastrigin, Griewank, Weierstrass and Schwefel, with dimension ranging from 25D~50D. |
| wcci2020 | Paper | Code | wcci2020 comprises 10 multi-task problem instances, each of which contains 50 basic problems. Optional basic problems include Shpere, Rosenbrock, Ackley, Rastrigin, Griewank, Weierstrass and Schwefel, which are all 50D. | |
| augmented-wcci2020 | Paper | Code | augmented-wcci2020 comprises 127 multi-task problems, each of which optinally contains 1~7 basic problems. Optional basic problems include Shpere, Rosenbrock, Ackley, Rastrigin, Griewank, Weierstrass and Schwefel, which are all 50D. | |
The PDF version of the paper is available here. If you find our MetaBox useful, please cite it in your publications or projects.
@inproceedings{metabox,
author={Ma, Zeyuan and Guo, Hongshu and Chen, Jiacheng and Li, Zhenrui and Peng, Guojun and Gong, Yue-Jiao and Ma, Yining and Cao, Zhiguang},
title={MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learning},
booktitle = {Advances in Neural Information Processing Systems},
year={2023},
volume = {36}
}
Here is our homepage and github. 🥰🥰🥰Please feel free to contact us—any suggestions are welcome!
If you have any question or want to contact us：
- 🌱Fork, Add, and Merge
- ❓️Report an issue
- 📧Contact WenJie Qiu (wukongqwj@gmail.com)
- 🚨We warmly invite you to join our QQ group for further communication (Group Number: 952185139).