This is the post-processing tool of the project Iterative Optimization Heuristics Profiler (IOHprofiler). This tool provides a web-based interface to analyze and visualization the benchmark data, collected from previous experiments. Importantly, we do support the widely used COCO data format (aka. Black-Box Optimization Benchmarking).
This tool is mainly built on R package Shiny, plotly and Rcpp. To use this tool, three options are available:
The details on the experimentation and post-processing tool can be found on arXiv.org.
To install the IOHProfiler R-package, please install R environment first. The binary file and installation manual for R can be found here https://cran.r-project.org/. Please start up the R console, which can be done (in case you’re not familiar with R) by either executing command R in your system terminal or open the R application.
The IOHanalyzer is now available on CRAN! To use the CRAN version, simply use the command in your R console:
The CRAN version is a stable version of the IOHanalyzer, but since this package is under continuous developement, it might make sense to use a developement version instead. The version in the master-branch of this repository is updated relatively frequently, and might be more usefull than the CRAN version. However, this version is likely to be more prone to errors. Please don’t hesitate to add such errors to the issues section on this github page.
To use the developement version, ‘devtools’ package is needed to install the sorftware. To install this package, please copy-paste and execute the following command into the R console
Error messages will be shown in your R console if there is any installation issue. Now, the IOHanalyzer package can be installed and loaded using the following commands:
This will install the package and all required dependencies. The GUI can be acessed using the command:
Alternatively, you can clone the source-code directely. This software is mainly written in R. To run it directly from the source code, please install R environment first. The binary file and installation manual for R can be found here https://cran.r-project.org/.
After R environment is correctly installed on you machine, several R packages are needed to execute the sorftware. Please start up the R console, which can be done (in case you’re not familiar with R) by either executing command R in your system terminal or open the R application. Once it is done, please copy-paste and execute the following commands into the R console to install all depedencies.
Error messages will be shown in your R console if there is any installation issue.
To allow for downloading of plots, orca[https://github.com/plotly/orca] and inkscape[https://inkscape.org/release/inkscape-0.92.4/] are needed.
Then, please clone (or downlaod) this repository into your own system. To clone the repository, please execute one of the the following commands in your system console (terminal):
> git clone git@github.com:IOHprofiler/IOHanalyzer.git> git clone https://github.com/IOHprofiler/IOHanalyzer.gitTo download, please click the green download button on this page.
To start the post-processing tool, please execute the following commands in the R console:
Alternatively, we have built a server to put this tool online, which is currently hosted in Leiden Institute of Advanced Computer Science, Leiden University. The server can be accessed via http://iohprofiler.liacs.nl.
Data preparation is fairly easy for this tool. Just compress the data folder obtained from the experiment into a zip file and uploaded it. Currently, we support two data formats:
In addition to the graphical user interface, it is possible to directly call several procedures to analyze the data.
> dsList <- read_dir('/path/to/data/folder', format = 'COCO')
> dsList
DataSetList:
1: DataSet((1+1)-Cholesky-CMA on f1 2D)
2: DataSet((1+1)-Cholesky-CMA on f1 5D)
3: DataSet((1+1)-Cholesky-CMA on f1 10D)
4: DataSet((1+1)-Cholesky-CMA on f1 20D)
5: DataSet((1+1)-Cholesky-CMA on f10 2D)
6: DataSet((1+1)-Cholesky-CMA on f10 5D)
7: DataSet((1+1)-Cholesky-CMA on f10 10D)
8: DataSet((1+1)-Cholesky-CMA on f10 20D)
9: DataSet((1+1)-Cholesky-CMA on f11 2D)
10: DataSet((1+1)-Cholesky-CMA on f11 5D)The return value is a list of DataSets. Each data set consists of:
summary:> summary(dsList[[1]])
DataSet Object:
Source: COCO
Algorithm: (1+1)-Cholesky-CMA
Function ID: 1
Dimension: 2D
80 instance found: 1,2,3,4,5,6,7,...,73,74,75,76,77,78,79,80
runtime summary:
target mean median sd 2% 5% 10% 25% 50% 75% 90% 95% 98% ERT runs ps
1: 7.010819e+01 1.0000 1.0 0.0000000 1 1 1 1 1 1 1 1 1 1.0000 80 1.0000
2: 6.642132e+01 1.0125 1.0 0.1118034 1 1 1 1 1 1 1 1 1 1.0125 80 1.0000
3: 6.298712e+01 1.1125 1.0 0.8999824 1 1 1 1 1 1 1 1 1 1.1125 80 1.0000
4: 6.254396e+01 1.1375 1.0 0.9242684 1 1 1 1 1 1 1 1 2 1.1375 80 1.0000
5: 6.173052e+01 1.2000 1.0 1.1295793 1 1 1 1 1 1 1 1 3 1.2000 80 1.0000
---
1478: 9.473524e-10 182.6000 182.0 24.0894168 145 145 145 145 181 195 195 210 210 3039.6000 5 0.0625
1479: 2.759535e-10 192.0000 188.5 13.5892114 181 181 181 181 182 195 210 210 210 3799.5000 4 0.0500
1480: 2.463310e-10 195.6667 195.0 14.0118997 182 182 182 182 195 195 210 210 210 5066.0000 3 0.0375
1481: 5.223910e-11 196.0000 196.0 19.7989899 182 182 182 182 182 210 210 210 210 7599.0000 2 0.0250
1482: 1.638512e-11 210.0000 210.0 NA 210 210 210 210 210 210 210 210 210 15198.0000 1 0.0125
function value summary:
algId runtime runs mean median sd 2% 5% 10%
1: (1+1)-Cholesky-CMA 1 80 2.019164e+01 1.313629e+01 1.872005e+01 3.976009e-01 1.186758e+00 2.267215e+00
2: (1+1)-Cholesky-CMA 2 80 1.672518e+01 1.171157e+01 1.626487e+01 3.412261e-01 5.649713e-01 1.313165e+00
3: (1+1)-Cholesky-CMA 3 80 1.341813e+01 7.960940e+00 1.466877e+01 1.177243e-01 1.778454e-01 5.553242e-01
4: (1+1)-Cholesky-CMA 4 80 1.100825e+01 6.439678e+00 1.261937e+01 9.206390e-02 1.492565e-01 5.221506e-01
5: (1+1)-Cholesky-CMA 5 80 9.326633e+00 5.492333e+00 1.213908e+01 7.246194e-02 1.031687e-01 3.482309e-01
---
90: (1+1)-Cholesky-CMA 229 80 6.321111e-09 4.609703e-09 8.318463e-09 1.648124e-10 9.137825e-10 1.336938e-09
91: (1+1)-Cholesky-CMA 231 80 5.823366e-09 4.609703e-09 7.618201e-09 1.648124e-10 9.137825e-10 1.336938e-09
92: (1+1)-Cholesky-CMA 238 80 5.549184e-09 4.506106e-09 7.258216e-09 1.648124e-10 9.137825e-10 1.336938e-09
93: (1+1)-Cholesky-CMA 251 80 4.902827e-09 4.506106e-09 2.863671e-09 1.648124e-10 9.137825e-10 1.336938e-09
94: (1+1)-Cholesky-CMA 257 80 4.737548e-09 4.461953e-09 2.526087e-09 1.648124e-10 9.137825e-10 1.336938e-09
25% 50% 75% 90% 95% 98%
1: 5.166078e+00 1.313629e+01 2.922941e+01 5.004596e+01 5.895140e+01 6.442948e+01
2: 3.506037e+00 1.171157e+01 2.470745e+01 3.956328e+01 5.012620e+01 6.002245e+01
3: 2.786120e+00 7.960940e+00 2.103983e+01 3.251354e+01 4.030147e+01 5.690653e+01
4: 1.284869e+00 6.439678e+00 1.636467e+01 2.453239e+01 3.043178e+01 4.718666e+01
5: 9.886750e-01 5.492333e+00 1.317642e+01 2.216466e+01 2.913811e+01 4.718666e+01
---
90: 2.799292e-09 4.609703e-09 6.554838e-09 8.992768e-09 1.235938e-08 2.771700e-08
91: 2.799292e-09 4.609703e-09 6.321677e-09 8.916086e-09 9.326487e-09 1.785183e-08
92: 2.799292e-09 4.506106e-09 6.099065e-09 8.854852e-09 9.180278e-09 1.041480e-08
93: 2.799292e-09 4.506106e-09 6.099065e-09 8.659254e-09 8.982758e-09 9.377720e-09
94: 2.799292e-09 4.461953e-09 5.970504e-09 8.547357e-09 8.925389e-09 9.234526e-09
Attributes: names, class, funcId, DIM, Precision, algId, comment, datafile, instance, maxRT, finalFV, src, maximizationget_RT_summary (RunTime) and get_FV_summary (FunctionValue):> get_RT_summary(dsList[[1]], ftarget = 1e-1)
algId target mean median sd 2% 5% 10% 25% 50% 75% 90% 95% 98% ERT runs ps
1: (1+1)-Cholesky-CMA 0.09986529 36.55 37.5 17.11236 4 5 14 22 37 49 57 67 68 36.55 80 1> get_FV_summary(dsList[[1]], runtime = 100)
algId runtime runs mean median sd 2% 5% 10% 25% 50% 75% 90% 95% 98%
1: (1+1)-Cholesky-CMA 100 80 0.0005886303 8.307195e-05 0.001165899 7.233939e-07 3.708489e-06 8.776148e-06 2.409768e-05 8.307195e-05 0.0004779061 0.001991386 0.002923357 0.004207976get_RT_sample (RunTime) and get_FV_sample (FunctionValue):> get_RT_sample(dsList[[1]], ftarget = 1e-1, output = 'long')
algId target run RT
1: (1+1)-Cholesky-CMA 0.1 1 69
2: (1+1)-Cholesky-CMA 0.1 2 39
3: (1+1)-Cholesky-CMA 0.1 3 38
4: (1+1)-Cholesky-CMA 0.1 4 34
5: (1+1)-Cholesky-CMA 0.1 5 67
---
76: (1+1)-Cholesky-CMA 0.1 76 52
77: (1+1)-Cholesky-CMA 0.1 77 22
78: (1+1)-Cholesky-CMA 0.1 78 26
79: (1+1)-Cholesky-CMA 0.1 79 33
80: (1+1)-Cholesky-CMA 0.1 80 25> get_FV_sample(dsList[[1]], runtime = 100, output = 'long')
algId runtime run f(x)
1: (1+1)-Cholesky-CMA 100 1 4.007000e-03
2: (1+1)-Cholesky-CMA 100 2 5.381801e-04
3: (1+1)-Cholesky-CMA 100 3 3.970844e-05
4: (1+1)-Cholesky-CMA 100 4 5.345724e-04
5: (1+1)-Cholesky-CMA 100 5 1.458869e-03
---
76: (1+1)-Cholesky-CMA 100 76 1.817090e-03
77: (1+1)-Cholesky-CMA 100 77 3.850201e-06
78: (1+1)-Cholesky-CMA 100 78 1.330411e-05
79: (1+1)-Cholesky-CMA 100 79 3.933669e-05
80: (1+1)-Cholesky-CMA 100 80 5.658113e-07Or output a wide format…
> get_RT_sample(dsList[[1]], ftarget = 1e-1, output = 'wide')
algId target run.1 run.2 run.3 run.4 run.5 run.6 run.7 run.8 run.9 run.10 run.11 run.12 run.13 run.14 run.15
1: (1+1)-Cholesky-CMA 0.1 69 39 38 34 67 3 36 41 14 30 41 47 31 48 53
run.16 run.17 run.18 run.19 run.20 run.21 run.22 run.23 run.24 run.25 run.26 run.27 run.28 run.29 run.30 run.31 run.32
1: 8 19 18 57 16 28 51 53 22 47 53 17 5 48 13 63 45
run.33 run.34 run.35 run.36 run.37 run.38 run.39 run.40 run.41 run.42 run.43 run.44 run.45 run.46 run.47 run.48 run.49
1: 15 24 46 65 44 71 52 31 17 18 45 19 5 37 50 33 24
run.50 run.51 run.52 run.53 run.54 run.55 run.56 run.57 run.58 run.59 run.60 run.61 run.62 run.63 run.64 run.65 run.66
1: 40 49 55 48 50 33 20 33 35 49 37 4 22 68 57 19 44
run.67 run.68 run.69 run.70 run.71 run.72 run.73 run.74 run.75 run.76 run.77 run.78 run.79 run.80
1: 38 48 31 14 41 50 67 21 43 52 22 26 33 25ggplot2 or plotly) using the provided plotting functions. The available functions are in the following style: Plot.{RT/FV/PAR}.{Plot_type}For more information on these functions, use the documentation available by executing the following type of command:
If you have any questions, comments, suggestions or pull requests, please don’t hesitate contacting us IOHprofiler@liacs.leidenuniv.nl!
The development team is:
When using IOHprofiler and parts thereof, please kindly cite this work as
Carola Doerr, Hao Wang, Furong Ye, Sander van Rijn, Thomas Bäck: IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics, arXiv e-prints:1810.05281, 2018.
@ARTICLE{IOHprofiler,
author = {Carola Doerr and Hao Wang and Furong Ye and Sander van Rijn and Thomas B{\"a}ck},
title = {{IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics}},
journal = {arXiv e-prints:1810.05281},
archivePrefix = "arXiv",
eprint = {1810.05281},
year = 2018,
month = oct,
keywords = {Computer Science - Neural and Evolutionary Computing},
url = {https://arxiv.org/abs/1810.05281}
}