Standard GA

Expected number of f-evaluations to reach target

Expected number of f-evaluations (ERT, lines) to reach fopt+∆f; median number of f-evaluations (+) to reach the most difficult target that was reached not always but at least once; maximum number of f-evaluations in any trial (×); interquartile range with median (notched boxes) of simulated runlengths to reach fopt+∆f; all values are divided by dimension and plotted as log10 values versus dimension. Shown is the ERT for targets just not reached by the artificial GECCO-BBOB-2009 best algorithm within the given budget k×DIM, where k is shown in the legend. Numbers above ERT-symbols indicate the number of trials reaching the respective target. The light thick line with diamonds indicates the respective best result from BBOB-2009 for the most difficult target. Slanted grid lines indicate a scaling with O(DIM) compared to O(1) when using the respective 2009 best algorithm.

ERT in number of function evaluations

5-D

#FEs/D 0.5 1.2 3 10 50 #succ
f1 2.5e+1:4.8 1.6e+1:7.6 1.0e-8:12 1.0e-8:12 1.0e-8:12 15/15
2.1 (2) 3.2 (2) 600 0/15
f2 1.6e+6:2.9 4.0e+5:11 4.0e+4:15 6.3e+2:58 1.0e-8:95 15/15
2.2 (3) 1.1 (0.9) 13 (11) 600 0/15
f3 1.6e+2:4.1 1.0e+2:15 6.3e+1:23 2.5e+1:73 1.0e+1:716 15/15
2.3 (2) 3.2 (4) 16 (29) 600 0/15
f4 2.5e+2:2.6 1.6e+2:10 1.0e+2:19 4.0e+1:65 1.6e+1:434 15/15
3.0 (3) 2.6 (3) 16 (27) 600 0/15
f5 6.3e+1:4.0 4.0e+1:10 1.0e-8:10 1.0e-8:10 1.0e-8:10 15/15
2.0 (1) 6.1 (6) 600 0/15
f6 1.0e+5:3.0 2.5e+4:8.4 1.0e+2:16 2.5e+1:54 2.5e-1:254 15/15
1.7 (3) 2.0 (5) 12 (26) 600 0/15
f7 1.6e+2:4.2 1.0e+2:6.2 2.5e+1:20 4.0e+0:54 1.0e+0:324 15/15
1.7 (0.8) 1.5 (1) 6.8 (4) 600 0/15
f8 1.0e+4:4.6 6.3e+3:6.8 1.0e+3:18 6.3e+1:54 1.6e+0:258 15/15
2.9 (1) 2.7 (3) 16 (14) 158 (190) 600 0/15
f9 2.5e+1:20 1.6e+1:26 1.0e+1:35 4.0e+0:62 1.6e-2:256 15/15
600 0/15
f10 2.5e+6:2.9 6.3e+5:7.0 2.5e+5:17 6.3e+3:54 2.5e+1:297 15/15
1.8 (2) 1.7 (3) 2.2 (3) 49 (73) 600 0/15
f11 1.0e+6:3.0 6.3e+4:6.2 6.3e+2:16 6.3e+1:74 6.3e-1:298 15/15
2.3 (3) 4.7 (4) 15 (51) 54 (63) 600 0/15
f12 4.0e+7:3.6 1.6e+7:7.6 4.0e+6:19 1.6e+4:52 1.0e+0:268 15/15
1.5 (1) 3.3 (2) 21 (15) 600 0/15
f13 1.0e+3:2.8 6.3e+2:8.4 4.0e+2:17 6.3e+1:52 6.3e-2:264 15/15
3.2 (2) 4.4 (4) 11 (12) 600 0/15
f14 1.6e+1:3.0 1.0e+1:10 6.3e+0:15 2.5e-1:53 1.0e-5:251 15/15
1.8 (0.9) 0.71 (0.8) 3.7 (5) 600 0/15
f15 1.6e+2:3.0 1.0e+2:13 6.3e+1:24 4.0e+1:55 1.6e+1:289 5/5
2.2 (3) 1.4 (0.8) 14 (21) 71 (87) 600 0/15
f16 4.0e+1:4.8 2.5e+1:16 1.6e+1:46 1.0e+1:120 4.0e+0:334 15/15
1.4 (1) 1.1 (3) 2.3 (4) 4.1 (3) 5.9 (9) 4/15
f17 1.0e+1:5.2 6.3e+0:26 4.0e+0:57 2.5e+0:110 6.3e-1:412 15/15
3.8 (3) 2.0 (2) 12 (19) 77 (124) 600 0/15
f18 6.3e+1:3.4 4.0e+1:7.2 2.5e+1:20 1.6e+1:58 1.6e+0:318 15/15
1.4 (1.0) 1.8 (0.8) 3.8 (3) 9.5 (12) 600 0/15
f19 1.6e-1:172 1.0e-1:242 6.3e-2:675 4.0e-2:3078 2.5e-2:4946 15/15
600 0/15
f20 6.3e+3:5.1 4.0e+3:8.4 4.0e+1:15 2.5e+0:69 1.0e+0:851 15/15
2.5 (5) 2.6 (3) 47 (39) 129 (230) 600 0/15
f21 4.0e+1:3.9 2.5e+1:11 1.6e+1:31 6.3e+0:73 1.6e+0:347 5/5
2.3 (3) 1.9 (3) 1.4 (1) 18 (21) 600 0/15
f22 6.3e+1:3.6 4.0e+1:15 2.5e+1:32 1.0e+1:71 1.6e+0:341 5/5
2.1 (3) 1.9 (2) 2.7 (2) 8.1 (19) 600 0/15
f23 1.0e+1:3.0 6.3e+0:9.0 4.0e+0:33 2.5e+0:84 1.0e+0:518 15/15
1.5 (1) 2.3 (2) 3.2 (6) 12 (23) 600 0/15
f24 6.3e+1:15 4.0e+1:37 4.0e+1:37 2.5e+1:118 1.6e+1:692 15/15
6.6 (12) 50 (56) 50 (59) 600 0/15

20-D

#FEs/D 0.5 1.2 3 10 50 #succ
f1 2.5e+1:4.8 1.6e+1:7.6 1.0e-8:12 1.0e-8:12 1.0e-8:12 15/15
2.1 (2) 3.2 (2) 600 0/15
f2 1.6e+6:2.9 4.0e+5:11 4.0e+4:15 6.3e+2:58 1.0e-8:95 15/15
2.2 (3) 1.1 (0.9) 13 (11) 600 0/15
f3 1.6e+2:4.1 1.0e+2:15 6.3e+1:23 2.5e+1:73 1.0e+1:716 15/15
2.3 (2) 3.2 (4) 16 (29) 600 0/15
f4 2.5e+2:2.6 1.6e+2:10 1.0e+2:19 4.0e+1:65 1.6e+1:434 15/15
3.0 (3) 2.6 (3) 16 (27) 600 0/15
f5 6.3e+1:4.0 4.0e+1:10 1.0e-8:10 1.0e-8:10 1.0e-8:10 15/15
2.0 (1) 6.1 (6) 600 0/15
f6 1.0e+5:3.0 2.5e+4:8.4 1.0e+2:16 2.5e+1:54 2.5e-1:254 15/15
1.7 (3) 2.0 (5) 12 (26) 600 0/15
f7 1.6e+2:4.2 1.0e+2:6.2 2.5e+1:20 4.0e+0:54 1.0e+0:324 15/15
1.7 (0.8) 1.5 (1) 6.8 (4) 600 0/15
f8 1.0e+4:4.6 6.3e+3:6.8 1.0e+3:18 6.3e+1:54 1.6e+0:258 15/15
2.9 (1) 2.7 (3) 16 (14) 158 (190) 600 0/15
f9 2.5e+1:20 1.6e+1:26 1.0e+1:35 4.0e+0:62 1.6e-2:256 15/15
600 0/15
f10 2.5e+6:2.9 6.3e+5:7.0 2.5e+5:17 6.3e+3:54 2.5e+1:297 15/15
1.8 (2) 1.7 (3) 2.2 (3) 49 (73) 600 0/15
f11 1.0e+6:3.0 6.3e+4:6.2 6.3e+2:16 6.3e+1:74 6.3e-1:298 15/15
2.3 (3) 4.7 (4) 15 (51) 54 (63) 600 0/15
f12 4.0e+7:3.6 1.6e+7:7.6 4.0e+6:19 1.6e+4:52 1.0e+0:268 15/15
1.5 (1) 3.3 (2) 21 (15) 600 0/15
f13 1.0e+3:2.8 6.3e+2:8.4 4.0e+2:17 6.3e+1:52 6.3e-2:264 15/15
3.2 (2) 4.4 (4) 11 (12) 600 0/15
f14 1.6e+1:3.0 1.0e+1:10 6.3e+0:15 2.5e-1:53 1.0e-5:251 15/15
1.8 (0.9) 0.71 (0.8) 3.7 (5) 600 0/15
f15 1.6e+2:3.0 1.0e+2:13 6.3e+1:24 4.0e+1:55 1.6e+1:289 5/5
2.2 (3) 1.4 (0.8) 14 (21) 71 (87) 600 0/15
f16 4.0e+1:4.8 2.5e+1:16 1.6e+1:46 1.0e+1:120 4.0e+0:334 15/15
1.4 (1) 1.1 (3) 2.3 (4) 4.1 (3) 5.9 (9) 4/15
f17 1.0e+1:5.2 6.3e+0:26 4.0e+0:57 2.5e+0:110 6.3e-1:412 15/15
3.8 (3) 2.0 (2) 12 (19) 77 (124) 600 0/15
f18 6.3e+1:3.4 4.0e+1:7.2 2.5e+1:20 1.6e+1:58 1.6e+0:318 15/15
1.4 (1.0) 1.8 (0.8) 3.8 (3) 9.5 (12) 600 0/15
f19 1.6e-1:172 1.0e-1:242 6.3e-2:675 4.0e-2:3078 2.5e-2:4946 15/15
600 0/15
f20 6.3e+3:5.1 4.0e+3:8.4 4.0e+1:15 2.5e+0:69 1.0e+0:851 15/15
2.5 (5) 2.6 (3) 47 (39) 129 (230) 600 0/15
f21 4.0e+1:3.9 2.5e+1:11 1.6e+1:31 6.3e+0:73 1.6e+0:347 5/5
2.3 (3) 1.9 (3) 1.4 (1) 18 (21) 600 0/15
f22 6.3e+1:3.6 4.0e+1:15 2.5e+1:32 1.0e+1:71 1.6e+0:341 5/5
2.1 (3) 1.9 (2) 2.7 (2) 8.1 (19) 600 0/15
f23 1.0e+1:3.0 6.3e+0:9.0 4.0e+0:33 2.5e+0:84 1.0e+0:518 15/15
1.5 (1) 2.3 (2) 3.2 (6) 12 (23) 600 0/15
f24 6.3e+1:15 4.0e+1:37 4.0e+1:37 2.5e+1:118 1.6e+1:692 15/15
6.6 (12) 50 (56) 50 (59) 600 0/15
f1 6.3e+1:24 4.0e+1:42 1.0e-8:43 1.0e-8:43 1.0e-8:43 15/15
746 (838) 1200 0/15
f2 4.0e+6:29 2.5e+6:42 1.0e+5:65 1.0e+4:207 1.0e-8:412 15/15
1.6 (2) 5.1 (4) 1200 0/15
f3 6.3e+2:33 4.0e+2:44 1.6e+2:109 1.0e+2:255 2.5e+1:3277 15/15
17 (37) 1200 0/15
f4 6.3e+2:22 4.0e+2:91 2.5e+2:250 1.6e+2:332 6.3e+1:1927 15/15
253 (182) 1200 0/15
f5 2.5e+2:19 1.6e+2:34 1.0e-8:41 1.0e-8:41 1.0e-8:41 15/15
9.4 (11) 251 (301) 1200 0/15
f6 2.5e+5:16 6.3e+4:43 1.6e+4:62 1.6e+2:353 1.6e+1:1078 15/15
38 (77) 399 (273) 1200 0/15
f7 1.0e+3:11 4.0e+2:39 2.5e+2:74 6.3e+1:319 1.0e+1:1351 15/15
1.8 (2) 45 (39) 54 (49) 1200 0/15
f8 4.0e+4:19 2.5e+4:35 4.0e+3:67 2.5e+2:231 1.6e+1:1470 15/15
313 (368) 1200 0/15
f9 1.0e+2:357 6.3e+1:560 4.0e+1:684 2.5e+1:756 1.0e+1:1716 15/15
1200 0/15
f10 1.6e+6:15 1.0e+6:27 4.0e+5:70 6.3e+4:231 4.0e+3:1015 15/15
49 (47) 209 (530) 1200 0/15
f11 4.0e+4:11 2.5e+3:27 1.6e+2:313 1.0e+2:481 1.0e+1:1002 15/15
1.8 (2) 5.3 (5) 1200 0/15
f12 1.0e+8:23 6.3e+7:39 2.5e+7:76 4.0e+6:209 1.0e+1:1042 15/15
136 (163) 460 (453) 1200 0/15
f13 1.6e+3:28 1.0e+3:64 6.3e+2:79 4.0e+1:211 2.5e+0:1724 15/15
188 (307) 1200 0/15
f14 2.5e+1:15 1.6e+1:42 1.0e+1:75 1.6e+0:219 6.3e-4:1106 15/15
209 (123) 405 (428) 1200 0/15
f15 6.3e+2:15 4.0e+2:67 2.5e+2:292 1.6e+2:846 1.0e+2:1671 15/15
76 (162) 265 (144) 1200 0/15
f16 4.0e+1:26 2.5e+1:127 1.6e+1:540 1.6e+1:540 1.0e+1:1384 15/15
3.9 (2) 41 (132) 1200 0/15
f17 1.6e+1:11 1.0e+1:63 6.3e+0:305 4.0e+0:468 1.0e+0:1030 15/15
18 (29) 133 (170) 1200 0/15
f18 4.0e+1:116 2.5e+1:252 1.6e+1:430 1.0e+1:621 4.0e+0:1090 15/15
21 (21) 1200 0/15
f19 1.6e-1:2.5e5 1.0e-1:3.4e5 6.3e-2:3.4e5 4.0e-2:3.4e5 2.5e-2:3.4e5 3/15
1200 0/15
f20 1.6e+4:38 1.0e+4:42 2.5e+2:62 2.5e+0:250 1.6e+0:2536 15/15
466 (764) 1200 0/15
f21 6.3e+1:36 4.0e+1:77 4.0e+1:77 1.6e+1:456 4.0e+0:1094 15/15
1200 0/15
f22 6.3e+1:45 4.0e+1:68 4.0e+1:68 1.6e+1:231 6.3e+0:1219 15/15
1200 0/15
f23 6.3e+0:29 4.0e+0:118 2.5e+0:306 2.5e+0:306 1.0e+0:1614 15/15
1.8 (0.9) 12 (15) 1200 0/15
f24 2.5e+2:208 1.6e+2:918 1.0e+2:6628 6.3e+1:9885 4.0e+1:31629 15/15
1200 0/15

Expected running time (ERT in number of function evaluations) divided by the best ERT measured during BBOB-2009. The ERT and in braces, as dispersion measure, the half difference between 90 and 10%-tile of bootstrapped run lengths appear in the second row of each cell, the best ERT in the first. The different target ∆f-values are shown in the top row. #succ is the number of trials that reached the (final) target fopt+ 10−8. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Bold entries are statistically significantly better (according to the rank-sum test) compared to the best algorithm in BBOB-2009, with p = 0.05 or p = 10−k when the number k > 1 is following the ↓ symbol, with Bonferroni correction by the number of functions.

Empirical cumulative distribution functions (ECDF)

Separable functions in 5-D

Misc. moderate functions in 5-D

Ill-conditioned functions in 5-D

Multi-modal functions in 5-D

Weak structure functions in 5-D

All functions in 5-D

Separable functions in 20-D

Misc. moderate functions in 20-D

Ill-conditioned functions in 20-D

Multi-modal functions in 20-D

Weak structure functions in 20-D

All functions in 20-D

Empirical cumulative distribution functions (ECDF), plotting the fraction of trials with an outcome not larger than the respective value on the x-axis. Left subplots: ECDF of number of function evaluations (FEvals) divided by search space dimension D, to fall below fopt+∆f where ∆f is the target just not reached by the GECCO-BBOB-2009 best algorithm within a budget of k×DIM evaluations, where k is the first value in the legend. Legends indicate for each target the number of functions that were solved in at least one trial within the displayed budget. Right subplots: ECDF of the best achieved ∆f for running times of 0.5D, 1.2D, 3D, 10D, 100D, 1000D,… function evaluations (from right to left cycling cyan-magenta-black...) and final ∆f-value (red), where ∆f and Df denote the difference to the optimal function value. Light brown lines in the background show ECDFs for the most difficult target of all algorithms benchmarked during BBOB-2009.

ERT loss ratios

f1f24 in 5-D, maxFE/D=120

#FEs/D best 10 % 25 % med 75 % 90 %
RLUS/D 1e2 1e2 1e2 1e2 1e2 1e2
2 0.71 1.4 2.1 2.9 4.9 10
10 1.6 3.1 3.8 5.1 6.8 50
100 4.2 12 20 26 32 95
1e3 15 55 87 1.4e2 2.2e2 9.5e2

f1f24 in 20-D, maxFE/D=60

#FEs/D best 10 % 25 % med 75 % 90 %
RLUS/D 60 60 60 60 60 60
2 1.0 3.6 11 37 40 40
10 7.0 7.4 15 1.5e2 2.0e2 2.0e2
100 17 51 91 2.4e2 2.0e3 2.0e3

ERT loss ratio versus the budget in number of f-evaluations divided by dimension. For each given budget FEvals, the target value ft is computed as the best target f-value reached within the budget by the given algorithm. Shown is then the ERT to reach ft for the given algorithm or the budget, if the GECCO-BBOB-2009 best algorithm reached a better target within the budget, divided by the best ERT  seen in GECCO-BBOB-2009 to reach ft. Line: geometric mean. Box-Whisker error bar: 25-75%-ile with median (box), 10-90%-ile (caps), and minimum and maximum ERT loss ratio (points). The vertical line gives the maximal number of function evaluations in a single trial in this function subset. See also the following figure for results on each function subgroup.

Separable functions in 5-D and 20-D

Moderate functions in 5-D and 20-D

Ill-conditioned functions in 5-D and 20-D

Multi-modal functions in 5-D and 20-D

Weak structure functions in 5-D and 20-D

ERT loss ratios (see the previous figure for details). Each cross (+) represents a single function, the line is the geometric mean.