Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests
Non-parametric tests can determine the better of two stochastic optimization algorithms when benchmarking results are ordinal—like the final fitness values of multiple trials—but for many benchmarks, a trial can also terminate once it reaches a prespecified target value. In such cases, both the time...
| Main Authors: | , , |
|---|---|
| Other Authors: | |
| Format: | Journal Article |
| Language: | English |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://hdl.handle.net/10356/174585 |
| _version_ | 1824454931229179904 |
|---|---|
| author | Price, Kenneth V. Kumar, Abhishek Suganthan, Ponnuthurai Nagaratnam |
| author2 | School of Electrical and Electronic Engineering |
| author_facet | School of Electrical and Electronic Engineering Price, Kenneth V. Kumar, Abhishek Suganthan, Ponnuthurai Nagaratnam |
| author_sort | Price, Kenneth V. |
| collection | NTU |
| description | Non-parametric tests can determine the better of two stochastic optimization algorithms when benchmarking results are ordinal—like the final fitness values of multiple trials—but for many benchmarks, a trial can also terminate once it reaches a prespecified target value. In such cases, both the time that a trial takes to reach the target value (or not) and its final fitness value characterize its outcome. This paper describes how trial-based dominance can totally order this two-variable dataset of outcomes so that traditional non-parametric methods can determine the better of two algorithms when one is faster, but less accurate than the other, i.e. when neither algorithm dominates. After describing trial-based dominance, we outline its benefits. We subsequently review other attempts to compare stochastic optimizers, before illustrating our method with the Mann-Whitney U test. Simulations demonstrate that “U-scores” are much more effective than dominance when tasked with identifying the better of two algorithms. We validate U-scores by having them determine the winners of the CEC 2022 competition on single objective, bound-constrained numerical optimization. |
| first_indexed | 2025-02-19T03:30:09Z |
| format | Journal Article |
| id | ntu-10356/174585 |
| institution | Nanyang Technological University |
| language | English |
| last_indexed | 2025-02-19T03:30:09Z |
| publishDate | 2024 |
| record_format | dspace |
| spelling | ntu-10356/1745852024-04-05T15:41:55Z Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests Price, Kenneth V. Kumar, Abhishek Suganthan, Ponnuthurai Nagaratnam School of Electrical and Electronic Engineering Engineering Two-variable non-parametric tests Evolutionary algorithms Non-parametric tests can determine the better of two stochastic optimization algorithms when benchmarking results are ordinal—like the final fitness values of multiple trials—but for many benchmarks, a trial can also terminate once it reaches a prespecified target value. In such cases, both the time that a trial takes to reach the target value (or not) and its final fitness value characterize its outcome. This paper describes how trial-based dominance can totally order this two-variable dataset of outcomes so that traditional non-parametric methods can determine the better of two algorithms when one is faster, but less accurate than the other, i.e. when neither algorithm dominates. After describing trial-based dominance, we outline its benefits. We subsequently review other attempts to compare stochastic optimizers, before illustrating our method with the Mann-Whitney U test. Simulations demonstrate that “U-scores” are much more effective than dominance when tasked with identifying the better of two algorithms. We validate U-scores by having them determine the winners of the CEC 2022 competition on single objective, bound-constrained numerical optimization. Published version Open Access funding provided by the Qatar National Library 2024-04-03T04:45:44Z 2024-04-03T04:45:44Z 2023 Journal Article Price, K. V., Kumar, A. & Suganthan, P. N. (2023). Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests. Swarm and Evolutionary Computation, 78, 101287-. https://dx.doi.org/10.1016/j.swevo.2023.101287 2210-6502 https://hdl.handle.net/10356/174585 10.1016/j.swevo.2023.101287 2-s2.0-85150777315 78 101287 en Swarm and Evolutionary Computation © 2023 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). application/pdf |
| spellingShingle | Engineering Two-variable non-parametric tests Evolutionary algorithms Price, Kenneth V. Kumar, Abhishek Suganthan, Ponnuthurai Nagaratnam Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_full | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_fullStr | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_full_unstemmed | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_short | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_sort | trial based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non parametric tests |
| topic | Engineering Two-variable non-parametric tests Evolutionary algorithms |
| url | https://hdl.handle.net/10356/174585 |
| work_keys_str_mv | AT pricekennethv trialbaseddominanceforcomparingboththespeedandaccuracyofstochasticoptimizerswithstandardnonparametrictests AT kumarabhishek trialbaseddominanceforcomparingboththespeedandaccuracyofstochasticoptimizerswithstandardnonparametrictests AT suganthanponnuthurainagaratnam trialbaseddominanceforcomparingboththespeedandaccuracyofstochasticoptimizerswithstandardnonparametrictests |