Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper−Parameters
Bayesian optimisation has gained great popularity as a tool for optimising the parameters of machine learning algorithms and models. Somewhat ironically, setting up the hyper-parameters of Bayesian optimisation methods is notoriously hard. While reasonable practical solutions have been advanced, the...
| Main Authors: | , |
|---|---|
| Format: | Report |
| Published: |
University of Oxford
2014
|
| _version_ | 1797061038283161600 |
|---|---|
| author | Wang, Z de Freitas, N |
| author_facet | Wang, Z de Freitas, N |
| author_sort | Wang, Z |
| collection | OXFORD |
| description | Bayesian optimisation has gained great popularity as a tool for optimising the parameters of machine learning algorithms and models. Somewhat ironically, setting up the hyper-parameters of Bayesian optimisation methods is notoriously hard. While reasonable practical solutions have been advanced, they can often fail to find the best optima. Surprisingly, there is little theoretical analysis of this crucial problem in the literature. To address this, we derive a cumulative regret bound for Bayesian optimisation with Gaussian processes and unknown kernel hyper-parameters in the stochastic setting. The bound, which applies to the expected improvement acquisition function and sub-Gaussian observation noise, provides us with guidelines on how to design hyper-parameter estimation methods. A simple simulation demonstrates the importance of following these guidelines. |
| first_indexed | 2024-03-06T20:25:23Z |
| format | Report |
| id | oxford-uuid:2f35eb0d-13dd-47ae-8bc1-6d0b64f97e91 |
| institution | University of Oxford |
| last_indexed | 2024-03-06T20:25:23Z |
| publishDate | 2014 |
| publisher | University of Oxford |
| record_format | dspace |
| spelling | oxford-uuid:2f35eb0d-13dd-47ae-8bc1-6d0b64f97e912022-03-26T12:54:02ZTheoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper−ParametersReporthttp://purl.org/coar/resource_type/c_93fcuuid:2f35eb0d-13dd-47ae-8bc1-6d0b64f97e91Department of Computer ScienceUniversity of Oxford2014Wang, Zde Freitas, NBayesian optimisation has gained great popularity as a tool for optimising the parameters of machine learning algorithms and models. Somewhat ironically, setting up the hyper-parameters of Bayesian optimisation methods is notoriously hard. While reasonable practical solutions have been advanced, they can often fail to find the best optima. Surprisingly, there is little theoretical analysis of this crucial problem in the literature. To address this, we derive a cumulative regret bound for Bayesian optimisation with Gaussian processes and unknown kernel hyper-parameters in the stochastic setting. The bound, which applies to the expected improvement acquisition function and sub-Gaussian observation noise, provides us with guidelines on how to design hyper-parameter estimation methods. A simple simulation demonstrates the importance of following these guidelines. |
| spellingShingle | Wang, Z de Freitas, N Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper−Parameters |
| title | Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper−Parameters |
| title_full | Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper−Parameters |
| title_fullStr | Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper−Parameters |
| title_full_unstemmed | Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper−Parameters |
| title_short | Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper−Parameters |
| title_sort | theoretical analysis of bayesian optimisation with unknown gaussian process hyper parameters |
| work_keys_str_mv | AT wangz theoreticalanalysisofbayesianoptimisationwithunknowngaussianprocesshyperparameters AT defreitasn theoreticalanalysisofbayesianoptimisationwithunknowngaussianprocesshyperparameters |