Towards a unified analysis of random fourier features
Random Fourier features is a widely used, simple, and effective technique for scaling up kernel methods. The existing theoretical analysis of the approach, however, remains focused on specific learning tasks and typically gives pessimistic bounds which are at odds with the empirical results. We tack...
| Main Authors: | , , , |
|---|---|
| Format: | Conference item |
| Published: |
ICML
2019
|
| _version_ | 1797076023992385536 |
|---|---|
| author | Li, Z Ton, J Oglic, D Sejdinovic, D |
| author_facet | Li, Z Ton, J Oglic, D Sejdinovic, D |
| author_sort | Li, Z |
| collection | OXFORD |
| description | Random Fourier features is a widely used, simple, and effective technique for scaling up kernel methods. The existing theoretical analysis of the approach, however, remains focused on specific learning tasks and typically gives pessimistic bounds which are at odds with the empirical results. We tackle these problems and provide the first unified risk analysis of learning with random Fourier features using the squared error and Lipschitz continuous loss functions. In our bounds, the trade-off between the computational cost and the expected risk convergence rate is problem specific and expressed in terms of the regularization parameter and the \emph{number of effective degrees of freedom}. We study both the standard random Fourier features method for which we improve the existing bounds on the number of features required to guarantee the corresponding minimax risk convergence rate of kernel ridge regression, as well as a data-dependent modification which samples features proportional to \emph{ridge leverage scores} and further reduces the required number of features. As ridge leverage scores are expensive to compute, we devise a simple approximation scheme which provably reduces the computational cost without loss of statistical efficiency. |
| first_indexed | 2024-03-06T23:58:19Z |
| format | Conference item |
| id | oxford-uuid:7505f2cf-0a2f-497c-b456-d3ccc6de8c6e |
| institution | University of Oxford |
| last_indexed | 2024-03-06T23:58:19Z |
| publishDate | 2019 |
| publisher | ICML |
| record_format | dspace |
| spelling | oxford-uuid:7505f2cf-0a2f-497c-b456-d3ccc6de8c6e2022-03-26T20:06:49ZTowards a unified analysis of random fourier featuresConference itemhttp://purl.org/coar/resource_type/c_5794uuid:7505f2cf-0a2f-497c-b456-d3ccc6de8c6eSymplectic Elements at OxfordICML2019Li, ZTon, JOglic, DSejdinovic, DRandom Fourier features is a widely used, simple, and effective technique for scaling up kernel methods. The existing theoretical analysis of the approach, however, remains focused on specific learning tasks and typically gives pessimistic bounds which are at odds with the empirical results. We tackle these problems and provide the first unified risk analysis of learning with random Fourier features using the squared error and Lipschitz continuous loss functions. In our bounds, the trade-off between the computational cost and the expected risk convergence rate is problem specific and expressed in terms of the regularization parameter and the \emph{number of effective degrees of freedom}. We study both the standard random Fourier features method for which we improve the existing bounds on the number of features required to guarantee the corresponding minimax risk convergence rate of kernel ridge regression, as well as a data-dependent modification which samples features proportional to \emph{ridge leverage scores} and further reduces the required number of features. As ridge leverage scores are expensive to compute, we devise a simple approximation scheme which provably reduces the computational cost without loss of statistical efficiency. |
| spellingShingle | Li, Z Ton, J Oglic, D Sejdinovic, D Towards a unified analysis of random fourier features |
| title | Towards a unified analysis of random fourier features |
| title_full | Towards a unified analysis of random fourier features |
| title_fullStr | Towards a unified analysis of random fourier features |
| title_full_unstemmed | Towards a unified analysis of random fourier features |
| title_short | Towards a unified analysis of random fourier features |
| title_sort | towards a unified analysis of random fourier features |
| work_keys_str_mv | AT liz towardsaunifiedanalysisofrandomfourierfeatures AT tonj towardsaunifiedanalysisofrandomfourierfeatures AT oglicd towardsaunifiedanalysisofrandomfourierfeatures AT sejdinovicd towardsaunifiedanalysisofrandomfourierfeatures |