HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource Constraints
Online data streaming has become one of the most common data forms in the modern world, which imposes a strong demand for developing hyperparameter optimization techniques for online learning algorithms. In fact, processing online streaming data can be considered as a constraint on the training proc...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2020-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9060908/ |
| _version_ | 1828890454609887232 |
|---|---|
| author | Renlong Jie Junbin Gao Andrey Vasnev Minh-Ngoc Tran |
| author_facet | Renlong Jie Junbin Gao Andrey Vasnev Minh-Ngoc Tran |
| author_sort | Renlong Jie |
| collection | DOAJ |
| description | Online data streaming has become one of the most common data forms in the modern world, which imposes a strong demand for developing hyperparameter optimization techniques for online learning algorithms. In fact, processing online streaming data can be considered as a constraint on the training process. Existing studies do not provide a clear framework for modeling appropriate constraints with hyperparameter optimization in this context. In this paper, we propose a framework, called HyperTube, based on a set of assumptions that clearly define the constraints and objective function for online hyperparameter optimization under limited computing resources. We also introduce a “micro-mini-batch training mechanism” to reuse online data mini-batches in an efficient manner. Numerical experiments compare the performances of different training settings under the constraints of HyperTube. The results on stationary data streams without concept drift indicate that training on incremental data samples with model selection efficiently uses computing power and gives satisfactory validation performance compared with training without model selection. Meanwhile, the results on data streams with significant concept drift indicate that parallel updating could lead to relatively good model performance. In both cases, with the best settings, HyperTube with the micro-mini-batch training mechanism significantly outperforms offline random search with the same amount of computational resources. These settings can be further improved by a modified genetic algorithm. Finally, we develop a systematic method for selecting suitable settings based on a set of criteria. |
| first_indexed | 2024-12-13T12:59:19Z |
| format | Article |
| id | doaj.art-710a9f6c48c54e599ec13fc6032f767a |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-13T12:59:19Z |
| publishDate | 2020-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-710a9f6c48c54e599ec13fc6032f767a2022-12-21T23:45:04ZengIEEEIEEE Access2169-35362020-01-018690386905710.1109/ACCESS.2020.29864569060908HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource ConstraintsRenlong Jie0https://orcid.org/0000-0003-0569-5158Junbin Gao1Andrey Vasnev2Minh-Ngoc Tran3Discipline of Business Analytics, The University of Sydney Business School, The University of Sydney, Camperdown, NSW, AustraliaDiscipline of Business Analytics, The University of Sydney Business School, The University of Sydney, Camperdown, NSW, AustraliaDiscipline of Business Analytics, The University of Sydney Business School, The University of Sydney, Camperdown, NSW, AustraliaDiscipline of Business Analytics, The University of Sydney Business School, The University of Sydney, Camperdown, NSW, AustraliaOnline data streaming has become one of the most common data forms in the modern world, which imposes a strong demand for developing hyperparameter optimization techniques for online learning algorithms. In fact, processing online streaming data can be considered as a constraint on the training process. Existing studies do not provide a clear framework for modeling appropriate constraints with hyperparameter optimization in this context. In this paper, we propose a framework, called HyperTube, based on a set of assumptions that clearly define the constraints and objective function for online hyperparameter optimization under limited computing resources. We also introduce a “micro-mini-batch training mechanism” to reuse online data mini-batches in an efficient manner. Numerical experiments compare the performances of different training settings under the constraints of HyperTube. The results on stationary data streams without concept drift indicate that training on incremental data samples with model selection efficiently uses computing power and gives satisfactory validation performance compared with training without model selection. Meanwhile, the results on data streams with significant concept drift indicate that parallel updating could lead to relatively good model performance. In both cases, with the best settings, HyperTube with the micro-mini-batch training mechanism significantly outperforms offline random search with the same amount of computational resources. These settings can be further improved by a modified genetic algorithm. Finally, we develop a systematic method for selecting suitable settings based on a set of criteria.https://ieeexplore.ieee.org/document/9060908/Hyperparameter optimizationpopulation based traininghyperbandmodel selectiononline data stream |
| spellingShingle | Renlong Jie Junbin Gao Andrey Vasnev Minh-Ngoc Tran HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource Constraints IEEE Access Hyperparameter optimization population based training hyperband model selection online data stream |
| title | HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource Constraints |
| title_full | HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource Constraints |
| title_fullStr | HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource Constraints |
| title_full_unstemmed | HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource Constraints |
| title_short | HyperTube: A Framework for Population-Based Online Hyperparameter Optimization with Resource Constraints |
| title_sort | hypertube a framework for population based online hyperparameter optimization with resource constraints |
| topic | Hyperparameter optimization population based training hyperband model selection online data stream |
| url | https://ieeexplore.ieee.org/document/9060908/ |
| work_keys_str_mv | AT renlongjie hypertubeaframeworkforpopulationbasedonlinehyperparameteroptimizationwithresourceconstraints AT junbingao hypertubeaframeworkforpopulationbasedonlinehyperparameteroptimizationwithresourceconstraints AT andreyvasnev hypertubeaframeworkforpopulationbasedonlinehyperparameteroptimizationwithresourceconstraints AT minhngoctran hypertubeaframeworkforpopulationbasedonlinehyperparameteroptimizationwithresourceconstraints |