TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search
In the realm of machine learning, the optimization of hyperparameters and the design of neural architectures entail laborious and time-intensive endeavors. To address these challenges, considerable research effort has been directed towards Automated Machine Learning (AutoML), with a focus on enhanci...
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2023-09-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/13/18/10138 |
| _version_ | 1797581435245166592 |
|---|---|
| author | Jiyong Shin Kyongseok Park Dae-Ki Kang |
| author_facet | Jiyong Shin Kyongseok Park Dae-Ki Kang |
| author_sort | Jiyong Shin |
| collection | DOAJ |
| description | In the realm of machine learning, the optimization of hyperparameters and the design of neural architectures entail laborious and time-intensive endeavors. To address these challenges, considerable research effort has been directed towards Automated Machine Learning (AutoML), with a focus on enhancing these inherent inefficiencies. A pivotal facet of this pursuit is Neural Architecture Search (NAS), a domain dedicated to the automated formulation of neural network architectures. Given the pronounced impact of network architecture on neural network performance, NAS techniques strive to identify architectures that can manifest optimal performance outcomes. A prominent algorithm in this area is Differentiable Architecture Search (DARTS), which transforms discrete search spaces into continuous counterparts using gradient-based methodologies, thereby surpassing prior NAS methodologies. Notwithstanding DARTS’ achievements, a discrepancy between discrete and continuously encoded architectures persists. To ameliorate this disparity, we propose TA-DARTS in this study—a temperature annealing technique applied to the Softmax function, utilized for encoding the continuous search space. By leveraging temperature values, architectural weights are judiciously adjusted to alleviate biases in the search process or to align resulting architectures more closely with discrete values. Our findings exhibit advancements over the original DARTS methodology, evidenced by a 0.07%p enhancement in validation accuracy and a 0.16%p improvement in test accuracy on the CIFAR-100 dataset. Through systematic experimentation on benchmark datasets, we establish the superiority of TA-DARTS over the original mixed operator, thereby underscoring its efficacy in automating neural architecture design. |
| first_indexed | 2024-03-10T23:05:25Z |
| format | Article |
| id | doaj.art-5b8e5cdd07ce45b38f898a69e856c88f |
| institution | Directory Open Access Journal |
| issn | 2076-3417 |
| language | English |
| last_indexed | 2024-03-10T23:05:25Z |
| publishDate | 2023-09-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj.art-5b8e5cdd07ce45b38f898a69e856c88f2023-11-19T09:23:33ZengMDPI AGApplied Sciences2076-34172023-09-0113181013810.3390/app131810138TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture SearchJiyong Shin0Kyongseok Park1Dae-Ki Kang2Department of Computer Engineering, Dongseo University, 47 Jurye-ro, Sasang-gu, Busan 47011, Republic of KoreaSuper Computing Cloud Center, Korea Institute of Science and Technology Information, Daejeon 34141, Republic of KoreaDepartment of Computer Engineering, Dongseo University, 47 Jurye-ro, Sasang-gu, Busan 47011, Republic of KoreaIn the realm of machine learning, the optimization of hyperparameters and the design of neural architectures entail laborious and time-intensive endeavors. To address these challenges, considerable research effort has been directed towards Automated Machine Learning (AutoML), with a focus on enhancing these inherent inefficiencies. A pivotal facet of this pursuit is Neural Architecture Search (NAS), a domain dedicated to the automated formulation of neural network architectures. Given the pronounced impact of network architecture on neural network performance, NAS techniques strive to identify architectures that can manifest optimal performance outcomes. A prominent algorithm in this area is Differentiable Architecture Search (DARTS), which transforms discrete search spaces into continuous counterparts using gradient-based methodologies, thereby surpassing prior NAS methodologies. Notwithstanding DARTS’ achievements, a discrepancy between discrete and continuously encoded architectures persists. To ameliorate this disparity, we propose TA-DARTS in this study—a temperature annealing technique applied to the Softmax function, utilized for encoding the continuous search space. By leveraging temperature values, architectural weights are judiciously adjusted to alleviate biases in the search process or to align resulting architectures more closely with discrete values. Our findings exhibit advancements over the original DARTS methodology, evidenced by a 0.07%p enhancement in validation accuracy and a 0.16%p improvement in test accuracy on the CIFAR-100 dataset. Through systematic experimentation on benchmark datasets, we establish the superiority of TA-DARTS over the original mixed operator, thereby underscoring its efficacy in automating neural architecture design.https://www.mdpi.com/2076-3417/13/18/10138temperature annealingdiscrete operator distributiondifferential architecture search |
| spellingShingle | Jiyong Shin Kyongseok Park Dae-Ki Kang TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search Applied Sciences temperature annealing discrete operator distribution differential architecture search |
| title | TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search |
| title_full | TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search |
| title_fullStr | TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search |
| title_full_unstemmed | TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search |
| title_short | TA-DARTS: Temperature Annealing of Discrete Operator Distribution for Effective Differential Architecture Search |
| title_sort | ta darts temperature annealing of discrete operator distribution for effective differential architecture search |
| topic | temperature annealing discrete operator distribution differential architecture search |
| url | https://www.mdpi.com/2076-3417/13/18/10138 |
| work_keys_str_mv | AT jiyongshin tadartstemperatureannealingofdiscreteoperatordistributionforeffectivedifferentialarchitecturesearch AT kyongseokpark tadartstemperatureannealingofdiscreteoperatordistributionforeffectivedifferentialarchitecturesearch AT daekikang tadartstemperatureannealingofdiscreteoperatordistributionforeffectivedifferentialarchitecturesearch |