Neural Architecture Search and Hardware Accelerator Co-Search: A Survey
Deep neural networks (DNN) are now dominating in the most challenging applications of machine learning. As DNNs can have complex architectures with millions of trainable parameters (the so-called weights), their design and training are difficult even for highly qualified experts. In order to reduce...
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2021-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/9606893/ |
| _version_ | 1818740279616733184 |
|---|---|
| author | Lukas Sekanina |
| author_facet | Lukas Sekanina |
| author_sort | Lukas Sekanina |
| collection | DOAJ |
| description | Deep neural networks (DNN) are now dominating in the most challenging applications of machine learning. As DNNs can have complex architectures with millions of trainable parameters (the so-called weights), their design and training are difficult even for highly qualified experts. In order to reduce human effort, neural architecture search (NAS) methods have been developed to automate the entire design process. The NAS methods typically combine searching in the space of candidate architectures and optimizing (learning) the weights using a gradient method. In this paper, we survey the key elements of NAS methods that – to various extents – consider hardware implementation of the resulting DNNs. We classified these methods into three major classes: single-objective NAS (no hardware is considered), hardware-aware NAS (DNN is optimized for a particular hardware platform), and NAS with hardware co-optimization (hardware is directly co-optimized with DNN as a part of NAS). Compared to previous surveys, we emphasize the multi-objective design approach that must be adopted in NAS and focus on co-design algorithms developed for concurrent optimization of DNN architectures and hardware platforms. As most research in this area deals with NAS for image classification using convolutional neural networks, we follow this trajectory in our paper. After reading the paper, the reader should understand why and how NAS and hardware co-optimization are currently used to build cutting-edge implementations of DNNs. |
| first_indexed | 2024-12-18T01:38:12Z |
| format | Article |
| id | doaj.art-c44f1794327f4d478272a0619e4c1a9d |
| institution | Directory Open Access Journal |
| issn | 2169-3536 |
| language | English |
| last_indexed | 2024-12-18T01:38:12Z |
| publishDate | 2021-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj.art-c44f1794327f4d478272a0619e4c1a9d2022-12-21T21:25:24ZengIEEEIEEE Access2169-35362021-01-01915133715136210.1109/ACCESS.2021.31266859606893Neural Architecture Search and Hardware Accelerator Co-Search: A SurveyLukas Sekanina0https://orcid.org/0000-0002-2693-9011Faculty of Information Technology, Brno University of Technology, Brno, Czech RepublicDeep neural networks (DNN) are now dominating in the most challenging applications of machine learning. As DNNs can have complex architectures with millions of trainable parameters (the so-called weights), their design and training are difficult even for highly qualified experts. In order to reduce human effort, neural architecture search (NAS) methods have been developed to automate the entire design process. The NAS methods typically combine searching in the space of candidate architectures and optimizing (learning) the weights using a gradient method. In this paper, we survey the key elements of NAS methods that – to various extents – consider hardware implementation of the resulting DNNs. We classified these methods into three major classes: single-objective NAS (no hardware is considered), hardware-aware NAS (DNN is optimized for a particular hardware platform), and NAS with hardware co-optimization (hardware is directly co-optimized with DNN as a part of NAS). Compared to previous surveys, we emphasize the multi-objective design approach that must be adopted in NAS and focus on co-design algorithms developed for concurrent optimization of DNN architectures and hardware platforms. As most research in this area deals with NAS for image classification using convolutional neural networks, we follow this trajectory in our paper. After reading the paper, the reader should understand why and how NAS and hardware co-optimization are currently used to build cutting-edge implementations of DNNs.https://ieeexplore.ieee.org/document/9606893/Automated designclassificationco-designdeep neural networkhardware acceleratorneural architecture search |
| spellingShingle | Lukas Sekanina Neural Architecture Search and Hardware Accelerator Co-Search: A Survey IEEE Access Automated design classification co-design deep neural network hardware accelerator neural architecture search |
| title | Neural Architecture Search and Hardware Accelerator Co-Search: A Survey |
| title_full | Neural Architecture Search and Hardware Accelerator Co-Search: A Survey |
| title_fullStr | Neural Architecture Search and Hardware Accelerator Co-Search: A Survey |
| title_full_unstemmed | Neural Architecture Search and Hardware Accelerator Co-Search: A Survey |
| title_short | Neural Architecture Search and Hardware Accelerator Co-Search: A Survey |
| title_sort | neural architecture search and hardware accelerator co search a survey |
| topic | Automated design classification co-design deep neural network hardware accelerator neural architecture search |
| url | https://ieeexplore.ieee.org/document/9606893/ |
| work_keys_str_mv | AT lukassekanina neuralarchitecturesearchandhardwareacceleratorcosearchasurvey |