Early Prediction of DNN Activation Using Hierarchical Computations
Deep Neural Networks (DNNs) have set state-of-the-art performance numbers in diverse fields of electronics (computer vision, voice recognition), biology, bioinformatics, etc. However, the process of learning (training) from the data and application of the learnt information (inference) process requi...
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2021-12-01
|
| Series: | Mathematics |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2227-7390/9/23/3130 |
| _version_ | 1797507415324753920 |
|---|---|
| author | Bharathwaj Suresh Kamlesh Pillai Gurpreet Singh Kalsi Avishaii Abuhatzera Sreenivas Subramoney |
| author_facet | Bharathwaj Suresh Kamlesh Pillai Gurpreet Singh Kalsi Avishaii Abuhatzera Sreenivas Subramoney |
| author_sort | Bharathwaj Suresh |
| collection | DOAJ |
| description | Deep Neural Networks (DNNs) have set state-of-the-art performance numbers in diverse fields of electronics (computer vision, voice recognition), biology, bioinformatics, etc. However, the process of learning (training) from the data and application of the learnt information (inference) process requires huge computational resources. Approximate computing is a common method to reduce computation cost, but it introduces loss in task accuracy, which limits their application. Using an inherent property of Rectified Linear Unit (ReLU), a popular activation function, we propose a mathematical model to perform MAC operation using reduced precision for predicting negative values early. We also propose a method to perform hierarchical computation to achieve the same results as IEEE754 full precision compute. Applying this method on ResNet50 and VGG16 shows that up to 80% of ReLU zeros (which is 50% of all ReLU outputs) can be predicted and detected early by using just 3 out of 23 mantissa bits. This method is equally applicable to other floating-point representations. |
| first_indexed | 2024-03-10T04:48:10Z |
| format | Article |
| id | doaj.art-36b06b4505eb475a85daf872434a9066 |
| institution | Directory Open Access Journal |
| issn | 2227-7390 |
| language | English |
| last_indexed | 2024-03-10T04:48:10Z |
| publishDate | 2021-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Mathematics |
| spelling | doaj.art-36b06b4505eb475a85daf872434a90662023-11-23T02:46:30ZengMDPI AGMathematics2227-73902021-12-01923313010.3390/math9233130Early Prediction of DNN Activation Using Hierarchical ComputationsBharathwaj Suresh0Kamlesh Pillai1Gurpreet Singh Kalsi2Avishaii Abuhatzera3Sreenivas Subramoney4Processor Architecture Research (PAR) Lab, Intel Labs, Bangalore 560048, IndiaProcessor Architecture Research (PAR) Lab, Intel Labs, Bangalore 560048, IndiaProcessor Architecture Research (PAR) Lab, Intel Labs, Bangalore 560048, IndiaCorporate Strategy Office, Intel, Haifa 3508409, IsraelProcessor Architecture Research (PAR) Lab, Intel Labs, Bangalore 560048, IndiaDeep Neural Networks (DNNs) have set state-of-the-art performance numbers in diverse fields of electronics (computer vision, voice recognition), biology, bioinformatics, etc. However, the process of learning (training) from the data and application of the learnt information (inference) process requires huge computational resources. Approximate computing is a common method to reduce computation cost, but it introduces loss in task accuracy, which limits their application. Using an inherent property of Rectified Linear Unit (ReLU), a popular activation function, we propose a mathematical model to perform MAC operation using reduced precision for predicting negative values early. We also propose a method to perform hierarchical computation to achieve the same results as IEEE754 full precision compute. Applying this method on ResNet50 and VGG16 shows that up to 80% of ReLU zeros (which is 50% of all ReLU outputs) can be predicted and detected early by using just 3 out of 23 mantissa bits. This method is equally applicable to other floating-point representations.https://www.mdpi.com/2227-7390/9/23/3130DNNReLUfloating-point numbershardware acceleration |
| spellingShingle | Bharathwaj Suresh Kamlesh Pillai Gurpreet Singh Kalsi Avishaii Abuhatzera Sreenivas Subramoney Early Prediction of DNN Activation Using Hierarchical Computations Mathematics DNN ReLU floating-point numbers hardware acceleration |
| title | Early Prediction of DNN Activation Using Hierarchical Computations |
| title_full | Early Prediction of DNN Activation Using Hierarchical Computations |
| title_fullStr | Early Prediction of DNN Activation Using Hierarchical Computations |
| title_full_unstemmed | Early Prediction of DNN Activation Using Hierarchical Computations |
| title_short | Early Prediction of DNN Activation Using Hierarchical Computations |
| title_sort | early prediction of dnn activation using hierarchical computations |
| topic | DNN ReLU floating-point numbers hardware acceleration |
| url | https://www.mdpi.com/2227-7390/9/23/3130 |
| work_keys_str_mv | AT bharathwajsuresh earlypredictionofdnnactivationusinghierarchicalcomputations AT kamleshpillai earlypredictionofdnnactivationusinghierarchicalcomputations AT gurpreetsinghkalsi earlypredictionofdnnactivationusinghierarchicalcomputations AT avishaiiabuhatzera earlypredictionofdnnactivationusinghierarchicalcomputations AT sreenivassubramoney earlypredictionofdnnactivationusinghierarchicalcomputations |