DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction Error
Deep belief networks (DBNs) of deep learning technology have been successfully used in many fields. However, the structure of a DBN is difficult to design for different datasets. Hence, a DBN structure design algorithm based on information entropy and reconstruction error is proposed. Unlike previou...
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2018-12-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/20/12/927 |
| _version_ | 1818007564795772928 |
|---|---|
| author | Jianjun Jiang Jing Zhang Lijia Zhang Xiaomin Ran Jun Jiang Yifan Wu |
| author_facet | Jianjun Jiang Jing Zhang Lijia Zhang Xiaomin Ran Jun Jiang Yifan Wu |
| author_sort | Jianjun Jiang |
| collection | DOAJ |
| description | Deep belief networks (DBNs) of deep learning technology have been successfully used in many fields. However, the structure of a DBN is difficult to design for different datasets. Hence, a DBN structure design algorithm based on information entropy and reconstruction error is proposed. Unlike previous algorithms, we innovatively combine network depth and node number and optimizes them simultaneously. First, the mathematical model of the structural design problem is established, and the boundary constraint for node number based on information entropy is derived by introducing the idea of information compression. Moreover, the optimization objective of the network performance based on reconstruction error is proposed by deriving the fact that network energy is proportional to reconstruction error. Finally, the improved simulated annealing (ISA) algorithm is used to adjust the DBN network layers and nodes simultaneously. Experiments were carried out on three public datasets (MNIST, Cifar-10 and Cifar-100). The results show that the proposed algorithm can design its proper structure to different datasets, yielding a trained DBN which has the lowest reconstruction error and prediction error rate. The proposed algorithm is shown to have the best performance compared with other algorithms and can be used to assist the setting of DBN structural parameters for different datasets. |
| first_indexed | 2024-04-14T05:17:19Z |
| format | Article |
| id | doaj.art-329d9e7dfc6e49de85c115c145491f75 |
| institution | Directory Open Access Journal |
| issn | 1099-4300 |
| language | English |
| last_indexed | 2024-04-14T05:17:19Z |
| publishDate | 2018-12-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj.art-329d9e7dfc6e49de85c115c145491f752022-12-22T02:10:19ZengMDPI AGEntropy1099-43002018-12-01201292710.3390/e20120927e20120927DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction ErrorJianjun Jiang0Jing Zhang1Lijia Zhang2Xiaomin Ran3Jun Jiang4Yifan Wu5National Digital Switching System Engineering and Technological Research Center (NDSC), Zhengzhou 450000, Henan, ChinaNational Digital Switching System Engineering and Technological Research Center (NDSC), Zhengzhou 450000, Henan, ChinaNational Digital Switching System Engineering and Technological Research Center (NDSC), Zhengzhou 450000, Henan, ChinaNational Digital Switching System Engineering and Technological Research Center (NDSC), Zhengzhou 450000, Henan, ChinaNational Digital Switching System Engineering and Technological Research Center (NDSC), Zhengzhou 450000, Henan, ChinaNational Digital Switching System Engineering and Technological Research Center (NDSC), Zhengzhou 450000, Henan, ChinaDeep belief networks (DBNs) of deep learning technology have been successfully used in many fields. However, the structure of a DBN is difficult to design for different datasets. Hence, a DBN structure design algorithm based on information entropy and reconstruction error is proposed. Unlike previous algorithms, we innovatively combine network depth and node number and optimizes them simultaneously. First, the mathematical model of the structural design problem is established, and the boundary constraint for node number based on information entropy is derived by introducing the idea of information compression. Moreover, the optimization objective of the network performance based on reconstruction error is proposed by deriving the fact that network energy is proportional to reconstruction error. Finally, the improved simulated annealing (ISA) algorithm is used to adjust the DBN network layers and nodes simultaneously. Experiments were carried out on three public datasets (MNIST, Cifar-10 and Cifar-100). The results show that the proposed algorithm can design its proper structure to different datasets, yielding a trained DBN which has the lowest reconstruction error and prediction error rate. The proposed algorithm is shown to have the best performance compared with other algorithms and can be used to assist the setting of DBN structural parameters for different datasets.https://www.mdpi.com/1099-4300/20/12/927deep learningDBNartificial intelligencestructure designinformation entropyreconstruction errorimproved simulated annealing algorithm |
| spellingShingle | Jianjun Jiang Jing Zhang Lijia Zhang Xiaomin Ran Jun Jiang Yifan Wu DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction Error Entropy deep learning DBN artificial intelligence structure design information entropy reconstruction error improved simulated annealing algorithm |
| title | DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction Error |
| title_full | DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction Error |
| title_fullStr | DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction Error |
| title_full_unstemmed | DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction Error |
| title_short | DBN Structure Design Algorithm for Different Datasets Based on Information Entropy and Reconstruction Error |
| title_sort | dbn structure design algorithm for different datasets based on information entropy and reconstruction error |
| topic | deep learning DBN artificial intelligence structure design information entropy reconstruction error improved simulated annealing algorithm |
| url | https://www.mdpi.com/1099-4300/20/12/927 |
| work_keys_str_mv | AT jianjunjiang dbnstructuredesignalgorithmfordifferentdatasetsbasedoninformationentropyandreconstructionerror AT jingzhang dbnstructuredesignalgorithmfordifferentdatasetsbasedoninformationentropyandreconstructionerror AT lijiazhang dbnstructuredesignalgorithmfordifferentdatasetsbasedoninformationentropyandreconstructionerror AT xiaominran dbnstructuredesignalgorithmfordifferentdatasetsbasedoninformationentropyandreconstructionerror AT junjiang dbnstructuredesignalgorithmfordifferentdatasetsbasedoninformationentropyandreconstructionerror AT yifanwu dbnstructuredesignalgorithmfordifferentdatasetsbasedoninformationentropyandreconstructionerror |